线性回归、Lasso回归、岭回归、逻辑回归的损失函数
线性回归:
J
(
θ
)
=
1
2
m
∑
i
=
1
m
(
h
(
x
(
i
)
)
?
y
(
i
)
)
2
J(\theta)=\frac{1}{2m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})^2
J(θ)=2m1?∑i=1m?(h(x(i))?y(i))2
Lasso回归：
J
(
θ
)
=
1
2
m
∑
i
=
1
m
(
h
(
x
(
i
)
)
?
y
(
i
)
)
2
+
λ
∑
j
=
1
n
∣
θ
∣
J(\theta)=\frac{1}{2m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})^2+\lambda\sum_{j=1}^{n}|\theta|
J(θ)=2m1?∑i=1m?(h(x(i))?y(i))2+λ∑j=1n?∣θ∣
岭回归：
J
(
θ
)
=
1
2
m
∑
i
=
1
m
(
h
(
x
(
i
)
)
?
y
(
i
)
)
2
+
λ
∑
j
=
1
n
θ
2
J(\theta)=\frac{1}{2m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})^2+\lambda\sum_{j=1}^{n}\theta^2
J(θ)=2m1?∑i=1m?(h(x(i))?y(i))2+λ∑j=1n?θ2
LR：
J
(
θ
)
=
?
1
m
∑
i
=
1
m
[
(
1
?
y
(
i
)
)
l
o
g
(
1
?
h
(
x
(
i
)
)
)
+
y
(
i
)
l
o
g
(
h
(
x
(
i
)
)
)
]
J(\theta)=-\frac{1}{m}\sum_{i=1}^m[(1-y^{(i)})log(1-h(x^{(i)}))+y^{(i)}log(h(x^{(i)}))]
J(θ)=?m1?∑i=1m?[(1?y(i))log(1?h(x(i)))+y(i)log(h(x(i)))]
推导LR
LR的损失函数推导
根据sigmoid函数的定义，
P
(
y
=
1
∣
x
,
θ
)
=
h
(
x
)
P(y=1|x,\theta)=h(x)
P(y=1∣x,θ)=h(x)，
P
(
y
=
0
∣
x
,
θ
)
=
1
?
h
(
x
)
P(y=0|x,\theta)=1-h(x)
P(y=0∣x,θ)=1?h(x)
因此，
P
(
y
∣
x
,
θ
)
=
h
(
x
)
y
[
1
?
h
(
x
)
]
1
?
y
P(y|x,\theta)=h(x)^y[1-h(x)]^{1-y}
P(y∣x,θ)=h(x)y[1?h(x)]1?y。
目标是最大化
P
(
y
∣
x
)
P(y|x)
P(y∣x)，即最大化其对数。
令似然函数L=
P
(
y
∣
x
)
P(y|x)
P(y∣x)，
l
n
L
=
y
l
o
g
(
h
(
x
)
+
(
1
?
y
)
l
o
g
(
1
?
h
(
x
)
)
lnL=ylog(h(x)+(1-y)log(1-h(x))
lnL=ylog(h(x)+(1?y)log(1?h(x))。
损失函数求最小化，注意加负号：
J
(
θ
)
=
?
1
m
∑
i
=
1
m
[
y
(
i
)
l
o
g
(
h
(
x
(
i
)
)
)
+
(
1
?
y
(
i
)
)
l
o
g
(
1
?
h
(
x
(
i
)
)
)
]
J(\theta)=-\frac{1}{m}\sum_{i=1}^m[y^{(i)}log(h(x^{(i)}))+(1-y^{(i)})log(1-h(x^{(i)}))]
J(θ)=?m1?∑i=1m?[y(i)log(h(x(i)))+(1?y(i))log(1?h(x(i)))]
LR的导数推导
对损失函数
J
(
θ
)
=
?
1
m
∑
i
=
1
m
[
y
(
i
)
l
o
g
(
h
(
x
(
i
)
)
)
+
(
1
?
y
(
i
)
)
l
o
g
(
1
?
h
(
x
(
i
)
)
)
]
J(\theta)=-\frac{1}{m}\sum_{i=1}^m[y^{(i)}log(h(x^{(i)}))+(1-y^{(i)})log(1-h(x^{(i)}))]
J(θ)=?m1?∑i=1m?[y(i)log(h(x(i)))+(1?y(i))log(1?h(x(i)))]求导：
J
′
(
θ
j
)
=
?
1
m
∑
i
=
1
m
[
y
(
i
)
h
θ
′
(
x
(
i
)
)
h
(
x
(
i
)
)
+
(
1
?
y
(
i
)
)
h
θ
′
(
x
(
i
)
)
1
?
h
(
x
(
i
)
)
]
=
?
1
m
∑
i
=
1
m
[
y
(
i
)
h
(
x
(
i
)
)
[
1
?
h
(
x
(
i
)
)
]
(
x
j
(
i
)
)
h
(
x
(
i
)
)
+
(
1
?
y
(
i
)
)
h
(
x
(
i
)
)
[
1
?
h
(
x
(
i
)
)
]
(
x
j
(
i
)
)
1
?
h
(
x
(
i
)
)
]
=
?
1
m
∑
i
=
1
m
[
(
y
(
i
)
?
h
(
x
(
i
)
)
)
x
j
(
i
)
]
\begin{aligned} J'(\theta_j)&=-\frac{1}{m}\sum_{i=1}^m[y^{(i)}\frac{h'_{\theta}(x^{(i)})}{h(x^{(i)})}+(1-y^{(i)})\frac{h'_{\theta}(x^{(i)})}{1-h(x^{(i)})}]\\[2ex] &=-\frac{1}{m}\sum_{i=1}^m[y^{(i)}\frac{h(x^{(i)})[1-h(x^{(i)})](x_j^{(i)})}{h(x^{(i)})}+(1-y^{(i)})\frac{h(x^{(i)})[1-h(x^{(i)})](x_j^{(i)})}{1-h(x^{(i)})}]\\[2ex] &=-\frac{1}{m}\sum_{i=1}^m[(y^{(i)}-h(x^{(i)}))x_j^{(i)}] \end{aligned}
J′(θj?)?=?m1?i=1∑m?[y(i)h(x(i))hθ′?(x(i))?+(1?y(i))1?h(x(i))hθ′?(x(i))?]=?m1?i=1∑m?[y(i)h(x(i))h(x(i))[1?h(x(i))](xj(i)?)?+(1?y(i))1?h(x(i))h(x(i))[1?h(x(i))](xj(i)?)?]=?m1?i=1∑m?[(y(i)?h(x(i)))xj(i)?]?
θ
j
=
θ
j
+
α
1
m
∑
i
=
1
m
[
(
y
(
i
)
?
h
(
x
(
i
)
)
)
x
j
(
i
)
]
\theta_j=\theta_j+\alpha\frac{1}{m}\sum_{i=1}^m[(y^{(i)}-h(x^{(i)}))x_j^{(i)}]
θj?=θj?+αm1?∑i=1m?[(y(i)?h(x(i)))xj(i)?]
对比线性回归对参数的导数：
J
′
(
θ
j
)
=
1
m
∑
i
=
1
m
(
h
(
x
(
i
)
)
?
y
(
i
)
)
x
j
(
i
)
J'(\theta_j)=\frac{1}{m}\sum_{i=1}^m(h(x^{(i)})-y^{(i)})x_j^{(i)}
J′(θj?)=m1?∑i=1m?(h(x(i))?y(i))xj(i)?
θ
j
=
θ
j
+
α
1
m
∑
i
=
1
m
[
(
y
(
i
)
?
h
(
x
(
i
)
)
)
x
j
(
i
)
]
\theta_j=\theta_j+\alpha\frac{1}{m}\sum_{i=1}^m[(y^{(i)}-h(x^{(i)}))x_j^{(i)}]
θj?=θj?+αm1?∑i=1m?[(y(i)?h(x(i)))xj(i)?]
可以发现二者虽然损失函数不同，但导数和梯度下降的公式却是相同的（神奇）
|