指数加权平均值Exponentially weighted averages

$v_0 = 0\\ v_1 = \beta v_0 + (1-\beta)\theta_1\\v_t = \beta v_{t-1} + (1-\beta)\theta_{t}$
$其中v表示第t个平均值，\theta_t表示第t个值，\beta\\ 是需要选定的超参数，大致可以理解为我们需要\\考虑\frac{1}{1-\beta}个之前的值的影响$
在机器学习和深度学习中,如果某个值既需要考虑以往的影响，又需要结合当前的价值，那么可以考虑使用加权平均数。
可以将 $v$ 想象为速度，将 $\theta$ 想象为加速度，当前的速度受到前一时间的速度和当前的加速度的影响。
在这里插入图片描述

上图是对温度和天气使用指数加权平均数进行预测。其中红色的 $\beta_1$ 选用的值更低，绿色的 $\beta_2$ 更高
假定 $\beta=0.2$
$v_1 = 0.2\theta_1\\ v_2 = 0.16\theta_1 + 0.2\theta_2$

可以看出上述平均值在 $t$ 很小的时候，该平均值不能很好拟合原模型，于是引入修正方法
$v_0 = 0\\v_1^{corrected} = \frac{v_1}{1-\beta^{1}}\\...\\v_t^{corrected} = \frac{v_t}{1-\beta^t}$

在之后将要介绍的三种优化器都是使用上述的类似方法进行优化。

带动量的梯度下降方法（Gradient Descent with Momentum）

相比于传统的梯度下降算法，这个新的算法引入了动量，
我们有
$para_{new} = para_{old} - \alpha grad\\其中\alpha为学习率。$
而动量采用了我们前面所谈到的指数加权平均数
$第t次迭代的更新公式为：\\ para_{new} = para_{old} - \alpha v_t\\ v_t = \beta v_{t-1} + (1-\beta) grad_t\\$
$其中\alpha为学习率，\beta是超参数，grad_t是第t才的梯度。\\\alpha往往会比我们在传统梯度下降时指定的学习率更低。$
在这里插入图片描述

# GRADED FUNCTION: update_parameters_with_momentum

def update_parameters_with_momentum(parameters, grads, v, beta, learning_rate):
    """
    Update parameters using Momentum
    
    Arguments:
    parameters -- python dictionary containing your parameters:
                    parameters['W' + str(l)] = Wl
                    parameters['b' + str(l)] = bl
    grads -- python dictionary containing your gradients for each parameters:
                    grads['dW' + str(l)] = dWl
                    grads['db' + str(l)] = dbl
    v -- python dictionary containing the current velocity:
                    v['dW' + str(l)] = ...
                    v['db' + str(l)] = ...
    beta -- the momentum hyperparameter, scalar
    learning_rate -- the learning rate, scalar
    
    Returns:
    parameters -- python dictionary containing your updated parameters 
    v -- python dictionary containing your updated velocities
    """

    L = len(parameters) // 2 # number of layers in the neural networks
    
    # Momentum update for each parameter
    for l in range(1, L + 1):
        
        # (approx. 4 lines)
        # compute velocities
        # v["dW" + str(l)] = ...
        # v["db" + str(l)] = ...
        # update parameters
        # parameters["W" + str(l)] = ...
        # parameters["b" + str(l)] = ...
        # YOUR CODE STARTS HERE
        v['dW'+str(l)] = beta*v['dW'+str(l)] + (1-beta)*grads['dW'+str(l)]
        v['db'+str(l)] = beta*v['db'+str(l)] +(1-beta)*grads['db'+str(l)]
        parameters['W'+str(l)] = parameters['W'+str(l)]-learning_rate*v['dW'+str(l)]
        parameters['b'+str(l)] = parameters['b'+str(l)]-learning_rate*v['db'+str(l)]
        # YOUR CODE ENDS HERE
        
    return parameters, v

RMSprop(Root mean sqare propagate)

相比于直接使用速度代替梯度进行计算，RMSprop采用一种类似累计均方的方法。
$第t次迭代：\\ S_0 = 0\\S_1 = \beta S_0 + (1-\beta )grad^2_1\\...\\S_t = \beta S_{t-1} + (1-\beta)grad^2_t\\para_{new} = para_{old} -\alpha \frac{grad}{\sqrt{S_t + \epsilon}}$
事实上RMSprop用处相比于Momentum和Adam，范围和用途都比较少。

Adam

Adam的方法相当于上述两个方法的结合体，同时使用了v和s。
$第t次迭代：\\ v_t = \beta_1 v_{t-1}+(1-\beta_1)grad_t\\ v_t^{corrected} = \frac{v_t}{1-\beta_1^t}\\ \\S_t = \beta_2 S_{t-1} + (1-\beta_2)grad_t\\ \\S_t^{correctd} = \frac{S_t}{1-\beta_2^t}\\ \\para_{new} = para_{old} - \alpha \frac{v_t^{correctd}}{S_t^{correted}}$

# GRADED FUNCTION: update_parameters_with_adam

def update_parameters_with_adam(parameters, grads, v, s, t, learning_rate = 0.01,
                                beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8):
    """
    Update parameters using Adam
    
    Arguments:
    parameters -- python dictionary containing your parameters:
                    parameters['W' + str(l)] = Wl
                    parameters['b' + str(l)] = bl
    grads -- python dictionary containing your gradients for each parameters:
                    grads['dW' + str(l)] = dWl
                    grads['db' + str(l)] = dbl
    v -- Adam variable, moving average of the first gradient, python dictionary
    s -- Adam variable, moving average of the squared gradient, python dictionary
    t -- Adam variable, counts the number of taken steps
    learning_rate -- the learning rate, scalar.
    beta1 -- Exponential decay hyperparameter for the first moment estimates 
    beta2 -- Exponential decay hyperparameter for the second moment estimates 
    epsilon -- hyperparameter preventing division by zero in Adam updates

    Returns:
    parameters -- python dictionary containing your updated parameters 
    v -- Adam variable, moving average of the first gradient, python dictionary
    s -- Adam variable, moving average of the squared gradient, python dictionary
    """
    
    L = len(parameters) // 2                 # number of layers in the neural networks
    v_corrected = {}                         # Initializing first moment estimate, python dictionary
    s_corrected = {}                         # Initializing second moment estimate, python dictionary
    
    # Perform Adam update on all parameters
    for l in range(1, L + 1):
        # Moving average of the gradients. Inputs: "v, grads, beta1". Output: "v".
        # (approx. 2 lines)
        # v["dW" + str(l)] = ...
        # v["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        v['dW'+str(l)] = beta1*v['dW'+str(l)]+(1-beta1)*grads['dW'+str(l)]
        v['db'+str(l)] = beta1*v['db'+str(l)]+(1-beta1)*grads['db'+str(l)]
        
        # YOUR CODE ENDS HERE

        # Compute bias-corrected first moment estimate. Inputs: "v, beta1, t". Output: "v_corrected".
        # (approx. 2 lines)
        # v_corrected["dW" + str(l)] = ...
        # v_corrected["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        v_corrected['dW'+str(l)]=v['dW'+str(l)]/(1-beta1**t)
        v_corrected['db'+str(l)]=v['db'+str(l)]/(1-beta1**t)
        # YOUR CODE ENDS HERE

        # Moving average of the squared gradients. Inputs: "s, grads, beta2". Output: "s".
        #(approx. 2 lines)
        # s["dW" + str(l)] = ...
        # s["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        s['dW'+str(l)]=beta2*s['dW'+str(l)]+(1-beta2)*grads['dW'+str(l)]**2
        s['db'+str(l)]=beta2*s['db'+str(l)]+(1-beta2)*grads['db'+str(l)]**2
        
        # YOUR CODE ENDS HERE

        # Compute bias-corrected second raw moment estimate. Inputs: "s, beta2, t". Output: "s_corrected".
        # (approx. 2 lines)
        # s_corrected["dW" + str(l)] = ...
        # s_corrected["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        s_corrected['dW'+str(l)]=s['dW'+str(l)]/(1-beta2**t)
        s_corrected['db'+str(l)]=s['db'+str(l)]/(1-beta2**t)
        
        # YOUR CODE ENDS HERE

        # Update parameters. Inputs: "parameters, learning_rate, v_corrected, s_corrected, epsilon". Output: "parameters".
        # (approx. 2 lines)
        # parameters["W" + str(l)] = ...
        # parameters["b" + str(l)] = ...
        # YOUR CODE STARTS HERE
        parameters['W'+str(l)]=parameters['W'+str(l)]-learning_rate*v_corrected['dW'+str(l)]/(np.sqrt(s_corrected['dW'+str(l)])+epsilon)
        parameters['b'+str(l)]=parameters['b'+str(l)]-learning_rate*v_corrected['db'+str(l)]/(np.sqrt(s_corrected['db'+str(l)])+epsilon)
        
        # YOUR CODE ENDS HERE

    return parameters, v, s, v_corrected, s_corrected

上诉的几种优化方法不仅可以加快训练速度，并且还可以一定程度上避免陷入局部最优值（可以想象为一个具有滑雪选手可以冲破一些较小的凹处、山谷，从而达到终点）

学习率下降（Learning Rate Decay）

在算法达到快要最优值得时候，我们会发现优化器会在最优值周围徘徊，因为学习率的设置，我们设置的步长往往太大了，然而初始设置步长很小又会导致速度便面，这个使用我们就可以采用一种很简单的方法使得学习率下降。
$\alpha_t = f(t)* \alpha_0\\ \\f是一个单调下降的函数$
有时为了防止学习率降为0，或者经常更新导致的速度问题， $f 函数$ 常常设置为阶段函数。
在这里插入图片描述