BN中的滑动平均是怎么做的
训练过程中的每一个batch都会进行滑动平均的计算[1]：
moving_mean = moving_mean * momentum + batch_mean * (1 - momentum)
moving_var = moving_var * momentum + batch_var * (1 - momentum)
式中的 momentum 为动量参数，在 TF/Keras 中，该值为0.99，在 Pytorch 中，这个值为0.9
初始值，moving_mean=0，moving_var=1，相当于标准正态分布，当然，理论上初始化为任意值都可以
在实际的代码中，滑动平均的计算会以一种更高效的方式，但实际上是等价的：
moving_mean -= (moving_mean - batch_mean) * (1 - momentum)
moving_var -= (moving_var - batch_var) * (1 - momentum)
滑动平均中 Momentum 参数的影响
整个训练阶段滑动平均的过程，(moving_mean, moving_var) 参数实际上是从正态分布，向训练集真实分布靠拢的一个过程。
理论上，训练步数越长是会越靠近真实分布的，实际上，因为每个batch并不能代表整个训练集的分布，所以最后的值是在真实分布附近波动。
一个更小的 momentum 值，意味着更大的更新步长，对应着滑动平均值更快的变化，能更快地向真实值靠拢，但也意味着更大的波动性，更大的 momentum 值则相反。
训练阶段使用的是 (batch_mean, batch_var)，所以滑动平均并不会影响训练阶段的结果，而是影响预测阶段的效果。关于BN在训练和测试时的差别可参考[2] 。
如果训练步数很短，一个大的 momentum 值可能会导致 (moving_mean, moving_var) 还没有靠拢到真实分布就停止了，这样对预测阶段的影响是很大的，也会是欠拟合的一个状态。如果训练步数足够，一个大的 momentum 值对应小的更新步长，最后的滑动平均的值是会更接近真实值的。
如果batch size 比较小，那单个batch的 (batch_mean, batch_var) 和真实分布会比较大，此时滑动平均单次更新的步长就不应过大，适用一个大的 momentum 值，反之可类比分析。
BN 前向过程代码实现
def batchnorm_forward(x, gamma, beta, bn_param):
"""
Forward pass for batch normalization.
During training the sample mean and (uncorrected) sample variance are
computed from minibatch statistics and used to normalize the incoming data.
During training we also keep an exponentially decaying running mean of the
mean and variance of each feature, and these averages are used to normalize
data at test-time.
At each timestep we update the running averages for mean and variance using
an exponential decay based on the momentum parameter:
running_mean = momentum * running_mean + (1 - momentum) * sample_mean
running_var = momentum * running_var + (1 - momentum) * sample_var
Input:
- x: Data of shape (N, D)
- gamma: Scale parameter of shape (D,)
- beta: Shift paremeter of shape (D,)
- bn_param: Dictionary with the following keys:
- mode: 'train' or 'test'; required
- eps: Constant for numeric stability
- momentum: Constant for running mean / variance.
- running_mean: Array of shape (D,) giving running mean of features
- running_var Array of shape (D,) giving running variance of features
Returns a tuple of:
- out: of shape (N, D)
- cache: A tuple of values needed in the backward pass
"""
mode = bn_param['mode']
eps = bn_param.get('eps', 1e-5)
momentum = bn_param.get('momentum', 0.9)
N, D = x.shape
running_mean = bn_param.get('running_mean', np.zeros(D, dtype=x.dtype))
running_var = bn_param.get('running_var', np.ones(D, dtype=x.dtype))
if mode == 'train':
sample_mean = x.mean(axis=0)
sample_var = x.var(axis=0)
running_mean = momentum * running_mean + (1 - momentum) * sample_mean
running_var = momentum * running_var + (1 - momentum) * sample_var
std = np.sqrt(sample_var + eps)
x_centered = x - sample_mean
x_norm = x_centered / std
out = gamma * x_norm + beta
cache = (x_norm, x_centered, std, gamma)
elif mode == 'test':
x_norm = (x - running_mean) / np.sqrt(running_var + eps)
out = gamma * x_norm + beta
else:
raise ValueError('Invalid forward batchnorm mode "%s"' % mode)
# Store the updated running means back into bn_param
bn_param['running_mean'] = running_mean
bn_param['running_var'] = running_var
return out, cache
注：代码参考[3]，原代码中 running_var 也初始化是 np.zeros，本文做了修改。
running_mean和 running_var 的初始值为正态分布参数值，可参考 Pytorch 代码中的 _NormBase 类
参考：
[1] https://jiafulow.github.io/blog/2021/01/29/moving-average-in-batch-normalization/
[2] https://zhuanlan.zhihu.com/p/61725100
[3] https://towardsdatascience.com/implementing-batch-normalization-in-python-a044b0369567
[4] 题图参考：https://kaixih.github.io/batch-norm/
