有关学习率的调整方式

当我们定义好优化器后，有关学习率的调整方式是比较头大的一个问题，除了自己手动定义函数来自定义的调整学习率的方法外，pytorch的optim库也提供了许多便捷的动态学习率的调整方式。

torch.optim.lr_scheduler提供了一些基于epoch数的学习率调整方法。
torch.optim.lr_scheduler.ReduceLROnPlateau 也允许基于模型的实时结果来动态调整学习率。

一、学习率的调整模板

常见模板1：

model = [Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = SGD(model, 0.1)
scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9)

for epoch in range(20):
    for input, target in dataset:
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
    scheduler.step()

常见模板2：

一次不仅可以只应用一个scheduler，也可以使用多个scheduler，如下所示：

model = [Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = SGD(model, 0.1)
scheduler1 = ExponentialLR(optimizer, gamma=0.9)
scheduler2 = MultiStepLR(optimizer, milestones=[30,80], gamma=0.1)

for epoch in range(20):
    for input, target in dataset:
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
    scheduler1.step()
    scheduler2.step()

通用模板：

scheduler = ...
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

二、自适应调整学习率方式 - ReduceLROnPlateau

这里讲一个比较实用的学习率调整方式。当某指标不再变化（下降或升高）时，再调整学习率，这是非常实用的学习率调整策略。
例如，当验证集的 loss 不再下降时，进行学习率调整；或者监测验证集的 accuracy，当accuracy 不再上升时，则调整学习率

torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=10, verbose=False, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0, eps=1e-08)

参数：

mode(str)- 模式选择，有 min 和 max 两种模式， min 表示当指标不再降低(如监测loss)， max 表示当指标不再升高(如监测 accuracy)。Default：min
factor(float)- 学习率调整倍数(等同于其它方法的 gamma)，即学习率更新为 new_lr = lr * factor。Default: 0.1
patience(int)- 忍受该指标多少个 step 不变化，当忍无可忍时，调整学习率。Default :10
verbose(bool)- 是否打印学习率信息， print(‘Epoch {:5d}: reducing learning rate of group {} to {:.4e}.’.format(epoch, i, new_lr))
threshold_mode(str)- 选择判断指标是否达最优的模式，有两种模式， rel 和 abs。
当 threshold_mode == rel，并且 mode == max 时， dynamic_threshold = best * ( 1 +threshold )；
当 threshold_mode == rel，并且 mode == min 时， dynamic_threshold = best * ( 1 -threshold )；
当 threshold_mode == abs，并且 mode== max 时， dynamic_threshold = best + threshold ；
当 threshold_mode == abs，并且 mode == max 时， dynamic_threshold = best - threshold； Default: ‘rel’.
threshold(float)- 配合 threshold_mode 使用。衡量新优化的阈值，只关注重大变化。Default: 1e-4.
cooldown(int)- “冷却时间“，当调整学习率之后，让学习率调整策略冷静一下，让模型再训练一段时间，再重启监测模式。Default: 0.
min_lr(float or list)- 学习率下限，可为 float，或者 list，当有多个参数组时，可用 list 进行设置。Default: 0.
eps(float)- 学习率衰减的最小值，当学习率变化小于 eps 时，则不调整学习率。Default: 1e-8.

参考

Pytorch官方
 学习率调整

附录：pytorch常见学习率调整函数：

lr_scheduler.LambdaLR

Sets the learning rate of each parameter group to the initial lr times a given function.

lr_scheduler.MultiplicativeLR

Multiply the learning rate of each parameter group by the factor given in the specified function.

lr_scheduler.StepLR

Decays the learning rate of each parameter group by gamma every step_size epochs.

lr_scheduler.MultiStepLR

Decays the learning rate of each parameter group by gamma once the number of epoch reaches one of the milestones.

lr_scheduler.ConstantLR

Decays the learning rate of each parameter group by a small constant factor until the number of epoch reaches a pre-defined milestone: total_iters.

lr_scheduler.LinearLR

Decays the learning rate of each parameter group by linearly changing small multiplicative factor until the number of epoch reaches a pre-defined milestone: total_iters.

lr_scheduler.ExponentialLR

Decays the learning rate of each parameter group by gamma every epoch.

lr_scheduler.CosineAnnealingLR

Set the learning rate of each parameter group using a cosine annealing schedule, where $\eta_{max}$ is set to the initial lr and $T_{cur}$ is the number of epochs since the last restart in SGDR:

lr_scheduler.ChainedScheduler

Chains list of learning rate schedulers.

lr_scheduler.SequentialLR

Receives the list of schedulers that is expected to be called sequentially during optimization process and milestone points that provides exact intervals to reflect which scheduler is supposed to be called at a given epoch.

lr_scheduler.ReduceLROnPlateau

Reduce learning rate when a metric has stopped improving.

lr_scheduler.CyclicLR

Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR).

lr_scheduler.OneCycleLR

Sets the learning rate of each parameter group according to the 1cycle learning rate policy.

lr_scheduler.CosineAnnealingWarmRestarts

Set the learning rate of each parameter group using a cosine annealing schedule, where $\eta_{max}$ is set to the initial lr, $T_{cur}$ is the number of epochs since the last restart and $T_{i}$ is the number of epochs between two warm restarts in SGDR: