|
最近在用L1loss做一个回归模型的训练，发现模型训练过程中loss及其不稳定，且训练效果很差，终于找到原因了！
原代码如下：
criterion = nn.L1Loss()
def train():
print('Epoch {}:'.format(epoch + 1))
model.train()
# switch to train mode
for i, sample_batched in enumerate(train_dataloader):
input, target = sample_batched['geno'], sample_batched['pheno']
# compute output
output = model(input.float().cuda())
loss = criterion(output, target.float().cuda())
# compute gradient and do SGD step
optimizer.zero_grad()
loss.backward()
optimizer.step()
以上代码问题出在：
loss = criterion(output, target.float().cuda())
我输入的batchsize是4，因此output的size是[4,1],也就是一个二维的数据；target的size是[4]。loss输出的结果是一个正确的数值。这也是我没发现问题的原因！我们看一下pytorch库里l1_loss的代码：
def l1_loss(input, target, size_average=None, reduce=None, reduction='mean'):
# type: (Tensor, Tensor, Optional[bool], Optional[bool], str) -> Tensor
r"""l1_loss(input, target, size_average=None, reduce=None, reduction='mean') -> Tensor
Function that takes the mean element-wise absolute value difference.
See :class:`~torch.nn.L1Loss` for details.
"""
if not torch.jit.is_scripting():
tens_ops = (input, target)
if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
return handle_torch_function(
l1_loss, tens_ops, input, target, size_average=size_average, reduce=reduce,
reduction=reduction)
if not (target.size() == input.size()):
warnings.warn("Using a target size ({}) that is different to the input size ({}). "
"This will likely lead to incorrect results due to broadcasting. "
"Please ensure they have the same size.".format(target.size(), input.size()),
stacklevel=2)
if size_average is not None or reduce is not None:
reduction = _Reduction.legacy_get_string(size_average, reduce)
if target.requires_grad:
ret = torch.abs(input - target)
if reduction != 'none':
ret = torch.mean(ret) if reduction == 'mean' else torch.sum(ret)
else:
expanded_input, expanded_target = torch.broadcast_tensors(input, target)
ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
return ret
代码里的warning，要求input和target的size必须一致，不然会出现不对的结果。我自己代码里把warning给ignore了，所以这个warning一直没看到！这里提醒大家，一定不要随意ignore warning，并且要好好看warning，不要只看error。。。。
我把代码改成以下，就没有问题了：
loss = criterion(output.squeeze(), target.float().cuda())
既然问题解决了，得知道为啥size不匹配会导致模型出错呀，不然找了那么久的bug不是白瞎了= =
我们先尝试错误输入，输入的size是[4,1],target的size是[4]：
input = tensor([[-0.3704, -0.2918, -0.6895, -0.6023]], device='cuda:0',
? ? ? ?grad_fn=<PermuteBackward>)
target = tensor([ 63.6000, 127.0000, 102.2000, 115.4000], device='cuda:0')
expanded_input, expanded_target = torch.broadcast_tensors(input, target)
ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
?返回 expanded_input：
tensor([[-0.3704, -0.2918, -0.6895, -0.6023],
? ? ? ? [-0.3704, -0.2918, -0.6895, -0.6023],
? ? ? ? [-0.3704, -0.2918, -0.6895, -0.6023],
? ? ? ? [-0.3704, -0.2918, -0.6895, -0.6023]], device='cuda:0',
? ? ? ?grad_fn=<PermuteBackward>)
返回?expanded_target：
tensor([[ 63.6000, ?63.6000, ?63.6000, ?63.6000],
? ? ? ? [127.0000, 127.0000, 127.0000, 127.0000],
? ? ? ? [102.2000, 102.2000, 102.2000, 102.2000],
? ? ? ? [115.4000, 115.4000, 115.4000, 115.4000]], device='cuda:0')?
返回ret：
tensor(102.5385, device='cuda:0', grad_fn=<PermuteBackward>)
接下来是正确输入,输入的size是[4],target的size是[4]：?
?input = tensor([-0.3704, -0.2918, -0.6895, -0.6023], device='cuda:0',
? ? ? ?grad_fn=<PermuteBackward>)
target = tensor([ 63.6000, 127.0000, 102.2000, 115.4000], device='cuda:0')
expanded_input, expanded_target = torch.broadcast_tensors(input, target)
ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
?返回 expanded_input：
?tensor([[-0.3704, -0.2918, -0.6895, -0.6023],
? ? ? ? [-0.3704, -0.2918, -0.6895, -0.6023],
? ? ? ? [-0.3704, -0.2918, -0.6895, -0.6023],
? ? ? ? [-0.3704, -0.2918, -0.6895, -0.6023]], device='cuda:0',
? ? ? ?grad_fn=<PermuteBackward>)
返回ret：
tensor(102.5385, device='cuda:0', grad_fn=<PermuteBackward>)
?经过mean求平均之后，返回的ret值是一样的，唯一不同的是expanded_input。这个中间值不一样，是否会导致梯度变化？为了验证这个想法，我们在代码中输出input的梯度值。
for name, parms in model.named_parameters():
print('name:', name)
print('grad_requirs:', parms.requires_grad)
print('grad_value:', parms.grad)
以下为错误输入，输入的size是[4,1],target的size是[4]：?
?===
name: module.linear1.bias
grad_requirs: True
grad_value: tensor([-0.1339, ?0.0000, ?0.0505, ?0.0219, -0.1498, ?0.0265, -0.0604, -0.0385,
? ? ? ? ?0.0471, ?0.0000, ?0.0304, ?0.0000, ?0.0000, ?0.0406, ?0.0066, ?0.0000,
? ? ? ? -0.0259, -0.1544, ?0.0000, -0.0208, ?0.0050, ?0.0000, ?0.0625, -0.0474,
? ? ? ? ?0.0000, ?0.0858, -0.0116, ?0.0777, ?0.0000, -0.0828, ?0.0000, -0.1265],
? ? ? ?device='cuda:0')
===
name: module.linear2.weight
grad_requirs: True
grad_value: tensor([[-0.9879, -0.0000, -1.0088, -0.1680, -0.7312, -0.0066, -0.3093, -0.7478,
? ? ? ? ?-0.3104, -0.0000, -0.1615, -0.0000, -0.0000, -0.3162, -0.1047, -0.0000,
? ? ? ? ?-0.4030, -0.3385, -0.0000, -0.1738, -0.0831, -0.0000, -0.3490, -0.1129,
? ? ? ? ?-0.0000, -0.8220, -0.0279, -0.3754, -0.0000, -0.3566, -0.0000, -0.5950]],
? ? ? ?device='cuda:0')
===
name: module.linear2.bias
grad_requirs: True
grad_value: tensor([-1.], device='cuda:0')
===
以下为正确输入,输入的size是[4],target的size是[4]得到的梯度：?
?===
name: module.linear1.bias
grad_requirs: True
grad_value: tensor([-0.1351, ?0.0000, ?0.0000, ?0.0000, -0.0377, ?0.0000, -0.0809, -0.0394,
? ? ? ? ?0.0000, ?0.0000, ?0.0000, ?0.0000, ?0.0000, ?0.0202, ?0.0098, -0.0365,
? ? ? ? -0.0263, -0.2063, -0.1533, -0.0626, ?0.0050, ?0.0000, ?0.0000, -0.0950,
? ? ? ? ?0.0000, ?0.0000, -0.0348, ?0.0000, ?0.0000, -0.1108, -0.0402, -0.1693],
? ? ? ?device='cuda:0')
===
name: module.linear2.weight
grad_requirs: True
grad_value: tensor([[-7.4419, ?0.0000, ?0.0000, ?0.0000, -1.9245, ?0.0000, -2.7927, -2.4551,
? ? ? ? ? 0.0000, ?0.0000, ?0.0000, ?0.0000, ?0.0000, -0.0309, -0.4843, -0.0211,
? ? ? ? ?-1.7046, -7.7090, -0.1696, -0.9997, -0.0862, ?0.0000, ?0.0000, -2.0397,
? ? ? ? ? 0.0000, ?0.0000, -0.3125, ?0.0000, ?0.0000, -3.9532, -0.0643, -6.5799]],
? ? ? ?device='cuda:0')
===
name: module.linear2.bias
grad_requirs: True
grad_value: tensor([-1.], device='cuda:0')
===
果然，梯度值不一样！！！经验教训：每一行代码都要深入理解其作用的机理，不要想当然！
|