Abstract
RoBERTa is a replication study of BERT that carefully messures the impact of many key heperparameters ans training data size.
improve performance by
- trainging the model longer, with bigger batches over more data
- removing the next sentence prediction objective
- training on longer sequences
- dynamically changing the masking pattern applied to the training data.
通过以上措施，发现其实之前 BERT 的潜力并未被充分发掘，一旦得到有效的训练，其性能匹配或超越在其后提出的若干模型。这一方面说明了之前被忽视的各种设计选择的重要性，一方面对近期提出来的一些号称通过对 BERT 进行而得到性能提升的模型发出疑问。
RoBERTa 代码(基于PyTorch实现)：https://github.com/pytorch/fair
