[人工智能]BERT模型自定义词汇以及token相关

1.加载bert模型及分词

from transformers import AutoModelForMaskedLM, AutoTokenizer
model = "bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model, use_fast=True)
model = AutoModelForMaskedLM.from_pretrained(model)

2.分词演示

• 这里是对COVID hospitalization分词

print(tokenizer.tokenize('COVID'))
print(tokenizer.tokenize('hospitalization'))
['CO', '##VI', '##D']
['hospital', '##ization']

• 如果让着两个词都能保持完整，而不被拆分，进行如下操作

# Let's increase the vocabulary of Bert model and tokenizer

new_tokens = ['COVID', 'hospitalization']
num_added_toks = tokenizer.add_tokens(new_tokens)

# Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.

model.resize_token_embeddings(len(tokenizer))
print(tokenizer.tokenize('COVID'))
print(tokenizer.tokenize('hospitalization'))

# with each new word added, a new vector of embeddings 
# with random values was added as well thanks to the 
# model.resize_token_embeddings(len(tokenizer)) method.

tokenizer.savepretrained("modle_dir")

3.自定义bert词表

• bert有自己的vocab配置可以人为修改以适应下游任务，参考看这篇https://blog.csdn.net/kyle1314608/article/details/10661204

• bert词表相关配置文件介绍
  https://www.jianshu.com/p/a6170d3408f5

参考链接

https://zhuanlan.zhihu.com/p/391814780