python—Hugginer管道是否在内部处理长文档？

twh00eeo 于 2021-09-29 发布在 Java

关注(0)|答案(0)|浏览(137)

我的设置如下所示：

MODEL_CHECKPOINT = "distilroberta-base"
tokenizer = AutoTokenizer.from_pretrained(PATH_TO_MY_MODEL, max_len=512, add_prefix_space=True)
model = AutoModelForTokenClassification.from_pretrained(MODEL_CHEKPOINT, num_labels=32)
ner_pipeline = pipeline(task="ner", tokenizer=tokenizer, model=model)

但是，我可以获得任意长度文档的ner预测。我想知道它是如何在内部实现的（可能是滑动窗口方法？）

python machine-learning nlp huggingface-transformers

来源：https://stackoverflow.com/questions/68545934/does-the-hugginface-ner-pipeline-internally-deal-with-long-documents