huggingface transformers 源码剖析： tokenizer篇

Table of Contents

1. 引言

2. tokenizer各类继承情况

20211123193710.png

2.1. PreTrainedTokenizerBase

Author: Zi Liang (liangzid@stu.xjtu.edu.cn) Create Date: Tue Nov 23 16:18:22 2021 Last modified: 2024-03-09 Sat 20:56 Creator: Emacs 28.1 (Org mode 9.5.2)