AutoTokenzier.from_pretrains needs setting with trust_remote_code inside load_module_map

[thincal]

System Info

lorax main

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

1. using a tokenizer class not registered in the Transformers
2. place the tokenizer config file and related files inside the adapter checkpoint folder
3. inference with adapter info pointed to this adapter folder
4. the tokenizer loading will go into the resolve_trust_remote_code logic inside the Transformers, where it will require user input to confirm if trust_remote_code is not True.

Expected behavior

loading the tokenizer directly if it is existed inside adapter.

[thincal]

The fix is easy, will prepare a PR for review soon.

[tgaddair]

Thanks @thincal!