Ministral-3-8B-Instruct tokenizer doesn't handle BPE markers properly

[yanancai]

System Info

Code:
from transformers import AutoTokenizer, AutoModelForImageTextToText, AutoProcessor
import torch

base_model = "mistralai/Ministral-3-8B-Instruct-2512-BF16"

model = AutoModelForImageTextToText.from_pretrained(base_model, dtype=torch.bfloat16)
model = model.to("cuda:1")
tokenizer = AutoProcessor.from_pretrained(base_model)

user_prompt = "hello how are you?"
messages = [
{"role": "user", "content": user_prompt},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text=text, return_tensors="pt").to(model.device, dtype=torch.bfloat16)
generate_ids = model.generate(**inputs, max_new_tokens=50, do_sample=False)
decoded_output = tokenizer.batch_decode(generate_ids[:, inputs["input_ids"].shape[1] :], skip_special_tokens=True)[0]
print(decoded_output)

Output:
Hello!ĠðŁĺĬĠI'mĠjustĠaĠvirtualĠassistant,ĠsoĠIĠdon'tĠhaveĠfeelings,ĠbutĠI'mĠhereĠandĠreadyĠtoĠhelpĠyouĠwithĠanythingĠyouĠneed!ĠHowĠaboutĠyouâĢĶhowĠareĠyouĠdoingĠtoday?ĠAnythingĠfunĠorĠinterestingĠon

Environments:
Python 3.12.7
transformers 5.0.0.dev0 (installed from main branch)
torch: 2.9.0
mistral_common: 1.8.6

The same code with MinistralCommonBackend loaded tokenizer works:
Code:
import torch
from transformers import AutoModelForImageTextToText, MistralCommonBackend

tokenizer = MistralCommonBackend.from_pretrained(base_model)
model = AutoModelForImageTextToText.from_pretrained(
base_model, torch_dtype=torch.bfloat16
)
model = model.to("cuda:2")

user_prompt = "hello how are you?"
messages = [
{"role": "user", "content": user_prompt},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text=text, return_tensors="pt").to(model.device)
generate_ids = model.generate(**inputs, max_new_tokens=50, do_sample=False)
decoded_output = tokenizer.batch_decode(generate_ids[:, inputs["input_ids"].shape[1] :], skip_special_tokens=True)[0]
print(decoded_output)

Output:
Hello! I'm just a program, so I don't have feelings, but I'm here and ready to help you with anything you need. How about you? How are you doing today?[😊]

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer, AutoModelForImageTextToText, AutoProcessor
import torch

base_model = "mistralai/Ministral-3-8B-Instruct-2512-BF16"

model = AutoModelForImageTextToText.from_pretrained(base_model, dtype=torch.bfloat16)
model = model.to("cuda:1")
tokenizer = AutoProcessor.from_pretrained(base_model)

user_prompt = "hello how are you?"
messages = [
{"role": "user", "content": user_prompt},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text=text, return_tensors="pt").to(model.device, dtype=torch.bfloat16)
generate_ids = model.generate(**inputs, max_new_tokens=50, do_sample=False)
decoded_output = tokenizer.batch_decode(generate_ids[:, inputs["input_ids"].shape[1] :], skip_special_tokens=True)[0]
print(decoded_output)

Expected behavior

Clean output with BPE markers handled properly

[Rocketknight1]

cc @itazap

[yanancai]

Just found it is the same behavior for deepseek-ai/DeepSeek-R1-Distill-Qwen-14B. The same code works with transformers 4.57.3, but not the latest transformers in main.

Code:
from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
import torch

base_model = "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"

model = AutoModelForCausalLM.from_pretrained(base_model)
model = model.to("cuda:1")
tokenizer = AutoProcessor.from_pretrained(base_model)

user_prompt = "hello how are you?"
messages = [
{"role": "user", "content": user_prompt},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text=text, return_tensors="pt").to(model.device)
generate_ids = model.generate(**inputs, max_new_tokens=50, do_sample=False)
decoded_output = tokenizer.batch_decode(generate_ids[:, inputs["input_ids"].shape[1] :], skip_special_tokens=True)[0]
print(decoded_output)

Output with transformers 4.57.3:
Alright, someone just said "hello how are you?" to me. I need to respond in a friendly and approachable way. I should acknowledge their greeting and let them know I'm here to help. Maybe something like, "Hello! I'm

Output with latest in main:
ĊĊĊĊHello!ĠI'mĠjustĠaĠvirtualĠassistant,ĠsoĠIĠdon'tĠhaveĠfeelings,ĠbutĠI'mĠhereĠandĠreadyĠtoĠhelpĠyouĠwithĠwhateverĠyouĠneed.ĠHowĠareĠyouĠdoing?ĠðŁĺĬ

[jurgisp]

Experiencing same issue with mistralai/Ministral-3-14B-Base-2512 - decode(encode(text)) produces text with symbols like "Ġ7Ġ" instead of spaces.

[JawadAliAI]

Hi @yanancai

I can reproduce this consistently on transformers main (5.x dev) and verified that it does not occur in 4.57.3.
It appears related to the tokenizer/processor refactor and improper SentencePiece/BPE decoding when using AutoProcessor.

I’m willing to dig into this and propose a fix.
Could you assign this issue to me?

Thanks!

[itazap]

pushed a fix for this! we need to just preserve the backend tokenizer behavior for those that use a LlamaTokenizer but change the decoder and such

[jurgisp]

Is it just a problem with how AutoProcessor constructs the tokenizer from the config? If so, is there a workaround to create it correctly manually in the current released version?