Qwen3.5: Difference between revisions
No edit summary |
No edit summary |
||
| (13 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
|release_date=February 15, 2026 | |release_date=February 15, 2026 | ||
|sizes=0.8B, 2B, 4B, 9B, 27B (dense), 35B-A3B (MoE), 122B-A10B (MoE), 397B-A17B (MoE) | |sizes=0.8B, 2B, 4B, 9B, 27B (dense), 35B-A3B (MoE), 122B-A10B (MoE), 397B-A17B (MoE) | ||
|architecture=[[Decoder-only Transformer | |architecture=[[Decoder-only Transformer]] | ||
|context_length=262,144 (up to 1M via API) | |context_length=262,144 (up to 1M via API) | ||
|modality=Image-Text-to-Text | |||
|thinking=Yes (toggleable) | |||
|license=Apache 2.0 | |license=Apache 2.0 | ||
|languages=201 languages and dialects | |languages=201 languages and dialects | ||
| Line 13: | Line 15: | ||
}} | }} | ||
'''Qwen3.5''' is an open-weight and native vision-language model series developed by Alibaba and released on February 15, 2026. | '''Qwen3.5''' is an open-weight and native vision-language foundation model series developed by Alibaba and released on February 15, 2026.<ref name="qwen3.5-blog">[https://qwen.ai/blog?id=qwen3.5 Qwen3.5: Towards Native Multimodal Agents]. Qwen Team, February 2026.</ref> It is build on a hybrid architecture using linear attention with [[Gated Delta Networks]] as well as sparse [[Mixture of Experts]]. The models support 201 languages and dialects, compared to 119 of their earlier [[Qwen3]] model series. | ||
Smaller versions up to 27B parameters are available as dense models, whereas the sizes 35B-A3B, 122B-A10B as well as the flagship model 397B-A17B use a [[Mixture of Experts]] architecture. | |||
== Benchmarks == | |||
Results for the flagship '''397B-A17B''' and '''9b''', '''4B''' as well as '''2B''' small models. | |||
{| class="wikitable" | |||
|- | |||
! Benchmark !! Category !! 397B-A17B !! 9B !! 4B !! 2B !! Claude Opus 4.6 | |||
|- | |||
| [[GPQA Diamond]]<ref>[https://artificialanalysis.ai/evaluations/gpqa-diamond?models=gemma-4-26b-a4b%2Cgemma-4-31b-non-reasoning%2Cgemma-4-e2b%2Cgemma-4-e4b-non-reasoning%2Cgemma-4-e4b%2Cgemma-4-e2b-non-reasoning%2Cclaude-opus-4-6-adaptive%2Cqwen3-5-2b%2Cqwen3-5-9b%2Cqwen3-5-397b-a17b%2Cqwen3-5-4b%2Cqwen3-5-2b-non-reasoning GPQA Diamond Benchmark Leaderboard: Results]. Artificial Analysis, April 2026.</ref> || style="text-align:right;" | Science || style="text-align:right;" | 89.3 || 80.6 || 77.1 || -- || style="text-align:right;" | 89.6 | |||
|- | |||
| [[SWE-bench Verified]] || style="text-align:right;" | Coding || style="text-align:right;" | 76.4 || -- || -- || -- || style="text-align:right;" | 80.8 | |||
|- | |||
| [[MMMU-Pro]] || style="text-align:right;" | Multimodal || style="text-align:right;" | 79.0 || 70.1 || 66.3 || 50.3 || style="text-align:right;" | 73.9 | |||
|- | |||
| [[MMMLU]] || style="text-align:right;" | Multilingual || style="text-align:right;" | 88.5 || 81.2 || 76.1 || 63.1 ||style="text-align:right;" | 91.1 | |||
|} | |||
== Tokenizer == | |||
The Qwen3.5 tokenizer has a total of 248,077 tokens (up from 151,669 in [[Qwen3]]), out of which 248,044 stem from the [[BPE]] vocabulary size, and 33 are added tokens. Note that Qwen3.5 uses 248,320 embeddings in its embedding table, slightly more than the total token number of the tokenizer. This might be due to performance improvements (it is a multiple of 512) and leaves room for adding additional tokens in the future. | |||
Thinking can be '''enabled''' and '''disabled''' through the chat template, which either appends <syntaxhighlight inline=1><think>\n</syntaxhighlight> or <syntaxhighlight inline=1><think>\n\n</think>\n\n</syntaxhighlight>. | |||
<syntaxhighlight lang="python"> | |||
from transformers import AutoTokenizer | |||
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B") | |||
len(tokenizer) | |||
# -> 248077 | |||
messages = [{"role": "user", "content": "Hi"}] | |||
tokenizer.apply_chat_template(messages, enable_thinking=False, tokenize=False, add_generation_prompt=True) | |||
# -> '<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n' | |||
tokenizer.apply_chat_template(messages, enable_thinking=True, tokenize=False, add_generation_prompt=True) | |||
# -> '<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\n' | |||
tokenizer.apply_chat_template(messages, enable_thinking=True, tokenize=True, add_generation_prompt=True) | |||
# -> {'input_ids': [248045, 846, 198, 12675, 248046, 198, 248045, 74455, 198, 248068, 198], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]} | |||
</syntaxhighlight> | |||
== References == | |||
<references /> | |||
Latest revision as of 13:19, 7 April 2026
| Qwen 3.5 | |
|---|---|
| Developer | Alibaba Cloud |
| Release Date | February 15, 2026 |
| Model Sizes | 0.8B, 2B, 4B, 9B, 27B (dense), 35B-A3B (MoE), 122B-A10B (MoE), 397B-A17B (MoE) |
| Architecture | Decoder-only Transformer |
| Modality | Image-Text-to-Text |
| Thinking | Yes (toggleable) |
| Context Length | 262,144 (up to 1M via API) |
| License | Apache 2.0 |
| Languages | 201 languages and dialects |
| Hugging Face | Qwen 3.5 |
| Paper | Link |
Qwen3.5 is an open-weight and native vision-language foundation model series developed by Alibaba and released on February 15, 2026.[1] It is build on a hybrid architecture using linear attention with Gated Delta Networks as well as sparse Mixture of Experts. The models support 201 languages and dialects, compared to 119 of their earlier Qwen3 model series. Smaller versions up to 27B parameters are available as dense models, whereas the sizes 35B-A3B, 122B-A10B as well as the flagship model 397B-A17B use a Mixture of Experts architecture.
Benchmarks[edit]
Results for the flagship 397B-A17B and 9b, 4B as well as 2B small models.
| Benchmark | Category | 397B-A17B | 9B | 4B | 2B | Claude Opus 4.6 |
|---|---|---|---|---|---|---|
| GPQA Diamond[2] | Science | 89.3 | 80.6 | 77.1 | -- | 89.6 |
| SWE-bench Verified | Coding | 76.4 | -- | -- | -- | 80.8 |
| MMMU-Pro | Multimodal | 79.0 | 70.1 | 66.3 | 50.3 | 73.9 |
| MMMLU | Multilingual | 88.5 | 81.2 | 76.1 | 63.1 | 91.1 |
Tokenizer[edit]
The Qwen3.5 tokenizer has a total of 248,077 tokens (up from 151,669 in Qwen3), out of which 248,044 stem from the BPE vocabulary size, and 33 are added tokens. Note that Qwen3.5 uses 248,320 embeddings in its embedding table, slightly more than the total token number of the tokenizer. This might be due to performance improvements (it is a multiple of 512) and leaves room for adding additional tokens in the future.
Thinking can be enabled and disabled through the chat template, which either appends <think>\n or <think>\n\n</think>\n\n.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-0.8B")
len(tokenizer)
# -> 248077
messages = [{"role": "user", "content": "Hi"}]
tokenizer.apply_chat_template(messages, enable_thinking=False, tokenize=False, add_generation_prompt=True)
# -> '<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
tokenizer.apply_chat_template(messages, enable_thinking=True, tokenize=False, add_generation_prompt=True)
# -> '<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\n'
tokenizer.apply_chat_template(messages, enable_thinking=True, tokenize=True, add_generation_prompt=True)
# -> {'input_ids': [248045, 846, 198, 12675, 248046, 198, 248045, 74455, 198, 248068, 198], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
References[edit]
- ↑ Qwen3.5: Towards Native Multimodal Agents. Qwen Team, February 2026.
- ↑ GPQA Diamond Benchmark Leaderboard: Results. Artificial Analysis, April 2026.