Reddit 60tps
https://www.reddit.com/r/LocalLLM/comments/1vvtquz/dgx_spark_qwen_38_27b_nvfp4_at_60toks_generation/
The version I'm using is this
https://forums.developer.nvidia.com/t/qwen3-8-27b-nvfp4-on-single-dual-dgx-spark-sglang-dflash2-fully-openai-compatible/380732
Workload (single stream) | 1x Spark | 2x Spark (TP=2) |
|---|
Code Generation | 52~61 tokens/sec | 87 tokens/sec |
Prose/Essay | 26 tokens/sec | 41 tokens/sec |
Thought Conversation | 34~49 tokens/sec | 49 tokens/sec |
Thinking about code, ("none") | 52 tokens/sec | approximately 80 tokens/sec |
TTFT, short prompt | approximately 0.16s | Similar |
TTFT, 16K prefix repeat | 0.44s | 0.74s |
Context Window | 262,144 | 262,144 |
I'm using more conservative settings (for quant, embedding, using 8B models together, etc.) and actual measurements show 36~46 tps
Flash 2+ SLang is really .. incredibly fast. What a technological advancement.
Today's tip: Qwen38 has a compact bug
When compressing a 150K token session with Qwen3.8 27B in DeepSeek Harness, a problem was reported where auto compact did not finish, almost the same as before.
Cause: The compressor inherits the reasoning setting of the original conversation and consumes the output budget during the process.
Solution: Request a summary only with reasoningEffort: off
Remove old reasoning, remove images, summarize large tool results.
This method was reported to have stabilized the compression of 128K·150K·256K sessions.
DeepSeek Harness Qwen3.8 compaction case
In Qwen Code, there was also a problem reported where auxiliary requests such as title, summary, and recap inherited the main model settings and unnecessarily engaged in thinking. The solution is "each auxiliary request should have its own model, listening, and extra body settings."
Qwen Code side-query setting inheritance problem
Another Qwen Code issue found that the general OpenAI request structure did not include Qwen-specific enable_thinking, so the OFF setting was not actually being sent to the server. In other words, simply displaying OFF in the UI is not enough; the provider adapter must explicitly insert it into the request body.
Qwen Code enable_thinking delivery defect
The following was added to Vector Build to resolve the issue:
{
"chat_template_kwargs": {
"enable_thinking": false,
"preserve_thinking": false
}
}
