Strata Benchmarked: A 125B MoE on a Gaming Laptop with 62 GB of RAM (Before/After Tuning)
A gaming laptop with 24 GB of VRAM and 62 GB of RAM running a 125B-parameter sparse MoE model — a year ago, that alone would have been "server-only" territory. But getting it to run is only the first step; running it right is another matter. On this machine, with the same weights and the same hardware, differences in configuration and engine version alone can make a 4–5× difference in speed.
This article documents that journey "from running to running right": before/after measurements for five changes, a single-variable sweep of 10+ engine parameters (which ones are real gains and which are just measurement noise), the quality/speed trade-off across three quantization levels, and how far the context length can be pushed.
1. Conclusions First
- The expert pack must be fully resident in RAM. This is the single most important thing about this engine. Before and after the fix, the same quantization level differed by 4.3× (15.6 → 66.6 tok/s).
- The engine version matters a lot: going from 0.1.40 → 0.1.43.1 boosted decoding by another 39% (66.6 → 92.9 tok/s), and the expert cache hit rate went from 94% to 98%.
- The highest-quality level can be fast too: IQ3_S, officially ranked first in quality, decodes at 96 tok/s with a 128K context once its expert pack is fully resident in RAM — only about 10% slower than the lightest level, while scoring 4 points higher in quality.
- Guard against noise when tuning: two baseline runs with the identical configuration differed by 6.7% on their own. Without interleaved controls, any "optimization" of ±3% may be fake — of the parameters I tested in this round, only 4 actually worked.
- Long context is usable: a 256K configuration was fed a 145,864-token prompt and passed a key-information recall check (that length exceeds 131,072, which a 128K configuration can't serve), with prefill holding steady above 1500 tok/s.
The figure above summarizes all the measurements in this article: ① the speed staircase from three fixes on the same quantization level; ② quality vs. speed across the three quantization levels; ③ which parameters in the sweep were effective and which were noise; ④ long-prompt tests at three lengths with a 256K context.
2. What Is Strata
Strata is an open-source local LLM inference engine (MIT, C++, built on top of llama.cpp/ggml), maintained by an independent developer. It was released in late 2026-09 and iterates extremely fast. Its positioning is narrow and explicit: it serves only one model, Qwen3.8-Flash-Next, and fully exploits that model's special architecture — which is why it's far faster on this model than general-purpose engines, at the cost of not being general-purpose.
Qwen3.8-Flash-Next is an experimental preview of the Qwen4 architecture (qwen4_exp):
| Item | Spec |
|---|---|
| Total params / active per token | ~125B / ~6B |
| MoE structure | 48 layers × 512 routed experts; 10 active per token + 1 shared expert |
| Attention | Hybrid of Gated DeltaNet linear attention + Qwen Sparse Attention (QSA) |
| Other | HyperConnections, PLE n-gram hash embedding table (51.2B params, ~28.8 GB) |
| Modality | Natively multimodal (BF16 vision projector, ~0.9 GB) |
The engine does several things specifically for this architecture. You need to understand them to make sense of the tuning that follows:
- Tiered expert caching: each layer's experts are packed into blobs and split into three tiers by hotness — the expert cache in VRAM, the resident arena in RAM, and the model file (SSD). Hits in VRAM or RAM never touch the disk; only misses read from the SSD.
- Row-wise streaming of the PLE table: the 28.8 GB n-gram table is a "lookup," not a matrix multiply, and each token reads only one row. So it can stay on the SSD and be read on demand instead of occupying RAM.
- MTP multi-token prediction (speculative decoding): a small draft head guesses several tokens at once, which the main model then verifies.
- int8 / streaming KV cache: compresses the VRAM footprint of long contexts.
To be clear about the requirements: officially, 12 GB of VRAM is enough to "get in the door," but that only means it runs. The real requirement is RAM — the expert pack is 34 GB for IQ2_XS, 40 GB for IQ3_XXS and 47 GB for IQ3_S; add code and runtime, and 64 GB of RAM is the comfortable baseline. You also need corresponding SSD space (the full model is 68–84 GB across two shards, of which the ~29 GB PLE table can stay on disk). More VRAM is faster, but VRAM is not the deciding factor.
3. Test Platform and Measurement Methodology
| Item | Configuration |
|---|---|
| GPU | RTX 5090 Laptop 24 GB (24463 MiB actually usable) |
| RAM | 62 GiB DDR5-5600 (2×32 GB; motherboard maximum 64 GB) |
| Storage | NVMe; model and expert packs both on a local drive |
| Engine | Strata 0.1.43.1 (0.1.40 before the upgrade) |
| Model | Qwen3.8-Flash-Next, three GSQ-RCO quantization levels (IQ2_XS / IQ3_XXS / IQ3_S) + BF16 vision projector |
| Sampling | Thinking disabled, temperature 0.3 |
Methodology (this matters — otherwise the numbers below aren't comparable):
- Decode speed =
completion_tokens ÷ total request time, discarding the first request after each engine restart (a cold-start outlier that can be as low as half the normal value). - Cross-restart drift of ±5–9%: restarting the same configuration twice for a baseline gave 113.4 and 106.3 tok/s. So any difference under ~7% is treated as noise.
- Prefill (prompt reading) is measured with a fixed 13K-token long prompt; repeated measurements of the same configuration are within ±1%, so this one is actually very stable.
- Every parameter change gets its own engine restart, discards the first run, and takes the median of multiple runs; key items are double-checked with A/B/A interleaving (baseline → variant → baseline).
4. Before and After: The Initial State Plus Five Changes
| # | Change | Decode speed | Change |
|---|---|---|---|
| 0 | Initial state: experts via file mapping + old engine (0.1.40) | 15.6 tok/s | — |
| 1 | Remove --mmap-experts, switch to the resident RAM arena |
66.6 tok/s | 4.3× |
| 2 | Engine upgrade 0.1.40 → 0.1.43.1 | 92.9 tok/s | +39% |
| 3 | Highest-quality level IQ3_S: remove --resident-budget-gib |
88.8 → 90–97 tok/s | +13% |
| 4 | --prefill auto:32768 (larger prompt chunks) |
13K prefill 1288 → 1410 tok/s | +15% |
| 5 | --vram-reserve-mib 700 → 1536 |
Eliminates the occasional VRAM-paging cliff down to single-digit tok/s | Stability |
Change 1: the experts must be in RAM. The decisive evidence came from the engine's own log:
expert tiers: … RAM 0 blobs, files 0 blobs, 138 GB read
Not a single expert was in RAM; everything was being read from disk. At the time the machine had only 31 GB of RAM, and the config included --mmap-experts (which means "opt out of the resident expert arena and use file mapping instead"). Once RAM was upgraded to 62 GB, this parameter went from a necessary compromise to pure dead weight — deleting it gave 4.3×.
Change 2: engine version. The 0.1.43 release explicitly lists changes such as "improved expert cache hit rate." In my measurements, the hit rate went from 94% to 98% and decoding rose 39% — same weights, same hardware.
Change 3: an easily misread "ceiling." The IQ3_S config originally had --resident-budget-gib 46, but whether I set it to 40 or 46, the engine log only ever reported:
resident RAM mode: 31.54 GiB of experts in RAM (page-locked)
It looked like "the RAM ceiling is only 31.5 GiB." Only after searching the strings in the engine binary did I understand: this parameter switches the engine to a different code path (the file-tier expert source), whose budget gets clamped by clamp_resident_budget to "currently available RAM − 4 GiB headroom − 0.25 GiB safety margin." Available RAM was only around 35 GiB because the page cache for the 28.8 GB PLE table was occupying memory at the time.
When you have enough RAM, you shouldn't set this budget at all. With it removed, the engine takes the default resident arena and locks the entire 47 GB expert pack into RAM in one go — verified via /proc/<pid>/smaps, the process shows a 46.84 GiB anonymous mapping, of which 46.76 GiB is backed by transparent huge pages (THP). Speed went from 88.8 to 90–97 tok/s.
Change 4: larger prompt chunks. This parameter only affects "prompt reading" (prefill), not generation speed. Four interleaved rounds with the same configuration:
--prefill |
Four rounds | Median |
|---|---|---|
auto (default) |
1230 / 1208 / 1255 / 1219 | 1228 tok/s |
auto:32768 |
1413 / 1421 / 1399 / 1404 | 1410 tok/s |
Each setting's spread within itself is <1%, and the two settings were interleaved — so this +15% is real. Interestingly, the engine itself prints a warning about it (saying that 32768-sized chunks lend cache slots to the prompt path, that those experts aren't in RAM, and that it could be 3× slower); but at the 13K scale the chunks fit, and the warning didn't materialize. It's worth re-testing when prompts reach tens of K.
Change 5: don't fill VRAM too full. This line showed up in the log, and everyone should pay attention to it:
WARNING: only 379 MiB of VRAM free at the start of this request (below 512 MiB):
when the card is this full the driver can page GPU memory to system RAM,
decode falls to a few tok/s and a window may never finish
After raising the VRAM reserve from 700 MiB to 1536 MiB, free VRAM went back to about 1.2 GiB, the warning disappeared, and speed was unchanged. This was also the root cause of the earlier occasional single-digit tok/s — not a model problem, but VRAM so full that it triggered driver paging.
5. Parameter Sweep: Over a Dozen Tested, Only 4 Actually Work
| Parameter | Local measurement | Verdict |
|---|---|---|
Use the resident arena (remove --resident-budget-gib) |
31.54 → 46.84 GiB resident, +13% | ✅ Effective |
--prefill auto:32768 |
Prefill +15% (verified with A/B/A/B) | ✅ Effective |
--vram-reserve-mib 700 → 1536 |
Eliminates the VRAM-paging cliff | ✅ Effective |
| Engine 0.1.40 → 0.1.43.1 | +39% | ✅ Effective |
--spec 6 |
105.4 tok/s (baseline 107.2) | ⬜ Noise |
--spec 8 |
100.0 tok/s | ⬜ Noise (slightly lower) |
--spec-min-p 0.7 |
106.3 tok/s | ⬜ Noise |
--expert-cache-per-layer |
+2.5% in a single run; after A/B/A, falls between the two baselines | ⬜ Disproven |
--ple-row-cache 8M |
109.6 tok/s (+2.2%) | ⬜ Noise |
--kv-resident 20480 |
107.5 tok/s (+0.3%) | ⬜ Noise |
--pool-workers 20 |
108.9 tok/s (+1.6%) | ⬜ Noise |
--pcie-frac 0.6 (manually overriding the auto value) |
99.3 tok/s, −7% | ⚠️ Worse |
--vram-reserve-mib 256 |
Engine fails to start (no room for the speculative draft head) | ⚠️ Not viable |
The most valuable row in this table is --expert-cache-per-layer: a single measurement showed +2.5%, which looked like a free gain; switching to an interleaved baseline → variant → baseline order put it squarely between the two baselines (113.4 / 106.3). A single sample + no control = reporting noise as a gain.
6. Choosing Among the Three Quantization Levels
The official model card provides per-level evaluations (GSQ-RCO quantization; all four levels allocate precision per tensor):
| Variant | Bits | Size | AIME25 | GPQA-D | LiveCodeBench v6 | Task average |
|---|---|---|---|---|---|---|
| BF16 base | 16.00 | 354 GB | 100.00 | 91.92 | 87.43 | 93.12 |
| Q2_0 | 2.40 | 66.4 GB | 96.67 | 89.39 | 81.14 | 89.07 |
| IQ2_XS | 2.50 | 68.0 GB | 96.67 | 87.37 | 83.43 | 89.16 |
| IQ3_XXS | 3.00 | 75.8 GB | 100.00 | 91.41 | 86.29 | 92.57 |
| IQ3_S | 3.50 | 83.6 GB | 100.00 | 92.93 | 86.86 | 93.26 |
IQ3_S is the officially recommended level: it matches the base model on AIME25, beats it by 1 point on GPQA-D, trails by only 0.57 on LiveCodeBench, and is less than a quarter of the base model's size.
Here's how speed and quality relate in a local deployment:
| Level | Expert pack | Steady-state decode (128K context) | Official quality average |
|---|---|---|---|
| IQ2_XS | 34 GB | 107 tok/s | 89.16 |
| IQ3_XXS | 40 GB | 92.9 tok/s | 92.57 |
| IQ3_S | 47 GB | 96 tok/s (90–97) | 93.26 |
Conclusion: if you have enough RAM, pick the highest-quality level. Once resident in RAM, IQ3_S essentially catches up with the lightest level on speed (a gap of about 10%, with 4 points higher quality). The only reason to choose a lighter level is insufficient RAM (e.g. a 32 GB machine).
7. Long Context: 256K in Practice
Raising the context from 128K to 256K comes with a small speed cost (IQ3_S drops from 96 to 83–86 tok/s — still within the ±7% drift band mentioned earlier, but directionally it is a bit slower), in exchange for being able to feed in longer material. I ran three lengths using the long-context speed test built into the engine/dashboard (a number is buried in the prompt and the model is asked to repeat it back, to verify it actually read the whole thing):
| Target length | Actual input | Time | Prefill speed | Key-information recall |
|---|---|---|---|---|
| 20K | 29,209 tokens | 18.8 s | 1556 tok/s | ✅ |
| 50K | 72,941 tokens | 47.0 s | 1551 tok/s | ✅ |
| 100K | 145,864 tokens | 95.3 s | 1530 tok/s | ✅ |
The third run's input of 145,864 tokens exceeds 131,072, meaning this request would fail outright with a limit error under the 128K configuration — that's the real benefit of the longer context.
The cost is small: the engine's resident memory grew from 52.6 GiB to 53.1 GiB (256K uses about 2.4 GiB more than 128K), swap usage was 39 MiB, and about 7 GiB of RAM remained available.
8. Final Configuration
Here are the complete parameters for running IQ3_S locally (paths such as --pack/--native omitted):
--expert-cache auto # VRAM expert slots sized automatically from available VRAM
--prefill auto:32768 # Prompt chunking (+15% in local tests)
--spec 4 --spec-min-p 0.5 # Speculative decoding: 4 draft tokens
--kv int8 # int8 KV cache
--kv-resident 20480 # KV streaming: keep 20480 cells per layer in VRAM
--max-context 262144 # 256K context
--vram-reserve-mib 1536 # Leave headroom for the driver to avoid VRAM paging
--pool-workers 15 # CPU-side worker threads
--vision # Enable vision (multimodal)
# No --mmap-experts, no --resident-budget-gib
How to verify (check both):
# 1) The engine reports resident mode itself
grep -a "resident RAM mode" strata.log # Expected: 46.xx GiB of experts in RAM
# 2) Does the process really have a ~47 GB anonymous mapping? (the log wording is the same across code paths, so it can mislead you)
awk '/^Rss:/{if($2>524288)print "Rss GiB",$2/1048576} \
/^AnonHugePages:/{print "AnonHuge GiB",$2/1048576}' /proc/$(pgrep -x strata)/smaps
# Expected: Rss 46.84 GiB / AnonHuge 46.76 GiB
On the engineering side, I integrated this into my self-built ops dashboard: switching quantization levels and context length via config files, auto-start on boot, and management via systemd. The dashboard also has two test entry points: a coding-ability comparison and a long-context speed test. Note that the dashboard's "memory mode" has three options (on-demand mapping / resident outside the GPU / fully resident), and only "fully resident" corresponds to the optimal configuration in this article — choosing either of the other two adds back the low-memory parameters removed above.
9. Limits and Applicability
- 12 GB of VRAM will run it, but don't expect speed: less VRAM means a lower expert cache hit rate, so speed depends on your RAM and SSD. This combination of a 24 GB card + 62 GB of RAM decodes at roughly 90–100 tok/s, with 13K-prompt prefill around 1400 tok/s.
- RAM is a hard requirement: the expert pack is 34–47 GB; add the OS and runtime, and a 32 GB machine can only run the lightest level, with some experts read from disk. 64 GB is the comfortable baseline.
- Mainly single-request: engines like this are designed for single-stream inference by default; you'll need to load-test multi-stream concurrency yourself.
- Quality limits: IQ3_S is essentially on par with the BF16 base in the official evaluations, but lower levels (IQ2_XS/Q2_0) lose 3–4 points on coding and reasoning benchmarks, so for coding tasks it's worth prioritizing the higher-quality level.
- An "uncensored" version is also possible: the community has an abliterated version of this model (
orcarouter/Qwen3.8-Flash-Next-Uncensored), and the engine docs include dedicated manual integration instructions (the hyperconnection/attention projections compressed in regular quantizations must be expanded back to BF16, the PLE table read via mmap, and the MTP draft head configured separately). The cost is a larger size (about 85 GB across two shards for IQ3_XXS) and higher resident RAM requirements (about 49.8 GiB) — it falls into the "runs, but takes manual work" category. - On "how much faster than llama.cpp": the engine's own docs don't promise to beat llama.cpp under identical conditions. Community re-tests under identical conditions land roughly in the 2× range, while the official comparison figures use a different methodology. Comparing speeds across different configurations, prompts and output lengths is meaningless; if you report numbers, report prefill and decode separately.
10. Three Lessons for Anyone Who Wants to Try This
- Read the logs before the speed charts. This engine tells you directly in its log whether the experts are "in RAM" or "being read from disk" (
RAM 0 blobs … 138 GB read), whether VRAM is too full, and whether the speculative draft head fits. 90% of "it's slow" problems can be answered from the log. - Don't carry old configs over to new hardware. Parameters like
--mmap-expertsare meant for low-memory machines; once RAM doubles, they become pure dead weight. After a hardware upgrade, re-examine every parameter that was there "to save memory back then." - Measurement discipline matters more than tuning tricks. First measure your machine's drift (restart the same configuration twice), then decide how large a difference is worth believing. In this round, a parameter that "looked like a +2.5% gain" dropped to zero under interleaved controls; of more than a dozen parameters, only 4 held up.
References
- Engine repository:
github.com/Niko1221/Strata(MIT; thedocs/folder covers low-memory mode, long context and third-party variants) - Quantization and evaluation:
huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF(includes per-item evaluations for all four quantization levels and VRAM recommendations) - Quantization method papers: GSQ (Gumbel-Softmax quantization, arXiv:2604.18556), RCO (bit-budget allocation under Riemannian constraints, arXiv:2605.00649)
- Architecture support on the general-purpose engine side: llama.cpp's
qwen4expsupport and MTP inference path (PR #27742 / #28243)
This article is purely a technical record and contains no purchase links.