Article · · 16 min read
A 125B model on a 1,400-euro computer. Putting Strata through my own benchmark
I got interested in running large language models locally without spending a fortune on a graphics card and a computer. My whole machine cost about 6,000 Polish złoty (roughly 1,400 euro): a server-grade Intel Xeon E5-2699 v4 from 2016, 121 GB of DDR4 RAM (that is what the system reports), an NVMe drive and a GeForce RTX 3060 with 12 GB of memory.
For a few weeks Strata kept showing up in my feed from different places: GitHub, Reddit, newsletters about local models. The promise was always the same: Qwen3.8-Flash-Next, Alibaba's open model with 125 billion parameters, on a single 12 GB card, at more than 50 tokens per second. I decided to measure that, and then to check with a benchmark of my own whether such a model is worth more than the four smaller ones that are the natural choice on a computer like this.
What Strata is
Strata is an open-source engine (MIT licence) by a single author under the pseudonym Niko1221, unaffiliated with the Qwen team, built around Qwen3.8-Flash-Next: a mixture-of-experts model in which about 6 billion of the 125 billion parameters do the work for each token. Strata keeps all the "experts" in RAM, leaves the attention layers and a stock of the most used experts on the graphics card, computes the rest on the CPU, and a small extra layer (MTP, multi-token prediction) proposes several tokens ahead, which the full model verifies in a single pass; according to the documentation the answer is the same, only faster (details of the mechanisms).
Installation and requirements
Installing through the strata command in a terminal needs no administrator rights and writes everything to the home directory:
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh
strata launch --plan --model iq3_s # shows the plan, changes nothing
strata launch --model iq3_s --context 128k --yes --no-tui
The script checks the SHA-256 sum and the release signature of the engine archive it downloads; the script itself, as always with curl | sh, arrives unverified, so I saved it to a file and read it before running it. The engine and the model with its extras are 90 GB to download; loading the model takes about a minute. The server exposes an OpenAI-compatible API (http://127.0.0.1:8080/v1), an Anthropic-compatible one, and a chat page in the browser.
The model comes in several levels of quantization, that is, weights stored with fewer bits instead of 16: smaller files, less RAM, slightly worse answers. The installer adds the RAM to the card's memory, subtracts 5 GB for the system and from a given threshold proposes the next variant:
| Installer threshold (RAM + VRAM − 5 GB) | Variant | Experts in RAM |
|---|---|---|
| 32 GB | Coder IQ1_M, half of the experts, chosen for programming | 23 GB |
| 40 GB | IQ2_XS | 35.5 GB |
| 48 GB | IQ3_XXS | 43 GB |
| 64 GB | IQ3_S, tested here | 50 GB |
RAM is what matters most, because the experts have to fit in it entirely: 50 GB for IQ3_S, and the documentation recommends 60 GB and says that with 64 GB the variant works when few other programs are open. On my machine the engine process used 83 GB, because with more than 107 GB of RAM Strata also loads into memory a 29 GB table of n-gram embeddings that normally stays on disk. The card needs at least 12 GB and the CPU must support AVX2; a bigger card makes the model faster, because it holds more experts, but does not lower the RAM requirement.
IQ3_S stores the weights at about 3.3 bits. The authors measure the quality of the compression on 5,000 tokens of text, checking how often a variant picks the same most likely token as the 8-bit version: IQ3_S in 87.7% of positions, the 4-bit UD-Q4_K_XL in 93.1%, IQ3_XXS in 84.5%, and the smallest Q2_0 in 80.6%. That is a different measure from a benchmark: a different token need not change the meaning of the answer, and how often it does, the documentation does not show. I had nothing to compare the full model against, so I do not know what the compression alone costs.
At start-up Strata warned that only 200–450 MB of the card's memory was left free, because the desktop takes some of it, and suggested --vram-reserve-mib 1024. I ignored the warning and nothing went wrong, but once the driver starts moving the card's memory into RAM, writing drops to a few tokens per second.
Speed: my measurements versus the documentation
The documentation gives 53 tokens per second of writing and 1,620 tokens per second of prompt reading for IQ3_S, measured on a computer with an RTX 5070 (also 12 GB), a Ryzen 5 7600 and 64 GB of RAM. On my machine, at temperature 0 (randomness off, the measurement repeatable) and with four kinds of text (a Python function, an essay in Polish, an essay in English, a list of 30 names), Strata wrote 49 tokens per second on average: from 43 in the Polish essay to 54 in code. Writing is therefore on a par with the documentation. The differences between kinds of text come from MTP: according to the statistics the server returns with every answer, in code the model accepted more than 70% of the proposed tokens, in Polish prose about 40%.
Reading long prompts came out twice as slow as in the documentation: 790–850 tokens per second for prompts of 17 to 115 thousand tokens. I have more cores than that machine (22 versus 6), but while reading, Strata ships experts to the card over PCIe, and my computer has PCIe 3.0 (measured at 10.5 GB/s), DDR4 memory and a CPU without the AVX-512 instructions the engine uses on newer processors (on mine, according to the log, it used AVX2). The documentation does not separate the effect of these factors, so I cannot say which one accounts for the difference.
For comparison, the same model file in llama.cpp (build b10809, with the experts in RAM and without the MTP layer, which llama.cpp does not use for this model) wrote 16 tokens per second, three times slower, and read prompts at 88–226 tokens per second.
Long context
The model natively supports 262,144 tokens of context; the installer sets 32K for cards under 14 GB, I started Strata with 128K. For the measurements I took random excerpts of Strata's English documentation, a different one each time so that the engine could not reuse a remembered prompt. Reading 115 thousand tokens took 140 s, and writing hardly depended on the context length: 46 tokens per second at 1,300 tokens and 42 at 115 thousand (43–54 in the short prompts of the previous section).
A Polish sentence with a password hidden in the middle was found by the model in 15 of 16 attempts; once, at 66 thousand tokens, it refused, because it took the sentence for a prompt injection, an instruction hidden in the data. In a harder test I placed two English sentences in the documentation's style ("the nightly regression tests run on machine ORCA-7", "machine ORCA-7 sits in the Frankfurt office") a quarter and three quarters of the way into the text, in five prompts of 9 to 122 thousand tokens; asked in which city the machine with the nightly tests stands, the model answered correctly every time.
Agents
The strata connect command prints or writes the configuration for Claude Code, Codex CLI, OpenCode, Hermes Agent and a few other tools; for Claude Code that is three environment variables: a local address compatible with the Anthropic API, a model name and a placeholder key. With --check it sends a test question and a tool call (tool calling, the model's request to run a function); on my machine both passed in 3–4 seconds.
Claude Code (in --bare mode, without plugins) got from me a small Python project with a bug in word counting, an unimplemented function and six tests, five of them failing. It fixed everything in 19 steps and 133 seconds without touching the test file; the model formulated every tool call correctly at the first attempt. The prompt cache helps: the engine remembers the already-read beginning of the conversation, so in the following steps it read only the appended part, in 0.3–1.1 s. This is one attempt on a simple task, not an assessment of work on a larger repository.
My own benchmark
Published benchmark results say little about a model compressed to 3 bits, run on hardware like this and asked questions in Polish. So I built my own set: 53 tasks in Polish in eight categories, matching what public test suites for language models measure.
| Category | Public model | Example tasks | Tasks |
|---|---|---|---|
| Mathematics | GSM8K, MATH | multiplying large numbers, a discount plus tax, compound interest, days between dates | 7 |
| Coding | HumanEval, MBPP | Python functions run against my tests: validating the Polish tax number NIP with its check digit, Roman numerals, numbers in Polish words | 6 |
| Reasoning | BBH, ARC | a liars puzzle, weighing coins, ordering by height, counting letters in a word | 9 |
| Knowledge | MMLU, SimpleQA | facts from history, science and computing, two of them Polish | 7 |
| Instruction following | IFEval | JSON only, capitals only, exactly five lines, 40–60 words with a prescribed last word | 7 |
| Working with text | DROP, SQuAD | questions about a given incident report: duration, cause, chronology; the sentiment of reviews | 6 |
| Writing | graded by hand | an email to a client, a four-line poem with AABB rhymes, a children's story, an official notice rewritten in a casual style | 6 |
| Resistance to making things up | TruthfulQA | questions about things that do not exist: the novel "The Glass Garden" by Bolesław Prus (a 19th-century Polish writer), a Polish physics Nobel laureate from 1978, the Python function itertools.chunked | 5 |
In the first six categories a script checks the answers: it compares a number or a word, parses JSON, counts lines and words, and runs the code against tests. The writing and the making-things-up tasks I graded myself, by two rules. A text passes when it does what was asked and is coherent; a single language error does not disqualify it, but a poem without rhymes or with two lines instead of four does. An answer about a non-existent thing passes only when the model says it does not exist and adds no invented facts; "Prus did not write it, it is a Reymont novel" does not pass.
The set is small and single-language, and I wrote the tasks myself, so it measures everyday, checkable tasks in Polish, not the limits of the models. The models answered at the default temperatures from their model cards, that is, the way they are used day to day, so the answers differ between passes; that is why I repeated the set and sum the results over all passes: Strata and gemma-4-E4B three times without thinking and twice with it, gemma-4-12B and Qwen3.6-35B-A3B twice without and once with, Qwen3.6-27B twice without thinking. In thinking mode, where the model writes out its reasoning before answering, it had a budget of 8,000 tokens for it (Strata) or 6,000 (the others, since they are slower); Qwen3.6-27B at its 5 tokens per second would need a dozen or so hours for such a pass, so I skipped it.
For the comparison I chose four open models that are the natural choice on a computer with a 12 GB card and plenty of RAM, all run through llama.cpp in Q4_K_M variants (about 4.8 bits per weight):
- gemma-4-E4B (5 GB): Google's small model, fits entirely on the card.
- gemma-4-12B (7.4 GB): the largest of these models that fits entirely on a 12 GB card.
- Qwen3.6-35B-A3B (21 GB): a mixture-of-experts model with 3 billion active parameters; as in Strata, the experts sit in RAM and the CPU computes them.
- Qwen3.6-27B (17 GB): a dense model, that is, one computing all its weights for every token; it fits on the card only partly.
The comparison is not entirely fair: the smaller models have their weights stored at about 4.8 bits, Strata at 3.3, so if compression breaks anything, it breaks the 125B model more.
Results
Without thinking mode, Qwen3.8-Flash-Next through Strata answered correctly in 89% of attempts (142 of 159 over three passes), and the whole set of 53 tasks took it 3.3 minutes. Gemma-4-12B had 82%, Qwen3.6-35B-A3B 79%, gemma-4-E4B 63%, and Qwen3.6-27B 79%.
Mathematics and coding separate the big model from the small ones most sharply. Strata slipped once in 21 maths attempts (in the trains problem it gave 20 minutes instead of 120) and not once in 18 coding attempts; gemma-4-12B and Qwen3.6-35B-A3B had 11 and 10 of 14 in maths and 10 of 12 each in code. The function that writes numbers in Polish words, with the inflection "dwa tysiące", "pięć tysięcy", was passed by none of the smaller models in any pass, and the multiplication 4567 × 8912 was got wrong every time without thinking by three of the four smaller models (only Qwen3.6-27B managed it); with thinking gemma-4-12B worked it out too. Gemma-4-E4B passed 7 of 18 coding tasks; its functions for parsing durations, validating NIP numbers and writing numbers in words failed the tests in every pass.
Reasoning went poorly for Strata without thinking: 23 of 27. The puzzle about three people, some of whom always lie, it got wrong in all three attempts, each time answering in under a second with a single name; with thinking it solved it two times out of two. Without thinking, all the smaller models also got that puzzle wrong in every pass. Counting the letters "r" in the word "przerażający" is a tokenization trap (the model sees the word in chunks, not in letters), which Strata passed 2 times out of 3 and both Gemmas not once.
Knowledge and instruction following do not separate the larger models: Strata, gemma-4-12B and Qwen3.6-35B-A3B had a full or nearly full score there. Gemma-4-E4B got facts wrong in 6 of 21 attempts and never once carried out the instruction "answer in capital letters only". Qwen3.6-35B-A3B twice exceeded the prescribed limit of 40–60 words.
Working with text is the only category where Strata without thinking lost to the smaller models: 15 of 18 against full scores for gemma-4-12B and Qwen3.6-35B-A3B. The whole difference is one question: the duration of the outage, which it three times answered as 105 minutes, counting from the certificate's expiry (14:30) instead of from the moment the system stopped accepting transactions (14:35). With thinking it had a full score.
Writing I graded by hand. The email, the technical explanation, the idiom translation and the rewritten notice came out well for Strata in every attempt; the four-line poem with AABB rhymes did not succeed without thinking even once (once I got two lines instead of four), and with thinking two times out of two. Rhyming in Polish turned out hard for everyone: none of the smaller models wrote a correct four-line poem, and the Gemmas put non-existent words into theirs ("gotę", "rozwoni"). Qwen3.6-35B-A3B twice added to the rewritten notice that the training is "on Wednesday" (3 June 2024 was a Monday).
Resistance to making things up was lost by every model. Strata without thinking passed 9 of 15 attempts, with thinking 3 of 10; gemma-4-12B and Qwen3.6-35B-A3B 4 of 10 each, gemma-4-E4B 5 of 15, and Qwen3.6-27B did best with 6 of 10. It was rarely a matter of summarising a non-existent novel from scratch (although gemma-4-E4B did that every time). The typical error looked different: the model rightly denied, and then added invented facts, for instance that "The Glass Garden" was written by Reymont (another Polish writer, a Nobel laureate), that "Shadows of Bridges" is a John le Carré novel from 2015, or it fell into a loop of "Jerzy Zurek (no, that's wrong)" repeated to the end of the answer. Ten attempts by Strata with thinking are too few to speak of a rule, but the pattern was the same each time: after longer reasoning the model answered more confidently, not more truthfully, and for example two times out of two described, with a code example, the function itertools.chunked, which Python does not have.
Thinking helps, then, where something has to be computed or checked: it raised Strata's score from 89% to 92%, and gemma-4-E4B's from 63% to 77%. It costs time: the set took Strata more than five times longer (18.4 minutes instead of 3.3), and one Python function took 141 s and 6,783 tokens of reasoning plus answer instead of 22 s. Gemma-4-12B thought for 26 minutes and in three tasks exhausted the 6,000-token budget, returning, among other things, an empty poem. Qwen3.6-35B-A3B with thinking came out the same as without (77% against 79%), because in nine tasks, four of them coding, it exhausted the budget and did not get to answer, and the set took 65 minutes. The budget is my choice; with a bigger one these models would probably answer better, but even more slowly.
Qwen3.6-27B, the only dense model in the comparison, answered correctly in 79% of attempts (84 of 106), but at 5 tokens per second the set took it 28 minutes, almost nine times longer than Strata without thinking. It was the only one of the smaller models to get the multiplication right in both passes and the best of all five on questions about non-existent things, but it twice got wrong the publication year of "The Doll" and the planet with the most moons, and never solved the liars puzzle or the numbers in words.
What follows from this
Speed on this computer is decided by three things: how many weights the model computes per token, whether they fit on the card, and whether the engine has multi-token prediction. Gemma-4-E4B writes 76 tokens per second, gemma-4-12B 38, Qwen3.6-35B-A3B with its experts in RAM 37, Strata 49, and Qwen3.6-27B, which does not fit on the card, 5; the same 125B model without MTP in llama.cpp wrote 16. On a 12 GB card the choice comes down to the pair Strata or gemma-4-12B: 7 percentage points and 11 tokens per second more, with the same errors in making things up, for the price of 60 GB of RAM occupied for the whole time it runs, 90 GB of disk, and dependence on a single author's engine built around one family of models. When the next Qwen comes out, llama.cpp will support it within days; Strata when Niko1221 adds it.
Being able to do part of the work entirely locally, on a model smaller than the leading cloud models, is tempting, and after this test I know which part: arithmetic, code, questions about checkable facts, and work on documents. Questions about things the model may not know I would not ask any of the five models locally without checking the answer.
Money is not the argument here. Under load the computer drew about 340 W according to a wattmeter, of which the card alone about 140 W; at Poland's electricity price of about 1 złoty per kilowatt-hour an hour of the model's work costs about 0.35 złoty (8 euro cents). The same model in Alibaba's API costs, according to the aggregators' price lists, 0.47 dollars per million generated tokens, and in an hour Strata generates at most 176 thousand of them; that is the same order of magnitude, without buying 60 GB of RAM. The argument is data that never leaves the computer, no rate limits, and work without a network. The limitation is that Strata by default handles one request at a time (I did not test several users) and that thinking mode, which improves the results in logic and code, makes the work five times longer.
Sources
- Strata, repository and documentation, Niko1221, version 0.2.3, engine 0.1.48: README (speed table),
docs/MODELS.md,docs/QUANTS.md,docs/HOW_IT_WORKS.md. - Qwen3.8-Flash-Next, model card, Qwen.
- llama.cpp, ggml-org, build b10809; comparison models in Q4_K_M variants from LM Studio Community.
- Qwen3.8-Flash-Next API price according to anotherwrapper.com, as of September 2026.