Local LLM benchmark: how much work fits in 48 GB?
Telling a company that does not want its data leaving the building to “just use a local model” is easy. The hard part is saying which jobs a local model can do and which it cannot. So I measured it — on my own hardware, with real coding tasks, across more than nine rounds.
What did I measure?
I measured open-weight language models on real software tasks: generating code from nothing, and making targeted edits inside an existing codebase. The measurement ran on an Apple M5 Max with 48 GB of unified memory using an MLX-based local service. The goal was not a leaderboard but removing guesswork from one question: can a local model do this job?
- Hardware
- Apple M5 Max · 48 GB unified memory
- Runtime
- MLX-based local service
- Rounds
- 9+ · series ongoing
- Task types
- Greenfield generation · targeted edit in an existing file
- Method
- Single-variable comparison
- Status
- Re-run whenever a new model lands
The clearest finding: the problem is the harness, not the model
Running the same model under different harnesses moved the score by roughly 11 points. Changing how the task and the prompt were written moved it by roughly 18. Changing sampling parameters such as temperature and top-p moved it by only 2–3 points. The setting people argue about most is the one that matters least.
| Variable | Score impact | What it means in practice |
|---|---|---|
| Task and prompt design | ≈ ±18 points | Highest-return investment. Describing the work well beats swapping models. |
| Harness (CLI or agent framework) | ≈ ±11 points | The same model behaves measurably differently under a different tool. |
| Sampling parameters | ≈ ±2–3 points | The most-tweaked knob makes the least difference. |
The consulting translation is this: when a client asks “which model should we use”, the answer is usually not a model name but a way of decomposing the work.
What can local models do, and what can they not?
Local models operate in a usable band for greenfield code generation: the average score sits around 86.5 out of 100 and moves about ±3 points between rounds. On targeted, surgical edits inside an existing file the same models drop to a 49–57 band: instead of changing the file precisely, they tend to rewrite it.
| Task type | Measured band (out of 100) | Reading |
|---|---|---|
| Greenfield generation | ≈ 86.5 (±3) | Usable in production, with review. |
| Targeted edit in an existing file | 49 – 57 | Do not give this to a local model; it rewrites the file. |
That distinction reduces the whole local-versus-hosted argument to one sentence: producing something new works on a local model; changing something that exists, surgically, does not. I use that line to decide which step of a client system goes where.
What limits do you hit at 48 GB?
48 GB of unified memory rules out a portion of the usable open-weight models outright. The file size on disk is also not the memory footprint — conflating the two is the most common planning error on Apple Silicon. Memory-utilisation and concurrency settings have hard edges too.
- Memory utilisation. At the threshold I measured, 0.75 runs and 0.80 crashes. That gap is observed, not theoretical.
- Concurrency. Accepting more than one request at a time causes an out-of-memory failure through double prefill; a single-request limit is a requirement, not a preference.
- Disk size ≠ memory. A 30 GB model file needs considerably more than that once context memory is added.
- Extended reasoning mode. For this model family, turning long reasoning off raised the score — counter-intuitive but reproducible.
- Timeout wall. The harness timeout, not the model, is what caps usable output length in practice.
How did I measure it?
The method is single-variable: each round changes exactly one thing and re-runs the same task set. The task set is derived from real software work, not synthetic puzzles. Every round is written down, and when a new model appears the same set runs again.
- Same task set, same hardware, same service version.
- One variable changes per round: model, harness, prompt design or sampling parameter.
- Results are scored against previous rounds; round number and settings are recorded.
- Configurations that hit a memory limit are recorded with the threshold they hit, not simply as “failed”.
- Results feed directly into the next client decision — this is a decision tool, not a research project.
This measurement was made on my own hardware with my own task set and makes no claim to be a general model ranking. Its purpose is to let me tell a client “a local model can do this” without guessing.
What is this for, in a project?
In a company that does not want data leaving the building, this data decides which step runs on a local model and which on a hosted one. Made by assumption, that decision fails in one of two ways: everything goes to the cloud and creates a data-transfer problem, or everything stays local and the quality is not good enough.
Last updated: