JustFit Runs a 27B Model With 200K-Token Context on a 24 GB Mac
An MLX runtime called JustFit stretched a 24 GiB MacBook's single-request context for Qwen3.8-27B from 30,720 to 212,992 positions by managing memory just in time.
TL;DR
- JustFit, an inference runtime for Apple's MLX framework, let a 24 GiB M4 Pro MacBook run Qwen3.8-27B with 196,608 input plus 16,384 output tokens [^1].
- That raised the longest completed single request from 30,720 positions with the mlx-vlm baseline to 212,992, or 6.93 times as many [^1].
Background
Capable open-weight models make running coding and reasoning assistants locally attractive, but long contexts and execution state quickly exhaust a laptop's memory [^1]. For repository-scale coding, source files, tool results and earlier attempts pile up over many calls; compacting them can drop details later edits need [^1]. The bottleneck is no longer only whether the model's weights fit, but whether the context and working state fit alongside them [^1].
What happened
In a paper by Yuhua Chen, JustFit combines three mechanisms: KVExec for executing on a compressed key-value cache, PhaseSwap for managing which components are resident in memory, and StateTrans for preserving state across serving transitions [^1]. Together they materialize data just in time and release it when it is no longer needed, independently of how the model weights are quantized [^1].
On a 24 GiB M4 Pro MacBook running Qwen3.8-27B in MXFP4 format, three independent runs completed 196,608 input and 16,384 output tokens [^1]. A separate run with two concurrent requests kept 229,376 positions in total [^1]. In performance tests, a 32K-token prompt generated 19.11 tokens per second, and a repeated 32K-plus-6K workload peaked at a median of 16,374 MiB [^1]. The runtime answered 29 of 30 AIME 2026 problems correctly [^1].
Why it matters
Running a strong model locally keeps private code and working state on the user's own machine instead of a server [^1]. The paper notes that Qwen3.8-27B reports benchmark scores close to frontier models on reasoning and coding, so the limiting factor for useful local assistants is increasingly memory management, not model quality [^1].
Practical example
Say you want an assistant to work through a large codebase on a laptop with 24 GB of memory. With a standard MLX setup, a single request topped out at 30,720 positions in the tests [^1]. With JustFit's approach, the same machine kept 212,992 positions in one request, enough to hold far more of the repository and its history at once [^1].