JustFit Enables 200K-Token LLM Serving on 24 GiB Laptops

TL;DR

• Researchers unveiled JustFit, an MLX-based inference engine that enables 200,000-token context window serving on standard 24 GiB consumer laptops [^1].
• JustFit combines compressed execution, component layer swapping, and just-in-time state reconstruction to eliminate memory crashes during long-context processing [^1][^2].

Background

Running large language models locally offers complete privacy and zero network latency for developers. However, processing expansive codebases or book-length documents creates massive key-value (KV) caches in system RAM. On laptops equipped with 24 gigabytes of unified memory, a 200,000-token context window quickly exhausts available memory overhead. This severe bottleneck forces system swapping, degrades response speed, or crashes the local operating system, preventing developers from running sophisticated reasoning models on personal hardware.

What happened

Computer science researchers introduced JustFit, an open-source inference runtime built on top of Apple's MLX framework that fundamentally alters local memory management [^1][^2]. The system enables mid-range laptops featuring 24 gigabytes of memory to process and serve massive 200,000-token context windows using open-weight models like Llama and Qwen [^1]. Previously, handling context windows of this magnitude on consumer devices required expensive workstation hardware with 64 gigabytes or more of unified memory.

JustFit solves memory starvation through three primary architectural innovations: KVExec, PhaseSwap, and StateTrans [^1]. The first component, KVExec, handles compressed key-value cache execution [^1]. Instead of maintaining full-precision context representations in active RAM, KVExec stores cache tokens in a heavily compressed state and materializes high-precision values only when specific attention layers evaluate active calculations [^1]. This prevents static context caches from consuming tens of gigabytes throughout long multi-turn interactions.

To prevent model weights from crowding out remaining system RAM, PhaseSwap dynamically alters component residency during inference pass cycles [^1]. Rather than keeping all model parameters continuously resident in memory, PhaseSwap streams specific model layers from solid-state storage into system memory right before execution and releases them immediately after [^1]. Finally, StateTrans manages state-preserving transitions during serving operations [^1]. It coordinates context prefilling and token generation steps without losing conversational history or forcing expensive recomputation passes across incoming requests [^1][^2].

Why it matters

Local AI deployment has consistently collided with physical hardware limitations. While enterprise cloud platforms handle multi-hundred-thousand token prompts by chaining clusters of enterprise graphics cards together across data centers, individual software developers work within fixed hardware boundaries. Until now, running long-context local models meant purchasing top-tier workstations with 64 or 128 gigabytes of RAM. JustFit demonstrates that software orchestration can bypass hardware barriers, bringing enterprise-grade context capacities to standard consumer devices.

This software shift carries significant implications for software engineering, financial services, and security research. Organizations operating under strict privacy mandates often forbid employees from uploading sensitive source code or proprietary financial ledgers to third-party cloud APIs. By enabling long-context serving on standard laptops, JustFit allows security teams and software engineers to run deep static code analysis and document review entirely offline on local physical hardware.

Additionally, JustFit challenges the conventional assumption that long-context processing requires static, high-bandwidth memory reservations. By demonstrating that layer swapping and just-in-time state materialization can maintain model coherence without performance destruction, the framework creates a template for future edge computing optimizations. As open-weight reasoning models grow in parameter size and context capacity, software runtime efficiency ensures consumer laptops remain capable development targets.

Practical example

Consider a freelance software engineer named Maya who works on a standard 24 GiB laptop. She needs to refactor an entire legacy application comprising forty interconnected files totaling 180,000 tokens of source code.

Normally, attempting to load this entire codebase into a local model causes her laptop to run out of memory, freeze, and crash her terminal session.

With JustFit installed, Maya pastes her full codebase into her local editor assistant. The engine splits the context safely. KVExec compresses the massive key-value history into a tiny memory footprint. As the model analyzes the logic across files, PhaseSwap streams model layers into memory on demand and vacates them instantly.

Maya's system memory usage stays safely below twenty gigabytes, leaving ample headroom for her code editor, browser, and background dev tools. Within seconds, her local model outlines a complete, multi-file refactoring plan without sending a single line of client code over the network.

Related gear

We recommend this book because it explains memory management, model serving architectures, and system optimization techniques.