inferwire
/
AI·3 min read

Llama-modchip Runs 284B DeepSeek Model on 64GB RAM CPUs

Llama-modchip enables local inference of 284B MoE models on 64GB CPU systems without GPUs using custom compression and memory tiling.

TL;DR

  • Llama-modchip enables running the 284B DeepSeek-V4-Flash model on a 64GB RAM CPU workstation without requiring any GPUs [^1].
  • The framework uses SwiGLU-H2 compression and custom memory tiling to achieve decode speeds up to 14.7 tokens per second [^1].

Background

Large language models relying on Mixture-of-Experts architectures traditionally require significant GPU memory to host hundreds of billions of parameters. Running these large models on standard desktop CPUs usually causes out-of-memory crashes or severely throttles performance due to memory bandwidth limits. To overcome hardware bottlenecks, developers explore model quantization, parameter pruning, and CPU cache optimizations to fit massive frontier models within standard workstation system RAM without dedicated graphics processing units.

What happened

Open-source project llama-modchip released version v2.4.0, allowing workstations with 64 GB of CPU RAM and zero GPUs to run frontier Mixture-of-Experts (MoE) models such as DeepSeek-V4-Flash 284B [^1]. Unpatched releases of standard IQ2_XXS quantizations require 76.87 GiB of RAM and crash from out-of-memory errors on 64 GB Linux machines [^1]. The new release compresses the 284.33B parameter model to 56.56 GiB on disk and holds peak memory usage to 57.61–58.9 GiB VmRSS at a full 256K context window [^1].

To shrink memory consumption and increase speed, llama-modchip employs Analytical SwiGLU-H2 Second-Order Hessian Compression, Fractal-64 Multi-Head Latent Attention transcoding, and whole-model register and cache tiling across L1, L2, and L3 CPU caches [^1]. On a 16-core AMD Zen 5 CPU without a GPU, the 284B model delivers prefill speeds of 44.36 tokens per second and pure CPU decode speeds ranging from 9.1 to 14.7 tokens per second when using Stockfish-PVS speculative drafting [^1].

The release also optimizes 30B MoE models like NVIDIA-Nemotron-3.5-30B-A3B, running it in 14.35 GiB of RAM at 25.0 to 60.4 tokens per second on Zen 5 CPUs and up to 111.6 tokens per second on AMD Threadripper processors [^1]. Additionally, llama-modchip introduces a zero-API Claude Code CLI integration (./modchip claude) that bypasses CPU prompt re-evaluation using /dev/shm Slot-0 KV Warm Restore, cutting warm turn latency to 18.9 ms on Nemotron-30B and 13.1–32.6 ms on DeepSeek-284B [^1]. Because the engine relies on block-64 Gray-coded tensors (GGML_TYPE_Q2_0, type ID 40), running these models in unpatched llama.cpp or Ollama will fail with an unknown tensor type error [^1].

Why it matters

Running 284-billion-parameter MoE models entirely on consumer-grade CPUs disrupts the hardware assumptions surrounding local artificial intelligence inference [^1]. Developers and organizations can run frontier-class models at practical decoding speeds on existing 64 GB RAM workstations, reducing reliance on expensive multi-GPU clusters or enterprise cloud APIs [^1].

Furthermore, combining second-order compression with multi-level CPU cache tiling demonstrates that memory hierarchy optimizations can compensate for memory bandwidth limits [^1]. Features like warm KV cache restores over shared memory (/dev/shm) solve high latency during multi-turn developer agent tasks, opening up local zero-API coding workflows without GPU hardware [^1].

Practical example

Say Sam is a software developer who wants to run large coding models locally on a desktop PC equipped with 64 GB of DDR5 RAM and no dedicated graphics card [^1].

Previously, attempting to load a standard quantization of DeepSeek-V4-Flash 284B required 76.87 GiB of memory, causing Sam's Linux system to run out of memory and crash [^1].

Using llama-modchip v2.4.0, Sam downloads the pre-compiled binary and the Fractal-64 GGUF checkpoint [^1]. The engine compresses the 284B parameter model on disk to 56.56 GiB and caps peak RAM usage at under 59 GiB at full 256K context [^1]. When Sam executes a prompt, the pure CPU setup achieves a 44.4 tok/s prefill speed and decodes responses at 9.1 to 14.7 tok/s using Stockfish-PVS [^1].

When Sam launches local coding agent workflows using ./modchip claude, /dev/shm Slot-0 KV Warm Restore re-activates context in 13.1 to 32.6 ms instead of waiting through 45-second CPU prompt re-evaluations [^1]. If Sam tries loading the custom block-64 tensor file directly into standard unpatched Ollama, the application returns an unknown tensor type 40 error, requiring Sam to use the llama-modchip binary [^1].

Related gear

We recommend this textbook for its deep mathematical coverage of deep learning architectures, matrix operations, and optimization algorithms that underpin modern model compression.

AdvertisementAmazon

Deep Learning

Sources

  1. [1]GitHub — GitHub - synchopate/llama-modchip: 🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB C
  2. [2]Hacker News — Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU