inferwire
/
AI·3 min read

Four Intel AI PCs Split Llama 3.1 70B Between Them

A paper with an Intel co-author splits LLMs by layer across idle AI PCs. Four Lunar Lake machines ran Llama 3.1 70B at 5.42 tokens per second, right at the threshold for interactive chat.

TL;DR

  • A paper by Tate Berenbaum of Not Community Labs and Muthaiah Venkatachalam of Intel splits a language model by layer across several Intel AI PCs [^1].
  • Four Lunar Lake machines ran Llama 3.1 70B at 5.42 tokens per second for one user, right at the authors' threshold for interactive chat [^1].

Background

Intel AI PCs have integrated GPUs, NPUs and 16 GB or more of unified memory, and often sit idle [^1]. That isn't enough for Llama 3.1 70B, which takes about 36 GB at 4 bits [^1]. Pipeline parallelism gives each machine a slice of the model's layers. Petals already ran BLOOM-176B across consumer GPUs over the internet at about one step per second [^2]. On consumer hardware, the network is the problem, since every token passes through all machines in turn [^1].

What happened

The system pre-compiles each slice into an INT4 OpenVINO graph and relies on three techniques [^1]. First, a naive per-slice export ran 13 to 23 percent slower than the unsplit model because it missed an OpenVINO GPU optimization; injecting a beam_idx Gather operation into each shard triggered that optimization and brought the shards within 4 percent of the unsplit model [^1]. Second, it makes speculative decoding work, in which a small draft model proposes several tokens and the large model checks them in one pass. Instead of physically deleting rejected draft tokens from the cache, which cost about 48 milliseconds per call, it masks them out, with bit-identical results at essentially no cost [^1]. Third, it interleaves requests from several users across the stages, each with its own cache [^1].

On three laptops connected over WiFi, two ASUS Zenbook S 14 with Lunar Lake and an HP OmniBook X 16 with Panther Lake, a two-machine pipeline running Llama 3.1 8B served two users at 43.97 tokens per second in total, 1.79 times the single-user speed of the unsplit model on the same hardware [^1]. For the large model, the authors split Llama 3.1 70B into four shards of about 9 GB and ran them on four AI PC instances in Intel's Tiber Cloud, connected through Tailscale's relay servers [^1]. Without speculative decoding the pipeline managed 1.74 tokens per second; with it, 5.42 tokens per second for one user, producing the same tokens [^1]. With a Panther Lake coordinator and two users, total throughput reached 6.43 tokens per second, or 3.21 per user [^1].

Why it matters

The appeal is cost and data locality: a fleet of office PCs that already exists could serve a model without a cloud dependency and at zero marginal cost [^1]. The numbers show how close to the edge that is. The authors put the floor for interactive chat at 5 tokens per second, so a four-machine 70B fleet sits right at the boundary for one user and drops below it when serving two [^1]. Speculative decoding helps most where the network is the bottleneck: it gave speedups of 1.24 to 1.44 times within a single process but 3.11 times over the relayed connection, because each check covers several tokens per round trip [^1]. The code, raw benchmark logs and reproduction scripts are published [^1].

Practical example

Say your office has four Lunar Lake laptops with 32 GB of memory each that sit idle after hours, and you want to run a 70B model without sending data to a cloud service. Following this paper, you would split the model into four INT4 shards of about 9 GB, put one on each machine and run a small draft model, such as Llama 3.2 1B, on the machine that coordinates [^1]. You could expect roughly five tokens per second for a single user, enough for chat at a measured pace but not for several people at once [^1].

Related gear

We recommend this book because it provides the fundamental principles of network architectures, pipeline coordination, and fault-tolerant distributed computing.

AdvertisementAmazon

Distributed Systems: Principles and Paradigms

Sources

  1. [1]arXiv — Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
  2. [2]arXiv — Petals: Collaborative Inference and Fine-tuning of Large Models