An M5 Max using Samsung's LPDDR5-PIM would generate 650 tokens/second

At Hot Chips 2026, Samsung announced LPDDR5X-PIM, a 16 GB memory package that performs matrix multiplication beside its DRAM banks. In a Llama 3.1 8B benchmark, its 614 GB/s internal bandwidth raised decode from 27 to 81.3 tokens per second.

Apple has announced no PIM Mac, but its eight-package M5 Max provides a useful thought experiment: could eight of Samsung's packages turn that result into 650 tok/s for one batch-one conversation?

We will compare replication, pipeline parallelism, speculative pipelines, and tensor parallelism to find which mapping can apply all eight packages to one token stream.

The hypothetical machine#

The 40-core M5 Max has a 512-bit LPDDR5X-9600 interface, 614.4 GB/s of memory bandwidth, and up to 128 GB of unified memory. Eight 16 GB Samsung packages match that capacity and interface width.

Swapping the packages would not give the M5 Max GPU a 4.9 TB/s memory bus. The SoC would still see 614 GB/s through the pins. The extra bandwidth exists inside the packages, where Samsung placed MAC units beside all sixteen banks. The weights stay in DRAM; small activation vectors and results cross the bus.

That is a large amount of inference hardware hiding in memory packages. If software can keep all eight packages busy, the useful decode rates look like this:

Even after calibrating against Samsung's silicon, the estimate reaches 650 tok/s. That result depends on making eight separate memory packages cooperate on one token stream.

None of those rates comes from plugging in the chips. They depend on how we divide one model across eight isolated compute units.

Where the estimates come from#

Decode reads each model weight about once per generated token. For a memory-bound dense model:

tokens/sec ≈ memory_bandwidth ÷ model_size

An 8B checkpoint occupies about 4 GB at SINT4. Reading 4 GB from the M5 Max's 614 GB/s bus gives 154 tok/s before overhead. At 80 percent utilization, that becomes the 123 tok/s baseline in the table.

Eight PIM packages provide 4.91 TB/s where the weights live. The raw 8B SINT4 ceiling becomes 1,228 tok/s. Samsung's benchmark sustained 81.3 tok/s against one package's 154 tok/s bandwidth ceiling, or 53 percent. Applying that measured efficiency across eight packages gives 650 tok/s.

The 53 percent factor is generous for an extrapolation, but it is more credible than quoting the bandwidth ceiling. It includes command sequences, PIM mode changes, vector operations, and host involvement measured on working silicon. Small models may fare worse because launch costs consume more of each token.

Long contexts also add KV-cache traffic. At 4K, grouped-query attention adds little next to reading the model weights. At 128K, it becomes a serious competitor for the same banks.

The first obstacle: eight packages are not one accelerator#

Each Samsung package owns its banks and MAC units. The packages have no direct link to each other. They communicate through the SoC, and Samsung's PIM hardware favors matrix-vector multiplication, or GEMV, with banks operating in lockstep.

Our inference mapping must satisfy three conditions:

1. Keep all eight packages working on every token.
2. Leave the large weight matrices inside their packages.
3. Express the work as GEMV rather than asking the PIM units to behave like a GPU.

Several familiar distributed-inference strategies fail one of these tests.

Replication: 650 tok/s across eight conversations#

An 8B SINT4 model occupies about 4 GB, so every package can hold a complete copy. Each complete checkpoint must fit within one 16 GB package; after reserving room for the KV cache and runtime data, that limits replication to roughly 20B to 27B SINT4 models. Eight users could each generate at Samsung's measured 81.3 tok/s, for about 650 tok/s in aggregate.

One user still gets 81.3 tok/s. Seven packages sit unused. Replication maximizes server throughput, not single-stream speed.

Replication proves that the hardware has plenty of aggregate throughput. We need a mapping that applies all of it to the same token.

Pipeline parallelism: 81 tok/s#

Divide the model's layers across the packages. Package 0 stores the first layers, package 1 stores the next group, and so on. Each layer group must fit one package, while the complete model can use all 128 GB.

pipeline: [L0–3][L4–7][L8–11] ... [L28–31]

For batch-one decode, token t can occupy only one stage at a time. Package 1 waits for package 0, then package 2 waits for package 1. Seven packages remain idle during each stage. The machine reads the model at one package's 614 GB/s internal rate, which returns us to Samsung's 81.3 tok/s result.

We replaced every memory package with PIM and made 8B decode slower than the standard M5 Max estimate of 123 tok/s.

Classic pipeline parallelism fills its stages with a batch of independent inputs. A local chat has one input, so it needs another source of tokens.

Speculative pipelines: 210 to 380 tok/s#

A speculative pipeline guesses future tokens so several tokens can occupy different stages at once. Package 7 finishes token t while package 0 starts a draft of token t+1. If the model accepts the draft, useful work advances through all eight packages. If it rejects the draft, the pipeline flushes. Like the ordinary pipeline, this mapping can spread the model across the combined 128 GB, though the draft head and runtime state consume some of that capacity.

Pipeline-Parallel Self-Speculative Decoding puts an early-exit draft head on the first stage. Speculative Pipeline Decoding drafts from the hidden states already moving through the pipeline. Both preserve the operation Samsung's PIM hardware wants: one activation vector moving through GEMV stages.

Acceptance rate controls the result. If a rejected guess costs an eight-stage refill, a 70 percent acceptance rate gives about 210 tok/s. At 90 percent, the estimate rises to 380 tok/s.

That is a useful recovery from 81 tok/s, but a deep pipeline carries a large flush penalty. We can shorten it.

A shallow hybrid: 590 tok/s#

Split the model into two pipeline stages instead of eight, then use four-way tensor parallelism inside each stage:

stage 0: [packages 0–3, each holds 1/4 of layers 0–15]
stage 1: [packages 4–7, each holds 1/4 of layers 16–31]

The model can use the combined 128 GB because each stage spans four packages.

Speculation keeps both stages occupied. A rejection now refills two stages instead of eight. At a 90 percent acceptance rate, the estimate reaches 590 tok/s.

This hybrid gives pipeline speculation a sensible role on PIM. It can approach full utilization while keeping every operation in GEMV. It still needs a trained draft head, rollback logic for the KV cache, and consistently high acceptance.

There is a simpler route to the last few tokens per second.

Eight-way tensor parallelism: 650 tok/s#

Give every package one eighth of every weight matrix. The complete model can use all 128 GB because each package stores only its shard. For each layer, the SoC broadcasts the activation vector to all eight packages. Each package multiplies its local shard, then the SoC gathers and reduces eight partial results.

tensor parallel: [1/8 of every layer on each package]
                 [all eight packages active on every token]

This mapping reaches the 650 tok/s estimate without predicting future tokens. All packages read weights at once, all remain in their efficient GEMV path, and a rejected draft can never flush the machine.

The repeated reductions sound expensive until we count the bytes. For an 8B transformer with a 4096-element hidden state and 32 layers, a Megatron-style split performs about 64 small reductions per token. Moving those vectors over a 614 GB/s SoC fabric takes microseconds. Reading 4 GB of weights still dominates the token.

Research systems have reached the same conclusion. CENT uses tensor parallelism across CXL-PIM modules when latency matters. LoL-PIM shards attention and feed-forward heads across PIM modules. The M5 Max SoC would provide a much tighter reduction fabric than a CXL switch.

For this hypothetical laptop, eight-way tensor parallelism is the best first implementation. It turns eight packages into one inference engine, uses a standard dense model, and requires no speculative training. The 650 tok/s estimate is an eight-fold extrapolation from Samsung's measured package, with the same 53 percent efficiency.

Why common token tricks do not stack cleanly#

Most modern speculative decoders verify several candidate tokens in one forward pass. That turns the large linear layers from GEMV into a narrow GEMM. GPUs benefit because GEMM uses their arithmetic units more efficiently. Samsung designed its bank-side units around GEMV, with roughly enough compute to match the bandwidth of one activation vector.

Verifying four candidates at once asks for about four times as much arithmetic per byte. The PIM units hit their compute ceiling, and lockstep banks must handle a matrix-shaped activation. Samsung provisioned about 19.2 TOPS across eight packages, close to what dense 8B decode needs at the 1,228 tok/s bandwidth ceiling. Multi-token verification can amortize control overhead and move the result above 650 tok/s, but it cannot multiply the hardware ceiling. LP-Spec and SpecPIM treat this GEMV-to-GEMM conversion as a central problem for PIM speculation.

This does not make multi-token prediction useless. It explains why the pipeline form fits this hardware while tree verification does not produce a clean multiplier on top of 650 tok/s.

Sparse mixture-of-experts models have a related problem. If a token selects two experts stored on two packages, six packages wait. Batch-one decode cannot hide that imbalance. A dense model wastes fewer PIM units than an expert-per-package MoE.

The physical limits#

Samsung has not published power figures for eight packages performing sustained MAC operations. A 16-inch laptop may reach its thermal or memory-power limit before all eight packages sustain 4.91 TB/s.

Control overhead may also scale poorly. Each package reaches 614 GB/s only while all of its banks execute the same PIM sequence. Eight packages multiply the command scheduling and readback work even when the transferred vectors remain small.

Long-context attention adds another bandwidth consumer. At some context length, dedicating banks or packages to the KV cache may beat giving all eight to weights. That crossover depends on model architecture and context length, so it does not belong in the headline estimate.

The 650 tok/s result should be read as a calibrated scaling target, not a benchmark of hardware that Apple sells. It assumes eight packages retain Samsung's single-package efficiency when driven together.

From 81 to 650#

Samsung has shown 81.3 tok/s from one PIM package. The standard M5 Max should reach about 123 tok/s on the same 4 GB SINT4 model through its 614 GB/s unified-memory bus. Replacing its memory with eight LPDDR5X-PIM packages creates 4.91 TB/s beside the weights.

Using that bandwidth well takes a progression of mappings:

The rates use an 8B SINT4 model and Samsung's measured 53 percent package efficiency. “Useful utilization” measures accepted-token work against the calibrated 650 tok/s capacity of all eight packages. Replication reaches 100 percent only with eight simultaneous users. A batch-one pipeline uses one package at a time; eight users can fill its stages without speculation.

The capacity column states where the weights may live. A 16 GB package can hold 32B SINT4 parameters by raw weight count, or roughly 20B to 27B after reserving space for KV cache and runtime data. The combined 128 GB has a raw 256B-parameter SINT4 limit, with the practical maximum set by context length and working memory.

Replication serves more users. Pipeline parallelism fits the model but idles the chips. Speculation fills the pipeline, then pays when guesses fail. Eight-way tensor parallelism keeps every package busy on every token without changing the model's computation.

Samsung demonstrated one PIM package with one 8B model. Eight packages would turn the memory subsystem into a small distributed inference machine. If a future laptop paired them with the right controller and runtime, roughly 650 tok/s on that workload looks possible.

The fastest credible path runs through the least exotic technique in the post: shard every matrix eight ways, run all packages in parallel, and keep the weights where the 4.91 TB/s of bandwidth lives.