Processing in Memory: DRAM Is About to Do Math

At Hot Chips 2026, Samsung presented a 16 GB LPDDR5X memory package that delivers 614 GB/s of internal bandwidth to its own compute units.

For scale, 614 GB/s matches the memory bandwidth of a top-spec Apple M5 Max across its entire unified memory system: the 40-core GPU configuration in the $3,499 MacBook Pro. Samsung claims that number from inside one memory package, against the 76.8 GB/s that escapes through its external pins. Tom's Hardware reported the resulting eightfold difference.

That eight-to-one ratio makes the case for Processing in Memory. DRAM banks already provide most of the bandwidth; the external pins cannot expose it.

The terms#

Bank. DRAM contains banks, independent arrays that can operate in parallel. A modern LPDDR5X die has dozens. Together they deliver enormous internal throughput, but they share a narrow external interface.

GEMV vs. GEMM. Matrix-vector multiply versus matrix-matrix multiply. Autoregressive decoding at batch size 1 uses GEMV: one token's activations multiplied against the entire weight matrix. Prefill and batched serving use GEMM. PIM helps most with GEMV.

PIM. Processing-in-memory hardware places compute units next to the banks, inside the memory die, where they can use bank-level bandwidth instead of interface bandwidth.

Arithmetic intensity. FLOPs performed per byte loaded. High intensity makes a workload compute-bound. Low intensity makes it bandwidth-bound, leaving matrix engines idle.

The bottleneck is memory bandwidth#

Generating one token requires reading every parameter from DRAM, multiplying it once, and discarding it. Each token depends on the preceding token, so batch-1 decoding cannot reuse the weights across tokens. Arithmetic intensity sits near the floor.

The first-order model:

tokens/sec ≈ memory_bandwidth_GB/s ÷ model_size_GB

At 614 GB/s against a 20 GB model, the hard ceiling sits around 30 tokens per second. The real number is lower because the KV cache grows with context and competes for the same bandwidth. During generation, most of the matrix hardware you paid for waits on DRAM.

Apple's local-inference performance centers on memory bandwidth. The M5 Pro moves 307 GB/s, the M5 Max moves 460 or 614 GB/s, and the M5 Ultra reaches 1.2 TB/s. Those numbers predict tokens per second better than TFLOPs do. Apple pays for a wider external interface.

Samsung's design keeps the largest transfers inside the memory package.

What Samsung built#

Sixteen PIM blocks sit beside the DRAM banks, with MAC trees running in parallel and an ALU handling floating-point and integer types. The package is a JEDEC-standard 561-ball part containing 16 GB across four dies per rank. Samsung targets servers, mobile devices, and clients using the same footprint as ordinary LPDDR5X.

The package processes data in four steps, each with consequences for software:

1. Activation write. Sixteen WRPB commands broadcast FP8 activation data from the host into source register files across the banks.
2. Weight load. PIMX_RD reads 32-byte weight elements from the DRAM cells, maps them across the MAC trees, and combines outputs in a vector register file.
3. Vector store. PIMX_WR moves partial sums back to the bank. The read-to-write ratio can vary instead of remaining fixed at 1:1.
4. Readback. The host switches to single-bank mode and issues up to 64 sequential reads to drain the 1-kbit vector register file.

Weights remain inside the package. Only the much smaller activations cross the bus, while the host sends commands to the PIM blocks.

Samsung supports fifteen precision combinations through MAC precision fields in a configuration register. SINT4 weights reach 2.4 TOPS; FP8 reaches about 1.2 TFLOPS per package.

Conventional memory controllers can use it#

Samsung calls its compatibility mechanism Address Align Mode. It maps DRAM addresses to MAC instructions so the part can work with a conventional DRAM controller. The system switches between single-bank mode for ordinary DRAM and multi-bank mode for PIM through predefined rows and PIM registers. Samsung says this approach switches modes faster and more reliably than its earlier HBM-PIM parts.

Requiring every SoC vendor to build a new memory controller could have kept PIM in research prototypes. Address Align Mode avoids that requirement, leaving software integration as the larger problem.

The measured result#

Samsung validated real silicon on its edge AI accelerator SoC. It compared LPDDR5X with LPDDR5X-PIM while running Llama 3.1 8B at a 320-token context using SINT8 activations, SINT4 weights, and SINT32 output.

This vendor benchmark covers one model, one short context, one accelerator, and integer quantization throughout. It demonstrates that the architecture works on silicon. It does not predict performance for every workload. The 320-token context in particular minimizes the KV cache problem.

Samsung has not disclosed package pricing, so the cost advantage over a wider memory interface remains a claim rather than a measured result.

Why memory vendors care#

Samsung opened its talk with a cost slide. Memory grew from 52% of AI chip component spending in Q1 2024 to 63% by Q4 2025, with HBM accounting for most of it.

LPDDR5X-PIM gives customers a route to HBM-class inference bandwidth without HBM's price, power use, or packaging complexity. Samsung has pursued this idea since the Aquabolt-XL HBM2-PIM proof of concept in 2021. LPDDR5X-PIM is its first LPDDR-based PIM design to reach productization.

The software problem#

No PIM backend exists in llama.cpp, vLLM, or another mainstream runtime. Samsung offers a simulator, a datasheet on request, and an SDK with reference tooling. Three obstacles stand between that vendor stack and a --pim flag.

1. K-quants do not map to the hardware#

llama.cpp gets much of its quality-per-bit advantage from GGUF k-quants such as Q4_K_M. These formats use block-wise quantization with per-block scales and superblock scales. CPUs and GPUs can perform the few dequantization operations per block on nearby general-purpose ALUs.

A DRAM die has little general-purpose logic. PIM MAC units remain small because DRAM manufacturing processes optimize for storage capacitors rather than logic transistors, and every added transistor reduces storage density. Research on in-memory activation quantization found that mainstream quantization schemes need extra scaling and control hardware on top of the MAC units. One cited design incurred about 126% area overhead for FP16 and INT32 support. Other studies of processing across the memory hierarchy and efficient sparse PIM show the same sensitivity to die-area overhead.

Samsung's minimal MAC units cannot execute the k-quant scaling operations. Its fifteen precision modes use uniform formats, including the SINT4 and SINT8 pairing in the benchmark. They do not implement Q4_K_M. Existing GGUF files would need re-quantization into a PIM-native layout, and we do not yet know how much of the k-quants' quality advantage would survive.

2. Physical memory layout#

mmap lets the kernel scatter pages across physical DRAM. PIM needs a weight matrix placed contiguously and aligned to the banks whose MAC trees will multiply it. Samsung's PIMX_RD reads 32-byte weight elements from cells and maps them across MAC trees, which only works when the data occupies the expected locations.

An operating system would need a huge-page-backed, bank-aware allocator exposed to user space. macOS, Linux, and Windows do not provide one today. llama.cpp's memory-mapped model loading, one of the features that makes it pleasant to use, conflicts with this requirement.

3. Bank-level PIM favors GEMV#

Current near-bank designs, including Samsung's HBM2-PIM, SK hynix's GDDR6-PIM, and LPDDR5X-PIM, target GEMV. That fits batch-size-1 decoding, where the bandwidth wall hurts most.

Modern inference also spends time between clean GEMV and dense GEMM. Batching combines GEMV operations across queries into GEMM. Grouped-query attention merges multiple GEMVs into narrow GEMM. These operations raise arithmetic intensity, but often remain too narrow to keep a GPU busy. The CENT architecture paper analyzes this operational-intensity gap.

GQA appears in most current models. Speculative decoding enters the same middle ground because verifying draft tokens in parallel converts GEMV into GEMM. Research on LPDDR-PIM speculative inference found that this conversion reduces the efficiency of GEMV-focused PIM designs. Small-batch local serving faces the same problem.

What about multi-token prediction?#

Multi-token prediction has two roles. A model can use it as a training objective, as DeepSeek-V3 does, and discard the extra prediction modules during ordinary inference. An inference runtime can also use those modules as a speculative drafter: they propose several future tokens, then the main model verifies the candidates in parallel. Google reports up to a 3x speedup from this approach with Gemma 4.

That verification step exposes the weak point in Samsung's design. Checking several candidate tokens in one pass changes the large linear layers from GEMV toward narrow GEMM. The in-bank MAC units can still perform some memory-bound work, but they cannot use their full bank-level throughput on those wider operations. Samsung's current design also blocks the NPU from accessing DRAM while PIM runs, preventing the two processors from splitting one layer concurrently.

MTP and PIM can work together in a heterogeneous system. PAPI proposes a scheduler that measures decoding parallelism at runtime and sends memory-bound kernels to PIM while keeping compute-bound kernels on a GPU. That research architecture uses more flexible hardware than Samsung's LPDDR5X-PIM package.

The speedups would not multiply cleanly because both techniques reduce the cost of reading model weights. MTP amortizes one weight read across several accepted tokens; PIM makes each remaining read faster. On hardware you can buy today, MTP may provide a larger and cheaper gain for models with accurate draft heads. Its benefit depends on draft acceptance rate and verification cost, while PIM's benefit depends on model layout, quantization, and the fraction of work that remains GEMV.

Mixture-of-experts models add uneven memory access. Routing sends tokens to a subset of experts. If those experts occupy a subset of banks, some banks become hot while others sit idle. Variable workloads also create changing GEMV-to-GEMM ratios that reduce hardware utilization.

Computer architects have explored several responses:

• Split the work. AttAcc, IANUS, and NeuPIMs keep an NPU for compute-intensive prefill GEMM and use PIM for memory-intensive decode GEMV. A survey of integrated NPU-PIM accelerators covers this heterogeneous approach.
• Move compute to a logic die. Duplex, which targets MoE, GQA, and continuous batching, puts 32 GEMM modules on the HBM logic die. Each has 512 FP16 MACs and an 8 KB buffer. Together they consume 17.80 mm², about 14.7% of a 121 mm² HBM3 die. A logic die can afford real GEMM units; the area beside a DRAM bank cannot.
• Build heterogeneous compute inside memory. CENT removes the GPU from a CXL system. HALO splits work by phase, mapping compute-bound prefill to compute-in-memory and memory-bound decode to compute-in-DRAM. Researchers also continue to improve GEMM libraries on real PIM hardware, but shipping bank-level devices still favor GEMV.

Samsung chose near-bank compute to reduce HBM cost with a JEDEC-standard 561-ball package and a conventional memory controller. That package only works if the logic remains small enough to sit beside the banks.

The resulting design targets decode-side GEMV and can ship in a familiar memory package, at the expense of GEMM flexibility. It fits batch-1 local inference on a dense model. MoE, speculative decoding, and batched inference may require a second generation.

llama.cpp
    │
    ├─ today ────────> NPU/GPU <══ 76.8 GB/s ══> passive DRAM
    │
    └─ PIM-aware ────> conventional DRAM controller
                            │
                            └─(commands)─> in-bank MACs @ 614 GB/s

Timeline#

JEDEC published the foundational LPDDR6 standard, JESD209-6, in July 2025. In April 2026, JEDEC said that JC-42.6 was nearing completion of a separate LPDDR6 PIM standard, alongside 512 GB densities, a narrower x6 per-die interface, and a SOCAMM2 module standard. JEDEC has not announced a publication date. Samsung's roadmap presents LPDDR5X-PIM as shipping now and LPDDR6-PIM as moving toward the completed standard.

Samsung has working silicon today, JEDEC has an unfinished standard, and open runtimes still lack the software stack.

Accelerator software often trails hardware by a year or more. PIM reaches into the kernel's page allocator and back through the quantization pipeline, which makes its integration harder than adding another compute backend. Adoption depends on someone building a bank-aware allocator and a re-quantization path in a runtime that people use.

A well-resourced vendor can fund that work. The likely route is a Samsung- or SK hynix-backed runtime fork that later moves upstream, much as vendor GPU backends did.

Why it matters#

Other responses to the memory wall buy bandwidth through wider buses, more channels, HBM stacks, or large unified-memory SoCs. Each solution adds power use, cost, and package complexity.

PIM places computation beside the DRAM banks, giving system designers another way to raise effective bandwidth without widening the memory interface.

Samsung's 3x vendor benchmark is less important than the 614 GB/s inside a 16 GB package. MTP may prove faster or cheaper for some models, and future systems may combine both techniques. PIM matters because it exposes bandwidth that current processors leave inside the DRAM die, giving hardware designers another option when software techniques stop scaling.