Sparsity, in-memory and analog compute

Digital accelerators have converged on dense matrix engines fed by a memory hierarchy, and their energy is dominated by data movement: in a classic breakdown, a DRAM access costs about 200× a MAC, a global-buffer access 6×, and a register-file access about 1×.¹ Even with good reuse, Murmann estimates that digital processing elements struggle to go below about 150–200 fJ per 8-bit MAC once local buffer traffic is counted, against about 75 fJ for a 16-bit MAC’s arithmetic alone in 28 nm.¹¹ That gap is what the three ideas here target:

• Sparsity reduces the number of MACs and bytes. The hard part is turning a zero count into wall-clock and energy savings on hardware that wants regular work.
• Compute-in-memory makes the weight-stationary dataflow total: weights never leave the array, so weight-fetch energy goes away and only activations and results move.
• Analog compute replaces adder trees with Kirchhoff summation or charge sharing. It wins only in the low-SNR regime, roughly below 8 bits, because thermal noise makes each extra bit cost about 4× the energy.¹¹

The chapter covers the patterns of sparsity and what hardware exploits, SRAM and resistive CIM, converter overheads, noise and accuracy, and what open prototypes actually measure. The recurring lesson is that the headline mechanism (skip a zero, sum a current) is cheap; the surrounding machinery (indices, load balance, ADCs, calibration) sets the real cost.

Zeros come from two places: static weight sparsity from pruning, and dynamic activation sparsity (ReLU zeros, which an accelerator can exploit by gating the MAC for about 45% energy savings, or by skipping the cycle for a 1.37× throughput gain in the designs Sze et al. survey).¹ Each can be unstructured or structured, and the pattern, more than the zero count, decides whether hardware benefits.

Unstructured: most accurate, least exploitable

Magnitude pruning plus retraining reaches 9–13× parameter reduction on older CNNs², and SparseGPT prunes 175-billion-parameter language models to 60% unstructured sparsity in one shot with negligible perplexity increase.¹⁰ But Mishra et al. note that the performance benefit of such patterns on matrix pipelines is “negligible and at times negative,” even at 95% sparsity.⁴ The obstacles:

1. Metadata. CSR/CSC formats store an index per nonzero; with 8-bit values and 16-bit column indices the overhead is up to 200%.⁴
2. Data-dependent access. Indirection adds latency to matrix reads and wastes cache lines.⁴
3. Imbalance. Per-row nonzero counts vary, so SIMD lanes or array rows idle while the busiest finishes.³

In software on GPUs, the best sparse kernels reached about 27% of a V100’s single-precision peak and gave 1.2–2.1× end-to-end speedups on sparse Transformers and MobileNets, at the moderate sparsity real models tolerate.⁵ Purpose-built hardware does better. EIE stored pruned fully connected layers in compressed sparse column form in on-chip SRAM, skipped zero activations, and attributed 10× of its energy advantage to sparsity.⁷¹ Dataflow designs with distributed SRAM can trigger a MAC only for each nonzero weight that arrives, so zeros are never streamed or stored.⁸

N:M structured sparsity

2:4 keeps at most two nonzeros per group of four along the reduction dimension. The compressed matrix is half the width, plus 2 bits per kept value saying which of the four positions it came from: 12.5% overhead for 16-bit values and 25% for 8-bit.⁴ The sparse matrix unit uses that metadata to select the matching half of the dense operand and runs a GEMM of half the depth, for twice the peak math rate.³ Because every group has the same count, there is no imbalance, and a value’s position follows from the compression rate with no pointer chasing. The recipe is to train dense, prune each group to its two largest magnitudes, and retrain with the original schedule, which matched dense accuracy across the vision, translation and language tasks reported.⁴³

Block and coarse structures

Block sparsity zeroes whole $b \times b$ tiles. A dense engine skips tiles with only a per-block index, and OpenAI’s kernels showed speedups near $1/(1-s)$ over cuBLAS at high sparsity with 32×32 blocks, while generic cuSPARSE was slower than simply multiplying the zeros.⁶ Coarser still, pruning whole channels or filters just yields a smaller dense layer, but larger groups lose more accuracy.¹³

CIM is the extreme of weight stationarity: weights are written once and every MVM reads them in place.¹ Designs split along two axes: the storage device and whether accumulation is analog or digital.

Resistive crossbars (RRAM, PCM, flash)

In a crossbar, row inputs (voltage levels, pulse widths or serial bits) meet a conductance matrix and each column current is $I_j = \sum_i V_i G_{ij}$.¹ Signed weights use a pair of devices, $w \propto G^+ - G^-$; IBM’s PCM chip reports that using two devices per sign improves SNR enough to move from roughly 3-bit-equivalent to 3–4-bit-equivalent weight precision relative to an 8-bit digital engine.¹⁶ The practical limits of large arrays are IR drop along wires (wire energy dominates at around 1k×1k), write energy and multiple programming pulses, and device-to-device and cycle-to-cycle variation with nonlinear conductance.¹ RRAM arrays usually put a series selector transistor in each cell (one-transistor-one-resistor, 1T1R); NeuRRAM integrated 3 million such devices with 130 nm CMOS.¹⁵ PCM drifts after programming (next section).¹⁶ In flash, each transistor acts as a programmable variable resistor holding a weight; Mythic presented a 40 nm design of this kind with 8-bit analog compute at Hot Chips 2018.¹⁸

SRAM CIM: analog and digital

Early analog SRAM CIM drove word lines with a 5-bit DAC and let bit-cell currents discharge the bit line, saving 12× in energy compared with reading binary weights out and computing separately. The bit-line discharge is nonlinear and cells vary, so that design needed a DAC that pre-distorts the input and boosting to combine weak classifiers.¹ Charge-domain designs sum on capacitors instead: each bit cell drives a local capacitor, and charge sharing across the column computes the sum. With about 1% capacitor matching, the cumulative error stays below one input step until $N$ exceeds about 10,000.¹¹ Jia et al. integrated a 590 kb charge-domain array with a CPU and used bit-parallel weights and bit-serial inputs to scale precision, reaching software-level CIFAR-10 accuracy at 1 and 4 bits.¹³

Digital CIM (DIMC) puts bitwise multipliers and adder trees next to the bit cells. It is noise-free and maps more flexibly, and in the Houshmand et al. survey its efficiency depends strongly on technology node and precision. For analog CIM (AIMC), by contrast, a newer node raises compute density but only marginally improves energy efficiency.¹⁴

The system problem

A CIM array computes only matrix-vector products against weights already written into it. Activation functions, products between two activation tensors, and data routing between arrays still need digital logic and buffers; IBM’s speech demonstration ran its vector operations and activations on a host.¹⁷ Retaining the array’s efficiency for a complete accelerator is the open challenge.¹¹ SRAM CIM capacity is small, so large models need off-chip weight buffers, which brings back the data movement CIM was meant to remove; dense non-volatile devices are the proposed fix.¹⁶

Precision has to be assembled from low-precision pieces, because a 16-bit multiply in one cell would need a 16-bit DAC, $2^{16}$ conductance levels and an ADC of more than 16 bits.¹² ISAAC’s scheme, typical of the field:

• Inputs bit-serial: $v = 1$ bit per cycle, so the DAC is an inverter and a 16-bit input takes 16 cycles, combined by shift-and-add. A 2-bit DAC would have added 63% chip area and 7% power for no throughput gain.¹²
• Weights bit-sliced: $w = 2$ bits per cell, so a 16-bit weight spans 8 cells whose column results are shifted and added digitally.¹²
• ADC resolution follows from the column sum: $A = \log_2 R + v + w$, minus 1 if $v$ or $w$ is 1. For $R = 128$ rows, $v = 1$, $w = 2$ this gives 9 bits; storing a column’s weights inverted when that keeps the MSB zero saves one bit, allowing an 8-bit ADC.¹²

Even so, ISAAC’s 96 ADCs per tile took 58% of tile power and 31% of area, and the authors found a 9-bit ADC never worth its cost.¹² Murmann’s amortized model makes the scaling explicit: $E_{\mathrm{MAC}} = E_{\mathrm{ADC}}/N + E_{\mathrm{cell}} + E_{\mathrm{logic}}$, with $E_{\mathrm{ADC}}$ fitted to published converters as $100\,\mathrm{fJ} \cdot \mathrm{ENOB} + 1\,\mathrm{aJ} \cdot 4^{\mathrm{ENOB}}$.¹¹¹⁹ For 4-bit operands over $N = 1152$ rows the predicted MAC energy is 3.8 fJ; beyond 8 bits it exceeds 200 fJ, which digital designs already reach, and above 6–7 bits competing with digital is “challenging.”¹¹ Published macros agree: the AIMC macro with the best peak efficiency, about 1,800 TOP/s/W, got there by optimizing its converters and using a large array so that the array, not the ADCs, dominated power.¹⁴

Alternatives to the classic column ADC trade differently: NeuRRAM integrates voltage-mode neurons that double as the ADC and activation function, avoiding large current-sinking amplifiers¹⁵; IBM’s 64-core PCM chip gives each of a core’s 256 outputs its own time-based current ADC, trimmed once at calibration, for fully parallel MVMs¹⁶; and IBM’s speech chip encodes inputs as pulse durations and uses ramp-based conversion.¹⁷

For a column summing $N$ products with independent per-device error of standard deviation $\sigma$ (as a fraction of the largest weight, the maximum conductance), the error variance is $\sigma^2 \sum x_i^2$ while the signal variance is $E[w^2] \sum x_i^2$, so the relative error $\approx \sigma / \operatorname{rms}(w)$ independent of $N$. Programming error therefore caps SNR no matter how many ADC bits you buy; the simulation below shows the plateau. The sources NeuRRAM lists are IR drop on input wires, drivers and crossbar wires; limited programming resolution; conductance relaxation; capacitive coupling; and ADC resolution and range. Modeling only relaxation and ADC effects overestimated CIFAR-10 accuracy by 2.32 points versus measurement.¹⁵

Drift

PCM drift follows $G(t) = G(t_0)\,(t/t_0)^{-\nu}$. A global per-core rescale of the MVM output corrects the mean, but $\nu$ depends on conductance state and device, so the residual error grows with time.¹⁶ RRAM instead relaxes abruptly within about a second of programming. NeuRRAM measured a relaxation spread near 10% of the maximum conductance, far larger than its write-verify tolerance; three rounds of re-programming cut it by 29%, and hardware-aware training largely mitigates the rest.¹⁵

Mitigations, by layer of the stack

• Device: iterative write-verify; two devices per signed weight to average error.¹⁵¹⁶
• Circuit: per-ADC trimming and on-chip calibration so converters share one linear transfer curve¹⁶; per-layer selection of input amplitude to fill the ADC range.¹⁵
• Training: noise injection using measured device statistics (25.34% → 85.99% CIFAR-10 in NeuRRAM’s evaluation), then chip-in-the-loop progressive fine-tuning (a further 1.99 points), where measured outputs of programmed layers train the layers not yet programmed.¹⁵
• System: keep precision-sensitive operations digital. IBM’s speech demonstration ran vector-vector products, biases, activations and the joint layer in software.¹⁷

Murmann notes that inference tolerates errors (and can be desensitized by noise injection) down to about 4 bits, but training does not; at least 8-bit floats tend to be required.¹¹ That makes analog CIM an inference technology. For reference, IBM’s 64-core chip measured precision equivalent to 3–4-bit weights in an 8-bit-I/O engine.¹⁶

Expert view adds the busiest-row count (load imbalance), a software sparse-kernel bar for unstructured sparsity, a rows-per-column slider $N$, output SNR (shown as equivalent bits), Murmann’s $B + \log_2 \sqrt{N}$ rule, and an ADC energy-per-conversion curve. All constants are illustrative; the ADC energy fit is Murmann’s.

• Find the unstructured sparsity where the index-driven engine first beats 1×, and the one where the software kernel breaks even. Compare with the 1/3 storage break-even.
• With $\sigma = 0$, compare the ADC bits you set with the equivalent bits shown at $N = 64$ and $N = 256$. Why is there a gap (the outputs don’t fill the ADC’s full scale), and how does it compare with the rule of thumb, which is the ADC’s own required ENOB?
• Raise $N$ from 16 to 256 at 10 bits and watch the ADC share of energy per MAC.

Calibrating the figures:

• 2:4 cuts weight storage to 62.5% of dense at 8 bits and 56% at 16 bits (including metadata); measured INT8 GEMM speedups on A100 approach 2× as GEMM-K grows at $M = N = 10{,}240$.⁴
• The Jia et al. figures are 1b-TOPS/W, normalized to 1-bit operations; throughput scales down linearly with the weight and input precisions.¹³ Houshmand et al. avoid bit-normalized metrics because they distort the landscape, and compare designs only at a stated 50% operand sparsity.¹⁴
• The HERMES chip’s 9.76 TOPS/W is for 1-phase MVMs at full utilization; the accuracy experiments used a higher-precision 4-phase read mode.¹⁶
• The speech system’s figures are estimates for the five analog chips plus modeled digital work: 6.94 TOPS/W with conventional weight mapping, where the analog-to-digital operation ratio was 325:1 (vector ops and activations in software), and 6.7 TOPS/W with the weight expansion that gave the 9.258% word error rate.¹⁷ Its 14× is a projection, relative to the best MLPerf energy-efficiency submission for that task.¹⁷

Sparsity: accuracy, pattern, speed

At equal sparsity, accuracy loss rises from unstructured to N:M to block to channel, while exploitability rises in the same order.¹⁴ 2:4 sits at a sweet spot because its constraint is local enough for retraining to absorb, which the A100 table shows: dense and 2:4 accuracy match within noise across ResNet, Mask R-CNN, GNMT, Transformer-XL and BERT.³ Its limits: it is fixed at 50%, its gain is in math only, and it applies to one operand (the second matrix and the output stay dense).⁴ Block sparsity typically recovers accuracy by enlarging the model, which is useful only when the larger dense model would have been impractical.⁴ Activation sparsity is known only at run time, so hardware must detect it on the fly, as the zero-gating and zero-skipping designs do, and its benefit varies with the input.¹

CIM: density, flexibility, utilization

• Array utilization. Large arrays amortize converters but are underused by depthwise and pointwise layers; small arrays map well but pay peripheral overhead, and many small macros raise feature-map traffic.¹⁴
• Capacity. Current SRAM CIM designs cannot hold more than about 10 MB of weights at reasonable area, so larger networks need off-chip weight buffers.¹⁶
• Write cost. Resistive devices need multiple programming pulses and have finite endurance, which rules out frequently rewritten weights.¹
• Programmability. Few analog CIM designs can be reconfigured for diverse models; NeuRRAM’s transposable array is one attempt.¹⁵

Analog: where the efficiency goes

Every analog design meets the same three-way tension. More rows per conversion amortize the ADC but grow the required ENOB by $\log_2 \sqrt{N}$ and worsen IR drop. More bits per cell or per input pulse cut cycles but exponentially shrink allowable rows (ISAAC’s $A = \log_2 R + v + w$). Higher precision costs 4× per bit in the thermal-noise regime.¹²¹¹ Murmann’s caution about binarized arrays is instructive: mixed-signal demonstrators reach about 4 fJ/MAC, but digital designs with similar architecture and dataflow get close, so mixed-signal is not necessarily a game changer for binarized arrays, and the case for it rests on multi-bit operands.¹¹

Five models, each simple enough to check by hand; the simulation implements most of them.

1. Sparse speedup on parallel hardware

Let $d$ be the density (1 − sparsity). For a dense engine, multiply time is unchanged: $T = T_{\mathrm{dense}}$. A software sparse kernel that reaches a fraction $\eta$ of dense peak gives speedup $S = \eta/d$, so it breaks even only at $d = \eta$. Gale et al.’s kernels reached 27% of single-precision peak on their best problems; taking $\eta \approx 0.27$ as a best case, this simple model needs more than about 73% sparsity.⁵ On hardware with $L$ lanes each handling one row, the step time is set by the busiest row:

where $C$ is the row length and $o$ the per-nonzero index-decoding overhead. Imbalance (max vs. mean nnz) and $o$ both eat into $1/d$. The simulation uses $C = 16$ and $o = 0.25$ (illustrative). For 2:4, $\mathrm{nnz}_r = C/2$ for every row and $o$ is absorbed into the multiplexer, so $S = 2$ exactly on the math pipeline.³ For blocks, $S = n_{\mathrm{blocks}} / \bigl(n_{\mathrm{nonzero}}\,(1 + o_{\mathrm{b}})\bigr)$, approaching $1/d$.⁶ In every case the end-to-end gain obeys Amdahl’s law over the non-GEMM and memory-bound parts.

2. Storage arithmetic

With $b$-bit values and an $i$-bit index per nonzero, storage relative to dense is $d\,(b + i)/b$.

• Unstructured, $b = 8$, $i = 16$: $3d$, so it saves memory only for $d < 1/3$.⁴
• 2:4, $b = 16$: $(2 \cdot 16 + 2 \cdot 2)/(4 \cdot 16) = 36/64$, about 44% saved; $b = 8$: $20/32$, about 38% saved.⁴
• Block $b_{\mathrm{s}} \times b_{\mathrm{s}}$ with one $i$-bit index per kept block: $d\,\bigl(1 + i/(b\, b_{\mathrm{s}}^2)\bigr)$.

3. Pruning recipes

1. Train–prune–retrain (Han et al.): train dense, remove weights below a magnitude threshold, retrain the survivors; iterate for higher sparsity.²
2. 2:4 workflow (Mishra et al.): train dense; in each group of four, zero the two smallest magnitudes; retrain with the same optimizer and learning-rate schedule as the original run, trading one extra training run for no hyperparameter search.⁴
3. One-shot, second-order (SparseGPT): prune layer by layer, solving a reconstruction problem so the remaining weights compensate, with no retraining; scales to 175B parameters and supports 2:4 and 4:8.¹⁰

4. Crossbar MVM, converters and SNR

Ideal crossbar: $I_j = \sum_i V_i \left(G^+_{ij} - G^-_{ij}\right)$.¹¹⁶ With bit-serial $v$-bit inputs and $w$-bit cells across $R$ rows, the column needs $A = \log_2 R + v + w$ $(-1 \text{ when } v \text{ or } w = 1)$ bits to be lossless.¹² In practice designs choose fewer, following an SNR argument. Murmann assumes uniform activations quantized to $B$ bits; the quantization noise of $N$ accumulated products sets an $\mathrm{SNR}_{\mathrm{sig}} = 2^{2B}$, and the ADC’s input-referred noise must be $k^2$ times smaller. For a full-scale range $\mathrm{FS}$, that gives¹¹

With $k = 2$ and $\mathrm{FS} = 0.5$ this is $B + \log_2 \sqrt{N}$, about 5 extra bits for $N = 1152$. Energy per MAC is then¹¹

with $k_1 = 100\,\mathrm{fJ}$ and $k_2 = 1\,\mathrm{aJ}$. Murmann fits this to published ADC measurements; the linear term covers the part of converter energy that grows with each added bit, and the $4^{\mathrm{ENOB}}$ term is the thermal noise limit: noise power is $kT/C$, so halving the step (one more bit) needs 4× the capacitance and 4× the energy.¹¹ At $\mathrm{ENOB} = 8$ the two terms are 800 fJ and 66 fJ; at 12 bits, 1.2 pJ and 16.8 pJ. The simulation uses this fit with $b = \mathrm{ENOB}$, adds an illustrative DAC term (44 fJ/bit × 4 bits shared by 64 columns, a per-bit figure Houshmand et al. fitted to published macros¹⁴) and 1 fJ per cell.

Device error adds independently: for per-device error $\sigma$, relative output error $\approx \sigma / \operatorname{rms}(w)$, so the output SNR is bounded by roughly $20 \log_{10}\!\bigl(\operatorname{rms}(w)/\sigma\bigr)\,\mathrm{dB}$ however fine the ADC. Effective bits follow from $\mathrm{SNR} = 6.02 \cdot \mathrm{ENOB} + 1.76\,\mathrm{dB}$. Combining the two error sources, $\text{total error}^2 \approx \sigma_{\mathrm{dev}}^2 + \sigma_{\mathrm{ADC}}^2$; the ADC should be sized so its term is a fraction of the device term, not driven to zero. That is the knee you can find in the simulation.

5. Drift compensation

For PCM, $G(t) = G(t_0)\,(t/t_0)^{-\nu}$. Global drift compensation rescales each core’s MVM results by an appropriate factor, removing the mean drift. The residual comes from the spread of $\nu$ across devices and states, and it grows over time.¹⁶