When CPUs Stopped Scaling: Why Hardware Got Complicated

When I was a kid, computers had a single core. Some did not come with a dedicated graphics card either, relying on integrated graphics built into the motherboard. What I remember most is that they simply got faster on their own. You bought a new machine every few years, everything you already owned ran better on it, and nobody had to explain why.

Two observations explained it. Moore’s Law predicted that the number of transistors on a chip would double roughly every two years. Dennard Scaling observed that as transistors shrank they drew less power, which meant clock speeds could rise alongside the transistor count.

Between them, they gave us performance that arrived whether or not anyone wrote better software. Manufacturers could cram more transistors into a single core, and that was enough to keep up with everything being asked of it.

When the Magic Stopped

In the mid-2000s, the physical limitations of processors began to show. We kept shrinking transistors, but voltage refused to drop enough. This led to two problems. First, making transistors smaller generated more heat in a smaller space, causing overheating of the entire processing chip. Second, electricity started leaking, as it could jump across smaller gaps more easily. Had the voltage continued to drop, the heating would have been manageable and electrical leakage minimised.

However, as this was not possible, processor speeds stalled around 3–5 GHz and desktop power froze near 100 W. We could still add more transistors, just not in a single core. At the same time, what we expected software to do kept growing: HD video streaming, complex 3D games, real-time data over the internet, early machine learning. The old habit of “waiting for a faster CPU” stopped working.

Multi-Core Helped, but Not Everywhere

The obvious quick fix was more cores. Even though we could not put more transistors into a single core, we could add more cores to the chip. This enabled computers to start multitasking. Previously, a computer only did one thing at a time, but could switch between tasks very quickly. So, while it may have looked like it was doing many things at once, it was really just switching between tasks.

With multi-core processors, the computer could now do multiple things simultaneously. But this also meant that software developers had to know how to divide programs effectively across all the cores. For example, a video editing application could use one core to decode video, another to apply effects, and a third to encode the final output. In other cases, it was harder to split the programs into smaller tasks.

Specialised Hardware

Since a program must now be divided into smaller tasks, there is a hard ceiling on what extra cores can buy you. Not everything can be broken into pieces that run at the same time. Some work is inherently sequential and has to happen in order, and that part does not get any faster no matter how many cores you give it. Gene Amdahl worked out the arithmetic for this in 1967, and the ceiling it puts on parallel hardware is a good deal lower than intuition suggests.

So we changed tactics. Graphics Processing Units (GPUs) emerged as a powerful tool for handling specific types of workloads. GPUs handle large amounts of identical maths. Although called Graphics Processing Units, their capabilities extend far beyond just graphics. What they really excel at is processing large blocks of data in parallel. So, if you do the same thing repeatedly, like decoding video or applying filters, a GPU can do it much faster than a CPU, because it can do it multiple times at once.

The CPU did not retire; it became the coordinator. But since GPUs are primarily good at working on large blocks of data, this did not resolve the issue of moving data between different parts of the system. GPUs require data to be read from storage by the CPU, placed into memory, and then processed by the GPU.

This required us to rethink computer architecture once again.

What “Computational Storage” Means

Moving data turns out to cost far more than most of us assume. Colin Scott maintains a website called Latency Numbers Every Programmer Should Know, which lays out the differences in latency between the various places a byte can live.

In 2020, reading a byte already sitting in the CPU’s cache costs on the order of a nanosecond. The same byte in main memory costs around a hundred. From an SSD, 16,000. Those are not gaps you close with a cleverer loop, and they change what is worth doing at all. The cost of moving data is routinely higher than the cost of processing it, and the fastest byte is the one you never move.

Computational Storage is one answer to this. It runs tiny, safe functions inside or right next to the SSD, doing simple but high-value work early: filter, compress, parse, summarise. Instead of pulling a million records up to the CPU in order to keep a few thousand, the SSD hands you the few thousand to begin with.

The example I fall back on when explaining this outside work is a spreadsheet. Imagine an Excel file with a million rows and a filter applied so you only see the thousand or so that matter. What the machine is actually doing is fetching all million and then throwing the other 999,000 away.

Computational Storage allows the system to do the filtering right at the storage level, so only the relevant rows are sent to the CPU in the first place. In essence, Excel asks the SSD to only return the rows that match the filter criteria, limiting the amount of bytes that need to be moved.

The major challenge with Computational Storage is standardisation. Different SSDs have different capabilities, and there is no universal way to express the kinds of computations you might want to run. This fragmentation makes it hard for developers to take advantage of these features without getting locked into a specific vendor’s ecosystem.

Where My Own Work Fits

During my PhD, we built Delilah, a storage device that can run small sandboxed functions. These functions filter, compress, and transform data at the storage level, cutting down what has to travel to the CPU. What made the work distinctive was the choice of eBPF as the language for them.

eBPF was built for network packet filtering and has since grown into a general tool used for security and monitoring as well. Two properties made it the right fit here. It is simple enough to be verified, so the drive can establish before running anything that a function is safe and cannot damage the data or the device. And it is vendor-neutral, so the same code runs on different hardware unchanged. That second property is what addresses the standardisation problem: a developer writes an eBPF function without needing to know whose drive it will end up on.

When eBPF on its own is not fast enough or cannot express the needed functionality, Delilah (or the program itself) can jump to device “registered functions”, which are hardware-specific or hardware-accelerated functions. These functions do not have the limitations of eBPF, as they are written in C by the vendor. While this breaks the vendor-neutral promise to a limited extent, it allows for high-performance operations when needed and access to hardware-specific features.

The thesis taught me something I had not expected going in. You do not win by cramming more raw compute into the drive. Just as with CPUs, you cannot out-compute the cost of moving data. What actually helps is reducing the movement itself, through good interfaces that give you a clean way to request work, and state kept on the device so it can remember and avoid re-reading. Push computation down to storage only where it reliably shrinks what you would otherwise have to move.

ARM’s Role

While GPUs and Computational Storage were helpful in speeding up computers, something else came with them: efficiency. Now that complex work was being done elsewhere, the CPU could focus on coordination and management tasks, optimising its workload and power consumption.

Until recently, almost all computers were equipped with x86 processors from Intel or AMD. These processors were powerful and advanced, but not particularly efficient. They consumed a lot of power and generated a lot of heat. ARM processors, on the other hand, were designed to be efficient and low-power. They were not as powerful as x86 processors, but they were good enough for many tasks including running the operating system and coordinating work.

If you have a MacBook with an M chip, you are already using an ARM processor. Apple switched from Intel x86 processors to their own ARM-based M1 chips in 2020, driven by the need for better performance per watt, which buys longer battery life and quiet operation without fans. My MacBook Air with an M3 chip is the best laptop I have owned, and by a wide margin the quietest. My previous x86 MacBook managed a few hours of battery and a fan that kicked in often; the M3 lasts days on a charge and has no fan at all.

What Developers Should Do

Most developers will not have the time or resources to build an entirely new storage stack for efficiency. Instead, they should focus on optimising their existing systems, with an understanding of the limitations and capabilities of their hardware.

Start by measuring data movement, not just CPU time. The processor is rarely the only bottleneck, and shifting bytes around can cost more than whatever you do with them once they arrive. Are you reading a lot of data? Can it be kept closer to the CPU?

Look for the places where you read mountains of bytes to keep a handful. Are you pulling many rows out of an SQL database only to filter and summarise them afterwards? Push that work into the query instead. It is the same idea as Computational Storage on a different layer: you describe what you want to the server, and only the answer travels back.

Where Data Sits Matters

Where you can, access data in a cache-friendly way. This means thinking about how data is laid out in memory and reading it in an order that suits the CPU cache. If you are processing a large array, walk it linearly rather than jumping around it. A CPU can predict a linear walk and fetch the next stretch before you ask for it.

Below are two nearly identical implementations of a matrix summation function in C:

// Version A: This is a great way to sum a matrix.
int sum_array_rows(int a[M][N])
{
    int i, j, sum = 0;

    for (i = 0; i < M; i++)
        for (j = 0; j < N; j++)
            sum += a[i][j];
    return sum;
}

// Version B: This is a terrible way to sum a matrix.
int sum_array_cols(int a[M][N])
{
    int i, j, sum = 0;

    for (j = 0; j < N; j++)
        for (i = 0; i < M; i++)
            sum += a[i][j];
    return sum;
}

Both produce the same result. The first is far more efficient, because it reads memory in the order memory is actually laid out. A matrix looks two-dimensional in the source code, but the memory holding it is one long line of bytes, so the order you walk it in decides how far the machine has to jump between reads.

Written out for a 4x4 matrix, where each row is stored end to end, the offsets each version touches look like this:

Version A walks straight through, one offset after the next. Version B jumps a full row each time, so the cache line the machine just fetched is thrown away before most of it gets used, and prefetching has nothing to work with.

This is the example I used to put in front of students. I would show both versions and ask which was faster, without having mentioned locality yet. Most said there was no difference, and the reasoning was sound: it is the same arithmetic on the same data, and walking a matrix end to end is O(n) whichever way you walk it. In a higher-level language that is usually where the analysis stops. Complexity analysis does not capture the access pattern at all, and on a large enough matrix the access pattern is what decides the runtime.

I ran both versions over a 13,000 x 13,000 matrix of integers, about 645 MiB, on my MacBook Air with an M3 chip. The array is filled before the clock starts, so no timed run pays for pages being handed out for the first time, and the two loops are exactly the ones above. These are the best of twenty-four runs. The size is deliberate: comfortably past the largest cache on the chip, so the column walk cannot be absorbed by one, but nowhere near large enough for the operating system to page any of it out. A matrix that no longer fits in free memory stops measuring the cache and starts measuring the memory manager.

13000 x 13000 ints, 645 MiB, best of 24

  row-major       0.010 s   0.062 ns/elem
  column-major    0.284 s   1.680 ns/elem
  ratio          27.31x slower

Nothing about the algorithm changed between those two runs. The only difference was the order in which it touched memory.

If you want to check this on your own hardware, the benchmark is here. The ratio you get depends on your cache line size and how much memory bandwidth the machine has, so it is worth running rather than taking my word for it.

The Point

We will not see the old “free-speed-every-two-years” miracle return. That is the outcome of physics, economics, and the way we now build chips. Instead of chasing higher clock rates, we win by putting computation where the data already sits, and by parallelising whatever can be parallelised.

The CPU has not gone anywhere. It still orchestrates threads, schedules I/O, and makes the high-level decisions. What it can no longer be expected to do is the heavy number-crunching, and what replaces it is knowing where your data lives and how much of it has to move.