Cache (computing): temporary storage to speed data access

The goals of using a cache are to reduce the access time and/or to reduce the number of accesses to a slow background medium. This means in particular that the use of caches is only worthwhile where the access time also has a significant influence on the overall performance. While this is the case, for example, with the processor cache of most (scalar) microprocessors, it does not apply to vector computers, where the access time plays a subordinate role. This is why caches are usually not used there, because they are of little or no benefit.

Another important effect of using caches is the reduction of the necessary data transfer rate to the connection of the background medium (see e.g. memory hierarchy); the background medium can therefore be "connected more slowly", which can result in lower costs, for example. Because the majority of requests can often be answered by the cache ("cache hit", see below), the number of accesses and therefore the necessary transmission bandwidth decreases. For example, a modern microprocessor without a cache would be slowed down even with a very small access time of the main memory by the fact that not enough memory bandwidth is available, because the number of accesses to the main memory and thus the demand on the memory bandwidth would increase greatly due to the omission of the cache.

With CPUs, the use of caches can thus contribute to reducing the Von Neumann bottle neck of the Von Neumann architecture. The execution speed of programs can thus be increased enormously on average.

A disadvantage of caches is the poorly predictable time behavior, since the execution time of an access is not always constant due to cache misses. If the data is not in the cache, the accessor must wait until it has been loaded from the slow background medium. With processors, this often happens when accessing data that has not yet been used or when loading the next program instruction during (long) jumps.

Since it is technically complex and thus usually not economically sensible to build a cache that is both large and fast, one can use several caches - e.g. a small fast cache and a significantly larger but somewhat slower cache (which is still much faster than the background memory to be cached). This allows the competing goals of low access time and large cache size to be achieved together. This is important for the hit rate.

If several caches exist, they form a cache hierarchy, which is part of the memory hierarchy. The individual caches are numbered according to their hierarchy level, i.e. Level1 ‑to Leveln ‑or L1, L2, etc. for short. The lower the number, the closer the cache is to the fast "user"; the lowest number therefore indicates the cache with the fastest access time, which is searched first. If the L1 cache does not contain the required data, the (usually slightly slower, but larger) L2 cache is searched, and so on. This continues until the data is either found in one cache level (a "cache hit", see below) or all caches have been searched without success (a "cache miss", see below). In the latter case, the slow background memory must be accessed.

If a cache hit occurs, for example, in the L3 cache, the requested data is delivered to the accessor and at the same time transferred to the L1 cache; for this, a cache line must give way there, which "sinks" into the L2 cache.

• With an inclusive cache, each cache level is transparent in itself, i.e. a cache line that is in the L1 cache is also present in the L2 and L3 caches. If the cache line is "displaced" from the L1 cache (overwritten with data from another address), nothing else needs to be done - it is still present in the L2 cache (provided no write-back or similar is necessary).
• In an exclusive cache, a cache line of an address exists only once in all cache levels. A cache line for address A in the L1 cache does not also exist in the L2 or L3 cache. If it is displaced from the L1 cache, it can either be discarded completely, or must be explicitly copied to the L2 cache. There, too, a (different) cache line is displaced to make room for the sinking one. This other cache line now sinks into the L3 cache, where a third cache line has to give way.

Exclusive cache hierarchies generate significantly more data traffic between the caches. In return, as many cache lines can be kept available as the sum of the L1, L2, and L3 cache size, whereas with the inclusive cache only the L3 cache size is decisive.

In the hardware area, modern CPUs in particular have two or three cache levels; other devices usually have only one cache level. In the software area, usually only one cache level is used, a prominent exception being web browsers, which use two levels (main memory and hard disk drive).