AlegsaOnline.com

Cache (computing): temporary storage to speed data access

Fast, temporary storage holding copies of data to reduce latency. Explains types, organization, replacement and write policies, performance metrics, locality, uses, and distinctions from buffers.

Author: Leandro Alegsa Creation date: November 13, 2022 Last updated: May 13, 2026

Overview

A cache in computing is a small, fast storage layer that retains copies of data originally kept in slower, larger storage. By keeping frequently accessed or recently computed items closer to the processor or application, a cache reduces the average time and cost to obtain information. A cache can hold results of computations, disk blocks, web responses, translated memory pages or other data that are expensive or slow to fetch from their authoritative source. When a requested item is present in the cache this is called a cache hit; when it is not, the system must fetch the item from the slower source and often place a copy into the cache, which is called a cache miss.

Key characteristics and components

Caches are defined by a few basic properties: capacity (size), granularity (the size of the stored unit, such as bytes or blocks), lookup method (how the cache locates an item), replacement policy (how it chooses what to evict when full), and write policy (how and when modified data are propagated back to the original store). Typical caches operate on the principle of locality: temporal locality (recently used items are likely to be used again) and spatial locality (items near recently used data are likely to be used soon). These patterns make caching effective in many layers of computing.

Common types of caches

CPU caches: Small, very fast caches located on or near the processor (L1, L2, L3 levels) that store instructions and data to reduce main memory accesses.
Disk and file-system caches: Memory buffers that cache disk blocks or file contents so that repeated reads do not require slow disk access.
Database caches: In-memory structures that keep frequently queried rows, pages or query results to accelerate database operations.
Web caches: Proxies, browsers and content delivery networks that store HTTP responses to reduce latency and bandwidth usage for subsequent clients.
Application-level caches: Framework or library-managed caches (in-memory key-value stores, memoization of function results) used to avoid recomputation or remote calls.

Organization and policies

Lookup structures vary from simple hash tables to multi-way associative or set-associative organizations in hardware caches. Replacement policies decide which entry to evict when space is needed; common strategies include least recently used (LRU), first-in-first-out (FIFO), least frequently used (LFU) and randomized eviction. Write policies determine when changes in the cache are written back to the backing store: write-through updates the backing store immediately, while write-back defers the update until eviction. Systems also implement coherence and invalidation mechanisms when multiple caches can hold the same underlying data.

Performance and metrics

Cache effectiveness is measured by hit rate (fraction of accesses served by the cache), miss rate, and average access latency. Misses are often classified as compulsory (first-time access), capacity (cache too small), or conflict (mapping constraints cause eviction despite available space). Designers balance cache size, associativity, and lookup latency: larger caches can hold more items but may cost more or take longer to search, while smaller caches are faster but miss more often. Simulation and profiling are common ways to evaluate caching strategies for specific workloads.

History, rationale and uses

The idea of caching arises from the need to bridge performance gaps between fast and slow layers of a system. Memory hierarchies and caches grew in prominence as processor speeds outpaced memory and storage access times. Today, caching underpins many optimizations: speeding program execution, reducing network traffic, enabling scalable web services, and improving responsiveness in interactive applications. Strategies such as memoization cache computed function outputs, while CDNs cache static web assets close to users to reduce latency.

Distinctions and notable considerations

Caches differ from buffers: a buffer temporarily holds data while it is being moved or processed and is typically managed explicitly by an application, whereas a cache transparently holds copies to speed repeated access. Caching can introduce complexity: stale data, consistency issues, and added memory pressure. Systems use time-to-live (TTL), validation headers, explicit invalidation or coherence protocols to reduce inconsistencies. Security and privacy also matter: caches can leak sensitive information if not carefully controlled, and side-channel attacks sometimes exploit cache behavior.

Benefit

The goals of using a cache are to reduce the access time and/or to reduce the number of accesses to a slow background medium. This means in particular that the use of caches is only worthwhile where the access time also has a significant influence on the overall performance. While this is the case, for example, with the processor cache of most (scalar) microprocessors, it does not apply to vector computers, where the access time plays a subordinate role. This is why caches are usually not used there, because they are of little or no benefit.

Another important effect of using caches is the reduction of the necessary data transfer rate to the connection of the background medium (see e.g. memory hierarchy); the background medium can therefore be "connected more slowly", which can result in lower costs, for example. Because the majority of requests can often be answered by the cache ("cache hit", see below), the number of accesses and therefore the necessary transmission bandwidth decreases. For example, a modern microprocessor without a cache would be slowed down even with a very small access time of the main memory by the fact that not enough memory bandwidth is available, because the number of accesses to the main memory and thus the demand on the memory bandwidth would increase greatly due to the omission of the cache.

With CPUs, the use of caches can thus contribute to reducing the Von Neumann bottle neck of the Von Neumann architecture. The execution speed of programs can thus be increased enormously on average.

A disadvantage of caches is the poorly predictable time behavior, since the execution time of an access is not always constant due to cache misses. If the data is not in the cache, the accessor must wait until it has been loaded from the slow background medium. With processors, this often happens when accessing data that has not yet been used or when loading the next program instruction during (long) jumps.

Cache Hierarchy

Since it is technically complex and thus usually not economically sensible to build a cache that is both large and fast, one can use several caches - e.g. a small fast cache and a significantly larger but somewhat slower cache (which is still much faster than the background memory to be cached). This allows the competing goals of low access time and large cache size to be achieved together. This is important for the hit rate.

If several caches exist, they form a cache hierarchy, which is part of the memory hierarchy. The individual caches are numbered according to their hierarchy level, i.e. Level1 ‑to Leveln ‑or L1, L2, etc. for short. The lower the number, the closer the cache is to the fast "user"; the lowest number therefore indicates the cache with the fastest access time, which is searched first. If the L1 cache does not contain the required data, the (usually slightly slower, but larger) L2 cache is searched, and so on. This continues until the data is either found in one cache level (a "cache hit", see below) or all caches have been searched without success (a "cache miss", see below). In the latter case, the slow background memory must be accessed.

If a cache hit occurs, for example, in the L3 cache, the requested data is delivered to the accessor and at the same time transferred to the L1 cache; for this, a cache line must give way there, which "sinks" into the L2 cache.

With an inclusive cache, each cache level is transparent in itself, i.e. a cache line that is in the L1 cache is also present in the L2 and L3 caches. If the cache line is "displaced" from the L1 cache (overwritten with data from another address), nothing else needs to be done - it is still present in the L2 cache (provided no write-back or similar is necessary).
In an exclusive cache, a cache line of an address exists only once in all cache levels. A cache line for address A in the L1 cache does not also exist in the L2 or L3 cache. If it is displaced from the L1 cache, it can either be discarded completely, or must be explicitly copied to the L2 cache. There, too, a (different) cache line is displaced to make room for the sinking one. This other cache line now sinks into the L3 cache, where a third cache line has to give way.

Exclusive cache hierarchies generate significantly more data traffic between the caches. In return, as many cache lines can be kept available as the sum of the L1, L2, and L3 cache size, whereas with the inclusive cache only the L3 cache size is decisive.

In the hardware area, modern CPUs in particular have two or three cache levels; other devices usually have only one cache level. In the software area, usually only one cache level is used, a prominent exception being web browsers, which use two levels (main memory and hard disk drive).

Questions and Answers

Q: What is caching?

A: Caching is a term used in computer science that refers to the practice of storing copies of data that is used often in order to access it faster than re-fetching or re-calculating the original data.

Q: How does caching work?

A: Caching works by using two kinds of storage media, one which is usually quite large but slow to access, and another which can be accessed much faster but generally smaller. The idea behind caching is to use the fast medium to store copies of data so that accessing the original data takes less time or is less expensive.

Q: What is a buffer?

A: A buffer is similar to a cache in that it stores copies of data for quicker access, however with a buffer, the client accessing the data knows there is a buffer and it's managed by an application whereas with a cache, clients need not be aware there's a cache.

Q: What does locality of reference mean?

A: Locality of reference means that when an application accesses certain blocks of structured data, they are also likely to access other blocks close to those originally accessed. This helps caches work well as they are typically small compared to all available data.

Q: Why do bigger caches take longer to lookup entries?

A: Bigger caches take longer because they contain more stored information and therefore require more time for lookups. They are also more expensive as they require more resources for storage.

Q: How can locality help make caches work better?

A: Locality helps make caches work better because when applications access certain blocks of structured data, they are likely also going to need other nearby blocks which can then be quickly retrieved from the cache instead of having to fetch them from elsewhere or recalculate them again.

Author

AlegsaOnline.com - Cache (computing) - Leandro Alegsa

Creation date: November 13, 2022
Last updated: May 13, 2026

URL: https://en.alegsaonline.com/art/15884