Computer Architecture

Module 4:Cache Performance

 Introduction to Cache
Memory
● Cache is a small, fast memory located
close to the CPU
● Stores frequently accessed data and
instructions
● Bridges the speed gap between CPU and
main memory
● How familiar are you with cache memory?
 The Need for Cache
● CPU speeds have increased faster than memory speeds
● Memory access can be a bottleneck in system performance
● Cache reduces average memory access time
● Why do you think memory access is slower than CPU
operations?
Cache Hierarchy

● Multiple levels of cache: L1, L2, L3

● L1 is smallest and fastest, closest to CPU
● L2 and L3 are larger but slower
● Can you guess why we have multiple
levels of cache?
 Locality of Reference
● Temporal locality: recently accessed data likely to be accessed
again
● Spatial locality: nearby data likely to be accessed
● Cache exploits these principles for better performance
● Can you think of examples of temporal and spatial locality in your
daily life?
 Cache Hit and Miss

● Cache hit: requested data found in cache

● Cache miss: data not in cache, must be
fetched from main memory
● Hit rate: percentage of memory accesses
found in cache
● How might a high miss rate affect system
performance?
 Types of Cache Misses
● Compulsory miss: first access to a memory block
● Capacity miss: cache is full and must evict data
● Conflict miss: multiple memory blocks map to same cache line
● Which type of miss do you think is hardest to avoid?
Cache Mapping
Techniques
● Direct mapping: each memory block has
one possible cache location
● Fully associative: memory block can be
placed anywhere in cache
● Set associative: compromise between
direct and fully associative
● What are the pros and cons of each
mapping technique?
 Replacement Policies
● Least Recently Used (LRU): replace least recently accessed
block
● First-In-First-Out (FIFO): replace oldest block
● Random: randomly select block for replacement
● Which policy do you think might work best? Why?
 Write Policies
● Write-through: update both cache and main memory immediately
● Write-back: update cache, mark block as dirty, update memory
later
● Write allocate vs. No-write allocate for cache misses
● How might these policies affect system performance?
 Cache Coherence

● Issue in multi-processor or multi-core

systems
● Ensures all copies of data in different
caches are consistent
● Protocols: MESI, MOESI, MESIF
● Why is cache coherence crucial in modern
computer systems?
 Cache Line Size
● Typical sizes range from 32 to 128 bytes
● Larger lines exploit spatial locality better
● But may increase miss penalty and conflict misses
● How would you decide on an optimal cache line size?
Cache Associativity

● Higher associativity reduces conflict

misses
● But increases hardware complexity and
access time
● Common associativities: 2-way, 4-way, 8-
way
● Can you explain why higher associativity
might slow down cache access?
 Prefetching
● Technique to fetch data into cache before it's needed
● Can be hardware-based or software-controlled
● Improves performance but may cause cache pollution
● What types of applications might benefit most from prefetching?
 Virtual Memory and Cache
● Virtual addresses must be translated to physical addresses
● Translation Lookaside Buffer (TLB) caches address translations
● Virtually indexed, physically tagged caches
● How does virtual memory impact cache design and
performance?
 Cache Performance
Metrics
● Average Memory Access Time (AMAT)
● Miss rate and miss penalty
● Hit time and bandwidth
● How would you use these metrics to
compare different cache designs?
 Optimizing Cache Performance
● Increase cache size and associativity
● Improve replacement and prefetching algorithms
● Optimize software for better cache utilization
● What software techniques might improve cache performance?
 Cache in Modern Processors
● Multi-level cache hierarchies
● Separate instruction and data caches
● Shared last-level caches in multi-core processors
● How has cache design evolved with modern processor
architectures?
 Future of Cache Design
● 3D stacked caches
● Non-volatile memory technologies
● Machine learning-based cache management
● What challenges do you foresee in future cache designs?
 Conclusion
● Cache is crucial for bridging the CPU-memory performance gap
● Involves complex trade-offs in design and implementation
● Continuous area of research and optimization
● How might cache design evolve to meet future computing
needs?
 Cache Inclusivity vs.
Exclusivity
● Inclusive: Lower level caches contain
copies of higher level cache data
● Exclusive: Each cache level contains
unique data
● Trade-offs between redundancy and hit
rate
● How might inclusivity or exclusivity affect
multi-core systems?
 Cache Partitioning
● Technique to divide cache among different processes or cores
● Reduces interference between workloads
● Can improve performance and predictability
● What scenarios might benefit most from cache partitioning?
Non-Uniform Cache
Access (NUCA)
● Large caches divided into banks with
varying access latencies
● Closer banks have lower latency than
farther ones
● Improves scalability of large caches
● How does NUCA relate to the concept of
locality?
 Cache Compression
● Storing compressed data in cache to increase effective capacity
● Trade-off between compression/decompression overhead and
capacity gain
● Various algorithms: frequent pattern compression, zero-value
compression
● Can you think of data types that might compress well in cache?
 Victim Caches

● Small fully-associative cache between

main cache and next level
● Stores recently evicted cache lines
● Reduces conflict misses in direct-mapped
or low associativity caches
● How might a victim cache improve
performance in a system with limited
associativity?
 Cache Oblivious Algorithms
● Algorithms designed to perform well without knowledge of cache
parameters
● Exploit locality of reference inherently
● Examples: cache-oblivious matrix multiplication, sorting
● Why might cache-oblivious algorithms be preferable in some
situations?
 Lockdown and Scratchpad Memory
● Lockdown: Pinning critical data in cache
● Scratchpad: Software-managed on-chip memory
● Used in real-time and embedded systems for predictability
● How do these techniques differ from traditional caching?
 Cache Side-Channel
Attacks
● Exploiting cache behavior to extract
sensitive information
● Examples: Flush+Reload, Prime+Probe
attacks
● Implications for security in shared cache
environments
● What security measures might help
mitigate cache side-channel attacks?
 Dynamic Cache Reconfiguration
● Adapting cache parameters at runtime
● Can adjust size, associativity, or replacement policy
● Responds to changing workload characteristics
● What challenges might arise in implementing dynamic
reconfiguration?
 Cache-Aware Operating Systems
● OS designs that consider cache behavior in scheduling and
memory management
● Cache-aware process scheduling and page coloring
● Optimizing system calls and context switches for cache
performance
● How might cache-awareness impact OS design decisions?
 Heterogeneous Cache
Architectures
● Combining different types of memory in
the cache hierarchy
● Example: SRAM for lower levels, eDRAM
or MRAM for higher levels
● Balances performance, power
consumption, and cost
● What are the potential advantages and
challenges of heterogeneous caches?
 Cache Coherence in GPUs
● Challenges in maintaining coherence across thousands of cores
● Techniques: scopes, release consistency, software-managed
coherence
● Impact on programming models and performance
● How does GPU cache coherence differ from CPU cache
coherence?
 Persistent Caches
● Caches that maintain data across power cycles
● Uses non-volatile memory technologies
● Potential for instant-on devices and improved energy efficiency
● What applications might benefit most from persistent caches?
Cache Modeling and
Simulation
● Tools for analyzing and optimizing cache
performance
● Trace-driven vs. execution-driven
simulation
● Popular simulators: gem5, SimpleScalar,
DineroIV
● How might cache simulation inform
hardware design decisions?
 Machine Learning for Cache Management

● Using ML algorithms to predict and optimize cache behavior

● Applications: prefetching, replacement policies, partitioning
● Potential for adaptive, workload-specific optimizations
● What challenges might arise in applying ML to cache
management?
 Near-Data Processing
● Performing computations close to where data is stored
● Reduces data movement and improves energy efficiency
● Implications for cache hierarchy and memory system design
● How might near-data processing change traditional cache
architectures?
 Cache-Conscious
Data Structures
● Designing data structures to optimize
cache utilization
● Examples: cache-oblivious B-trees,
memory-aligned structures
● Impact on algorithm performance and
software optimization
● Can you think of a common data structure
that could be made more cache-
conscious?
 Quantum Computing and Caching
● Challenges in applying classical caching concepts to quantum
systems
● Quantum memory hierarchies and coherence issues
● Potential for quantum-inspired classical caching techniques
● How might caching principles evolve in the era of quantum
computing?
 Future Directions in Cache Research
● Neuromorphic computing and brain-inspired caching
● Integration with emerging memory technologies (e.g.,
memristors)
● Caching for domain-specific architectures and accelerators
● What do you think will be the most significant challenge in future
cache design?
 Conclusion: The Evolving Role of Caches

● Caches remain crucial for bridging performance gaps

● Increasing complexity and specialization in cache design
● Interdisciplinary nature of modern cache research
● How do you envision caches adapting to future computing
paradigms?