Lecture No.
3
Parallel Computing Memory Models
Memory Hierarchy
CPU
Registers
Cache Memory
RAM
Hard disk
Flash, CD-ROM, BIOS, Tape Drive, Etc
Memory Hierarchy
● the memory hierarchy
separates computer storage
into a hierarchy based on
response time.
●
● Since response time,
complexity, and capacity are
related, the levels may also be
distinguished by their
performance and controlling
technologies.
Memory Models in Parallel Systems
● There are two types of memory models in parallel computers:
○ Distributed memory systems
○ Shared memory systems
And can also be a combination of
Distributed-shared memory model
● Hybrid Memory Model
Distributed Memory Model/Architecture
● distributed memory refers to
a multiple-processor computer system in
which each processor has its own
private memory.
● Computational tasks can only operate on
local data, and if remote data is required, the
computational task must communicate with
Distributed Memory Model/Architecture
Distributed Memory Model/Architecture
With Multiprocessors
Shared Memory Model
Distributed and Shared memory Differences
● shared memory architecture (SMA) refers to
a multiprocessing design where several processors
access globally shared memory.
● Shared memory architectures may use:
○ Uniform Memory Access (UMA): all the processors share the
physical memory uniformly.
○ Non-Uniform Memory Access (NUMA): memory access time
depends on the memory location relative to a processor.
○ Cache-only memory architecture (COMA): the local
memories for the processors at a node is used as cache.
● In a SMA system processors communicate by reading and writing memory
locations.
● The two key problems in scaling an SMA system are:
○ performance degradation due to "contention" when several processors try to access the
same memory location.
○ lack of "coherence" if memory is cached and goes out of synch with the original values as
modifications take place.
Cache Coherence
● In a shared memory multiprocessor with a
separate cache memory for each processor , it is
possible to have many copies of any one instruction
or operand :
○ one copy in the main memory and one in each cache memory.
● When one copy of an operand is changed, the other
copies of the operand must be changed also.
● Cache coherence is the technique that ensures that
changes in the values of shared operands are
propagated throughout the system in a timely fashion.
Cache Coherence
Lecture No. 3
Parallel Computing Memory Models
Memory Hierarchy
CPU
Registers
Cache Memory
RAM
Hard disk
Flash, CD-ROM, BIOS, Tape Drive, Etc
Memory Hierarchy
● the memory hierarchy
separates computer storage
into a hierarchy based on
response time.
●
● Since response time,
complexity, and capacity are
related, the levels may also be
distinguished by their
performance and controlling
technologies.
Memory Models in Parallel Systems
● There are two types of memory models in parallel computers:
○ Distributed memory systems
○ Shared memory systems
And can also be a combination of
Distributed-shared memory model
● Hybrid Memory Model
Distributed Memory Model/Architecture
● distributed memory refers to
a multiple-processor computer system in
which each processor has its own
private memory.
● Computational tasks can only operate on
local data, and if remote data is required, the
computational task must communicate with
Distributed Memory Model/Architecture
Distributed Memory Model/Architecture
With Multiprocessors
Shared Memory Model
Distributed and Shared memory Differences
● shared memory architecture (SMA) refers to
a multiprocessing design where several processors
access globally shared memory.
● Shared memory architectures may use:
○ Uniform Memory Access (UMA): all the processors share the
physical memory uniformly.
○ Non-Uniform Memory Access (NUMA): memory access time
depends on the memory location relative to a processor.
○ Cache-only memory architecture (COMA): the local
memories for the processors at a node is used as cache.
● In a SMA system processors communicate by reading and writing memory
locations.
● The two key problems in scaling an SMA system are:
○ performance degradation due to "contention" when several processors try to access the
same memory location.
○ lack of "coherence" if memory is cached and goes out of synch with the original values as
modifications take place.
Cache Coherence
● In a shared memory multiprocessor with a
separate cache memory for each processor , it is
possible to have many copies of any one instruction
or operand :
○ one copy in the main memory and one in each cache memory.
● When one copy of an operand is changed, the other
copies of the operand must be changed also.
● Cache coherence is the technique that ensures that
changes in the values of shared operands are
propagated throughout the system in a timely fashion.
Cache Coherence
● There are three distinct levels of cache coherence:
○ Every write operation appears to occur instantaneously.
○ All processes see exactly the same sequence of changes of values for each separate
operand.
○ Different processes may see an operand assume different sequences of values. (This is
considered noncoherent behavior.)
Some Examples of Shared memory
systems
Symmetric multiprocessing (SMP) Systems:
● It is a hardware and software architecture where two or more
identical processors connect to a single, shared main memory,
have full access to all I/O devices, and are controlled by a
single operating system instance that treats all processors
equally, reserving none for special purposes.
● Most multiprocessor systems today use an SMP architecture.
● In the case of multi-core processors, the SMP architecture
applies to the cores, treating them as separate processors.
● SMP (symmetric multiprocessing) is the processing of programs by
multiple processors that share a common operating system and memory.
In symmetric (or "tightly coupled") multiprocessing, the processors share
memory and the I/O bus or data path.
● A single copy of the operating system is in charge of all the processors.
● SMP systems are tightly coupled MIMD multiprocessor systems with a pool
of homogeneous processors running independently, each processor
executing different programs and working on different data and with
capability of sharing common resources (memory, I/O device, interrupt
system and so on) and connected using a system bus or a crossbar.
● SMP systems have centralized shared memory called main
memory (MM) operating under a single operating system with two or
more homogeneous processors.
● Usually each processor has an associated private high-speed memory
known as cache memory (or cache) to speed up the main memory
data access and to reduce the system bus traffic.
● Processors may be interconnected using buses, crossbar
switches or on-chip mesh networks.
● The bottleneck in the scalability of SMP using buses or crossbar
switches is the bandwidth and power consumption of the interconnect
among the various processors, the memory, and the disk arrays.
Multi-core processor
● A multi-core processor is a single computing component
with two or more independent actual processing
units (called "cores"), which are the units that read and
execute program instructions.
● The instructions are ordinary CPU instructions such as add,
move data, and branch, but the multiple cores can run
multiple instructions at the same time, increasing overall
speed for programs amenable to parallel computing.
●
● Manufacturers typically integrate the cores onto a
single integrated circuit die (known as a chip multiprocessor
or CMP), or onto multiple dies in a single chip package.
● A multi-core processor implements multiprocessing in a single
physical package.
Designers may couple cores in a multi-core device tightly or loosely.
● For example, cores may or may not share caches, and they may
implement message passing or shared-memory inter-core
communication methods.
● Common network topologies to interconnect cores include bus, ring,
two-dimensional mesh, and crossbar. (Will be discussed later)
● Homogeneous multi-core systems include only identical
cores, heterogeneous multi-core systems have cores that are not
identical.
● Just as with single-processor systems, cores in multi-core systems
may implement architectures such as VLIW, superscalar, vector,
or multithreading.
Some Details Regarding Multicore
Systems
● Processors were originally developed with only one core.
● In the mid 1980s Rockwell International manufactured versions of
the 6502 with two 6502 cores on one chip as the R65C00, R65C21,
and R65C29, sharing the chip's pins on alternate clock phases.
● Other multi-core processors were developed in the early 2000s
by Intel, AMD and others.
● Multi-core processors may have two cores (dual-core CPUs, for
example, AMD Phenom II X2 and Intel Core Duo), three cores
(tri-core CPUs, for example, AMD Phenom II X3), four cores
(quad-core CPUs, for example, AMD Phenom II X4,
Intel's i5 and i7 processors), six cores (hexa-core CPUs, for
example, AMD Phenom II X6 and Intel Core i7 Extreme Edition
980X), eight cores (octa-core CPUs, for example, Intel Xeon
E7-2820 and AMD FX-8350), ten cores (deca-core CPUs, for
example, Intel Xeon E7-2850), or more.
An AMD Athlon X2 6400+ dual-core processor.
Assignment No. 2
● Discuss about different techniques to solve cache coherence
problems in Shared memory Systems.
●
● Write down a comprehensive note about the difference between
multi-core and multi-processing systems.
Examples of Distributed memory
Systems
Computer cluster:
A computer cluster consists of a set of loosely or tightly
connected off-the-shelf computers or workstations that work together so
that, in many respects, they can be viewed as a single system.
● Unlike grid computers, computer clusters have each node set to perform the
same task, controlled and scheduled by software.
● The components of a cluster are
usually connected to each other
through fast local area
networks ("LAN"), with
each node (computer used as a
server) running its own instance of
View of a computer cluster at the Chemnitz University
of Technology, Germany
● The computer clustering approach usually
(but not always) connects a number of readily
available computing nodes (e.g. personal
computers) via a fast local area network.
● The activities of the computing nodes are
orchestrated by "clustering middleware", a
software layer that sits atop the nodes and
allows the users to treat the cluster as by and
large one cohesive computing unit, e.g. via
● A computer cluster may be a simple two-node system which just connects
two personal computers, or may be a very fast supercomputer.
● A basic approach to building a hhhhjhjhis that of a Beowulfcluster which
may be built with a few personal computers to produce a cost-effective
alternative to traditional high pehhhrformance computing.
● An early project that showed the viability of the concept was the
133-node Stone Soupercomputer.
● The developers used Linux, the Parallel Virtual Machine toolkit (PVM) and
the Message Passing Interface (MPI) library to achieve high performance at
a relatively low cost.
●