Chapter 9

Multiple Core Computers

1
 Topics
9.1 Multiple core organization

9.2 Graphics Processing Unit (GPU) design

9.3 Usage of GPU as a co-processor

2
9.1 Multiple core organization
• A multiple core or multi-core organization is one that combines two
or more independent processors into a single package, often a single
integrated circuit (IC).
• A dual-core device contains two independent microprocessors.
• In general, multi-core organization allow a computing device to
exhibit some form of thread-level parallelism (TLP) without including
multiple microprocessors in separate physical packages.

3
Major Technology Providers
• The latest versions of many architectures use multi-core, including PA-RISC (PA-
8800), IBM POWER (POWER7), SPARC (UltraSPARC IV), and various processors
from Intel and AMD.
• There is some controversy as to whether multiple cores on a chip is the same
thing as multiple processors. Major technology providers are divided on this
issue.
• IBM considers its dual-core POWER4 and POWER5 to be two processors, just
packaged together.
• Sun Microsystems, in contrast, considers its UltraSPARC IV to be a multi-threaded
rather than multi-processor chip.
• Intel considers their multi-core designs to be a single processor.
• This is not an idle debate, because software is often more expensive when
licensed for more processors.
4
Single-core computer

5
Multi-core architectures

• Replicate multiple processor cores on a single die.

Core 1 Core 2 Core 3 Core 4

Multi-core CPU chip 6

Multi-core CPU chip
• The cores fit on a single processor socket
• Also called CMP (Chip Multi-Processor)

c c c c
o o o o
r r r r
e e e e

1 2 3 4

7
The cores run in parallel
thread 1 thread 2 thread 3 thread 4

c c c c
o o o o
r r r r
e e e e

1 2 3 4

8
Within each core, threads are time-sliced (just
like on a uniprocessor)
several several several several
threads threads threads threads

c c c c
o o o o
r r r r
e e e e

1 2 3 4

9
Interaction with OS
• OS perceives each core as a separate processor

• OS scheduler maps threads/processes to different cores

• Most major OS support multi-core today

10
Why multi-core ?
• Difficult to make single-core clock frequencies even higher
• Many new applications are multithreaded
• General trend in computer architecture
(shift towards more parallelism)

11
Instruction-level parallelism
• Parallelism at the machine-instruction level
• The processor can re-order, pipeline instructions, split
them into microinstructions, do aggressive branch
prediction, etc.
• Instruction-level parallelism enabled rapid increases in
processor speeds over the last 15 years

12
Thread-level parallelism (TLP)
• This is parallelism on a more coarser scale
• Server can serve each client in a separate thread
(Web server, database server)
• A computer game can do AI, graphics, and physics in three
separate threads
• Single-core superscalar processors cannot fully exploit TLP
• Multi-core architectures are the next step in processor
evolution: explicitly exploiting TLP

13
General context: Multiprocessors

• Multiprocessor is any
computer with several
processors
• SIMD
• Single instruction, multiple data
• Modern graphics cards Lemieux cluster,
• MIMD Pittsburgh
supercomputing center
• Multiple instructions, multiple data
14
Multiprocessor memory types
• Shared memory:
In this model, there is one (large) common shared memory
for all processors

• Distributed memory:
In this model, each processor has its own (small) local
memory, and its content is not replicated anywhere else

15
Multi-core processor is a special kind of a multiprocessor:
All processors are on the same chip

• Multi-core processors are MIMD:

Different cores execute different threads (Multiple Instructions),
operating on different parts of memory (Multiple Data).

• Multi-core is a shared memory multiprocessor:

All cores share the same memory

16
What applications benefit from multi-core?
• Database servers Each can
run on its
• Web servers (Web commerce) own core
• Telecommuncation markets: 6WINDGate
(datapath and control plane)
• Multimedia applications
• Scientific applications, CAD/CAM
• In general, applications with
Thread-level parallelism
(as opposed to instruction-level parallelism)
17
More examples

• Editing a photo while recording a TV show through a digital

video recorder
• Downloading software while running an anti-virus program
• “Anything that can be threaded today will map efficiently to
multi-core”
• BUT: some applications difficult to parallelize

18
9.2 Graphics Processing Unit (GPU) design

 GPU is a graphical processing unit which enables

running high definitions graphics on Personal
Computers.
 Similar to Central Processing Unit (CPU), GPU is a single-
chip processor.
 The GPU has hundreds of cores as compared to the 4 or
18 in the latest CPUs.
 The primary job of the GPU is to compute 3D functions.
 GPU parallelism is similar to multicore parallelism

19
Parallelism

20
Heterogeneous Computing

Host: the CPU and its memory Device: the GPU and its memory

21
GPU vs CPU
• A GPU is tailored for highly parallel operation while a CPU executes programs
serially
• For this reason, GPUs have many parallel execution units and higher transistor
counts, while CPUs have few execution units and higher clockspeeds
• A GPU is for the most part deterministic in its operation (though this is quickly
changing)
• GPUs have much deeper pipelines (several thousand stages vs 10-20 for CPUs)
• GPUs have significantly faster and more advanced memory interfaces as they
need to shift around a lot more data than CPUs

22
GPU Architectures
• Processing is highly data-parallel
• GPUs are highly multithreaded
• Use thread switching to hide memory latency
• Less reliance on multi-level caches
• Graphics memory is wide and high-bandwidth
• Trend toward general purpose GPUs
• Heterogeneous CPU/GPU systems
• CPU for sequential code, GPU for parallel code
• Programming languages/APIs
• DirectX, OpenGL
• Compute Unified Device Architecture (CUDA)
23
The GPU pipeline
• The GPU receives geometry information from the CPU as an input and
provides a picture as an output
• Let’s see how that happens

host vertex triangle pixel memory

interface processing setup processing interface

24
Host Interface
• The host interface is the communication bridge between the CPU and
the GPU
• It receives commands from the CPU and also pulls geometry
information from system memory
• It outputs a stream of vertices in object space with all their associated
information (normals, texture coordinates, per vertex color etc)

host vertex triangle pixel memory

interface processing setup processing interface

25
Vertex Processing
• The vertex processing stage receives vertices from the host interface in object
space and outputs them in screen space
• This may be a simple linear transformation, or a complex operation involving
morphing effects
• Normals, texcoords etc are also transformed
• No new vertices are created in this stage, and no vertices are discarded
(input/output has 1:1 mapping)

host vertex triangle pixel memory

interface processing setup processing interface

26
Triangle setup
• In this stage geometry information becomes raster information
(screen space geometry is the input, pixels are the output)
• Prior to rasterization, triangles that are backfacing or are located
outside the viewing frustrum are rejected
• Some GPUs also do some hidden surface removal at this stage

host vertex triangle pixel memory

interface processing setup processing interface

27
Triangle Setup (cont)
• A fragment is generated if and only if its center is
inside the triangle

• Every fragment generated has its attributes computed

to be the perspective correct interpolation of the
three vertices that make up the triangle

host vertex triangle pixel memory

interface processing setup processing interface

28
Fragment Processing
• Each fragment provided by triangle setup is fed into fragment
processing as a set of attributes (position, normal, texcoord etc),
which are used to compute the final color for this pixel
• The computations taking place here include texture mapping and
math operations
• Typically the bottleneck in modern applications

host vertex triangle pixel memory

interface processing setup processing interface

29
Memory Interface
• Fragment colors provided by the previous stage are written to the framebuffer
• Used to be the biggest bottleneck before fragment processing took over
• Before the final write occurs, some fragments are rejected by the zbuffer, stencil
and alpha tests
• On modern GPUs, z and color are compressed to reduce framebuffer bandwidth
(but not size)

host vertex triangle pixel memory

interface processing setup processing interface

30
[Link]

[Link] 31
Programmability in the GPU
• Vertex and fragment processing, and now triangle set-up, are
programmable
• The programmer can write programs that are executed for every
vertex as well as for every fragment
• This allows fully customizable geometry and shading effects that go
well beyond the generic look and feel of older 3D applications

host vertex triangle pixel memory

interface processing setup processing interface

32
Diagram of a modern GPU
Input from CPU

Host interface

Vertex processing

Triangle setup

Pixel processing

Memory Interface

64bits to 64bits to 64bits to 64bits to

memory memory memory memory

33
CPU/GPU interaction
• The CPU and GPU inside the PC work in parallel with each other
• There are two “threads” going on, one for the CPU and one for the
GPU, which communicate through a command buffer:

GPU reads commands from here

Pending GPU commands

CPU writes commands here

34
CPU/GPU interaction (cont)
• If this command buffer is drained empty, we are CPU limited and the
GPU will spin around waiting for new input. All the GPU power in the
universe isn’t going to make your application faster!
• If the command buffer fills up, the CPU will spin around waiting for
the GPU to consume it, and we are effectively GPU limited

35
CPU/GPU interaction (cont)
• Another important point to consider is that programs that use the
GPU do not follow the traditional sequential execution model
• In the CPU program below, the object is not drawn after statement A
and before statement B:
•Statement A
•API call to draw object
•Statement B

• Instead, all the API call does, is to add the command to draw the
object to the GPU command buffer

36
Grids, Blocks, and Threads

37
CUDA Memory

Faster, per-block

Fastest, per-thread

Slower, global

Read-only, cached

38
Graphics Hardware Abstraction

• OpenGL and DirectX provide an abstraction of the hardware.

39
Programming using CUDA
• Compute Unified
Device Architecture

• do_something_on_host();
• kernel<<<nBlk, nTid>>>(args);
…
• cudaDeviceSynchronize();
• do_something_else_on_host();

Highly parallel

40
Example: NVIDIA Tesla Streaming
multiprocessor

8 × Streaming
processors

41
Example: NVIDIA Tesla
• Streaming Processors
• Single-precision FP and integer units
• Each SP is fine-grained multithreaded
• Warp: group of 32 threads
• Executed in parallel,
SIMD style
• 8 SPs
× 4 clock cycles
• Hardware contexts
for 24 warps
• Registers, PCs, …

42
Modern GPU has more ALU’s

43
9.3 Usage of GPU as a co-processor

The GPU architecture model. The GPU is a coprocessor to the CPU.

It consists of multiple SIMD multiprocessors, and has a large amount of device memory.
This model is applicable to both NVIDIA's CUDA and AMD's CTM .

44