NX Nastran

PARALLEL PROCESSING GUIDE

1
 Proprietary & Restricted Rights Notice

© 2007 UGS Corp. All Rights Reserved. This software and related documentation are
proprietary to UGS Corp.
NASTRAN is a registered trademark of the National Aeronautics and Space
Administration. NX Nastran is an enhanced proprietary version developed and
maintained by UGS Corp.
MSC is a registered trademark of [Link] Corporation. [Link] and
[Link] are trademarks of [Link] Corporation.
All other trademarks are the property of their respective owners.

2
CONTENTS
Page

Preface About this Book 4

Introduction
1 Fundamentals of parallel processing
Introduction of Parallel Processing 5
NX Nastran Parallel Solution Methods 10
2 Running parallel solutions
Parallel Processing Status in NX Nastran 12
DMP System Prerequisites in NX Nastran 13
Running Parallel Jobs 14
Running SMP Jobs 15
Running DMP Jobs 16
Parallel Solution Sequences 30
3 Computational methods
Foundation of Computational Methods 32
General Recommendations for the Selection of 41
the DMP Methods
4 Examples
Linear Static Analysis: SOL 101 43
Normal Mode Analysis: SOL 103 46
Direct Frequency Response Analysis: SOL 108 57
Modal Frequency Response Analysis: SOL 111 58
Modal Transient Response Analysis: SOL 112 63
Optimization: SOL 200 64
Industrial Case Study 66
Reference 70

3
PREFACE

About this Book

NX NASTRAN is a general-purpose finite element program which solves a wide variety

of engineering problems. The NX NASTRAN Parallel Processing Guide is intended to
help you choose among the different parallel processing and computational methods, and
ultimately increase the performance of analysis by reducing CPU time, memory and disk
space requirements.

This main material in this book covers the parallel processing methods for the linear
static, normal modes, direct frequency response, modal frequency response, modal
transient response, and design optimization.

This book will continue to be revised to reflect future enhancements to NX Nastran

parallel processing methods. Changes and additions are encouraged, and can be
communicate through UGS technical support [Link]

Introduction

The NX Nastran Parallel Processing Guide provides information on NX Nastran parallel

methods on Linux and UNIX systems. This book is composed of 4 chapters:

1. Fundamentals of parallel processing

2. Running parallel NX Nastran solutions
3. Distributed memory parallel processing computational methods
4. Examples for distributed memory parallel processing solution

To effectively use this book, it is important for you to be familiar with the basic structure
of NX Nastran. For more information about the mathematical foundation, refer to NX
Nastran Numerical Methods User’s Guide.

4
Chapter 1
FUNDAMENTALS OF PARALLEL PROCESSING

1.1 Introduction of Parallel Processing

Why Use Parallel Computing?

In many applications today, the volume of analysis is going up, the physics being
modeled is increasingly complex, and the time allocated to development is under
increasing pressure. Therefore, researchers always need higher processing speed and
more memory in order to investigate increasingly complex problems. Optimization
of processes and data flows helps, but there are still limitations on computational
performance in serial computation.

Parallel processing is a method that allows simultaneous use of multiple processors

on one or more machines to decrease solution times. The benefits of parallel
processing include:

• Reduced solution time on large problems

• Reduced hardware requirements by utilizing smaller and less expensive machines
• Decreased turnaround time for each solution; allowing you to analyze more
designs under more conditions in a shorter amount of time

Types of Parallel Computing

Generally, parallel programming normally proceeds according to one of the following

models in Figure 1. Single Instruction, Single Data (SISD) is conventional serial
execution that processes only one stream of instructions and one stream of data.
Single Instruction, Multiple Data (SIMD) refers to a parallel execution model in
which all processors execute the same operation at the same time, but each processor
is allowed to operate upon its own data. In Multiple Instruction, Single Data (MISD),
each processor executes independent streams of instructions. However, all the
processing elements work on the same data stream. Multiple Instruction, Multiple
Data (MIMD) refers to a parallel execution model in which each processor is
essentially acting independently. The processors work on their own data with their
own instructions. Tasks executed by different processors can start or finish at
different times.

5
 SISD SIMD

Single Instruction, Single Data Single Instruction, Multiple Data

MISD MIMD

Multiple Instruction, Single Data Multiple Instruction, Multiple Data

Figure 1.1 Types of parallel computing

Machine Architectures for Parallel Computing

• Shared Memory Parallel (SMP) Machines

A Shared Memory Parallel (SMP) machine is defined as a single machine with

multiple processors that share a common memory and I/O as illustrated in Figure 1.2.
Multiple processors can operate independently, but changes in a memory location
effected by one processor are visible to all other processors. The primary
disadvantage is the lack of scalability between memory and processors. It becomes
increasingly difficult and expensive to design and produce shared memory machines
with ever increasing numbers of processors.

The OpenMP Application Program Interface (API) supports multi-platform SMP

programming in C/C++ and Fortran on all architectures, including Unix platforms
and Windows NT platforms. Jointly defined by a group of major computer hardware
and software vendors, OpenMP is a portable, scalable model that gives shared-
memory parallel programmers a simple and flexible interface for developing parallel
applications for platforms ranging from the desktop to the supercomputer.

Figure 1.2 SMP Machine architecture

6
• Distributed Memory Parallel (DMP) Machines

A Distributed Memory Parallel (DMP) machine uses multiple machines or clusters

with one or more processors communicating over a network, or multiprocessors with
multiple I/O channels. Figure 1.3 shows a typical architecture of a DMP machine.
Each machine has its own memory and one or more disks. In DMP, data is private to
each node and it is necessary to exchange data across different nodes. Therefore, you
need to decide which data is to be sent/received to/from which node. The main
advantage is that memory is scalable with multiple processors.

Message passing is a widely used communication model in programming parallel

processor machines for DMP processing. You can access the communication
hardware only through the message passing library. Message passing libraries like
Message Passing Interface (MPI) or Parallel Virtual Machine (PVM) send and
receive data from a program running on one node to a program running on another
node. In particular, MPI is a standard message passing library that has been
efficiently implemented on a variety of platforms.

MPI is portable, efficient, expressive, and provides thread safety. In addition,

• Point-to-point communications handle data transmission between any two
processors in a communicator.
• Collective communications handle simultaneous communication between all
processors in a communicator.

Figure 1.3 DMP Machine architecture

7
Differences between DMP and SMP

The differences between DMP and SMP are listed in the table below.

Feature DMP SMP

Hardware environment High performance Shared memory multi-

workstation clusters processor workstations

Parallelism level Partition finite element Subdivided matrix and

model vector operations

Software mechanism Message Passing Interface OpenMP API

(MPI)

Performance of Parallel Processing

The performance of a parallel processing is rated by its speedup or efficiency. The

speedup Sp is defined as:

Ts
Sp = = E* p
Tp

where Ts is the time to run a serial job, and Tp is the time it takes to run the same job
with p processors.

The efficiency E is defined as:

Ts
E=
( p * Tp )

For better performance, it is helpful to minimize the communication overhead by using

high speed network switches. The ideal is to have a high bandwidth and a low latency.

8
Domain Decomposition for Parallel Processing

In the parallel processing of finite element applications, domain decomposition has

widely been used. Domain decomposition divides the data according to the number of
processors. For the boundary of each subdivided data, it is necessary to communicate
with other processors. It is important to maintain load balancing and minimize
communication in domain decomposition.

As shown in the table below, by subdividing a large problem using domain

decomposition and running the job in parallel, we can greatly reduce elapsed execution
time since p3 is much greater than p. We also have fewer resource requirements to solve
the subdivided case since the problem size n is much less than N.

Undivided Subdivided

Problem size N n = N/p

Operation cost N3 = p3 n3 pn3
Remarks p3 >> p

9
1.2 NX Nastran Parallel Solution Methods
NX Nastran offers the ability to run certain solution sequences in parallel. The following
solution methods are available for running NX Nastran in parallel.

• SOL 101 Linear static

• SOL 103 Normal modes
• SOL 108 Direct frequency response
• SOL 111 Modal frequency response
• SOL 112 Modal transient response
• SOL 200 Design optimization

For various NX Nastran parallel solutions, the following computational methods are
available.

• Geometric Domain Static Analysis (GDSTAT) method

• Load Domain Static Analysis (LDSTAT) method
• Frequency Domain Normal Modes Analysis (FDMODES) method
• Geometric Domain Normal Modes Analysis (GDMODES) method
• Hierarchic Domain Normal Modes Analysis (HDMODES) method
• Frequency Domain Frequency Response analysis (FDFREQR) method

10
Chapter 2
RUNNING PARALLEL NX Nastran SOLUTIONS

NX Nastran parallel methods achieve their solution speed by dividing the FE model into
smaller pieces to be solved simultaneously. This division, called “hierarchic”, is
performed with respect to geometry or frequency range individually or both at the same
time. Although each processor is working on its own partition of the geometry or
frequency range, it communicates with the others to share information. Once the solution
is complete, the results are merged, creating a single result file.

NX Nastran DMP methods are available with solution sequences 101, 103, 108, 111, 112,
and 200, in which DMP methods for SOL 200 are available from version 5. The SMP is
used only for lower level operations such as matrix decompositions and matrix
multiplications for all solution sequences. Most of the discussions and examples in this
book focus on the DMP computational methods and solution methods.

11
2.1 Parallel Processing Status in NX Nastran

Parallelization in NX Nastran
The parallelization process in NX Nastran consists of three different categories: high
level with DMP, medium level with DMP, and low level with SMP.

Level Application
DMP High level Hierarchic domain

Medium level Geometry domain

Frequency domain

SMP Low level Kernel (DCMP, MPYAD module)

For the mathematical details of various parallel methods, refer to the NX Nastran
Numerical Methods User’s Guide.

Message Passing Interface (MPI)

NX Nastran offers the ability to run certain solution sequences in parallel using a
Message Passing Interface (MPI), an industry-wide standard library for C and Fortran
message-passing programs. MPI programs can be run on SMP computers, DMP
computers, and a cluster of computers supported by the MPI package. In most cases, NX
Nastran uses the hardware vendor’s MPI implementation. While this usually results in the
highest performance levels, it also limits a DMP job to computers supported by the
vendor’s MPI package. For example, you cannot use a mixture of IBM and Sun machines
to run a single NX Nastran DMP job.

Further information on the MPI standard is available online at the MPI forum website:
[Link]

12
2.2 DMP System Prerequisites in NX Nastran

DMP System Prerequisites

NX Nastran supports the following system:

• HP UX Itanium
• AIX Power
• Linux x86
• Linux Itanium
• Linux Opteron/EM64T
• SGI Altix Itanium

Note that NX Nastran DOES NOT support DMP for Windows system.

The detailed requirements of hardware and software are described in the NX Nastran
Installation and Operation Guide.

13
2.3 Running Parallel Jobs
An NX Nastran parallel job can be selected by the nastran command with either
keywords or a system cell. The nastran command permits the keywords, dmp and
parallel, to request DMP and SMP runs, respectively. Alternatively, system cells 231 and
107 can also be used for DMP and SMP runs.

It is strongly recommended to use keywords to request a parallel job. The following table
provides keywords and their descriptions as well as the system cells that can be used for
DMP and SMP runs.

Keywords Description System Cell

dmp Default = 0 (deselect DMP processing)
(dmparallel) Specifies the number of tasks for a DMP
DMP 231
analysis
1-256 processors are available in a DMP job
parallel Default =0 (deselect SMP processing)
(smparallel) Specifies the maximum number of CPUs
SMP 107
selected for SMP processing
1-1023 processors are available in a SMP job.

14
2.4 Running SMP Jobs
The following example illustrates how to run an NX Nastran job named ‘[Link]’ in
an SMP environment with p processors.

nastran example parallel=p

or

nastran example system107=p

15
2.5 Running DMP Jobs
When running an NX Nastran DMP job, keywords are used to select (1) the
computational method, and (2) the finite element model partitioning method for each
parallel solution sequence.

nastran example [computational method] [ finite element model partitioning

method]

• The available keywords for computational method are “dmp”, “numseg”, and
“nclust.”
• The keyword for selecting the partitioning method for finite element model is
“gpart”. The default is “gpart=0.”

¾ General requirements for a DMP run

The following are some general requirements for running NX Nastran DMP jobs.

• NX Nastran must be properly installed on all the hosts listed by the “hosts”
keyword or in the ‘[Link]’ file.
• The “master” node is the first computer named by the “hosts” keywords or listed
in the “[Link]” file, and “slave” nodes are the remaining systems.
• The MPI program start command must be available in the path of the local host.
(“dmpruns” on Compaq, “poe”on AIX, “mprun” on Sun, and “mpirun” on others)
• The input data file must be accessible on the local host.
• The scratch directory can be on a global or local file system. NX Nastran
recommends the scratch directory be local to each node.
• The nastran keywords are processed in both the local and master/slave system.

The detailed requirements of hardware and software are described in the “NX Nastran
Installation and Operations Guide”.

¾ Managing Host-database Assignments

The performance of an NX Nastran parallel job is very much dependent on the CPU,
memory system, and I/O system performance. A DMP job is extremely sensitive to I/O
system performance since each task independently accesses the I/O system. Especially,
the performance of the disk subsystem that contains the permanent and SCRATCH
DBSets can have a significant impact on NX Nastran performance. The impact is even
greater if multiple tasks are using the same file system. It is recommended that the “dbs”,
“hosts”, and “sdirectory” be local to each node.

16
The following example illustrates how to run ‘[Link]’ in parallel with four
processors with “dbs”, “hosts”, and “sdirectory” local to each node.

nastran example dmp=4 gpart=1 \

hosts=node1:node2:node3:node4 \
sdirectory=/scratch1:/scratch2:/scratch3:/scratch4 \
dbs=/scratch1:/scratch2:/scratch3:/scratch4

• It is strongly recommended to use the nastran keyword “slaveout=yes” to print

out analysis procedures from all processors. With “slaveout=yes”, the f04, .f06,
and .log files contain the outputs of master and slave processors in the following
format.

• The nastran keyword “mergeresults=yes” specifies the results from each DMP
task are to be merged into the standard files from the master node.

¾ Managing files in DMP jobs

When the NX Nastran job is running, the input file is directly read by each MPI task.
Each node that cannot read the input file reads a local copy of the file that is copied to the
job’s scratch directory before the job begins. In order to access another node, “r-”
commands such as rsh, rcp, and rlogin must be available and properly configured on each
node. Similarly, any host that can write to the output directory directly writes
its .f04, .f06, .log and other output files to that directory. Any host that cannot see the
output directory writes its default output files to the job’s scratch directory. These files
are then copied back to the output directory at the end of job.

17
2.5.1 Finite element model partitioning method

In the parallel processing of a finite element application, finite element model data should
be distributed to each processor. There are two different methods for model partitioning
in an NX Nastran parallel processing task: grid-based partitioning and degree of freedom-
based partitioning. The module SEQP performs grid-based partitioning, in which p
domains are created by an automatic partitioner from the grid-based graph. The module
GPARTN performs graph partitioning for the graph based on degrees of freedom.

It should be noted that the module SEQP cannot handle a finite element model that
includes acoustic fluid or virtual mass. In addition, the module SEQP provides poor
performance if the finite element model includes many Multi-Point Constraints (MPCs).
Generally, the module GPARTN gives better performance than the module SEQP. It is
recommended to use the module GPARTN if the finite element model includes many
MPCs, acoustic fluids, and/or virtual masses.

The partitioning method can be selected by the nastran command keyword ‘gpart’ in the
following way.

Module Keyword
SEQP gpart = 0 default

GPARTN gpart= 1

Each module provides two different algorithms for graph partitioning. The algorithm can
be selected by the DMAP parameter oldseq.

Module Method DMAP parameter

EXTREME param, oldseq = 9

SEQP
MLV param, oldseq = 11 default

METIS param, oldseq = 9

GPARTN
MLV param, oldseq = 11 default

18
2.5.2 Parallel Computational Methods

Based on the partitioned finite element model, there are five different types of DMP
computational methods in NX Nastran:

• Geometric Domain Static Analysis (GDSTAT)

• Load Domain Static Analysis (LDSTAT)
• Frequency Domain Normal Modes Analysis (FDMODES)
• Geometric Domain Normal Modes Analysis (GDMODES)
• Hierarchic Domain Normal Modes Analysis (HDMODES)
• Frequency Domain Frequency Response Analysis (FDFREQR)

GDSTAT and LDSTAT are for static analysis. The other four DMP methods are for
dynamic analyses.

The overall flow of the finite element model partitioning and parallel computational
methods is illustrated in Fig. 2.1, in which the approaches for SOL 101, 103, 108, 111,
and 112 are described. Fig. 2.2 through Fig. 2.5 describes the modal detailed flow for
each solution sequence. Fig. 2.6 represents the structural optimization DMP task flow.
The shaded boxes at each analysis flow correspond to DMP task.

The details of various computational methods are described in Chapter 3.

19
 Finite Element Model

SOL 101 SOL 108

SOL 103
GPART=0 GPART=1 SOL 111
SOL 112
SOL 103
SOL 111 SOL 103
SOL 101 SOL 112 SOL 111
SOL 112

LDSTAT GDSTAT GDMODES HDMODES FDMODES

FDFREQR

Fig. 2.1 Overview of NX Nastran DMP solution task except SOL 200

20
ƒ Linear static analysis DMP task

SOL 101

Finite Element Model

GPART=1 GPART=0

SEQP

LDSTAT GDSTAT

Fig. 2.2 Linear static analysis (SOL 101) DMP task

21
ƒ Normal mode analysis DMP task

SOL 103

Finite Element Model

GPART=0 GPART=1
GPART

SEQP GPARTN

GDMODES HDMODES FDMODES

Fig. 2.3 Normal mode analysis (SOL 103) DMP task

22
ƒ Direct frequency response analysis DMP task

SOL 108

Finite Element Model

FDFREQR

Fig. 2.4 Direct frequency response analysis (SOL 108) DMP task

23
 ƒ Modal frequency response analysis DMP task

SOL 111

Finite Element Model

GPART=0 GPART=1
GPART

SEQP GPARTN

GDMODES HDMODES FDMODES

FDFREQR

Fig. 2.5 Modal frequency response analysis (SOL 111) DMP task

Note that you do not need to explicitly request FDFREQR in the FDMODES,
GDMODES, and HDMODES runs for SOL 111. For GDMODES or HDMODES, the
FDFREQR will be always executed automatically when “gpart=1” is specified. The
FDMODES does not need to specify the keyword “gpart” to run the FDFREQR
automatically.

24
 ƒ Modal transient response analysis DMP task

SOL 112

Finite Element Model

GPART=0 GPART=1
GPART

SEQP GPARTN

GDMODES HDMODES FDMODES

Transient Response Analysis

In SERIAL

Fig. 2.6 Modal transient response analysis (SOL 112) DMP task

Once the normal mode calculation portion is finished in parallel, then the modal transient
response analysis is performed in serial.

25
 ƒ Design optimization DMP task

SOL 200

INITIAL
DESIGN

ANALYSIS

DMP normal mode anaysis

gpart=1 gpart=0

GDMODES HDMODES FDMODES

OPTIMIZATION

IMPROVED
DESIGN

Fig. 2.7 Design optimization (SOL 200) DMP task

Note that only the normal mode calculation portion of an optimization can now use DMP.

26
• Geometric Domain Static Analysis (GDSTAT)

GDSTAT is available only with the SEQP module, which is the default. The
model in the GDSTAT method is automatically partitioned into p domains based
on geometry, where p is the number of processors. Each domain contains that
portion of the geometry plus the boundary. You can think of geometric
decomposition as an automated super-element approach. The keyword “dmp”
defines the number of processors.

Partitioning module
SEQP nastran example dmp=p

• Load Domain Static Analysis (LDSTAT)

LDSTAT is useful when large numbers of load cases exist. In the LDSTAT, the
large number of load cases are automatically partitioned into p sets of cases,
where p is the number of processors. Each processor runs the static analysis with
its own load cases. Note that each processor contains the full model, so that finite
element model partitioning is not required. The keyword “dmp” defines the
number of processors. The value of the keyword “gpart” shoule be 1.

Partitioning module
nastran example dmp=p gpart=1

• Geometric Domain Normal Modes Analysis (GDMODES)

A finite element model is automatically partitioned into p domains, where p is the

number of processors. Each domain contains that portion of the geometry plus the
boundary. Each processor performs the same normal mode analysis using the
local matrices corresponding to the partitioned domains. Both the SEQP and
GPARTN modules are available for the GDMODES.

Partitioning module
SEQP nastran example dmp=p
GPARTN nastran example dmp=p gpart=1

Note that it is important to use the EIGRL card, not EIGR. In addition, the
number of processors p should be greater than 1. The value of the keyword “dmp”

27
 must be an integer greater than or equal to 2, and a power of 2. For example, 2, 4,
8, 16, etc. are valid.

• Frequency Domain Normal Modes Analysis (FDMODES)

The frequency range of interest specified on the EIGRL entry is automatically

decomposed into multiple frequency segments—one for each processor. The
modes for each of these segments are then solved separately within each
processor. Note that each processor contains the full model, so that finite element
model partitioning is not required. The value of the keyword “numseg” for the
number of frequency segment should be the same as the value of the keyword
“dmp”.

Partitioning module
- nastran example dmp=p numseg=p
- or specify NUMS entry in EIGRL card

For DMP processing, the frequency range between V1 and V2 of EIGRL must be
specified. If both NUMSEG and NUMS are specified, NUMS takes precedence.

• Hierarchic Domain Normal Modes Analysis (HDMODES)

The HDMODES solution sequence combines two existing techniques of DMP

processing: GDMODES and FDMODES. While the geometry partitions are
solved within a set of processors called a cluster, frequency segments are also
solved in parallel across multiple clusters. The advantage is a faster eigenvalue
problem solution time for very large models.
The keyword “dmp” defines the number of processors p, and the keyword
“nclust” defines the number of clusters c. The number of geometry partitioning g
in a cluster does not have to be defined explicitly. Note that 1<c<p and
p=c*g. The number of clusters c should be properly selected so that g is an
integer greater than or equal to 2, and a power of 2. For example, 2, 4, 8, 16, etc.
are valid

Partitioning module
SEQP nastran example dmp=p nclust=c
GPARTN nastran example dmp=p nclust=c gpart=1

Alternatively, there is a way that uses keywords “gdoms” and “fsegs”. The
keyword “gdoms” defines the number of geometric domains used in HDMODES.
The value “gdoms” must be an integer greater than or equal to 2, and a power of 2.
The "fsegs" keyword must also be defined when "gdoms" is defined. The product

28
 of "fsegs" and "gdoms" should be the value of the "dmp" keyword. There is no
default value.

Partitioning module
SEQP nastran example gdoms=g fsegs=f
GPARTN nastran example gdoms=g fsegs=f gpart=1

• Frequency Domain Frequency Response analysis (FDFREQR)

The excitation frequencies, as specified on the FREQi entries, are split among the
processors as evenly as possible, and the responses due to these excitation
frequencies are then calculated within each of the respective processors. Each
processor contains the full model.

Partitioning module
- nastran example dmp=p

Note that, for SOL 111, running a job with GDMODES, FDMODES, or
HDMODES automatically performs FDFREQR without requesting it explicitly.
Performing FDFREQR partitions the excitation frequencies among the number of
processors defined with the “dmp” keyword.

Note that FDMODES, GDMODES, and HDMODES perform the parallel processing task
for the normal mode analysis as well as the modal frequency response analysis only when
the module GPART is used. Without the module GPART, gpart=0, the frequency
analysis part runs in serial.

For more details of the various computational methods, refer to Chapter 3.

29
2.6 Parallel Solution Sequences

Several DMP computational methods are available with solution sequences 101, 103, 108,
111, 112, and 200 (ver.5). Table 2-1 shows the available parallel computational methods
for various solution sequences. For details of various solution sequences for DMP, refer
to Chapter 4.

Table 2.1 Available parallel computational methods for various solution

sequences

Parallel computational methods

Solution GDSTAT LDSTAT GDMODES FDMODES HDMODES FDFREQR
+
101 O O

103 O O O
108 O
111 Ox O Ox O*
112 O O O
200 O+ O O+

O+ : You must specify the “gpart=1” keyword in the NX Nastran run.

Ox : You should specify the “gpart=1” keyword to run FDFREQR.

With “gpart=0”, except FDMODES, frequency response analysis part
runs in serial instead of using FDFREQR.

O* : You do not need to explicitly request FDFREQR.

- GDMODES or HDMODES automatically performs FDFREQR
with the “gpart=1” keyword.
- FDMODES also runs FDFREQR automatically without “gpart” keyword.

30
Chapter 3
DMP COMPUTATIONAL METHODS

Chapter 3 provides details on the computational methods used for running an NX

Nastran job in parallel.

31
3.1 Foundation of Computational Methods

3.1.1 Geometric Domain Static Analysis (GDSTAT)

The GDSTAT provides an efficient parallel solution for the linear static analysis of
large models. The static solution is performed on the u l set. The l-set is identical to the
a-set if there is no rigid body support (r-set). The mathematical expression for the static
equilibrium of the finite element model can be expressed as:

K ll u l = Pl

where K ll is the global stiffness matrix, Pl is the loads, and u l is the displacements.

The finite element model in the GDSTAT method is automatically partitioned into p
domains, where p is the number of processors. Figure 3.1 shows each domain that
contains the portion of the geometry (O1, O2, O3, or O4) plus the boundary t. Think of
geometric decomposition as an automated super-element approach. After the domain is
partitioned, each processor performs the linear static analysis only on its local domain
with boundary. GDSTAT allows a problem that is not possible to solve on one
processor to be solved in the DMP environment by reducing a significant amount of
disk space and memory.

O1 O2

O3 O4

t
Figure 3.1 Partitioning a finite element model for geometry domain static analysis
(GDSTAT)

32
3.1.2 Load Domain Static Analysis (LDSTAT)
The LDSTAT is useful when there are large number of load cases in the linear static
analysis problem K ll u l = Pl . Instead of partitioning the finite element model, the load
matrix Pl is partitioned among the processors as evenly as possible, and the linear
solution is calculated within each of the respective processors for its own load cases.
The mathematical expression for the linear static analysis can be expressed as:

K ll u k = Pk , k = 1,", p

where p is the number of processors. Once all processors finish its own linear solution,
the master processor collects and forms the overall solution as

u l = [u1 , u 2 , " , u p ]

33
3.1.3 Frequency Domain Normal Mode Analysis (FDMODES)

To analyze the dynamic behavior of the structures, modal solution techniques are
commonly used. The essence of the modal solution is efficient calculation of the
mechanical system's free, undamped vibration. That is the eigenvalue analysis problem
of:

( K aa − λM aa )φ a = 0

Here the eigenvalue λ represents a natural frequency, and the eigenvector φ is a free
vibration shape of the finite element model. The a subscript refers to the a-set, which
is the analysis partition of the finite element model. This is one of the most time
consuming computations of large scale global analyses in the automobile and the
aerospace industries.

The frequency range of interest specified on the EIGRL entry is automatically

decomposed into multiple frequency segments—one for each processor. NX Nastran
also allows you to directly input the number of frequency segments with the NUMS
entry on the EIGRL entries. Table 3.1 shows the frequency segment of each processor
when s processors are available.

Table 3.1 Partition of frequency for FDMODES parallel run

Frequency Lower Upper

Processor
segment frequency frequency
1 1 f0 f1
2 2 f1 f2
# #
j j fj-1 fj
# #
s s fs-1 fs

Note that each processor contains the full model in the FDMODES computation, so that
the mode shapes in the individual frequency segments are independent of each other.
Figure 3.2 represents the schematic diagram: each processor solves the full model
within its frequency segment. The only communication needed is when gathering the
results for the master processor. Table 3.2 shows the eigensolutions obtained from each
processor.

Although FDMODES reduces the frequency range for a process by decomposing the
frequency domain, it is still ineffective with respect to a large problem size.

34
 ( K aa − λM aa )φa = 0

Figure 3.2 Frequency domain partitioning for the normal mode analysis of a
whole finite element model

Table 3.2. The distribution of eigensolution ( Φ, Λ ), Φ = {Φ 1 , Φ 2 ," , Φ s }

and Λ = diag{Λ 1 , Λ 2 ,", Λ s } , with FDMODES parallel run

Frequency domain

Segment 1 … Segment j … Segment s

(Φ 1 , Λ 1 ) … (Φ j , Λ j ) … (Φ s , Λ s )

35
3.1.4 Geometry Domain Normal Mode Analysis (GDMODES)

GDMODES is executed by automatically subdividing the geometry obtained from the

finite element model. Such a subdivision of a finite element model is shown in Figure
3.3. Here the o partition refers to the interior of the domains and the t partition is the
common boundary shared by the domains. NX Nastran provides two modules, SEQP
and GPARTN, to execute this partitioning, as described in Chapter 2.

The geometric domain decomposition is mathematically represented with the following

reordering for the p partitioned domains.

⎡ K oo
1
− λM oo
1
K ot1 − λM ot1 ⎤ ⎡ φ o1 ⎤
⎢ ⎥⎢ ⎥
⎢ K oo2 − λM oo2 K ot2 − λM ot2 ⎥ ⎢φ o2 ⎥
⎢ % # ⎥⎢ # ⎥ = 0
⎢ p ⎥⎢ p ⎥
⎢ K oop − λM oop K ot − λM ot ⎥ ⎢φ o ⎥
p

⎢ K T ,1 − λM T ,1 T ,2
− λM oo
T ,2
" K oo
T,p
− λM oo
T,p
K tt − λM tt ⎥⎦ ⎢⎣φt ⎥⎦
⎣ oo oo K oo

This partitioned eigenvalue problem may be solved by a special formulation of the

Lanczos method. Once each processor contains the portion of the local domain with
boundary, each processor computes its part of the global eigensolution and exchanges
boundary components with other processors.

Table 3.3 shows the eigensolution obtained from each processor, in which the global
eigensolutions are obtained by merging all distributed eigensolutions.

Note that the formulation yields a computationally exact solution. The efficiency
depends on the relative size of the boundaries with respect to the interiors.

36
 frequency range

[ fmin fmax ]

O1 O2

O3 O4

Figure 3-3. Geometry domain partitioning for the normal mode analysis of a finite
element model

Table 3.3 The distribution of eigensolution ( Φ, Λ ) with GDMODES parallel

run, where Φ = {Φ 1 , Φ 2 ,", Φ p }T .

partition 1 (Φ 1 , Λ )

# #
Geometry
partition i (Φ i , Λ )
domain

# #

partition p (Φ p , Λ )

Note that the geometric domain decomposition is important to reduce the very large
problem sizes, but it does not affect the frequency spectrum.

37
3.1.5 Hierarchic Domain Normal Mode Analysis (HDMODES)

The HDMODES scheme simultaneously executes the two previously presented

methods, GDMODES and FDMODES. With this approach, a subset of processors or a
cluster solves the eigenvalue problem for the local geometry while communicating with
other subsets of processors or other clusters in order to consider the other frequency
ranges.

The HDMODES computation is based on the processor assignment shown in Table 3.4,
where we have p * s processors available. For each frequency segment, p processors are
used for GDMODES. In other words, s processors are used for FDMODES for each
geometric portion. For example, the ((j - 1) p+i)-th processor computes the eigenvalues
of the j-th frequency segment Λ j and the i-th geometric partition of the corresponding
eigenvectors Φ ij . The selection of the s and p value is problem dependent.

Note that HDMODES solves the eigenvalue problem computationally exactly as the
GDMODES and FDMODES methods do.

Table 3.4 Hierarchic domain decomposition concept

Frequency domain
segment segment Segment
… …
1 j s
partition 1 (j-1)*p+1 (s-1)*p+1
1 (Φ 1 , Λ 1 )
1
(Φ 1j , Λ j ) (Φ 1s , Λ s )

# # # #
Geometry partition i (j-1)*p+i (s-1)*p+i
domain i (Φ , Λ 1 )
i
1
(Φ ij , Λ j ) (Φ sj , Λ s )

# # # #
partition p j*p s*p
p (Φ , Λ 1 )
p
1
(Φ pj , Λ j ) (Φ sp , Λ s )

The preferred hardware environment for HDMODES is a cluster of multiprocessor

workstations that is usually tied together by either a hardware switch or a network, as
illustrated in Fig. 3.4. The preferred arrangement is to retain the geometric domain
component of the hierarchic approach on the workstations and spread the frequency
domain segments across the network.

38
Assuming m workstations with n processors each, based on the scheme of Table 3.4,
the tasks of each column (the geometric partitions of a particular frequency segment)
reside on one workstation. The tasks of each row (the various frequency segments of a
particular geometry partition) are spread across the cluster, as shown in Fig. 3.5.

Figure 3-4. A scheme of cluster of multi-processor workstations

Figure 3-5. Mapping the hierarchic domain decomposition with n geometric

domain partition and m frequency partition (Pi: geometric domain partition and
Sj: frequency segment) to a cluster of multi-processor workstations.

39
3.1.5 Frequency Domain Frequency Response Analysis
(FDFREQR)

The excitation frequencies, as specified on the FREQi entries, are split among the
processors as evenly as possible, and the responses for the partitioned excitation
frequencies are then calculated within each of the respective processors as shown in.
Fig.3-6. The whole range of excitation frequencies is partitioned into many sub-
intervals such as [Fi Fi+1]. Note that each processor contains the full model.

Figure 3.6 Excitation frequency domain partitioning for (direct or modal)

frequency response analysis of a whole finite element model

For direct frequency response analysis (SOL 108), you need to explicitly request
FDFREQR in the nastran command, as explained in Chapter 2. However, in the modal
frequency response analysis (SOL 111), any parallel computational method
automatically runs FDFREQR without requesting it. In other words, GDMODES,
FDMODES, or HDMODES implicitly invokes FDFREQR.

40
3.2 General Recommendations for the Selection of the
DMP Methods

SOL 101 has two DMP methods for static analysis. It is recommended to use
GDSTAT if you have a very large model. LDSTAT is useful when there are large
numbers of load cases.

For SOL 108, if there are many forcing frequencies, FDFREQR should be selected. In
FDFREQR, the source requirements such as disk space and memory are as high as the
source requirements of the serial run.

In SOL 111, it is recommended that you use FDMODES if you have a small model and
a large frequency range of interest. If you have a very large model and insufficient disk
space, use either GDMODES or HDMODES. Especially, HDMODES is recommended
for a cluster of multiprocessor workstations.

SOL 200 follows the same recommendation as for SOL 111.

Fig. 3-7 illustrates the general guide for selecting a suitable DMP computational
method.

FDMODES
Frequency HDMODES
range

GDMODES

FE model
size

Figure 3.7 General guideline for the selection of DMP computational methods

41
Chapter 4
Examples for
Distributed Memory Parallel Processing Solution
The NX Nastran DMP computational method described in Chapter 3 is available for the
following solution sequence.

• SOL 101 Linear static

• SOL 103 Normal modes
• SOL 108 Direct frequency response
• SOL 111 Modal frequency response
• SOL 112 Modal transient response
• SOL 200 Design optimization

This chapter presents several examples for each solution sequence in DMP processing.

42
4.1 Linear Static Analysis: SOL 101

4.1.1 GDSTAT

The static analysis is performed for a finite element cube model that has 2197 grid points
for 1728 CHEXA elements and 6048 CQUAD4 elements. GDSTAT is implemented in a
DMP run. The module SEQP partitioned the global finite element model into sub-
domains and a boundary. The following information in the .f06 describes the detailed
statistics of partitioning with two and four processors, respectively.

STATISTICS FROM AUTOMATIC MODEL PARTITIONER

THE NUMBER OF DOMAINS CREATED USING EXTREME IS 2

THE NUMBER OF GRID POINTS IN THE GLOBAL BOUNDARY IS 283
THE NUMBER OF ELASTIC ELEMENTS IN THE RESIDUAL IS 0
THE NUMBER OF RIGID ELEMENTS IN THE RESIDUAL IS 0
DOMAIN ID # INTERNAL GRID POINTS # EXTERNAL GRID POINTS # OF ELEMENTS
--------- ---------------------- ---------------------- -------------
1 1014 169 4032
2 900 283 3744

STATISTICS FROM AUTOMATIC MODEL PARTITIONER

THE NUMBER OF DOMAINS CREATED USING EXTREME IS 4

THE NUMBER OF GRID POINTS IN THE GLOBAL BOUNDARY IS 361
THE NUMBER OF ELASTIC ELEMENTS IN THE RESIDUAL IS 0
THE NUMBER OF RIGID ELEMENTS IN THE RESIDUAL IS 0
DOMAIN ID # INTERNAL GRID POINTS # EXTERNAL GRID POINTS # OF ELEMENTS
--------- ---------------------- ---------------------- -------------
1 468 174 2060
2 468 169 2016
3 468 169 1880
4 432 205 1820

Each processor performs the static analysis with the corresponding local domain and the
corresponding boundary, and the master processor collects the results that you requested
through communications. These outputs are printed only on the master processor.

D I S P L A C E M E N T V E C T O R

POINT ID. TYPE T1 T2 T3 R1 R2 R3

30060 G 1.578281E-05 -9.678060E-05 -2.301274E-04 -4.363790E-06 -3.554464E-06 -4.259573E-09
40066 G -1.578241E-05 -9.678057E-05 -2.301267E-04 -4.363756E-06 3.554413E-06 4.257577E-09
70660 G -1.578268E-05 9.677987E-05 -2.301273E-04 -4.363777E-06 -3.554452E-06 4.259614E-09
80666 G 1.578229E-05 9.677984E-05 -2.301266E-04 -4.363744E-06 3.554401E-06 -4.257619E-09

43
It is strongly recommended to use the nastran keyword “slaveout=yes” to print out
analysis procedures from all processors. With “slaveout=yes”, the f04, .f06, and .log files
contain the outputs of master and slave processors in the following format.

#
⎫
⎬ master processor
⎭
* * * END OF JOB * * *

****************
S L A V E 1
****************
#
⎫
⎬ slave 1 processor
⎭
* * * END OF JOB * * *

****************
S L A V E 2
****************
#
⎫
⎬ slave 2 processor
⎭
* * * END OF JOB * * *

****************
S L A V E 3
****************
#
⎫
⎬ slave 3 processor
⎭
* * * END OF JOB * * *

44
4.1.2 LDSTAT

For a model with 500 subcases, 125 subcases are assigned to each processor when four
processors are available. Without partitioning a domain, each processor performs the
static analysis with the partitioned load cases. The outputs are printed only on the master
processor after the master processor collects the results that you requested through
communications.

SUBCASE 101
D I S P L A C E M E N T V E C T O R
POINT ID. TYPE T1 T2 T3 R1 R2 R3
151000 G 3.392543E-07 1.089757E-21 1.783046E-08 .0 -1.204824E-07 -3.644054E-2
151010 G 3.392543E-07 1.426177E-21 -1.783046E-08 .0 1.204824E-07 -1.336866E-2

SUBCASE 102

D I S P L A C E M E N T V E C T O R
POINT ID. TYPE T1 T2 T3 R1 R2 R3
150205 G -6.509690E-07 -2.062356E-07 8.779039E-22 .0 -1.718401E-21 -1.224331E-07
150206 G -5.072370E-07 -1.837194E-07 -4.060963E-08 .0 1.252286E-07 2.982923E-08

SUBCASE 500
D I S P L A C E M E N T V E C T O R
POINT ID. TYPE T1 T2 T3 R1 R2 R3
150205 G -1.303690E-06 -6.062316E-08 9.779011E-20 .0 -3.418412E-22 -8.223322E-06
150206 G -2.172310E-06 -1.937191E-06 -3.060233E-05 .0 4.652283E-08 6.975915E-06

45
4.2 Normal Mode Analysis: SOL 103

A finite element plate model with 110 grid points and 100 CQUAD4 elements is
executed in parallel with SOL 103 analysis. The total number of degrees of freedom is
660. Four processors are used for the DMP run.

The parallel normal mode analysis is performed with FDMODES, GDMODES, and
HDMODES. For each computational method, both SEQP and GPARTN are implemented
for the domain decomposition.

4.2.1 FDMODES

FDMODES does not require domain decomposition. The .f06 file shows the number of
eigenvalues calculated in each processor. Each processor computes 94, 61, 26, and 27
eigenvalues, respectively. The total number of modes found is 208.

The master processor collects the eigenvalues and eigenvectors of slave processors, and
prints out the merged eigensolutions.

FREQUENCY DOMAIN PARALLEL LANCZOS METHOD

94 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 1

61 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 2

26 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 3
27 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 4

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 5
NUMBER OF ROOTS FOUND ................ 208
NUMBER OF SOLVES REQUIRED ............ 45

$-------------------------------------------------------------------------------
$ The list of all eigenvalues collected from the slave processor
$-------------------------------------------------------------------------------

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED GENERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

With “slaveout=yes”, you can see the information about the eigenvalue problem on each
slave processor. It also shows the number of eigenvalues found on the slave processors.

46
****************
S L A V E 1
****************

FREQUENCY DOMAIN PARALLEL LANCZOS METHOD

#
61 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 2

****************
S L A V E 2
****************

FREQUENCY DOMAIN PARALLEL LANCZOS METHOD

#
26 E IGENVALUES FOUND IN DISTRIBUTED SEGMENT # 3

****************
S L A V E 3
****************

FREQUENCY DOMAIN PARALLEL LANCZOS METHOD

#
27 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 4

47
4.2.2 GDMODES

GDMODES is implemented with the nastran keyword “gpart”. The default is “gpart=0”,
which uses SEQP module. The option “gpart=1” calls the GPARTN module.

¾ gpart=0 (default)

When the domain is partitioned, the output from the SEQP module is shown as:

STATISTICS FROM AUTOMATIC MODEL PARTITIONER

THE NUMBER OF DOMAINS CREATED USING EDSMLV IS 4

THE NUMBER OF GRID POINTS IN THE GLOBAL BOUNDARY IS 30
THE NUMBER OF ELASTIC ELEMENTS IN THE RESIDUAL IS 0
THE NUMBER OF RIGID ELEMENTS IN THE RESIDUAL IS 0
DOMAIN ID # INTERNAL GRID POINTS # EXTERNAL GRID POINTS # OF ELEMENTS
--------- ---------------------- ---------------------- -------------
1 20 16 25
2 20 16 25
3 20 16 25
4 20 16 25

GEOMETRY DOMAIN PARALLEL LANCZOS METHOD

The partitioning statistics shows the number of domains created, grid points, and
elements for each sub-domain.

Once the results of eigenvalue analysis on the slave processors are collected, the master
processor prints the summary of merged eigensolutions.

208 EIGENVALUES FOUND IN DOMAIN # 1

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 208
NUMBER OF SOLVES REQUIRED ............ 41

$-------------------------------------------------------------------------------
$ The list of all eigensolutions collected from the slave processor
$-------------------------------------------------------------------------------

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

48
With the gpart=0 option, the summary of eigenvalue analysis from the master processor
is broadcasted to the slave processors, so that all processors print the same summary
information.

****************
S L A V E 1
****************
#
209 EIGENVALUES FOUND IN DOMAIN # 2

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 208

NUMBER OF SOLVES REQUIRED ............ 41

$-------------------------------------------------------------------------------
$ The list of all eigensolutions collected from the slave processor
$-------------------------------------------------------------------------------

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

¾ gpart=1

The partitioning statistics from the GPARTN module is shown as

RESULT OF SESET PARTITIONS:

TOTAL PARTITIONS TOTAL GRIDS TOTAL BDY. GRIDS MOVES

4 496 138 0
PARTITION INTERIOR GRIDS BDY. GRIDS FRACTION
1 88 74 .8409
2 91 74 .8132
3 88 74 .8409
4 91 74 .8132

GEOMETRY DOMAIN PARALLEL LANCZOS METHOD

Note that the output is related to the degrees of freedom (DOF), not grid points, even
though the output uses GRIDS terminology instead of DOF. This example has 88 internal
degrees of freedom and 74 boundary degrees of freedom for domain 1. There are 91 and
74, 88 and 74, 91 and 74 degrees of freedom for the interior and boundary of domain 2, 3,
and 4, respectively.

49
The .f06 file of the master processor prints the summary of eigenvalue analysis.

208 EIGENVALUES FOUND IN DOMAIN # 1

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 208
NUMBER OF SOLVES REQUIRED ............ 41
$-------------------------------------------------------------------------------
$ The list of eigensolutions collected from the slave processor
$-------------------------------------------------------------------------------

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

However, with the gpart=1 option, the master processor does not broadcast the collected
eigenvalues and/or eigenvectors to the slave processors, so no eigensolutions are printed
in the.f06 file except the eigenvalue summary.

#
****************
S L A V E 1
****************
#
208 EIGENVALUES FOUND IN DOMAIN # 2

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3

NUMBER OF ROOTS FOUND ................ 208

NUMBER OF SOLVES REQUIRED ............ 41

****************
S L A V E 2
****************
#
208 EIGENVALUES FOUND IN DOMAIN # 3

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

50
 BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3

NUMBER OF ROOTS FOUND ................ 208

NUMBER OF SOLVES REQUIRED ............ 41

****************
S L A V E 3
****************
#
208 EIGENVALUES FOUND IN DOMAIN # 4

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3

NUMBER OF ROOTS FOUND ................ 208

NUMBER OF SOLVES REQUIRED ............ 41

51
4.2.3 HDMODES

In the example below, HDMODES is executed with ‘dmp=4’ and ‘nclust=2’ keywords.
Among the four processors, the first processor is the master processor, and the other
processors are the slave processors.

Note that HDMODES defines the local master and local slave processors inside of each
cluster.

¾ gpart=0 (default)

The partitioning statistics show the information of domains for each cluster. There are
two domains for each cluster. The first domain of the cluster has 50 internal grids and 10
external grids, and the second domain has the same local size. Generally, the size of each
domain is different.

STATISTICS FROM AUTOMATIC MODEL PARTITIONER

THE NUMBER OF DOMAINS CREATED USING EDSMLV IS 2
THE NUMBER OF GRID POINTS IN THE GLOBAL BOUNDARY IS 10
THE NUMBER OF ELASTIC ELEMENTS IN THE RESIDUAL IS 0
THE NUMBER OF RIGID ELEMENTS IN THE RESIDUAL IS 0
DOMAIN ID # INTERNAL GRID POINTS # EXTERNAL GRID POINTS # OF ELEMENTS
--------- ---------------------- ---------------------- -------------
1 50 10 50
2 50 10 50

HIERARCHIC DOMAIN PARALLEL LANCZOS METHOD

In this example, cluster 1 includes processors 1 and 3, and processors 2 and 4 are in
cluster 2. Processors 1 and 2 are the local masters in clusters 1 and 2, respectively. In
cluster 1, 155 eigenvalues are collected in processor 1. Similarly, in cluster 2, 53
eigenvalues are collected in processor 2. Once each local master processor collects the
results for each cluster, the master processor (processor 1) collects the information from
the local masters.

In the .f06 file, the master processor prints the summary of the eigenvalue analysis and
the list of eigenvalues. Be careful when interpreting the “number of roots found”
information in the summary. This information concerns the cluster in which the master is
included. For example, in the model below, interpret the “number of roots found”
message as indicating that cluster 1 found 155 eigenvalues. The master processor lists all
208 eigenvalues that are merged from all of the local master processors. You should
determine the number of modes from the list of eigenvalues.

52
 155 EIGENVALUES FOUND IN DOMAIN # 1

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 155
NUMBER OF SOLVES REQUIRED ............ 41
$-----------------------------------------------------------------------------------
$ The list of all eigensolutions collected from the slave processor
$-----------------------------------------------------------------------------------

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

Again, be cautious when interpreting the output from slave processors. The summary of
eigenvalue analysis for slave processors is confined to the corresponding processor. In
the list of eigenvalues on the slave processors, the mode number does not represent the
global mode number.

Note that, with the gpart=0, the master process broadcasts the collected output only to
the local masters. In the example below, the local slave, processor 2, prints 208
eigenvalues, while another local slave, processor 4, prints 53 eigenvalues.

#
****************
S L A V E 1
****************
#
53 EIGENVALUES FOUND IN DOMAIN # 1

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 2
NUMBER OF ROOTS FOUND ................ 53
NUMBER OF SOLVES REQUIRED ............ 39

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.632620E+09 4.040570E+04 6.430766E+03 1.000000E+00 1.632620E+09
2 2 1.633399E+09 4.041533E+04 6.432300E+03 1.000000E+00 1.633399E+09
#

53
 52 52 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
53 53 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

****************
S L A V E 2
****************
#
155 EIGENVALUES FOUND IN DOMAIN # 2

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 155
NUMBER OF SOLVES REQUIRED ............ 41

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

****************
S L A V E 3
****************
#
53 EIGENVALUES FOUND IN DOMAIN # 2
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)
BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 2

NUMBER OF ROOTS FOUND ................ 53
NUMBER OF SOLVES REQUIRED ............ 39

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.632620E+09 4.040570E+04 6.430766E+03 1.000000E+00 1.632620E+09
2 2 1.633399E+0 4.041533E+04 6.432300E+03 1.000000E+00 1.633399E+09
#
52 52 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
53 53 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

54
¾ gpart=1

The partitioning statistics with GPARTN describe the number of degrees of freedom for
each cluster. In the example below, the first cluster has 223 degrees of freedom in the
interior of domain 1, and 50 degrees of freedom in the boundary. The second cluster has
the same local size as the first one.

TOTAL PARTITIONS TOTAL GRIDS TOTAL BDY. GRIDS MOVES

2 496 50 0
PARTITION INTERIOR GRIDS BDY. GRIDS FRACTION
1 223 50 .2242
2 223 50 .2242

HIERARCHIC DOMAIN PARALLEL LANCZOS METHOD

The interpretation of the .f06 file for the master processor is the same as that of the
gpart=0 option.

155 VALUES FOUND IN DOMAIN # 1

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 155
NUMBER OF SOLVES REQUIRED ............ 41

$-------------------------------------------------------------------------------
$ The list of all eigensolutions collected from the slave processor
$-------------------------------------------------------------------------------

R E A L E I G E N V A L U E S
MODE EXTRACTION EIGENVALUE RADIANS CYCLES GENERALIZED NERALIZED
NO. ORDER MASS STIFFNESS
1 1 1.696349E+07 4.118676E+03 6.555076E+02 1.000000E+00 1.696349E+07
2 2 1.848026E+07 4.298867E+03 6.841859E+02 1.000000E+00 1.848026E+07
#
207 207 3.073819E+09 5.544203E+04 8.823873E+03 1.000000E+00 3.073819E+09
208 208 3.123167E+09 5.588530E+04 8.894422E+03 1.000000E+00 3.123167E+09

With gpart=1, the master process does not broadcast the collected output to slave
processors, so no eigenvalues are listed in the information from the slave processors.

55
 #
****************
S L A V E 1
****************
#
53 EIGENVALUES FOUND IN DOMAIN # 1
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 2
NUMBER OF ROOTS FOUND ................ 53
NUMBER OF SOLVES REQUIRED ............ 40

****************
S L A V E 2
****************
#
155 EIGENVALUES FOUND IN DOMAIN # 2
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 3
NUMBER OF ROOTS FOUND ................ 155
NUMBER OF SOLVES REQUIRED ............ 41

****************
S L A V E 3
****************
#
53 EIGENVALUES FOUND IN DOMAIN # 2

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

BLOCK SIZE USED ...................... 7

NUMBER OF DECOMPOSITIONS ............. 2
NUMBER OF ROOTS FOUND ................ 53
NUMBER OF SOLVES REQUIRED ............ 40

56
4.3 Direct Frequency Response Analysis: SOL 108

For a finite element plate model that has 110 grid points and 100 CQUAD4 elements,
SOL 108 analysis is performed in parallel with the FDFREQR method. The total number
of excitation frequencies is 89 in the entry of FREQ1.

FREQ1 10 3.0 3.0 88

The .f06 file describes the partitioned frequency range. In the example below, processors
1, 2, 3, and 4 execute 23, 22, 22, and 22 excitation frequency ranges. Once each
processor finishes its own analysis, the master processor collects the results and prints the
output.

#
DISTRIBUTED MEMORY PARALLEL FREQUENCY RESPONSE
NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES ON LOCAL PROCESSOR (ID= 1) = 23
#
$------------------------------------------------------------------------------
$ The list of all responses collected from the slave processor
$------------------------------------------------------------------------------

R E S P O N S E O U T P U T
****************
S L A V E 1
****************
#
DISTRIBUTED MEMORY PARALLEL FREQUENCY RESPONSE
NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES ON LOCAL PROCESSOR (ID= 2) = 22
#
****************
S L A V E 2
****************
#
DISTRIBUTED MEMORY PARALLEL FREQUENCY RESPONSE
NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES ON LOCAL PROCESSOR (ID= 3) = 22
#
****************
S L A V E 3
****************
#
DISTRIBUTED MEMORY PARALLEL FREQUENCY RESPONSE
NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES ON LOCAL PROCESSOR (ID= 4) = 22
#

57
4.4 Modal Frequency Response Analysis: SOL 111

In SOL 111, the FDMODES, GDMODES, and HDMODES methods will automatically
perform FDFREQR after the eigenvalue analysis is finished, so you do not need to
request FDFREQR explicitly for the modal frequency response. Therefore, the parallel
SOL 111 job can be run exactly the same way as the parallel SOL 103. In the example
below, SOL 103 is executed in parallel and is run with four processors.

4.4.1 FDMODES and FDFREQR

Each processor computes 94, 61, 26, and 27 eigenvalues, respectively, for the whole
finite element model. For the modal frequency response analysis, 19 excitation
frequencies are split to 5, 5, 5, and 4, and distributed to each processor.

#
$------------------------------------------------------------------------------
$ The number of eigenvalues found at each processor
$------------------------------------------------------------------------------

94 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 1

61 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 2

26 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 3

27 EIGENVALUES FOUND IN DISTRIBUTED SEGMENT # 4

#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)
#
$------------------------------------------------------------------------------$
The number of excitation frequencies partitioned for each processor
$------------------------------------------------------------------------------

PERFORMANCE SUMMARY TABLE FOR DISTRIBUTED MEMORY FREQUENCY RESPONSE

NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES = 19
PROCESSOR # FREQ. CPU (SEC) ELAPSED (SEC)
--------- ------- --------- -------------
1. ugs001 5 1.57 2.22
2. ugs002 5 1.87 2.22
3. ugs003 5 1.75 2.22
4 ugs004 4 1.49 2.20
#
$------------------------------------------------------------------------------
$ The list of all responses collected from the slave processor
$------------------------------------------------------------------------------

R E S P O N S E O U T P U T

#
****************
S L A V E 1
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

58
 ****************
S L A V E 2
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

****************
S L A V E 3
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

4.4.2 GDMODES and FDFREQR

Once the eigenvalue analysis is finished in parallel in SOL 103, both the gpart=0 and the
gpart=1 options use the same method to partition the range of excitation frequencies.
Once each processor finishes the modal frequency response analysis, the master process
collects all response output and prints it in the .f06 file.

¾ gpart=0 (default)

#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
PERFORMANCE SUMMARY TABLE FOR DISTRIBUTED MEMORY FREQUENCY RESPONSE
NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES = 19
PROCESSOR # FREQ. CPU (SEC) ELAPSED (SEC)
--------- ------- --------- -------------
1. ugs001 5 5.61 11.09
2. ugs002 5 5.16 11.08
3. ugs003 5 5.70 11.08
4. ugs004 4 5.33 11.07
#
$-----------------------------------------------------------------------------------
$ The list of all responses collected from the slave processor
$-----------------------------------------------------------------------------------

R E S P O N S E S U M M A R Y

59
¾ gpart=1

#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

PERFORMANCE SUMMARY TABLE FOR DISTRIBUTED MEMORY FREQUENCY RESPONSE

NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES = 19

PROCESSOR # FREQ. CPU (SEC) ELAPSED (SEC)

--------- ------- --------- -------------
1. ugs001 5 5.61 11.09
2. ugs002 5 5.16 11.08
3. ugs003 5 5.70 11.08
4. ugs004 4 5.33 11.08

#
R E S P O N S E S U M M A R Y

4.4.3 HDMODES and FDFREQR

For a SOL 111 run with HDMODES and FDFREQR, it is strongly recommended to run
with gpart=1 because gpart=1 takes full advantage of all computational resources.

¾ gpart=0 (default)

In the gpart=0 option, only local master processors are involved in the FDFREQR
method. Because the slave processors are idle in the modal frequency response analysis,
you cannot obtain the maximum performance. Once the local master processors finish the
calculation of responses, the master processor collects them and broadcasts to the local
masters. In the example below, two local master processors, processor 1 and 3, contain
the response results in their entirety.

#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

60
 PERFORMANCE SUMMARY TABLE FOR DISTRIBUTED MEMORY FREQUENCY RESPONSE

NUMBER OF FREQUENCY DOMAINS = 2

NUMBER OF FREQUENCIES = 19

PROCESSOR # FREQ. CPU (SEC) ELAPSED (SEC)

--------- ------- --------- -------------
1. ugs001 10 4.19 4 .18
2. ugs002 9 4.17 4.18
#
$-----------------------------------------------------------------------------------
$ The list of all responses collected from the slave processor
$-----------------------------------------------------------------------------------

R E S P O N S E O U T P U T

#
****************
S L A V E 1
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
$------------------------------------------------------------------------------
$ The list of all responses collected from the slave processor
$------------------------------------------------------------------------------

R E S P O N S E O U T P U T

#
****************
S L A V E 2
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
****************
S L A V E 3
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

¾ gpart=1

61
With the gpart=1 option, all processors are involved in the FDFREQR method, so you
can obtain the maximum performance. The master processor collects the results from the
slave processors, and prints the complete response results.

#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
PERFORMANCE SUMMARY TABLE FOR DISTRIBUTED MEMORY FREQUENCY RESPONSE
NUMBER OF FREQUENCY DOMAINS = 4
NUMBER OF FREQUENCIES = 19

PROCESSOR # FREQ. CPU (SEC) ELAPSED (SEC)

--------- ------- --------- -------------
1. ugs001 5 1.82 1.98
2. ugs002 5 1.98 1.97

3. ugs003 5 1.82 1.97

4. ugs003 4 1.66 1.97
#
$-----------------------------------------------------------------------------------
$ The list of all responses collected from the slave processor
$-----------------------------------------------------------------------------------

R E S P O N S E O U T P U T

#
****************
S L A V E 1
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
****************
S L A V E 2
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
****************
S L A V E 3
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

62
4.5 Modal Transient Response Analysis: SOL 112

In the parallel SOL 112 run, once the eigenvalue analysis is performed in parallel, the
modal transient response analysis is executed in serial. During the modal transient
response analysis, all slave processors are idle. The output of responses in the .f06 file is
printed only in the master processor.

#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
$-----------------------------------------------------------------------------------
$ Only the master processor run the transient responses analysis
$-----------------------------------------------------------------------------------

R E S P O N S E O U T P U T
#

****************
S L A V E 1
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
****************
S L A V E 2
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
****************
S L A V E 3
****************
#
E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

63
4.6 Optimization: SOL 200

In the optimization procedures, several different types of analyses are involved. If any
eigensolution is required, the eigenvalue analysis may be performed in parallel with the
computational methods, FDMODES, GDMODES, and HDMODES, used in SOL 103.

In the iterations of optimization procedure, the optimization algorithm runs in serial at

each processor except the eigenvalue analysis. Whenever the eigenvalue analysis is
performed in parallel, the collected and merged eigensolutions from the master
processors are broadcast to all processors. This approach results in the same optimization
results for every processor, because all processors proceed the optimization using the
same eigensolutions in the optimization loop.

Note that GDMODES and HDMODES are valid with only the gpart=1 option.

The following .f06 is a typical output format in SOL 200 with DMP, in which all
processors print the same output.

#
i-th I T E R A T I O N
#
$-----------------------------------------------------------------------------------
$ run FDMODES, GDMODES, or HDMODES, and collect the local eigensolutions
$ from the slave processors
$-----------------------------------------------------------------------------------

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
O P T I M I Z A T I O N O U T P U T
#
j-th I T E R A T I O N
#
****************
S L A V E 1
****************
#
i-th I T E R A T I O N
#
$-----------------------------------------------------------------------------------
$ run FDMODES, GDMODES, or HDMODES, and have the same eigensolutions
$ as the master processor
$-----------------------------------------------------------------------------------

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
O P T I M I Z A T I O N O U T P U T

64
 #

j-th I T E R A T I O N
#
****************
S L A V E 2
****************
#
i-th I T E R A T I O N
#
$-----------------------------------------------------------------------------------
$ run FDMODES, GDMODES, or HDMODES, and have the same eigensolutions
$ as the master processor
$-----------------------------------------------------------------------------------

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
O P T I M I Z A T I O N O U T P U T
#
j-th I T E R A T I O N
#
****************
S L A V E 3
****************
#
i-th I T E R A T I O N
#
$-----------------------------------------------------------------------------------
$ run FDMODES, GDMODES, or HDMODES, and have the same eigensolutions
$ as the master processor
$-----------------------------------------------------------------------------------

E I G E N V A L U E A N A L Y S I S S U M M A R Y (READ MODULE)

#
O P T I M I Z A T I O N O U T P U T
#
j-th I T E R A T I O N

65
4.7 Industrial Case Study

A trimmed car body FE model with SOL 103 is used as a case study to analyze the
technologies presented in NX Nastran. This type of car model has all major components
of the car, such as wheels, engine, etc. incorporated. Tables 4-1 and 4-2 present the
details for this finite element model.

Table 4-1 Model statistics of trimmed car body FE model

Number of Number of Number of Number of

nodes shell elements solid elements rigid elements
380,007 361,249 3,762 9,056

Table 4-2 The size of sets for the trimmed car body FE model

The size of set

g-set n-set f-set a-set

2,280,042 2,223,139 2,223,109 1,937,282

Table 4-3 demonstrates the effect of the automated geometric domain decomposition on
this model by showing the number of interior and boundary nodes of the partitions.
Several observations can be made. The interior range size depends on the quality of the
automated partitioning. The boundary size increases with the number of partitions.
Finally, the boundary size is at least two orders of magnitude smaller than the interior
size, which is important to the computational efficiency.

Table 4-3 Results of geometry domain decomposition

Number of Maximum Minimum Maximum Minimum

Partitions Interior Interior Boundary Boundary

2 197,460 182,199 354 354

4 104,788 90,791 572 418
8 54,496 40,067 518 450

66
The automated frequency domain decomposition results are shown in Table 4-4. There
are 840 modes in the frequency range of interest.

Table 4-4. Frequency domain decomposition statistics

Number of Minimum number of Maximum number of

segments modes among segments modes among segments
2 380 460
4 193 252
6 133 165
7 105 141
8 92 119

The automated geometric domain partitioning techniques usually provide only even, and
preferably binary, numbered domains. This is because these techniques are primarily
based on binary graph partitioning. This is not a restriction, as shared memory
workstations tend to have an even number of processors. On the other hand, in the
frequency domain decomposition, odd numbers of segments are also allowed. This
technology is insensitive to that issue and enables the use of odd-numbered workstations
via the hierarchic technology in a workstation cluster environment.

The analysis was executed on a cluster of eight workstations, each containing eight
processors with a 1.5 GHz clock cycle. The cluster had a one gigabyte Ethernet network
connection. The option gpart=1 is used.

Table 4-5. Execution times on workstation cluster with HDMODES

Number of Number of Number of Elapsed

I/O (GB) Speedup
processors partitions segments time[min:sec]
1 1 1 1,028.4 528:58 1.00
4 2 2 431.4 167:15 3.13
8 4 2 266.9 83:41 6.26
16 8 2 191.1 45:16 11.57
32 8 4 98.3 34:07 15.35
48 8 6 77.8 27:14 19.23
56 8 7 67.1 24:41 21.22
64 8 8 61.4 27:00 19.40

67
The task of finding the natural frequencies and mode shapes of such a model is an
enormous one. It is an overnight job with more than a terabyte of I/O operations. The
execution on a single processor is impractical considering the work environment and time
schedule at automobile companies.

Fig. 4.1 and Fig.4.2 show the elapsed time and disk I/O of HDMODES with 8 geometry
partitions for the different number of frequency segment. With 56 processors, 8 geometry
partitions in each workstation and 7 frequency segments across workstations are used.
The elapsed time for 32 processors is already a practical execution. The efficiency above
decreases, but the speedup is still increasing. It peaks at 56 processors, although a wider
frequency range for this model may extend that peak to 64 or higher. It means that 7
frequency segments with 8 geometry partitioning for each frequency segment is the most
suitable partitioning for this test FE model in this workstation cluster environment.

HDMODES
(8 geometry partitions)

600

550

500

450

400
Elapsed time (min.)

350

300

250

200

150

100

50

0
1 16 32 48 56 64
Number of processors

Fig. 4.1 Elapsed time of HDMODES with 8 geometry partitions for the different number
of frequency segment.

68
 HDMODES
(8 geometry partitions)

1100

1000

900

800

700
Disk I/O (GB)

600

500

400

300

200

100

0
1 16 32 48 56 64
Number of processors

Fig. 4.2 Disk I/O of HDMODES with 8 geometry partitions for the different number of
frequency segment.

69
References

[1] L. Komzsik, The Lanczos Method: Evolution and Application, 2003, SIAM.

[2] W. Gropp, E. Lusk, and A. Skjellum, Using MPI: Portable Parallel Programming with
the Message Passing Interface, 1997, MIT.

[3] NX Nastran Numerical Methods User’s Guide, 2005, UGS.

70