Enhancements in DFSORT and ICETOOL
Enhancements in DFSORT and ICETOOL
October, 2010
Frank L. Yaeger
DFSORT Team
IBM Systems Software Development
San Jose, California
Internet: yaeger@[Link]
For papers, online books, news, tips, examples and more, visit the DFSORT home page at URL:
[Link]
ii New in DFSORT
Abstract
Over the years, DFSORT has provided important enhancements in many areas. This paper provides a quick intro-
duction to selected DFSORT and ICETOOL enhancements available as of October, 2010. Examples using the
various features are included, where appropriate.
Abstract iii
iv New in DFSORT
Contents
What's New in DFSORT 1
Introduction 1
JOINKEYS Application 2
ICETOOL Features 3
Changing Installation Options via PARMLIB 13
Reformatting Features (INREC, OUTREC, OUTFIL) 13
OUTFIL-Only Features 20
INCLUDE and OMIT Features 24
Symbols for Fields and Constants 27
Symbols for Output Columns 28
Symbols for %nn Parsed Fields 29
System Symbols 29
SET and PROC Symbols 30
Symbols for RACF, DFSMSrmm and DCOLLECT 30
SUM Features and Extensions 30
SORT and MERGE Extensions 31
Larger Fields and Constants 31
Free Form Formats 31
Managed Tape Processing 32
Improvements for RACF's IRRUT200 Utility 32
64-Bit Architecture 33
Multiple Hiperspaces 33
Easier Migration from Other Sort Products 33
VSAM Processing 34
Processing z/OS Unix Files 35
Larger Tape Block Sizes 35
Long Variable-Length Output Records 36
Signed/Unsigned Zero 37
Time-of-Day Installation Options Control 37
More Work Data Sets 38
More Merge Data Sets 38
Simplified Installation and Customization 38
Improvements for Copy, Merge and ICEGENER Applications 39
Incomplete Spanned Records 39
LRECL Padding and Truncation 39
Contents v
vi New in DFSORT
What's New in DFSORT
Introduction
DFSORT is IBM's high performance sort, merge, copy, analysis and reporting product. DFSORT is an optional
feature of z/OS.
DFSORT, together with DFSMS and RACF, form the strategic product base for the evolving system-managed
storage environment. DFSMS provides vital storage and data management functions. RACF adds security func-
tions. DFSORT adds the ability to do faster and easier sorting, merging, copying, reporting and analysis of your
business information, as well as versatile data handling at the record, field and bit level.
This paper provides a quick introduction to selected DFSORT and ICETOOL enhancements available as of
October, 2010. Examples using the various features are included, where appropriate.
Complete information on the newest features of DFSORT and DFSORT's ICETOOL can be found in User Guide
for DFSORT PTFs UK90025 and UK90026 (October, 2010).
Complete information on all of the features of DFSORT and DFSORT's ICETOOL through September, 2010 can be
found in z/OS DFSORT Application Programming Guide (SC26-7523-05) and z/OS DFSORT Installation and
Customization (SC26-7524-03). If you're not familiar with DFSORT, DFSORT's ICETOOL or DFSORT Symbols,
you should consider going through z/OS DFSORT: Getting Started (SC26-7527-05). You can access these docu-
ments online by clicking the Publications link on the DFSORT home page at URL:
[Link]
JOINKEYS Application
The new JOINKEYS application support is a significant addition to DFSORT's join and match capabilities. A
JOINKEYS application makes it easy to create joined records in a variety of ways including inner join, full outer
join, left outer join, right outer join, and unpaired combinations. The two input files can be of different types
(fixed, variable, VSAM, and so on) and have keys in different locations.
Three new control statements can be used for a JOINKEYS application. One JOINKEYS statement is required for
each input file to indicate the ddname of the file, describe the keys, indicate whether the file is already sorted by
those keys, and so on. An inner join is performed by default, but a JOIN statement can be used to specify a
different type of join. A REFORMAT statement is used to describe the fields from the two files to be included in
the joined records, and optionally an indicator of where the key was found ('B' for both files, '1' for file1 only or '2'
for file2 only).
The records from the two input files can be processed in a variety of ways before and after they are joined.
Here's an example of a Cartesian join of two files that have their keys in different locations.
2 New in DFSORT
//JK EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//VBIN DD DSN=... VB input file
//FBIN DD DSN=... FB input file
//SORTOUT DD DSN=... FB output file
//SYSIN DD *
* Control statements for JOINKEYS application
JOINKEYS F1=VBIN,FIELDS=(18,16,A),SORTED
JOINKEYS F2=FBIN,FIELDS=(1,16,A)
REFORMAT FIELDS=(F2:22,12,F1:5,12,F2:1,16)
* Control statements for main task (joined records)
OPTION EQUALS
SORT FIELDS=(13,12,CH,A)
/*
ICETOOL Features
DFSORT's popular multipurpose ICETOOL utility offers many important features:
The RESIZE operator is a significant addition to ICETOOL's wide range of capabilities. For fixed-length
records, RESIZE lets you create a larger record from multiple shorter records, or multiple shorter records from
a larger record. You just indicate the desired length of the output records and RESIZE automatically resizes
the records based on the input length.
For example, the following ICETOOL job creates 40-byte output records from 10-byte input records:
//INCRS EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN DD DSN=... input file (FB/10)
//OUT DD DSN=... output file (FB/40)
//TOOLIN DD *
RESIZE FROM(IN) TO(OUT) TOLEN(40)
/*
If IN contained the following records:
Bird
Bluejay
4
Charlie
Rodent
Rat
2
Sara
OUT would contain the following records:
Bird Bluejay 4 Charlie
Rodent Rat 2 Sara
For the DISPLAY operator, the length for CH values has been raised from 1500 bytes to 4000 bytes, the length
for HEX values has been raised from 1000 bytes to 2000 bytes, the number of ON fields has been raised from
20 to 50, and the line length has been raised from 2048 bytes to 8192 bytes. This lets you view and analyze
more of your data.
The JKFROM operand makes it easy to perform one or more JOINKEYS applications using ICETOOL's
COPY or SORT operators.
4 New in DFSORT
//SELN EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//INPUT DD DSN=... input file
//OUTPUT DD DSN=... output file
//TOOLIN DD *
SELECT FROM(INPUT) TO(OUTPUT) ON(1,5,CH) FIRST(3) USING(CTL1)
/*
//CTL1CNTL DD *
SORT FIELDS=(1,5,CH,A,11,3,ZD,D)
/*
If INPUT contained the following records:
JUNE 025
APRIL 077
JUNE 072
JUNE 124
APRIL 051
APRIL 058
JUNE 082
JUNE 095
APRIL 005
OUTPUT would contain the following records:
APRIL 077
APRIL 058
APRIL 051
JUNE 124
JUNE 095
JUNE 082
USING(xxxx) can be used to supply DFSORT control statements INCLUDE, OMIT, INREC, OPTION, SORT
and OUTFIL for SELECT processing. INCLUDE and OMIT can be used to delete records that you don't want
to SELECT. INREC can be used to reformat the input records before they are selected. OPTION can be used
to specify optional parameters such as MAINSIZE. SORT can be used to specify fields you want to sort on,
but not select on. OUTFIL statements can be used to further process the selected records.
In the following example, an OMIT statement is used to ensure that records with C'NO' in positions 25-26 are
not processed by the SELECT operator, and a SORT statement is used to sort/select the records ascending by
the field in positions 12-16 and sort the records descending by the field in positions 35-40.
SELECT FROM(IN) TO(OUT) ON(12,5,CH) FIRSTDUP USING(CTL1)
//CTL1CNTL DD *
OMIT COND=(25,2,CH,EQ,C'NO')
SORT FIELDS=(12,5,CH,A,35,6,CH,D)
In the following example, OUTFIL statements are used to reformat selected and discarded records for certain
record types.
SELECT FROM(INPUT1) TO(OUT1) DISCARD(OUT2) -
ON(31,6,ZD) FIRST USING(CTL2)
//CTL2CNTL DD *
OUTFIL FNAMES=OUT1,INCLUDE=(6,1,BI,EQ,+1,OR,6,1,BI,EQ,+5),
BUILD=(1,20,21,4,PD,TO=ZD,90:X)
OUTFIL FNAMES=OUT2,INCLUDE=(6,1,BI,EQ,+12),
BUILD=(1,12,13,5,PD,TO=ZD,X,42,4,FI,TO=ZD,120:X)
FIRSTDUP and LASTDUP let you select the first or last record of each set of duplicates.
6 New in DFSORT
Vezinaw 207
Terradista 503
Van Noorden 207
Yaeger 503
Paulsen 806
Samuels 806
Wilson 207
We want to join the records with matching city codes in positions 1-3 of IN1 and positions 14-16 of IN2 to
produce an output data set with RECFM=FB and LRECL=60 that contains the following records:
Vezinaw 207 New York NY 1992-05-18
Van Noorden 207 New York NY 1992-05-18
Wilson 207 New York NY 1992-05-18
Terradista 503 San Jose CA 1967-12-24
Yaeger 503 San Jose CA 1967-12-24
Paulsen 806 Denver CO 2001-10-03
Samuels 806 Denver CO 2001-10-03
The following ICETOOL job uses the SPLICE operator to join the records.
//S1 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN1 DD DSN=... input file1
//IN2 DD DSN=... input file2
//T1 DD DSN=&&T1,UNIT=SYSDA,SPACE=(CYL,(5,5)),DISP=(MOD,PASS)
//OUT DD DSN=... output file
//TOOLIN DD *
* Reformat IN1 to contain:
* | blanks | key | IN1 data |
COPY FROM(IN1) TO(T1) USING(CTL1)
* Reformat IN2 to contain:
* | IN2 data | key | blanks |
COPY FROM(IN2) TO(T1) USING(CTL2)
* SPLICE the matching IN1/IN2 records to produce:
* | IN2 data | key | IN1 data |
SPLICE FROM(T1) TO(OUT) ON(14,3,CH) -
WITHALL WITH(1,13)
/*
//CTL1CNTL DD *
OUTREC BUILD=(14:1,3,17:4,26,60:X)
/*
//CTL2CNTL DD *
OUTREC BUILD=(1,20,60:X)
/*
In the next SPLICE example, the input records from data sets FILE1 and FILE2 are separated into the fol-
lowing output data sets:
– BOTH: records that appear in FILE1 and FILE2
– F1ONLY: records that only appear in FILE1
– F2ONLY: records that only appear in FILE2
Here's the ICETOOL job:
8 New in DFSORT
Frank
Paul
F2ONLY contains the following records:
Karen
Mary
For the COUNT operator, WRITE(countdd) lets you write the count of records to an output data set.
TEXT('string') can be used to insert a text string before the count of records, and DIGITS(d) or
EDCOUNT(formatting) can be used to format the count of records.
For example, the following COUNT operator writes a record in CT1 with text and the count of records from
IN1:
COUNT FROM(IN1) WRITE(CT1) TEXT('Record count is ') -
EDCOUNT(A1,U08)
If IN1 contained 865234 records, CT1 would contain this record:
Record count is 865,234
SUB(n) can be used to subtract n from the count of records and ADD(n) can be used to add n to the count of
records. This is especially useful for dealing with data sets that contain header and/or trailer records.
For example, if IN2 has a header, data records and a trailer record, the following COUNT operator writes a
record in CT2 with the count of data records:
COUNT FROM(IN2) WRITE(CT2) SUB(2) DIGITS(3)
If IN2 contained 375 records, CT2 would contain this record:
373
A return code of 12, 8 or 4 can be set if a specified data set is EMPTY, NOTEMPTY, HIGHER(n),
LOWER(n), EQUAL(n) or NOTEQUAL(n), where n is a specified number of records (for example, 50000).
This makes it easy to control the execution of downstream operators or steps using JCL facilities like IF or
COND. A return code of 12 is set with RC12 or by default. RC8 can be used to set a return code of 8
instead of a return code of 12. RC4 can be used to set a return code of 4 instead of a return code of 12.
For example, in the following job, the COUNT operator stops STEP2 from being executed by setting a return
code of 12 if the IN data set is empty.
//STEP1 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//IN DD DSN=...
//TOOLIN DD *
* SET RC=12 IF THE 'IN' DATA SET IS EMPTY, OR
* SET RC=0 IF THE 'IN' DATA SET IS NOT EMPTY
COUNT FROM(IN) EMPTY
/*
// IF [Link] = 0 THEN
//*** STEP2 WILL RUN IF 'IN' IS NOT EMPTY
//*** STEP2 WILL NOT RUN IF 'IN' IS EMPTY
//STEP2 EXEC ...
...
// ENDIF
The DISPLAY and OCCUR operators create useful reports with a minimum of work. Various options let
you create even better looking reports.
10 New in DFSORT
06/17/2008 Accounts Report for First Quarter
Account
Amount Id Number Date
----------- --- ------- -----
932.71 1 15932 01/06
1,376.22 2 00187 01/28
831.47 3 15932 02/12
1,832.61 4 02158 02/17
763.89 5 00187 03/05
9,200.13 6 15932 03/19
Format Result
DT1 SMF date interpreted as Z'yyyymmdd'
DT2 SMF date interpreted as Z'yyyymm'
DT3 SMF date interpreted as Z'yyyyddd'
DC1 TOD date interpreted as Z'yyyymmdd'
DC2 TOD date interpreted as Z'yyyymm'
DC3 TOD date interpreted as Z'yyyyddd'
DE1 ETOD date interpreted as Z'yyyymmdd'
DE2 ETOD date interpreted as Z'yyyymm'
DE3 ETOD date interpreted as Z'yyyyddd'
TM1 SMF time interpreted as Z'hhmmss'
TM2 SMF time interpreted as Z'hhmm'
TM3 SMF time interpreted as Z'hh'
TM4 SMF time interpreted as Z'hhmmssxx'
TC1 TOD time interpreted as Z'hhmmss'
TC2 TOD time interpreted as Z'hhmm'
TC3 TOD time interpreted as Z'hh'
TC4 TOD time interpreted as Z'hhmmssxx'
TE1 ETOD time interpreted as Z'hhmmss'
TE2 ETOD time interpreted as Z'hhmm'
TE3 ETOD time interpreted as Z'hh'
TE4 ETOD time interpreted as Z'hhmmssxx'
Number of cities: 4
FL format lets you display signed hexadecimal floating point values, totals, minimums, maximums and aver-
ages as their corresponding signed integer values. The integer values can be further edited using formatting
items. This makes it easy to display hexadecimal floating point values and statistics as integers.
Here's an example of displaying hexadecimal floating point values with DISPLAY:
DISPLAY FROM(SMF71) LIST(SMF71RPT) -
TITLE('Low impact central storage frames') -
HEADER('Min Frames') ON(925,8,FL,U10) -
HEADER('Max Frames') ON(933,8,FL,U10) -
HEADER('Avg Frames') ON(941,8,FL,U10) -
BLANK PAGE
12 New in DFSORT
Changing Installation Options via PARMLIB
ICEPRMxx members in concatenated PARMLIB can now be used to specify changes to DFSORT s installation
options. Each ICEPRMxx member can contain options to be changed for any or all of DFSORT s eight installation
environments (JCL, INV, TSO, TSOINV and TD1-TD4). Up to ten ICEPRMxx members can be activated by a
START ICEOPT started task command. The options in the activated members will be merged with the ICEMAC
defaults at run-time.
A different ICEPRMxx member, or combination of ICEPRMxx members, for different LPARs can be activated at
IPL time by including a START ICEOPT command in an appropriate COMMNDxx member in PARMLIB, or at
any time by issuing a START ICEOPT command from the console.
ICEPRMxx members are are easier to use and more flexible then the old method using the ICEMAC macro and
usermods.
The ICETOOL DEFAULTS operator can be used at any time to produce a report showing the merged
PARMLIB/ICEMAC installation default values for each environment that will be used at run-time, as well as the
active ICEPRMxx and ICEMAC values.
14 New in DFSORT
Here's an example of converting an mmddyy date to a ccyyddd date and a 3-character weekday string, and
converting a ccyyddd date to a ccyy/mm/dd date and 1-digit weekday string.
OPTION COPY,Y2PAST=1996
INREC BUILD=(1,6,Y2W,TOJUL=Y4T,X,
1,6,Y2W,WEEKDAY=CHAR3,X,
9,7,Y4T,TOGREG=Y4T(/),X,
9,7,Y4T,WEEKDAY=DIGIT1)
INREC, OUTREC and OUTFIL can create reformatted records in one of the following ways using unedited,
edited, or converted input fields and a variety of constants, as appropriate:
– FINDREP: Reformat each record by replacing character or hexadecimal input constants anywhere in the
record with character, hexadecimal or null output constants. FINDREP lets you do a variety of find and
replace operations on your records.
Here's an example of FINDREP with OUTREC:
OUTREC FINDREP=(IN=(C'D27',C'A52',C'X31'),OUT=C'INVALID')
– BUILD: Reformat each record by specifying all of its item one by one. BUILD gives you complete
control over the items you want in your reformatted INREC records and the order in which they appear.
You can delete, rearrange and insert fields and constants.
Here's an example of BUILD with INREC:
INREC BUILD=(1,20,C'ABC',26:5C'*',
15,3,PD,EDIT=([Link]),21,30,80:X)
Note: For INREC or OUTREC, you can use FIELDS or BUILD. For OUTFIL, you can use OUTREC
or BUILD.
– OVERLAY: Reformat each record by specifying just the items that overlay specific columns.
OVERLAY lets you change specific existing columns without affecting the entire record.
Here's an example of OVERLAY with OUTREC:
INREC OVERLAY=(45:45,8,TRAN=LTOU)
Lowercase letters in positions 45-52 of the input record will be changed to uppercase letters in positions
45-52 of the output records. All other bytes in the input record will be copied to the output record without
change.
– IFTHEN clauses: Reformat different records in different ways by specifying how FINDREP, BUILD or
OVERLAY items are applied to records that meet given criteria. IFTHEN clauses let you use sophisti-
cated conditional logic to choose how different record types are reformatted.
Here's an example of IFTHEN clauses with OUTFIL:
OUTFIL IFTHEN=(WHEN=(1,5,CH,EQ,C'TYPE1'),
BUILD=(1,40,C'**',+1,TO=PD)),
IFTHEN=(WHEN=(1,5,CH,EQ,C'TYPE2'),
BUILD=(1,40,+2,TO=PD,X'FFFF')),
IFTHEN=(WHEN=NONE),OVERLAY=(45:C'NONE'))
IFTHEN also lets you do various types of operations involving groups of records by making it easy to
propagate fields from the first record of a group to the other records of the group, or add an identifier
and/or sequence number to each record of the group. These functions are useful by themselves, and can
also facilitate other types of group operations such as sorting by groups, including or omitting groups, and
so on.
Here's an example of including groups of records consisting of a HDR record, data records and a TRL
record, while excluding records before, between and after the groups.
16 New in DFSORT
Operand Constant
DATE C'mm/dd/yy'
DATE1, DATE1-d, DATE1+d C'yyyymmdd'
DATE1(c), DATE1(c)-d, DATE1(c)+d C'yyyycmmcdd'
DATE1P, DATE1P-d, DATE1P+d P'yyyymmdd'
DATE2, DATE2-m, DATE2+m C'yyyymm'
DATE2(c), DATE2(c)-m, DATE2(c)+m C'yyyycmm'
DATE2P, DATE2P-m, DATE2P+m P'yyyymm'
DATE3, DATE3-d, DATE3+d C'yyyyddd'
DATE3(c), DATE3(c)-d, DATE3(c)+d C'yyyycddd'
DATE3P, DATE3P-d, DATE3P+d P'yyyyddd'
DATE4 C'[Link]'
DATE5 C'[Link]'
TIME C'hh:mm:ss'
TIME1 C'hhmmss'
TIME1(c) C'hhcmmcss'
TIME1P P'hhmmss'
TIME2 C'hhmm'
TIME2(c) C'hhcmm'
TIME2P P'hhmm'
TIME3 C'hh'
TIME3P P'hh'
This makes it easy to insert timestamps of various types into your output records. For example, if you used the
following OUTREC statement for a DFSORT run on March 16, 2005 at 03:46:21pm:
OUTREC BUILD=(2X,YDDD=(D4.),X,TIME1(:),X,1,10)
your SORTOUT records would look as follows:
075.2005 15:46:21 data
INREC, OUTREC and OUTFIL can convert signed hexadecimal floating point values (FL) into their corre-
sponding signed integer values. The integer values can be further edited using edit masks or edit patterns, or
further converted to BI, FI, PD, PDF, PDC, ZD, ZDF, ZDC or FS/CSF values. This makes it easy to display
and do arithmetic on hexadecimal floating point values.
Here's an example of displaying a hexadecimal floating point value as an integer using an edit mask:
INREC BUILD=(1,40,41:51,4,FL,M12)
INREC, OUTREC and OUTFIL can display a numeric field in various ways using DFSORT's M0-M26 pre-
defined edit masks, or edit patterns you define yourself using EDIT=(pattern) and SIGNS=(signs). You can
edit BI, FI, PD, PD0, ZD, CSF/FS, UFF, SFF, FL or Y2x input fields using edit masks or edit patterns. The
length of the output field can be defaulted or specified. As a simple example, if you specify:
Format Result
DT1 SMF date interpreted as Z'yyyymmdd'
DT2 SMF date interpreted as Z'yyyymm'
DT3 SMF date interpreted as Z'yyyyddd'
DC1 TOD date interpreted as Z'yyyymmdd'
DC2 TOD date interpreted as Z'yyyymm'
DC3 TOD date interpreted as Z'yyyyddd'
DE1 ETOD date interpreted as Z'yyyymmdd'
DE2 ETOD date interpreted as Z'yyyymm'
DE3 ETOD date interpreted as Z'yyyyddd'
TM1 SMF time interpreted as Z'hhmmss'
TM2 SMF time interpreted as Z'hhmm'
TM3 SMF time interpreted as Z'hh'
TM4 SMF time interpreted as Z'hhmmssxx'
TC1 TOD time interpreted as Z'hhmmss'
TC2 TOD time interpreted as Z'hhmm'
TC3 TOD time interpreted as Z'hh'
TC4 TOD time interpreted as Z'hhmmssxx'
TE1 ETOD time interpreted as Z'hhmmss'
TE2 ETOD time interpreted as Z'hhmm'
TE3 ETOD time interpreted as Z'hh'
TE4 ETOD time interpreted as Z'hhmmssxx'
18 New in DFSORT
The interpreted values can be further edited using edit masks or edit patterns, or converted to BI, FI, PD, PDF,
PDC, ZD, ZDF, ZDC or FS/CSF values. This makes it easy to show SMF, TOD and ETOD date and time
values in meaningful ways.
The following example shows how TOD date and time values can be converted to readable form:
* Display a TOD date as C'yyyy/mm/dd' and a
* TOD time as 'hh:mm:[Link]'.
OUTREC BUILD=(X,26,8,DC1,EDIT=(TTTT/TT/TT),X,
26,8,TC4,EDIT=(TT:TT:[Link]))
The SORTOUT output might look as follows:
2005/02/09 10:27:04
2005/02/10 06:13:21
2005/03/05 12:07:33
2005/03/22 06:43:08
INREC, OUTREC and OUTFIL can combine numeric fields (p,m,f), decimal constants (+n and -n), operators
(MIN, MAX, MUL, DIV, MOD, ADD, SUB) and parentheses to form arithmetic expressions. The results can
be further edited using editing masks or editing patterns, or converted to BI, FI, PD, PDF, PDC, ZD, ZDF,
ZDC or FS/CSF values.
Here's an example with an arithmetic expression:
OUTFIL FNAMES=OUT,
BUILD=(5:C'% REDUCTION FOR ',21,8,C' IS ',
((11,6,ZD,SUB,31,6,ZD),MUL,+1000),DIV,11,6,ZD,
EDIT=(SIIT.T),SIGNS=(+,-))
INREC, OUTREC and OUTFIL can generate sequence numbers in output records. You can create BI, PD, ZD
or CSF/FS sequence numbers. Starting values and increment values can be defaulted or specified. You can
restart the sequence number at the starting value whenever a value in a specified field changes. As a simple
example, if you specify:
INREC OVERLAY=(81:SEQNUM,8,ZD,START=1000,INCR=100,RESTART=(25,5))
zoned decimal sequence numbers 00001000, 00001100, 00001200, and so on will be generated in positions
81-88 of the output records. Each time the value in positions 25-29 changes, the sequence number will start
again at 00001000. Note that an F-sign is used for zoned decimal sequence numbers so they will be displayable
and printable.
By default, DFSORT uses the OUTREC or INREC record length as the SORTOUT LRECL when the
SORTOUT LRECL is unavailable, instead of using the SORTIN length. This can eliminate
unexpected/unwanted padding and truncation. For example, previous to this change, if you had a fixed-length
input data set with an LRECL of 80 and specified:
OUTREC BUILD=(1,50)
you'd get a SORTOUT LRECL of 80 and output records padded with binary zeros from positions 51-80. Now,
instead by default, you'll get a SORTOUT LRECL of 50 and no padding.
The SOLRF installation and run-time options allow you to specify whether you want the SORTOUT LRECL
to be set the new way or old way when INREC or OUTREC is specified, as follows:
– SOLRF=YES (installation) or SOLRF (run-time) tells DFSORT to use the INREC or OUTREC length as
the SORTOUT LRECL. This is the IBM-supplied default and is usually what you want when you specify
INREC or OUTREC. This is also the way OUTFIL OUTREC always works.
– SOLRF=NO (installation) or NOSOLRF (run-time) tells DFSORT not to use the INREC or OUTREC
length as the SORTOUT LRECL. This is the way DFSORT used to work and may cause
unexpected/unwanted padding or truncation of the SORTOUT records.
20 New in DFSORT
OUTFIL FNAMES=(PIPE1,PIPE2,PIPE3),SPLIT
OUTFIL FNAMES=(OUT1,OUT2,OUT3),SPLITBY=50
The first record is written to the PIPE1 pipe, the second record is written to the PIPE2 pipe, the third record is
written to the PIPE3 pipe, the fourth record is written to the PIPE1 pipe, and so on.
Records 1-50 are written to the OUT1 data set, records 51-100 are written to the OUT2 data set, records
101-150 are written to the OUT3 data set, records 151-200 are written to the OUT1 data set, and so on.
Signed hexadecimal floating point values can be converted into their corresponding signed integer values for
calculating total, minimum, maximum and average statistics with OUTFIL's TRAILERx parameters. The
integer values can be further edited using edit masks or edit patterns, or further converted to BI, FI, PD, PDF,
PDC, ZD, ZDF, ZDC or FS/CSF values. This makes it easy to display statistics for hexadecimal floating point
values.
Here's an example of displaying a total and average for a hexadecimal floating point value:
OUTFIL TRAILER1=(C'Total:',10:TOT=(51,8,FL,M12),/,
C'Average:',10:AVG=(51,8,FL,M12))
The BLKCCH1, BLKCCH2 and BLKCCT1 parameters of OUTFIL can be used to suppress page ejects for
reports in various ways.
BLKCCH1 allows you to avoid forcing a page eject at the start of the report header; the ANSI carriage control
character of '1' (page eject) in the first line of the report header (HEADER1) is replaced with a blank.
BLKCCH2 allows you to avoid forcing a page eject at the start of the first page header; the ANSI carriage
control character of '1' (page eject) in the first line of the first page header (HEADER2) is replaced with a
blank.
BLKCCT1 allows you to avoid forcing a page eject at the start of the report trailer; the ANSI carriage control
character of '1' (page eject) in the first line of the report trailer (TRAILER1) is replaced with a blank.
For example, with:
OUTFIL BLKCCH2,
HEADER1=('Control report',2X,DATE,2/),
HEADER2=('Page ',PAGE=(EDIT=(IIT)),/,
'Account',12:'Revenue')
DFSORT does a page eject at the start of the report header, and at the start of the second and subsequent page
headers, but suppresses the page eject at the start of the first page header. Thus, the report header and first
page header appear on the same page.
OUTFIL can produce reports without the ANSI carriage control characters that indicate actions to be taken on a
printer (for example, page eject, skip a line, and so on). If the REMOVECC operand is specified, DFSORT
"removes" the carriage control character from each record of the report. This makes it easy to remove the
printer controls when they're not needed because the output will be viewed or written to a list data set, rather
than printed. When REMOVECC is specified, the RECFM does not need to include 'A' for ANSI (for
example, it can be FB instead of FBA) and the LRECL does not need to include an extra byte for the carriage
control character.
As a simple example, if you specify the following for an input data set with 5723 records:
OUTFIL FNAMES=TOTCNT,NODETAIL,
TRAILER1=(COUNT=(M11,LENGTH=6))
TOTCNT would contain:
1005723
where '1' is the carriage control character for a page eject and '005723' is the count of total records. If you
were passing the count to another program that expected it to start in column 1, you could specify:
22 New in DFSORT
blanks
blanks
Field 2 contains BBB
Field 1 contains AAA
blanks
blanks
Field 2 contains DDD
Field 1 contains CCC
Note that four output records are produced for each input record.
The SAMPLE=n and SAMPLE=(n,m) parameters of OUTFIL can be used to sample records in a variety of
ways. The sample consists of the first m records in every nth interval. STARTREC=x and ENDREC=y can be
used with SAMPLE to select a range of records to be sampled. SAMPLE=n writes every nth record starting at
the STARTREC record and ending at or before the ENDREC record. SAMPLE=(n,m) writes m records every
nth record starting at the STARTREC record and ending at or before the ENDREC record.
Here's an example of OUTFIL sampling:
OUTFIL FNAMES=OUT1,SAMPLE=5
OUTFIL FNAMES=OUT2,SAMPLE=(1000,2),ENDREC=2500
OUTFIL FNAMES=OUT3,STARTREC=23,ENDREC=75,SAMPLE=25
OUTFIL FNAMES=OUT4,STARTREC=1001,SAMPLE=(100,3)
The input records written to each output data set are as follows (remember that the default for STARTREC is 1
and the default for ENDREC is the last record in the data set):
– OUT1: 1, 6, 11, and so on
– OUT2: 1, 2, 1001, 1002, 2001, 2002
– OUT3: 23, 48, 73
– OUT4: 1001, 1002, 1003, 1101, 1102, 1103, and so on
The REPEAT=n parameter of OUTFIL lets you write each output record multiple times. The repeated records
are identical, unless the SEQNUM operand is used to create different sequence numbers for the repeated
records.
Here's an example of OUTFIL repetition:
OUTFIL FNAMES=RPT50,REPEAT=5000
OUTFIL FNAMES=RPTSQ,REPEAT=20,OVERLAY=(81:SEQNUM,8,ZD)
5000 identical copies of each input record are written to the RPT50 output data set.
20 reformatted output records are written to the RPTSQ output data set for each input record. The reformatted
output records consist of input positions 1-80 and an 8-byte ZD sequence number that starts at 1 and is incre-
mented by 1 for each output record, including the repeated records.
OUTFIL can convert fixed-length input records (for example, FB) to variable-length output records (for
example, VB). If the new FTOV operand is specified without OUTREC or BUILD, the entire fixed-length
record is used to build the variable-length record. If FTOV is used with OUTREC or BUILD, the specified
fields from the fixed-length record are used to build the variable-length record. This makes it easy to use all of
DFSORT's features when converting from FB to VB.
For example, FTOV might be used as follows:
OUTFIL FNAMES=VAROUT,FTOV
OUTFIL FNAMES=V1,FTOV,BUILD=(1,20,26:21,10,6C'*')
For ease-of-use, VTOF can be used as an alias for CONVERT. Thus, FTOV can be used to convert from FB
to VB, and VTOF (or CONVERT) can be used to convert from VB to FB.
24 New in DFSORT
– Character (FS format): Test for '0'-'9' in all bytes. For example, if you used the following INCLUDE
statement:
INCLUDE COND=(21,5,FS,EQ,NUM)
a record with a value of '02579' in positions 21-25 would be included, whereas records with values of
'02A79', '-2579' and '0257 ' would not be included.
– Zoned decimal (ZD format): Test for X'F0'-X'F9' for all bytes except the last, and X'F0'-X'F9',
X'D0'-X'D9' or X'C0'-X'C9' in the last (sign) byte. For example, if you used the following OMIT state-
ment:
OMIT COND=(31,3,ZD,NE,NUM)
a record with a value of X'F1F3D8' in positions 31-33 would not be omitted, whereas records with values
of X'F1F3A8' and X'404040' would be omitted.
– Packed decimal (PD format): Test for 0-9 for all digits and F, D or C for the sign. For example, if you
used the following INCLUDE statement:
INCLUDE COND=(11,2,PD,EQ,NUM)
a record with a value of X'832C' in positions 11-12 would be included, whereas records with values of
X'3A2F' and X'4290' would not be included.
This makes it easy to include or omit records based on whether they contain valid numeric data.
INCLUDE and OMIT can compare appropriate fields against generated run-time constants for current, past and
future dates in the following forms (d is days, m is months and c is any character except a blank):
Operand Constant
DATE1, DATE1-d, DATE1+d C'yyyymmdd'
DATE1(c), DATE1(c)-d, DATE1(c)+d C'yyyycmmcdd'
DATE1P, DATE1P-d, DATE1P+d +yyyymmdd
DATE2, DATE2-m, DATE2+m C'yyyymm'
DATE2(c), DATE2(c)-m, DATE2(c)+m C'yyyycmm'
DATE2P, DATE2P-m, DATE2P+m +yyyymm
DATE3, DATE3-d, DATE3+d C'yyyyddd'
DATE3(c), DATE3(c)-d, DATE3(c)+d C'yyyycddd'
DATE3P, DATE3P-d, DATE3P+d +yyyyddd
DATE4 C'[Link]'
Y'DATE1', Y'DATE1'-d, Y'DATE1'+d Y'yymmdd'
Y'DATE2', Y'DATE2'-m, Y'DATE2'+m Y'yymm'
Y'DATE3', Y'DATE3'-d, Y'DATE3'+d Y'yyddd'
This makes it easy to include or omit records based on whether they contain dates equal to, lower than or
higher than the date of the run, a date before the run or a date after the run.
DATEn and DATEn(c) generate a character string (C'string') for today's date that can be used in comparisons
just like any other character string. Likewise, DATEn-r and DATEn(c)-r generate a character string for a past
date and DATEn+r and DATEn(c)+r generate a character string for a future date.
26 New in DFSORT
2. If NOVLSCMP and VLSHRT are in effect, DFSORT treats the entire INCLUDE/OMIT logical expression
as false if any field is short. Thus, comparisons involving non-short fields are ignored if any comparison
involves a short field.
3. If NOVLSCMP and NOVLSHRT are in effect, DFSORT issues an error message, terminates and gives a
return code of 16 if a short INCLUDE/OMIT field is found. VLSCMP=NO and VLSHRT=NO are the
IBM-supplied installation defaults.
To illustrate how this works, suppose you specify:
INCLUDE COND=(6,1,CH,EQ,C'1',OR,70,2,CH,EQ,C'T1')
If a variable-length input record has a length less than 71 bytes, the field at bytes 70-71 is short.
– If you specify:
OPTION VLSCMP
the record is included if byte 6 of the input record is C'1' or omitted if byte 6 is not C'1'. The comparison
of bytes 70-71 equal to C'T1' is false because bytes 70-71 contain either X'hh00' (for a record length of 70
bytes) or X'0000' (for a record length of less than 70 bytes). The comparison involving the non-short field
is performed even though a short field is present.
– If you specify:
OPTION NOVLSCMP,VLSHRT
the record is omitted because any short field makes the entire logical expression false. The comparison
involving the non-short field is not performed because a short field is present.
– If you specify:
OPTION NOVLSCMP,NOVLSHRT
DFSORT terminates because any short field results in termination.
DFSORT symbols turn DFSORT's syntax into a high level language. Symbols can help to standardize your
DFSORT applications and increase your productivity. You can use a symbol anywhere you can use a field or
constant in any DFSORT control statement or ICETOOL operator. DFSORT symbols can be up to 50 characters,
are case-sensitive and can include underscore (_) and hyphen (-) characters. Thus, you can create meaningful,
descriptive names for your symbols, such as Price_of_Item (or Price-of-Item) making them easy to remember, use
and understand.
Field symbols define a field in terms of its position, length and format. Constant symbols define a field in terms of
its literal, numeric or bit value. Once you make a symbol available, you free yourself from the sometimes tedious
process of figuring out its position, length, format or value. No more confusion over offsets versus positions and
whether to add 4 for the RDW or not. No more recoding positions in statements when you rearrange fields in your
data.
To give you a quick idea of how easy it is to use DFSORT symbols, let's look at the JCL and control statements
for a simple DFSORT job tha uses symbols for fields and constants.
Here's the JCL and control statements for a DFSORT job that uses the symbols in [Link]:
//EXAMP JOB ...
//RUNIT EXEC PGM=ICEMAN
//** SYMNAMES POINTS TO ONE OR MORE SYMBOL DATA SETS.
//SYMNAMES DD DSN=[Link],DISP=SHR
//** SYMNOUT LISTS THE ORIGINAL SYMBOL STATEMENTS AND
//** THE SYMBOL TABLE DFSORT BUILDS FROM THEM.
//SYSOUT DD SYSOUT=*
//SORTIN DD DSN=ACCOUNT,DISP=SHR
//SORTOUT DD ...
//SYSIN DD *
INCLUDE COND=((Type,EQ,Loan,AND,Balance,GT,Level1),OR,
(Type,EQ,Check,AND,Balance,LE,Penalty))
SORT FIELDS=(Full_Name,A,Type,A,Account_Number,D)
/*
The following example shows how symbols can be used for output columns.
SYMNAMES:
Amt,8,5,ZD
Dept,27,5
New_col,52
Control statements:
28 New in DFSORT
OPTION COPY
INREC OVERLAY=(New_col:Amt,MUL,+2,TO=FS,LENGTH=8,
Dept:Dept,JFY=(SHIFT=RIGHT),
Amt:Amt,TO=FS,LENGTH=8)
SYMNAMES:
Name,%00
Amt,%01
Dept,%02
Name_col,5
Amt_col,21
Dept_col,41
Control statements:
OPTION COPY
OUTREC PARSE=(Name=(ENDBEFR=C',',FIXLEN=10),
%=(ENDBEFR=C','),
Amt=(ENDBEFR=C',',FIXLEN=8),
Dept=(ENDBEFR=C',',FIXLEN=12)),
BUILD=(Name_col:Name,
Amt_col:Amt,SFF,TO=FS,LENGTH=10,
Dept_col:Dept,TRAN=UTOL)
System Symbols
You can use system symbols in character constants defined as symbols. symbol,s'string' or symbol,S'string' can be
used to define a string containing any combination of EBCDIC characters and system symbols you want to use to
form a character string. You can use dynamic system symbols (e.g. &JOBNAME and &DAY), system-defined
static symbols (e.g. &SYSNAME and &SYSPLEX) and static system symbols defined by your installation.
The following example shows how system symbols can be used in symbol constants:
SYMNAMES:
Title,S'Jobname: &JOBNAME., Sysplex: &SYSPLEX., System: &SYSNAME.,'
Totday,S'Total for &WDAY.:'
Control statements:
OPTION COPY
OUTFIL REMOVECC,
HEADER2=(Title,X,' Page: ',PAGE=(EDIT=(TT)),/),
TRAILER1=(/,Totday,20:TOT=(21,4,FS,TO=FS,LENGTH=5))
The following example shows how SET symbols XDSN and YDSN can be used in a DFSORT OMIT statement:
// SET XDSN='[Link]',YDSN='[Link]'
//S1 EXEC PGM=SORT,PARM='MSGDDN=MYOUT,JP1"&XDSN",JP2"&YDSN",LIST'
//MYOUT DD SYSOUT=*
...
//SYSIN DD *
OPTION COPY
OMIT COND=(1,44,CH,EQ,JP1,OR,1,44,CH,EQ,JP2)
/*
30 New in DFSORT
The number of fields you can use with the frequently used SUM statement has been increased significantly.
This allows you to increase the number of fields you total for a sort or merge application.
You have new ways to handle an overflow condition for SUM fields. The new OVFLO installation and
run-time option allows you to specify what you want DFSORT to do when BI, FI, PD or ZD sum fields
overflow, as follows:
– OVFLO=RC0 tells DFSORT to issue an informational message, set a return code of 0 and continue proc-
essing when sum fields overflow. The pair of records involved in the overflow is left unsummed.
– OVFLO=RC4 tells DFSORT to issue an informational message, set a return code of 4 and continue proc-
essing when sum fields overflow. The pair of records involved in the overflow is left unsummed.
– OVFLO=RC16 tells DFSORT to issue an error message, terminate and give a return code of 16 if sum
fields overflow.
For example, if you wanted to display the maximum, minimum, average and total of a 25-byte ZD field and an
18-byte PD field in your input records, you could use this ICETOOL operator:
STATS FROM(IN) ON(5,25,ZD) ON(41,18,PD)
DFSORT can use the information passed to it from DFSMSrmm or ICETPEX, when appropriate, to improve its
processing of managed tapes in the following ways:
DFSORT can obtain accurate input filesize information for managed tapes. This can result in improved sort
performance and more accurate dynamic workspace allocation.
Additionally, you don't have to supply the input filesize to DFSORT when this information is available from
DFSMSrmm or ICETPEX. DFSORT will automatically use the filesize it obtains from DFSMSrmm or
ICETPEX to override any FILSZ=En or SIZE=En value you specify. However, you must remove any
FILSZ=n, FILSZ=Un, SIZE=n or SIZE=Un value you specify in order for DFSORT to use the filesize it
obtains from DFSMSrmm or ICETPEX.
DFSORT can obtain input and output attributes such as RECFM, LRECL and BLKSIZE for managed tapes.
As a result, you don't have to specify these attributes explicitly for input and output tape data sets when this
information is available from DFSMSrmm or ICETPEX.
32 New in DFSORT
64-Bit Architecture
DFSORT can exploit 64-bit real architecture by using memory objects for sort applications, when appropriate.
DFSORT can exploit 64-bit real architecture by backing storage and data spaces in real storage above 2 gigabytes,
and by using central storage instead of expanded storage for Hipersorting.
Multiple Hiperspaces
DFSORT can use multiple Hiperspaces for external storage requirements, increasing DFSORT's ability to use
Hipersorting for large sort applications when sufficient system resources are available.
Changing an installation option (via an ICEPRMxx member in PARMLIB) changes the way DFSORT works
globally by default. Specifying a run-time option changes the way DFSORT works for a specific application.
VSAM Processing
DFSORT gives you new ways to process VSAM data sets as follows:
The new RESET=YES installation option and RESET run-time option tells DFSORT to process a VSAM
output data set defined with REUSE as a NEW data set. The high-used RBA is reset to zero and the output
data set is effectively treated as an initially empty cluster. RESET=YES is the IBM-supplied installation
default.
The new RESET=NO installation option and NORESET run-time option tells DFSORT to process a VSAM
output data set defined with REUSE as a MOD data set. The high-used RBA is not reset and the output data
set is effectively treated as an initially non-empty cluster.
34 New in DFSORT
The new VSAMEMT=YES installation option and VSAMEMT run-time option tells DFSORT to accept an
empty VSAM input data set and process it as having zero records. VSAMEMT=YES is the IBM-supplied
installation default.
The new VSAMEMT=NO installation option and NVSAMEMT run-time option tells DFSORT to issue an
error message, terminate and give a return code of 16 if an empty VSAM data set is found.
The new VSAMIO=YES installation option and VSAMIO run-time option tells DFSORT to allow a sort appli-
cation to use the same VSAM data set for input and output, provided that RESET is in effect and the VSAM
data set was defined with REUSE. The VSAM data set is processed as NEW for output and will contain the
sorted input records, that is, it will be sorted in-place.
The new VSAMIO=NO installation option and NOVSAMIO run-time option tells DFSORT to issue an error
message, terminate and give a return code of 16 if the same VSAM data set is specified for input and output.
VSAMIO=NO is the IBM-supplied installation default.
DFSORT supports the VSAM extended addressability function for extended format VSAM data sets, which
provides the capability of VSAM data sets larger than four gigabytes.
The installation SDB option has been expanded to allow selection of system-determined optimum block sizes
greater than 32760 bytes for output tape data sets. SDB can also be used as a run-time option. If you want to use
system-determined block sizes for DASD and tape output data sets, specify one of the following values:
SDB=LARGE if you want DFSORT to select tape output block sizes greater than 32760 bytes.
SDB=YES or SDB=SMALL if you want DFSORT to select tape output block sizes up to 32760 bytes.
SDB=INPUT if you want DFSORT to select tape output block sizes greater than 32760 bytes only if the tape
input block size is greater than 32760 bytes. SDB=INPUT is the IBM-supplied installation default.
If you don't want DFSORT to use system-determined block sizes, specify SDB=NO (not recommended).
DFSORT's ICEGENER, like IEBGENER, will use the parameters SDB=LARGE, SDB=YES, SDB=SMALL,
SDB=INPUT and SDB=NO if you specify them. Here's an example of an IEBGENER job that selects a system-
determined block size greater than 32760 bytes for a SYSUT2 tape output data set:
//EXAMP JOB ...
//SDBOUT EXEC PGM=IEBGENER,PARM='SDB=LARGE'
//SYSPRINT DD SYSOUT=*
//SYSUT1 DD DSN=[Link],DISP=SHR
//SYSUT2 DD DSN=OUTPUT3,DISP=(NEW,KEEP),UNIT=3590,VOL=SER=075836,
// LABEL=(,SL),BLKSZLIM=1G
//SYSIN DD DUMMY
This same job can use DFSORT's more efficient ICEGENER facility if your site has installed ICEGENER to be
invoked by the name IEBGENER. Alternatively, you can specify PGM=ICEGENER to ensure that ICEGENER is
used.
36 New in DFSORT
Signed/Unsigned Zero
DFSORT lets you treat numeric -0 and +0 values as signed (that is, different) or unsigned (that is, the same) for
collation, comparisons, editing, conversions, minimums and maximums.
The new SZERO=YES installation option and SZERO run-time option tells DFSORT to treat numeric zero
values as signed for INCLUDE, INREC, MERGE, OMIT, OUTFIL, OUTREC and SORT statement processing.
-0 and +0 are treated as different values, that is, -0 is treated as a negative value and +0 is treated as a positive
value. SZERO=YES is the IBM-supplied installation default.
The new SZERO=NO installation option and NOSZERO run-time option tells DFSORT to treat numeric zero
values as unsigned for INCLUDE, INREC, MERGE, OMIT, OUTFIL, OUTREC and SORT statement proc-
essing. The new UZERO option tells ICETOOL to treat numeric zero values as unsigned for DISPLAY,
OCCUR, SELECT and UNIQUE operator processing (overriding the default of treating zero values as signed
for these operators). -0 and +0 are treated as the same value, that is, -0 and +0 are both treated as positive
values.
Four time-of-day installation modules (ICETD1-4) can be used independently to activate different sets of installation
defaults on different days at different times. Each environment installation module (ICEAM1-4) can enable one or
more time-of-day installation modules.
This capability allows new levels of control for DFSORT installation defaults. For example, you could use the
ICEPRM01 member below (in PARMLIB) to set up larger DSA and TMAXLIM limits for batch program-invoked
DFSORT applications that run off-shift (6:00pm-5:59am) during the week, and all weekend:
INV
ENABLE=TD1
SVC=(,ALT)
DSA=48
TD1
WKDAYS=(1800,559)
WKEND=ALL
SVC=(,ALT)
DSA=96 TMAXLIM=8388608
By setting your installation defaults appropriately, you can fine-tune DFSORT's resource usage for special situations
at your site.
The new limit of 255 work data sets and the new ddnames increase significantly the amount of data you can sort in
a single application. Use more work data sets only when you need them for extremely large sorts.
All features can be installed together, and the mode of operation (resident or nonresident) can be chosen at IPL
time. The number of FMIDs has been reduced from 10 to 3 and the number of libraries required to install
DFSORT has been reduced from 40 to 26. These changes eliminate many decisions and speed up installation and
customization.
IBM's DFSORT and DFSMS teams have simplified the process of replacing IEBGENER with DFSORT's popular
ICEGENER facility. You only need to apply DFSMS PTF UW48193 to supply an alias of "IEBGENR" for
IEBGENER, and place ICEGENER with an alias of "IEBGENER" ahead of IEBGENER in the system's search
order for programs. This new process removes the need to monitor IEBGENER PTFs relative to ICEGENER.
38 New in DFSORT
Improvements for Copy, Merge and ICEGENER Applications
Storage above 16MB virtual can be used for copy, merge and ICEGENER applications. This provides virtual
storage constraint relief and may provide improved performance for these applications.
Copy and merge modules reside above 16MB virtual. This provides additional virtual storage constraint relief.
Option-in-effect messages are produced for copy and merge applications as well as for sort applications. This
makes it easier to determine the options used for a particular run.