Linux/UNIX Operations Guide

BASE24-eps®

Version 3.0.13, February 18, 2022

Table of Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
About this publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
What’s new in this publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Release 3.0.13 (February 2022) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Release 3.0.12 (November 2021). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Release 3.0.10 (May 2021) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Release 3.0.8 (November 2020). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1. Message flows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Diagram notations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
On-us transaction_ interchange or a device other than an ATM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Not-on-us transaction_ interchange or a device other than an ATM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
On-us transaction_ ATM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Not-on-us transaction_ ATM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Java Device Handler (JDH) configuration sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
User interface messages for logon and logoff requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
User interface messages for requests other than logon and logoff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
User interface messages for an XML passthrough request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2. General operations guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Startup and shutdown procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Application logon user IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Application startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Starting order for application processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
What to do if the SITIMRP process is stopped . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Start-up scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Startup sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Check system status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Application shutdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Shutdown sequence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
BASE24-eps licensing on Linux and UNIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Expiring license notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Update the BASE24-eps license file for Linux and UNIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Stored message extracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3. c-tree utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
env_vars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
CTACCESS utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
User operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Group operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
File operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
CTADMNX utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
CTAVAILABLE utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
 CTBLD utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
CTBLD setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Preparing the configuration files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Creating the special data file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Setting up the CTBLD run file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Running CTBLD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Running CTBLD in build mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Running CTBLD in append mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
CTBLOCK utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
CTCMDSET utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Creating a set file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
CTERRGET utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
CTFILEINFO utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
CTREFRESH utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
CTREFRESH setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Preparing the configuration file ([Link]) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Creating the special data file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
Setting up the CTREFRESH run file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Running CTREFRESH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
CTSRVRINFO utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
CTVERIFY utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Error 14 correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Running the CTVERIFY utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4. Oracle utilities and stored procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Oracle database utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
EPS_TRUNC_TBL stored procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Load IPF refresh data for an Oracle database using SQL*Loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5. PostgreSQL guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
PostgreSQL settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Server and virtual machine (VM) sizes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Red Hat Enterprise Linux (RHEL) server operating system settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
PostgreSQL settings ([Link]). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Disable autovacuum in PostgreSQL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
PostgreSQL monitoring and maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Monitor disk space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Monitor transaction ID exhaustion (Wraparound). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Monitor multiple transaction wraparound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Monitor Heap-only-tuple (HOT) updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Monitor page and table bloat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
View page cache usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
View the number of sessions open on a server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
View the top CPU and IO consuming queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
PostgreSQL VACUUM command guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6. Managing WebSphere MQ queue managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Starting and stopping a WebSphere MQ queue manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Starting a WebSphere MQ queue manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Stopping a WebSphere MQ queue manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Checking on channels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
WebSphere MQ tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Tips for MQ server binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7. JMS MQ configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Setting up JMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Prerequisite checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Create the binding file for JMS and MQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
MQ client connection details for JMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Start the MQ listener. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8. Extract processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Journal Query Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Auto restart options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
ESBLDJNL script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Start automatic extract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Next extract run time calculation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Scenario 1: Query execution configuration modified . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Scenario 2: EOPP restart_ no configuration changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Stop an automatic extract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Starting a manual extract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Manual query execution - Start option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Manual query execution - Restart option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
9. File partition processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Preparing the input file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Running the CTBuild utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
10. Partial refresh processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Processing steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Loading the prepared input into the input file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
11. Contingency processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Transaction Contingency Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
User interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Linux/UNIX configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Primary (transaction processing) system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Backup (transaction contingency) system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
User interface contingency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
User interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Environment variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Linux/UNIX configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Primary (users modified database) system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Backup (user interface contingency) system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
12. Understanding the c-tree server configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
c-tree server configuration parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Sample c-tree server configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
13. Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
ACI Worldwide HELP24 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
eSupport access. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Telephone access. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Customer responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Information requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Customer severity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Priority scheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Before you call . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Obtaining software version information from BASE24-eps JAR files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
JAR files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Opening JAR files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Obtaining software version information from BASE24-eps C++ programs/libraries . . . . . . . . . . . . . . . . . . . . . . . 114
Running jwhat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
Rebuilding BASE24-eps executables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Rebuilding the $MWORK/[Link] makefile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Troubleshooting BASE24-eps on Linux/UNIX (Solaris_ AIX_ HPUX). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Data backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Threading model of BASE24-eps applications on Linux/UNIX platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Layout of the BASE24-eps software on the platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Typical layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Env_vars file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Admf file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Metadata files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
[Link] file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
MQ files and scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
MQ directory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
Location of the c-tree database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Memory table sizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Location of the ICE-XS configuration files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Where do events get logged?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Where do core files get generated? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
AIX considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Linux/UNIX commands that can be used outside of the BASE24-eps product for troubleshooting . . . . . . . . . 125
Man command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Ps command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Top or topas command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Netstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Ndd command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Which and whence commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
 Prstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Iostat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Vmstat command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
Sar command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Df command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Taking stacks and cores of running processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Diagnostic tools outside of the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Message monitoring with MMON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
Configure ICE-XS processes for MMON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Turn off MMON User Exit tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Example .nof file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
ICE-XS user exit log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
MMON MQ queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
MMRD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
MMON database tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
MMON CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
MMON CLI flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Message layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
MQ diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
c-tree diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
NOF-XS for ICE-XS troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
Exception log in BASE24-eps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Journal Perusal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
JLFScan on your journal data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
Journal monitoring with JMON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
Default behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Running JMON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
JMON command reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
JMON example scripts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
Problem determination methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
Moving USEC/Version Checker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Miscellaneous troubleshooting tips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Version Checker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
User interface configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Accessing another application from the same desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Change the desktop title . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Removing the desktop sounds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
User security issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
User has locked their account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Password recovery if a user has locked out their account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Time zone description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
External connection configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
External connection issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
 Out of memory errors when accessing multiple UI windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Determining the SIS version of an executable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Firewall issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
14. Maintenance on the Linux/UNIX platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Database cleanup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Process control CLEANUP commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Java properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Manage MQ queue context entries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
User audit log (UALOGD/USRAULOG) cleanup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
15. Miscellaneous operational tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
ACI desktop terminal services implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Desktop section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Common section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
System property available at startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Summary (if TerminalServices is enabled) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Linux/UNIX platform-specific tuning considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Enable direct I/O on AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Security considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Overview of BASE24-eps processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Recommended security settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Linux/UNIX-related recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
IBM MQ-related recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
c-tree- and other database-related recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
External communications-related recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
Considerations for running TDAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
16. Performance and availability considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Single server - single site deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Infrastructure components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
IBM MQ configuration from a performance perspective. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Tuning parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
ICE-XS communication server process for Linux/UNIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Datasources in the BASE24-eps product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
c-tree server performance and availability considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
c-tree server availability considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
c-tree server performance considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
File partitioning and data layout considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Setup and configuration of the BASE24-eps application processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Multiple servers - single-site deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Network considerations for a multi-server deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
IBM MQ configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Server binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Client binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
c-tree server performance and availability considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
 Active/passive c-tree server deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Multiple servers - two-site deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
c-tree server performance and availability considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Active/active c-tree server deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Network considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Notices
ACI Worldwide

Offices in principal cities throughout the world.

[Link]

Americas +1 402 390 7600

Asia Pacific +65 6334 4843

Europe, Middle East, Africa +44 (0) 1923 816393

© Copyright ACI Worldwide, Inc. 2022

All information contained in this documentation, as well as the software described in it, is confidential and proprietary
to ACI Worldwide, Inc., or one of its subsidiaries, is subject to a license agreement, and may be used or copied only
in accordance with the terms of such license. Except as permitted by such license, no part of this documentation
may be reproduced, stored in a retrieval system, or transmitted in any form or by electronic, mechanical, recording,
or any other means, without the prior written permission of ACI Worldwide, Inc., or one of its subsidiaries.

ACI, ACI Worldwide, ACI Payments, Inc., ACI Pay, Speedpay and all ACI product/solution names are trademarks or
registered trademarks of ACI Worldwide, Inc., or one of its subsidiaries, in the United States, other countries or both.
Other parties' trademarks referenced are the property of their respective owners.

1
About this publication
The BASE24-eps Linux/UNIX Operations Guide contains information to assist in the daily operation of the BASE24-
eps product on the Linux/UNIX platform.

2
What’s new in this publication
Here is what’s new in this publication.

Release 3.0.13 (February 2022)

Here are the topics and changes made for release 3.0.13.

Updates the PostgreSQL settings topic for additional Red Hat Enterprise Linux (RHEL) server operating system
settings and to clarify that, in general, all the recommended PostgreSQL settings ([Link]) are already
implemented in TPAexec.

Release 3.0.12 (November 2021)

Here are the topics and changes made for release 3.0.12.

Throughout guide

Removes references to IBM System z.

PostgreSQL monitoring and maintenance

Adds a reference to the Grafana tool and additional information for monitoring and maintenance.

Adds the Message monitoring with MMON topic to support using the Message Monitor (MMON) to trace and
monitor messages that are sent from and received by the solution.

Release 3.0.10 (May 2021)

Here are the topics and changes made for release 3.0.10.

MQ files and scripts

Adds files and scripts in the MQ directory after installation.

Journal monitoring with JMON

Adds the JMON utility, which enables you to monitor/examine/interrogate transactions (rows) written to your Journal
tables in real time or retrospectively. For customers migrating from the HP NonStop platform to the Linux/UNIX
platform, this utility replaces some of the key functionality provided on the NonStop platform by the third-party utility
Monitor.

PostgreSQL settings

Clarifies that the work_mem setting value in the PostgreSQL settings ([Link]) is calculated as (25% of
RAM) / max_connections.

3
Release 3.0.8 (November 2020)
Here are the topics and changes made for release 3.0.8.

PostgreSQL monitoring and maintenance

Adds a script for setting up a cron job.

4
Section 1. Message flows
This section provides examples of transaction message flows through the BASE24-eps system in the following
processing situations.

• On-us transaction acquired from an interchange or a device other than an ATM.

• Not-on-us transaction acquired from an interchange or a device other than an ATM.
• On-us transaction acquired from an ATM.
• Not-on-us transaction acquired from an ATM.
• Java Device Handler (JDH) configuration.
• User interface messages for logon/logoff requests and UI requests.
• User interface messages for an XML passthrough request.

Diagram notations
The BASE24-eps system is made up of a few individual processes running independently on the Linux/UNIX Server.
Internally, all these entities communicate via WebSphere MQ Queues (identified as MQ’s on the following diagrams).
The different entities that participate in processing an online transaction include:

• The ICE-XS Protocol Handler handles downstream and upstream communication to the BASE24-eps Server
using the necessary protocol. Some ICE-XS instances are used for communicating with Host Security Module
(HSM) entities.
• In ATM transactions, the Java device handler (JDH) preprocesses downstream ATM transactions.
• The Integrated Server process (IS) performs the core online functions (authorization, routing, and so on.) of the
BASE24-eps application.
• The Timer process starts and stops shared timers.

On-us transaction_ interchange or a device other than an

ATM
In the diagram below, the transaction is an on-us transaction acquired from an interchange or a device other than an
ATM.

5
Steps Processing
01 An inbound transaction arrives at the ICE-XS protocol handler.
02 The ICE-XS protocol handler routes the transaction to the IS process™ queue. All
message traffic to the IS is routed to one queue that is shared among all the threads
of all the instances of the IS process.
03 One of the IS threads picks up the request and processes the transaction.
03.1 If any encryption-related activity is deemed necessary, the processing IS thread sends
a synchronous message to the queue for the ICE-XS process connected to the HSM
devices.
03.2 The ICE-XS process picks the message off the queue.
03.3 The ICE-XS process sends the message to the HSM device.
03.4 The ICE-XS process receives a response from the HSM device.
03.5 The ICE-XS process responds to the specific thread that issued the HSM request. To
be able to receive synchronous responses, each thread of each instance of the IS
process has a specific queue (apart from the shared queue from which they all receive
asynchronous messages).

6
Steps Processing
04 The IS thread returns the message to the queue for the inbound ICE-XS process that
originated the message.
05 The ICE-XS process picks up the message from its queue.
06 The ICE-XS process delivers the response to the external entity that originated the
request.

Not-on-us transaction_ interchange or a device other than

an ATM
In the diagram below, the transaction is a not-on-us transaction acquired from an interchange or a device other than
an ATM.

Steps Processing
01 An inbound transaction arrives at the ICE-XS protocol handler.

7
Steps Processing
02 The ICE-XS protocol handler routes the transaction to the IS process queue. All
message traffic to the IS is routed to one queue that is shared among all the threads
of all the instances of the IS process.
03 One of the IS threads picks up the request and processes the transaction.
03.1 If any encryption-related activity is deemed necessary, the processing IS thread sends
a synchronous message to the queue for the ICE-XS process connected to the HSM
devices.
03.2 The ICE-XS process picks the message off the queue.
03.3 The ICE-XS process sends the message to the HSM device.
03.4 The ICE-XS process receives a response from the HSM device.
03.5 The ICE-XS process responds to the specific thread that issued the HSM request. To
be able to receive synchronous responses, each thread of each instance of the IS
process has a specific queue (apart from the shared queue from which they all receive
asynchronous messages).
04 The IS thread sets a shared timer by placing a timer message in the input queue for
the Timer process.
05 The Timer process picks up the start timer message and updates its internal memory
collection.
06 The IS process determines the upstream destination for the message and delivers the
message to the queue for the respective outbound ICE-XS process.
07 The outbound ICE-XS process picks up the message from its queue.
08 The ICE-XS sends the message out for authorization in the required protocol.
09 The ICE-XS entity receives the response from the external host.
10 The ICE-XS process places the response in the shared queue for the IS process.
10.1 If a response from the external authorizing entity does not reach the IS process in
time, the shared timer process sends a message to the IS process to inform it of this
timeout event.
11 The IS process picks up the message from its shared queue.
12 The IS process sends a message to the Timer process queue to cancel the shared
timer set earlier during request processing.
13 The Timer process picks up the message and updates its internal memory collection.
14 The IS process prepares the response message and delivers it to the inbound ICE-XS
process that originated the request.
15 The ICE-XS process picks up the message from its queue.
16 The ICE-XS process delivers the response to the external entity that originated the
request.

8
On-us transaction_ ATM
In the diagram below, the transaction is an on-us transaction acquired from an ATM.

Steps Processing
01 An inbound transaction arrives at the ICE-XS protocol handler.
02 The ICE-XS protocol handler routes the transaction to the JDH process input queue.
03 The JDH process picks up the request message from its input queue and begins
processing.
04 After initial processing, transactions are routed to the IS process queue. All message
traffic to the IS is routed to one queue that is shared among all the threads of all the
instances of the IS process.
05 One of the IS threads picks up the request and processes the transaction.

9
Steps Processing
05.1 If any encryption-related activity is deemed necessary, the processing IS thread sends
a synchronous message to the queue for the ICE-XS process connected to the HSM
devices.
05.2 The ICE-XS process picks the message off the queue.
05.3 The ICE-XS process sends the message to the HSM device.
05.4 The ICE-XS process receives a response from the HSM device.
05.5 The ICE-XS process responds to the specific thread that issued the HSM request. To
be able to receive synchronous responses, each thread of each instance of the IS
process has a specific queue (apart from the shared queue from which they all receive
asynchronous messages).
06 The IS thread performs necessary validations and then returns the response message
to the reply queue for the JDH process that originated the message in step 4.
07 The JDH process pulls the message off its reply queue.
08 The JDH process returns the message to the queue for the ICE-XS process where the
message originated.
09 The ICE-XS process picks up the message from its queue.
10 The ICE-XS process delivers the response to the external entity that originated the
request.

Not-on-us transaction_ ATM

In the diagram below, the transaction is a not-on-us transaction acquired from an ATM.

10
Steps Processing
01 An inbound transaction arrives at the ICE-XS protocol handler.
02 The ICE-XS protocol handler routes the transaction to the JDH process input queue.
03 The JDH process picks up the request message from its input queue and begins
processing.
04 After initial processing, transactions are routed to the IS process queue. All message
traffic to the IS is routed to one queue that is shared among all the threads of all the
instances of the IS process.
05 One of the IS threads picks up the request and processes the transaction.
05.1 If any encryption-related activity is deemed necessary, the processing IS thread sends
a synchronous message to the queue for the ICE-XS process connected to the HSM
devices.
05.2 The ICE-XS process picks the message off the queue.

11
Steps Processing
05.3 The ICE-XS process sends the message to the HSM device.
05.4 The ICE-XS process receives a response from the HSM device.
05.5 The ICE-XS process responds to the specific thread that issued the HSM request. To
be able to receive synchronous responses, each thread of each instance of the IS
process has a specific queue (apart from the shared queue from which they all receive
asynchronous messages).
06 The IS thread sets a shared timer by placing a timer message in the input queue for
the Timer process.
07 The Timer process picks up the start timer message and updates its internal memory
collection.
08 The IS process determines the upstream destination for the message and delivers the
message to the queue for the respective ICE-XS process.
09 The ICE-XS process picks up the message from its queue.
10 The ICE-XS process sends the message out for authorization in the required protocol.
11 The ICE-XS process receives the response from the external host.
12 The ICE-XS process places the response in the shared queue for the IS process.
12.1 If a response from the external authorizing entity does not reach the IS process in
time, the shared timer process sends a message to the IS process to inform it of this
timeout event.
13 The IS process picks up the message from its shared queue.
14 The IS process sends a message to the Timer process queue to cancel the shared
timer set earlier during request processing.
15 The Timer process picks up the message and updates its internal memory collection.
16 The IS process prepares the response message and delivers it to the reply queue for
the JDH process that originated the message in step 4.
17 The JDH process pulls this message off its reply queue.
18 The JDH process returns the message to the queue for the ICE-XS process that
originated the message.
19 The ICE-XS process picks up the message from its queue.
20 The ICE-XS process delivers the response to the external entity that originated the
request.

Java Device Handler (JDH) configuration sample

The following diagram represents a sample configuration of the Java Device Handler (JDH) on a Linux/UNIX system.

12
Inter-process communication between the Java Device Handler (JDH) and other processes such as IS, JTIMER,
and UI is handled by JMS-MQ and TCP/IP. ICE-XS provides the interface between TCP/IP and the JMS-MQ.

• Transaction input from all TCP-connected Diebold ATMs is normally configured to be routed through ICE-XS,
then through the TCSRQST and TCSRPLY queues. All TCP-connected Diebold ATMs therefore utilize a single
listening port and listener.
• Similarly, all TCP-connected NCR ATMs are configured to use their own separate, common port and listener,
routed via the NDCRQST and NDCRPLY queues.
• The BASE24-eps integrated server (IS) process provides authorization and routing services. TSEC is the
security component within IS that provides an interface between the JDH and the security module (not shown in
diagram).
• The BASE24-eps end of period processor (EOPP) is a separate process that sends messages to the JDH to
initiate cutover processing at the required times each day.
• The Java Timer (JTIMER) process is the DeSAF process that monitors the reversal SAF and cutover SAF and
sends stored messages to the IS process.
• The TCSDCT, NDCDCT, CHNADM and FWKADM are listener ports where the JDH receives ATM operational
control commands, which originate from the ACI desktop.

13
User interface messages for logon and logoff requests
The diagram below illustrates user interface messages for logon and logoff requests.

Steps Processing
1 The ACI desktop sends the request over HTTP/XML to the WebSphere Application
Server (WAS).
2 The UI Java Server receives the request and routes it to the User Security (USEC)
process using Java Remote Method Invocation (RMI) over the configured [Link]
and [Link] defined in the [Link] file. The UI Java Server is instructing
the USEC process to update the BASE24-eps c-tree database.
3 The USEC process updates the BASE24-eps c-tree database with the user’s logon or
logoff information, then sends the response to the UI Java Server process using RMI.
4 The UI Java Server places the reply in an HTTP message and sends it to the ACI
desktop.

14
User interface messages for requests other than logon and
logoff
The diagram below illustrates user interface messages for requests for units of work other than logon and logoff.

Steps Processing
1 The ACI desktop sends the request over HTTP/XML to the WebSphere Application
Server (WAS).
2 The UI Java Server receives the request and routes it to the User Security process
using Java Remote Method Invocation (RMI) over the configured [Link] and
[Link] defined in the [Link] file. The UI Java Server is requesting the
USEC process to determine whether the user has appropriate security to perform the
requested unit of work.
3 The USEC process checks the user’s security permissions to determine if the user is
enabled to perform the requested function, then sends the response to the UI Java
Server process using RMI.
4 The UI Java Server performs the requested unit of work against the 
c-tree
database, formats an XML response, and replies to the ACI desktop.

15
User interface messages for an XML passthrough request
The diagram below illustrates user interface messages for an XML passthrough request.

Steps Processing
1 The ACI desktop sends the request over HTTP/XML to the WebSphere Application
Server (WAS).
2 The UI Java Server receives the request and routes it to the User Security process
using Java Remote Method Invocation (RMI) over the configured [Link] and
[Link] defined in the [Link] file. The UI Java Server is requesting the
USEC process to determine whether the user has appropriate security to perform the
requested unit of work.
3 The USEC process checks the user’s security permissions to determine if the user is
enabled to perform the requested function, then sends the response to the UI Java
Server process using RMI.
4 The UI Java Server sends the XML request to the configured External Connection
(extrcnct) host:port. The ICE-XS process receives the XML request.
5 The ICE-XS process sends the XML request to the configured MQ Queue.
6 The XML Server obtains the XML request from the MQ Queue.

16
Steps Processing
7 The XML Server performs the requested unit of work against the 
c-tree database
and puts the response back on the MQ Queue.
8 The ICE-XS process removes the response from the MQ Queue.
9 The ICE-XS process sends the response to the UI Java Server.
10 The UI Java Server formats the XML response in an HTTP message and replies to
the ACI desktop.

17
Section 2. General operations guidelines
This section contains procedures for startup, shutdown, and extracting stored messages.

Startup and shutdown procedures

Startup and shutdown procedures for BASE24-eps are implemented through simple shell scripts. All relevant scripts
are designed to run using the Korn shell.

Essentially, the BASE24-eps application is a set of Linux/UNIX processes, some multi-threaded, each of which is
configured to run with its own set of command-line arguments. For ease of use, a run script is provided for each of
the BASE24-eps processes. The values for the different command-line arguments defined in each of the run scripts
are set using environment variables. These environment variables are defined in a separate file by the name,
env_vars.

All example commands mentioned in this section assume that the user is using the Korn shell.
NOTE Consult relevant Linux/UNIX documentation for the corresponding commands on other Linux/UNIX
shells.

Application logon user IDs

The user logons and IDs for Linux/UNIX servers are typically decided prior to installation in conjunction with the
system administrators at the customer sites. These typically do not change. There are also various classes of
BASE24-eps operators who access the system using the BASE24-eps desktop. These operator logons and IDs
have no relation with the logons and IDs created on the Linux/UNIX servers. Operators with sufficient privileges can
add, modify, or delete operator IDs.

There is nothing special about the Linux/UNIX user account that needs to be created for a BASE24-eps installation.
The specific username and security settings are typically worked out with the system administration personnel at the
installation site. The user needs to have the ability to access all of the file systems and directories that are required
to be able to install the BASE24-eps product.

Typically, a single user account is used to install the system. This would also be the user account used for startup
and shutdown of processes and to do any maintenance on the system.

In addition to the default user and group settings for this logon, this user needs to be a part of the mqm group. This
needs to be set up when this logon is created on the Linux/UNIX system.

NOTE This presupposes that MQ has already been installed on the system and the group mqm is defined.

Application startup
BASE24-eps application processes have to be started in a certain order to initialize properly.

Starting order for application processes

The main criteria is that the SITIMRP process must be started first. All the other processes try to start timers on
startup, so the timer process must be started first to reply to these requests. The IS, XMLS, JQRY, CMCP, and

18
SAFM processes should be started next (in no particular order).

The EOPP process should be started last. When the EOPP process is started, it queues messages to the various
processes to kick off certain activities like extracts, journal cleanup, interface initialization, and so on. These
messages are sent as process control deliver commands. If these messages are not processed in the first few
minutes of being sent, they are dropped as stale messages. It is important to have the other application processes
started before the EOPP process so they are there to catch these process control commands when the EOPP
process is started.

When the EOPP process is started, it sends out the Interface Manager INITIALIZE command, which tells all the
interfaces that log on automatically to log on, and starts all of the interface store-and-forward timers.

What to do if the SITIMRP process is stopped

BASE24-eps has a lot of processing that is triggered by timer expirations. If the SITIMRP process is stopped or
crashes for some reason, the application loses all of its timers. Many of these timers are reset when they expire so if
they never expire, none of this processing ever happens. If a system is up and running but the SITIMRP process
stops, it is crucial that you get all of the timers restarted. To do this, you should stop and restart the EOPP process.
This restarts the cutover timers, restart scheduled extracts, and reinitialize the interface records so that the extract
timers are restarted.

Start-up scripts

The first step to set up the Linux/UNIX environment for a user to administer BASE24-eps should be to source the
env_vars file. This file is located in the bin directory under the BASE24-eps (ES_HOME) directory tree. Follow the
commands below to perform this operation:

$> cd <ES_HOME>/bin
$> . ./env_vars

From this point onward, this guide uses environment variables to see certain directories and
configuration parameters. These variables and many other parameters are defined in the
NOTE
env_varsfile defined above. See that file for definitions and a short description for each variable
when required.

The complete start-up of a BASE24-eps system involves executing each of the different run scripts that have been
configured to run for that particular instance of BASE24-eps. To facilitate this process, many runscripts have been
provided in the$BINdirectory. When executed, these scripts start up each of the required BASE24-eps processes. A
sample run script is shown below (runxml):

19
 nohup $OUTPUT/[Link] \
-evtcmplto=999 \
-symname="$XML_INQNAME"1 \
-qmgrname=$QMNGR \
-inqname=$XML_INQNAME \
-numthreads=$XML_NUMTHRD \
-mdlname="$XML_INQNAME"[Link] \
-cfgpath=$DATA/ \
-mdbv=csv \
-ai=ES \
-timrqname=$TIMRQNAM \
-strt=$DATA >> $LOG/[Link] 2>$LOG/[Link] &

To execute therunxmlscript, type in the following command:

$> runxml

All the run scripts except therunisscript take no run line argument. The runis script typically takes a
numeric argument (usually 1 or higher). The number denotes the instance number of the
NOTE [Link]. For redundancy and volume needs of various customers, it might be necessary to
have more than [Link] running. This numeric argument is used to differentiate between
the various different instances of the [Link].

Startup sequence

1. Source in the env_vars.

cd <ES_HOME > /bin

. ./env_vars

2. Start MQ Manager.

start_manager.sh

3. Start c-tree Server.

ctstart

4. Start SITIMRP.

runtimrp

5. Start IS, XML, etc.

runis 1

runxml

6. Start EOPP.

runeopp

20
 7. Start ICE-XS Listener processes.

runicexs t052_gui

8. Start USEC and VersionChecker.

cd $ES_HOME/ESUI

9. Start JTIMER (if running JDH).

runjtimer

10. Start JDH/IFX (if running JDH).

runjdh 1

runifx 1

11. Start ESWeb-Java Server.

cd /app/WebSphere/AppServer/bin

./[Link] ES_T052

Check system status

The esstatus script shows you the status of your BASE24-eps system.

This script connects to the ACIJMX agent to retrieve BASE24-eps statuses from JMX. A user ID and password is
required to connect to the ACIJMX agent. The esstatus script prompts you for a user ID and password and asks if
you want to save the user ID and password as an encrypted file. If you enable the esstatus script to save them as an
encrypted file, the script creates a $ESJMX/config/[Link] file that contains the user ID and encrypted password. For
subsequent esstatus sessions, the esstatus script accesses the user ID and password from this file rather than
prompting you for every new session.

To continuously refresh, use esstatusl to loop every five seconds. Use Ctrl-c to break.

Esstatusl

Below is an example of the output from the esstatus script:

21
 *** WebSphere MQ Manager ***
[Link] is running
There is 1 runmqlsr Running
There is 1 runmqtrm Running
*** c-tree server ***
J52DSVR is running
*** USEC and VersionChecker ***
Checking USEC ...
There is/are 1 Security Servers Running
Checking VersionChecker ...
There is/are 1 Version Checkers Running
*** ES Processes for System: J52D ***
There is 1 [Link] Running
There is 1 [Link] Running
There is 1 sitimrp Running
There are 2 sievtq Running
*** JDH/IFX processes ****
There is 1 JDH Running
There is 1 JTIMER Running
*** JMX processes ****
There is 1 ACIJMX Running

Application shutdown
As always, before proceeding with any Linux/UNIX command line BASE24-eps-related activity, the user should
“source” the env_vars file as described above in the Application Startup topic.

The complete shutdown of a BASE24-eps system involves terminating each of the different BASE24-eps processes.

Shutdown sequence

1. Source in the “env_vars”.

cd <ES_HOME

. ./env_vars

2. Stop IS, XML, etc.

stopis 1

stopxml

stopeopp

3. Stop ICE-XS Listener processes.

stopicexs t052_gui

4. Stop USEC and VersionChecker.

cd $ES_HOME/ESUI

22
 ./KillES

5. Stop JTIMER (if running JDH).

stopjtimer

6. Stop JDH/IFX (if running JDH).

stopjdh 1

stopifx 1

7. Stop ESWeb-Java Server.

cd /app/WebSphere/AppServer/bin

./[Link] ES_T052

8. Stop SITIMRP.

stoptimrp

9. Stop MQ Manager.

stop_manager.sh

10. Stop c-tree Server.

ctend

BASE24-eps licensing on Linux and UNIX

The BASE24-eps license is provided in a DSXL (Digitally Signed XML) file that is supplied by ACI Worldwide
separate from the distribution media (pak/[Link] file). This licensing file authorizes BASE24-eps execution on specific
machines and operating systems in your organization until an expiration date. The license contains machine names,
operating system names, and an expiration date. BASE24-eps host processes will not start unless they are
configured with a license that has not expired more than six months ago and the license authorizes execution on that
machine and operating system.

The location of the BASE24-eps license file is specified by the LICENSE_FILE environment attribute - if no value is
configured in the Environment Configuration UI, the default value is used. The license file must contain valid license
details for the machine on which your BASE24-eps runs. If you have licensed BASE24-eps to run on multiple
machines, ACI may deliver to you a separate file for each machine, or a single license file containing licenses for
multiple machines.

The default value for the LICENSE_FILE environment attribute designating the default location for the license file is
/usr/aci/b24eps/[Link]. The use of soft symbolic link for the license file would make it easier to re-configure
BASE24-eps when updated or renewed license files are installed.

For information about updating your license key, see Update the BASE24-eps license file for Linux and UNIX .

23
Expiring license notifications
BASE24-eps issues event messages to notify you when your license is nearing expiration. Thirty days prior to a
license expiring, warning messages are issued to alert you of a pending license expiration. Error messages are
logged once a minute during the seven days before expiration and after the license has expired. You must obtain
and load a new license for your system before the license expires. Once the license expires, processes using an
expired license can run for 180 days before self-terminating. Once 180 days past the expiration date is reached,
BASE24-eps processes cannot be started, and if they are running, they will immediately terminate.

You can check the license expiration date using the EXPIRY DATE Info command.

Update the BASE24-eps license file for Linux and UNIX

Before you begin:

Before using the new license file, backup the original license as a precaution against corruption or accidentally
overwriting the file. Any changes to the file, including the addition or removal of a space character, corrupt the
license because the checksum changes, rendering the licenses digital signature invalid.

Upon receiving your new license file from ACI Worldwide, make a backup copy of it and transfer it to your system.
The license file must be transferred as an ASCII file.

When you have obtained a new license file, install the new license by performing the following steps:

1. Replace the old license file with the new one by copying the new license file to the directory and file name that is
used by your old license file.

Alternatively, you may update the LICENSE_FILE environment attribute to indicate the new license file.

If you use a symbolic link for the license file, you should just update the symbolic link to point to the new license
file

2. Use Process Control to issue a LICENSE_FILE Alter command on the Environment component to notify the
BASE24-eps processes to refresh the license file details.
◦ If the update of the license file is completed successful, a confirmation event will be logged.
◦ If the update fails, events indicating the cause of the failure are logged - in this situation revert to the
previous license file setup.
3. Use Process Control to issue an EXPIRY DATE Info command on the License Check component to check
that the expiration date has been updated to that of the new license.
4. Monitor events log for events indicating license file problems. Follow the recovery action described for the
events that are logged and, if necessary, restore the old license file, restart the processes, then contact ACI
Worldwide, Inc. to resolve the issue.

Stored message extracts

Time insensitive messages like advices, reversals and file updates are stored during processing. If the interface is
logged on and has store-and-forward timers pending, the stored messages are sent without any intervention.
However, if the interface is not logged on, interfaces have never been initialized by the INITIALIZE command (and
therefore has no store-and-forward timers pending), or if the SITIMRP is not running, the stored records are never

24
sent.

If you see messages in an interface SAF data source that you want to get sent, you should log on the interface and
send the SAFSEND <interface name> deliver command to the Interface Manager component. Sometimes a further
step is required to get the SAF messages sent. If we have been having communication problems or the queueing
layer (MQ or XPNET) had been incorrectly configured and was not getting responses back to the right queue, our
SAF counts may be wrong. If this is the case, the SAFRESET <interface name> deliver command should be sent to
the interface manager component. This causes the SAF Manager to reread the SAF files and to verify that the
counts are correct.

25
Section 3. c-tree utilities
The BASE24-eps product includes the following ACI-developed utilities for use with c-tree server. This section
describes these utilities and how to use them.

• CTACCESS
• CTADMNX
• CTAVAILABLE
• CTBLD
• CTBLOCK
• CTCMDSET
• CTERRGET
• CTFILEINFO
• CTREFRESH
• CTSRVRINFO
• CTVERIFY

The CTCMDSET utility is a Faircom utility distributed with the c-tree server. It is included here
NOTE
because of its use with several other utilities.

Location
With the exception of the CTCMDSET utility, the executables for these utilities are located in the SIS object directory.
This directory is defined as an environment variable named $SISLIB. The CTCMDSET utility is distributed by
Faircom with c-tree server.

env_vars
The env_vars file, located in the bin directory under the BASE24-eps (ES_HOME) directory tree, should be loaded
prior to using the utilities in this section. Follow the commands below to perform this operation:

$> cd <ES_HOME>/bin
$> . ./env_vars

CTACCESS utility
The CTACCESS utility is an enhanced version of the Faircom sa_admin utility and can be used by system or
database administrators to manage c-tree users, groups, and files. It can also be used to monitor users logged on to
the c-tree server and to disconnect any user from the c-tree server.

This utility is a command line version of the system administrator program (ctadmn). It enables operations to be
performed one-at-a-time interactively, or operations can be batched and run from a shell script.

26
Syntax

ctaccess -s<srvr name> -1<set file name> -l<log file name> [ <command syntax> |
i<script file name> ]

Where:

-s <srvr name>: The c-tree server name.

-1 <set file name>: The name of a set file that contains the administrator’s encrypted
user ID and password. 
 To use this argument, the user must first create a set file
using the CTCMDSET utility. See the CTCMDSET utility documentation for
information about creating a set file.

-l <log file name>: The name of the output file to which the command output is written.

<command syntax>: The required command syntax for a single operation to be

performed.
-i <script file name>: The name of a script file containing a batched set of commands
to be run.

Operations can be performed one at a time, in which case the specific <command syntax> is
entered. Or commands can be batched in a script file, in which case, the <script file name> is
NOTE
provided. The same command syntax is used in either case. Applicable command syntax is
provided below for the various user, group, and file operations supported by the CTACCESS utility.

User operations
The following CTACCESS user operations enable changes to user information. The command syntax can be
entered directly in the ctaccess command or placed in a script file.

Operations Command syntax

List user accounts -oul

Show user account -ous <userid>

information
Delete a user account -our <userid>

Change a user account -oud <userid> -d<description>

description
Add a user account to a -oug <userid> -g<group>
group
Remove a user account -oux <userid> -g<group>
from a group
Change a user account -oup <userid> -w<password>
password

27
Operations Command syntax
Change a user account -oum <userid> -m<memory> [-u<rule>]
memory limit
Change a user account -oue <userid> [-b<begdat>] [-e<enddat>] [-l<loglimit>] [-
extended settings t<mustlogon>] [-r<rsmlogon>]

Add a user account -oua <userid> [-d<description>] [-w<password>] [-g<group>] [-

m<memory>] [-u<rule>] [-b<begdat>] [-e<enddat>] [-l<loglimit>]
[-r<rsmlogon>] [-t<mustlogon>]

where:

<userid> The user ID of the user account.

-d: <description> The description assigned to the user account.

-g: <group> The group to which the user account belongs.

-w: <password> The password for the user account.

-m: <memory> The memory limit to be assigned to a group.

-u: <rule> An optional rule value associated with the memory limit. Values are A (absolute), D
(default), or G (guideline). The following is an example specifying an absolute memory
limit of 10 MB:
-m10485760 -uA

-b: <begdat> The first date (mm/dd/yyyy) on which the user account is considered valid.

-e: <enddat> The last date (mm/dd/yyyy) on which the user account is considered valid.

-l: <loglimit> The maximum number of consecutive failed logon attempts enabled - after which the
account is blocked.

-r: <rsmlogon> The number of minutes the user account is temporarily blocked, starting from the time
the command is processed. Specifying a value of block (-r block) blocks the account
indefinitely - until it is unblocked by an administrator Specifying a value of unblock (-r
unblock) unblocks the account immediately.

-t: <mustlogon> The interval, in minutes, during which the user must log on at least once, starting from
the time the command is processed. If the user does not log on at least once within the
number of minutes specified, the account is blocked.

Group operations
The following CTACCESS group operations enables changes to group information. The command syntax can be
entered directly in the ctaccess command or placed in a script file.

28
Operations Command syntax
List groups -ogl

Show group information -ogs <group>

Delete a group -ogr <group>

Change a group description -ogd <group> -d<description>

Change a group memory -ogm <group> -m<memory> [-u<rule>]
limit
Add a group -oga <group> [-d<description>] [-m<memory>] [-u<rule>]

where:

<group> The name of the group for which the action is to occur.

-d: <description> The description to be assigned to a group.

-m: <memory> The memory limit to be assigned to a group.

-u: <rule> An optional rule value associated with the memory limit. Values are A (absolute), D
(default), or G (guideline). The following is an example specifying an absolute memory
limit of 10 MB-m10485760 -uA.

File operations
The following CTACCESS file operations enable changes to file information. The command syntax can be entered
directly in the ctaccess command or placed in a script file.

Operations Command syntax

List files -ofl <filename>

Change file group -ofg <filename> <group>

Change file owner -ofo <filename> <owner>

Change file password -ofp <filename> <password>

Set file permissions -ofs <filename> <permission> <permission> …

Add/remove file permissions -ofs <filename> +/-<permission> +/-<permission>…

where:

<filename> The name of the file on which the action is to be taken.

<group> The name of the group to which the file is to be assigned.

29
 <owner> The owner to which the file is to be assigned.

<password> The new password to be assigned to the file.

<permission> The permissions, separated by spaces, to be assigned to the file. Preceding a set of
permissions with a + or - adds or removes the corresponding permissions from the file.

CTADMNX utility
The CTADMNX utility is an ACI extension to the Faircom CTADMN utility. All of the functionality available in the
Faircom version of the utility is available in the extended version provided by ACI. However, the ACI extended
version includes the following options under the Monitor Clients function:

• List attached clients - enables viewing all attached clients in order of connection.
• List by Process ID (PID) - enables viewing all client connections in process ID (PID) order. The PID is derived
from the information in the Node field, which is seen when viewing with option 1. The information contained in
the Node field is placed there by the BASE24-eps application when connecting to the c-tree Server.
• List by given Process ID (PID) - enables viewing all client connections for a specific PID. When this option is
selected, the user is prompted for the PID. All entries associated with the entered PID are displayed.
• Kill client.

The Faircom CTADMN utility supports only options 1 and 4. Otherwise, the CTADMN and
NOTE CTADMNX utilities are identical. For documentation on the remainder of the CTADMNX functions,
see the applicable Faircom documentation.

CTAVAILABLE utility
The CTAVAILABLE utility is used to determine if a c-tree server is up and available. It does this by attempting to
connect to the server and generating a message based on the response.

NOTE This utility can be used in scripts to start the c-tree server. An example is provided further below.

Syntax

ctavailable - b - s<srvr name> [-u<userid>] [-p<password>] [-1<set file name>]

where:

-b Do not display program banner information.

-s: <srvr name> The name of the c-tree server.

-u: <userid> The user ID for the c-tree server.

30
 -p: <password> The password associated with the user ID for the c-tree server.

-1: <set file name> The name of a set file that contains the encrypted user ID and password associated
with the c-tree server. If this argument is provided in the run command, the - u and - p
arguments cannot be entered. To use this argument, the user must first create a set file
using the CTCMDSET utility. See the CTCMDSET utility documentation for information
about creating a set file.

Responses:

The CTAVAILABLE command can generate the following response messages:

INFO -> c-tree server <srvr name> is available!

The server is up and available.
WARNING -> c-tree server <srvr name> is not ready for connection!
The server is down; there is no listener on the port queried.
ERROR -> c-tree server <srvr name> is not available! Error Code = <error
code>

The server is not available; an error response was received for the port queried. The error code is provided in the
response messages.

Example Script:

The following is an example of a script to start the c-tree server that uses the CTAVAILABLE utility. In this case, the
CTAVAILABLE utility generates a message once the c-tree server is available.

nohup $CTREE/erver/ctsrvr &

until $SISOBJ/ctavailable -b -s#10505 -1$CONFIG/[Link]
do
print "WARNING -> Waiting for c-tree Server to come ready!"
sleep 2
done

CTBLD utility
The CTBLD utility is used to create c-tree data and index files used by the BASE24-eps applications.

The CTBLD utility can run in two different modes:

Build Mode The utility creates empty data and index files for an application. In this mode, the utility can
process any number of data and index files in a given run. When completed, each data file
and its associated indexes are valid c-tree data files, but are left empty to be populated by
some other external application or procedure.

31
 Append Mode The utility creates empty data and index files, appends a special type of data file to the end
of the empty data file, and rebuilds the contents of the indexes from the data. In this mode,
only one file can be processed in a given execution of the utility.

CTBLD setup
Several files that must be prepared or created in advance for the CTBLD utility to run:

• Configuration files ([Link], [Link], and [Link])

• Special data file - required for append mode only
• CTBLD run file

Preparing the configuration files

To run the CTBLD utility, the user must prepare the three configuration files described in the following table. These
configuration files are generated using the Metadata Manager program (MetaMan), which places the metadata in
these files required by the CTBLD utility.

File name Description

[Link] A text file containing two types of information:

1. The specific location of each data file to be

accessed by the BASE24-eps applications.
2. Operational parameters needed by the BASE24-eps
applications.

These parameters can be used to help define the

operating environment of a specific BASE24-eps
application.

A user should never modify this file

directly. All modifications should be
made using MetaMan. The slightest
NOTE error in modifying the information in this
file causes strange behavior when a
BASE24-eps application attempts to
execute.

32
File name Description
[Link] A text file containing the metadata required to define
each table, each field within a table, and each index
associated with a table.

A user should never modify this file

directly. All modifications should be
made using MetaMan. The slightest
NOTE error in modifying the information in this
file causes strange behavior when a
BASE24-eps application attempts to
execute.

[Link] The [Link] file is specific to the c-tree environment.

MetaMan generates this file in a special format needed
to create data files and their associated indexes. This file
contains one set of definitions for each data file to be
created.

In Append Mode, only one data file can

be processed per run and only one file
can be defined in the [Link] file - the file
being created. The other two
NOTE
configuration files can contain
definitions for more than one data file
as long as one of the tables defined in
the file is the one being built.

Creating the special data file

The special data file is only required by the CTBLD utility when running in append mode. It is the file that contains
the actual data to be appended to the newly created file.

NOTE If you are running in build mode only, you can skip ahead to the next section.

The special data file must be built by some external process or procedure that formats the file and its data records
as defined by the metadata configuration files for the specific table. It can be created in one of two specific formats,
dictated by whether the new file is to contain fixed- or variable-length data records.

Creating a special data file with fixed-length records

If the new file is to contain fixed-length records, the special data file must contain all fixed-length records, all
contiguously following one other, and each with a fixed length as defined in metadata.

This concept is illustrated in the following example for a 102-byte, fixed-length data record format:

33
 Record 1 - length = 102 bytes
Record 2 - length = 102 bytes
Record 3 - length = 102 bytes
. . .
. . .
. . .
Record n - length = 102 bytes

In the fixed-length format, each record must contain the data as defined by the metadata for that table. The following
is the structure for the example 102-byte records shown above. In this case, the special data file would contain some
number of fixed-length, 102-byte records, structured as shown below, all contiguously following one another.

Table Name = DataTypes

Record max length = 102 bytes
Record definition =
Delete flag 16 bit integer
Thread ID 32 bit integer
Index 32 bit integer
SIS Data Type 32 bytes of character data
SQL Data Type 25 bytes of character data
C Data Type 20 bytes of character data
c-tree Data Type 15 bytes of character data

Creating a Special Data File with variable-length records

In the variable-length format, each data record requires a 10-byte header to define the actual length of the data
record. The format of a 10-byte header is as follows:

Header ID 2 byte binary - Contains 0xFAFA

Length 1 4 byte integer - Contains the total length of the data plus the length
of the 10-byte header.
Length 2 4 byte integer - Contains the total length of the data only.

The concept of variable-length records is illustrated in the following example. In this case, a variable-length field has
been added to the 102-byte record for comments that can contain up to 48 bytes of character data. Thus, the data
records that can vary in length from 102 bytes to 150 bytes:

10 byte header for record 1

Record 1 - length = 102 bytes
10 byte header for record 2
Record 2 - length = 120 bytes
10 byte header for record 3
Record 3 - length = 109 bytes
. . .
. . .
. . .
10 byte header for record n
Record n - length = 135 bytes

Similar to the fixed-length format, for the variable-length format, each record must contain the data as defined by the

34
metadata for that table. The following is the structure for the example variable-length records shown above.

Table Name = DataTypes

Record max length = 150 bytes
Record definition =
Delete flag 16 bit integer
Thread ID 32 bit integer
Index 32 bit integer
SIS Data Type 32 bytes of character data
SQL Data Type 25 bytes of character data
C Data Type 20 bytes of character data
c-tree Data Type 15 bytes of character data
Comments 0 to 48 bytes of character data

The header portion of each variable-length record would be as follows for the example provided above.

Header ID Length 1 Length 2

Record 1 0xFAFA 112 102
Record 2 0xFAFA 130 120
Record 3 0xFAFA 119 109
Record n 0xFAFA 145 135

Setting up the CTBLD run file

The CTBLD utility is usually executed using a run file. A sample run file, by the name of runctbld , is provided with
the BASE24-eps distribution under the $BIN directory. The syntax and structure of the run file is shown in the
example below.

Syntax

$SISOBJ/ctbld \
$CONFIG/[Link] \
[ -u<userid> -p<password> |
-1<set file name> ] \
-mdbv=csv \
-ai=$AI \
-strt=$CONFIG [ \
-append=/db01/visa/ctree/visasrvr4/testdata/rawdata ]

where:

Line 1 The location of the CTBLD utility executable.

Line 2 The location of the [Link] configuration file.

-u c-tree server’s user ID.

-p c-tree server’s password.

35
 -1 The name of a set file containing the encrypted user ID and password associated with the c-tree
server. If this argument is provided in the run command, the - u and - p arguments cannot be
entered. To use this argument, the user must first create a set file using the CTCMDSET utility.
See the CTCMDSET utility documentation for information about creating a set file.

-mdbv= Specifies whether metadata is loaded from a .csv file or a preloaded memory table. Only the first
option is currently supported. This value should always be set to “csv”.

-ai= The application ID, as defined in MetaMan, for the metadata used to generate the [Link] file.
The application ID can be obtained from the first section of the [Link] configuration file.

-strt= The location of the [Link] and [Link] configuration files.

-append= The presence of this line in the run file indicates that the CTBLD utility is to be run in append
mode. The information entered here is the location of the special data file to be appended to a
newly created c-tree data file. If CTBLD is to be run in append mode, the user must also ensure
the specified [Link] configuration file contains only a single entry for the table to be built and
appended to.

Running CTBLD
To run the CTBLD utility, you enter the following command specifying the runfile that has been created (as described
above):

Syntax

$BIN/runctbld

Running CTBLD in build mode

In build mode, the CTBLD utility reads each set of definitions specified in the [Link] file, one file at a time, and
creates internal tables needed to create the data file. Once the file information is compiled for a table, the CTBLD
program sends the information to the c-tree server, which creates the empty data file and associated indexes.

Successful completion

If the execution of the CTBLD utility completes successfully, there are two lines of information directed to STDOUT
for each table in the [Link] configuration file. The ifil processing is complete now! message in the last line indicates
that the program has completed its run. An example is shown below:

36
 ACI's Ctree Data File Create/Build Utility
Version 2.04.00
July 23, 2003
-- Initilize c-tree server: ALSRVR1@ACIO-CISSUN18
-- Created data and index files of DATA_file:
/home/mydir/visa/bcvtst/data/bcvfile1
-- Initilize c-tree server: ALSRVR1@ACIO-CISSUN18
-- Created data and index files of DATA_file: /home/ mydir
/visa/bcvtst/data/bcvfile2
-- Initilize c-tree server: ALSRVR1@ACIO-CISSUN18
-- Created data and index files of DATA_file: /home/ mydir
/visa/bcvtst/data/bcvjournal
-- ifil processing is complete now!

Correcting errors

If an error occurs while processing a specific data file, STDOUT shows the error, and CTBLD continues to the next
data file definition. CTBLD attempts to build all the data files it can and reports errors for the ones it is unable to
build.

If the build encounters an error for one or more of the data files in the configuration, the errors can be corrected and
CTBLD can be re-executed using the same run file. In the subsequent run, error number (19), file already exists ,
messages populate for those files created in the initial run. These error messages can be ignored.

Running CTBLD in append mode

In append mode, the CTBLD utility performs the build step first in the same manner as it does for the build mode for
a single file only (since only one data file can be processed per run). Then, the CTBLD utility appends the special
data file as an additional step.

The CTBLD utility generates a “Starting the APPEND processing!” message when it starts the append processing
and an END OF RUN → Data File creation is now complete! message when it completes the append processing.

Since only one data file can be processed per run, only one data file can be defined in the [Link]
file. If there are more than one data file definitions in the [Link] file, the append step does not occur.
NOTE
There is no specific error message generated for this condition. However, the Starting the APPEND
processing! message does not appear.

CTBLOCK utility
The CTBLOCK utility is used to send a block command to a c-tree server to place the server in, or remove the server
from, block mode.

In block mode, the server does not enable applications to open data files and indexes. Block mode is required
occasionally when there is a need for exclusive access to files under the control of a specific c-tree server (for
example, the CTVERIFY utility requires this type of exclusive access to files).

When placed in block mode, the server enables a certain amount of time for all current opens to be closed (if any are
encountered). After that, the server closes the files that are still open.

Syntax

37
 ctblock - srvr=<c-tree server name> [-u<uid>] [-p<password>]
[-1<set>]
-secu=<ON | OFF>

where:

-srvr=: <c-tree servername> The name of the c-tree server to receive the block command.

-u: <uid> The user ID for the c-tree server.

-p: <password> The password associated with the user ID for the c-tree server.

-1: <set> The name of a set file that contains the encrypted user ID and password
associated with the c-tree server.

If this argument is provided in the run command, the - u and - p arguments cannot be entered. To
NOTE use this argument, the user must first create a set file using the CTCMDSET utility. See the
CTCMDSET utility documentation for information about creating a set file.

-secu
This argument sets or resets the block mode for the c-tree server. If ON is specified, the specified server starts to
block any new opens of data files under its control. If OFF is specified, the c-tree server removes the block and
enables file opens by non-administrative users.

Output

Depending on whether block mode is being turned on or off, one of the following messages is sent to STDOUT:

ON "INFO → Blocked non-admin group user logons!"

OFF "INFO → Unblocked non-admin group user logons!"

CTCMDSET utility
The CTCMDSET utility is used to create a set file containing an encrypted user ID and password.

Set files can be passed to certain utilities as an alternative for entering the user ID and password in the clear.

The CTCMDSET utility is a Faircom utility distributed with the c-tree server. It is not with the other
NOTE ACI utilities in this section. Documentation is provided here because set files are used by several
other utilities.

Creating a set file

To create a set file, the user must first create a text file, named with a file extension of “.cfg” (for example,
<textfile>.cfg). The file must contain the following two lines:

38
 USERID <user ID>
PASSWD <user password>

Once the text file is created with the appropriate <user ID> and <user password>, the CTCMDSET utility can be run
using this text file as input. The command syntax is as follows:

$CTREE/utils/ctcmdset $DB/ctree/[Link]

If the utility is successful, it displays a message indicating the set file was created. The set file has the same name
as the input file except that its file extension is “.set”.

[Link]

CTERRGET utility
The CTERRGET utility is used for displaying detailed information pertaining to specific c-tree server error codes.

At times, BASE24-eps issues errors due to underlying system failures and problems being experienced by the c-tree
server.

The CTERRGET utility expands on the BASE24-eps error codes to provide additional information on the error.

Syntax

cterrget
<error number>

where:

<error number> The number associated with the error message.

Response

The utility responds with a text message providing an explanation of the error. Below is an example of the output
provided by this utility.

Error: 12 ( FNOP_ERR )
Could not open file: not there or locked. Either file does not exist, filnam
points to incorrect file name, or file is locked by another process. Get errno
which is stored in c-tree global variable sysiocod and look up in compiler
documentation or errno.h. When, in server mode, FNOP_ERR is caused by conflicting
open requests the value of sysiocod will be set to FCNF_COD (-8). For example:
your open failed because another process had the file open in exclusive mode.
When, in server mode version 6.05.26a or later, the FNOP_ERR was caused by a
device access error the sysiocod value will be set to FDEV_COD (-9).

39
CTFILEINFO utility
The CTFILEINFO utility provides the following information pertaining to data files on a c-tree server:

• Record count for a file

• List of file locks
• List of opens on a file
• Detailed file structure informationSyntax

ctfileinfo -s<srvr name> [ -u<userid> -p<password> ] (or)

[ -1<set file name> ]
[ -f<filename> ] <display mode> [ -a ]

Where:

-s: <srvr name> The c-tree server name.

-u: <userid> The administrator’s user-id.

-p: <password> The administrator’s password.

-1: <set file name> The name of a set file that contains the administrator’s encrypted user ID and
password. If this argument is provided in the run command, the - u and - p arguments
cannot be entered. To use this argument, the user must first create a set file using the
CTCMDSET utility. See the CTCMDSET utility documentation for information about
creating a set file.

-f: <filename> The file name to search for. The presence of this argument makes all other options
relative to the specified file. If no file name is included, all other options pertain to all
files opened by the c-tree server.

-a Specifies that the output is to be appended to the end of an existing output file. This
argument is used with the open file and lock dump options only. The CTFILEINFO utility
produces two disk files in the current working directory: the “lockdump” file and the
“[Link]” file. If this runtime option is selected, the utility does not delete either of
these output files (if they already exist) prior to the current run request and instead
append the information to the end of the existing output files. If this runtime option is not
selected, the utility purges and recreates the output files prior to writing the information.

<display mode> - Requires one of the following runtime arguments:

-c
Display a record count for the specified file (requires -f argument)

40
-cx
Same as -c except no header info is displayed.

-d
Display detailed file information for the specified file (requires -f argument).

-l
Display the lock dump table

-o
Display all files opened by the c-tree server.

-o1
Display only the data files opened by the c-tree server.

-o2
Display only the index files opened by the c-tree server.

CTREFRESH utility
The CTREFRESH utility refreshes large data files with data provided by the customer. Processing is similar to using
the append mode of the CTBLD utility. The difference is that CTREFRESH is a stand-alone c-tree utility and does
not require a c-tree server to be running nor rely on any running c-tree server for its processing.

• The CTREFRESH utility creates empty data and index files, appends a special type of data file to the end of the
empty data file, and rebuilds the contents of the indexes from the data.

The CTBLD utility is still functional and provides a level of security not available with the
NOTE CTREFRESH utility. The advantage of CTREFRESH is its performance as it is approximately four
times faster than CTBLD in the append mode.

CTREFRESH setup
Several files that must be prepared or created in advance for the CTREFRESH utility to run:

• Configuration file ([Link])

• Special data file
• CTREFRESH run file

Preparing the configuration file ([Link])

To run the CTREFRESH utility, the user must prepare the [Link] configuration file. This configuration file is generated
using the Metadata Manager program (MetaMan), which places the metadata in the files required by the
CTREFRESH utility.

The [Link] file is specific to the c-tree environment. MetaMan generates this file in the special format needed to
create data files and their associated indexes. This file contains one set of definitions for the data file to be
refreshed.

41
 For this utility, only one data file can be processed per run, and only one file can be defined in the
NOTE
[Link] file, the file being created.

Creating the special data file

The special data file contains the actual data to be appended to the newly created file. This file is required by the
CTREFRESH utility. It is also required by the CTBLD utility when running in append mode.

The special data file must be built by an external process or procedure that formats the file and its data records as
defined by the metadata configuration files for the specific table. It can be created in one of two specific formats
depending on whether the new file is to contain fixed- or variable-length data records.

Creating a special data file with fixed-length records

If the new file is to contain fixed-length records, the special data file must contain all fixed-length records, all
contiguously following one other, and each with a fixed length as defined in metadata.

This concept is illustrated in the following example for a 106-byte, fixed-length data record format:

Record 1 - length = 106 bytes

Record 2 - length = 106 bytes
Record 3 - length = 106 bytes
. . .
. . .
. . .
Record n - length = 106 bytes

In the fixed-length format, each record must contain the data as defined by the metadata for that table. The following
is the structure for the example 102-byte records shown above. In this case, the special data file would contain some
number of fixed-length, 102-byte records, structured as shown below, all contiguously following one another.

Table Name = DataTypes

Record max length = 106 bytes
Record definition =
Version num 32 bit integer
Delete flag 16 bit integer
Thread ID 32 bit integer
Index 32 bit integer
SIS Data Type 32 bytes of character data
SQL Data Type 25 bytes of character data
C Data Type 20 bytes of character data
c-tree Data Type 15 bytes of character data

Creating a special data file with variable-length records

In the variable-length format, each data record requires a 10-byte header to define the actual length of the data
record. The format of a 10-byte header is as follows:

42
 Header ID 2 byte binary - Contains 0xFAFA
Length 1 4 byte integer - Contains the total length of the data plus the length
of the 10-byte header.
Length 2 4 byte integer - Contains the total length of the data only.

The concept of variable-length records is illustrated in the following example. In this case, a variable-length field has
been added to the 106-byte record for comments that can contain up to 48 bytes of character data. Thus, the data
records that can vary in length from 102 bytes to 154 bytes:

10 byte header for record 1

Record 1 - length = 106 bytes
10 byte header for record 2
Record 2 - length = 124 bytes
10 byte header for record 3
Record 3 - length = 113 bytes
. . .
. . .
. . .
10 byte header for record n
Record n - length = 139 bytes

Similar to the fixed-length format, for the variable-length format, each record must contain the data as defined by the
metadata for that table. The following is the structure for the example variable-length records shown above.

Table Name = DataTypes

Record max length = 154 bytes
Record definition =
Version_num 32 bit integer
Delete flag 16 bit integer
Thread ID 32 bit integer
Index 32 bit integer
SIS Data Type 32 bytes of character data
SQL Data Type 25 bytes of character data
C Data Type 20 bytes of character data
c-tree Data Type 15 bytes of character data
Comments 0 to 48 bytes of character data

The header portion of each variable-length record would be as follows for the example provided above.

Header ID Length 1 Length 2

Record 1 0xFAFA 116 106
Record 2 0xFAFA 134 124
Record 3 0xFAFA 123 113
Record n 0xFAFA 149 139

Setting up the CTREFRESH run file

The CTREFRESH utility is usually executed using a run file. A run file, by the name of runctrefresh , can be added
under the $BIN directory. The syntax and structure of the run file is shown in the example below.

43
Syntax

$SISOBJ/ctrefresh \
$CONFIG/[Link] \
-file=/db01/visa/ctree/visasrvr4/testdata/rawdata ]

Where:

Line 1 The location of the CTREFRESH utility executable.

Line 2 The location of the [Link] configuration file.

-file= The fully qualified name of the special data file.

Running CTREFRESH
To run the CTREFRESH utility, enter the following command specifying the runfile that has been created (as
described above).

Syntax

$BIN/runctrefresh

The CTREFRESH utility performs the build step first for a single file (since only one data file can be processed per
run). Then, the utility appends the special data file as a second step.

Successful completion

If the execution of the CTREFRESH utility completes successfully, information is directed to STDOUT for the table
defined in the [Link] configuration file. An example is shown below:

ACI's Ctree Data File Refresh Utility

Version 1.00.01
August 4, 2008
INFO -> Created data and index files for data file: filea
INFO -> Appending data file!
INFO -> Rebuilding indexes!
INFO -> Data File Refreshed!

CTSRVRINFO utility
The CTSRVRINFO utility provides the following types of information pertaining to a running version of the c-tree
server:

• Statistical information about the server (simple and extended).

44
 • Lock dump information.
• It can also be used to start and stop the c-tree server monitors.

Syntax

ctsrvrinfo -s<srvr name> [ -u<userid> -p<password> ] (or) [-1<set file name>] ]

[ [-c [-v=<value>]] (or) [-f [-v=<value>]] (or) [-m [-v=<value>]] ]
[ - l [-a] ]
[-x ]

Where:

-s: <srvr name> The c-tree server name.

-u: <userid> The administrator’s user-id.

-p: <password> The administrator’s password.

-1: <set file name> The name of a set file that contains the administrator’s encrypted user
ID and password. If this argument is provided in the run command, the
- u and - p arguments cannot be entered. To use this argument, the
user must first create a set file using the CTCMDSET utility. See the
CTCMDSET utility documentation for information about creating a set
file.

-c: Toggle the Checkpoint monitor This action starts or stops the Checkpoint monitor depending on its
current state. If it is running, the action stops it; if it is not running, the
action starts it. A value of

• v=DETAIL can be used with the - c argument to expand the

amount of information generated by the Checkpoint monitor.

-f: Toggle the Function monitor This action starts or stops the Function monitor depending on its
current state. If it is running, the action stops it; if it is not running, the
action starts it. A value of

• v=<output file name> can be used with the - f argument to specify

the name of the output file the Function monitor is to create. If no
file name is specified, the Function monitor creates its output file in
the default location for the FCS files.

45
 -m: Toggle the Memory monitor This action starts or stops the Memory monitor depending on its
current state. If it is running, the action stops it; if it is not running, the
action starts it. A value of - v=<memory threshold value> can be used
with the -m argument to specify the memory threshold limit for the
Memory monitor. If this value is set, the Memory monitor generates a
message whenever the specified threshold is exceeded.

-l Display lockdump information.

-a Specifies that the lockdump output is to be appended to the end of an

existing output file. This argument is used with the lockdump option
only. The CTSRVRINFO utility produces a disk output file, called
“lockdump,” in the current working directory If this runtime option is
selected, the utility does not delete the existing lockdump output file (if
it already exists) prior to the current run request and instead append
the information to the end of the existing output file. If this runtime
option is not selected, the utility purges and recreates the lockdump
output files prior to writing the information.

-x Display the extended version of the server information.

CTVERIFY utility
The CTVERIFY utility is used to rename data files or address error 14 conditions generated by the c-tree server.

It is recommended that you run the CTVERIFY utility for any major system operational activity to ensure that the
data sources are not affected with the operational activity. You should do this whenever you see a#c-tree 407# error
on your database.

When there is a SAN disc migration, it is recommended that you copy the $B24DATA directory contents onto a new
disc, then validate the data source’s integrity by running the CTVERIFY utility before discarding the contents from
the old discs.

It is also recommended that you use the CTDUMP utility to create a backup database rather than using the#cp -p#
command to avoid corrupted files.

Error 14 correction
The c-tree server generates an error 14 whenever it opens a data file and the update flag within the data files header
is set improperly. Although very unlikely, this error can occur due to specific problems during a server restart. When
the c-tree server restarts, it uses its transaction logs to restore any files that had some action in progress. If there is
an issue with a transaction log or a transaction log is removed from the system, the restart may not be able to clean
up certain files. In this case, when an application attempts to open such a file, it receives an error 14 and the c-tree
server does not open the file.

It is highly recommended that transaction logs not be removed from the systems, and as such,
NOTE error 14 should be a rare occurrence. If you run into error 14 and you are using BASE24-eps
version 04.4 code stream and beyond, contact ACI.

46
To solve this problem, run the CTVERIFY utility after the server has been restarted and before any applications are
started.

As a general rule, customers are encouraged to have the CTVERIFY utility run as part of any startup scripts or
procedures. The program takes only a few seconds to run if there are no errors encountered. If errors are
encountered, the program corrects error 14 conditions and rebuilds the indexes associated with the data files. Note
that although CTVERIFY reports other types of errors, which can be helpful for identifying issues, it only fixes error
14 conditions.

Running the CTVERIFY utility

The c-tree server must be restarted before running the CTVERIFY utility. Additionally, there must not be any
applications started that use the database files. The CTVERIFY utility program must have exclusive access to the
data files.

Syntax

ctverify [Link]
[ -u<userid> -p<password> |
-1<set file name> ]
<-rename=[YES|Yes|yes | NO|No|no ]>

Where:

The [Link] file, generated by MetaMan, containing the file information of the files being rebuilt.

-u: <userid> The administrator’s user-id.

-p: <password> The administrator’s password.

-1: <set file name> The name of a set file that contains the administrator’s encrypted user ID and
password. If this argument is provided in the run command, the - u and - p arguments
cannot be entered. To use this argument, the user must first create a set file using the
CTCMDSET utility. See the CTCMDSET utility documentation for information about
creating a set file.

-rename= A Yes argument instructs the utility to change the location information and rebuild the
index. A No argument instructs the utility to try to rebuild the file but not change or
rename location information inside the IFIL structure.

47
Section 4. Oracle utilities and stored procedure
The BASE24-eps product uses Oracle database utilities when installing and using an Oracle database. ACI also
provides a stored procedure created during installation for truncating Oracle tables in production.

Oracle database utilities

The BASE24-eps installation program requires access to the SQL*Plus Oracle database utility. Thus, you must use
the SQL *Plus Oracle database utility if you use the BASE24-eps installation program to create the Oracle tables.
The SQL*Plus Oracle database utility, which can be run from a Linux/UNIX shell, allows database configuration
(CREATE TABLESPACE, CREATE TABLE, etc.) commands to be run from a command line. During the installation,
you have the option to defer the Oracle table creation work to your Oracle DBA using whatever tool is desired. You
can also use the SQL*Loader Oracle database utility to load data into your Oracle database using several different
supported methods.

The Oracle Call Interface (OCI) provides a native C language interface to an Oracle database. BASE24-eps
applications use OCI to connect to the Oracle database.

The SQL*Loader Oracle database utility enables you to load data into an Oracle database for large tables that need
to be migrated from BASE24-eps installations that used a C-tree or DB2 database. This utility is also used to replace
an Oracle table on the BASE24-eps system when performing a full file refresh from a host.

The Oracle 32-bit Full Client has all the files needed to run the Oracle utilities mentioned above.

EPS_TRUNC_TBL stored procedure

During installation, the ESSETUP program creates a stored procedure named EPS_TRUNC_TBL that the BASE24-
eps SIS OCI layer uses to perform truncates of tables in the Oracle database. By default it can be used to truncate
any BASE24-eps table. At a minimum, BASE24-eps applications must have the ability to truncate Journal tables
before they are reused. The stored procedure code includes a commented out example of how you could implement
the code in production to make it more restrictive. Database administrators may want to implement tighter
restrictions regarding which tables are allowed to be truncated. The EPS_TRUNC_TBL stored procedure is installed
in the $CONFIG/sis_ora_trunc.sql file.

Load IPF refresh data for an Oracle database using

SQL*Loader
Use SQL*Loader to load data from an external file into an Oracle file.

48
sqlldr cust2/cust2@//nrc3lcustdbs01vm:1521/EPS2000 control=[Link] data=[Link]
log=[Link] discard=[Link]

WHERE:
cust2/cust2@//nrc3lcustdbs01vm:1521/EPS2000 <= userid/password to access the
Oracle DB on nrc3lcustdbs01vm DB Server

[Link] - the input FIT file from Maestro

[Link] file is as follows:

load data
into table ipmdi
fields terminated by "," optionally enclosed by '"'
( stream_element )

The ipmdi Oracle table is defined as follows:

CREATE TABLE [Link]

(
stream_element CLOB
DEFAULT ''
)
TABLESPACE EPS_cust2_Dynamic;

49
Section 5. PostgreSQL guidelines
Before you configure and tune the PostgreSQL server, ensure you have a good understanding of the PostgreSQL
server concepts, page space usage, and vacuuming process.

NOTE Do not manually run the#VACUUM# command unless ACI specifically advises you to do so.

Adhere to the following guidelines to ensure success in configuring and tuning the PostgreSQL server:

• Ensure that no invasive workload is running on the PostgreSQL servers (VMs). These workloads can have a
negative impact on database server functioning.
• Set the PostgreSQL logging level to "error only." Ensure that is set up according to the local infrastructure and
security best practices.

The following topics contain PostgreSQL guidelines:

• PostgreSQL settings
• Disable autovacuum in PostgreSQL
• PostgreSQL monitoring and maintenance
• PostgreSQL VACUUM command guidelines

PostgreSQL settings
ACI labs validated the recommended PostgreSQL settings, and these settings proved to offer the best combination
of performance, database space management, and resource utilization up to high-processing volumes, that is, 5K
transactions per second.

The BASE24-eps MetaMan utility generates the DDL to create all the objects needed for BASE24-eps (such as
tables and indexes). It also contains preconfigured settings around vacuuming and page space usage. The
MetaMan utility automatically generates the following settings:

• Fillfactor
• Toast_tuple_target
• Autovacuum_vacuum_threshold

The recommended PostgreSQL settings are derived from ACI testing labs. Use them as a guide. ACI can provide
more specific settings based on sizing activity.

Server and virtual machine (VM) sizes

The following table describes the server and virtual machine (VM) sizing correlated to PostgreSQL settings.

System resource Capacity

Memory 256 GB

50
System resource Capacity
CPU 8 vCPUs (hardware threads) out of an Intel Xeon Gold 6146 CPU @ 3.20
GHx

Disk storage High-performance, low-latency SSD array (500 GB available to PostgreSQL

$PGDATA)

Red Hat Enterprise Linux (RHEL) server operating system settings

For Red Hat Enterprise Linux (RHEL) server operating system settings, use Linux kernel 3.10.0-957.el7.x86_64,
which is usually set in /etc/[Link]. Change these settings only if ACI advises you to do so or based on the
sizing effort or other measurements.

Setting Recommended value

vm.dirty_ratio 10

vm.dirty_background_ratio 5

vm.dirty_expire_centisecs 499

kernel.sched_migration_cost_ns 5000000

kernel.sched_autogroup_enabled 0

[Link] 0

vm.overcommit_memory 2

vm.dirty_background_bytes 67108864

vm.dirty_bytes 536870912

net.ipv4.tcp_keepalive_intvl 10

net.ipv4.tcp_keepalive_probes 3

net.ipv4.tcp_keepalive_time 10

PostgreSQL settings ([Link])

Retain the PostgreSQL settings ([Link]) that are not mentioned here. Where settings are not specified in
comments, do not change the recommended values unless ACI advises you to do so.

51
In general, all these settings are already implemented via TPAexec and they are presented here for reference.

Setting Recommended value Comments

listen_addresses Your host IP addresses List all addresses (NICs) where you want the
PostgreSQL server to be reached.

max_connections 1000 Can be changed based on sizing activity.

shared_buffers 48 GB Approx. 20% of RAM. Set it higher only if

ACI advises.

work_mem 64MB (25% of RAM) / max_connections

maintenance_work_ 1 GB
mem

effective_io_concurre 8 Set to value matching your OS/disk storage

ncy performance. Usually this value is good. If
needed, ACI may advise customers to use
the pgbench tool to determine the
appropriate value for a specific deployment.

max_worker_process 16
es

checkpoint_timeout 60 min

max_wal_size 100 GB It can go near 50% of RAM. Change only if

ACI advises.

min_wal_size 80 MB

checkpoint_completio 0.93
n_target

random_page_cost 1.5

effective_cache_size 200 GB Approx. 80% of RAM. Change only based on

available RAM as ACI advises during the
sizing exercise.

autovacuum on Do not change.

52
Setting Recommended value Comments
log_autovacuum_min 0
_duration

autovacuum_max_w 10
orkers

autovacuum_naptime 5 min

autovacuum_vacuum 50000
_threshold

autovacuum_analyze 25000
_threshold

autovacuum_vacuum 0
_scale_factor

autovacuum_analyze 0
_scale_factor

autovacuum_vacuum 5 ms
_cost_delay

autovacuum_vacuum 700
_cost_limit

autovacuum_freeze_ 1000000000
max_age

autovacuum_multixac 1000000000
t_freeze_max_age

Disable autovacuum in PostgreSQL

After you install PostgreSQL, modify the database (DB) tables as required. These may include journal tables and
other tables that either are not frequently updated or are not updated at all.

1. Consult the ACI project team to determine which journal tables contain unnecessary indexes, and then drop
those indexes.
2. Consult the ACI project team to identify the set of tables that are not frequently updated or not updated at all,
such as admin card and active sessions.
3. For all journal tables and for the other tables you identify, execute the following query to disable#autovacuum# :
ALTER TABLE eps_pg_schema.a0jr01 SET ( autovacuum_enabled = false,

53
 toast.autovacuum_enabled = false ); .

PostgreSQL monitoring and maintenance

To see how effectively the PostgreSQL database is working, continuously monitor its activity and analyze its
performance. You can use a variety of tools, but ACI does not recommend a specific one. Use the tools with which
you are most comfortable and that are the least invasive to the PostgreSQL server operation, such as Grafana,
Zabbix, Nagios, Munin, and others. Plain SQL queries are also an option.

Monitor disk space

One of the most important monitoring tasks, for the PostgreSQL server is to make sure that disk doesn’t become full.
Constant monitoring is key and keeping tablespaces on dedicated disks (and monitoring those) is a good practice.
OS disk quotas, if used, must be configured in such a way that they prevent disk full events.

Monitor transaction ID exhaustion (Wraparound)

For more details, see the PostgreSQL 11 at ACI - PostgreSQL XID Wraparound Maintenance Guide on Salesforce.

A transaction ID is a unique identifier that is given to each transaction. When all of the two billion available
transaction IDs have been used, the transaction IDs start over at one, which results in wraparound issues.
Transaction IDs that have wrapped around can cause severe data loss. If the age value approaches two billion, that
is, half of 2^32 space, either tune auto-vacuuming or take other measures to avoid the transaction ID wraparound,
also known as the “frozen transaction ID" problem.

To monitor the frozen transaction ID and to determine how close it is to wraparound, execute the following query:

SELECT datname, age(datfrozenxid) AS age FROM pg_database;

Monitor multiple transaction wraparound

Multiple transaction IDs, or multixact IDs, support row locking by multiple transactions. Since space is limited in a
tuple header to store lock information, that information is encoded a multixact ID whenever there is more than one
transaction concurrently locking a row. Information about which transaction IDs are included in any multixact ID is
stored separately in the pg_multixact subdirectory, and only the multixact ID appears in the xmax field in the tuple
header. Like transaction IDs, multixact IDs are implemented as a 32-bit counter and corresponding storage, all of
which requires careful aging management, storage cleanup, and wraparound handling. There is a separate storage
area that holds the list of members in each multixact, which uses a 32-bit counter, and requires you to manage it.

To see the value of the multixact ID, execute the following query:

54
 SELECT
oid::regclass::text AS table,
age(relfrozenxid) AS xid_age,
mxid_age(relminmxid) AS mxid_age,
least(
(SELECT setting::int
FROM pg_settings
WHERE name = 'autovacuum_freeze_max_age') - age(relfrozenxid),
(SELECT setting::int
FROM pg_settings
WHERE name = 'autovacuum_multixact_freeze_max_age') - mxid_age
(relminmxid)
) AS tx_before_wraparound_vacuum,
pg_size_pretty(pg_total_relation_size(oid)) AS size,
pg_stat_get_last_autovacuum_time(oid) AS last_autovacuum
FROM pg_class
WHERE relfrozenxid != 0
AND oid > 16384 AND oid::regclass::text like 'eps_pg_schema%'
ORDER BY tx_before_wraparound_vacuum;

The following query shows the next autovacuum start and details about values of transaction ID and multixact ID.
This example filters the table names with name in eps_pg_schema:

SELECT
oid::regclass::text AS table,
age(relfrozenxid) AS xid_age,
mxid_age(relminmxid) AS mxid_age,
least(
(SELECT setting::int
FROM pg_settings
WHERE name = 'autovacuum_freeze_max_age') - age(relfrozenxid),
(SELECT setting::int
FROM pg_settings
WHERE name = 'autovacuum_multixact_freeze_max_age') - mxid_age
(relminmxid)
) AS tx_before_wraparound_vacuum,
pg_size_pretty(pg_total_relation_size(oid)) AS size,
pg_stat_get_last_autovacuum_time(oid) AS last_autovacuum
FROM pg_class
WHERE relfrozenxid != 0
AND oid > 16384 AND oid::regclass::text like 'eps_pg_schema%'
ORDER BY tx_before_wraparound_vacuum;

ACI recommends that the XID and multixact-ID wraparound is monitored (and actions taken) daily. For this, the ACI
provided script ACI_AAWV.sh can be used and setup as a Linux cron job.

An example on how to set this up in cron, is:

0 2 * * * sh /path/to/script/ACI_AAWV.sh -T 50 -X -A -d epsdb101 2>&1

>/path/to/script/output/AAWV_script_`date +\%Y\%m\%d_\%H\%M\%S`.log

This example is setting up a cron job that runs the ACI_AAWV.sh script with the following characteristics:

55
 • Runs every day at 2:00 AM
• Vacuums the top 50 tables with oldest XIDs
• Runs on a database name epsdb101
• The cron job should be setup for the Postgres user, any other DB admin user or the user which is the database
owner.

Before using in production, ensure these details are customized to match the target environment.

Monitor Heap-only-tuple (HOT) updates

Heap-only-tuple (HOT) is an optimization that PostgreSQL uses to reduce the amount of I/O necessary for updates.
You can execute the following query to examine HOT updates as part of performance tuning, especially on tables
like Merchant, Usages, Positive Balance, and others that are similar. The percentage should be relatively high in
favor of HOT updates.

`SELECT schemaname, relname, n_tup_ins, n_tup_upd, n_tup_hot_upd, n_tup_del

FROM pg_stat_user_tables
WHERE schemaname like 'eps_pg_schema%' ORDER BY n_tup_upd DESC;`

Monitor page and table bloat

PostgreSQL can suffer from table bloat from dead tuples due to table update or delete activity. It can also experience
bloat from data type alignment padding. You can estimate table bloat on a heavily used PostgreSQL table, but there
is no query for which the result is 100% accurate.

The following query results in an estimated bloat percentage in a database on a table-by-table basis, and it provides
other useful information. Use this query to gauge how well autovacuum is keeping bloat under control during and
after a test, as well as how much alignment bloat there is in a newly vacuumed table.

Periodically, choose a set of tables and run this query against them. The query in this example runs against the
Usage tables (see last filter ‘usg%’). Tune it to your own needs:

WITH constants AS (
-- define some constants for sizes of things
-- for reference down the query and easy maintenance
SELECT current_setting('block_size')::numeric AS bs, 23 AS hdr, 8 AS ma ),
no_stats AS (
-- screen out table who have attributes
-- which dont have stats, such as JSON
SELECT table_schema, table_name,
n_live_tup::numeric as est_rows,
pg_table_size(relid)::numeric as table_size
FROM information_schema.columns
JOIN pg_stat_user_tables as psut
ON table_schema = [Link]
AND table_name = [Link]
LEFT OUTER JOIN pg_stats
ON table_schema = pg_stats.schemaname
AND table_name = pg_stats.tablename
AND column_name = attname
WHERE attname IS NULL

56
 AND table_schema NOT IN ('pg_catalog', 'information_schema')
GROUP BY table_schema, table_name, relid, n_live_tup ),
null_headers AS (
-- calculate null header sizes
-- omitting tables which dont have complete stats
-- and attributes which aren't visible
SELECT
hdr+1+(sum(case when null_frac <> 0 THEN 1 else 0 END)/8) as nullhdr,
SUM((1-null_frac)*avg_width) as datawidth,
MAX(null_frac) as maxfracsum,
schemaname,
tablename,
hdr, ma, bs
FROM pg_stats CROSS JOIN constants
LEFT OUTER JOIN no_stats
ON schemaname = no_stats.table_schema
AND tablename = no_stats.table_name
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
AND no_stats.table_name IS NULL
AND EXISTS ( SELECT 1
FROM information_schema.columns
WHERE schemaname = columns.table_schema
AND tablename = columns.table_name )
GROUP BY schemaname, tablename, hdr, ma, bs ),
data_headers AS (
-- estimate header and row size
SELECT
ma, bs, hdr, schemaname, tablename,
(datawidth+(hdr+ma-(case when hdr%ma=0 THEN ma ELSE hdr%ma END))):
:numeric AS datahdr,
(maxfracsum*(nullhdr+ma-(case when nullhdr%ma=0 THEN ma ELSE nullhdr%ma
END))) AS nullhdr2
FROM null_headers ),
table_estimates AS (
-- make estimates of how large the table should be
-- based on row and page size
SELECT schemaname, tablename, bs,
reltuples::numeric as est_rows, relpages * bs as table_bytes,
case when exists (select 1 from (select pg_options_to_table(pg_class
.reloptions)::varchar as opts)
a where [Link] like '(fillfactor,%') then CEIL((reltuples*(datahdr +
nullhdr2 + 4 + ma -
(CASE WHEN datahdr%ma=0
THEN ma ELSE datahdr%ma END)
)/(bs-20))) * bs / 1-(select replace(right(opts,3),')','')::int
/100::numeric
from (select pg_options_to_table(pg_class
.reloptions)::varchar as opts ) a
where [Link] like '(fillfactor,%')
else CEIL((reltuples*
(datahdr + nullhdr2 + 4 + ma -
(CASE WHEN datahdr%ma=0
THEN ma ELSE datahdr%ma END)
)/(bs-20))) * bs end AS expected_bytes,
reltoastrelid
FROM data_headers
JOIN pg_class ON tablename = relname
JOIN pg_namespace ON relnamespace = pg_namespace.oid
AND schemaname = nspname
WHERE pg_class.relkind = 'r' ),
estimates_with_toast AS (

57
 -- add in estimated TOAST table sizes
-- estimate based on 4 toast tuples per page because we dont have
-- anything better. also append the no_data tables
SELECT schemaname, tablename,
TRUE as can_estimate,
est_rows,
table_bytes + ( coalesce([Link], 0) * bs ) as table_bytes,
expected_bytes + ( ceil( coalesce([Link], 0) / 4 ) * bs ) as
expected_bytes
FROM table_estimates LEFT OUTER JOIN pg_class as toast
ON table_estimates.reltoastrelid = [Link]
AND [Link] = 't' ),
table_estimates_plus AS (
-- add some extra metadata to the table data
-- and calculations to be reused
-- including whether we cant estimate it
-- or whether we think it might be compressed
SELECT current_database() as databasename,
schemaname, tablename, can_estimate,
est_rows,
CASE WHEN table_bytes > 0
THEN table_bytes::NUMERIC
ELSE NULL::NUMERIC END
AS table_bytes,
CASE WHEN expected_bytes > 0
THEN expected_bytes::NUMERIC
ELSE NULL::NUMERIC END
AS expected_bytes,
CASE WHEN expected_bytes > 0 AND table_bytes > 0
AND expected_bytes <= table_bytes
THEN (table_bytes - expected_bytes)::NUMERIC
ELSE 0::NUMERIC END AS bloat_bytes
FROM estimates_with_toast
UNION ALL
SELECT current_database() as databasename,
table_schema, table_name, FALSE,
est_rows, table_size,
NULL::NUMERIC, NULL::NUMERIC
FROM no_stats ),
bloat_data AS (
-- do final math calculations and formatting
select current_database() as databasename,
schemaname, tablename, can_estimate,
table_bytes, round(table_bytes/(1024^2)::NUMERIC,3) as table_mb,
expected_bytes, round(expected_bytes/(1024^2)::NUMERIC,3) as expected_mb,
round(bloat_bytes*100/table_bytes) as pct_bloat,
round(bloat_bytes/(1024::NUMERIC^2),2) as mb_bloat,
table_bytes, expected_bytes, est_rows
FROM table_estimates_plus )
-- filter output for bloated tables
SELECT databasename, schemaname, tablename,
can_estimate,
est_rows,
pct_bloat, mb_bloat,
table_mb FROM bloat_data
-- this where clause defines which tables actually appear
-- in the bloat chart
-- example below filters for tables which are either 50%
-- bloated and more than 20mb in size, or more than 25%
-- bloated and more than 4GB in size
WHERE ( pct_bloat >= 50 AND mb_bloat >= 10 )

58
 OR ( pct_bloat >= 25 AND mb_bloat >= 1000 )
AND tablename like 'usg%'
ORDER BY pct_bloat DESC;

To determine the number of open sessions on a server, and if you know the BASE24-eps DB username , execute
the following query:

`select count(*) from pg_stat_activity where datname = 'eps_postgres' and usename

= 'eps_pg_user';`

View page cache usage

To view page cache usage, execute the following query. This example is for Usage tables, filtered as ‘usgd%’.

`SELECT relname,cast(heap_blks_hit as numeric) / (heap_blks_hit + heap_blks_read)

AS hit_pct, heap_blks_hit, heap_blks_read
FROM pg_statio_user_tables WHERE (heap_blks_hit + heap_blks_read)>0 AND relname
like 'usgd%' ORDER BY hit_pct;`

View the number of sessions open on a server

To view the number of open sessions on a server when you know the BASE24-eps DB username , execute the
following query:

`select count(*) from pg_stat_activity

where datname = 'eps_postgres' and usename = 'eps_pg_user';`

View the top CPU and IO consuming queries

The `pg_stat_statements` extension can be used to view the top 20 CPU consuming
queries:

SELECT substring(query, 1, 50) AS short_query,

round(total_time::numeric, 2) AS total_time,
calls,
round(mean_time::numeric, 2) AS mean,
round((100 * total_time / sum(total_time::numeric) OVER ())::numeric, 2) AS
percentage_cpu
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 20;

To look at this from I/O point of view, pg_stat_statements can be queried like:

59
 SELECT substring(query, 1, 30) AS query,
calls,
round(total_time::numeric, 2) AS total_time,
round(blk_read_time::numeric, 2) AS io_read_time,
round(blk_write_time::numeric, 2) AS io_write_time,
round((100 * total_time / sum(total_time) OVER ())::numeric, 2) AS
percentage
FROM pg_stat_statements
ORDER BY blk_read_time + blk_write_time DESC
LIMIT 20;

PostgreSQL VACUUM command guidelines

PostgreSQL server is configured with special settings for autovacuum. For more information about those settings,
see PostgreSQL settings. Even with those settings, it is a good practice to run the manual#VACUUM# command on
a regular basis.

NOTE Do not run the#VACUUM FULL# command unless ACI advises you to do so.

The following list describes when to run the#VACUUM# command:

• On static tables, or tables that are updated with low frequency, run the#VACUUM# command once a week, best
during a special maintenance window on the weekend.
• On frequently updated tables, such as usages, rolling usages, merchant or others based on your business
transaction flow, run the#VACUUM# command once a day, best during a special maintenance window at night.
• BASE24-eps does not perform DDL operations at runtime, except when metadata changes. Make every effort to
avoid any DDL operation during the maintenance window when#VACUUM# commands are run.

For details, see the PostgreSQL 11 at ACI - ACI_AAWV.sh Script User Guide in Salesforce.

60
Section 6. Managing WebSphere MQ queue
managers
This section describes the commands to manage WebSphere MQ queue managers within a BASE24-eps
environment.

Starting and stopping a WebSphere MQ queue manager

Commands are available for starting and stopping a WebSphere MQ queue manager.

Starting a WebSphere MQ queue manager

Enter the following command to start a WebSphere MQ queue manager.

start_manager.sh

Stopping a WebSphere MQ queue manager

Before issuing the command to stop the queue manager, you must make sure that all processes that have
references to the queues or queue manager are stopped. If any process has the queue manager open the stop
command may hang until the queue manager is released. DO NOT ISSUE A LINUX/UNIX KILL COMMAND on any
of the WebSphere MQ processes. Killing the queue manager processes can result in a queue manager that is
corrupted and cannot be restarted. Instead, find the process that has the queue manager open and stop that
process. Stop the IS processes or the TPC/IP process that has the queue manager open. Also stop any runmqsc
sessions that are running against the queue manager. Once all the processes that are using the queue manager are
stopped, the queue manager finishes shutting down normally. Enter the following command to stop a WebSphere
MQ queue manager:

stop_manager.sh

Checking on channels
The client channels needed for remote connectivity are displayed in this queue.

runmqsc [Link]

DISPLAY CHSTATUS (*)

61
 1 : DISPLAY CHSTATUS (*)
AMQ8417: Display Channel Status details.
CHANNEL(TO.B2C) XMITQ([Link])
CONNAME(acio-cisaix2(4030)) CURRENT
CHLTYPE(CLUSSDR) STATUS(RUNNING)
RQMNAME([Link])
AMQ8417: Display Channel Status details.
CHANNEL(TO.B2) XMITQ( )
CONNAME([Link]) CURRENT
CHLTYPE(CLUSRCVR) STATUS(RUNNING)
RQMNAME([Link])
AMQ8417: Display Channel Status details.
CHANNEL(TO.B2) XMITQ( )
CONNAME([Link]) CURRENT
CHLTYPE(CLUSRCVR) STATUS(RUNNING)
RQMNAME([Link])
AMQ8417: Display Channel Status details.
CHANNEL(TO.B2) XMITQ( )
CONNAME([Link]) CURRENT
CHLTYPE(CLUSRCVR) STATUS(RUNNING)
RQMNAME(QM_source_control)

A status of STOPPED indicates that the channel was explicitly stopped with the STOP CHANNEL command OR the
channel has reached the limit of retry attempts at establishing a connection and the following command must be
issued to restart the channel:

START CHANNEL ( [Link])

It is not recommended that the STOP CHANNEL command be used unless you are performing maintenance on the
other end of the channel (that is, taking down the queue manager for an extended period of time.) You must
remember to manually start any channel that has been stopped with the STOP CHANNEL command. WebSphere
MQ does not automatically restart channels that have been stopped explicitly.

A full description of the CHSTATUS command including explanations of all possible status values
NOTE
can be found in the IBM WebSphere MQ Script (MQSC) Command Reference Manual.

WebSphere MQ tips
Create your queue managers once and only once. Creating a MQ manager is required only if you plan to run in
server binding mode or there is another local infrastructure requirement that mandates this. If BASE24-eps runs in
MQ client binding mode, it does not use the MQ manager created on the same server. In that case, the deployment
can use the Linux/UNIX environment variable MQSERVER or CCDT (client channel definition table) or any other
client connection method preferred by the MQ administrator.

Tips for MQ server binding

Running a script that creates a queue manager more than one time creates multiple instances of the same queue
manager. Only one instance is current and active, but all of the instances and information about queues, channels,
and so on, that go along with the queue manager is stored in the repositories. This creates, at best, a list of objects
that must be verified to be available (only one of which is available) and at worst complete confusion resulting in an
unknown object error when WebSphere MQ tries to resolve a cluster queue name. After a queue manager has been
created, you should only need to use ALTER, DELETE, and DEFINE commands from within runmqsc to maintain

62
the objects associated with the queue manager. Contact technical support for advice if you feel there is a need to
recreate an existing queue manager unless you are absolutely certain that you understand all possible side effects
of recreating queue managers in a clustered environment.

Making changes to .mqsc scripts in the mq_config directory does not result in a changed configuration in the queue
manager. These scripts are executed only when you create a queue manager. Any adds, changes, or deletes on an
existing queue manager need to be made from within the runmqsc command processor using the ALTER, DELETE,
AND DEFINE commands.

If you make a change to a queue manager configuration using runmqsc , make sure you update the associated
.mqsc file with the changes in case someone needs to recreate the system and has to run the create scripts to
create a queue manager.

Issue the display clusqmgr(*) command from within runmqsc to see if you have multiple queue managers defined
with the same name if you have unexplained errors when trying to resolve a cluster queue name.

Issue the display qcluster(*) command from within runmqsc to see a list of the cluster queues that this queue
manager knows about. If you are looking for a specific queue, it might not show up in the list until a program tries to
open it and the name is resolved to a local queue manager that hosts the queue. If it is a cluster queue that exists on
more than one queue manager within the cluster, it shows up multiple times in the list, once for each queue manager
that has been resolved.

Queue manager names should be unique in the first 12 characters. The first 12 characters of a queue manager
name are used in the algorithm that automatically generates message IDs and correlation IDs within WebSphere
MQ. The only way to guarantee unique message IDs and correlation IDs is to have unique queue manager names.

63
Section 7. JMS MQ configuration
Websphere MQ is widely used on multiple platforms to perform message services between different components.
This section illustrates how to set up JMS.

Setting up JMS
Perform the following steps to set up JMS.

Prerequisite checklist
The following prerequisites must be met before JMS and MQ can be set up:

1. Verify that the Websphere MQ is installed on the server box.

Normally the install directory is /opt/mqm.

2. Verify that the Websphere MQ for Java components are installed.

Normally the directory is /opt/mqm/java/lib.

This is a list of jars that should come with standard:

◦ [Link] [Link] [Link]

◦ [Link]
◦ [Link]
◦ [Link]
◦ [Link]
◦ [Link]
◦ [Link]
◦ [Link]
◦ [Link]

The following directory snapshot shows the list of files as an example. It shows that the [Link] was
installed/copied at the same time with the other files, and there is a jdbc directory also.

64
 3. Verify if the following jars are installed and the directory or file names are included in the class path.

From ACI, [Link] should be in the install directory with other JSF jars.

From a third party, [Link] should be in the install directory.

Create the binding file for JMS and MQ

Perform the following steps to create the binding file for JMS and MQ.

1. Create an mq_jms_admin directory in the main install directory.

2. Create an empty directory JNDI-Directory under mq_jms_admin.
3. Create an environment variables (env_vars) file identifying the MQ JMS environment in the mq_jms_admin
directory.

Template:

65
 JAVA_HOME=/usr/java
MQJMS=/opt/mqm/java
CP=$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]
CP=$CP:$MQJMS/lib/[Link]

LD_LIBRARY_PATH=$LD_LIBRARY_

PATH:
$MQJMS/lib
export JAVA_HOME MQJMS CP

LD_LIBRARY_PATH

4. Create a file to start the IBM implementation of the queue administration utility (runadmin) in the mq_jms_admin
directory.

Template:
#!/bin/sh
. ./env_vars
# ----------------------------------------------------------
# IBM Websphere MQ Support for Java Message Service
# Installation Verification Test - Setup script
#
# Licensed Materials - Property of IBM
#
# 5648-C60 5724-B4 5655-F10
#
# (c) Copyright IBM Corp. 1999. All Rights Reserved.
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
# ----------------------------------------------------------
$JAVA_HOME/bin/java -cp $CP -DMQJMS_LOG_DIR=$MQ_JAVA_DATA_PATH/log
-DMQJMS_TRACE_DIR=$MQ_JAVA_DATA_PATH/trace
-DMQJMS_INSTALL_PATH=$MQ_JAVA_INSTALL_PATH [Link]

5. Change the permissions to make this runnable:

chmod 755 runadmin

6. Create a file named [Link] or borrow it from the MQ distribution from IBM and edit.

Template (the provider URL needs to be changed):

# -----------------------------------------------------------------
# IBM Websphere MQ Support for Java Message Service
#
# This is the default configuration file for the IBM Websphere MQ Classes

66
# for Java Message Service Administration Tool.
#
# Licensed Materials - Property of IBM
#
# 5648-C60 5724-B4 5655-F10
#
# (c) Copyright IBM Corp. 2002. All Rights Reserved.
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
# ------------------------------------------------------------------
#
# The following line specifies which JNDI service provider is in use.
# It currently indicates an LDAP service provider. If a different
# service provider is used, this line should be commented out and the
# appropriate one should be uncommented.
#
#INITIAL_CONTEXT_FACTORY=[Link]
#INITIAL_CONTEXT_FACTORY=[Link]
INITIAL_CONTEXT_FACTORY=[Link]
#
# The following line specifies the URL of the service provider's initial
# context. It currently refers to an LDAP root context. An example of a
# file system URL is also shown, commented out.
#
#PROVIDER_URL=ldap://polaris/o=ibm,c=us
PROVIDER_URL=file:/db01/b24dev/ui/mq_jms_admin/JNDI-Directory
#
# The following line specifies the security authentication model in use,
# and may be 'none' (for anonymous authentication), 'simple', or 'CRAM_MD5'.
#
SECURITY_AUTHENTICATION=none
#
# If you don't have SECURITY_AUTHENTICATION=none, then JMSAdmin will
# prompt you for the User DN and password. If you want to bypass these
# prompts then you can specify one or both of the values here. Since
# the password here is in cleartext this is not normally recommended
# except for testing. You should replace these values with your own.
#
#PROVIDER_USERDN=cn=Manager,o=ibm,c=uk
#PROVIDER_PASSWORD=secret
#
#
# The following line determines whether to use an InitialDirContext, or an
# InitialContext. Takes value of TRUE or FALSE.
#USE_INITIAL_DIR_CONTEXT=TRUE
#
# The following line specifies a prefix to add to names when carrying out
operations such
# as lookup/bind.
#NAME_PREFIX=cn=
#
# The following line specifies a marker at which names will be truncated when
viewing
# the contents of the Context.
#NAME_READABILITY_MARKER=..
#
# The three standard types of InitialContextFactory have the following
defaults;
# Note that these defaults will be set automatically if the flags are not
present, but

67
 # will be overrided by including the flags.
#
# LDAP FSCONTEXT WEBSPHERE
#
-------------------------------------------------------------------------------
-----
# USE_INITIAL_DIR_CONTEXT TRUE FALSE FALSE
# NAME_PREFIX cn= omitted omitted
# NAME_READABILITY_MARKER omitted omitted ..
#

7. Create a [Link] file that has definition of the JMS MQ connection.

Template (change the port number to something that is available):

DEF QCF(JmsQueueConnectionFactory)

HOSTNAME(acio-cissun18)

TRANSPORT(CLIENT)

QMANAGER([Link])

PORT( 28888 )

NOTE
For MQ client set-up, the TRANSPORT parameter must be set to CLIENT for an MQ client or SERVER for
an MQ server. The .in file contains the following:

DEF QCF(JmsQueueConnectionFactory)

HOSTNAME(<hostname_or_IP>)

TRANSPORT(CLIENT)

QMANAGER(<Q_manager_name>)

CHAN(<MQ_channel_name>)

PORT(<MQ_listener_port>)

The channel name

8. Execute the runadmin. At the prompt, paste in the content of the [Link] and press the Enter key.

InitCtx> DEF QCF(JmsQueueConnectionFactory)

HOSTNAME(acio-cissun18)

TRANSPORT(CLIENT)

QMANAGER([Link])

68
 PORT( 28888 )

This creates the .binding file in the JNDI_Directory.

9. Display and check the definition.

InitCtx> display qcf(*)

QCF(JmsQueueConnectionFactory)

FAILIFQUIESCE(YES)

HOSTNAME(acio-cissun18)

PORT(28888)

QMANAGER([Link])

USECONNPOOLING(YES)

CCSID(819)

TEMPMODEL([Link])

MSGBATCHSZ(10)

TRANSPORT(CLIENT)

SYNCPOINTALLGETS(NO)

TEMPQPREFIX(AMQ.*)

MSGRETENTION(YES)

RESCANINT(5000)

POLLINGINT(5000)

LOCALADDRESS()

VERSION(2)

CHANNEL([Link])

10. Enter ‘end’ to exit from the prompt.

InitCtx> end

NOTE If in need to regenerate the .binding file, the previous one has to be deleted first.

69
MQ client connection details for JMS

When the MQ client connection is used, the TRANSPORT parameter must be set to CLIENT.

The [Link] file should look like the following:

DEF QCF(JmsQueueConnectionFactory) +
HOSTNAME(<hostname_or_IP>) +
TRANSPORT(CLIENT) +
QMANAGER(<Q_manager_name>) +
CHAN(<MQ_channel_name>) +
PORT(<MQ_listener_port>)
//TODO Developer, please complete the language
[source,<language>]

The <MQ_channel_name> must match the channel name used within the MQ server configuration.

Start the MQ listener

Add nohup runmqlsr -m $QMNGR -t TCP -p 28888>> [Link] & to start_manager.sh, the MQ start
script, and restart MQ. Another option is running it at the command prompt, which starts the MQ listener for the
selected port.

70
Section 8. Extract processing
This section contains procedures for configuring, starting, and stopping automatic extracts based on a fixed number
of hours on a Linux/UNIX system. For further information about extracts, see the BASE24-eps Extract and Reporting
Users Guide.

The physical transfer of the extracted data from the Linux/UNIX system to another host system for processing is
accomplished through a File Transfer Protocol (FTP) or tape mechanism.

Journal Query Configuration

To add an extract query to the BASE24-eps application, log on to the BASE24-eps application UI and navigate to
Journal Query Configuration ( Configure > Journal > Journal Query Configuration ). An example extract query is
shown below. In this example, the query is configured to begin the extract based on a fixed number of hours.

71
Auto restart options
The End-of-Period Process (EOPP) uses the following information to schedule automatic extracts:

Auto Start Timer Checking this box enables automatic extracts.

Restart Period Extracts restart every fixed number of hours.

Restart Time This specifies the restart time that the scheduled extracts are based from.

Number of Hours The number of hours between extracts. This field should be set to a number that evenly
divides into 24 (1, 2, 3, 4, 6, 8, 12) so that the extract times are predictable.

ESBLDJNL script
After you have configured your Journal Profiles (see Configuring Journal Profiles section within the Operations
Guide ), you can run the “esbldjnl” script to actually create these Journal files and update your BASE24-eps
Metadata configuration (that is, $CONFIG/[Link] and $CONFIG/[Link]) files.

To use the script:

1. Enter your new Journal assign names into the Journal Profile window in the UI.
2. The script uses file JOURNAL_TEMPLATE as a template. This filename should already exist in your
$CONFIG/[Link] and $CONFIG/[Link] files. It uses the file definition in $CONFIG/[Link] as the basis for what
the new Journal files are created like. You may want to look at the $CONFIG/[Link] and the attributes associated
with the JOURNAL_TEMPLATE file, to ensure it fits the way you want these Journal files to look for your site.
3. Run the esbldjnl from any directory. The “esbldjnl” script is in your $BIN directory. Your $BIN directory is
automatically searched because it was placed in your PATH variable at installation time.

When esbldjnl runs, it modifies your $CONFIG/[Link] and $CONFIG/[Link] files to include the new Journal
assigns you created in your Journal Profile UI windows. It also saves a copy of $CONFIG/[Link] and
$CONFIG/[Link] with appending a date/time to their name.

Below is a session with additional comments (in red) on how to use it:

> esbldjnl
Processing Journal Profile: JLF-BNK1-A40
Journal Profile Name: JLF-BNK1-A40
JLF_BNK1_A40_1
JLF_BNK1_A40_2
JLF_BNK1_A40_3
JLF_BNK1_A40_4
Do you want to process the above listed Journal File(s)? (y/n):y

The above files were not in the $CONFIG/[Link] file and are displayed to alert you for possible processing.

By telling it to process this Profile, it prompts you for what alternate indexes to use for these new Journal files. You
are only asked these questions once for each esbldjnl run.

72
 Do you want to use the Account Key? (y/n):y
Do you want to use the Channel Key? (y/n):n
Do you want to use the First Clerk Key? (y/n):n
Do you want to use the Second Clerk Key? (y/n):n
Do you want to use the Merchant Key? (y/n):y
Do you want to use the PAN Key? (y/n):y
Enter prefix for physical filename (## is appended):bnk1a40

The above prefix for physical filename is the filename you see if you go to the directory location.

/db01/b24dev/jkreife/j62a/db/jnldata
Use the above default directory path? (y or type in directory):/db01/b24dev/jkrei
fe/j62a/db/newdata

The above lines show you your default Journal directory, which comes you are your $JNLDATA variable, and if you
want to create your new Journal files there. You can respond by entering a new directory, or enter y to accept
default.

ACI's Ctree Data File Create/Build Utility

Version 3.00.00
July 26, 2004
-- Created data and index files for data file: /db01/b24dev/jkreife/j62a/db/newd
ata/bnk1a401
Using C-tree server: J62ASVR@ACIO-CISSUN17
-- Created data and index files for data file: /db01/b24dev/jkreife/j62a/db/newd
ata/bnk1a402
Using C-tree server: J62ASVR@ACIO-CISSUN17
-- Created data and index files for data file: /db01/b24dev/jkreife/j62a/db/newd
ata/bnk1a403
Using C-tree server: J62ASVR@ACIO-CISSUN17
-- Created data and index files for data file: /db01/b24dev/jkreife/j62a/db/newd
ata/bnk1a404
Using C-tree server: J62ASVR@ACIO-CISSUN17
INFO -> Number of data files created: 4
END OF RUN -> Data File creation is now complete!

The above lines show the c-tree build output for the new journal files.

Do you want to continue to the next profile? (y/n):y

You can now exit esbldjnl or proceed to the next Journal Profile record.

Processing Journal Profile: JLF-BNK1-A99 Do you want to continue to the next profile?
(y/n):n

If there are no Journals in the next profile that need building, choose exit. The above session created my new
Journal Files, with using only four indexes.

To use your new Journal Files, recycle your BASE24-eps processes so that the new updated $CONFIG/[Link] is
loaded.

73
Start automatic extract
This section describes procedures for starting an extract that runs automatically at the specified times.

If the EOPP is not running, start it. If the Auto Start Timer box is checked on the Journal Query Configuration
window, the EOPP starts a timer to perform the next extract.

If the EOPP is running and a change is made to the Journal Query Configuration (for example, checking the Auto
Start Timer), then automatic extracts can be restarted by altering the Journal Query Configuration data source.

Altering the Journal Query Configuration data source causes the EOPP to reread the Journal Query Configuration
data source. Any timers that were started are cancelled and restarted, and any timers that have not started are
started. To alter the Journal Query Configuration data source, perform an Alter Data Source command from the
Process Control UI shown below.

Next extract run time calculation

The next run time calculation is based on the Auto Restart Options configured on the Journal Query Configuration
UI. The next run time is calculated by incrementing the previous run time by the fixed number of hours configured. If
the previous run time is zero, then the current time is used along with the restart time. The previous run time is set to
zero whenever a change is made on the Journal Query Configuration UI.

Once an automatic extract is initiated, it continues to run every configured number of hours until it is stopped. The
extract runs at the next available time based on the restart hour. For example, if the restart hour is 7:02 a.m. and the

74
number of hours is 2, then extracts run at 7:02 a.m., 9:02 a.m., 11:02 a.m., 1:02 p.m., 3:02 p.m., 5:02 p.m., 7:02
p.m., 9:02 p.m., 11:02 p.m., 1:02 a.m., 3:02 a.m., and 5:02 a.m. Automatic extracts do not wait until the restart hour
to run nor do they stop at midnight and resume at the restart hour of the next day.

Scenario 1: Query execution configuration modified

A configuration change has been made to the Journal Query Configuration UI and the EOPP is currently running.
When the Journal Query Configuration data source is altered or a manual extract is performed, the next run time is
calculated as follows:

Restart time Number of hours Current time Next run time

7:02 a.m. 2 1:00 a.m. 1:02 a.m.
7:02 a.m. 2 2:30 p.m. 3:02 p.m.
7:02 a.m. 2 5:30 a.m. 7:02 a.m.
10:30 a.m. 3 8:00 a.m. 10:30 a.m.
10:30 a.m. 3 5:20 p.m. 7:30 p.m.
12:00 p.m. 4 7:00 a.m. 8:00 a.m.
12:00 p.m. 4 4:01 p.m. 8:00 p.m.
8:45 a.m. 6 5:00 a.m. 8:45 a.m.
8:45 a.m. 6 9:00 a.m. 2:45 p.m.
2:35 p.m. 8 8:35 a.m. 2:35 p.m.
2:35 p.m. 8 11:00 p.m. 6:35 a.m. (next day)
7:00 p.m. 12 1:00 p.m. 7:00 p.m.
7:00 p.m. 12 9:00 p.m. 7:00 a.m. (next day)

Scenario 2: EOPP restart_ no configuration changes

The Journal Query Configuration has not been changed but the EOPP is currently stopped. When the EOPP
process is started, the next run time is calculated as follows:

Last run date/time Number of hours Current date/time Next run date/time
Day 1, 9:02 a.m. 2 Day 1, 5:10 p.m. Day 1, 7:02 p.m.
Day 1, 5:02 p.m. 2 Day 4, 8:30 a.m. Day 4, 9:02 a.m.
Day 1, 7:30 p.m. 3 Day 1, 10:45 p.m. Day 2, 1:30 a.m.
Day 1, 4:30 a.m. 3 Day 2, 8:15 a.m. Day 2, 10:30 a.m.
Day 1, 4:00 p.m. 4 Day 1, 7:00 p.m. Day 1, 8:00 p.m.
Day 1, 8:00 a.m. 4 Day 3, 2:20 p.m. Day 3, 4:00 p.m.
Day 1, 8:45 a.m. 6 Day 1, 7:30 p.m. Day 1, 8:45 p.m.

75
Last run date/time Number of hours Current date/time Next run date/time
Day 1, 2:45 p.m. 6 Day 2, 10:50 a.m. Day 2, 2:45 p.m.
Day 1, 6:35 a.m. 8 Day 1, 7:10 a.m. Day 1, 2:35 p.m.
Day 1, 2:35 p.m. 8 Day 4, 3:00 p.m. Day 4, 10:35 p.m.
Day 1, 7:00 p.m. 12 Day 1, 11:15 p.m. Day 2, 7:00 a.m.
Day 1, 7:00 a.m. 12 Day 3, 8:30 a.m. Day 3, 7:00 p.m.

Stop an automatic extract

Automatic extracts are stopped by unchecking the Auto Start Timer box, located in the Auto Restart Options section
of the Journal Query Configuration UI.

Extracts can be stopped immediately by altering the Journal Query Configuration data source.

If one more extract is acceptable after the Auto Start Timer box is unchecked, then the user can wait until this extract
is performed, after which no more extracts are scheduled.

76
Starting a manual extract
Another method of starting an extract is through manual initiation. To manually initiate an extract, log on to the
BASE24-eps application and navigate to the Query Execution UI ( System Operations Query Execution ). Select
the name of the query that you want to start manually.

If the retain position is not checked in the Journal Query Configuration UI, a manual extract selects from all of the
transactions in the journal for the current day.

Manual query execution - Start option

When starting a manual query extract (Start button selected), the extract is run based on current journal position
information that was built on the previous extraction run, if the retain position box is selected on the Journal Query
Configuration UI.

Performing a manual extract (Start) does not cause the timer for the automatic extract to be cancelled or
recomputed.

77
Manual query execution - Restart option

When restarting a query extract manually (Restart button selected), a list of previous run dates and times is
displayed. The restart option works only when the retain position box is selected on the Journal Query Configuration
UI. The user then selects the date and time of the previous extract to be restarted and clicks Select. The manual
extract restart is run based on the selected last query date/time and the selected last queries associated journal
position information.

A user would select to restart an extract when trying to reproduce an extract that was previously generated.

Performing a manual extract (restart) does not cause the timer for the automatic extract to be cancelled or
recomputed.

78
Section 9. File partition processing
Performing a full file refresh on a Linux/UNIX system involves loading a prepared file into a c-tree table so that it is
available to the application. Loading a c-tree table is done most efficiently by using the CTBLD utility with the append
option.

The GoldenGate product uses the concept of a c-tree database transaction to perform replication. If
NOTE the mechanism being used for full file refresh is based on a file replacement, the file is not
replicated on the backup system(s).

For full file refresh to function correctly, this operation must be performed on each of the individual c-tree database
instances, for both local and remote contingency, as applicable in the deployed system.

Preparing the input file

The input file must be prepared with a c-tree header on each record. The format of the c-tree header is:

Position Field Description

1-2 Record Marker The value 0xFAFA.
3-6 Total Record Length The length of the data record plus the10-byte header.
7 - 10 Utilized Length The length of the data record without the 
10-byte
header.

This layout is best verified by displaying your input file in hex (od - x filename). A properly formatted input file with c-
tree headers looks like the sample shown below.

The data highlighted in green is the c-tree header. The data highlighted in yellow is the sis_tbl_ver field, which
should match the version number of your input.

79
 0000000

fafa 0000 03ca 0000 03c00000 0003

3030
0000020 3030 3120 2020 2020 2020 2020 2020 2020
0000040 2020 3433 3230 3031 3030 3030 3030 3030
0000060 3030 3438 3520 2020 2020 2020 2020 3030
0000100 3030 0000 000b abcf eb0c 3035 3035 3033
0000120 3134 3332 3133 4150 4231 3130 3458 2020
0000140 2020 2020 2020 2020 2020 2020 2020 2020
0000160 2020 2020 2020 3030 3030 3030 3030 3030
0000200 3030 3030 3039 3230 2020 2020 2020 2020
0000220 2020 5643 3030 3035 3031 3237 3137 3033
0000240 3136 3438 3336 2020 2020 2020 2020 3035
0000260 3031 3237 3137 3033 3136 3230 3035 3031
0000300 3238 3230 3036 3031 2020 2020 2020 2020
0000320 2020 2020 2020 2020 2020 2020 2020 2020
0000340 2020 2020 7878 7878 2020 2020 2020 2020
0000360 2020 2020 2020 2020 2020 2020 2020 2020
0000400 2020 2020 0000 0000 0000 0000 0000 0000
0000420 0000 2020 2020 2020 2020 2020 2020 2020
0000440 2020 0000 0000 0000 0000 0000 0000 0000
0000460 2020 2020 2020 2020 2020 2020 2020 2020
0000500 0000 0000 0000 0000 0000 0000 0000 2020
0000520 2020 2020 2020 2020 2020 2020 2020 0000
0000540 0000 0000 0000 0000 0000 0000 2020 2020

Running the CTBuild utility

Once you have a file with properly formatted c-tree headers on each record, you can run the CTBuild utility. CTBLD
recreates the file, so delete the file that you are going to load first using the rm command:

rm $B24DATA/crd01b.*

You need a definitions file for the table you are loading. The following definitions file is for a Card table with assign
CARD_01B.

80
 [
SYSINFO]
num_datafile = 1
num_indexes ??
[DATAFILE]
# c-tree creation script for table: Card (crdd)
# Assign: CARD_01B created on 3/1/2005 10:27:43 AM
filename = T052SVR@oma3h001:/db01/b24dev/SystemTest/t052/db/b24data/crd01b

max_rec_len = 1960

min_rec_len = 4

audit = Y
num_index = 1
[INDEX]
index_name = T052SVR@oma3h001:/db01/b24dev/SystemTest/t052/db/b24data/crd01b_Key
1
key_len = 51
key_type = primary
key_unique = Y
num_sgmnt = 3
[KEYSEG]
# Key Element Name: inst_id
offset = 4
length = 20
[KEYSEG]
# Key Element Name: pan
offset = 24
length = 28
[KEYSEG]
# Key Element Name: crd_seq_num
offset = 52
length = 3

You run the CTBLD utility using a script. Here is one that uses the above definitions file ([Link]) and loads it
using input BECARDF3.

$SISOBJ/ctbld \
$CONFIG/[Link] \
-uADMIN -pADMIN \
-mdbv=csv \
-ai=$AI \
-strt=$CONFIG \
-append=$DB/BECARDF3

Now that the file is loaded, you are ready to make it the active file used by the application using the directions in the
BASE24-eps File Partitioning User Guide.

81
Section 10. Partial refresh processing
Partial refreshes are batches of updates to an application file that follow a defined format. This section describes the
processing performed.

The following files are involved in a partial refresh:

• Input file
• Target file
• Resulting report

The basic processing is to read a record of the input file, apply the update to the target file, and note any errors or
exceptions to the report.

Processing steps
Follow these steps:

1. Load your prepared input into the input file.

2. Initiate the processing by sending process control command to the RFSH process. The command is a “Deliver”
command to component ID _ Partial Refresh (RFSH)_ with the text Input [input file assign] <Report [report file
assign]>. The assigns must be existing entries in your [Link].
3. The refresh runs.
4. Check the report.

To do a partial Star IPF Refresh, the command has to go to component ID Star IPF Refresh
NOTE
(STARIPF).

Loading the prepared input into the input file

The only files that ACI supports for reads on UNIX are those in c-tree, so the input file needs to be loaded into a c-
tree file. This is done most efficiently by recreating the file using CTBLD, specifying the append option. The data
that the file is loaded with must be prepared with c-tree headers if it is to be used as the input to CTBLD.

For this purpose, a utility called headeradd which is found at …\Server\utils\partial_refresh. This utility reads your
prepared input file and outputs it into a new file with c-tree headers for each line of input.

There are two assigns in the standard [Link] for use as partial refresh input files: REFRESH_INPUT1 and
REFRESH_INPUT2.

Follow these steps.

1. Get an input file from your host or other source. Let us call this file input.
2. Run the#headeradd input input_w_hdr# command to create a file with c-tree headers.
3. Obtain a definition file for the file you want to load.

82
 Here is the contents of the definition file if you want to load REFRESH_INPUT1:

[SYSINFO]
num_datafile = 1
num_indexes ??
[DATAFILE]
# c-tree creation script for table: Stream (strmd)
# Assign: REFRESH_INPUT1 created on 3/1/2005 10:27:43 AM
filename = T052SVR@oma3h001:/db01/b24dev/SystemTest/t052/db/b24data/refr1

max_rec_len = 4062

min_rec_len = 0

audit = N
num_index = 0

4. Run CTBLD with the - append option using your input with headers and the definitions file.

Here is a sample script that uses a definition file called [Link] and an input file called input_w_hdr:

$SISOBJ/ctbld \

$CONFIG/[Link] \

-uADMIN -pADMIN \

-mdbv=csv \

-ai=$AI \

-strt=$CONFIG \

-append=$DB/input_w_hdr

5. The file is now loaded, you can verify by doing a select using DALCI from REFRESH_INPUT1.

83
Section 11. Contingency processing
This section describes BASE24-eps contingency processing, including details on how to set up the following
BASE24-eps contingency processing:

• Transaction Contingency Interface

• User Interface Contingency

Transaction Contingency Interface

This section contains details and examples of how to configure BASE24-eps Contingency Interface processing on a
primary system (processing transactions) and a backup system (logging transaction updates via the transaction
contingency interface). Details on the configuration of the Contingency Interface UI and the required contingency
setup are shown below.

User interface
Log on to the BASE24-eps application and navigate to the Contingency Interface Configuration UI (Configure >
Interface > Host > Contingency). Enter the Contingency interface configuration data for the primary system on the
Contingency Interface UI.

84
On the Store and Forward (SAF) tab, enter the names of the SAF file assigns that are set up in [Link].
These are the store-and-forward files used by the BASE24-eps Contingency Interface for processing.

85
No user-initiated log on to the Contingency Interface is required (End of Period Process (EOPP) must be running).
The Contingency Interface starts sending transactions to the backup system as soon as the system is up and
running and communication to the backup system has been established. The EOPP needs to be started before the
Contingency Interface starts sending data to the remote system for processing.

The SYSP_CNTGY_SAF file is where transactions being sent to the backup system are queued.

On the Station(s) tab, configure the stations used for BASE24-eps Transaction Contingency Interface processing.

86
87
On the Contingency tab, select the Interface Enabled box for the primary system.

88
From the Process Control UI, rebuild the following On-Line Transaction Processing (OLTP) files.

• INTERFACE_OLTP
• OUTBOUND_STATION_OLTP
• INBOUND_STATION_OLTP

Repeat the following Process Control commands for these OLTPs.

89
Log Message Output:

60 0DSLDR: Data source CONTINGENCY_INTERFACE_OLTP

successfully loaded.

Instruct any running IS and XML processes to use the new Contingency Interface OLTP that has been built.

90
Log Message Output:

70 0DASRC: Warmboot not needed for CONTINGENCY_INTERFACE_OLTP

70 0DASRC: Warmboot not needed for CONTINGENCY_INTERFACE_OLTP
70 0DASRC: Warmboot not needed for CONTINGENCY_INTERFACE_OLTP
60 0DASRC: Successful warmboot of data source:
CONTINGENCY_INTERFACE_OLTP

Linux/UNIX configuration
This section describes the BASE24-eps setup required on the Linux/UNIX platform for the primary and backup
contingency systems.

Primary (transaction processing) system

The following entry is included on the primary system in the /config/admf file. The primary system queues transaction
contingency records to the MQ queue of an Integrated Server on the backup system for contingency processing.

CNTNGNCY [Link]

Backup (transaction contingency) system

The following entry is included on the backup system in the /config/admf file. The ADMF entry on the backup system
points to an MQ queue of an Integrated Server on the primary system, to send back a response for each Transaction
contingency record processed on the backup system.

CNTNGNCY [Link]

91
 ACI designs contingency systems to have two types of Integrated Servers running on each system
- ones that primarily perform online transaction processing and ones that perform only contingency
NOTE
processing. The queues for these two IS types are separate to minimize the impact of contingency
work on online transaction throughput.

User interface contingency

This section contains details on how to configure BASE24-eps user interface contingency processing for a primary
system (database changes being made by users) and a backup system (logging user database changes). Shown
below are details on configuring the User Interface Contingency processing and the setup required on the
Linux/UNIX platform.

UI contingency processing starts as soon the BASE24-eps system is started or a UI database update is placed in
the UI Contingency Store and Forward file (SAF). No user logon is required.

User interface
Login to the BASE24-eps application and navigate to the Contingency Store and Forward Configuration (System
Operations > Contingency SAF).

Enter the User Interface Contingency configuration data for the primary system. Enter the name of the User Interface
Contingency Station Name. This is used in the ADMF configuration on the following page.

92
Environment variable
An environment variable must be set up for UI contingency processing. Shown below are the settings for the primary
system. Log on to the BASE24-eps application and navigate to the Environment variables (System Operations >
Environment), shown below.

System UI_CNTGY_LEVEL Value

Primary System 1

Backup System 0

Linux/UNIX configuration
This section describes how to set up BASE24-eps application on the Linux/UNIX platform for the primary and
backup User Interface contingency systems.

Primary (users modified database) system

The following entries are included on the primary system in the /config/admf file. The primary system queues User
Interface contingency records to the MQ queue of the Integrated Server on the backup system for contingency
processing.

93
 UICTGY [Link]
XMLI [Link]

The primary system needs to have the environment entry (in the environment data file) for
UI_CONTINGENCY_LEVEL set to 1.

Backup (user interface contingency) system

The following entries are included on the backup system in the /config/admf file. The ADMF entry on the backup
system points to an MQ queue of an Integrated Server on the primary system, to send back response for each User
Interface contingency record processed on the backup system.

UICTGY [Link]

The backup system needs to have the environment entry (in the environment data file) for
UI_CONTINGENCY_LEVEL set to 0.

94
Section 12. Understanding the c-tree server
configuration file
This section describes the parameters available for configuring a c-tree server and provides a sample c-tree server
configuration.

c-tree server configuration parameters

The following parameters are available for configuring a c-tree server.

Each parameter description identifies whether that parameter has a default value.

SERVER_NAME

Default value: FAIRCOMS

A name assigned to c-tree Server, instead of FAIRCOMS.

Example:

SERVER_NAME <NAME>

LOCAL_DIRECTORY

Default value: The server’s working directory.

One of two mutually exclusive ways to supply the c-tree server with the name of a directory path for processing files
without absolute names. Absolute names include a specific volume or drive reference as part of the name (for
example, d:\fairserv\data\). The trailing slash is required. If a LOCAL_DIRECTORY name is defined in the
configuration script, the name is attached to the beginning of any file name that is not absolute. If neither
LOCAL_DIRECTORY nor SERVER_DIRECTORY is supplied, database and system files are stored relative to the c-
tree server working directory. LOCAL_DIRECTORY and SERVER_DIRECTORY cannot be used together.

The LOCAL_DIRECTORY does not become a permanent part of the file name. The name entered
NOTE into the transaction log does not include the LOCAL_DIRECTORY. LOCAL_DIRECTORY does not
affect the location of the c-tree server status log, [Link].

Example:

LOCAL_DIRECTORY <Path>

Make sure each instance of c-tree server has defined different path for the LOCAL_DIRECTORY to
NOTE
avoid overwriting c-tree log files.

COMM_PROTOCOL

Default value: Platform dependent (most servers load TCP/IP as default)

Specifies a communications module loaded by the server. Some c-tree servers support several protocols

95
simultaneously (that is, Windows and Macintosh). For example, the c-tree server could be communicating with users
through a telephone line, others on a Novell network, and still others on an Ethernet connection. All that is needed is
a separate “COMM_PROTOCOL” line in the configuration script for each communication module to be loaded by the
c-tree server.

Example:

The following example loads TCP/IP, NetBIOS and IPX/ for the communications options available for your platform:

COMM_PROTOCOL F_TCPIP
COMM_PROTOCOL FNETBIOS
COMM_PROTOCOL FSPX

FILES

Default value: 100

The maximum number of data files and indexes where each index, whether or not in a separate file, counts toward
this total. For example, an index file which supports (that is, contains as separate index members) three different
keys counts as three files toward the FILES total. There is no effective limit to the number of files supported by the c-
tree server, except for any limits imposed by the available system memory.

Example:

FILES <Number of Files>

CONNECTIONS or USERS

Default value: Lower of 128 or activation limit.

The maximum number of connections to the c-tree server. Typically, c-tree servers are activated to support up to one
of the following values for concurrent user connections: 8, 16, 32, 64, 128, 256, 512, or 1024. However, your
particular c-tree server may be customized with a different value for connections. Specifying a number of users
greater than the actual number of users needed results in inefficiencies (for example, unused memory), so the goal
is to keep this number as low as feasible on the system. The Activation Key flier displays the allowable number of
users for your c-tree server, as does the c-tree server startup window.

Example:

CONNECTIONS <Number of Connections>

DAT_MEMORY

Default value: See note below.

The memory allocated to the data cache, specified in bytes. Within the memory constraints of the hardware, there is
no effective limit.

Example:

DAT_MEMORY <bytes>

96
 The default value for both the standard c-tree server and the c-treeSQL server is 600 *
NOTE PAGE_SIZE. Assuming a default page_size of 8192, the default DAT_MEMORY would be 4915200
for both the standard c-tree server and the c-treeSQL server.

NOTE Make sure system has enough memory before adding this parameter.

IDX_MEMORY

Default value: See note below.

The memory allocated to the index cache, specified in bytes. Within the memory constraints of the hardware, there is
no effective limit. High-speed buffer search routines ensure quick access to the entire cache.

Example:

IDX_MEMORY <bytes>

The default value for both the standard c-tree server and the c-treeSQL server is 600 *
NOTE PAGE_SIZE. Assuming a default page_size of 8192, the default DAT_MEMORY would be 4915200
for both the standard c-tree server and the c-treeSQL server.

NOTE Make sure system has enough memory before adding this parameter.

MAX_DAT_KEY

Default value: 32

Maximum number of indexes per data file.

Example:

MAX_DAT_KEY <Max Indexes per Data File>

MAX_KEY_SEG

Default value: 12

Maximum number of key segments allowed per index.

Example:

MAX_KEY_SEG <Max Segments per Index>

LOCK_HASH

Default value: 16

The number of hash bins available to the lock hash algorithm. A lock table exists holding all lock entries for each
user. A hash algorithm is employed to search this table. The LOCK_HASH value specifies the number of 8-byte
hash bins available for use by this algorithm. This value should only be increased with careful consideration. There is

97
a marginally decreasing return for increasing values.

LOG_SPACE

Default value: 10

This is the number of megabytes of disk space allocated to storing active transaction logs, starting with a minimum
of 2. The c-tree server maintains up to 4 active log files, which consume, in the aggregate, up to LOG_SPACE
megabytes of disk space. Log files are numbered consecutively starting with one. The log file names are in the form
[Link].

Example:

LOG_SPACE <Megabytes>

BUFR_MEMORY

Default value: 64000

Specifies the size of memory blocks the c-tree server uses in conjunction with data and index cache buffers. To
minimize interaction with the underlying system memory manager, the c-tree server manages its own blocks of
memory out of which the buffer pages are allocated. The c-tree server acquires one large block of memory and
allocates smaller pieces as needed. If you are attempting to limit memory use by reducing IDX_MEMORY and/or
DAT_MEMORY, set BUFR_MEMORY to about one eighth of the smaller of IDX_MEMORY and DAT_MEMORY.

Example:

BUFR_MEMORY <bytes>

PAGE_SIZE

Default value: 8192

The number of bytes per buffer page (maximum 65536 bytes). This value is rounded down to a multiple of 128. To
encourage compatibility across different c-tree Plus environments, we suggest not modifying the PAGE_SIZE.
However, if performance is of concern, this value can be modified to suit the characteristics of the operating system.
Generally, this is a matter to discuss with the application programmer.

Example:

PAGE_SIZE <bytes>

CACHE_LINE

Default value: 16

Ensure the cache line setting matches the setting for your equipment to maximize the performance of the c-tree
server under multi-CPU systems.

A cache-line is the smallest amount of memory a processor retrieves and stores in its highest speed cache. Typical
cache-line sizes are 16, 32 or 64 bytes.

Example:

98
CACHE_LINE <size>

NODEQ_SEARCH

Default value: 50

Provides for more efficient cleanup of deleted index nodes.

When an index node becomes empty, an entry for the node is added to the delete node queue. An internal server
thread reads entries from this queue and makes the empty nodes available for reuse. This parameter value specifies
how many delete node queue entries are searched for redundancy when adding an entry to the delete node queue.
Increasing the value of this parameter has the effect of reducing unnecessary duplication in the delete node queue.

CHECKPOINT_INTERVAL

Default value: 2MB

This keyword can speed up recovery at the expense of performance during updates. The interval between
checkpoints is measured in bytes of log entries. It is ordinarily about one-third the size of one of the active log files
(L000….FCS). Reducing the interval speeds automatic recovery at the expense of performance during updates.

The entry is interpreted as bytes if greater than 100 or as megabytes if less than 100. For example,
CHECKPOINT_INTERVAL 2 sets an approximate 2MB interval and CHECKPOINT_INTERVAL 150000 sets an
approximate 150,000 byte interval.

Example:

CHECKPOINT_INTERVAL <interval in bytes or MB>

COMPATIBILITY TCPIP_CHECK_DEAD_CLIENTS

Default value: No check

The c-tree server normally recognizes when a client disconnects. However, the server relies on a chain of events
controlled by the operating system To recognize the disconnection. The client computer must notify the server host
computer that the connection has been dropped.

Example:

When a user closes an application, the socket is closed by the operating system, which sends a message to the
server host machine. However, if the network connection is temporarily interrupted or if the client machine is
powered down suddenly, this message is not sent and the server host machine cannot recognize that the client
connection has dropped.

COMPATIBILITY SYNC_LOG

Instructs the c-tree server to open its transaction logs in synchronous write (direct I/O on Solaris) mode. In this
mode, writes to the transaction logs go directly to disk (or disk cache), avoiding the file system cache, so the server
is able to avoid the overhead of first writing to the file system cache and then flushing the file system cache buffers
to disk. This keyword also causes flushed writes for data and index files to use direct I/O. Using this keyword
enhances performance of transaction log writes.

COMPATIBILITY FDATASYNC

99
On appropriate Linux/UNIX systems, this keyword instructs the c-tree server to use the fdatasync() file system call
rather than the fsync() file system call. The fdatasync() call can provide a more efficient flushing of data to disk for
some database activities. The effect of this call on Solaris system is described below:

Solaris man page: The fdatasync() function forces all currently queued I/O operations associated with the file
indicated by file descriptor files to the synchronized I/O completion state.

The functionality is as described for fsync(3C) (with the symbol _XOPEN_REALTIME defined), with the exception
that all I/O operations are completed as defined for synchronised I/O data integrity completion.

COMPATIBILITY EXTENDED_TRAN_ONLY

This keyword enforces the use of only files with extended transaction numbers, and causes a R6BT_ERR(745) on
an attempt to create or open a non-extended-transaction-number file. A read-only open is not a problem since the
file cannot be updated. This configuration option has no effect on access to non-transaction files, as transaction
numbers are not relevant to non-transaction files.

TRANSACTION_FLUSH

Default value: 100

This keyword provides control for the maximum number of updates to a buffer (data or index) before it is flushed.
The buffer may well be flushed prior to this number of updates because of the LRU (Least Recently Used) scheme
or because of the checkpoint limit. This system parameter affects only buffers holding images for transaction
controlled files. Reducing this value reduces the amount of buffering, slowing system performance; but decreases
the amount of work to be performed during recovery. A value of zero causes the buffer to be flushed upon update.

Example:

TRANSACTION_FLUSH <# of updates>

MEMORY_FILE

The c-tree server supports creating memory files using a server configuration keyword. This feature enables
developers to create memory files using their existing application code, provided the file is created using an Xtd8
create function such as CreateIFileXtd8. The MEMORY_FILE keyword is useful to quickly test how a file or set of
files behave as memory files.

To create a memory file using the server configuration keyword, specify one or more entries of the form shown in the
example.

Example:

MEMORY_FILE <file name>#<max size>

The file name variable can include wildcard characters, and the maximum size variable is optional. If no maximum
size is specified, then 4GB is used. If a file is being created and matches one of the MEMORY_FILE file name
entries, it is created as a memory file unless it is a superfile host, superfile member, mirrored, segmented, or
partitioned file.

To cause all possible files to be created as memory files, add the following configuration entry: MEMORY_FILE *

CHECKPOINT_FLUSH

100
Default value: 2

This keyword controls the aging of updated buffers based on the number of checkpoints that have occurred since
the buffer was last flushed. CHECKPOINT_FLUSH <num_chkpnts> sets the maximum number of checkpoints to be
written before a data or index cache page holding an image for a transaction controlled file is flushed. The default
value is 2. Increasing this value avoids repeated flushing of updated cache pages that can occur in a system that
maintains high transaction rates. When CHECKPOINT_FLUSH is increased, the c-tree server automatically detects
the reliance on previous transaction logs and increases the active log count as needed provided that the
FIXED_LOG_SIZE server configuration keyword is not enabled.

COMPATIBILITY TDATA_WRITETHRU COMPATIBILITY TINDEX_WRITETHRU

These keywords force transaction controlled data files and index files, respectively, to be written directly to
disk(whenever c-tree determines that they must be flushed from the c-tree buffers), and the calls to flush their OS
buffers are skipped.

COMMIT_DELAY (for sleep times expressed in milliseconds) COMMIT_DELAY_USEC (for sleep times expressed
in microseconds)

Default values: 10 milliseconds for COMMIT_DELAY 10 microseconds for COMMIT_DELAY_USEC

NOTE Keep these default values in mind when setting a time limit for aborting transactions.

The keywords control the length of time after a given transaction completes that the transaction manager waits
before flushing the transaction to disk. By waiting, more than one transaction (that is, the first one and all others that
complete before the delay period expires) may be committed at the same time, reducing disk-access overhead. On
average, the longer the delay, the greater the number of transactions committed.

Increasing the delay time increases the chances of losing data in a catastrophe (that is, power
NOTE
loss).

Examples:

COMMIT_DELAY <milliseconds | -1>

COMMIT_DELAY_USEC <microseconds | -1>

If both forms of the commit delay keyword are used, then the last entry in the configuration file prevails. One
millisecond is 1000 microsecond.

Not all systems support arbitrarily short sleep times. For example, FairCom has found that on
NOTE Solaris, unless using real-time capabilities of the operating system, the minimum achievable sleep
time is 10 milliseconds even if a shorter sleep time is requested.

LOG_TEMPLATE

Default value: 0

Enable the log template feature by specifying the server keyword LOG_TEMPLATE in the server configuration file.

101
Example:

LOG_TEMPLATE <n>

The variable <n> is the number of log templates you want the server to maintain. The default is 0,which means no
use of log templates. For instance, a value of two (2) means that two blank logs ([Link] and [Link])
are created at first server startup in addition to the template ([Link]).

Prior to using the log template feature, existing transaction logs must be deleted To cause the server to create log
templates. To do this, shut down the server cleanly and delete [Link], [Link], and [Link].
When the server is restarted after adding this keyword, startup may take longer due to creation of template log files
(*.FCT).

Limitations:

Log templates are not supported when mirrored logs or log encryption is in use.

Background on Creating c-tree Transaction Logs:

Information concerning ongoing transactions is saved on a continual basis in a transaction log file. A chronological
series of transaction log files is maintained during the operation of the c-tree server. Transaction log files containing
the actual transaction information are saved as standard files. They are provided names in sequential order, starting
with [Link] (which can be thought of as “active c-tree Server log, number 0000001”) and incrementing
sequentially (that is, the next log file is [Link], and so on.). By default, the c-tree server saves up to four
active logs at a given time.

A transaction log must be 0xff filled to ensure known contents. Also, the logs are extended and flushed to ensure log
space is available. Without the use of the log template feature, the 0xff filling of the log file (and forcing its directory
entries to disk) occurs during log write operations. This means that the log file is tied up during this fill/extension
processing, which can lead to increased latency for transactions that are in progress when log extension occurs.

With the transaction log template feature enabled, a new empty log template named [Link] is created at
server startup to serve as a log template. The first actual full size log, [Link], is copied from the template
and a blank log, [Link], is copied from the template. Whenever a new log is required, the corresponding
blank log file is renamed from [Link] to [Link] and, asynchronously, the next blank log,
[Link], is copied from the template.

This feature is designed only for use in very high speed, high throughput systems where it is desirable to be more
than one log ahead in case the template copy operation runs slowly.

DIAGNOSTICS TRAP_COMM

This keyword should be used primarily for debugging since its corresponding processing consumes
NOTE
additional overhead.

The DIAGNOSTICS TRAP_COMM keyword enables developers to “record” communication traffic coming in to a
server and play it back with the cttrap utility.

When the DIAGNOSTICS TRAP_COMM keyword is active, the trap file, [Link], is created in the server
directory by default. To prepend a path onto the trap file name (to route it to a separate disk or directory), add an
entry of the form DIAGNOSTIC_STR <trap file path>. For example, if DIAGNOSTIC_STR /bigdisk/ were in the
configuration file, then the trap file would be /bigdisk/[Link].

102
When the DIAGNOSTICS TRAP_COMM keyword is activated, this keyword instructs the Faircom server to log
incoming communications packets to [Link] prior to execution. This log can be played back using the
cttrap utility and a debug server to observe the results of the client requests, enabling the developer to exactly
duplicate and repeat client activities.

Syntax

DIAGNOSTICS TRAP_COMM

_cttrap_ is a multithreaded client application that “plays back” a TRAP_COMM log

file. Whenever a Multithreaded Client library is created, cttrap is also be
generated. The default TRAP_COMM file name is*[Link]*.

cttrap <TrapCommLogFileName> [ <ServerName> ]

FUNCTION_MONITOR

This keyword should be used primarily for debugging since its corresponding processing consumes
NOTE
additional overhead.

The FUNCTION_MONITOR keyword displays the client number, function number, function name, and file name in a
scrolling fashion on the c-tree server console window.

Alternatively, the same information, along with the return value and error codes for each function, can be routed to a
file by specifying a file name. Default value for this keyword is NO.

Syntax

FUNCTION_MONITOR <YES | NO | file_name>

Sample c-tree server configuration

With the transaction log template feature enabled, a new empty log template named [Link] is created at
server startup to serve as a log template.

SERVER_NAME < >

;Limit [Link] to 2 Mb and only keep one [Link] file
CTSTATUS_SIZE -2048000
;TCP/IP for communication protocol
COMM_PROTOCOL F_TCPIP
;Increase page buffer size
PAGE_SIZE 32768
;Encrypt the [Link] file to hide passwords etc.
ADMIN_ENCRYPT
;Disabel "Guest" Login's
GUEST_LOGON No
;Allow up to 3 failed login attempts
LOGON_FAIL_LIMIT 3
;After LOGON_FAIL_LIMIT times don’t allow logins for LOGON_FAIL_TIME in minutes.
(-1 is forever)
LOGON_FAIL_TIME 30
LOCAL_DIRECTORY /db01/b24dev/ctree/log/

103
;Number of simultaneous connections -

Tune this parameter based on license agreement.

CONNECTIONS 512
;Hash bins for file locks
LOCK_HASH 32
; Space for active transaction logs(Mb)
LOG_SPACE 240
;Desired checkpoint interval is every 1/3 of a transaction log.
;For 96 MB of log space and 4 active log files, this is 96MB/12=8MB.
CHECKPOINT_INTERVAL 20000000
CHECKPOINT_FLUSH 37
;Sun cache line of 32
CACHE_LINE 32
;Data and index cache -

Tune this parameter depending on memory availability.

DAT_MEMORY 2000000000
IDX_MEMORY 3500000000
BUFR_MEMORY 1048576
; Number of files opened -

No C-tree limit - except for any limits imposed by the available system memory.

FILES 2000
; To effectively disable Transaction Flush
TRANSACTION_FLUSH 10000
; Enhanced commit delay.
COMMIT_DELAY_USEC 500
; For more efficient cleanup of deleted index nodes.
NODEQ_SEARCH 400
; For pre-allocating files
LOG_TEMPLATE 2
; For memory-mapped context files
MEMORY_FILE /db01/b24data/ctx*dat
MEMORY_FILE /db01/b24data/ctx*idx
; fdsync() call
COMPATIBILITY FDATASYNC
; Enable directio() function
COMPATIBILITY SYNC_LOG
; To ensure use of only 6-byte tran files.
COMPATIBILITY EXTENDED_TRAN_ONLY
; By-pass file system caches for data and index file writes
COMPATIBILITY TDATA_WRITETHRU
COMPATIBILITY TINDEX_WRITETHRU
;For diagnostic log at server shutdown.
DIAGNOSTICS SNAPSHOT_SHUTDOWN

104
Section 13. Troubleshooting
This section presents several tools that should be useful in identifying and resolving problems with a BASE24-eps
system, including the following:

• Using HELP24
• Obtaining software version information from BASE24-eps JAR files
• Obtaining software version information from C++ programs and libraries
• Troubleshooting BASE24-eps on Linux/UNIX (Solaris, AIX, HPUX), including the following topics:
◦ Data backups
◦ Layout of the BASE24-eps software on a typical install
◦ Event logging
◦ Linux/UNIX commands that can be used outside of the BASE24-eps product for troubleshooting
◦ Diagnostic tools outside of the operating system
◦ Problem determination methodology
• Miscellaneous troubleshooting tips

ACI Worldwide HELP24

HELP24 and HELP24 InfoLink are excellent sources of assistance and information. The following access options
appear on the support page of the ACI Worldwide, Inc. Internet website ([Link]

• A current listing of HELP24 phones worldwide.

• An option to send an e-mail message to HELP24.
• An option to sign up for HELP24 InfoLink.
• An option to access HELP24 InfoLink once you have received a user ID and password.

eSupport access
Once you have your HELP24 eSupport logon and password, you can do the following:

• Access the HELP24 database to open a new Clarify case.

• Access installation and fix release procedures online.
• Access all BASE24, BASE24-eps, and ICE-XS documentation online.

The ESUPPORT Quick Reference Guide contains procedures for opening a new case, editing an existing case,
accessing documentation online, searching solution articles, and searching fix information. Once you log on to
eSupport, click the Quick Reference Guide link under Portal Help.

Once you log on to eSupport, the path to the installation and fix release procedures is Product Central > Product
Information > BASE24-eps 2.1. Once you have opened the library, search for "Linux/UNIX" or "Version" to access
the following documentation.

105
 Fix Procedures Procedures for updating a software release or version that you have already installed.

Install Guides Procedures for installing a software release or version for the first time.

Version Upgrades Procedures for upgrading BASE24 Transaction Security Services once you have
installed a BASE24-eps software release or version.

Telephone access
When accessing HELP24 by telephone, you may occasionally be greeted by a recorded message. If this occurs,
listen carefully to the complete message, select BASE24 as product and follow instructions. A support analyst will
promptly be in touch with you.

Customer responsibilities
To ensure effective use of ACI support resources, it is recommended that you initiate the following activities prior to
or soon after reporting an incident to ACI:

• Perform initial diagnosis to isolate incident area (for example, operational, environmental, custom software
modification, device handler, and so on).
• Gather supporting data, trace information, screen prints or dump files.
• Determine and report recent environment changes.
• Identify any custom code linked to the suspected problem.
• Eliminate any associated non-ACI software conflicts (for example, hardware, third-party vendor software or
platform issues, host based processing issues, and so on).

Information requirements
You are encouraged to report any incident related to the ACI supported environment to HELP24 as soon as
possible. We recommend that you perform the necessary incident determination steps specific to your environment
and provide the relevant incident descriptive information.

HELP24 requires the following information to record and begin analysis on the reported incident:

Platform The hardware platform on which the incident is occurring (for example, IBM AIX,
IBM CICS, HP, SUN, and so on).

Database The type of database (for example, CTREE, ORACLE, ENSCRIBE, DB2, and so
on).

Module ID The actual module affected by the incident (for example, VISA, SPDH, Integrated
Server (IS), XMLS, or other processes).

Release/ Version The release level of the product.

106
 Incident Name Your own brief description of the problem. Future communication references your
description and understanding of the incident.

Test/Prod/Certification The nature of the incident. This is helpful in assessing the priority.

Incident Description A detailed description of the incident and its symptoms.

Business Impact Assessment of impact on network, customers, transaction flow, and so on.

Customer Incident # The customer reference number associated with this incident (optional).

Customer severity
The following customer severity levels are available to help prioritize a problem when it is reported.

Urgent Issue needs attention right now.

High Issue needs attention today.

Medium Issue needs attention this week.

Low Issue needs attention some time in the future.

Priority scheme
The following priority scheme is used to prioritize a problem when it is reported.

• Critical (also known as priority 1)

◦ ACI provides 24 x 7 support for critical problems.
◦ Total system outage or serious degradation on a production system.
• Serious (also known as priority 2)
◦ Serious incident on production system. A majority of on-line transactions are being approved. However,
other required processing is impacted.
◦ Incident on test system that impacts a critical go-live date.
◦ Unable to complete final certification with an interchange.
◦ Significant impacts to the business relationship of our customer with their customer.
• Moderate (also known as priority 3)
• No apparent danger of major service interruption or minimal business impact.
◦ Infrequent occurrence with easy recovery.
◦ Informational (also known as priority 4)

107
 • HELP24 must perform an initial evaluation of requests of this type to determine the scope of any request. Once
this is understood, a target response date is set and communicated to the customer. These problems usually
require less effort to resolve and ACI attempts to provide a solution within four business days.

Before you call

• Search the solution database on the HELP24 website to see if a solution is already available.
• Create a case on the HELP24 website to obtain a case number. If Internet access is not available, send a
message to the HELP24 e-mail mailbox. The HELP24 analyst responds with a case number by e-mail.
• Attach any supporting documentation to the case (traces, log file, dump files or screen prints).
• Let the HELP24 analyst know how often you would like to receive follow-up calls on the issue.
• The HELP24 analyst performs first level diagnosis and additional ACI support staff may be called upon to assist.
When incidents are reported after hours, an ACI technical support analyst is paged immediately. We allocate
resources immediately for priority 1 issues. Priority 2, 3, and 4 incidents are handled on the next business day.
• The HELP24 analyst or software engineer working on the incident reports the status of an incident on the case
query function on the website.
• When the incident has been resolved, information about the resolution is logged to the ACI tracking database
and reported to you.

At times, your HELP24 analyst may ask you to determine the software version information.

Obtaining software version information from BASE24-eps

JAR files
This section describes how to determine the JAR versions for the BASE24-eps device handlers written in Java.

The BASE24-eps device handlers are written in the Java programming language. All Java code is delivered in JAR
(Java Archive) files. The JAR files have an associated manifest, which describes the contents of the JAR. By
examining the manifest, the version of the JAR file can be determined.

The reader should also review the Java JAR File specification freely available from SUN Microsystems. It is
available for download, with the SDK documentation, from the [Link] website.

JAR files
The BASE24-eps JAR files contain precompiled Java classes. The classes are arranged in a directory structure that
matches the class packages. The JAR file is actually a compressed archive file, commonly referred to as a ZIP file.

The BASE24-eps device handlers each comprise a set of JAR files. Each JAR file has version information
embedded within it. All device handlers use the following JAR files:

• [Link]
• [Link]
• [Link]
• [Link]

108
 • [Link]

The device-specific software, for each device type is isolated in a device-specific JAR file:

• [Link]
• [Link]

Optional features are distributed in additional JAR files:

• [Link]

Opening JAR files

JAR files can be opened using WinZip or any similar tool. The diagram below shows how the files can be opened,
and their contents. The procedure, on an NT platform, is as follows.

1. Locate the lib directory, where the software JAR files are located.
2. Right click any of the JAR files. A menu appears. Select Open With.
3. Select WinZip Executable as the program to open the JAR file (if the check box labeled Always Use This
Program To Open These Files is checked, uncheck it).

The image below depicts the opening process for the [Link] file, using WinZip on a Windows platform.
Similar tools can be used on other hardware platforms (for example, SUN). Optionally, a JAR file can be transferred
to an NT workstation and its contents examined there.

109
A JAR file opened in WinZip appears as shown in the following diagram.

110
Notice that the Path information matches the package names for each class. Java uses the structure in the JAR file
to locate the proper classes for execution.

The first file in the list is the [Link] file. This file contains a description of the JAR file contents. *The
[Link] file contains the version of the software being run. * The [Link] file can be opened from the JAR
file and its contents examined. Right clicking on the file name and selecting Open in the menu opens the file.

The window shows the opening process for [Link].

111
Click Open.

112
The text box below contains the manifest data from the [Link] file. A definition of the data in the
manifest follows.

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.5.3
Created-By: 1.3.1_04-b02 (Sun Microsystems Inc.)
Implementation-Title: Java Device Handler - Base24 Common
Implementation-Version: 04.1.0 - Build 20040305-1035
Implementation-Vendor: ACI Worldwide, Inc.

Manifest-Version: 1.0 is the version of the manifest for the JAR file, currently version 1.0.

Ant-Version: Apache Ant 1.5.3 is the version of the Ant system used to build the JAR files. Currently, ACI Worldwide
uses version 1.5.3 of Ant. Ant is a tool widely used to compile Java projects and generate JARs for delivery. It is
freely available from the Jakarta Apache group.

Created-By: 1.3.1_04-b02 (Sun Microsystems Inc.) is the version of the JVM in which the Ant task is running while
creating the JAR files.

113
Implementation-Title: Java Device Handler - Base24 Common is the name of the product (Java Device Handler) and
the name of the JAR file (Base24 Common) to which this manifest pertains.

Implementation-Version: 04.1.0 - Build 20040305-1035 is the version of the product (04.1.0) and the build version
(Build 20040305-1035) of the JAR file. Note that the build version is the date (2004/03/05 or March 5, 2004) and the
time (10:35) when the JAR file was built. This is the most important data in the manifest file. This identifies the
specific version of the code being run.

Implementation-Vendor: ACI Worldwide, Inc. is the name of the organization that created the software.

Obtaining software version information from BASE24-eps

C++ programs/libraries
This section describes how to determine the version of every C++ header and implementation source module that
was used to make a BASE24-eps executable program or library object.

BASE24-eps executable programs such as the Integrated Server and the End-Of-Period Process are written in C.
Every C header (.h) and implementation (.cpp) source component of these programs contains a text string that
identifies the exact version of the component that was used to build this particular version of the executable. These
version number strings are compiled into the final library or executable and can be displayed using the jwhat utility.

The jwhat utility is a Java program that displays the version strings embedded in a program/library object. It is similar
to the what program found on most Linux/UNIX systems.

Running jwhat
The jwhat utility is a Java program and requires that the Java runtime be executed from the command line.
Therefore, before running jwhat, make sure that the Java runtime is available. To verify this, type java at the
command line. A usage message should be displayed.

To run jwhat, type the command followed by the name of the executable or library object whose contents are to be
displayed. For example:

jwhat [Link]

The utility displays the version of each source module that was used to build the executable program or library
object. The module names are sorted into alphabetical order after the name of the object being displayed. The
output can either be displayed on the console or can be routed to a disk file.

Hint: Saving the output from jwhat to disk enables the contents of one version of an executable to be compared with
a later version to identify exactly which modules have been changed from one version to the next.

The following is an extract from the output produced by running jwhat with the name of the [Link] program as the
parameter:

114
 [Link]:
%full_filespec: Sitdm.h,2:incl:1 %
%full_filespec: [Link],5:c++:1 %
%full_filespec: [Link],4:c++:1 %
%full_filespec: acct1.h,3:incl:1 %
%full_filespec: [Link],3:c++:1 %
%full_filespec: acct2.h,3:incl:1 %
%full_filespec: [Link],3:c++:1 %
%full_filespec: acct3.h,3:incl:1 %
%full_filespec: [Link],7:c++:1 %
%full_filespec: acctb.h,4:incl:1 %
%full_filespec: [Link],8:c++:1 %
%full_filespec: acctky.h,11:incl:1 %
%full_filespec: acctnum.h,3:incl:1 %
%full_filespec: accttyp.h,3:incl:1 %
…

Rebuilding BASE24-eps executables

BASE24-eps is installed with a script in the $BIN location, called runlink, that executes the $MWORK/[Link]
makefile. You can execute runlink as needed to rebuild the BASE24-eps executables.

Runlink requires a tag parameter for execution. You can specify the IS tag to recreate the $LIB/[Link] only.

Example: runlink IS

Or, you can specify the ALL tag to recreate all executables.

Example: runlink ALL

Rebuilding the $MWORK/[Link] makefile

If necessary, the acilink script can be executed to recreate the $MWORK/[Link] makefile. The acilink script
rebuilds the makefile based on the BASE24-eps component objects that have been installed.

Following the rebuilding of the makefile, the runlink script must be executed to actually perform the make.

Troubleshooting BASE24-eps on Linux/UNIX (Solaris_ AIX_

HPUX)
This section provides information on the following troubleshooting topics.

• Data backups
• Layout of the BASE24-eps software on a typical install
• Event logging
• Linux/UNIX commands that can be used outside of the BASE24-eps product for troubleshooting
• Diagnostic tools outside of the operating system
• Problem determination methodology

115
Data backups
ACI recommends performing a full backup of BASE24-eps data following a new installation and following each
version upgrade. We also recommend some type of incremental daily backups.

The c-tree server provides several configuration options for performing incremental or full backups. See the c-tree
Server Administrator’s Guide on the Faircom website.

Threading model of BASE24-eps applications on Linux/UNIX platforms

The BASE24-eps applications on the Linux/UNIX platform are multithreaded. Each process has the following
threads:

• The main thread is the primary thread that starts when the application is invoked.
• The signal handling thread handles normal termination of the application when it is shutdown or when error
conditions force the application to terminate. In the latter case, debug information in the form of process stacks
and core files are generated by this thread prior to the application’s termination.
• The application thread handles all processing and business logic for the application.
• The message thread reads messages from the application input queue and forwards them to the application
thread for processing.
• The command thread reads messages from the application’s command queue and forwards them to the
application thread for processing.

For each numthread parameter specified in the application command line, an application thread, a message thread,
and a command thread are spawned in addition to the main thread and the signal handling thread. For example, if
numthreads=1 is specified in the application command line, there are five threads (Light Weight Processes or LWPs)
running in the process space: 1 + 1 + (1 x 3) = 5. If numthreads=5 is specified in the application command line, 17
threads occupy the process space: 1 + 1 + (5 x 3) = 17.

Layout of the BASE24-eps software on the platform

This section introduces the layout of the BASE24-eps software on a typical install.

Typical layout

A typical layout of the BASE24-eps package on any Linux/UNIX system has a few base sets of directories where the
software, corresponding configurations, some shell scripts and so on reside.

In a BASE24-eps 04.1 system these may look like the following:

$ ls
Server bin ctree logs mwork scripts
b24data config ice-xs mq_config output

In a BASE24-eps 04.4 system these may look like these:

116
 $ ls
SRC_ST44 comm db jsf logs queuing
bin config esdh lib mwork

Looking at the env_vars file is a good start to understanding the system and familiarizing yourself with the layout.

• In a 04.1 installation this file exists in the config directory.

• In a 04.4 installation this file exists in the bin directory.

To start exploring the directory structure, source the env_vars file as follows:

$ . <path to the env_vars file/>env_vars

This sets up your environment and makes it easier for you to navigate around the installation.

Env_vars file

If you open up the env_vars file, you would see information such as follows. This example may not match your
system exactly, but there should be enough commonality for you to navigate the system.

This file sets up environment variables that are used by the run scripts in $BIN to start up and shut down the system.

These also have definitions for standard message and command queues for the different processes.

One variable of note here that is site dependent is SIS_TZ. This is a posix form of the TZ variable setup in your
system. This is used by most of the BASE24-eps application processes with the exception of ICE-XS on the
Linux/UNIX system to determine time zone offsets.

Another useful variable of note is FCSRVR_CFG that points to the location of the c-tree server config file that is
being used on your system.

Another useful variable here is $LOG that defines where you would find most of your BASE24-eps application
generated event files. More on this in a later section.

Admf file

BASE24-eps applications use station names to describe and recognize their end points.

The mapping between the station names and the actual MQ queue names is provided in the admf file.

This file usually has four fields per record. It looks like this:

IS AS1QUEUE H 01
PCTL PCTLEVT H 01

The first field is the station name. The second field is the queue name. The third field determines the type of station
and messaging. The fourth field is reserved.

Valid values for the third field are as follows:

117
 H This station takes messages with an MDS header.

N Keeps original MQ header, and indicates that the MQ object is a namelist. Suppresses IBM MQ event
logging.

R MQ is used to communicate with an external entity that is not a BASE24-eps application.

Metadata files

In $DATA there are three files that provide metadata descriptions for the c-tree tables used by the product. These
files should not be hand modified. It is best to always go through MetaMan and generate new files for any metadata
changes that you may need. The files are [Link], [Link] and [Link].

[Link] file

Typically this file resides in the c-tree directory of the standard layout described above. In addition to the [Link]
file in this directory, when you have c-tree running you should have other files with the extension .FCS and .FCT.

The only .FCS file in this directory that has readable text is [Link].

You should be able to read through the entries in this file and this should give you some broad perspective of the
status of the c-tree server program running on this machine.

If you see a [Link] file, then this file has settings that override any settings or modifications that you may make
to your [Link] file.

The settings in this directory should be modified only by someone with c-tree DBA privilege in the system and should
typically be done in consultation with ACI and/or Faircom support. Look through the Faircom documentation for
further details.

MQ files and scripts

In the mq_config directory, you would find config files and scripts as shown below.

118
 $ ls
[Link]
chstatloop
configure_devenv_as1_manager.mqsc
configure_devenv_as1_manager.sh
configure_devenv_as1_racal.mqsc
create_mq_managers.sh
ipcrm_clean

[Link]

restart_mq_managers.sh

start_mq_managers.sh
stop_mq_managers.sh
[Link]
view_channel_status.mqsc
view_channel_status.sh
view_cluster_as1_queues.mqsc
view_cluster_queues.sh
vqdloop

These files are provided by ACI for setting up and configuring your messaging system. Your particular site may have
a MQ administrator who may follow different methods to set up the queue managers and queues. Then the files
above should provide you with a reference for how to setup and configure queues and should be guidelines for
setting up the MQ system.

The .ini files have tuning parameters in them and need to be in place prior to creating the queue managers and
queues. Any changes to these files would require that you delete and recreate your queue managers and queues.
Therefore refrain from making too many changes to these files and consult your MQ administrator or ACI support
prior to making any changes.

The vqdloop script in this directory provides a way to monitor queue depths. This is provided as a sample and can
be used as a reference to create other scripts that monitor queues of specific interest and defined periodicity.

MQ directory

After installation, the following files and scripts are placed in the mq directory.

[pkodev PKO4]:-> ls -ltar

total 148
-rwxr-xr-x. 1 pkodev b24dev 1024 Nov 12 06:09 view_qmodel_depth.sh
-rwxr-xr-x. 1 pkodev b24dev 421 Nov 12 06:09 view_clstr_queue_depth.sh
-rwxr-xr-x. 1 pkodev b24dev 402 Nov 12 06:09 view_channel_status.sh
-rwxr-xr-x. 1 pkodev b24dev 20 Nov 12 06:09 view_channel_status.mqsc
-rwxr-xr-x. 1 pkodev b24dev 34 Nov 12 06:09 [Link]
-rwxr-xr-x. 1 pkodev b24dev 101 Nov 12 06:09 [Link]
-rwxr-xr-x. 1 pkodev b24dev 2186 Nov 12 06:09 cfg_svcloc.mqsc
-rwxr-xr-x. 1 pkodev b24dev 597 Nov 12 06:09 cfg_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 5953 Nov 12 06:09 cfg_interface.mqsc
-rwxr-xr-x. 1 pkodev b24dev 517 Nov 12 06:09 cfg_clstr_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 1153 Nov 12 06:09 view_queue_depth.sh
-rw-r--r--. 1 pkodev b24dev 44 Nov 12 06:09 [Link]
-rw-r--r--. 1 pkodev b24dev 5116 Nov 12 06:09 cfg_manager_BSI.mqsc

119
-rw-r--r--. 1 pkodev b24dev 24 Nov 12 06:09 view_qmodel_depth.mqsc
-rw-r--r--. 1 pkodev b24dev 33 Nov 12 06:09 view_queue_depth.mqsc
-rw-r--r--. 1 pkodev b24dev 657 Nov 12 06:09 def_namelst.mqsc
-rw-r--r--. 1 pkodev b24dev 3160 Nov 12 06:09 cfg_ent.mqsc
-rw-r--r--. 1 pkodev b24dev 23370 Nov 12 06:09 cfg_clstr_manager.mqsc
drwxr-xr-x. 4 pkodev b24dev 36 Nov 12 06:10 ..
-rw-r--r--. 1 pkodev b24dev 7837 Nov 23 10:08 cfg_context.mqsc
-rw-r--r--. 1 pkodev b24dev 33334 Nov 23 10:08 cfg_manager.mqsc
drwxr-xr-x. 2 pkodev b24dev 4096 Dec 17 07:54 .

cd $BIN
[pkodev PKO4]:-> ls -lta
total 832
drwxr-xr-x. 36 pkodev b24dev 4096 Feb 4 02:49 ..
drwxr-xr-x. 4 pkodev b24dev 4096 Feb 3 06:21 .
-rwxr-xr-x. 1 pkodev b24dev 3108 Feb 2 02:06 rundciu
drwxr-xr-x. 2 pkodev b24dev 4096 Feb 2 01:51 linkaci
-rwxr--r--. 1 pkodev b24dev 906 Feb 1 10:02 runjmu
-rwxrwxrwx. 1 pkodev b24dev 234 Jan 15 06:50 cleartrc
-rwxrwxrwx. 1 pkodev b24dev 32 Jan 15 06:11 prep_reg
-rwxrwxrwx. 1 pkodev b24dev 184 Jan 15 06:04 obeydal
-rwxr--r--. 1 pkodev b24dev 906 Jan 4 09:35 runjm
-rwxr--r--. 1 pkodev b24dev 2762 Dec 17 07:50 EvtAdapter_env_vars
-rwxr--r--. 1 pkodev b24dev 3382 Dec 17 07:50 runtdal
-rwxr--r--. 1 pkodev b24dev 74 Dec 17 07:50 [Link]
-rwxr--r--. 1 pkodev b24dev 21679 Dec 8 02:43 env_vars
-rwxr-xr-x. 1 pkodev b24dev 22789 Dec 1 09:22 jmx_eps_reload
-rwxr--r--. 1 pkodev b24dev 90 Nov 12 06:09 icexs_env_vars
-rwxr-xr-x. 1 pkodev b24dev 2557 Nov 12 06:09 stopicexs
drwxr-xr-x. 2 pkodev b24dev 8192 Nov 12 06:09 linksvt
-rwxr-xr-x. 1 pkodev b24dev 5240 Oct 27 08:55 runicexs
-rwxr-xr-x. 1 pkodev b24dev 9503 Oct 15 08:50 eshaimpo
-rwxr-xr-x. 1 pkodev b24dev 8724 Oct 15 08:50 eshacopy
-rwxr-xr-x. 1 pkodev b24dev 1456 Sep 14 08:16 stoptdal
-rwxr-xr-x. 1 pkodev b24dev 1429 Sep 2 11:01 acilinit
-rwxr-xr-x. 1 pkodev b24dev 1459 Sep 2 11:01 runatbm
-rwxr-xr-x. 1 pkodev b24dev 1556 Sep 2 11:01 runeopp
-rwxr-xr-x. 1 pkodev b24dev 1497 Sep 2 11:01 runjqbtch
-rwxr-xr-x. 1 pkodev b24dev 1531 Sep 2 11:01 runrfsh
-rwxr-xr-x. 1 pkodev b24dev 1535 Sep 2 11:01 runsafm
-rwxr-xr-x. 1 pkodev b24dev 1496 Sep 2 11:01 runtbp
-rwxr-xr-x. 1 pkodev b24dev 1650 Sep 2 11:01 runttlm
-rwxr-xr-x. 1 pkodev b24dev 15399 Sep 2 11:01 [Link]
-rwxr-xr-x. 1 pkodev b24dev 1532 Sep 2 11:01 runbaut
-rwxr-xr-x. 1 pkodev b24dev 4634 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 4611 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 7229 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 5165 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 1462 Aug 10 02:47 runtssv
-rwxr-xr-x. 1 pkodev b24dev 14359 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 1274 Aug 10 02:47 create_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 2428 Aug 10 02:47 dalci
-rwxr-xr-x. 1 pkodev b24dev 24927 Aug 10 02:47 esbldjnl
-rwxr-xr-x. 1 pkodev b24dev 35009 Aug 10 02:47 eschgprc
-rwxr-xr-x. 1 pkodev b24dev 20596 Aug 10 02:47 eshelper
-rwxr-xr-x. 1 pkodev b24dev 38510 Aug 10 02:47 esinfo
-rwxr-xr-x. 1 pkodev b24dev 34471 Aug 10 02:47 esrevert
-rwxr-xr-x. 1 pkodev b24dev 3687 Aug 10 02:47 essecure
-rwxr-xr-x. 1 pkodev b24dev 12245 Aug 10 02:47 esstatus
-rwxr-xr-x. 1 pkodev b24dev 3651 Aug 10 02:47 esunsec

120
-rwxr-xr-x. 1 pkodev b24dev 4100 Aug 10 02:47 jmx_add_nof
-rwxr-xr-x. 1 pkodev b24dev 453 Aug 10 02:47 restart_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 1617 Aug 10 02:47 runcnfg
-rwxr-xr-x. 1 pkodev b24dev 1430 Aug 10 02:47 runep
-rwxr-xr-x. 1 pkodev b24dev 1675 Aug 10 02:47 runis
-rwxr-xr-x. 1 pkodev b24dev 10344 Aug 10 02:47 runmeta
-rwxr-xr-x. 1 pkodev b24dev 9051 Aug 10 02:47 runscptcc
-rwxr-xr-x. 1 pkodev b24dev 2003 Aug 10 02:47 startesweb
-rwxr-xr-x. 1 pkodev b24dev 807 Aug 10 02:47 start_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 1337 Aug 10 02:47 stopcmcp
-rwxr-xr-x. 1 pkodev b24dev 1361 Aug 10 02:47 stopcnfg
-rwxr-xr-x. 1 pkodev b24dev 1337 Aug 10 02:47 stopeopp
-rwxr-xr-x. 1 pkodev b24dev 1234 Aug 10 02:47 stopep
-rwxr-xr-x. 1 pkodev b24dev 1928 Aug 10 02:47 stopevtadapter
-rwxr-xr-x. 1 pkodev b24dev 1482 Aug 10 02:47 stopevtlog
-rwxr-xr-x. 1 pkodev b24dev 469 Aug 10 02:47 stopfpid
-rwxr-xr-x. 1 pkodev b24dev 846 Aug 10 02:47 stopifx
-rwxr-xr-x. 1 pkodev b24dev 1486 Aug 10 02:47 stopis
-rwxr-xr-x. 1 pkodev b24dev 836 Aug 10 02:47 stopjdh
-rwxr-xr-x. 1 pkodev b24dev 1343 Aug 10 02:47 stopjqbtch
-rwxr-xr-x. 1 pkodev b24dev 748 Aug 10 02:47 stopjtimer
-rwxr-xr-x. 1 pkodev b24dev 633 Aug 10 02:47 stop_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 459 Aug 10 02:47 stopqm
-rwxr-xr-x. 1 pkodev b24dev 1337 Aug 10 02:47 stoprfsh
-rwxr-xr-x. 1 pkodev b24dev 1337 Aug 10 02:47 stopsafm
-rwxr-xr-x. 1 pkodev b24dev 1571 Aug 10 02:47 stopsvcloc
-rwxr-xr-x. 1 pkodev b24dev 1327 Aug 10 02:47 stoptbp
-rwxr-xr-x. 1 pkodev b24dev 1340 Aug 10 02:47 stoptimrp
-rwxr-xr-x. 1 pkodev b24dev 1221 Aug 10 02:47 stoptssv
-rwxr-xr-x. 1 pkodev b24dev 1667 Aug 10 02:47 stopttlm
-rwxr-xr-x. 1 pkodev b24dev 1332 Aug 10 02:47 stopxml
-rwxr-xr-x. 1 pkodev b24dev 1056 Aug 10 02:47 alist
-rwxr-xr-x. 1 pkodev b24dev 735 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 324 Aug 10 02:47 bldqscripts
-rwxr-xr-x. 1 pkodev b24dev 1199 Aug 10 02:47 create_clstr_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 564 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 130 Aug 10 02:47 esstatusl
-rwxr-xr-x. 1 pkodev b24dev 571 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 491 Aug 10 02:47 loadoltp
-rwxr-xr-x. 1 pkodev b24dev 354 Aug 10 02:47 restart_clstr_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 1499 Aug 10 02:47 runaacr
-rwxr-xr-x. 1 pkodev b24dev 1636 Aug 10 02:47 runap18bfi
-rwxr-xr-x. 1 pkodev b24dev 1514 Aug 10 02:47 runcmcp
-rwxr-xr-x. 1 pkodev b24dev 2986 Aug 10 02:47 runctbld
-rwxr-xr-x. 1 pkodev b24dev 3130 Aug 10 02:47 rundalci
-rwxr-xr-x. 1 pkodev b24dev 443 Aug 10 02:47 rundaliscd
-rwxr-xr-x. 1 pkodev b24dev 3351 Aug 10 02:47 runemtbld
-rwxr-xr-x. 1 pkodev b24dev 639 Aug 10 02:47 runevtadapter
-rwxr-xr-x. 1 pkodev b24dev 840 Aug 10 02:47 runevtlog
-rwxr-xr-x. 1 pkodev b24dev 698 Aug 10 02:47 runfpid
-rwxr-xr-x. 1 pkodev b24dev 921 Aug 10 02:47 runfpsb
-rwxr-xr-x. 1 pkodev b24dev 498 Aug 10 02:47 runfpsbv
-rwxr-xr-x. 1 pkodev b24dev 498 Aug 10 02:47 runfpsbw
-rwxr-xr-x. 1 pkodev b24dev 750 Aug 10 02:47 runjlfscan
-rwxr-xr-x. 1 pkodev b24dev 1514 Aug 10 02:47 runkldr
-rwxr-xr-x. 1 pkodev b24dev 508 Aug 10 02:47 runlink
-rwxr-xr-x. 1 pkodev b24dev 681 Aug 10 02:47 runqm
-rwxr-xr-x. 1 pkodev b24dev 1533 Aug 10 02:47 runsvcloc
-rwxr-xr-x. 1 pkodev b24dev 1492 Aug 10 02:47 runtimrp
-rwxr-xr-x. 1 pkodev b24dev 847 Aug 10 02:47 runtrcv
-rwxr-xr-x. 1 pkodev b24dev 1556 Aug 10 02:47 runxml

121
 -rwxr-xr-x. 1 pkodev b24dev 546 Aug 10 02:47 start_clstr_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 1153 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 382 Aug 10 02:47 start_system
-rwxr-xr-x. 1 pkodev b24dev 1553 Aug 10 02:47 stopaacr
-rwxr-xr-x. 1 pkodev b24dev 1338 Aug 10 02:47 stopatbm
-rwxr-xr-x. 1 pkodev b24dev 1559 Aug 10 02:47 stopbaut
-rwxr-xr-x. 1 pkodev b24dev 438 Aug 10 02:47 stop_clstr_manager.sh
-rwxr-xr-x. 1 pkodev b24dev 467 Aug 10 02:47 stopesweb
-rwxr-xr-x. 1 pkodev b24dev 577 Aug 10 02:47 stopfpsb
-rwxr-xr-x. 1 pkodev b24dev 316 Aug 10 02:47 vcqd
-rwxr-xr-x. 1 pkodev b24dev 381 Aug 10 02:47 vqd
-rwxr-xr-x. 1 pkodev b24dev 26758 Aug 10 02:47 acilink
-rwxr-xr-x. 1 pkodev b24dev 83 Aug 10 02:47 chstatloop
-rwxr-xr-x. 1 pkodev b24dev 104 Aug 10 02:47 net
-rwxr-xr-x. 1 pkodev b24dev 286 Aug 10 02:47 runevtparser
-rwxr-xr-x. 1 pkodev b24dev 450 Aug 10 02:47 [Link]
-rwxr-xr-x. 1 pkodev b24dev 313 Aug 10 02:47 vchd
-rwxr-xr-x. 1 pkodev b24dev 83 Aug 10 02:47 vchdloop
-rwxr-xr-x. 1 pkodev b24dev 86 Aug 10 02:47 vcqdloop
-rwxr-xr-x. 1 pkodev b24dev 80 Aug 10 02:47 vqdloop
-rwxr-xr-x. 1 pkodev b24dev 1659 Aug 10 02:44 sis_diag_abort.sh
-rwxr-xr-x. 1 pkodev b24dev 1063 Aug 10 02:44 sis_diag_msgtimeout.sh
-rwxr-xr-x. 1 pkodev b24dev 1709 Aug 10 02:44 sis_packcore.sh

Location of the c-tree database

If you are integrating BASE24-eps USEC with a third-party authentication and authorization engine
(for example, an LDAP Active Directory) for single sign-on authentication and authorization, you
must load or replicate the existing USEC configuration to the third party product and subsequently
NOTE
perform all user security configuration maintenance and operations using the third party product
rather than using USEC and the ACI desktop user security windows. For more information, refer to
the BASE24-eps Java Server Reference Guide.

If you open up your [Link] or your [Link] you would be able to figure out the location of your c-tree database files.

An entry looks like the following in [Link]:

"ACQUIRER_ISSUER_RELATION","Acquirer_Issuer_Relation","1.0","USDVMS01@[Link]:
/fsd/data/harnessdata/aird","ctree","",""

This entry reads as follows: Assign name ACQUIRER_ISSUER_RELATION has a table name
Acquirer_Issuer_Relation, is of version 1.0, exists on a c-tree server of name USDVMS01, which is on a server with
IP address [Link], and is located on that server at the location /fsd/data/harnessdata/aird. This is a c-tree
file.

An entry looks like the following in [Link]:

"ACQUIRER_TXN_ALLOWED_OLTP","Acquirer_Txn_Allowed_OLTP","1.0","/fsd/ctree/data/harnes
sdata/aqtxod","HASHDS","-MT_SIZE=500000","ACQUIRER_TXN_ALLOWED"

This entry reads similarly except that this is a Hash data source and loaded up as a hash table in memory. Hash
tables and Radix tables are typically read-only and in memory. Typically they represent a memory image of a c-tree
data file that exists on disk. In this case this is ACQUIRER_TXN_ALLOWED.

The file /fsd/ctree/data/harnessdata/aqtxod represents a disk image of the memory file. All OLTP files (that is, Hash

122
or Radix files) typically end with the suffix od. The tables typically have a .dat extension. Depending on how many
indexes that have been defined for the table, you are likely to see one or more .idx files.

The [Link] should give you some idea of the number of indexes per table and their broad layout. [Link] defines the
actual table structure.

When you do a load oltp in DALCI or use the UI to alter a table and load it, there are two operations that happen.
The deliver command to load the OLTP file generates a …*od file on disk and the alter command instructs the
application threads to refresh the image of this file that they have in their local memory.

Memory table sizing

The MT_SIZE variable for assigns in the CONFCSV ([Link]) represents the initial memory size that you are
reserving per IS process (on HP NonStop platforms) or per thread (on Unix platforms) for this table. The Data
Source Loader component dynamically resizes the table until it’s large enough for all the data if the initial value is too
small. If the initial value is too small, the Data Source Loader component increases the MT_SIZE by 100,000 bytes
or by 10%, whichever is larger. This repeats until there are no more memory errors due to too small of a memory
size.

The Data Source Loader component does not update the MT_SIZE value in the [Link]. The actual size used is
written to disk when the OLTP is successfully built. You can see the actual table size used on the processing
summary report with a label of “Table Size”. You can optionally reset the MT_SIZE in the confcsv close to this size
and if you see messages that the size is increasing over time. This will reduce the number of times to attempt to
rebuild the OLTP for subsequent loads, but the load will only fail if there is not enough memory in the system to do
the build.

Location of the ICE-XS configuration files

Typically in your installation you have a comm or an ICE-XS directory. This directory holds all of the ICE-XS
configuration files. In this directory, you see a set of files with a .nof extension. These files have the details of the
ICE-XS configuration. Look at your ICE-XS documentation for further details.

The files with the .param extension are internal configuration files for ICE-XS and define the location of your license
file and a maximum memory size for ICE-XS.

These files are typically set up once and not touched subsequently. If you need to make any changes here, consult
with your network administrator (for ip/port config changes), your application support personnel (for any changes to
station configurations) and support personnel from ACI or Insession.

Where do events get logged?

Most BASE24-eps applications log events in files in the $LOG directory as set up in the env_vars file.

Events logged by the applications are typically in the form of an xml message in the Event queue. The Event queue
is typically set up using evtdest setup in the environment table. This Event Destination is typically an MQ queue. The
env_vars file can also have a default event queue set up. If both these options are set up incorrectly, the application
can, by default, put event messages to stdout and stderr.

The run scripts in the $BIN directory can redirect the messages coming to stdout and stderr to specified log files.
Look at the specific run file to determine the location of these files.

ICE-XS logs errors to the syslog.d file. This is typically /var/adm/messages on Solaris and AIX and

123
/var/adm/syslog/[Link] on HP-UX.

c-tree typically logs its errors in the [Link] file.

ACI highly recommends that customers attach these event files along with any problem report that is opened with
HELP24.

Where do core files get generated?

In general, keep a watch on core files generated by any BASE24-eps process. With the core naming feature added
as part of the 04.4 version, core files generated by any BASE24-eps process are found in a standard $LOGS
directory. As part of this feature, core files are renamed with a name that indicates the name and process ID of the
process generating the core file.

A special Event Completion Timeout generates an additional set of process stack and core files. In the command
line parameters for BASE24-eps, specify an event completion timeout in the evtcmplto parameter. This value is a
measure of time in seconds.

The threading model of a BASE24-eps application is described earlier in this section. When the message thread
reads a message and forwards it to the application thread for processing, the evtcmplto parameter specifies the
maximum amount of time the message thread waits for the application thread to finish processing the message and
for control to return to the message thread so it can pick up the next message from the input queue. If this timeout
expires, the message thread determines something abnormal is happening in the application thread and signals the
process to terminate abnormally. A set of process stack files are generated prior to sending this signal, To take a
current snapshot of the process. Subsequently, the abnormal termination also causes the signal handling thread to
take a snapshot of the process prior to process termination.

These two sets of files can provide two different views of the process and are valuable in helping determine the
issue. The files also follow core naming conventions and include the word “timeout” in their names, differentiating
them from files generated through process termination.

As part of the procedures to open a support case with ACI, execute the following script to package BASE24-eps-
generated core files before attaching it to the case:

sis_packcore.sh <program> <core file> <destination directory>

Where:

<program> Name of the process that generated the core file.

<core file> Name of the core file that was generated

<destination directory> Directory where you want the package to be generated (optional)

This script packages the core file along with other system libraries that enable ACI support staff to debug the
problem efficiently. It produces a single file name packcore_<date>.tar.Z (on Solaris and HP-UX ) , packcore_.[Link]
on RHEL , or snapcore_<pid>.pax.Z (on AIX ) . ACI highly recommends that customers attach these files along with
any problem report that is opened with the HELP24.

If there is a critical c-tree error, there is also a pstack file and a core file. Sending these files to ACI assists in a

124
quicker resolution of the issue.

AIX considerations

It is recommended that AIX systems are configured with the system attribute fullcore set to true. This enables
complete (full) core files to be produced when a process crashes. When the system attribute is set to false , core
files generated do not contain data sections. This limits problem diagnosis. The only impact of such a change is that
it takes a slightly longer time to terminate the process as the large core file is written.

The system attribute fullcore can be set to true via the following shell commands:

$ lsattr -El sys0 -a fullcore

fullcore false Enable full CORE dump True
$ sudo chdev -l sys0 -a fullcore=true
sys0 changed
$ lsattr -El sys0 -a fullcore
fullcore true Enable full CORE dump True

Linux/UNIX commands that can be used outside of the BASE24-eps product for
troubleshooting
There are a fairly large number of useful Linux/UNIX commands and many good books out in the market that can
point you in the right direction. The ones mentioned here are to give you a running start if you need one.

Man command

Most Linux/UNIX systems come with man pages installed for most if not all of these commands. These should give
you a bit more information on the command.

Invocation is as follows:

$ man <cmd_name>

If you are not sure what the command name is but know a keyword that may help, use the following command:

$ man - k <key_word>

To get more information on man and its options:

$ man man

Ps command

Running the ps - ef command may give you information that looks like this.

$ps -ef
UID PID PPID C STIME TTY TIME CMD
root 0 0 0 Jun 16 ? 0:04 sched
root 1 0 0 Jun 16 ? 91:39 /etc/init -

125
The UID is the person who started the process, the PID is the process ID of the process, and the PPID is the
process ID of the parent process. If you invoke the command from a shell, the PID has the process ID of the
command and the PPID has the process ID of the shell. The TIME indicates the CPU time that the process has had
since it was started and CMD is typically a brief window into the command name and maybe a few of the parameters
that it was started with.

Look through the man page of ps for more information.

Top or topas command

On Solaris and on HP-UX, the system administrator may have already installed or may be amenable to install a
version of a program called top. A similar program called topas is available on the AIX platform. The output of the top
program looks as below with some explanations. There is a lot of data presented in the output.

$/usr/local/bin/top
last pid: 9162; load averages: 1.74, 2.27, 3.10
755 processes: 728 sleeping, 16 zombie, 8 stopped, 3 on cpu
CPU states: 68.8% idle, 26.0% user, 4.8% kernel, 0.3% iowait, 0.0% swap
Memory: 32G real, 8797M free, 19G swap in use, 8771M swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
19593 uitid9 1 0 0 18M 16M cpu/11 159:09 12.48% workshop

The output shows information of how many processes are currently running on the system, how many active and
how many sleeping. A Zombie process is a special kind of a process typically in a hung or not very useful state.
Here, the parent process has been killed and the process is now owned by root and may show up as <defunct>.
These are best killed and cleaned up so that they do not use system resources.

There is information about CPU utilization by user, kernel, iowait and so on. High iowait times means that the CPU is
not doing much and waiting for work. You typically should not be running systems which are over 85-90% busy.

There is information about memory in the system, real memory and swap space and how much of this is used. You
typically do not want to see too much thrashing in swap space because that is a good indication for tuning or
upgrading the system.

PID is process id. Username is self explanatory. THR is the thread ID of the thread running. Typically we do not look
at PRI for priority or NICE for nice level which are methods of specifying process priority. SIZE and RES are memory
sizes of the application. These are maximum sizes that the application ever reached and would typically not indicate
the fact if you allocate huge chunks of memory and release them. Obviously if you see these numbers grow over a
period of time, that is an indication of a memory leak.

CPU state tells you which processor that the application is currently running on. This is typically not very useful
information unless you have created processor sets and bound your application. Time is the time that that the
process has had on the CPU since it was started. CPU % gives you a % of CPU utilization across the entire system.

Look through the man page of top for more information.

Netstat command

The netstat command is a method of determining the state of a socket. The following is a sample of what you see as
output from netstat.

126
 $netstat - a
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp 0 0 localhost.5524 localhost.53537 ESTABLISHED
tcp 0 0 oma3h001.52170 oma3h001.15037 ESTABLISHED
tcp 0 0 oma3h001.5571 oma3h002.58192 ESTABLISHED
tcp 0 0 *.24131 *.* LISTEN

The output contains the following information. In each case we are using a tcp protocol. Typically Recv-Q and
Send-Q should be zero. Numbers here indicate an issue with the network connection. The Local Address and
Foreign Address indicate the hostname or IP address of the local/remote box and the socket on which the
connection is established.

The last field in each of the entries indicates the state of the connection. Listen indicates that the tcp server is waiting
for a connection. The last entry in the output indicates that there is a tcp listener listening on socket 24131 on the
local machine and accepts connections from anyone (.). Established indicates that the connection has been
established. A TIME_WAIT indicates that the connection has been terminated abruptly and the socket is in the
process of terminating.

Look through the man page of netstat for more information.

Ndd command

The ndd command is used to get and set the parameters of any device. You can use get with no harm. Do not use
set without checking with the system administrator and/or consulting with ACI support.

Typically you use ndd to figure out information on the tcpip driver, as shown in the following sample.

$ ndd /dev/tcp \?
? (read only)
tcp_time_wait_interval (read and write)
tcp_conn_req_max_q (read and write)
tcp_conn_req_max_q0 (read and write)
tcp_conn_req_min (read and write)
tcp_conn_grace_period (read and write)
tcp_cwnd_max (read and write)
tcp_debug (read and write)
tcp_smallest_nonpriv_port (read and write)
tcp_ip_abort_cinterval (read and write)
tcp_ip_abort_linterval (read and write)
tcp_ip_abort_interval (read and write)
tcp_ip_notify_cinterval (read and write)
tcp_ip_notify_interval (read and write)
tcp_ipv4_ttl (read and write)
tcp_keepalive_interval (read and write)
tcp_maxpsz_multiplier (read and write)
tcp_mss_def_ipv4 (read and write)
tcp_mss_max_ipv4 (read and write)
tcp_mss_min (read and write)
tcp_naglim_def (read and write)
tcp_rexmit_interval_initial (read and write)
tcp_rexmit_interval_max (read and write)
tcp_rexmit_interval_min (read and write)
tcp_deferred_ack_interval (read and write)
tcp_snd_lowat_fraction (read and write)

127
 tcp_sth_rcv_hiwat (read and write)
tcp_sth_rcv_lowat (read and write)
tcp_dupack_fast_retransmit (read and write)
tcp_ignore_path_mtu (read and write)
tcp_rcv_push_wait (read and write)
tcp_smallest_anon_port (read and write)
tcp_largest_anon_port (read and write)
tcp_xmit_hiwat (read and write)
tcp_xmit_lowat (read and write)
tcp_recv_hiwat (read and write)
tcp_recv_hiwat_minmss (read and write)
tcp_fin_wait_2_flush_interval (read and write)
tcp_co_min (read and write)
tcp_max_buf (read and write)
tcp_strong_iss (read and write)
tcp_rtt_updates (read and write)
tcp_wscale_always (read and write)
tcp_tstamp_always (read and write)
tcp_tstamp_if_wscale (read and write)
tcp_rexmit_interval_extra (read and write)
tcp_deferred_acks_max (read and write)
tcp_slow_start_after_idle (read and write)
tcp_slow_start_initial (read and write)
tcp_co_timer_interval (read and write)
tcp_sack_permitted (read and write)
tcp_trace (read and write)
tcp_compression_enabled (read and write)
tcp_ipv6_hoplimit (read and write)
tcp_mss_def_ipv6 (read and write)
tcp_mss_max_ipv6 (read and write)
tcp_rev_src_routes (read and write)
tcp_ndd_get_info_interval (read and write)
tcp_rst_sent_rate_enabled (read and write)
tcp_rst_sent_rate (read and write)
tcp_wroff_xtra (read and write)
tcp_extra_priv_ports (read only)
tcp_extra_priv_ports_add (write only)
tcp_extra_priv_ports_del (write only)
tcp_status (read only)
tcp_bind_hash (read only)
tcp_listen_hash (read only)
tcp_conn_hash (read only)
tcp_acceptor_hash (read only)
tcp_host_param (read and write)
tcp_time_wait_stats (read only)
tcp_host_param_ipv6 (read and write)
tcp_1948_phrase (write only)
tcp_reserved_port_list (read only)
tcp_close_wait_interval(obsoleted- use tcp_time_wait_interval) (no read or write)

Use the man page of ndd to get more information

Which and whence commands

The which and whence commands tell you the location from where your command is being run. For example, if you
have multiple versions of a program at different locations and are trying to quickly determine where a program would
be run from without specifying the complete path, try:

$ which <cm_name>

128
or

$ whence <cmd_name>

Prstat command

The prstat command is a Solaris command. There may be open source equivalents available on the AIX and HP-UX
platforms.

The following is a sample of the prstat command output.

$prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
29611 muruganj 3344K 2248K cpu2 42 0 0:00.06 0.4% top/1
25134 jkreife 89M 55M sleep 28 10 0:00.01 0.2% java/55
123 flemingg 87M 54M sleep 28 10 0:00.01 0.1% java/41
15336 cones 94M 59M sleep 28 10 0:00.01 0.1% java/62

415 balakris 1992K 1744K cpu3 58 0 0:00.00 0.1% prstat/1

17775 shrestha 139M 19M sleep 58 0 0:00.00 0.1% [Link]/6

The output is similar to the top or ps - ef command, but with one additional piece of information in the /NLWP column
that indicates the number of threads in the process.

Iostat command

The iostat command displays i/o statistics, as shown in the following sample.

$ >iostat -xtc 5 2
extended device statistics tty cpu
device r/s w/s kr/s kw/s wait actv svc_t %w %b tin tout us sy wt id
md10 0.3 0.3 6.9 5.1 0.0 0.0 22.7 0 1 1 1259 1 1 1 97
md11 0.2 0.3 5.8 0.3 0.0 0.0 15.6 0 0
md12 5.8 0.3 40.9 5.0 0.0 0.0 3.2 0 1
md20 0.0 0.0 0.0 0.0 0.0 0.0 13.7 0 0
md21 0.0 0.0 0.0 0.0 0.0 0.0 13.6 0 0

In this example, device is the name of the disk, and r/s and w/s are reads and writes per second. Kr/s and kw/s are
kilobytes read and written per second, which are of use where there is a huge amount of i/o that you are trying to
find. Wait is the average number of transactions waiting for service (queue length), actv is the average number of
transactions actively being serviced (removed from the queue but not yet completed), and svc_t is average service
time in milliseconds. %w is percent of time transactions are waiting for service (queue non-empty), and %b is
percent of time the disk is busy (transactions in progress)

This command is useful for finding or not finding disk activity where you expect it to be.

129
Vmstat command

The vmstat command reports virtual memory statistics regarding process, virtual memory, disk, trap, and CPU
activity, as shown in the following sample. On multi-processor systems, the vmstat command averages the number
of CPUs into the output.

$ vmstat
procs| memory | page | disk | faults |
cpu
r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us
sy id
0 0 0 42393360 13834632 125 388 109 1 1 0 0 1 0 6 0 454 3097 1754 1
1 98

The vmstat command display contains the following fields.

Field Description
r procs The number of processes in the run queue.
b procs The number of processes blocked for resources I/O, paging, and so forth.
w procs The number of processes in swapped state.
swap memory The amount of swap space currently available, expressed in kilobytes.
free memory The size of the free list, expressed in kilobytes.
re page Page reclaims per second. See the -S option for how this field is modified.
mf page Minor faults per second. See the -S option for how this field is modified.
pi page Kilobytes paged in per second.
po page Kilobytes paged out per second.
fr page Kilobytes freed per second.
de page Anticipated short-term memory shortfall (kilobytes per second).
sr page Pages per second scanned by clock algorithm.
m1 diskm1 diskm1 diskm2 Report the number of disk operations per second. There are slots for up to four disks,
disk labeled with a single letter and number. The letter indicates the type of disk (s = SCSI i
= IPI, and so forth); the number is the logical unit number.
in faults Non-clock device interrupts per second.
sy faults System calls per second.
cs faults CPU context switches per second.
us cpu User time, expressed as a percentage of CPU time. On multiprocessor systems, this
is an average across all processors.
sy cpu System time, expressed as a percentage of CPU time. On multiprocessor systems,
this is an average across all processors.

130
Field Description
id cpu Idle time, expressed as a percentage of CPU time. On multiprocessor systems, this is
an average across all processors.

Use the man page of vmstat to get more information.

Sar command

The sar command is a system activity reporter that samples cumulative activity counters in the Linux/UNIX operating
system. Look at the sar man pages for more details

Df command

The df command displays the amount of disk space occupied by mounted or unmounted file systems, the amount of
used and available space, and how much of the file system total capacity has been used.

$df - k

This gives you usage in kilobytes.

Taking stacks and cores of running processes

The commands available for taking stacks and cores of running processes vary by hardware platform.

Solaris and RHEL

On Solaris or RHEL on an x86-64 server, you can use the following commands to take stacks and cores. Determine
the stack of a running process using the following $pstack command.

$pstack <pid >

Take a core dump of a running process using the following $gcore command.

$gcore [-o filename] pid

Determine stack of a core file using the following $pstack command.

$pstack <corefile>

AIX

On AIX, you can use the following commands to take stacks and cores. Determine the stack of a running process
using the following $procstack command.

$procstack <pid >

Take a core dump of a running process using the following $gencore command.

$gencore pid

These commands are useful when you think that the application process is hanging or has some other anomalous
behavior. All of these operations can affect performance of the running application and should be used with caution.

131
Look at main pages for the individual commands to get more details.

Diagnostic tools outside of the operating system

This section details a few of the diagnostic tools outside of the operating system that may give you useful information
to troubleshoot your issue.

Message monitoring with MMON

Use Message Monitor (MMON) to trace and monitor messages that are sent from and received by the solution.
MMON is available on the Linux platform, starting with BASE24-eps 3.0.12.

MMON “catches” messages at the edge of the solution, as they pass through ICE-XS. This enables messages that
might not reach an IS (Integrated server) trace to be traced and monitored.

MMON can parse and “pretty print” messages in a human-readable format.

For Linux platform users, MMON offers functionality provided on the HP NonStop platform by the NET24-XPNET
auditing facility and the third-party utility Explode.

MMON can be used in test environments to monitor and examine messages arriving at and departing from the
solution. ACI does not recommend that you use MMON in production systems.

MMON is composed of the following components and pieces of infrastructure:

• Message Monitor ICE-XS User Exit (MMIXUE)

• Dedicated queue in an IBM MQ queue manager
• Message Monitor Reader (MMRD)
• Dedicated tables in the database
• Message Monitor Command Line Interface (MMON CLI)

The following image shows the components of MMON:

132
A: Message Monitor ICE-XS User exit (MMIXUE). MMIXUE places copies of messages passing through ICE-XS in
an IBM MQ Message Monitor queue. It uses two ICE-XS User Exit IMP (Intermediate Message Processor)
definitions per ICE-XS process (for example Visa ICE-XS process and BankNet ICE-XS process) to monitor
messages: one IMP for incoming messages and one IMP for outgoing messages. Written in C as a shared library.

B: NOF files. The User Exit IMPs are configured in .nof files so you can choose which ICE-XS processes to
configure for message monitoring. You can enable or disable the User Exit IMPs with the nofxs utility.

C: Message Monitor Reader (MMRD). A process that gets messages from the Message Monitor queue and inserts
them into Message Monitor tables (filled in rotation). Written in C++ on top of SIS layer.

D: JSON files. Describe the layout of the messages, which the Message Monitor Command Line Interface can parse
and display. For example, they describe field names, field sizes, field types, possible field values, and their
meanings.

E: Message Monitor Command Line Interface (MMON CLI). A command-line tool that reads the messages from the
tables in the database and shows the messages to the user. Positions, filters, and selects from the table to which it
has been pointed to by the user. Pretty-prints (parse and format) mesages if the application understands the type of
message. Written in Python.

F: The output on the terminal. Users can view "pretty-printed" (exploded) messages, with or without their raw
representation, and restrict the ouput using different options.

MMON is delivered to customers on the Linux platform starting with BASE24-eps 3.0.12:

• MMIXUE is delivered as a shared library ([Link]). During installation, it is copied onto the Linux server in the
$LIB directory.
• MMRD is delivered as an archive (mmon.a) and is linked into an executable object ([Link]) at installation.
• MMON CLI is delivered as an executable object (mmon). You can find the binary mmon and its stubs library in
the static directory in $LIB.

133
Configure ICE-XS processes for MMON

To use MMON, you must configure the ICE-XS processes for which you want to trace messages. To do this, edit the
.nof files for the ICE-XS processes to make the following changes:

• Use ENDPOINTs, not STATIONs.

• Add two EXITPROFILE definitions: one for inbound messages, and one for outbound messages. Specify the
location of [Link] as the LIBRARY, trace_msg_ue as the FUNCTION, and set SRC, DEST, and
ENABLED in OPTIONS. Inbound messages must have the symbolic name of the endpoint as the SRC, and the
name of the BASE24-eps component ID (that is, interface component ID, ATM device handler component ID,
and so on) as the DEST. Outbound messages must have the name of the BASE24-eps component ID as the
SRC, and the symbolic name of the endpoint as the DEST.
• Add two IMP definitions: one for inbound messages, and one for outbound messages. Reference the correct
EXITPROFILEs.
• Add an IMPREQUEST to each ROUTERULE. Reference the correct IMP.
• Change the RRSELECTION option from FIRSTMESSAGE to EVERYMESSAGE in the ENDPOINT definition.
• Restart (cold start) the ICE-XS process to pick up changes to the .nof file.

See the example of the ICE-XS .nof file with the MMON changes made.

Turn off MMON User Exit tracing

After the tracing function is configured, use one of the following ways to turn it off (that is, enable/disable the user
exit):

1. Stop the ICE-XS process, comment out the IMPREQUEST parameter specification of the ROUTERULE section,
and restart (cold start) the ICE-XS process to pick up new changes.
2. Enable or disable the user exit without stopping the ICE-XS process. To do this, use NOF-XS (the command line
interface for ICE-XS). Run commands similar to the following:

$ICEXS/nofxs
open PRO1_BNET
alter EXITPROFILE EXP_MMON_INBND
OPTIONS("SRC=S1C^BNET","DEST=INTFBNET","ENABLED=Y")

a. Set the ENABLED parameter to Y or N to enable or disable the user exit.

Example .nof file

The following is an example ICE-XS .nof file with changes made to configure and enable MMON:

OPEN PRO1_BNET
ADD MSGROUTER MSGR1
ADD DEVPROTOCOLGENERIC2 DEVP_BNET
ADD DEVPROTOCOLMDS DEVP_MDS

ADD CONNPROFILETCPIP PRO1_BNET, AUTHZADDR *, LPORT 40026, MAXCONNS 10

ADD CONNPROFILEMQ CP_MQ_IS, RECVQMGR [Link], RECVQ [Link], SENDQMGR
[Link], SENDQ [Link]

134
ADD ENDPOINT EP_MQ_IS1, CONNPROFILE CP_MQ_IS, DEVPROTOCOL DEVP_MDS, CONTACT OUT,
ISTATUS STARTED,URI URI_MQ_IS1
ADD ENDPOINT EP_MQ_IS2, CONNPROFILE CP_MQ_IS, DEVPROTOCOL DEVP_MDS, CONTACT OUT,
ISTATUS STARTED,URI URI_MQ_IS2
ADD ENDPOINT EP_MQ_BNET, CONNPROFILE PRO1_BNET, DEVPROTOCOL DEVP_BNET, CONTACT IN,
ISTATUS ACTIVE, RRSELECTION EVERYMESSAGE, URI URI_BNET 1

== Exit profile resources 2

ADD EXITPROFILE EXP_MMON_INBND, &

TYPE LIBRARY, &
LIBRARY /app/b24dev/users/PRO1/lib/[Link], &
FUNCTION trace_msg_ue, &
OPTIONS ("SRC=S1C^BNET","DEST=INTFBNET","ENABLED=Y")

ADD EXITPROFILE EXP_MMON_OUTBND, &

TYPE LIBRARY, &
LIBRARY /app/b24dev/users/PRO1/lib/[Link], &
FUNCTION trace_msg_ue, &
OPTIONS ("SRC=INTFBNET","DEST=S1C^BNET","ENABLED=Y")

== Intermediate Message Processor resources 3

ADD IMP IMP_MMON_INBND, &

TYPE EXIT, &
EXITPROFILE EXP_MMON_INBND

ADD IMP IMP_MMON_OUTBND, &

TYPE EXIT, &
EXITPROFILE EXP_MMON_OUTBND

ADD ROUTERULE RR_1, &

IMPREQUEST IMP_MMON_INBND, & 4
TYPE STATIC, &
ORIGENDPOINTURI URI_BNET, &
DESTENDPOINTURI URI_MQ_IS1, &
WEIGHT 1, &
ENABLE YES, &
MEP REQUESTONLY, &
MDSSYMSRC S1C^BNET, &
MDSSYMDEST INTFBNET

ADD ROUTERULE RR_2, &

IMPREQUEST IMP_MMON_INBND, & 5
TYPE STATIC, &
ORIGENDPOINTURI URI_BNET, &
DESTENDPOINTURI URI_MQ_IS2, &
WEIGHT 2, &
ENABLE YES, &
MEP REQUESTONLY, &
MDSSYMSRC S1C^BNET, &
MDSSYMDEST INTFBNET&

ADD ROUTERULE RR_3, &

IMPREQUEST IMP_MMON_OUTBND, & 6
TYPE STATIC, &
ORIGENDPOINTURI URI_MQ_IS1, &
DESTENDPOINTURI URI_BNET, &
WEIGHT 3, &
ENABLE YES, &

135
 MEP REQUESTONLY, &
MDSSYMSRC INTFBNET, &
MDSSYMDEST S1C^BNET&

ADD ROUTERULE RR_4, &

IMPREQUEST IMP_MMON_OUTBND, & 7
TYPE STATIC, &
ORIGENDPOINTURI URI_MQ_IS2, &
DESTENDPOINTURI URI_BNET, &
WEIGHT 4, &
ENABLE YES, &
MEP REQUESTONLY, &
MDSSYMSRC INTFBNET, &
MDSSYMDEST S1C^BNET&

1 Adds RRSELECTION EVERYMESSAGE,

2 Adds Exitprofile resources.
3 Adds IMP resources.
4 Adds IMPREQUEST inbound location.
5 Adds IMPREQUEST inbound location.
6 Adds IMPREQUEST outbound location.
7 Adds IMPREQUEST outbound location.

After changes are made to the .nof file, the ICE-XS processes start putting copies of messages that pass through
the ICE-XS processes into the MQ queue.

If tracing ATMs, in the ROUTERULE section of the .nof file, in addition to the IMPREQUEST parameter you must
also have an IMPRESPONSE parameter. The RRSELECTION parameter must be set to CONNECTION, not
EVERYMESSAGE.

The following is a sample ATM ICE-XS .nof file with MMON changes made:

136
 == ====================================================================
==
== %name: [Link] %
== %version: 064_3 %
== %created_by: jkreife %
== %date_created: Tue Dec 18 11:52:27 2018 %
==
== ====================================================================

OPEN PRO1_ANCR
ADD DEVPROTOCOLGENERIC2 DEVP_ANCR
ADD DEVPROTOCOLMDS DP-MDS
ADD VSOCKET VS_NDC, KEEPALIVE YES, TCPKEEPALIVE 30

ADD CONNPROFILETCPIP PRO1_ANCR, AUTHZADDR *, LPORT 40099, MAXCONNS 1, LVSOCKET

VS_NDC
ADD CONNPROFILEMQ CP-MQ, RECVQMGR [Link], RECVQ PRO1.S1C_NDC, SENDQMGR
[Link], SENDQ [Link]

ADD ENDPOINT EP-NDC, CONTACT IN, CONNPROFILE PRO1_ANCR, DEVPROTOCOL DEVP_ANCR,

ISTATUS STARTED, FMM YES, RRSELECTION CONNECTION 1
ADD ENDPOINT EP-BASE24EPS, CONNPROFILE CP-MQ, DEVPROTOCOL DP-MDS, CONTACT OUT,
ISTATUS STARTED, FMM NO

== Exitprofile resources 2

ADD EXITPROFILE EXP_MMON_INBND, &

TYPE LIBRARY, &
LIBRARY /app/b24dev/users/prodev/PRO1/lib/[Link], &
FUNCTION trace_msg_ue, &
OPTIONS ("SRC=S1C^NDC","DEST=NCRDH","ENABLED=Y")

ADD EXITPROFILE EXP_MMON_OUTBND, &

TYPE LIBRARY, &
LIBRARY /app/b24dev/users/prodev/PRO1/lib/[Link], &
FUNCTION trace_msg_ue, &
OPTIONS ("SRC=NCRDH","DEST=S1C^NDC","ENABLED=Y")

== Intermediate Message Processor resources 3

ADD IMP IMP_MMON_INBND, &

TYPE EXIT, &
EXITPROFILE EXP_MMON_INBND

ADD IMP IMP_MMON_OUTBND, &

TYPE EXIT, &
EXITPROFILE EXP_MMON_OUTBND

ADD ROUTERULE RR-1, IMPREQUEST IMP_MMON_INBND, IMPRESPONSE IMP_MMON_OUTBND, TYPE

STATIC, WEIGHT 1, ENABLE YES, ORIGENDPOINTURI EP-NDC, DESTENDPOINTURI EP-
BASE24EPS, MDSSYMSRC S1C^NDC, MDSSYMDEST NCRDH 4
ADD ROUTERULE RR-2, IMPREQUEST IMP_MMON_OUTBND, IMPRESPONSE IMP_MMON_INBND, TYPE
STATIC, WEIGHT 2, ENABLE YES, ORIGENDPOINTURI EP-BASE24EPS, DESTENDPOINTURI EP-
NDC, MDSSYMSRC NCRDH, MDSSYMDEST S1C^NDC 5

1 Adds RRSELECTION CONNECTION.

2 Adds Exitprofile resources.
3 Adds IMP resources.

137
4 Adds inbound IMPREQUEST and IMPRESPONSE locations.
5 Adds outbound IMPREQUEST and IMPRESPONSE locations.

ICE-XS user exit log

The ICE-XS user exit produces log messages in a file such as $LOGS/mmon_ue.PRO1_BNET.log. In this example,
PRO1_BNET is the ICE-XS symbolic name (the log filename is specific to each ICE-XS process).

MMON MQ queues

At installation, the regular queues, queue <PRFX>.MMON and service queue <PRFX>.MMON1 are created. If
necessary, you can create them manually. You can adjust the maximum queue depth for the <PRFX>.MMON queue
to suit the level of message traffic in the system. Adjust the MAXMSGL parameter of the <PRFX>.MMON queue to suit
the maximum message length that the system will support. MMRD and the database support messages up to 16K in
length.

MMRD

You must start MMRD to get the copies of the messages off the MQ queue and insert them into a table in the
database. You can use the runmmrd script, located in $BIN, to start MMRD or Application Management in the ACI
Desktop.

• MMRD keeps doing this job until it is stopped. Use either the stopmmrd script (also present in $BIN) or
Application Management in the ACI desktop, to stop MMRD.
• MMRD logs events to the BASE24-eps event log.
• MMRD fills the tables in the database in rotation. For example:
◦ When the current table is full, MMRD switches to the next table, empties it, and starts filling it with new
messages.
◦ When the last table is full, MMRD cycles back to the first table.
• You can adjust the number of messages that MMRD will store in each table in the database to suit the level of
message traffic in the system. The default number of messages that MMRD will store in each table in the
database is 10000. To configure the number of messages that MMRD will store in each table in the database:

If Application Management is used by the BASE24-eps system, then:

1. In the BASE24-eps ACI desktop, go to System Operations > Application Management.

2. When the window opens, go to Categories > Process Manager > UNIXProcess> SISUNIXProcess and
select MMRD.
3. On the right, click Edit.
4. Find the row containing the Name Args and add the argument -tblmaxrow= along with the number of
messages that you want to store in each table. Omit quotes.

138
 5. Click Save.
6. Restart the MMRD process.

If Application Management is not used by BASE24-eps, then:

1. In the runmmrd script, update the value for the -tblmaxrow argument.
2. Restart the MMRD process.

139
In addition to configuring the number of messages which MMRD stores in each table in the database, you can also
increase (or decrease) the number of tables used in rotation to store messages in the database. To do this, configure
the -tblringsize argument the same way that you configure the -tblmaxrow argument (described previously).
The default number of tables is three. You can adjust the number of tables used in rotation to store messages in the
database to suit the level of message traffic in the system.

You can also change the table names to which the messages are logged. For example, if you want to change the
table names to MMON_BNET_TRC_00, MMON_BNET_TRC_01, and MMON_BNET_TRC_02, specify the following startup
argument: -tblrootname= MMON_BNET_TRC_.

MMON database tables

At installation, the database tables are created. In a standard installation, there are three tables for messages, used
in rotation. The default table names are MMON_TRACE_00, MMON_TRACE_01, and MMON_TRACE_02. A table also
stores control information. Its default table name is MMON_TRACE_INFO.

You can control and configure the names and number of tables.

MMON CLI

After the copies of the messages are in the tables in the database, you can use the Message Monitor Command
Line Interface (MMON CLI - mmon) to view them. You can run the MMON CLI on any Linux machine that has
access to the database. You do not have to run MMON CLI on the Linux server where BASE24-eps runs, nor on the
Linux server which runs the database. Although MMON CLI’s binary mmon and its stubs repository (static/) are
provided under $LIB, ACI recommends that you relocate these artifacts to a suitable dedicated directory. To run
MMON CLI, execute it from the command line in the directory where mmon is found:

./mmon

MMON CLI must be provided with at least one flag, otherwise, as it is the case with the above
NOTE
example, it defaults to showing the help page.

The first time MMON CLI is run using option -t, the user is prompted to enter the following:

• Database Type (PostgreSQL or Oracle)

• PostgreSQL Database Name (for example, eps_postgres) or Oracle Service Name
• PostgreSQL Schema (for example, eps_pg_schema) or Oracle Client Path
• Database IP Address
• Database Port
• Database User Name (for example, eps_pg_mmon_user)
• Database User Password

ACI recommends that you configure a dedicated database user that has been created by a DBA to have select
access only, and only to the MMON tables in the database.

MMON CLI saves the information in [Link] so you do not have to specify this information on subsequent
runs. If you must change this information later, you can edit [Link], or delete [Link] and run
the MMON CLI again. [Link] is stored in the same directory as the MMON CLI mmon executable object.

140
MMON CLI writes to its own log file ([Link]). MMON CLI also stores its logging configuration options in
[Link]. MMON CLI automatically adds the logging configuration options when MMON CLI creates
[Link]. You can edit them if necessary. MMON CLI logging configuration options are:

• Level of logging: INFO, DEBUG, WARNING or CRITICAL.

• Maximum number of log entries.
• Number of rotating backup log files to keep.
• Location of the [Link] file.

If the logging level is DEBUG, the MMON writes more information to the [Link] file, and the raw messages read
from the tables in the database will be shown to the user (see Examples of MMON CLI output).

In addition to reading and pretty-printing messages from tables in the database, MMON CLI can also read and
pretty-print messages captured in BASE24-eps Integrated Server (IS) trace files (typically named mtrcrc) with the -f
flag. If using MMON CLI to read and pretty-print messages captured in BASE24-eps trace files, the BASE24-eps
trace must have been configured with the External Message trace level enabled. MMON CLI cannot read BASE24-
eps trace files from older versions of BASE24-eps.

MMON CLI flags

To get MMON CLI help and view the flags supported, use the -h flag:

./mmon -h

141
 MMON - Started @ 2021-07-13T09:46:51

usage: mmon [-h] [-a [FINDASCII ...]] [-d [DESTINATION ...]] [-e ENDTIME] [-f
FILE]
[-n] [-o [SOURCE ...]] [-p] [-r] [-s STARTTIME] [-t [TBNAME]]
[-v]

Utility that pretty prints BASE24-eps transactions based on parameters.

optional arguments:
-a [FINDASCII ...], --findascii [FINDASCII ...]
Find and display messages that contain the given ASCII
string.
Multiple values separated by a space are allowed.
-d [DESTINATION ...], --destination [DESTINATION ...]
Filter the output per given destination. Multiple values
separated by a space are allowed. e.g. -d DEST1 DEST2
-e ENDTIME, --endtime ENDTIME
Use the given datetime in YYYYMMDDhhmmss format, to
determine
at which entry to stop the display.
-f FILE, --file FILE Use the BASE24-eps trace in the given FILE-path as input
for
message parsing.
-n, --nlogon Exclude network management messages from the output.
-o [SOURCE ...], --source [SOURCE ...]
Filter the output per given source. Multiple values
separated
by a space are allowed. e.g -o SOURCE1 SOURCE2
-p, --paging Activate paging of the output.
-r, --realtime Monitor and display messages as they become available from
the
MMON table until CTRL+C is entered.
-s STARTTIME, --starttime STARTTIME
Use the given datetime in YYYYMMDDhhmmss format, to
determine
from which entry to start the display.
-t [TBNAME], --tbname [TBNAME]
Read messages from the specified table. If no table is
specified, the table into which messages are currently being written is used.
-h, --help Show this help message and exit.
-v, --version Print this utility's current version number and exit.

Example :
For table :
mmon -t MMON_TRACE_00 <other options> # executes options on given table.
mmon -t <other options> # executes options on current working table.
mmon -t -r <other options> # executes options in real-time on the current
working table.
mmon -t MMON_TRACE_00 -r <other options> # executes options in real-time
on
data from the given table.
For BASE24-eps traces:
mmon -f file_name <other options> # executes options on given file name.

Flags other than -v, -h , -t and -f, require either the -f or -t flag.

• Flags -t and -f are mutually exclusive.

142
 • Flags -r and -f are mutually exclusive.

When using the -r flag, MMON CLI polls the database at regular intervals to fetch new messages. The interval
between polls defaults to 0.5 seconds. You can adjust realtimepolling parameter in [Link] to
change the interval. Values under 0.1 seconds are invalid and will be ignored by mmon. A value of 0.1 seconds will
be used instead.

Values provided for the filter flags -o, -d, and -a can be wildcarded (with * and ?).

The filter flag -a searches for strings in the displayed output. For example, if you want to search for a PAN, you must
search for it in its masked format, which is how it is displayed.

Message layout

MMON CLI uses JSON files to describe the layout of messages for each interface and enables MMON CLI to parse
the messages and pretty-print them on the page. The MMON CLI comes with JSON files for all the interfaces that
MMON currently supports. You can edit the JSON files (for example, to specify field names in local language). Be
sure to take backups. The JSON files are in the static/stubs/ subdirectory under the location of the MMON CLI
mmon executable object.

MMON CLI can parse either ASCII or EBCDIC encoded messages. Depending on whether an endpoint uses ASCII
or EBCDIC encoding, you can update the ext_char_set setting in the JSON file to indicate to MMON CLI which
encoding to apply. For example:

You can define each symbolic station name of the endpoint in the JSON file, if you must differentiate them, in the
MMON CLI. For example, if an endpoint has one station using ASCII encoding and one station using EBCDIC
encoding, you can implement a configuration similar to the following:

143
Examples of MMON CLI output

The following are examples of MMON CLI output:

MMON - Started @ 2021-10-26T10:07:18

**********************************

2021-05-28 05:22:30.814703 FROM: [S1C^BNET] TO: [INTFBNET]

0120 INTFBNET

Raw data:
[F0F1F2F0767B46018EE1A21AF1F6F5F3F2F1F7F1F0F0F0F0F0F0F0F3F6F9F0F0F0F0F0F0F0F0F0F0F
0F0F0F9F0F7F3F3F0F0F0F0F0F0F0F9F0F7F3F3F0F5F2F8F1F0F2F2F1F3F6F1F0F0F0F0F0F0F0F5F2F
0F2F1F1F2F2F2F3F0F0F5F2F8F0F5F2F8F0F5F2F8F5F5F4F2F0F5F1F0F0F0F0F6F5F5F5F5F5F5F0F6F
6F6F6F6F6F6F0F5F2F8F0F0F0F5F2F0F2F1F1F4F2F2F5F0F0F0C2D5E3F3F3F0F0F7C2D5E34BF3F34BF
0F0F78240404040C2D5E3F3F3F0F0F740E3819592969481A3404040404040E68199A2A981A68140404
0404040D7D6D3F0F3F9D9F1F5F1F0F0F5F2F8F1F2F2F2F3F0F2F0F0F1D7F6F3F1F5D4C3C3F2F1F7F0F
0F4F9F9F0F9F1F2F9F8F5F9F8F5F1F0F15F2A0209859A032105289C01009F020600000001626D9F030
60000000000009F1A0206169F10120110200000044000000000000000000000009F2701809F3403420
1009F360202989F370433E7E58082025800950500000400009F2608226194410CAA2D16F0F0F3F1F9F
1F0F2F5F1F0F0F0F0F0F4F0F0F2F4F0F0F0F5F6F9F0F2F1F0F1F2F3F4F0F0F9D4C4E2F1F2F2F2F3F0]

Bitmap : [767B46018EE1A21A]
002 Primary Account Number (PAN) : [5321********0369]
003 Processing Code : [000000]
003 - Transaction Code : [00]
003 - From Account : [00]
003 - To Account : [00]
004 Transaction Amount : [000000090733]
006 Amount, Cardholder Billing : [000000090733]
007 Transmission Date and Time : [0528102213]
010 Conversion Rate, Cardholder Billing : [61000000]
011 System Trace Audit Number (STAN) : [052021]
012 Time, Local Transaction : [122230]
013 Date, Local Transaction : [0528]
015 Date, Settlement : [0528]
016 Date, Conversion : [0528]
018 Merchant Type : [5542]
022 Point-of-Service (POS) Entry Mode : [051]
023 Card Sequence Number : [000]
032 Acquiring Institution ID Code : [555555]
033 Forwarding Institution ID Code : [666666]
037 Retrieval Reference Number : [052800052021]
038 Authorization ID Response : [142250]
039 Response Code : [00]
041 Card Acceptor Terminal ID : [BNT33007]
042 Card Acceptor ID Code : [BNT.33.007b ]
043 Card Acceptor Name/Location : [BNT33007 Tankomat
Warszawa POL]
048 Additional Data-Private Use :
[R151005281222302001P6315MCC217004990912]
049 Currency Code, Transaction : [985]
051 Currency Code, Cardholder Billing : [985]
055 Integrated Circuit Card (ICC) Data :
[5F2A0209859A032105289C01009F020600000001626D9F03060000000000009F1A0206169F1012011
0200000044000000000000000000000009F2701809F34034201009F360202989F370433E7E58082025

144
800950500000400009F2608226194410CAA2D16]
Tag [5F2A] - Len [02] : [0985]
Tag [9A] - Len [03] : [210528]
Tag [9C] - Len [01] : [00]
Tag [9F02] - Len [06] : [00000001626D]
Tag [9F03] - Len [06] : [000000000000]
Tag [9F1A] - Len [02] : [0616]
Tag [9F10] - Len [12] :
[011020000004400000000000000000000000]
Tag [9F27] - Len [01] : [80]
Tag [9F34] - Len [03] : [420100]
Tag [9F36] - Len [02] : [0298]
Tag [9F37] - Len [04] : [33E7E580]
Tag [82] - Len [02] : [5800]
Tag [95] - Len [05] : [0000040000]
Tag [9F26] - Len [08] : [226194410CAA2D16]
060 Advice Reason Code : [191]
061 Point-of-Service (POS) Data : [1000004002400056902101234]
061 - POS Terminal Attendance : [1]
061 - Reserved for Future Use - 1 : [0]
061 - POS Terminal Location : [0]
061 - POS Cardholder Presence : [0]
061 - POS Card Presence : [0]
061 - POS Card Capture Capabilities : [0]
061 - POS Transaction Status : [4]
061 - POS Transaction Security : [0]
061 - Reserved for Future Use - 2 : [0]
061 - Cardholder-Activated Terminal Level : [2]
061 - POS Card Data Term Input Cap Ind : [4]
061 - POS Authorization Life Cycle : [00]
061 - POS Country Code : [056]
061 - POS Postal Code : [902101234]
063 Network Data : [MDS122230]
063 - Financial Network Code : [MDS]
063 - Banknet Reference Number : [122230]

**********************************

2021-05-28 05:22:30.864604 FROM: [INTFBNET] TO: [S1C^BNET]

0130 INTFBNET

Raw data:
[F0F1F3F0766302018A81A002F1F6F5F3F2F1F7F1F0F0F0F0F0F0F0F3F6F9F0F0F0F0F0F0F0F0F0F0F
0F0F0F9F0F7F3F3F0F0F0F0F0F0F0F9F0F7F3F3F0F5F2F8F1F0F2F2F1F3F6F1F0F0F0F0F0F0F0F5F2F
0F2F1F0F5F2F8F0F5F2F8F0F0F0F0F6F5F5F5F5F5F5F0F6F6F6F6F6F6F6F0F5F2F8F0F0F0F5F2F0F2F
1F0F0C2D5E3F3F3F0F0F7F0F3F9D9F1F5F1F0F0F5F2F8F1F2F2F2F3F0F2F0F0F1D7F6F3F1F5D4C3C3F
2F1F7F0F0F4F9F9F0F9F1F2F9F8F5F9F8F5F0F0F9D4C4E2F1F2F2F2F3F0]

Bitmap : [766302018A81A002]
002 Primary Account Number (PAN) : [5321********0369]
003 Processing Code : [000000]
003 - Transaction Code : [00]
003 - From Account : [00]
003 - To Account : [00]
004 Transaction Amount : [000000090733]
006 Amount, Cardholder Billing : [000000090733]
007 Transmission Date and Time : [0528102213]
010 Conversion Rate, Cardholder Billing : [61000000]
011 System Trace Audit Number (STAN) : [052021]

145
 015 Date, Settlement : [0528]
016 Date, Conversion : [0528]
023 Card Sequence Number : [000]
032 Acquiring Institution ID Code : [555555]
033 Forwarding Institution ID Code : [666666]
037 Retrieval Reference Number : [052800052021]
039 Response Code : [00]
041 Card Acceptor Terminal ID : [BNT33007]
048 Additional Data-Private Use :
[R151005281222302001P6315MCC217004990912]
049 Currency Code, Transaction : [985]
051 Currency Code, Cardholder Billing : [985]
063 Network Data : [MDS122230]
063 - Financial Network Code : [MDS]
063 - Banknet Reference Number : [122230]

**********************************

Number of transactions displayed: 2

End of MMON Session

MQ diagnostics

You can determine a number of things about the status of your queues and channels and definitions using a few mq
commands. Use these commands and play with them after consulting your MQ administrator.

$ mqver

Displays the version and patch level of MQ installed on your system

$runmqsc <q_manager_name>

If this command does not report an error, your queue manager does exist. When this command works, it takes you
to a blank prompt.

Typing the following command gets you out of this prompt and returns you to the Linux/UNIX shell.

end

You can display all queues present in the mq system by doing the following command:

$runmqsc <q_manager_name>
display qlocal(*)
…
..
..
.
end
$

This gives a summary display of your queues. Getting more details about a specific queue in this queue manager
requires you to use the following command:

146
 $runmqsc <q_manager_name>
display qlocal(<your_queue_name>)
…
end
$

You can determine the channels that you have using the following commands:

$runmqsc <q_manager_name>
display channel(*)
…
..
..
.
end
$

Display the channel status using the following command:

display chstatus(*)

Again, as before, these give you summary descriptions. Specify the actual channel name in the parentheses to get
details.

In typical installations, ACI provides a script by the name vqdloop that displays the queue depth in the system every
one to three seconds. This is a useful tool to see bottlenecks in the system, if any. If the queue is building up then
the process that reads messages off the queue is either down or is severely constrained from a performance
perspective.

If you see messages in a queue and want to determine what these messages are, MQ provides a sample program
by the name amqsbcg in /opt/mqm/samp/bin (the installation at your site may have it at a different location). Invoke
this program using the following command:

$ amqsbcg <q_name> <q_manager_name>

This command reads the MQ messages without removing them from the queue and write the details out to stdout.

Use your MQ manuals to get more information.

c-tree diagnostics

A method to check if your specific c-tree server is up or not can be done using

147
 $ctadmn
Enter Administrator User ID (and/or press RETURN) >>

<your_admin_id>

Enter Administrator Password (and/or press RETURN) >>

<your_admin_password>

Enter Optional File Password (and/or press RETURN) >>

Enter Optional Server Name (and/or press RETURN) >>

<name_of_your_c-tree_server>

If this works, that should tell you that your c-tree server is alive. This tool is menu driven and you can explore a
number of different options with your c-tree server using this method.

For example, after the above command works, it would bring up a menu that reads as follows:

**** FairCom(R) Server Administration Utility ****

Copyright 1990-2001 FairCom Corporation
All Rights Reserved
1. User Operations
2. Group Definitions
3. File Security
4. Monitor Clients
5. Server Information (IOPERFORMANCE)
6. Server Configuration (SystemConfiguration)
7. Stop Server

Choosing Monitor Clients tells you if any clients are attached to your c-tree server. This is useful information if there
are clients that are attached when you don’t expect them or vice-versa.

Look through your Faircom documentation for more details.

NOF-XS for ICE-XS troubleshooting

ICE-XS has a utility packaged with it that is called NOF-XS. Invocation to NOF-XS is as follows:

148
 $nofxs
NOF-XS 1 → open <symname>
ICE-XS:<symname> 2 → status station *
STATION Msgrouter State
< symname > <state>
ICE-XS:<symname> 3 → exit
NOF-XS - 1001 Process stopping
$

In the example above, we did a display of all configured stations.

If you have questions regarding whether a particular station is connected or not, this would be a quick way to figure
this out.

Look at your ICE-XS documentation for more details on NOF-XS and how to use it.

Exception log in BASE24-eps

BASE24-eps has an exception log that can be accessed using dalci, as follows:

$rundalci
1: select * from EXCEPTION_LOG insertform;
…
..
.
2: exit

If there are messages that are sent to BASE24-eps where the switch does not understand the message format, it
would write these messages out to the exception log. So if you are missing messages, this may be a place to go to
find them.

Journal Perusal

Journal Perusal is an application within the BASE24-eps UI that gives a view to the data that has been logged to the
application journal files. It can be used for after-the-fact problem resolution and to investigate disputed transactions.
The display of the Journal Perusal Summary and Detail windows are formatted using scripts that can be altered
onsite to give custom views to the journal data. The Journal Perusal UI enables transactions to be looked up by PAN
or Account Number for card issuers, and by Channel ID for terminal acquirers. The look-ups can be limited to a
specific posting date or a date/time range from the current time back to the retention period of the journal files for an
issuer.

For more information on Journal Perusal and the steps required to configure a UI user so that they can use specific
perusal scripts, see the Journal Perusal User Guide in the BASE24-eps documentation under the Operations Menu.

For more information on writing custom Journal Perusal Scripts, see the Scripting Manual in the BASE24-eps
documentation under the Core Menu.

JLFScan on your journal data

Another way to view journal data that does not require access to the BASE24-eps UI is a utility called JLFScan. This

149
utility can select transactions from a specific journal file and display the contents of a range of transaction data
elements.

To run JLFScan, you must know the assign name of the journal file that you wish to select transactions from. Journal
file assigns are configured in the Journal Profile Configuration UI of the BASE24-eps desktop, or can be found in the
[Link] (for more information on the configuration of journal file assigns, see the Journal User Guide of the
BASE24-eps documentation under the Core Menu). The assign for a Stream file is necessary in the [Link] (a
popular one is named MTRCRC).

JLFScan is run by executing the runjlfscan command in the $BIN directory. JLFScan runs interactively and
remembers the last entry used from one run to the next. Many times this last entry shows up in parentheses when
requesting input and is the default value if you press the enter key. The following example shows the dialog between
a user and the JLFScan program

>./runjlfscan
JLFScan Version 1.3
Enter the parameters for this run. 'Enter' key indicates use default.
Enter the Journal [Link] assign or Q to quit (JLF_BNK1_P99_5):
[Here you must enter the assign name of a journal file from the [Link]]
Count Rows in Journal (Y/N), or Q to quit (N):
[Entering Y will cause the JLFSCan program to just count the number of records in
the journal file and display the result. Entering N allows you to proceed to
displaying journal record contents.]
Filter read using constraints (Y/N), or Q to quit (N):
[Entering N will display data for every journal record in the file. Entering Y
allows you to select filtering constraints.]

Filtering constraints can be placed on the following fields which are part of the journal file keys:

• pan
• onl_key_amt
• seq_num_char
• iss_inst_id
• acct_num
• tran_dat_tim
• acq_inst_id
• mrch_id
• chan_id

The dialog for entering constraints is:

Enter the field name or Q to quit :

mrch_id [One of the field names from the list above]
Enter the operator or Q to quit :
== [operators ==, <, <=, >, >=, !=, contains, starts]
Enter the value or Q to quit :
MRCH2 [the value to use in the constaint]

150
Once your constraints are entered, the following dialog tells JLFScan what data to report upon:

Enter first row to be displayed or Q to quit (1):

1 [Start at the beginning of the file, or skip records to the count specified]
Enter the number of rows to be displayed or Q to quit (7):
1 [Tells how many records to display]
Specify TDEs to be displayed: A (all) or R (range) or S (select) or Q to quit
A [ Tells which TDEs to display. A displays all the TDES, R or S allows you to
select certain TDES by the TDE object ID]
Specify the number of TDEs to display at a time, or Q to quit (5):
[Tells how many TDEs to display between prompts]
Enter a Report Stream [Link] assign (if required) or Q to quit (MTRCRC):
[An optional report stream to capture the output if desired.]

The output looks like this:

******************************************************************************
pan onl_key_amt seq_num_char discrim
9999999999999999999999999999 133221096004931830 999999999999 0
num_jrnl_rec iss_inst_id acct_num tran_dat_tim acq_inst_id bus_lvl1
1 00001 2004121120000061 00001
bus_lvl2 bus_lvl3 bus_lvl4 bus_lvl5 mrch_id chan_id shift_num btch_num
MRCH2 SPDH-01 1 22
rec_frmt clerk_id
5 CLRK01
jrnl_body
0x0000083d000003e8000000434d52434832202020202020202020202000000348303030303120
------------------------------------------------------------------------------
Object id = 2101
TRANSACTION AMOUNT = TXN AMT VAL = 133221096004931830, TXN AMT CRNCY CDE = 840
; ACQUIRER: CRNCY CDE = 840, CNV RTE/FEE FLG = 1, 0, SCLE/RTE = 0, 1
; CRDHLDER 1 BILLING: CRNCY CDE = 840, CNV RTE/FEE FLG = 1, 0, SCLE/RTE = 0, 1
; CRDHLDER 2 BILLING: CRNCY CDE = 840, CNV RTE/FEE FLG = 1, 0, SCLE/RTE = 0, 1
; ISSUER: CRNCY CDE = 840, CNV RTE/FEE FLG = 1, 0, SCLE/RTE = 0, 1
; LOGICAL NET: CRNCY CDE = 840, CNV RTE/FEE FLG = 1, 0, SCLE/RTE = 0, 1
; TERMINAL: CRNCY CDE = 840, CNV RTE/FEE FLG = 1, 0, SCLE/RTE = 0, 1
------------------------------------------------------------------------------
Object id = 2102
DISCRIMINANT = 0
------------------------------------------------------------------------------
Object id = 2104
PAN = 9999999999999999999999999999, Algo = 0
------------------------------------------------------------------------------
Object id = 2105
CAPTURE DATE = 04/12/12
------------------------------------------------------------------------------
Object id = 2106
SEQUENCE NUMBER = 999999999999
More (Y/N) (Y):

Journal monitoring with JMON

Use the Journal Monitor (JMON) utility to monitor transactions in your Journal files using a command line interface.
This utility replaces some of the key functionality provided on the NonStop platform by the third-party utility Monitor.
JMON provides similar functionality on the Linux platform.

151
The JMON utility enables you to monitor/examine/interrogate transactions (rows) written to your Journal tables in
real time or retrospectively. In summary, JMON enables you to watch transactions as they are logged to your
BASE24-eps system or to examine transactions from open or closed Journals.

The JMON utility can be run interactively from a command line or from a shell script. You can filter the transactions
you want to monitor through a number of different criteria and specify different output formats or customize the
output displayed according to your specific business requirements. JMON is intended to be used for both testing and
troubleshooting. You can run it in secure or unsecure modes. When you run JMON in secure mode, you must
specify your username and password before you can run the utility. You should always use JMON in secure mode
when monitoring Journal transaction on a production system.

The utility uses the PCI_NUM_UNMASK_LEFT and PCI_NUM_UNMASK_RIGHT Environment attributes to mask PANs
and other sensitive data recorded in the output. If these parameters are both set to 0, no sensitive data is masked. If
these attributes are not set, they default to 6 to the left and 4 to the right.

JMON does not use the above Environment attributes for masking when using the SCRIPT
NOTE formatter. When using the SCRIPT formatter, you must perform any sensitive data masking within
the scripts themselves.

You can monitor transactions in the Journal in real time or for specified date/time ranges. As long as the Journal
transactions you want to monitor are in the ring of Journals, you can access them using JMON. You cannot use
JMON to access Journals outside the ring of Journals.

You can view the stream output from the utility from the operating system prompt or you can direct the output to a file
using standard Linux commands.

Default behavior

If you run JMON without any runtime parameters, the default behavior of the utility is to read the current date Journal
(or Journals if no issuer institution values are provided) from the start to end of file and display the resulting entries
found in the file(s) using the DEFAULT display formatter. That is, when you run JMON without any parameters, it
assumes the following:

Capture Date=Current Posting Date

Issuer Institution=ALL
Display Formatter=DEFAULT
Masking=true
Realtime Mode=false
Paging Prompting=false
Journal Assigns Display=false
Journal Profiles Display=false

Running JMON

You can run JMON in secure or unsecure mode from a command line or from an ACI-provided run script. You can
use one of the built-in output formats or create your own customized format in a script compiled from the Script
Editor.

By default, the output returned from the JMON utility is displayed at the command prompt (STDOUT).

Using the '>' symbol, you can re-direct the STDOUT content to a file.

152
Executables

ACI provides two executables for the JMON utility. The [Link] runs JMON in secure mode, and the [Link] runs
JMON in unsecure mode. The [Link] executable is installed in the $LIB directory while [Link] can be found
under the $ES_HOME/secure directory.

Run scripts

ACI provides run scripts for each JMON executable. runjmonu executes the [Link] file and runjmon executes the
[Link] file. Run script runjmon is installed in the $BIN directory while runjmonu can be found in
$ES_HOME/secure. You can use these scripts as guidance when writing your own scripts.

Run JMON in unsecure mode

After installation, place an executable copy of $ES_HOME/secure/runjmonu under $BIN and one of
$ES_HOME/secure/[Link] under $LIB. You can use the runjmonu script to run JMON in unsecure mode. You can
edit the runjmonu script to set the runtime parameters as desired for your environment.

IMPORTANT You should never use unsecure mode in a production environment.

cd lgnt/bin
runjmonu

Run JMON in secure mode

Use the runjmon script to run JMON in secure mode. You can edit the runjmon script to set the runtime parameters
as desired for your environment.

cd lgnt/bin
runjmon

You are first prompted to authenticate with your username and password before you can run the utility.

Accessing help for JMON

Use the run script with the -help parameter to display detailed help for running the utility.

runjmonu -help

The help content displayed is essentially the same as that provided in the JMON command reference topic.

User selection display

Whenever you run JMON, the first output displayed is a summary of the parameters you specified when starting
JMON.

153
Because Journal files can contain a large number of records, ACI recommends that you always use filter options to
narrow the number of records displayed in the output.

When you run JMON without any filters, you are prompted with a warning message before continuing as the output
can be excessive as shown in the following image.

When you see this prompt, press C to continue or any other key to exit.

NOTE Refer to the JMON command reference topic for details on setting up filters.

Redirecting JMON output to a file

To redirect the streaming output from JMON to a file instead of the screen, you simply use the standard UNIX
platform redirection symbol '>' and a named text file in the startup command.

Because redirection to a file is controlled by the standard UNIX redirection operator ‘>’, only the standard streamed
output is written to the file, which excludes any errors, warnings, and informational messages generated by the
JMON process.

Also, depending on the JMON options selected, a prompt could be triggered that prompts you to enter a choice.
Therefore, a best practice is to avoid using the -prompt parameter when redirecting to a file.

For example, using the following startup command:

runjmonu -acqinst=*VIS* -issinst=BANK -prefix=484

The screen output looks similar to the following:

154
You could redirect the same output to a text file by using the UNIX redirection operator ‘>’ as follows:

runjmonu -acqinst=*VIS* -issinst=BANK -prefix=484 > [Link]

The screen output will now appear as follows, with most of the output written to the [Link] file in the shell script’s
current working directory:

JMON Journal profiles and assigns output

JMON can list the Journal profiles or Journal assigns accessed in processing the output if you select to run JMON
with the -profiles or -assigns parameters. The Journal profiles accessed are determined by the issuer institution(s)
and capture date selected. The assigns listed represent the data sources accessed in the ring of journals based on
the capture date. This is just informative information provided in addition to the Journal entries returned.

If no issuer institutions are specified, JMON reads the Journals configured for all institutions. If you do not specify a
capture date, JMON uses the current date.

The following image depicts the output when you run JMON with the -profiles and -assigns parameters.

155
JMON DEFAULT output formatting

The following image depicts JMON output when using the DEFAULT format. The DEFAULT format is used if you do
not specify the -formatter runtime parameter or explicitly set it to DEFAULT (that is, -formatter=DEFAULT).

The following table describes the fields displayed for the DEFAULT format.

Field Description
Tran Date/Time The transaction date and time.

MTI The transaction message type identifier.

156
Field Description
Acquirer The transaction acquirer institution.

Issuer The transaction issuer institution.

Card Number The transaction card number.

TC The transaction code.

Amount The transaction amount and currency.

B24 The transaction BASE24-eps internal action code.

Iss The transaction external issuer action code.

Acq The transaction external acquirer action code.

RRN The transaction retrieval reference number.

JMON RAW output formatting

The following image depicts JMON output when using the RAW format (that is -formatter=RAW). This format
displays the unformatted raw data from the jrnl_body of Journal records. Refer to the Transaction Data Element
(TDE) Reference Guide for more information.

157
JMON SCRIPT output formatting

The example outputs depicted in this topic are dependent on the operations performed within the
scripts configured for the JMON session. These examples are based on example JMON scripts
NOTE described in this guide. Any masking of sensitive data must be performed within your scripts. The
PCI_NUM_UNMASK_LEFT and PCI_NUM_UNMASK_RIGHT Environment attributes are not used with
the SCRIPT format.

The following images depict JMON output when using the SCRIPT formatter and compiled scripts.

The first image depicts JMON output when using a data script only with no headers.

158
 A data script is required, but does not necessarily need to output any data. If all that you require is
a summary printed by the trailer script at the end of the run, then you can use the data script to
NOTE
accumulate totals using exported script variables (see the BASE24-eps Scripting Manual for
details)

The next image depicts JMON output with an optional header script.

The following image depicts JMON output with an optional trailer script.

159
For more information on setting the -formatter parameter for scripts, refer to JMON command reference topic. For
sample data, header, and trailer scripts, refer to JMON example scripts.

JMON TDE output formatting

The image in this section depicts JMON output when using the TDE formatter (that is, -formatter=TDE). The TDE
format provides the formatted data from the jrnl_body of Journal records and has the same format as JLFSCAN.
TDEs are identified in the Journal data by their object IDs and TDE names provided on the next line following the
object ID. You can find further information on TDEs in the BASE24-eps Transaction Data Element (TDE) Reference
Guide.

160
JMON command reference

This topic describes all of the optional parameters and filters you can use when running JMON. Some optional
parameters require values and some do not. If a parameter requires a value, the value must be preceded by an '='.
For filters, you can additionally provide multiple values, with each value separated by commas.

You can provide a combination of the following optional parameters and filters to display the Journal records you
want in the desired format.

Each parameter or filter must be entered with a leading '-' (for example, -assigns -prompt).

Optional parameters without values

The following optional parameters are entered without values. For most of these parameters, the default behavior is
the same as the parameter not being provided.

161
-assigns Lists all Journal and Journal Continuation assign names that were processed in the output above
all the Journal entries. The Journal assigns processed depends on how you set other parameters
that determine the journal files processed, such as -issinst, -plusone, or -captrdat. Otherwise, all
assigns for the target day are processed.

-help Displays help text for all optional parameters and filters and quits.

-nomask Enables you to disable the default masking of sensitive data (such as the PAN) set by the
PCI_NUM_UNMASK_LEFT and PCI_NUM_UNMASK_RIGHT Environment attributes.

These Environment attributes do not apply when using the SCRIPT format. Any
NOTE
masking sensitive data must be performed within the scripts themselves.

-plusone Processes the next day’s Journal file in addition to the target day’s Journal file, in order to capture
all transactions for a specific date (for example, where terminals or institutions have already cut
over).

NOTE You cannot use this parameter in conjunction with the -realtime parameter.

-profiles Lists all Journal Profile names that that were processed in the output above all the Journal entries.
The Journal profiles processed can vary, depending on how other parameters are set, such as the
-issinst parameter.

-prompt Enables the utility to prompt the user after each screen page of output. When you provide this
parameter, the following options are available on each prompt:

• Space: Next page

• C/c: Disable further prompting
• Q/q: Quit (exit JMON)

NOTE You cannot use this parameter in conjunction with the -realtime parameter.

162
 -realtime Enables the utility to report transactions in near-real-time. The utility process periodically polls the
Journals to display new transaction records as they are logged. The poll interval defaults to 5
seconds, but you can change it to a different interval using the -poll parameter described below.

Use Control^C to exit. Otherwise, the process continues to run until logical network cutover
occurs. Without this option, the process terminates when the end-of-file is reached on the target
Journal.

You cannot use the -realtime parameter in the same JMON query with any of the following
parameters:

• -captrdat
• -prompt
• -endtime
• -plusone

Optional parameters with values

The following optional parameters accept values.

-captrdat=<YYYYMMDD> Specifies the capture date, which determines the BASE24-eps

Journal(s) from which to fetch entries to display. Defaults to today’s
date if you run JMON prior to logical network cutover, or tomorrow’s
date if you run JMON after logical network cutover.

You cannot use this parameter in conjunction with the

NOTE
-realtime parameter.

-endtime=<YYYYMMDDhhmmss> Specifies the transaction date/time at which to stop fetching

transactions to display for a given capture date or current date if no
capture date is provided. Defaults to the end-of-file.

You cannot use this parameter in conjunction with the

NOTE
-realtime parameter.

163
 -formatter=<FRMT> Displays the retrieved Journal data in the specified format. Valid
<FRMT> values:

• DEFAULT — Displays the Journal data on a single line for each

transaction. If you do not specify the -formatter parameter, the
DEFAULT value is used.
• RAW — Displays the raw unformatted Journal data as it is written
in the jrnl_body of Journal records.
• SCRIPT: <scpt_data>,<scpt_hdr>,<scpt_trlr>

<scpt_data>: Required. Specifies the script that displays the

Journal data.

<scpt_hdr>: Optional. Specifies the script that displays header

data.

<scpt_trlr>: Optional. Specifies the script that displays trailer data.

No spaces are allowed within the SCRIPT string.

• TDE — Displays the formatted data from the jrnl_body of Journal

records in TDE format. Provides the same formatting as
JLFSCAN.

Refer to the JMON DEFAULT output formatting, JMON RAW output

formatting, JMON SCRIPT output formatting, and JMON TDE output
formatting topics above to view sample output for each of the above
FRMT values.

-poll=<seconds> Sets the time interval, in seconds, that the process waits before polling
the Journals for new transactions, until the end-of-file on all Journals is
reached when running in real-time mode. You can only use this
parameter in conjunction with the -realtime parameter.

-starttime=<YYYYMMDDhhmmss> Specifies the transaction date/time at which to start fetching

transactions to display for a given capture date or current date if no
capture date is provided. Defaults to the start of file of each Journal for
the provided capture date (or today if no capture date is provided).

Filter parameters

Filter parameters enable you to control the Journal output to display. When you define one or more filters, only
Journal records that match the provided filter(s) are displayed. You can provide either a single value or multiple
values for any filter. If the latter is desired, you must provide the multiple values in a comma separated list (with NO
blank spaces) as follows:

-<filter>=<value[,<value>,<value>,...]>

164
 -acqinst Filters the output by acquirer institution ID(s).

-issinst Filters the output by issuer institution ID(s).

This parameter does not allow values with special characters and also restricts
NOTE
the journal profiles processed to those profiles configured for the issuer.

-actcde Filters the output by the 3-digit numeric internal action code(s) logged for a transaction. Action
code values must be numeric with a fixed length of 3.

-mti Filters the output by the internal numeric message type identifier(s). MTI values must be numeric
with a fixed length of 4.

-prefix Filters the output by prefix(es). Prefix values must be numeric and can be up to a maximum length
of 28. This variable length allows for the value to vary from the one configured in BASE24-eps for
that prefix. That is, you can specify any number of PAN digits, including the entire PAN, to match
for inclusion in the output.

-txncde Filters the output by the internal transaction code. Values must be alphanumeric with a fixed length
of 2.

-termid Filters the output by terminal ID. Values can be up to 16 characters in length.

Special characters

You can use the following special characters with a filter value, excluding the -issinst filter.

! — Use as a prefix to exclude a specified value from the output. (NOT allowed in a multiple value comma
separated list).

* — Wildcard that represents one or more characters.

? — Wildcard that represents a single character.

Special characters examples with JMON:

runjmon -actcde=!000 (displays all declined transactions)

runjmon -txncde=1* (displays all transactions with a transaction code beginning
with 1)
runjmon -actcde=09? (displays all transactions with an action code that starts
with 09
runjmon -txncde=1?,!00 (NOT allowed)
runjmon -txncde=00,1*
runjmon -txncde=1*,0*
runjmon -issint=BAN* (NOT allowed)

165
JMON example scripts

You can re-use and customize your existing Journal perusal or Journal Query scripts for the JMON utility. ACI
provides a predefined set of perusal scripts at installation.

The scripts you use must be compiled in the Script Editor and reside in the Script Repository.

The following are examples of JMON scripts.

Example JMON header script

The following is an example of a JMON header script named JMTEST_HDR. This script provides headings for the
columns provided by the data script and initializes counters for the total number of transactions, the number of
transactions for two institutions (BANK and NOFI), and the number of transactions for all other institutions. These
variables are incremented in the example JMON data script and printed in the example JMON trailer script.

166
 void JMTEST_HDR;

TEXT ("PAN ");

spaces( 1 );

TEXT ("Issuer Institution ");

spaces( 1 );

TEXT ("Terminal Id ");

spaces( 1 );

TEXT ( "Date and Time " );

spaces( 1 );

TEXT ( "Transaction " );

spaces( 1 );

TEXT ( "Billed Amt " );

spaces( 1 );

TEXT ( "Action" );
spaces( 3 );

TEXT ( "MTI" );
spaces( 2 );

TEXT ("Account Nr ");

spaces( 1 );

TEXT ( "Seq Nr" );

spaces( 1 );

spaces( 8 );
TEXT ( "Rsn" );
spaces( 1 );

#
# Set Exported Script Variables
#
VAR.BIN_SET( "TXN_CNT", 0 );
VAR.BIN_SET( "BANK_TXN_CNT", 0 );
VAR.BIN_SET( "NOFI_TXN_CNT", 0 );
VAR.BIN_SET( "OTHER_TXN_CNT", 0 );

Example JMON data script

The following is an example of a JMON data script named JMTEST_DATA. This script selects and formats the
transaction data to be displayed for each transaction that meets the criteria you used to run JMON. It also
increments the variables initialized in the header script.

void JMTEST_DATA;
#
number lnum;
number dec_plcs;
number bill_amt;
number amt_1;
number amt_2;

167
number amt_3;
number aed_amt;
number amt_orig;
number txn_amt;
number amt_current;
string curr_cde;
#
string func_cde;
string mti_cde;
string iss;
#
string mrc;
#
STRING pan;
NUMBER pan_lgth;
STRING pan_first_part;
STRING pan_last_part;
#
boolean is_admin_txn;
#
is_admin_txn = false;

VAR.BIN_INCR( "TXN_CNT", 1);

if (exists(TDE.PAN_TAG))
{
pan = rtrim( [Link] );
pan_lgth = strlen( pan );
pan_first_part = left( pan, 6 );
pan_last_part = right( pan, 4 );
TEXT( pan_first_part );
fill( "*", pan_lgth - 10 );
TEXT( pan_last_part );
SPACES( 19 - pan_lgth );
}
else
{
spaces (19);
}
#
if (exists(TDE.ISS_TAG))
{
#text( substr ((TDE.ISS_INST_ID),0,4 ));
text(TDE.ISS_INST_ID);
spaces( 1 );
iss= rtrim(TDE.ISS_INST_ID);
switch(iss)
{
case "BANK":
VAR.BIN_INCR( "BANK_TXN_CNT", 1);
case "NOFI":
VAR.BIN_INCR( "NOFI_TXN_CNT", 1);
case DEFAULT:
VAR.BIN_INCR( "OTHER_TXN_CNT", 1);
}
}
else
{
spaces (5);
}

168
if (exists( TDE.TERM_ID_TAG))
{
TEXT( TDE.TERMINAL_ID );
spaces( 1 );
}
else
{
spaces (17);
}
#
if (exists( TDE.TSTAT_TAG))
{
# Log time is yyyy/mm/dd hh:mn:[Link]
# display dd/mm hh:mn:ss
# substr offsets are base 0.
TEXT( substr(TDE.LOG_TIM,8,2) + substr(TDE.LOG_TIM,4,3) + " " +
substr(TDE.LOG_TIM,11,8));
spaces( 1 );
}
else
{
if (exists(TDE.CAPTR_DAT_TAG) )
{
prnt_dat( TDE.CAPTR_DAT,2,true );
spaces( 1 );
}
else
{
if (exists(TDE.LOCAL_DAT_TIM_TAG) )
{
prnt_dat( TDE.LOCAL_DAT_TIM,2,true );
spaces( 1 );
}
else
{
spaces (9);
}
}
if (exists(TDE.LOCAL_DAT_TIM_TAG))
{
prnt_tim( TDE.LOCAL_DAT_TIM,1,true );
spaces( 1 );
}
else
{
spaces (9);
}
}
#
# Check For Admin Transaction
#
if ( EXISTS( TDE.INSTRM_TYP_TAG ) )
{
if ( TDE.INSTRM_TYP == "AD" )
{
is_admin_txn = true;
}
}
#
# Transaction code description
#

169
if (exists(TDE.PROC_CDE_TAG) )
{
switch ( TDE.PROC_CDE_TXN_CDE )
{
case "00":
TEXT ( "Purchase ");
case "01":
TEXT ( "Withdrawal ");
case "02":
TEXT ( "Debit Adjustmnt");
case "09":
TEXT ( "Purch Cashback ");
case "0A":
TEXT ( "Mobile TopUp ");
case "0B":
TEXT ( "Fee Collection ");
case "17":
TEXT ( "Fast Cash ");
case "20":
TEXT ( "Refund ");
case "21":
TEXT ( "Deposit ");
case "22":
TEXT ( "Credit Adjstmnt");
case "24":
TEXT ( "Cheque Deposit ");
case "2A":
TEXT ( "Funds Disbrsmnt");
case "2E":
TEXT ( "Cash Deposit ");
case "30":
TEXT ( "Avail Funds ");
case "31":
if ( is_admin_txn )
{
TEXT ("Admin Bal Txn ");
}
else
{
TEXT ( "Balance Enq ");
}
case "3B":
TEXT ( "Mini Stmnt ");
case "3N":
TEXT ( "Acct Selection ");
case "40":
TEXT ( "Transfer ");
case "50":
TEXT ( "Bill Payment ");
case "90":
TEXT ( "Pin Change ");
case "9I":
TEXT ( "Check Proof Lst");
case "93":
TEXT ( "Admin Day Close");
case "98":
TEXT ( "Admin Cash Adj ");
case "99":
TEXT ( "EMV Script Mgt ");
case "9W":
TEXT ( "Cheque Book Req");

170
 case "9Y":
TEXT ( "Admin Bal Stand");
case "9Z":
TEXT ( "Admin Bal Curr ");
case "A1":
TEXT ( "Log Only ");
case "RD":
TEXT ( "Card Activate ");
case "R1":
TEXT ( "Tran Prepare ");
case "R2":
TEXT ( "Ben Detail Inq ");
case "R3":
TEXT ( "Cust AC Det Inq");
case "R4":
TEXT ( "Bills Inq ");
case "R5":
TEXT ( "Beneficiary Inq");
case "R6":
TEXT ( "Manage Reg Bill");
case "R7":
TEXT ( "Bill ACCT Sel ");
case "R8":
TEXT ( "Pay All Bills ");
case "R9":
TEXT ( "Billers Inq ");
case "RA":
TEXT ( "List Val Inq ");
case "RB":
TEXT ( "Cust Acct Inq ");
case "RC":
TEXT ( "Biller Valid ");
case "RE":
TEXT ( "Statement Req ");
DEFAULT:
TEXT (TDE.PROC_CDE_TXN_CDE);
spaces( 13 );
}
}
else
{
if ( ([Link] == 1304)
|| ([Link] == 1314) )
{
TEXT ( "File Update ");
if exists (tde.fnct_cde_tag )
{
func_cde = itoa(tde.fnct_cde );
switch ( func_cde)
{
case "300":
text( substr ((func_cde),0,3 ));
case "301":
text ("ADD");
case "302":
text ("CHG");
case "303":
text ("DEL");
case "304":
text ("RPL");
case "305":

171
 text ("INQ");
case "306":
text ("FRP");
case "307":
text ("FAD");
case "308":
text ("FDE");
case "309":
text ("CAD");
DEFAULT:
text( substr ((func_cde),0,3 ));
}
}
else
{
TEXT ( "NFC");
}
}
else
{
spaces (15);
}
}
#
# Extract the Transaction BILLED Amount
#
if( ([Link] >= 1400)
&& ([Link] < 1500) )
{
#
# When first two bytes of the MTI are "14":
# Set amt_orig using TDE.AMT_ORIG
#
aed_amt = 0;
if (exists (TDE.AMT_ORIG_TAG))
{
amt_orig = tde.amt_orig;
}
else
{
amt_orig = 0;
}
amt_current = 0;
bill_amt = 0;
if (exists( TDE.TXN_AMT_TAG ))
{
amt_current = tde.txn_amt;
bill_amt= tde.txn_amt;
}
else
{
amt_current = bill_amt;
}
if ( (amt_current == 0)
|| (amt_current == amt_orig) )
{
aed_amt = bill_amt;
}
else
{
txn_amt = amt_current - bill_amt;

172
 if (txn_amt == 100)
{
aed_amt = amt_current;
}
else
{
amt_1 = 0;
amt_2 = 0;
amt_3 = 0;
aed_amt = 0;
if (TDE.TXN_AMT_CRNCY_CDE == 784 )
{
aed_amt = amt_current;
}
else
{
if (exists ( tde.txn_amt_tag ) )
{
amt_1 = TDE.TXN_AMT;
}
else
{
amt_1 = amt_current;
}
if (exists(TDE.ACQUIRER_AMOUNT_TAG))
{
amt_2 = TDE.ACQ_AMT;
}
else
{
amt_2 = amt_orig;
}
}
}
}
prnt_crncy( bill_amt, 10," ", 2 );
spaces( 1 );
spaces( 4 );
} # End Reversal
else
{
if ( exists( TDE.ACQUIRER_AMOUNT_TAG ) )
{
dec_plcs = 2;
prnt_crncy( TDE.ACQ_AMT , 10," ",dec_plcs );
spaces( 1 );
curr_cde = itoa(TDE.CURR_CDE );
text( substr ((curr_cde),0,3 ));
spaces ( 1 );
}
else
{
if ( exists( TDE.TXN_AMT_TAG ) )
{
dec_plcs = 2;
prnt_crncy( TDE.TXN_AMT , 10," ",dec_plcs );
spaces( 1 );
curr_cde = itoa(TDE.TXN_AMT_CRNCY_CDE );
text( substr ((curr_cde),0,3 ));
spaces ( 1 );
}

173
 else
{
spaces (15);
}
}
}
#
# Extract the mcc
#
if ( exists( TDE.CRD_ACCPT_BUS_CDE_TAG ) )
{
# rt_Just(TDE.CRD_ACCPT_BUS_CDE, 4 );
# spaces( 1 );
}
else
{
# spaces (5);
}
#
# Response Code
#
if (exists(TDE.ACT_CDE_TAG) )
{
lnum = tde.act_cde;
if ( lnum == 0 )
{
fill( "000", 3 );
}
else
{
rt_just( tde.act_cde, 3 );
}
spaces(1);
SWITCH ( tde.act_cde )
{
#
# Approved
#
CASE 000:
text( "App " );
CASE 600:
text( "App " );
CASE 083:
text( "OAR " );
CASE 082:
text( "Don " );
CASE 096:
text( "Act " );
CASE 107:
text( "Ref " );
CASE 300:
if ( [Link] == 1314 )
{
text( "App " );
}
else
{
text( "Dec " );
}
CASE 400:
text("Rev ");

174
 DEFAULT:
text( "Dec " );
}
spaces( 1 );
}
else
{
text( " NRP " );
}
if (exists(TDE.MTI_TAG) )
{
mti_cde = itoa([Link]);
}
else
{
if (exists(TDE.MTI_ORIG_TAG) )
{
mti_cde = itoa(tde.mti_orig );
}
else
{
mti_cde = "UNKN";
}
}
text( substr ((mti_cde),0,4 ));
spaces (1);
#
#
# Transaction Account Number
#
if ( (TDE.PROC_CDE_TXN_CDE == "2E")
|| (TDE.PROC_CDE_TXN_CDE == "24") )
{
if (exists(TDE.ACCT2_TAG))
{
lft_just( TDE.ACCT2_NUM,28);
spaces( 1 );
}
else
{
spaces (29);
}
}
else
{
if (exists(TDE.ACCT1_TAG))
{
lft_just( TDE.ACCT1_NUM,28 );
spaces( 1 );
}
else
{
spaces (29);
}
}
if (exists(TDE.SEQ_NUM_TAG))
{
text( ltrim(TDE.SEQ_NUM) );
}
spaces (1);
#

175
 if (exists(TDE.MSG_RSN_CDE_TAG))
{
mrc = itoa(tde.msg_rsn_cde);
}
else
{
mrc = " ";
}
text( substr ((mrc),0,4 ));

Example JMON trailer script

The following is an example JMON trailer script. It simply displays the number of transactions included for the
variables initialized in the example header script.

void JMTEST_TRLR;

#
# Report
#
end_line;
TEXT ("Number of transactions included: " );
rt_just( VAR.BIN_GET( "TXN_CNT" ), 7 );
END_LINE;
TEXT ("Number of BANK Issuer transactions: " );
rt_just( VAR.BIN_GET( "BANK_TXN_CNT" ), 4 );
END_LINE;
TEXT ("Number of NOFI transactions: " );
rt_just( VAR.BIN_GET( "NOFI_TXN_CNT" ), 11 );
END_LINE;
TEXT ("Number of Other Issuer transactions: " );
rt_just( VAR.BIN_GET( "OTHER_TXN_CNT" ), 3 );
END_LINE;
#
# Clear
#
VAR.BIN_CLEAR( "TXN_INLCUDE_CNT" );
VAR.BIN_CLEAR( "TXN_EXCLCUDE_CNT" );

Problem determination methodology

Issues can be, on a broad basis, separated out as startup issues or runtime issues. A startup issue is one where
either you are bringing up the system for the first time or are trying to do something new for the first time on a
running system.

A runtime issue is when something used to work in a running system but does not work anymore.

Most startup issues are related to either configuration being incorrect or of functionality not available in the current
implementation of the product.

In startup issues, it is often useful to narrow down the location of the issue so that there is a shorter time to problem
determination. The commands and processes described in the previous sections should help you narrow down the
issue.

176
When doing problem determination, it is a good idea to take things one step at a time, especially for startup issues.
In any transaction path terminating within the BASE24-eps application, there is at least one hop over ICE-XS and
one hop to the application. In a transaction path not terminating within the application, there are at least four hops
(inbound-outbound-inbound-outbound) over ICE-XS and at least one hop over the application.

1. Determine if you have connectivity to and from the endpoint where your transaction is originating from or going
to. This is communication-level connectivity, and netstats and nofxs should help you figure this out. Errors and
exceptions should be noted in the syslog.d file where ICE-XS writes its error messages.
2. If you do have connectivity but are not sure if ICE-XS is delivering the message to the queue that you desire,
bring down the application and send the transaction. If the message shows up in a queue where you expect your
application to read the message, then ICE-XS does not have a problem. You can use amqsbcg to determine the
message. Now with the message still in the queue, bring up your application. If the message disappears from
the queue then your admf file and the run script for the application are fine. If there is still an issue, you may see
an exception in the exception log or an event written to the application event file in $LOG.
3. If you are consistently seeing timeouts on your messages in the application, your most likely culprit is SIS_TZ
not matching your system TZ.
4. Once you have ruled out ICE-XS from the mix, the application could have a configuration problem (tables not set
up correctly) or may not have the desired functionality.

Runtime problems fall in two categories:

• Related to time and volume and so on

• Something changing in the environment

Runtime problem troubleshooting typically requires a very good understanding of the particular application and the
environment.

Typical runtime problems related to time and volume include:

• Growth in memory footprint of the application (meaning that the application has a memory leak),
• Application hanging (this could be a startup or a runtime issue)
• Performance degrading over a period of time
• Other problems

For this class of problems, determine the factor in the system and environment that is providing the constraint.
Process stacks and core dumps of running processes would be useful in these scenarios.

If nothing has changed from the transaction’s perspective, and we are not able to see any obvious application issue
like the application abending or hanging, then there may be something that has changed in the environment. Look
for any upgrades or downgrades in the system, modifications or tuning of parameters, and so on.

Moving USEC/Version Checker

The default installation places USEC and Version Checker (VC) on the same Linux/UNIX box as the IS and c-tree
server. Some installations have their ESWeb-Java Server installed on a different Linux/UNIX machine and want their
USEC/VC on this same machine.

Below are instructions to move USEC and VC from the default machine to the machine where the ESWeb-Java

177
Server is installed.

1. Tar up ESUI directory.

cd $ES_HOME

tar cf [Link] ESUI

2. FTP to new Linux/UNIX machine.

ftp acio-cissun18

cd <ES_HOME>

binary

put [Link]

quit

3. Un-tar the ESUI tar file on new Linux/UNIX system.

cd $ES_HOME

tar xf [Link]

4. Modify USEC’s [Link] file.

cd $ES_HOME/ESUI/USEC

change VERSIONCHECKER "host" location to new Linux/UNIX machine.

5. Modify ESWeb’s [Link] file.

cd $ES_HOME/ESWeb/ESConfig

change the "[Link]" location to the new Linux/UNIX machine.

6. Modify Desktop’s [Link] file.

Use Windows File Explorer and navigate to your ES\Desktop directory.

change VERCHECKER "Host" location to the new Linux/UNIX machine.

7. If Java is in a different directory path, do the following:

cd $ES_HOME/ESUI/SharedComponents

change the “JREBin” and “JREHOME” to the Java directory paths

178
Miscellaneous troubleshooting tips
This section contains a number of troubleshooting tips for you to use.

Version Checker
Note that version checking is disabled when running any component as a windows service. To version check the jar
files against the jar repository, you must stop the service and run it using the shortcut or batch file. Once version
checking is complete, you can stop the component and restart the service.

User interface configuration

The following topics provide tips for configuring the user interface.

Accessing another application from the same desktop

To access another application using the same desktop, modify the [Link] file. Go to the Desktop folder on your
PC, modify the [Link] file and add the SecondService entry:

[Hosts]
ESNCService=acio-cissun18:25320/ESWeb/ESHTTPServlet
SecondService=<HOST>:<PORT #>/ESWeb/ESHTTPServlet

<HOST> is the host name or IP address on which the application is running. <PORT #> is the application http port
number.

Once you have stopped and started the desktop, the Connection button is displayed. Click it to select which
connection you want to use.

Change the desktop title

The title that appears at the top of each desktop window consists of two parts that are separated by a hyphen, as
shown in the following example:

To change the first half of the title (that is, ES UI 05.2), edit the [Link] file. Revise the Title entry, as
shown below.

179
 # TITLE
# -----
# The name for the Application
Title=

ES UI 05.2

//TODO Developer, please complete the language

[source,<language>]

To change the second half of the title (that is, Production), edit the [Link] file. In the [Desktop] section, change the
Host entry from Production to the desired text. In the [Hosts] section, change the first entry so it matches the change
you made in the [Desktop] section, as shown below.

[Desktop]
Logo=
AppPath=E:\ESUI\__ES_CritFix05.2_NSK\ES\Desktop\Applications
HelpSet=ACI_Help.hs
Host=

Production

.
.
.
[Hosts]

Production
=K9:29974/ESWeb/ESHTTPServlet

Removing the desktop sounds

To permanently remove the sound when the desktop starts up, perform the following steps. This can only be done
after the initial startup of the desktop.

Edit the <machine name>.cfg file within the Desktop folder on the PC, add the following line to the [DESKTOP]
section, and save the <machine name>.cfg.

SoundTheme=:

This disables the startup sound for this particular desktop. To disable the sounds for every desktop communicating
with an ESWeb process, the above line must be added to the [Link] file within the OSS environment under
the ESWeb/ESConfig subdirectory.

The sound settings established in the <machine name>.cfg and [Link] files can be overridden for individual
desktops through the Preferences > Sounds dialog, which is located on the File menu.

User security issues

The following procedures are available when users have locked their account or need to have their password reset.

180
User has locked their account

If a user has locked out their User Security account due to inactivity or incorrect password attempts:

1. Log onto the ACI desktop with a different user that has access to the User Security Profile UI.
2. Read up the User ID of the user.
3. Change the status on the first tab to Active.
4. Save the record.

Another procedure to reset the User Security account can be used if there are no other User IDs that can be
found to log onto the ACI desktop that have User Security Profile access. This procedure uses DALCI and
should be used only if there is no other way to gain access to the ACI desktop. Perform the following:

Move to the bin subdirectory where the installation was done.

Source in the env_vars file to the current Linux/UNIX Shell.
rundalci
update USERS set UserStatusCode = "A " where UserID == "<user ID>";
update USERS set BadLoginTries = 0 where UserID == "<user ID>";
exit

This resets the user’s status back to active and reset the BadLoginTries to 0.

At this point, the user should be able to log into the UI again.

Password recovery if a user has locked out their account

If the password needs to be reset:

1. Log into the ACI desktop with a different user that has access to the User Security Profile UI.
2. Read up the User ID of the user.
3. Assign the user a new password on the third tab.
4. Save the record.

Another procedure to reset the user account can be used if there are no other user IDs that can be found to log
into the ACI Desktop that have User Security Profile access. This procedure uses DALCI and should be used
only if there is no other way to gain access to the ACI desktop. Perform the following:

Move to the bin subdirectory where the installation was done.

Source in the env_vars file to the current Linux/UNIX Shell.
rundalci
update USERS set UserStatusCode = "A " where UserID == "<user ID>";
update USERS set BadLoginTries = 0 where UserID == "<user ID>";
update PASSWORD set Password = "jJ57hwD+gMN4RD8Cijmv6xPSYAA=" where UserID ==
"<user ID>";
exit

This resets the user’s status to active and resets their password to the value newuser01. The user can log into
the UI again and change their password to a unique value.

181
Time zone description
The standard form of the time zone description (POSIX) uses the following variables.

zone A three or more letter name for the time zone in normal (winter) time.

[-]offset A signed time telling the offset of the time zone westwards from Greenwich. The time
has the form hh[:mm[:ss]] with a one- or two-digit hour, and optional two-digit minutes
and seconds.

dst The name of the time zone when daylight saving is in effect. It can be followed by an
offset telling how big the clock correction is other than the default of one hour.

start/time,end/time Specifies the start and end of the daylight saving period. The start and end fields
indicate on what day the changeover occurs. These fields must be in one of the
following formats: J n: The Julian day n (1 ⇐ n ⇐ 365) ignoring leap days (that is, there
is no February 29).

N: The zero-based Julian day (0 ⇐ n ⇐ 365) including leap days. M m.n.d: This
indicates month m and the n -th occurrence of day d (1 ⇐ m ⇐ 12, 1 ⇐ n ⇐ 5, and 0 ⇐ d
⇐ 6, where 0=Sunday). The fifth occurrence means the last occurrence of that day in a
month. For example, M4.1.0 is the first Sunday in April and M9.5.0 is the last Sunday in
September.

The time field indicates the time the changeover occurs on a given day. See examples below:

TZ=GMT0 This is Greenwich Mean Time (GMT).

TZ=CET-1CEST,M3.5.0/2,M10.5.0/3 This is Central European Time, one hour east from Greenwich.
Daylight saving starts on the last Sunday in March at 2 a.m. and ends
on the last Sunday in October at 3 a.m.

TZ=GMT0BST,M3.5.0/1,M10.5.0/2 This is British time, daylight saving starts and ends at the same
moment as CET, but in an earlier time zone.

TZ=EST5EDT,M4.1.0/2,M10.5.0/2 This is U.S. Eastern Standard Time, five hours west from Greenwich.
Daylight saving starts on the first Sunday in April at 2 a.m. and ends on
the last Sunday in October at 2 a.m.

External connection configuration

To configure the External Connection:

1. Navigate to the External Connection user interface.

182
2. Select the Process ID.
3. Insert a row.
4. Add the process name and description. Be sure to use the same name for both the Process ID and the External
Connection ID.

183
5. For the C++ XML Server to be configured successfully, two entries on the EXTRCNCT table must be added.
◦ BASE24-ES/C++XMLSERVER
◦ BASE24-ES/C++XMLSERVER FORMATTED
6. Display the General Details tab. Enter BASE24-es/C++ XML SERVER. Using the next box and list, select the
process ID you added, Connection Description and Message Format as indicated below. Increase the Retry
Attempts to two and Maximum Connections to 10.

184
7. Display the Protocol Details tab. Select Protocol Code TCP/IP, Protocol Options .HDR2. (don’t forget the first
and last period), the IP address and Port Number of the Linux/UNIX Server, and the Max Connections Allowed
value of one. Click Save.
8. Display the General Details tab. Enter BASE24-ES/C++XMLSERVER FORMATTED and select the process ID
you added. Repeat steps 6 and 7 to add the General Details and Protocol Details for this entry.

185
 In order for these external connection settings to take effect, the WebSphere Application Server
NOTE
must be restarted.

External connection issues

If a user can log into the UI successfully, but is unable to access the Environment UI, the External Connection values
must be verified.

To verify the External Connection values, launch the External Connection Configuration UI:

System Operations -> Server Management -> External Connection

Select the following value for Connection ID: BASE24-ES/C++XMLSERVER

The second list (Process ID) must contain the value of the ESWeb process ID within the Java Server environment.
For Linux/UNIX and CICS systems, this value must be BASE24-ESUI. If this value is not contained in the list, it must
be added via the Process Configuration UI by clicking on the Hyperlink label named Process ID from this window.

Verify the values of Destination IP Address and Port Number on the Protocol Details tab. These values must match
the IP address of the system and port number where the XML server receives its requests.

Repeat the above steps for the Connection ID: BASE24 ES/C++XMLSERVER FORMATTED

Out of memory errors when accessing multiple UI windows

The following error is displayed when you are trying to launch a UI window and there is insufficient memory
available:

186
There is not enough memory to satisfy your request. Please close all open windows and
try again.

Correct the error by editing the [Link] file. In that file, locate the [Link]
parameter and add - Xmx96m to the end.

Example:

[Link]=-DSC=true - DchannelTrace=true - DSCH=true - Xmx96m

Determining the SIS version of an executable

You can determine the version of SIS used for any BASE24-eps executable using the -siver parameter. Knowing the
SIS version can be helpful for troubleshooting and when reporting problems to HELP24.

1. Enter the name of any BASE24-eps executable following by the -siver parameter at the Linux/UNIX command
prompt:

TSPESPA:ACW5:lib 45> ./[Link] -siver

or

$ /ccm_wa/swm/b24/es/Server,intc_aix_2.1.0/eshome/lib/[Link] -siver

The SIS version and build date are returned to the screen.

SIS version: sis/int_zos_2.1

SIS build date: 03 Mar 2015

or

SIS version: sis/int_aix_2.1

SIS build date: 03 Mar 2015

What to do next:

Convey the SIS version information when reporting a case to HELP24.

Firewall issues
If there is a firewall between the desktop and the Version Checker server, some additional configuration must be
performed on the Version Checker server. There are two situations where these additional parameters are needed.

• A firewall that blocks ports

• A firewall that performs Network Address Translation (NAT) mapping of IP addresses from internal to external

For a firewall that blocks ports, the following parameter needs to be added to the [Link] file within the
[Registry] section:

187
RMIServerPort=<port number>

By default, RMI uses arbitrary port(s) to establish connections directly between a client and a remote service. In this
case, a firewall would block arbitrary ports. If specified, RMI uses this static port when clients communicate the RMI
services. If not specified, arbitrary ports are used. The port number specified in the RMIServerPort parameter needs
to be opened up through the firewall, as well as the Registry port.

For a firewall that performs Network Address Translation, the following parameter needs to be added to the
[Link] file within the [Registry] section:

RMIServerHost=<external IP address or name>

188
Section 14. Maintenance on the Linux/UNIX
platform
In a BASE24-eps system on the Linux/UNIX platform, you should periodically perform certain maintenance tasks to
keep the system running in a healthy, effective manner.

Periodically perform the following tasks to maintain the BASE24-eps system, keep it running effectively, and
maximize performance:

• Database cleanup
• User audit log (UALOGD/USRAULOG) cleanup

Database cleanup
BASE24-eps provides numerous process control CLEANUP commands or Java properties to clean up expired or
inactive data that is no longer needed.

BASE24-eps contains a separate process control CLEANUP command or set of Java properties for each data source
with date-sensitive data. Run these commands or configure the Java properties so that expired or unused records
do not consume excess space for the data source over time. If you do not run them, the performance of BASE24-
eps degrades and consumes system resources.

You can automate the execution of the CLEANUP commands. By definition, the Java properties automate the
cleanup processing.

Process control CLEANUP commands

BASE24-eps provides the following components to run the process control CLEANUP commands:

• Balance Impacts Cleanup component

• Context Cleanup component
• Journal Query Cleanup component
• Preauthorization Hold Cleanup component
• Rolling Usage Cleanup component
• Stop Payment Cleanup component
• Usage Cleanup component
• Voice Authorization Cleanup component

For detailed information about process control CLEANUP commands, see the BASE24-eps Process Control User
Guide.

Java properties
BASE24-eps provides the following components to automatically perform cleanup processing based on Java
property configuration:

189
 • User Audit Log Cleanup component
• Database Cache Cleanup component
• Message Context Cleanup component
• Event Adapter Cleanup component

For detailed information about Java properties for cleanup processing, see the BASE24-eps Java Server Reference
Guide. .

Manage MQ queue context entries

If a transaction message sits in the context queue for a period defined by the OLDREC setting in [Link] , it is
automatically marked as expired and not read on subsequent read (MQGET) calls, when BASE24-eps (SIS MQDS)
reads the context. Without proper setup, these expired messages can accumulate.

To automatically clear the expired messages:

• For Linux/UNIX, set the parameter ExpiryInterval under the section TuningParameters for your MQ manager in
the [Link] file.

NOTE Ensure the local MQ administrator sets up an appropriate queue depth for the context queue.

Due to the nature of context processing, the context queue of a healthy system should have few messages, as the
messages are written and read quickly. However, any timeouts or problems connecting or getting responses from
external entities (host or else) can result in expired messages if the problem persists more than the number of
seconds defined in OLDREC.

User audit log (UALOGD/USRAULOG) cleanup

BASE24-eps provides several Java properties that you can configure to retain User Audit Log data source
(UALOGD) records for as long as you need or as is required (for example, at least a year per PCI requirements).

Optional user auditing enables you to audit all online operator activity performed through the ACI desktop user
interface. The database auditing mechanism enables you to track the following information to the application
database by user ID and time of change:

• Adds
• Updates (before and after images)
• Deletes
• User log outs
• Password verifications
• Password changes

When a Transaction Security Services (TSS) file is audited, records are written to a User Audit Log File (UALOGD)
data source. You can configure the files audited when your ACI system is installed and control the extent to which
files are audited.

190
 • When you configure full auditing, then header information and detail information about the before and after
images of the affected record is logged to the UALOGD.
• When you configure medium auditing, only the header information is logged to the UALOGD.

The Java User Interface servlets (ESWEB) process writes user auditing records directly to the UALOGD. The C++
User Interface (XML Server) process can either write auditing records directly to the UALOGD or write NOWAIT
messages to an Audit Store and Forward File (ASAFD) for transmission to the User Audit application, depending on
the value of the USE_LOCAL_USER_AUDIT Environment attribute.

• If this attribute is set to a value of Y, the process writes auditing records directly to the UALOGD.
• If this attribute is set to a value of N, the process writes NOWAIT messages to the ASAFD for transmission to
the UAUD application. In this case, the Audit Store and Forward Configuration window on the ACI Desktop
user interface enables you to configure how the ASAFD is used (for example, the maximum number of records
that can be outstanding in the ASAFD at one time).

When the UAUD application receives an auditing message, it routes the message through WebGate to the ESWEB
process to be logged to the UALOGD.

By default, the setting is for the XML Server to write auditing records directly to the UALOGD.

The UALOGD, which is the input file for daily audit reports, has several Java cleanup properties to control how long
records are retained and how often old records are removed. You can set the [Link]
property to 365 (one year) as long as you confirm the file is large enough, which depends on the activity in the logs.
The following are the Java cleanup properties:

• [Link]
• [Link]
• [Link]
• [Link]
• [Link]

See the BASE24-eps Java Server Reference Guide for detailed descriptions of these Java properties.

When you run the ESWEB process under OSS (on platform), the properties file used is
no_ems_config.properties. This properties file is necessary because processes started
NOTE
under OSS cannot log to EMS. Thus, you need to configure the above properties in the
no_ems_config.properties file.

Ensure you have the correct process ID set-up (either on the Configuration Properties window or, for
versions 07.4 and older, in the /{ESHOME}/ESWeb/ESConf/<prefix>/<node>/[Link]. For
example, [Link]=BASE24-ESUI (Older versions P1A^ESWEB01).

You also must have at least one [Link]* specified (either on the Application window, or, for version
07.04 and older, in the {ESHOME}/ESWeb/ESConf/<prefix>/<node>/[Link]. For example,
[Link]=[Link]).

You also must confirm that the [Link] in the [Link] matches the
-[Link]= entry in the RunESWebTacl and/or RunESWeb files under OSS or the -[Link]= entry in the
[Link] for off platform. Otherwise, the cleanup will not work.

191
You then have to stop/start the ESWEB process to pick up the new configuration.

192
Section 15. Miscellaneous operational tips
This section contains operational tips and additional information about utilities.

ACI desktop terminal services implementation

This section addresses the configuration options used to support the running of the BASE24-eps UI Desktop(s) from
one machine over Windows Terminal Services remote connections.

A BASE24-eps desktop can be installed on a server machine that enables multiple users to connect via terminal
services. A desktop can be executed independently (one per user) of other instances (processes) of the same
desktop executable(s). To partition preferences, log files, and configuration, the following “local” config (or desktop
config) properties have been introduced. These properties can be set at the time of a desktop being installed for the
sole purpose of sharing over terminal services.

Desktop section
TerminalServices=<true | false

This property must be set to true if the desktop is installed for the purpose of terminal service accessibility

If true, the local config (desktop config) and system default preferences do not get written out. Each startup of the
desktop begins with the same host connection and system default preferences for every user.

This does not reflect “user” specific preferences. Those can still be saved to a configured path
NOTE
location. See the PreferencesPath property described below.

If false, the desktop handling of the configuration and preferences are unchanged. They are saved accordingly so
that the next startup reflects the user’s connection settings and system preferences.

Example:

TerminalServices=true

PreferencesPath=<directory path name

This property should be specified if users connecting through terminal services are limited to write access to specific
location(s), which should be a common location on the machine or network.

If the specified path name does not exist, the desktop attempts to create it. The desktop must have write access to
the specified path.

The default order to determine where a user’s preferences are stored is as follows:

• Specified preference path

• By local config (desktop config ) path location
• By home directory

Example:

193
PreferencesPath=c:\ACI_Desktop\prefs

PreferencesDisabled=<false | true

This property defaults to false, meaning user preferences are always enabled and saved.

If set to true, user preferences cannot be modified or saved at all.

Example:

PreferencesDisabled=false

Common section
LogPath=<log directory path

This property specifies the Log Path location of where the desktop’s log files (.err, .dbg, .ent) are created and used.

If the directory does not exist, the desktop attempts to create the directory path.

This property should be specified if terminal service is enabled and the specified path should enable write access to
the desktop.

If not specified, the log files are created in the default home location.

Example:

LogPath=c:\ACI_Desktop\logs

LogFileName=<log file name

This property specifies the name of the log file prefix to which the .err, .dbg and .ent files suffixes are appended. The
three files are created based on the log file name within the LogPath location specified (if any).

If Terminal Service is enabled (true), then the <log file name> specified has the following token appended to it:
<USERNAME> (if not already present within the log file name specified). At the time the log file name is obtained,
the <USERNAME> token is replaced with the system property “username,” as set by the JVM. This enables each
terminal service user to create and utilize their own desktop log files without impacting other instances of desktop log
files.

Example:

LogFileName=desktop

System property available at startup

-DNoNavTree=<true|false

If true, every time a desktop starts, the Navigation Tree collapses (not present). If false, the Navigation Tree is
dependent upon the default preference setting to determine if it is visible or not.

194
 The User Preference Window has the OK button disabled anytime it is determined that Preferences
NOTE
cannot be saved when TerminalService is enabled (set to true).

Examples:

• Prior to logging on.

• If the PreferencedDisabled property is true.

Summary (if TerminalServices is enabled)

1. Local Config file (Desktop Config) is not saved. The same default host connection is used every time the
desktop starts up for every terminal service user.
2. System name (default) preferences are not enabled and therefore cannot be saved. All users shares the same
default preferences prior to logging in.
3. There is an option to remove the Navigation Tree at startup. The DNoNavTree=true system property must be set
if desired.
4. The LogPath and PreferencesPath properties enable a user to specify a directory path (whether it exists or not)
for log files and user preferences to be created (stored ), respectively. The directories must be accessible for
creation, reading, and writing.
5. Once a user is logged on, user preferences can be saved for each logged on user within the Preferences Path
location. This is disabled only if PreferencesDisabled=true is specified, otherwise it functions as described.
6. Log files are created and written to under the specified LogPath. The log file name is modified with the
<USERNAME> token and file suffixes, respectively (.err, .dbg, .ent). Once the Log files are to be created, the
system username replaces the <USERNAME> token.

Linux/UNIX platform-specific tuning considerations

Enable direct I/O on AIX
On an AIX platform, if a Journaled files system (JSF and JSF2) is used, the operating system caches the files stored
on that file system to reduce the disk access frequency.

Since the database also caches the file in memory, there may be a possibility for memory contention and
degradation of performance if the database files are created on the Journaled file system.

To resolve this, enable Direct I/O on the Journaled file system. This enables the database files to be on files systems
while bypassing the operating system’s buffer cache. This is accomplished by using the mount option dio.

Security considerations
This section addresses some of the data security considerations in a BASE24-eps production installation. It is not
intended to be a definitive guide to securing all data in the production environment. The material in this section is
entirely focused on certain aspects of data security as it relates to passing of information between the various
components of the BASE24-eps application.

195
Overview of BASE24-eps processes
The BASE24-eps application is a distributed application with multiple processes interacting with each other. There
are different types of interfaces to be considered as part of data security.

• On all Linux/UNIX platforms, the various BASE24-eps application processes either use the IBM MQ series
product or an application-level protocol over TCP/IP for inter-process communication (IPC). This is true
irrespective of the number of Linux/UNIX servers that are present in the deployment configuration.
• The ICE-XS process from the ACI Communication Services for BASE24-eps acts as a communications handler,
providing endpoints with a communication interface (TCP/IP, X.25, SNA) on one side and the IBM MQ interface
on the other side to communicate with other BASE24-eps application processes.
• The ATM Device Handlers, where deployed, use the JMS MQ interface to communicate with BASE24-eps
application components.
• The database of choice on the Linux/UNIX platforms is c-tree. The BASE24-eps applications use a “client-
server” model using TCP/IP to communicate with the c-tree database server. This is true irrespective of whether
the c-tree database server is deployed on the same Linux/UNIX server that the application is running on or if the
c-tree database is running on a different Linux/UNIX server.
• If the GoldenGate replication product is a part of this deployment, this presents another type of interface which
uses TCP/IP to communicate across servers.
• Another category of interface is based on the specific Hardware Security Module (HSM) device that is used in
this particular deployment.

There are different considerations involved in securing the various interfaces described above and the need for data
security should be balanced with attention to ease of operations, performance, and availability considerations. Prior
to production, the specific sets of procedures used at a particular deployment needs to be worked out by all the
stakeholders involved.

Recommended security settings

Below are some of the procedures to secure a deployment. This is not an exhaustive list.

Linux/UNIX-related recommendations

Tighten the Linux/UNIX file system security privileges on each of the files comprising the BASE24-eps deployment
by following the steps below:

• Create a secure Linux/UNIX user and group and ensure that all the BASE24-eps, c-tree, and GoldenGate
executables have the same user and group ownership. This way, only one user can execute the applications on
the system.
• Ensure that all BASE24-eps, c-tree, and GoldenGate executables have only - r-x------ permissions.
• Ensure that the configuration files have only -r- - - - - - - - permission.
• Set rwxr-x-- permission to LOG, CTREE LOG, GoldenGate LOG directories and set only r-xr-x- - - permission to
all other directories. This restricts the user from moving/deleting the existing files and creating new files with the
same names (mainly configuration files).
• Ensure that the umask settings on the system are set appropriately so that the database files are created with
only - rw------- permissions.
• Ensure that UNIX System Auditing is turned on so there is an Audit log which can be used to trace back.

196
 • Disable all File transfer access (like FTP, RSH, SCP) on the system. If needed temporarily, remember to always
disable them after such use.
• Enables only sudo access to the normal user (operator) with access to a restricted set of Linux/UNIX commands
(without access to commands like chmod, vi, rm, umask).
• Secure the servers so that only the right set of operational staff are able to access the system, files, network
connections, and so on.

IBM MQ-related recommendations

Ensure a high degree of trust and security is placed on the staff performing the administration of the IBM MQ in the
system. Assign the “mqm” group privileges to a limited number of trusted staff, as any user with mqm group
permissions has full access to all MQ Administration functions and all MQ objects within the system, as well as to the
security control through the MQ’s Object Authority Manager (OAM).

Use the setmqaut OAM command to grant only the required permissions to the group/user with privileges to run the
application. The setmqaut OAM command can be used to restrict access to the queue manager itself by this
group/user. This ensures that a non-authorized user cannot create a new queue on this queue manager.

For BASE24-eps implementations spanning multiple Linux/UNIX servers and/or involving the use of a WebSphere
MQ cluster, channel security exits can be written to secure the cluster receiver channel, thereby preventing
unauthorized queue managers from joining the cluster.

When using MQ client connectivity you must verify that the security on the client channel is set according to the local
infrastructure policies

SSL can be enabled for the MQ cluster communication with the penalty of increased CPU consumption and
transaction latency.

ACI does not provide documentation for these configurations. See the information provided at the
NOTE
link below for further assistance on this product:

[Link]
[Link]/support/knowledgecenter/SSFKSJ/mapfiles/product_welcome_wmq.htm]

For channel exit information, search for "Channel exit programs”.

For cluster workload information, search for "Cluster workload user exit".

c-tree- and other database-related recommendations

There currently exists a proprietary handshake mechanism between the c-tree client library and the c-tree server
used to validate the client and the server. This handshake mechanism has been implemented to ensure that non-
ACI c-tree clients cannot communicate with or interrogate the c-tree server deployed by ACI.

The following steps can also be used to further secure the system:

• Ensure that the [Link] (or equivalent set file) has only read permissions for the owner of all the application
executables and no other user has any permission to read or use this file. This ensures that only the right set of
application processes can use or see this file.
• Ensure that the metadata files are secure and, specifically, that the [Link] file uses the above settings file in

197
 the parameters section and does not have a c-tree server username and password in this file.

Ensure that the [Link] and the dalcom applications are secure in the system. After deployment and initial testing,
these should either be removed from the production environment or highly secured. These executables give a SQL-
like interface to the c-tree database and can be used to query, extract and modify information in the c-tree data files.

External communications-related recommendations

Critical fields in the messages to and from the HSM are encrypted, as per requirements laid out by the HSM vendor.
The actual manner in which the communication to the HSM device is initiated and maintained is dictated by the HSM
vendor. If there are specific concerns in this area, these should be addressed directly to the HSM vendor.

The number of ports to be opened for external endpoints to connect to the ICE-XS process in the system needs to
be discussed. There is no limit on the number of connections that can be handled by a single ICE-XS process;
therefore it is possible to limit the number of ports opened. However, there are aspects of operational manageability
to be taken into account when determining the right number of ports required. These would vary by deployment and
must be discussed prior to going live with the BASE24-eps system.

Considerations for running TDAL

Describes how the TCP/IP DAL (TDAL) process is started and the options and parameters that control it.

When runtdal is run, it starts a module called silisten . At the same time, the port is assigned from the -port
parameter. When connections come in from any AJI (Java) processes for database connections, a [Link]
(assigned from the -object parameter) is created for each connection. The parameters inside [Link]
contain the information needed for [Link] for the metadata and database connection.

The following two examples show the [Link] and silisten modules in memory:

[Link] 4 /users/b24cnt/bin/[Link] off

silisten -port=56008 -object=/users/b24cnt/lib/[Link]

-argfile=/users/b24cnt/bin/[Link] -trace=off

In the first example, the argument "4" is the socket descriptor passed from silisten to the newly spawned [Link].
The argument tells TDAL where to send responses and where to take new queries from.

The total number of [Link] programs running on a system is determined by the [Link] and
[Link] settings in AJI applications (such as ACIJMX, ESWeb, or EvtAdapter). For each
DBConnection object created by AJI, there will be a new [Link] program running.

The following table describes each of the parameters that can be passed to [Link].

-port Opens the port.

-object Creates a new instance of [Link] .

198
-trace Used for debugging purposes.

Places each database activity into the

$LOGS/[Link] file.

This can get very large. Do not use the

NOTE
-trace parameter unless told to do so.

-uncommitread Plays a role in handling of DB errors, most notably,

instead of triggering the reconnection logic, it just returns
an error (fs_abort) and expects the client to re-create to
the datasource (much like AJI does).

199
Section 16. Performance and availability
considerations
This section addresses some of the performance and availability considerations for a BASE24-eps installation.

The intent of the section is to explain some of the principles and software architecture forming the basis on which a
particular deployment is determined.

There are a large number of variables feeding into the process to determine any specific deployment and this
section is not intended to be an exhaustive account, but a guiding first step.

There are also aspects of this section relating to the previous section and security implications in a BASE24-eps
deployment.

The actual determination of the layout of a deployment is reached through discussion between the ACI project teams
and the customer. This section is not intended to replace this dialog. Rather, the intent is for the reader to attain a
broad understanding in the determination of an optimal deployment.

BASE24-eps has a loosely coupled, distributed architecture which enables distributing components of the application
to different servers. It is even possible to have hybrid environments with certain caveats that must be true.

The intent of the BASE24-eps architecture and the various tunable components and/or parameters in the system is
to enable a consistent end-to-end transaction latency while managing reasonable availability requirements and CPU
costs in the system.

The material presented in the following topics is organized on the basis of the number of servers and the number of
sites involved in a particular deployment. Each topic builds on principles and considerations laid out in prior topics
and an attempt is made not to repeat information. Therefore, it would benefit readers of this document to read this
section in sequence.

Components of the BASE24-eps application are described only as is relevant to performance and availability
considerations. There is no attempt here to provide exhaustive detail of the functionality provided by each these
components.

Single server - single site deployment

Single server - single site deployment is the simplest of the deployment models for the BASE24-eps product.

The deployment model is typically used in test and certification configurations. It is also used in production systems
where the transaction volume is relatively low and there is only a single site involved in the deployment. It is also
possible to handle a fairly large volume of transactions by purchasing a large Linux/UNIX server with a large number
of CPUs. There may be some consideration of this model as a better option than going with multiple servers.

In either case, discussion has revealed that hardware and/or network failure is unlikely or not a very high priority.
Also, replication of data from the online c-tree database is not a requirement.

In this model, all parts of the BASE24-eps application including the c-tree database server, the ICE-XS
communication handlers, the BASE24-eps applications, the Web Server to run the UI Server, and the ATM Device
Handlers (where applicable) all run on one physical hardware box.

200
On Linux/UNIX platforms, the components of the BASE24-eps applications that are a part of the online transaction
include the following:

• The ICE-XS communication handler that communicates to the endpoints downstream and, in the case of a
switched environment, to the upstream endpoints
• IBM MQ provides the Inter Process Communication (IPC) and messaging infrastructure, enabling
communication between the ICE-XS process and components of the BASE24-eps application
• The BASE24-eps Integrated Server executable ( [Link] )
• Any HSMs and the associated ICE-XS communications handlers that may be involved in communication with
these devices
• The c-tree database server ( ctsrvr )
• The shared timer process ( sitimrp ) holding all shared timers in the system
• Where applicable, the ATM Device Handlers managing the interface to a particular class of devices

There are a number of other components of the BASE24-eps application that are also a part of the deployment.
These include the following:

• The ACI desktop infrastructure deployed as a rich client communicating to a web server with access to the c-tree
database.
• The End of Period Processing (EOPP) component managing the processing required for end of day settlement.
• The DALCI executable providing an SQL-like interface to the c-tree database. It is recommended this not be
used in production as it is very costly from an overall system performance perspective and has implications for
the performance of the c-tree database server.
• The Real Time Feed process providing a near real time feed of the Journal data to external processes to
perform back-office processing (as an example).
• The SAFMGR process managing Store and Forward of specific transactions and so on.

Infrastructure components
Certain components of the deployment which provide the basic infrastructure for the BASE24-eps applications need
to be managed by external entities (hardware, software, or both). Specifically, these include the availability of the
following components:

• Hardware platform
• Operating system
• Network infrastructure
• Disk I/O infrastructure
• Web Server
• IBM MQ
• c-tree server

If any of these components fail, the BASE24-eps application components that rely on them, will fail in an
unpredictable fashion.

201
It is recommended that system monitoring be a regular part of the operational procedures, and used to monitor the
availability of each of these components and perform the appropriate actions as discussed and documented in the
requirements for this deployment.

Inherently, these components are fairly stable and certain things can be done to help the availability of these
components. For example, the minimum hardware configuration for BASE24-eps applications requires two CPUs
and eight gigabytes of memory.

Having more than one network card installed on the Server with each network card configured over a different part of
the network infrastructure is also recommended. This ensures that a failure of one network card or failure of one part
of the network infrastructure does not remove the server from the network.

Data should be stored on a high speednetwork-attached storage (NAS)with some form of RAID implemented to
ensure that failure of a single disk or disk controller does not take the whole database or the files offline.

It is strongly recommended for single server production deployments that prior to production, this
NOTE
aspect of availability be discussed and documented.

IBM MQ configuration from a performance perspective

There are certain performance considerations from an IBM MQ perspective worth noting.

The BASE24-eps application uses the IBM MQ for IPC. It is critical from a performance perspective to ensure this
process is as fast as possible. This dictates some of the considerations in the setup, configuration, and tuning of the
IBM MQ deployment.

You can install the MQ server locally on the same machine as BASE24-eps or remotely, letting BASE24-eps connect
as a client. Client connection is not recommended for cross-site communication; it is only for connectivity within
same site where the relative distance between the BASE24-eps server and MQ server is minimal. While it may be
possible to port the application to use the MQ client API and use an off-platform server to run the MQ server, the
performance cost is fairly high and hence, this option is, by design, not used in BASE24-eps.

All MQ queues used within the system are non-persistent memory-based queues unless specified as persistent
queues. Where persistent queues are used internally within the IBM MQ product, there is considerable disk I/O
involved in the operation. This is more expensive from a transaction-latency and CPU-cost perspective.

BASE24-eps does not rely on MQ’s guaranteed delivery mechanisms. Instead, BASE24-eps operates under the
end-to-end business transaction consistency rules applying to all financial transaction systems. These cover more of
the potential areas where transactions could be lost than can be covered by the MQ product’s facilities.

There are some specific tuning parameters that we recommend for the IBM MQ product.

Where the ACI personnel create and setup the queue managers in the deployment, the following tuning parameter is
copied into the [Link] file specific to the queue manager(s) used by the BASE24-eps applications. This needs to be
set up prior to creation of high throughput queues.

Tuning parameters

DefaultQBufferSize=512000 (for all UNIX platforms)

DefaultQBufferSize=25952256 (Linux platform only)

202
The DefaultQBufferSize parameter is specified in 4K pages and represents the maximum size of a buffer that MQ
uses to keep queue content in memory.

If the queue manager is being created by someone outside of the BASE24-eps automated install program, this
parameter should be set up in the [Link] file prior to setting up the queues for the first time. This is not a parameter
that can be tuned once queue manager is running and the queues have already been set up. The only way to modify
this parameter after a queue manager is running is to delete the queue where this needs to be applied, stop the
queue manager, apply the parameter, start the queue manager, and recreate the queue.

This chunk of memory is now blocked off for this high throughput queue.

In small systems where available memory is not very high, this would be one of the parameters needing to be
managed.

ICE-XS communication server process for Linux/UNIX

The ICE-XS communication server process is a multithreaded process that enables concurrency in handling
transactions. It must run at a high priority.

The ICE-XS server is typically a very small percentage of the cost of the transaction from both the transaction
latency perspective and from the CPU cost perspective. Therefore, manageability and operational ease is of greater
significance than performance alone.

Configure a separate instance of the ICE-XS server for each type of endpoint and class of device. If there are many
of the same type of devices, you should have some criteria for dividing these into separate chunks and assigning
each chunk to a separate instance of the ICE-XS server. The issues to be weighed here are manageability of the
interfaces and devices against manageability of some number of Linux/UNIX processes.

BASE24-eps also has the capability to handle a 95xx connection message from ICE-XS in the case of asynchronous
communication links. As part of handling this message, BASE24-ps can dynamically handle changes in MQ
message queue configurations associated with the communication link to an endpoint. This feature enables an
endpoint to potentially establish links to two separate instances of the ICE-XS server, thus increasing the availability.

When a link is established, the action of establishing a connection or disconnection results in generating 95xx
connect and disconnect messages. This process enables the application to update its internal routing table to point
to the queue associated with the instance of the ICE-XS process that is currently connected to the endpoint. This
facility is currently unavailable for endpoints requiring synchronous communication; for example, the HSM devices.

NOTE
Do not run a single instance of the ICE-XS communication server process to manage all communication
interfaces and stations to the system.

Also, when you have a large number of interfaces and stations configured in a single ICE-XS server process, the
number of threads the server spawns to handle the workload increases. Based on the workload, this increase in
the number of threads is beneficial from a performance perspective, up to a certain threshold, which varies by the
specific Linux/UNIX platform. Beyond the threshold, as the number of threads is increased, the performance
degrades rapidly because of contention over CPU resources.

Datasources in the BASE24-eps product

There are different types of datasources that are a part of the BASE24-eps application architecture.

203
All of the data typically resides in a physical database. On the UNIX platform, the physical database of choice is c-
tree. The proper configuration, tuning, and management of the c-tree database and the data files on disk are critical
to the optimal performance of the system.

The BASE24-eps product has two different types of memory-based data sources available. These are read-only
data sources, which typically hold data organized as a Hash Table or a Radix Table. The factors driving a particular
table to be configured as a Hash Table or a Radix Table are dependent on which form provides the easiest access or
best performance in a particular situation.

These datasources are typically used to store static configuration data that does not change frequently in the
system.

Data is configured into a c-tree database table. If this table is configured to be stored as a Hash Table or a Radix
Table, the application creates a disk image to the corresponding Hash or Radix table on disk. This happens either at
application startup or when a#Deliver# command is issued.

After startup when each thread issues the first open on the Hash or Radix data source, the corresponding disk
image is loaded into memory for that thread. An#Alter# command forces each thread to reload the current disk
image of the datasource into memory.

Each application thread of the process uses its own copy of the Hash or Radix tables in memory.

There are two factors to consider here.

The fact that each application thread has its own memory image of the Hash/Radix tables implies that the memory
footprint of the application is large and has a potential to grow larger if more threads are configured or used. This is a
factor used in the initial sizing of the system and among the factors involved in the configuration done for
deployment.

The second factor relates to the frequency, the number of tables, and the volume of data that is changing during a
warm boot of a system.

While the application has been designed to minimize the impact of a warm boot on the online transactions by having
the application threads update their memory image one at a time, the application still absorbs a performance hit
since the number of threads available to service the workload is reduced for the duration of this process. So it is
advisable to schedule warm boots of the application to periods of time when the transaction volume is low.

c-tree server performance and availability considerations

c-tree server availability considerations

The c-tree server is an important part of the infrastructure required for the functioning of the BASE24-eps
application. In single server environments, the c-tree server process is co-located with the BASE24-eps applications.
There is typically only one instance of the c-tree server running in this environment. If there is any outage of the c-
tree server, all of the BASE24-eps applications fail except the ICE-XS communication handler processes. At this
time, incoming transactions would be processed by the ICE-XS communication handler and placed into the
BASE24-eps application input queue. The messages would continue to get queued waiting for the application
processes to be restarted.

Monitoring availability of the c-tree server is critical here to ensure the system handles this particular condition and is
able to restart not only the c-tree server but also all of the BASE24-eps application processes that terminated when
the c-tree server process went down.

204
In-flight messages are lost or timed-out in this scenario. The “Business Transaction” integrity however is maintained.
c-tree uses the concept of a transaction log file which would be replayed when the c-tree server is brought back up.
The “Business Transaction” would be handled as an atomic operation which would either be applied in its entirety or
not at all into the physical disks that comprise the database table.

c-tree server performance considerations

The tuning parameters available in the c-tree database server configuration are described in Section 11. The specific
tuning parameters applicable to a particular deployment depend on a number of different criteria, including available
hardware, memory and disk sizes, and the specific business needs that are to be solved. These would typically be
set up and tuned by the ACI project team prior to production.

Modifying these parameters should only be attempted by personnel who are very familiar with the c-tree database
server administration concepts and are highly cognizant of specific solution/deployment. In any case, changes to the
parameters should undergo extensive regression testing with a specific focus on availability and performance before
they are rolled into a production environment.

In a single server environment, the single instance of the c-tree database server handles different types of
workloads, including online transactions, batch transactions, and potential UI transactions. Care should be taken to
ensure that from an operational perspective, these different workloads do not interfere with each other.

A typical online transaction holds a lock on specific tables and rows for a period of time typically in the order of a few
milliseconds.

A batch transaction could potentially hold a lock for a longer period of time, which is completely dependent on the
number of records/tables that are affected.

Avoidance of extensive UI transactions operating on the c-tree database server which is handling a reasonable
production load is recommended. Depending on the type of operation being performed, this operation could hold a
critical lock that can have severe implications for both the performance and the availability of the BASE24-eps
applications.

All these factors are usually considered part of the original sizing of the solution. If there is a change, the tuning
parameters and assumptions about the CPU costs or memory requirements need to be revisited to ensure the
changes do not have a severe negative impact on the production environment.

File partitioning and data layout considerations

In addition to the tuning and availability considerations of the c-tree server process, a significant factor for
consideration is the partitioning of critical files in the database and the physical layout of these files in the disk
subsystems.

In the BASE24-eps product, each table consists of a separate data file with the actual physical records comprising
the table as well as a number of index files carrying information about the potential indexes configured on the table.

It is recommended that all of the c-tree database configuration files and the actual data and index files be placed on
anetwork-attachedstorage (NAS) to ensure there is minimal I/O latency in the system and to enhance the
performance of the system. It is also recommended that the c-tree log files (denoted by the LOCAL_DIRECTORY
configuration parameter in the c-tree configuration file) be isolated from the BASE24-eps data and index files. These
two file groups should be located in two separate disk subsystems.

Additional factors include having some kind of RAID implemented on these NAS disk subsystems to ensure

205
availability of the I/O infrastructure and to mitigate product outages caused by bad disks or disk controllers.

In addition, other mechanisms are available to partition files in the system. These are typically performed on critical
files. Configuration of these partitions is usually performed by the ACI project teams at the time of deployment. Some
regular examples of these partitioning mechanisms are explained below.

• Partitioning of the Journal Files. This is typically done to ensure that you do not run into contention issues on a
single journal disk file during transaction processing. The basic intention is to normalize the I/O load across
multiple files. For better performance, multiple physical layouts of the files should be managed to ensure they all
do not reside on the same physical disk subsystem. The actual partitioning scheme can be based on the
business considerations at the particular deployment. This partitioning mechanism is a part of the static
configuration in the system at the time of deployment.
• For files undergoing refreshes or updates, there is a second file partitioning mechanism available. Here, there
are typically two versions of the data file available: the currently active data file and the backup version of this
data file. The application layer is insulated from this partitioning by an abstraction layer in between which points
a virtual data source name to the current active version of the data file. The refresh or update is typically done
on the backup version of the data file and then a switch is done in the abstraction layer to point to this new
version of the data file. The tables that are typically partitioned in this mechanism are CARD and USAGE. This
configuration is typically done by the ACI project teams at the time of deployment.

Setup and configuration of the BASE24-eps application processes

The factors discussed in this section relate to availability and performance considerations associated with the
number of threads configured for the BASE24-eps application processes and the number of instances of the
processes configured in the system for availability.

Most of the application processes operating in a BASE24-eps deployment run with a single application thread.
However, the architecture of the BASE24-eps application processes enables them to run with multiple threads or
have multiple instances running in the system.

NOTE
Due to the nature of the processing they perform, the following application processes must always run single-
threaded:

[Link] (End-of-Period Processing)

[Link] (Store and Forward Manager)

[Link] (Transaction based Pricing)

The multiple threads provide for concurrency of transaction processing. This takes advantage of the Symmetric Multi
Processing (SMP) capabilities of the Linux/UNIX operating system variants that we currently support.

The multiple instances of processes provide for availability of critical processes in the system. Multiple instances
ensure that when one instance of an application either fails or is brought down for a specific reason, the other
instances of the application continue processing the transaction load.

ACI benchmark results have shown that the best performance is attained when the BASE24-eps Integrated Server
(IS) process is run with 12 threads per instance. Performance starts degrading when more than 12 threads are
added to an IS process. Configuring multiple instances and breaking down the number of threads configured
sometimes helps balance the transaction load well and provides for a better performance characteristic. If more
threads are needed, it is better to add IS processes than to increase the number of threads beyond 12 per process.

206
Running with multiple threads or multiple instances requires some setup and configuration steps to be performed.
The configuration is typically decided by the ACI project team and takes into account the requirements for a
particular deployment.

Prior to deciding whether a particular process needs to run “multi threaded” or have multiple instances running on a
system, there are specific factors in need of analysis. Having multiple instances or multiple threads in processes
increases the memory resources required by the system. The setup and configuration of the environment is more
complex. Also, having multiple threads or multiple instances of the process cause the Linux/UNIX operating system
scheduler to work harder in the system. All these factors need to be balanced with the requirements of concurrent
processing, end-to-end transaction latency requirements, and availability requirements for the system.

If an instance of an IS process goes down for any reason, it can typically be restarted without a major availability and
performance impact to the operations. However, it is still recommended to have at least two instances of the IS
process running in the system.

For certain application processes, it makes sense to have multiple instances of the application threads configured.
Typically, these are the BASE24-eps Integrated server process ([Link]), the BASE24-eps UI server, and the ATM
Device Handlers (if these are a part of the deployment).

The number of threads is configured in the numthrds parameter passed on the command line or part of the startup
script for the application process. For the Java-based applications (UI server or the ATM Device handler), this is
configured as the “MaxConnNum” parameter in the LISTENER or equivalent file for the process.

The BASE24-eps Integrated Server process is the prime conduit for all of the online transactions. From an
availability perspective, at least two instances of this process are usually configured in the system.

There is one additional factor determining the number of instances or threads we configure in the system. Each
thread configured uses two client connections to the c-tree server. The number of available c-tree clients licensed is
an additional factor for consideration.

Multiple servers - single-site deployment

In this model, there are at least two Linux/UNIX servers involved.

This model is referred to as a High Availability Configuration in the BASE24-eps sizing request that was sent for
this particular deployment. The hardware requirements for this model are higher than the requirements for a Single
Server since each of the available components of the solution needs to have the ability to handle the full load.

This model is typically used when there is a single site in the deployment but additional consideration has been
given to availability requirements. This is a more typical production type of deployment.

Analysis of Single Points of Failure needs to happen in conjunction with the customer requirements and the actual
deployment model needs to be discussed and agreed upon prior to deployment. What is deemed as an “available”
component in the system varies by deployment and to some extent is based on the specific requirements.

There is one obvious “Single Point of Failure” not considered in this deployment model. This would be anything
taking the site out of commission.

Since this is a single site, any factor causing the whole site to go out of commission needs to be monitored and
reported upon. Also, potential “down time” needs to be considered and mechanisms need to be put in place to
ensure this meets or beats the customer’s requirements from this perspective.

207
Based on the requirements, the customers could choose to go to an Active/Passive model or an Active/Active
model. Among the factors involved in making this choice is the requirement of time within which failure is detected
on an Active system and the switchover of transaction traffic to the other system happens.

All of the factors mentioned from an availability perspective in the previous section hold true for this model of
deployment also. In addition to the factors mentioned above, certain additional considerations are applicable to a
multi-server environment.

Network considerations for a multi-server deployment

Various aspects of operation of the ACI BASE24-eps product rely on client-server communication over TCP/IP. For
example, the BASE24-eps application components communicate with the c-tree database server using a client-
server model of communication.

IBM MQ relies on TCP/IP based communication when communicating between servers.

These components rely on a high speed local network (LAN) which is highly available. There are mechanisms built
in to ensure recovery on a temporary network outage. However, this does come at the cost of performance and, in
some cases, availability of the BASE24-eps applications.

The actual mechanisms for ensuring network performance, availability and reliability are outside the scope of
discussion in this document.

This is a critical consideration and part of the infrastructure requirement for a BASE24-eps deployment.

IBM MQ configuration
BASE24-eps can connect to the IBM MQ infrastructure using server or client binding.

Server binding

BASE24-eps expects to have a local installation of IBM MQ on each server where an application component of
BASE24-eps is running. The only exception to this rule is if the c-tree database server is the only component of the
BASE24-eps deployment that would ever run on the particular server. In this case, we do not need to have IBM MQ
installed on this system.

BASE24-eps typically uses the MQ Cluster technology to set up and configure a network of MQ managers in the
deployment. The reason for choosing the MQ cluster technology is to help ease the MQ configuration in deployment.

Certain customers have chosen to implement the IBM-distributed queuing mechanism to achieve the same
configuration. Distributed queuing configurations are typically more complex to set up.

There are advantages and disadvantages to each method of operation. The discussion of these is outside the scope
of the current document.

If the distributed queuing mechanism is chosen, ACI typically encourages this implementation be performed by a
skilled MQ Administrator who is a part of the customer’s staff.

There are some specific tuning parameters that we recommend for the IBM MQ product in a cluster configuration.

Where the ACI personnel create and set up the queue managers in the deployment, the following tuning parameter
is copied into the _qm.ini _ file specific to the queue manager(s) used by the BASE24-eps applications.

208
Channel:

AdoptNewMCA=ALL

AdoptNewMCATimeout=60

AdoptNewMCACheck=ALL

PipeLineLength=2

ExitProperties:

CLWLMode=FAST

If the queue manager is being created by someone outside of the BASE24-eps automated install program, this
parameter should be set up in the [Link] file prior to setting up the queues for the first time. This is not a parameter
that can be tuned once queue manager is running and the queues have already been set up. The only way to modify
this parameter after a queue manager is running is to delete the queue where the parameter needs to be applied,
stop the queue manager, apply the parameter, start the queue manager, and recreate the queue.

There must be an MQ server for each BASE24-eps server.

Client binding

When using client binding, you can configure the deployment to use the Linux/UNIX environment variable
MQSERVER or CCDT (client channel definition table) or any other client connection method preferred by the MQ
administrator.

c-tree server performance and availability considerations

In a multi-server configuration, the c-tree database is a potential candidate to be made an available component. One
important reason for this has been mentioned in a previous section. Outage of the c-tree server causes all BASE24-
eps applications to fail.

There are different deployment models that could be used based on specific customer requirements. The choice
should be made based on specific customer requirements and the option decided prior to configuration and
deployment.

This model is referred to as Local Contingency.

Active/passive c-tree server deployment

In this model, there is only one instance of the c-tree server active at any time. This instance is called the Primary
instance of the c-tree Server.

All applications connect to this c-tree server for data base access. The c-tree configuration files and the physical
data files are placed on a high-speed NAS. All Linux/UNIX servers where a c-tree database server could potentially
run should have high speed network access to this NAS. In addition, the directory structures or paths set up to
access these files should be identical to all of these UNIX servers.

In this scenario, when the primary c-tree server goes down, this needs to be immediately detected and another
instance of the c-tree server needs to be started either on the same Linux/UNIX Server or on an alternate

209
Linux/UNIX Server. The c-tree server needs to be deployed and set up on this alternate Linux/UNIX server, ready to
go.

On startup, this process uses the same configuration and activates the same set of database tables from the same
directory path as was configured for the primary server. This new instance would then replay the transaction logs to
ensure the data files are in a “consistent” state, and then become available for operation.

All BASE24-eps application processes that were connected to the Primary c-tree server could potentially have failed
when the c-tree server went down. These processes would then need to be restarted and would then connect to this
new instance of the c-tree server which would now become the primary database server.

The detection and restarting of the c-tree server can be automated by using a Linux/UNIX Cluster failover
mechanism. These mechanisms vary by vendor.

Customers should use Sun Cluster software on the Sun Solaris Platforms and High Availability Cluster
Multiprocessing (HACMP) software on the IBM AIX platforms.

Multiple servers - two-site deployment

In this model of deployment, there are at least two sites involved with a degree of geographic separation.

The number of Linux/UNIX servers deployed could vary from one server per site to any of a number of different
servers at each site.

The availability considerations could apply between the two sites or they could be layered with availability options
within each site with the alternate site providing an additional availability and failover mechanism.

This model is labeled as Maximum Availability Configuration in the BASE24-eps sizing request.

All of the availability and performance considerations of the previous sections apply to this model.

c-tree server performance and availability considerations

In the multi-site deployment, each site needs to have at least one active c-tree server.

Even if there is only one primary site processing transactions, the secondary site should have a c-tree server active
and participating in data replication. GoldenGate for data replication is a requirement in this model of deployment.

ACI strongly recommends against having the BASE24-eps applications connect to the c-tree database server over a
Wide Area Network (WAN). Only the replication data should be transported over the WAN.

This model is referred to as Remote Contingency.

As before, the distribution of traffic between the two sites is external to the deployment of ACI software and beyond
the scope of this document.

Also, the lag in replication over the WAN could be a critical factor that needs to be monitored and managed.

Active/active c-tree server deployment

In this model, there are two active instances of the c-tree server deployed on two different UNIX Servers.

210
It is possible to have only one site with one instance of the c-tree server actively processing transactions, whereas
the other site with the other instance of the c-tree server participates as a replicated system. In this model, the disk
subsystem is not shared and thus adds a layer of availability, since two physical copies of the data are available at
any time. This model requires the deployment of the GoldenGate product with at least one-way replication.

In this model, when the primary site fails, the transactions are immediately pointed to the backup site. The BASE24-
eps applications are also started up right when the primary site failure is detected. Since the c-tree database server
is already active, this reduces the downtime in the switchover.

Network considerations

In addition to the requirements from the previous sections, another consideration is the speed, availability, and
reliability of the Wide Area Network (WAN) between the two sites. This is a primary consideration because the
GoldenGate replication software relies on the WAN to keep the two sites in sync.

211