Monitoring, Reporting and

Maintenance
Version 6.7.191
Table of Contents
Legal and Compliance Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Copyright SEEBURGER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Allowed use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Prohibited use. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Classification of Relevant Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Operating System, Database, JVM, BIS Runtime System and Process Executions . . . . . . . . . . . . . 7
Errors and Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Analyzing Results and Coming to a Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Tracking System Behavior, Performance and Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Check List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Check List: Guidance and Thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Check List Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Tools and Services: Gathering Relevant Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Runtime Command REST API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
General Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Enabling the REST Endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Usage Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
BIS Portal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Control Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Scheduler Execution State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
SEEBURGER Support Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Configuration options for specific collectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Collector process-errors-radar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
JBoss JMX Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Server Info . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Destination Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Instance Metrics (Prometheus) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
List of Prefixes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Example Metrics Explained. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Host Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Command Line Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Automatic Historic Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Collection Best Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Configure Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Configure log-level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
 Configure log settings during Installation and Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Prevention from log file poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Centralized Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Diagnostic Database Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Processes by States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Number of JMS Messages (Sorted by Queue Name) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Processes by Exec. Time (Sorted by Type, State and Prio.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Examples of Errors and Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Not Enough Tablespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
OutOfMemory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Process Inspector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
TPM-Lookup: Missing Entity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
BIS Inspector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
XPath: Missing Field. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
XPath: Invalid System Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
XPath: Invalid Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
XML-Schema: Missing Element . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
StoreService: Missing Access Rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
DT Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Example Adapter Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Mail Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Invalid Mail Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
SAP Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Connection to Gateway Broke Down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Connection to SAP Gateway Was Interrupted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
FTP-Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
Login Fault:. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
FTP-Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Wrong Password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
AS2-Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
No Connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
AS2-Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Unknown Partner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
OFTP Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
No Transfer Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Further Classification of Communication Faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Database Connection Interrupted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
 Scope

Legal and Compliance Notices

Copyright (c) [year] SEEBURGER AG ([Link] All rights reserved.

Scope
SEEBURGER products and services, including all associated content such as images, text, web
content, and all types of software-related documentation and manuals (hereinafter "SEEBURGER
Materials"), are protected by global intellectual property laws, including copyright laws. SEEBURGER
Materials are owned by SEEBURGER AG, a SEEBURGER-affiliated company, and/or third parties who
have granted SEEBURGER the right to use their content. SEEBURGER Materials may only be used in
accordance with the guidelines below or as covered by an agreement between you and SEEBURGER.

SEEBURGER cannot grant usage rights for content owned by third parties. Any reproduction,
modification, distribution, public display, licensing, or sale is permitted only within the scope of the
explicit license terms provided. Unless such a license is granted, any use of the content is prohibited.
For further information, consulting a copyright attorney is recommended.

Copyright SEEBURGER
Allowed use
SEEBURGER Customers may use SEEBURGER Materials in accordance with their licensing or other
agreement with SEEBURGER. SEEBURGER Partners may use SEEBURGER Materials in accordance
with their Partner agreement with SEEBURGER.

You may use, share, and include the SEEBURGER Materials in AI training, provided that the use is
non-commercial, the SEEBURGER Materials remain unmodified, and are shared only within your
company.

You must include the following statement to identify the SEEBURGER copyrighted work in your
materials: “© [year] SEEBURGER AG or an SEEBURGER affiliate company. All rights reserved.”
(Replace “year” with the publication year of the materials.)

Prohibited use
Your use may not be obscene or pornographic, and you may not be disparaging, defamatory, or
libelous to SEEBURGER, any of its products, or any other person or entity.

You must not use any automated data gathering or extraction methods designed to scrape or
extract data, including for text and data mining purposes.

Your use may not directly or indirectly state or imply sponsorship, affiliation, or endorsement of your
product or service by or with SEEBURGER.

You acknowledge that any rights granted to you herein do not constitute a transfer of title. You do
not obtain any ownership rights, title, or other interest in SEEBURGER Materials or trademarks by
downloading, copying, or otherwise using the SEEBURGER Materials.

Notes
The hereby provided information is only a brief cutout from the full "Legal Information" document.
For comprehensive details regarding legal matters, refer to the "Legal Information" document.

SEEBURGER expressly declares that the document "Legal Information" (also delivered with your BIS

Monitoring, Reporting and Maintenance 4

 Notes

installation media) is an integral part of the documentation.

Monitoring, Reporting and Maintenance 5

 Motivation

Motivation
This document aims to prevent the BIS 6 system from unnecessary down times by monitoring and
reporting bottlenecks, identifying possible pitfalls of process designs, observing improper system
and database configurations (which lead to lack of performance), and also to detect possible bugs,
errors or exceptions (which may occur in some situations).

In general, for many issues and system bottlenecks, there are hot fixes, patches, workarounds,
solutions and/or configurations already available. In order to improve the effectiveness of your
operations, as well as communicating efficiently with SEEBURGER Support, we highly recommend
that you:

• Keep your system up-to-date (with latest patches and releases).

• Monitor the performance of your system and database.
• Check system limitations and resource shortage.
• Document and fix errors and exceptions.
• Have the necessary information at hand in case of opening an incident.

Every BIS system installation is unique (customized), if a problem occurs the SEEBURGER Support
team responds in the shortest time possible. If you are able to provide detailed and accurate
information on demand, it helps to keep the down time at a minimum, and prevents (in a worst case
scenario) the system from crashing.

Development is an ongoing process, and hot fixes, patches, as well as new releases, improve the
performance and robustness of the system.

We recommend new users, who still have limited BIS 6 system knowledge, to start working with the
BIS Front-end, and collect some experience first, before going into depth and operating the JMX
console and the database.

This document does not contain all relevant information which may be needed
by SEEBURGER Support. Also the information on trouble shooting is not provided
 in details, and we recommend contacting the SEEBURGER Support or
Professional Service, or consulting the SEEBURGER Knowledgebase, if you have
questions, or experience serious problems.

Monitoring, Reporting and Maintenance 6

 Classification of Relevant Information

Introduction
The BIS system provides various types of information for monitoring and analyzing the system.
Mostly the relevant data is accessed through tools and services varying on the heterogeneous
platform (mainly for those which are platform-dependent). This document does not feature possible
cases, but it describes examples on various systems, and how the relevant information can be
located.

The next section describes the tools and services that will help you to locate the required
information.

Classification of Relevant Information

The tables below give an overview of relevant information sorted by different categories and
information, which is relevant for the performance and analyzing the system behavior. In addition, a
list of errors and exceptions is given. There are also suggestions on tools and services which are
related to a particular version.

Operating System, Database, JVM, BIS Runtime System and Process

Executions
Relevant Information Classification Tools/Services Version
CPU (average) load OS Task Manager, perform (depends on OS)
(Windows);
nmon, top, (Unix)
BIS System Front-end: Dashboard since BIS 6.3.2
Free RAM OS Task Manager, perform (depends on OS)
(Windows)
BIS System Front-end: since BIS 6.3.2
Monitoring Dashboard
Active threads OS Task Manager, perform (depends on OS)
(Windows)
BIS System Front-end: Dashboard since BIS 6.3.1
VM Thread dump since BIS 6.2.2
Active workers VM Thread dump since BIS 6.2.2
Available bandwidth OS Task Manager, perform (depends on OS)
(Windows®);

nmon, (Unix)
Java VM Thread states VM Thread dump since BIS 6.2.2
Queued JMS messages BIS System Metrics since BIS 6.5.2
Exec. time of SQL Database SQL query (depends on DB)
statements
Database Automatic Database Oracle
Diagnostic Monitor
(ADDM)
Garbage collection VM GC log, jstat, jcmd, (depends on VM)
metrics
BIS System Front-end: Dashboard since BIS 6.3.2

Monitoring, Reporting and Maintenance 7

 Analyzing Results and Coming to a Conclusion

Relevant Information Classification Tools/Services Version

Number of exec. BIS System Front-end: Process since BIS 6.2.2
processes, Inspector, metrics
Trial/Process execution Database SQL query, metrics since BIS 6.2.2
time

Errors and Exceptions

Relevant Information Classification Tools/Services Version

Adapter errors BIS System Adapter Error Monitor since BIS 6.2.2
Available disk space OS / FS dir (Windows), df (Unix) (depends on OS)
Available file handles OS / FS ulimit (Unix) (depends on OS)
Available network OS / Network Task Manager, perform (depends on OS)
bandwidth (Windows);
nmon, top, (Unix)
Available tablespace Database SQL query (depends on DB)
Database Database Control Oracle
Center
In-Doubt transaction Database SQL query (depends on DB)
BIS System Log files since BIS 6.2.2
Failed process BIS System Front-end: Process since BIS 6.2.2
executions Inspector
BIS System Log files (depends on DB)
Database SQL query (depends on DB)
Errors and exceptions Client VM Java console since BIS 6.2.2
of interaction
belonging to BIS Front- BIS System Log files since BIS 6.2.2
end

Analyzing Results and Coming to a Conclusion

By collecting relevant information in the previous section, you can draw some conclusions. In
general, there is no need to gather all data, because you can determine much by looking at specific
information, e.g. the JMS messages which are used for internal communication.

The most relevant information defines the throughput (number of process executions per time
period). You can easily determine this by taking the number of executed processes divided by the
trial execution time (both of those numbers are available the BIS 6 Front-end).


With respect to performance, it is a good practice to have a reference process to
obtain a baseline.

As a very high level type of guideline:

Monitoring, Reporting and Maintenance 8

 Analyzing Results and Coming to a Conclusion

Parameter Comments
CPU (average) load The average CPU load depends on the hardware sizing, the process
designs and many other parameters, but a full load of up to 100% on
instance in latest BIS version can be feasible. Simple AdapterEngines
(either BIC and/or DT) usually have a lesser load (up to ~ 50%).
Free RAM The free RAM space depends on the VM setting and the hardware
(very often more than 10% are left free). By running into
OutOfMemory, processing e.g. conversion on own instances is
proposed.
Active threads The BIS system has a normal load between 400 and 1000 active
threads. The critical number is higher than 1500.
Exec. time of SQL The performance of the BIS system depends very much on the
statements performance of the database and the response times. Databases
such as Oracle have many key parameters and configurations which
have a major impact. The number of processes and sessions are
some of the key parameters.
Garbage collection Long garbage collections, and in particular full GC, have a negative
effect, and lead to a bad throughput. A good indication of
performance is to look at a period of time, and determine how often
a full GC runs, and how long it takes.
Trial/Process execution time The total execution time (by trial or average time of a process) is
some valuable information and a good indication of performance.

Monitoring, Reporting and Maintenance 9

 Check List

Tracking System Behavior, Performance and

Errors
In order to track the behavior, performance, and errors of the BIS system; a check list is proposed.
The check list raises several questions, which may help to collect important and accurate
information, to qualify a support request.

In case of a trouble shooting request, we also provide hints about where to look at more specifically,
and to define some alerts in advance, in order to get some early information, before a problem may
occur.

Logging information is overwritten after some time. It is a good practice to

 increase the length of tracing log data to a history of more than 72 hours (which
is important when a problem occurs on Friday or during a weekend).

Check List
The next table provides a step by step guided introduction of how to test and verify the status of the
system. Going through the checklist, you ensure that you check all aspects which might be related to
a problem. If you require further information, or some examples, check the other sections in this
document.

Check List: Guidance and Thresholds

Reference Action
A 1. 1. Start a web browser and connect to the bis system,
[Link] where <bis_host> = the host name of the
bis server, e.g. [Link] # the BIS 6 system is
not available
A 1.1 If A 1) failed, please check:

1. Start a browser, or SQL CLI, and connect to your database #

the database is not available.
2. If you do not know how to connect, please contact your
database administrator for further information.

Tools:

• Oracle SQL Developer Studio sqldeveloper (available for free:

h_ttp://[Link]/technology/software/products/sql/ind
ex.html_)
• MS SQL Server, e.g. SQL Server Management Studio Express

Monitoring, Reporting and Maintenance 10

 Check List

Reference Action
A 1.2 If A 1) failed, please check:

1. Connect to the host of your BIS installation, and check if your BIS
is up and running.
2. If the service is running, and you cannot connect:
◦ Check the connection from a different PC (you may have a
problem with a firewall).
◦ Restart the bis service (and check log files [Link] in
_BIS_HOME/log_).
◦ Start a web browser, and connect to the BIS (refer to A 1).
3. If the service is not running:
◦ Check the log files in directory BIS_HOME/log/
◦ Start up the service.
◦ Start a web browser and connect to the BIS (refer to A 1) #
BIS is not available.
A 1.3 1. After you have logged into the system, switch to the control
center, and check the availability of adapters.
2. When the landscape contains several machines, check the
availability of all instances.
3. If the required adapters are not activated:
◦ Start all adapters (it might be necessary to do it several
times, before they actually appear)
◦ Check log files in directory BIS_HOME/log/ # the instance
is not available.
B 1.1 1. Connect to the BIS host and check the CPU utilization (several
minutes).
2. Check the current load of the system.
3. If the CPU load is less then 90% (for a while) start some
processing.
4. If the load is reaching 90%, the instance seems to be working
fine.

Tools:

• OS specific tools (monitoring CPU)

• BIS Front-end (monitoring CPU at dashboard)
•# The instance does not have a peak of 90% utilization.
•# The instance keeps 100% utilization (for 15 minutes and
longer).

Monitoring, Reporting and Maintenance 11

 Check List

Reference Action
B 2.1 Check the number of OS threads during process execution, and
particularly when having an increasingly high load that might lead to
some resource shortage (if a configuration is wrong, limiting the
system from executions).

Tools:

• Microsoft: Windows Task Manager

• Unix: glance
•# Having more than 1200 active threads.
•# Having more than 2000 active threads.
B 2.2 The number of threads during a process execution, and particularly
when having an increasingly high load that might lead to some
resource shortage (if a configuration is wrong, limiting the system
from executions).

Tools:

1. Open the BIS Front-end and select the dashboard to see the
number of active threads:
◦ ThreadDump (Client console, java tools) monitoring the
thread states.
◦# Having plenty of threads in state ‘blocked’ and
‘waiting’.
◦# Having more than 1000 active threads.
◦# Having more than 1500 active threads.
B 4.1 Out of memory will bring the system into an unknown state.
Therefore it is necessary to have sufficient memory left.

1. Check the memory usage.

Tool:

• BIS Front-end (Dashboard)

B 4.2 1. Monitor GC activity.
2. Check GC activity.

Tool:

• BIS_HOME/log/run/dir/[Link]
•# Long full garbage collections > 20 seconds.
•# Long full garbage collections > 30 seconds.

Monitoring, Reporting and Maintenance 12

 Check List

Reference Action
B5 Investigations of the database will be requested on demand.

1. Execute query, take results, and note response time.

Tool:

• Oracle SQL Developer Studio sqldeveloper

(available for free: [Link]
products/sql/[Link])
• MS SQL Server e.g. SQL Server Management Studio Express
C1 The default setting of file handles in an enterprise environment is
significant.

1. Check the number of file handles of each system.

Tool:

• Unix: ulimit; lsof

C2 The system is storing, and removing data, on disks of all hosts.

1. Check the available storage of the instances.

2. Check the available storage of the table space on the database.

Tool:

• Windows: Explorer
• Unix: df, du
• Oracle Enterprise Manager
•# Less than 10-20 % space left.
•# Less than 5-10 % space left.
C3 The number of waiting sessions of your database is an indication of
performance.

1. Ensure that you have enough load in your system by starting a

float load.
2. Check how many sessions are waiting.

Tool:

• Oracle Enterprise Manager

•# More than 40 waiting sessions.
•# More than 70 waiting sessions.
C4 1. If the database reports high watermarks, which has an impact to
the overall performance, check if there is HWM contention
reported.

Tool:

• Oracle Enterprise Manager

Monitoring, Reporting and Maintenance 13

 Check List

Reference Action
C5 Free field to report other resource shortage and limitations.
D 1 – D5 If errors and exception appear, they will be written in log files.

1. Check the occurrence of Exception, ERROR, WARN, ORA-.

2. Analyze the log files of all instances BIS_HOME/log/.
3. Make a copy.
4. Classify them by type and report in the fields D 6 – D 20.

Tool:

• Unix: grep
•# OutOfMemoryException, In-doubt transaction, no more
handles.
E Make some additional notes, and describe the incident briefly in your
own words.

Check List Report

Monitoring, Reporting and Maintenance 14

 Check List

Monitoring, Reporting and Maintenance 15

 Check List

Monitoring, Reporting and Maintenance 16

 Runtime Command REST API

Tools and Services: Gathering Relevant

Information
There are many tools and services available which make it easy to collect the relevant information in
various ways. The next sections introduce tools and services, and explain how they interact, and
how to gather the relevant information.

Runtime Command REST API

The Runtime Command REST API offers a secure, auditable, and Kubernetes-compatible method
to execute Karaf shell commands remotely via HTTP. This API exposes a single REST endpoint that
allows the execution of existing Karaf shell commands, eliminating the need for SSH or interactive
shell access. This is especially useful in environments such as Kubernetes, where SSH access is often
restricted for security and compliance reasons.

General Notes
• Purpose: Execute Karaf shell commands remotely via HTTP.
• Permission: To use the REST API, we recommend a dedicated user. The required permissions
are rest:access and runtime-commands-rest:access.
• Authentication: All commands are executed within the context of the authenticated user. Some
system commands are restricted to the admin, manager, or viewer roles in the Karaf shell. To
enable these roles, you need to assign the corresponding permissions to the user.

Enabling the REST Endpoint

The REST endpoint is served via the Remote API endpoint of the BIS system listener but must be
enabled separately. To enable it, update the configuration file [Link]
as follows:

[client-command]
[Link] = true

Usage Constraints
• Single Command Execution: Only one command can be executed per REST call.
• No Multi-Command Support: Chained or batch command execution is not supported.
• Synchronous Execution Only: Streaming or asynchronous execution is not available.

Endpoint
POST /api/runtime/commands

Description

Provides a generic access to the Runtime commands of the BIS.

Parameters

Monitoring, Reporting and Maintenance 17

 BIS Portal

Headers

• Accept string (Optional):

application/xml or application/json are supported.
• Content-Type string (Optional):
application/xml or application/json are supported.

Body parameters

• command json Required:

The command that should be executed.

Example Request

POST /files/upload/my-folder
Content-Type: application/json
{
"command": "vfs-admin:ls vfs://shardmft-LS000/"
}

Example Response

HTTP/1.1 200 OK
Content-Type: application/json
{
"status": "SUCCESS",
"durationMs": 66,
"message": "vfs://shardmft-LS000/default",
"timestamp": "2025-11-06T10:02:24Z",
"executedBy": "seeadmin",
"runOn": "vfs-testserver"
}

HTTP Response Status Codes

Status Meaning
200 OK Command executed successfully
400 Bad Request No command was given or command is forbidden
401 Unauthorized Access denied
403 Forbidden Insufficient permissions or endpoint is disabled
500 Internal Server Error Unexpected server error

BIS Portal
Control Center
The Control Center web app is logical system specific. It allows to monitor and control runtime
aspects of the BIS system.

Monitoring, Reporting and Maintenance 18

 BIS Portal

You need the read-only or modify/admin roles contained in the sample groups SEE_CC_VIEW_000
or SEE_CC_ADMIN_000 to view or interact with the control center.

When you start the Control Center it starts with last active screen, or the Dashboard view. On the left
you have menu entries for the different screens.

Dashboard

The Dashboard screen in the Control Center gives a quick overview of the BIS System performance
and includes metrics for the Instances (CPU, Memory) as well as Adapter Execution statistics.

The statistics are collected for a short time range. For more elaborate and long-term trend analysis
we recommend to collect metrics on a dedicated system monitoring solution using the Instance
Metrics (Prometheus).

With the Configure Dashboard button you open the dialog which allows to configure what graphs
are shown in the Dashboard.

If you hover over a chart details for the selected data point are show.

Scheduler Execution State

The Scheduler Execution State menu entry in the Control Center web app shows an overview of all
scheduler tasks and their execution information. Scheduler tasks are defined in the BIS Masterdata
Navigator.

The overview table only shows task execution of entries in your specific logical system. Scheduler
tasks needed to maintain the BIS system and shipped by default are hosted and executed in the
default logical system 000.

The columns in the execution overview table have the following meanings:

Column Name Description

State Green checkmark - the task was executed and the last run was
successful
Red cross - the last task execution was not successful.
Grey stop sign - the task is not active.

Monitoring, Reporting and Maintenance 19

 BIS Portal

Column Name Description

Active Checkmark if the task is active - it will be regularly scheduled. You
can deactivate scheduler tasks in the BIS Masterdata Navigator if
they are not needed or should be paused. Do not delete system tasks
as they will be restored on updates. The activation state will be
preserved.
Task The name of the task.
Instance Tasks of execution type clustered will be listed with an empty
Instance field. If a task has an execution type Each there will be one
entry per BIS Instance (which matches the Instance Selector).
Number of Executions The number of executions in the history. This number is reset
periodically, it allows to check for new executions and allows to see
how often the tasks run relative to other tasks, but the absolute
value has no meaning.
Last success Timestamp of the last successful execution of this scheduler task (on
a given instance or cluster wide).
Last failed Timestamp of the last failed execution of this scheduler task (on a
given instance or cluster wide).
Number of failed Executions The number of failed executions in the history. This number is reset
periodically.
Next Execution The estimated timestamp when this will be executed next time. This
is empty for de-activated tasks.
Created Timestamp of the creation of the scheduler task (or first system
startup).

The table is not updated automatically, you need to use the Refresh button on the top right if you
want to monitor execution counts or timestamps.

In the table settings, you can select the visible columns and their order. If you click on a column
header, you can change the sort order.

When you enable the quick filter, you can add filter criteria in the table row below the title to limit
the results shown and search for specific entries. For example, you can enter “BQ” in the task (name)
filter to see only task executions that are used for the batch queue.

If you double-click a specific task or select a row and then click the Show History button, you can
view the history of the selected task. This view will show a list of all recorded executions.

Drill-down: Execution History of Task

In the execution history for a specific task (and Instance), the following information is displayed:

Monitoring, Reporting and Maintenance 20

 BIS Portal

Column Name Description

Status Success - this task execution completed successfully
Failed - this task execution completed abnormally
Skipped - this task did not run due to lack of resources, previous runs
that were too long, or restarts.
Instance Instance ID where the task was executed. In case of tasks with
execution type Clustered, each execution can be on a different
instance. For tasks of type Each, you see the Instance you selected
on the overview screen.
Start Timestamp when the execution began
End Timestamp when the execution finished
Message Optional completion message or error text provided by the executed
task listener.
Workload factor Number provided by the task listener to summarize the complexity
of the work performed. Not all listeners provide this information, and
the meaning depends on the listener.
Next Execution Timestamp calculated by the scheduler engine for the next
invocation.

Reorganization of Task Execution History

The execution history is regularly cleaned up, the system scheduler task
SCHEDULER_EXECUTION_REORG has a SchedulerExecutionReorgListener with the
configuration settings "Max Days" (Default 10) and "Max number" (Default 1000). If the execution
history of a task is longer or older than the specified values, the entries will be removed. After that
the number of executions and failures is re-calculated.

Monitoring, Reporting and Maintenance 21

 SEEBURGER Support Agent

Use the BIS Masterdata Navigator in logical system 000, navigate to the Scheduler tree entry and
open "System Tasks", search for the SCHEDULER_EXECUTION_REORG task and configure the
numbers in the Task Settings section.


Increasing the execution history limits can negatively affect the performance of
the scheduler.

SEEBURGER Support Agent

The SEEBURGER Support Agent (SSA) is the standard way of collecting support information for
incidents as well as remote monitoring information.

In case of an incident, SSA provides a simple means of aggregating system health information into a
single zip archive. It can be triggered:

• by the client shell via diagnostic:run-profile (see help for details)

• through the Web UI of an Installation Server (Web Installer / BIS Landscape Manager)
• by manually triggering one of the COLLECTOR_* system schedulers through the BIS Front-end

When running any diagnostic profile, it is sufficient to do so on a single instance. It will automatically
gather relevant data remotely from all reachable instances in the system. In case an instance is not
reachable due to connectivity problems, the procedure can be repeated on the non-reachable
instance to retrieve the missing data for support. In this case, specify the -i INSTANCEID
parameter to limit collection to this instance.

If you specify the --send option for automatically sending the collected support ZIP to SEEBURGER
support systems (or if you want to send the regular monitoring data to SEEBURGER remote
monitoring services) you need to configure the endpoint.

Details about the diagnostic:run-profile client command can be found in the Client
Commands reference manual or with the --help option.

If the Installation Server is used to trigger the SSA collection (button Generate
Support ZIP), it will connect to the selected instance and invoke the diagnostic
profile run (collection and optional sending) there. See the Landscape
Management manual for additional description.
If you trigger the collection on the system entry it will pick any running instance
(prefers the instance with primary AdminServer role). The AdminServer is
preferred, so that you only need to configure one instance, which is typically not
 used for transaction processing. To make it more predictable which instance is
called, select the instance entry instead.
From there collecting from all instances in the system (and optionally sending to
SEEBURGER) is initiated. For this this the endpoint- and possibly proxy
configuration as well as required communication permissions must be provided
to this installation.
No matter if you specify the send option or not, the Installation Server offers a
Browser download at the end of the ZIP generation job.

Configuration
Typically, SSA required no special configuration and works out of the box to gather information and
store a zip file locally. The configuration is required if SSA should automatically encrypt and send
data to the SEEBURGER support.

The upload URL, credentials and PGP encryption key required for the automatic sending are

Monitoring, Reporting and Maintenance 22

 SEEBURGER Support Agent

provided in a signed configuration file that SEEBURGER provides. The file needs to be stored on
${[Link]}/license/[Link] on every instance of the BIS system (which might be
used to collect and send).

Typically, the following URL for uploading the support data via https is used. Your Internet-visible IP-
Address must be whitelisted by SEEBURGER support. You cannot intercept this TLS connection for
security reasons:

[Link]

In case your firewall does not allow direct outgoing connections, it is possible to use a proxy server.
The proxy server can either be configured globally in ${[Link]}/[Link] or
overridden in the scheduler configuration for automatic scheduled sending.

The following settings configure the standard proxy settings for Security Zone, Proxy Mode and
Alias. Make sure to uncomment the options you want to specify

• If you select proxyMode=SPECIFIC the proxyAlias property must be filled with the name of
the proxy server (from BIS system configuration).
• If you select proxyMode=GROUP the name of the proxy group from the BIS system configuration
must be specified.
• If DEFAULT is configured, the default proxy for the HTTP protocol is used.
• If you use empty proxyMode= or keep it commented out, will not use a BIS proxy.

Details about proxy configuration can be found in the Adapter Master Configuration guide.

# The security zone property used when sending the info to

seeburger
#securityZone=
# The proxy alias property used when sending the info to
seeburger
#proxyAlias=
# The proxy mode property used when sending the info to
seeburger.
# DEFAULT, SPECIFIC, GROUP can be used here
#proxyMode=DEFAULT

Connections to SEEBURGER are initiated from the collecting instance, therefore the BIS Secure Proxy
(BSP) integration on this instance must be installed and configured, if a proxy (by name,default or
group) is configured which uses the BIS Secure proxy integration infrastructure.

If you run automated collections for the SEEBURGER Remote Management Service (COLLECTOR_*
BIS system scheduler tasks) make sure to configure the correct instance to execute this collection.
By default, it happens on any AdminServer roles (type Clustered, so it is only executed once).

Profiles
SSA comes equipped with multiple profiles that determine what data is gathered. Currently, it ships
with these default profiles.

• monitoring - typically executed automatically in 5-minute intervals to send monitoring

Monitoring, Reporting and Maintenance 23

 SEEBURGER Support Agent

information for RMS. This scheduler is deactivated by default. To make use of it, enable the
scheduler and make sure that the [Link] file is present
• analysis - typically executed automatically in 24-hour intervals to send long term analysis
information for RMS. This scheduler is deactivated by default. To make use of it, enable the
scheduler and make sure that the [Link] file is present
• incident - the scheduler is deactivated by default and intended for manual execution in case
of a support incident. If the automatic sending is activated in the scheduler, the [Link]
file needs to be available on the instance
• structural-backup - this profile is used to collect information about the landscape structure,
such as the configuration files of the instances, masterdata backup, recipe and resources
provided by BIS LM. Its being used to create a backup of the landscape.

The collected data contains basic telemetry and support relevant information including but not
limited to

• logs
• instance details
• metrics
• version history and patch level

Sensitive information like passwords will not be collected. But keep in mind that log files and
especially heap dumps will contain information which requires protection. To verify the contents of
an archive, you can execute any of the profiles (without the send option) which will create a local zip
archive that can be inspected.

The performance impact of the monitoring and analysis profiles is minimal. The incident profile is
slightly more resource-intensive, specifically, it will produce some disk IO to collect the log files, and
CPU to compress the results.

If the incident profile is run with the option to create heap dumps (relevant for
out of memory troubleshooting), it will cause a processing delay of several
seconds/minutes on the instance(s). It also requires disk capacity for generating
 the heap dumps and preparing the archives. It also increases the chance of
lockups or crashes, so it should only be used if absolutely necessary. Typically,
you should also plan a restart of all instances, once you have requested a heap
dump for this reason.

Configuration options for specific collectors

This configuration section describes the configuration options for specific collectors that are used by
the SSA. This is achieved by adding or editing the etc/[Link] file. These
configurations can be made profile specific by adding them in a
<profile>.[Link] file instead, for example,
[Link]. Enabling a collector, that is disabled by default, in
[Link] instead of <profile>.[Link] will make it a part of all
profiles.

Collector process-errors-radar

The process-errors-radar collector is used to collect information about process errors in the
system. It collects data from a directory specified in the radar-file-bridge configuration or by parsing
the output of the list-processes | get-log command. The collector is intended to be used in
the monitoring profile and is disabled by default. It can be enabled in the
[Link] or [Link] file, which will be called "the

Monitoring, Reporting and Maintenance 24

 JBoss JMX Console

configuration file" from now on, by adding the following line:

• [Link]=true

Here are the configuration options for the process-errors-radar collector:

• [Link]

If this option is set to true, the collector will use the files generated by the radar-file-bridge to
collect process errors. If it is set to false, it will use the output of the list-processes | get-log
command. The default value is true.

• [Link]

This is the directory where the radar file bridge will generate its output files. It is configurable by
using the [Link] option in the configuration
file. The specified value is a directory path, relative from ${[Link]} as the root, and must
match the value specified in the radar-file-bridge configuration. The default value is
${[Link]}/radar/errors.

• [Link]

With this option the files from the [Link]

directory are included in the collected ZIP under the process-events subdirectory. Enable it by using
the [Link]=true in the configuration file. The
default value is false.

• [Link]

This option allows to specify additional fields to be collected from the processes. It is configurable
by using the [Link] option in the configuration file. The
specified value is a list of field names separated by commas or semicolons. The specified field
names must match the values in the .json file of the process to be collected. The default value is
empty, which means no additional fields are collected.

• [Link]

This option allows to specify the retention time for the files generated by the radar-file-bridge. It is
configurable by using the [Link] option in the
configuration file. The specified value is a duration in minutes. The default value is 15, which means
the files are deleted if they were created more than 15 minutes ago.

• [Link]

If [Link] is set to false, the process-errors-radar

collector will use the list-processes | get-log command to collect process errors. By using
[Link], the default arguments to list-processes can
be overridden. By default, the command is executed with --past "PT15M" to collect processes
from the last 15 minutes. Specifying the -c or --column argument is not supported, and the
collector will ignore all other specified arguments.

JBoss JMX Console

The JMX console is accessible from the Portal, and provides plenty of relevant information.

Monitoring, Reporting and Maintenance 25

 JBoss JMX Console

Tool/Service Relevant Information?

ServerInfo: thread dump Active threads.
Java VM Thread states.
Counting active workers.
DestinationManager (on Running and queued JMS.
JBoss 4.0.4/5), ServerPeer
(on JBoss 4.2.3)

Searching for “serverinfo” and “DestinationManager” provides additional information.

Server Info
The ServerInfo (under [Link] > Type =ServerInfo) provides several methods, with detailed
information, such as listMemoryPools(). The result delivers the current state of different memory
pools (which are also shown in the Front-end | Dashboard).

Furthermore, the listThreadDump()-operation takes a snap shot of all executed threads. This
document contains a lot of useful and valuable information:

• The number of total threads.

• The number of thread groups:
• The number of thread states: By searching the threads with the key word blocked, the document
might show the following information below. Collecting such information is useful for later
inquiries.

Example of JMS-message with state blocked:

Thread: JBossMQ Cache Reference Softner : priority:5,

Monitoring, Reporting and Maintenance 26

 JBoss JMX Console

demon:true, threadId:31,
threadState:BLOCKED,
threadLockName:[Link]@5eebd50e
[Link](Mes
[Link])
[Link]([Link])
[Link]([Link])
<more>

In addition, there is the possibility to count worker threads of a particular queue, when the system is
getting too slow and JMS-messages are queued (shown in DestinationManager).

Worker Keyword Worker (equivalent to DestinationManager)

[Link] InitiatorReceiver
ssage
[Link] CorrelatorReceiver
Message
MRestartPointReceiverEJB.o RestartPointReceiver
nMessage
[Link] WLHReceiver

A short example is shown below:

Thread: JMS SessionPool Worker-17 : priority:5, demon:true,

threadId:261,
threadState:RUNNABLE, lockName:null
[Link].socketRead0(Native Method)
[Link]([Link])
[Link](Unknown Source)
[Link](Unknown Source)
[Link](Unknown Source)
[Link](Unknown Source)
[Link](Unknown Source)
[Link](Unknown Source)
<more>

[Link]
ssage([Link])
[Link](Unknown Source)
[Link](DelegatingMetho
[Link])
[Link]([Link])
[Link]([Link]
)
<more>

Monitoring, Reporting and Maintenance 27

 JBoss JMX Console

Destination Manager
The Destination Manager (DM) shows useful information about the number of JMS-messages
running, and queued, in the system. The monitor provides information as to what components are
involved in processing, where the messages are at a particular point (queued/running), and how
much JMS-load the system has to cope with.

1. Select the JBoss JMX Console | search for and select the page service=DestinationManager |
invoke listMessageCounter().

The Destination Manager contains the state of running, and queued, JMS-messages sorted by the
different queues (by name). The field count is an absolute counter of handled JMS-messages. The
column Depth provides the number of queued messages; the delta for both gives a number of
messages which are newly appended, and/or added, or removed from a queue, for a given time
interval (which can be manually done by clicking the Re-invoke button).

Example

The following list of queues explains the general principle of counts at queues:

Monitoring, Reporting and Maintenance 28

 JBoss JMX Console

Type Name Sub. Durable Count CountDel Depth DepthDel Last Add
ta ta
Queue BatchQue - - 1713 - 0 - 11/26/08
uePT 2:22:10
PM
Queue MCorrelat - - 32231 130 4 - 11/26/08
orReceive 2:22:10
r PM
Queue MInitiator - - 40 - 0 - 11/26/08
2:22:10
PM
Queue MInitiator - - 46290 152 14050 3 11/26/08
Receiver 2:22:10
PM
Queue MInternal - - 1286 - 2 - 11/26/08
Receiver 2:22:10
PM
Queue MRestart - - 30004 141 1 1 11/26/08
PointRec 2:22:10
eiver PM
Queue OrdersHa - - 62 - 0 - 11/26/08
ndlerBatc 2:22:10
hQueueP PM
T
Queue OrdersHa - - 95 - 0 - 11/26/08
ndlerDtA 2:22:10
S2Compo PM
nent
Queue RecMonit - - 189 - 17 -1 11/26/08
or 2:22:10
PM
Queue ToNode - - 2452 10 1237 - 11/26/08
2:22:10
PM
Queue ToSeeWo - - 1213 5 0 - 11/26/08
rklistHan 2:22:10
dler PM
Queue WLHRece - - 28394 251 9 1 11/26/08
iver 2:22:10
PM
Queue toDtAS2C - - 7910 8 0 - 11/26/08
omponen 2:22:10
t PM
Queue toDtHPSB - - 11720 119 0 - 11/26/08
Compone 2:22:10
nt PM
Queue tobicNod - - 8768 121 0 - 11/26/08
ePT 2:22:10
PM

Monitoring, Reporting and Maintenance 29

 Instance Metrics (Prometheus)

Type Name Sub. Durable Count CountDel Depth DepthDel Last Add
ta ta
Queue tostoreN - - 0 - 0 - 11/26/08
odePT 2:22:10
PM
172545 938 15451 5 Total

Queues which are not in use (since start up) have the value 0 (in column count), like queue
tostoreNodePT.

Queues which are currently involved in processing have the value in column countDelta, like
MInitiatorReceiver = 152.

Queues which have values in the column Depth have message queued, like MInitiatorReceiver =
14050.

Queues which have a positive count in the DepthDelta column, such as MInitiatorReceiver = 3,
increase the number of messages (means they add to a queue). Queues which have a negative
count in the DepthDelta column, such as RecMonitor = -1, reduce the number of queued messages
(means jobs are taken from a queue).

The number of running jobs, in the given time interval, can be calculated by the total countDelta =
938 (minus 5 of depthDelta, because they are queued), or increase the number when the value is
negative.

By keeping the refresh time of Re-invoke operation (the button is shown on upper side of the dialog)
of the view constant (like some browser such as Opera do), the values of the columns countDelta
and depthDelta are indications of performance and throughput of the JMS. To the best of our
knowledge, a good performance (which obviously depends on the design of a process, and the
configuration of the system as well) has a throughput of more then 50 JMS-messages per second.

Instance Metrics (Prometheus)

Each BIS instance (since SP60) exports runtime metrics in Prometheus text-based exposition format.
This is a list of counters, gauges, histograms or summaries which can be queried by http. The query
is a lightweight operation and can be done from multiple monitoring systems with frequency as high
as every second or for a more long term recording every minute.

The purpose of this metrics interface is to allow technical monitoring for capacity planning,
availability alerts and to some lesser extent SLA monitoring (the metrics are well suited for real-time
statistics but cannot drill into business cases or monitor specific SLA check rules). The metrics can
be stored in time-series databases, aggregated, graphed or directly used for alerting on thresholds.

Each instance has its own set of metrics, depending on the components installed. Some metrics
might be collected from shared resources (like the database or host) and can show up in the list for
multiple instances. See the below list for shared or separate metrics.

The metric endpoint is exposed via the "Management" app registered to the "Management
(INSTANCEID)" listeners. Recent installations activate this listener with portal authentication (Basic
Authentication) by default, for updated installations you might need to activate the listener (and
allow network access from your monitoring system).

The user accessing the metrics requires the metrics:view permission, which is by default contained in
the group BIS_ADMIN.

The URL constructed from the Admin Port ([Link]=13000

Monitoring, Reporting and Maintenance 30

 Instance Metrics (Prometheus)

[Link]), instance hostname and the fixed path to the metrics servlet like this:

The metric names (with a namespace and subsystem prefix) and zero to multiple dimensional
parameters (labels). Each metric is accompanied by a short clear text description. Units of metrics
are normally bytes, seconds or count. The numbers are double precision floating point numbers in
scientific representation.

When setting up a Prometheus server to scrape metrics you should add labels to all metrics from the
same endpoint describing the actual source. This can be instanceid, hostname, system and possibly
type of landscape (prod, staging). This way you can easily sum up all metrics by any of those
dimensions. The metric servlet itself does not add repeating properties to describe the source.

List of Prefixes
Generally the list of metrics is dynamic and might also change per version. You should therefore
query your specific installation to inspect the metrics available. However, the following table gives
you an orientation about the different types of metric providers possibly registered.

Metric Prefix Description On Roles Sharing

Level
bis_hardware basic information about disk space machine
and processor count, refreshed every
~15s.
bis_jms statistics about various internal JMS AS,PE, WLH
queues for this JMS provider
bis_inbox Performance metrics of the INBOX PE
component
bis_inbox_tasks Performance of Inbox per initiator.
bis_dashboard Will be removed AS db
bis_mapping Performance counters for BIS Mapping
engine
jvm Metrics about JVM performance and
load including threadcounts, GC times
and counts, buffers, classes and
memory pool usage. jvm_info is a
pseudo metric which lists the actual
JVM used as properties.
guava_cache In-Memory cache statistics.

• bis_landscape_instance_cache``
performance of landscape API
lookup cache
• bis_caching_config_provider``
performance of SeeConfig cache
(requires system property
[Link]
[Link]
ORD_STATS=true)
• masterdataNavigatorCache-000``
BIS Navigator App in-memory
cache of master data per LS
bis_adapter_status Local adapter state. AE

Monitoring, Reporting and Maintenance 31

 Instance Metrics (Prometheus)

Metric Prefix Description On Roles Sharing

Level
bis_adapter_monitoring Job counts for local adapters. AE
bis_adapter_recovery Recoveries for local adapters. AE
bis_adapter_messageids MessageIDStore usage for local AE
tore adapter.
bis_adapter_direct_mod Statistics of executed direct mode AE
e orders
bis_aqm Statistics related to the AQM AE
bis_engine Execution statistics of Business PE
Process Engine.
bis_ftpserver Statistics for FTP server AE
upload/download
bis_httplistener Statistics for http requests on the ALL
httplistener
bis_order Statistics of created/executed orders PE
for DirectMode, Sync Mode and AQM
bis_sftpserver Statistics for SFTP server AE
upload/download
bis_sftpclient Statistics for SFTP Client SSH login AE
bis_ds_server Server side metrics of DataStore server DS
bis_ds_client Client side metrics of DataStore clients ALL
bis_is Information Server and Client ALL
statistics
com_seeburger_bisfx Various metrics of the BIS FX AE (FX)
application (some without the
com_seeburger_bisfx prefix) are
described in the BIS FX
Administration and Installation
manual in the Metrics chapter..

Example Metrics Explained

The following curl (http command line client) command can be used to retrieve the current Inbox
queue depth on the specific instance. It includes two commented lines of descriptions. First, the
basic authentication header for user "test" with password "secret" is calculated, and this can be
used in all following invocations:

$ echo -n "test:secret" | openssl base64

dGVzdDpzZWNyZXQ=
$ set AUTH=Authorization: basic dGVzdDpzZWNyZXQ=
$ curl -s -H "$AUTH" [Link]
| grep bis_inbox_current_message_count_gauge
# HELP bis_inbox_current_message_count_gauge Current number of
...
# TYPE bis_inbox_current_message_count_gauge gauge
bis_inbox_current_message_count_gauge 0.0

Monitoring, Reporting and Maintenance 32

 Instance Metrics (Prometheus)

Host Statistics

The metrics exported by each instance are mainly about the JVM and the Installed components. It
does not contain much details about the machine performance (hardware, hypervisor, and
operating system statistics).

If you set up a Prometheus based metrics collection (into a time series database) you typically will
also start a "node exporter" component which is offered by the community for various operating
systems. This node exporter will provide detailed and specific statistics about key OS metrics.

SEEBURGER recommends the following two node exporters for this usage:

• Windows: WMI Exporter

• Linux: Prometheus Node Exporter - [Link]

More exporters can be found on the Prometheus project site:

[Link]

Note that the most detailed metrics collection by the Prometheus Node Exporter happens on the
Linux operating system (you might need more specific monitoring agents for AIX or Solaris).

Command Line Query

In addition to the traditional Prometheus metrics servlet, you can query the metrics also from the
(OSGi) command console ([Link]=8090). With the diagnostic:show-metric
command. This is mostly used for interactive use since it is a bigger overhead to start the SSH
console connection and query for single metrics.

If you plan to call it less often, you can use it to script the metric retrieval, for example with the client
script:

$ .\[Link] -- diagnostic:show-metric -f CSV -m

'bis_inbox_current*'
"Name","Labels","Type","Value","Description"
"bis_inbox_current_message_count_gauge","","GAUGE","0.0","Curre
nt number of JMS
messages in the BP engine-related queues"

As you can see metric names are specified with the -m parameter and * is supported for substring
matches.

Automatic Historic Collection

When the instance is started it automatically dumps the metrics in regular intervals to the local
filesystem. This data can be used to do post mortem analysis and will be collected by SEEBURGER
support in case it is needed.

The compressed daily archives of this

can be found in
<BIS_HOME>/log/monitoring/metrics/<yy-mm-dd>.ms[.gz]. It is a promrec/promplay
formatted (GitHub) appended journal. It is recommended to not use this collection for your
monitoring since the frequency is coarse and picking up the files is more complicated than just
polling the metrics endpoint.

Currently snapshots are created every 60s, this can be configured with the system property
[Link] (in seconds). Use 0 to turn it off. The files are kept by default for 7days

Monitoring, Reporting and Maintenance 33

 Configure Logging

(can be configured with system property [Link]).

Collection Best Practice

While it is generally possible to query an instance, calculate the metrics and alert on it (especially
since the text protocol is very easy to implement), the preferred method is to set up a Prometheus
Server and use it for alerts. This has the advantage that metrics can be accumulated or filtered over
time. Besides the Built-In Prometheus alerting other monitoring systems (like Nagios) can also use
Plugins to query the Prometheus store and alert based on conditions.

Having a Prometheus data store allows to collect fine grained time series information over a longer
time period from thousands of nodes with only limited resurces. In addition to collecting it can also
argument the recorded metrics with calculated metrics. In the Prometheus Ecosystem, you can
comfortably set up Dashboards (for example Grafana). However currently SEEBURGER does not
offer distributions or support for Prometheus as part of a regular product maintenance contract.

A counter is a good method to communicate the performance of a system, even when polled
unreliable/irregular. However counters (especially error counters) can not directly be used for
alerting (since a condition like error_count > 10 will only work one time for a running system and
not recognize when the system has recovered). With Prometheus (or other time series databases)
you can easily query for derivates or rates over certain time windows error_count[10min] > 10.
This is therefore the recommended method. Recording all error counters for post-mortem analysis is
also helpful to be prepared to root-cause analyze unexpected issues

Configure Logging
Configure log-level
Please refer to the Logging and Tracing section in the Adapter Configuration manual for detailed
info about how to manage log levels. The Adapter Control center UI is the recommended way of
configuring log levels during operation, especially for communication troubleshooting.

You can also use the log:set console command. This will be temporary persistent (until the next
update -c run).

Configure log settings during Installation and Update

Every component creating a log-appender delivers its logger properties to file

<BIS_HOME>/etc/[Link]. This file should not be modified since it is
regularly replaced.

For modification of logging, there is file <BIS_HOME>/etc/custom-

config/[Link] available, which allows overriding and extending the
existing default logging configuration. New keys can be added, and existing ones can be replaced.

Settings from this custom logging configuration will be merged with the default
[Link] during update (-c).

While overriding and extending are simple, a more advanced use case is to configure adapters
individually. This is shown as an example in <BIS_HOME>/etc/custom-
config/[Link] below.

#
# Logging Customization File
#

Monitoring, Reporting and Maintenance 34

 Configure Logging

# You can use this file to customize the logger configuration.

It will be merged with [Link] during update
(-c)
# New keys can be added and existing ones can be replaced.
# Configuration is done in property format as described in
[Link]
#

# More advanced example on how to exclusively configure one

adapter (AS2 Controller in this example) individually instead
of just editing the common adapter appender.
# Each adapter has its own module name, which can be seen in
filename of existing log (replace underscores with blanks),
e.g. filename AS2_Controller.lgw indicates module name "AS2
Controller".

# Exclude the module "AS2 Controller" from general log by

overriding existing excludes and adding "AS2 Controller"
[Link] = AS2
Controller,datastore,MID-Reorg,KSM-Reorg,ErrorMonitor-
Reorg,aqm,aqmmigration,vfs,inbox_receiver,initiator,component_r
eceiver,message_receiver,waiting_messages_receiver,restart_poin
t_receiver,requests_to_initiator,alarm_receiver,correlator,inte
rnal_receiver,bspremotelogger,bsplogger,Backend,Transport

# Add the new appender (see bwlow) with name "AS2_Controller"

to rootLogger
[Link]=AS2_Controller,RollingFile,Error
RollingFile,ServerBootRollingFile,Adapter,DatastoreRollingFile,
ChangeRequestRollingFile,MIDReorgRollingFile,KSMReorgRollingFil
e,ErrorMonitorReorgRollingFile,BSPRollingFile,BSPErrorRollingFi
le,BSPRemoteLogger,BSPErrorRemoteLogger
[Link].AS2_Controller.ref =
AS2_Controller
[Link].AS2_Controller.[Link] = INFO

# Define new appender "as2controller" with name

"AS2_Controller" for Module "AS2 Controller"
# AS2 Controller Rolling file appender
[Link] = RollingRandomAccessFile
[Link] = AS2_Controller
[Link] =
\${[Link]}/log/adapters/AS2_Controller/AS2_Controller.lgw
[Link] =
\${[Link]}/log/adapters/AS2_Controller/AS2_Controller.lgw.%
[Link]
[Link] = true
[Link] = PatternLayout
[Link] = Instance home:

Monitoring, Reporting and Maintenance 35

 Configure Logging

${[Link]}\nInstance information:
[Link]=${[Link]}, [Link]=${[Link]},
[Link]=${[Link]},
[Link]=${[Link]}\nAdditional information:
[Link]=${[Link]}, [Link]=${[Link]},
[Link]=${[Link]}, [Link]=${[Link]},
[Link]=${[Link]}\n
[Link] =
${[Link]}
[Link] = Policies
[Link] =
OnStartupTriggeringPolicy
[Link] =
SizeBasedTriggeringPolicy
[Link] = 10MB
[Link]=DefaultRolloverStra
tegy
[Link]=min
[Link] = 30
[Link] =
ThreadContextMapFilter
[Link] = ACCEPT
[Link] =
KeyValuePair
[Link] = Module
[Link] = AS2
Controller

Prevention from log file poisoning

Depending upon logging component and log level, user derived content is logged. For example, an
URL parameter of an incoming AS2 message.
An attacker could use this to inject special characters such as newlines into the message, which is
then logged to the BIS log file.

This issue could be exploited for social engineering attacks against


administrators by injecting malicious messages into log files. These could include
false error conditions that instruct administrators to contact someone or even
interact with services or systems by restarting them.

From a down-to-earth point of view, however, it seems unlikely an administrator looking at BIS log
files would be tempted to interpret it in any other way than it is meant. Especially, the component
that issues the logs is always mentioned, so it becomes always clear if a log message is coming from
a communication adapter, for example.

However, it is possible to filter log messages, in general, to not allow special characters at all. This is
not active by default because it involves expensive parsing and potential modification of each log
message and therefore has a performance impact.

However, if there is a need for configuration, follow the below procedure.

Monitoring, Reporting and Maintenance 36

 Centralized Logging

• The predefined patterns are located at the beginning of file

<BIS_HOME>/etc/[Link]
• Edit file for logging configurations, as described above: <BIS_HOME>/etc/custom-
config/[Link]
• Replace/add one or more patterns to replace the desired characters.
• For example, to replace the predefined main pattern:
[Link] = %d{yyyy-MM-dd’T’HH:mm:[Link]}\t %p\t %c\t %t\t
[%X{[Link]}:%X{[Link]}]\t %X{Module}\t %X{env}\t %X{user}\t
%X{contextId}\t %m%n
with a pattern that filters newlines, redefine it in etc/custom-
config/[Link] with
[Link] = %d{yyyy-MM-dd’T’HH:mm:[Link]}\t %p\t %c\t %t\t
[%X{[Link]}:%X{[Link]}]\t %X{Module}\t %X{env}\t %X{user}\t
%X{contextId}\t %replace{%m}{[\r\n]+}{}%n

Reference: [Link]

Centralized Logging
By default, BIS instances use the local filesystem for storage of log files. This is a fast and reliable
way. BIS offers a remote viewer for logfiles and also manages logfiles generation including roll-over
and compression. It uses its own log-format so that the log viewer can show the logfiles in a
structured way. The SEEBURGER Support Agent can also compress all log files from the instances
into a collector archive, which allows SEEBURGER support to easily analyze them.
However, if you need to manage multiple BIS instances, if you want some automated log
monitoring, or if the BIS instances are not stateful (as in some cloud or container scenarios), this
distributed and localized log storage may not be sufficient. In this case, we recommend setting up
log collectors (agents or sidecars) that read the BIS logfiles for centralized storage or search.
If you go this route, keep the following in mind:

• In support cases, you must be able to reproduce the contents of the logfile so that you can send
it to SEEBURGER (an online search in your internal log management system may not be
sufficient).
• The main logfile of each instance is <BIS_HOME>/log/[Link] - if you scrape this file
often, you will get most of the important information. However, some information is missing
from this file, so you may want to scrape the other files as well:
◦ <BIS_HOME>/log/{audit,changerequests}/* - Audit information. Due to sensitivity
and lifecycle, it is not included in [Link].
◦ <BIS_HOME>/log/pl/* - SQL traces. Only used when enabled, high volume, and
sensitivity. Should normally not be needed.
◦ <BIS_HOME>/log/reorg/ - BIS reorganization runs. If you need to monitor and
troubleshoot reorrg issues, make sure you collect them separately. Alternatively, you can
access the files on a running instance if requested by support.
• <BIS_HOME>/log/monitoring/* - contains data files that are not directly logfiles
(prometheus dumps and .csv statistic). They are not part of the [Link].
• The additional logfiles such as [Link], [Link], adapter/, aqm/,
datastore/, portal/, smt/, and vfs/ do not contain additional/different information and
therefore do not need to be scraped. They are just stored separately (duplicating the [Link])
to keep the information longer.
• <BIS_HOME>/log/{client,diffdb,update} - contains logfiles related to installation and
update, these cannot be found in [Link].
• <BIS_HOME>/log/[Link] - boot time recording of system state, not contained in

Monitoring, Reporting and Maintenance 37

 Diagnostic Database Queries

[Link]. It is usually sufficient to provide the current versions from a running instance for
support reasons.
• <BIS_HOME>/log/run/dir - default working dir, may contain third-party traces and dumps
(SAP JCo, MQSeries traces, Java dumps) - which are not part of the [Link]. You should at least
monitor the directory for old files. If you need to troubleshoot a crash (especially for stateless
container instances), you may need to mount this directory as an external volume to preserve its
contents across restarts. During normal operation, you will not need most of the contents and
should clean it out regularly. If you want to monitor for Java GC performance, you can use the
<BIS_HOME>/log/run/dir/[Link] which is in the OpenJDK Unified logging format.

Using logging configuration, you can also add a custom logfile that contains essentially the same
information as [Link], but with a different format or reorganization. This can help with certain
log-collector agents.

If you plan to turn off the additional log duplicates in the above mentioned files, make sure to
coordinate the support process with SEEBURGER before doing so, as standard support is based on
these files being part of the SSA collector logs.

The instance console log (karaf command) does not include all the severities or columns which are
written to the [Link] log, so it cannot be used as a sole recording of system operation (unless
customized - we have no experience with performance when using this in a container coordinator or
docker environment).

Diagnostic Database Queries

Some of the processing information of the BIS system is stored in the system database. While we
generally not offer the database structure for interaqction as a stable interface, there are some
queries you can use in case of troubleshooting:

• Processes by states.
• Process execution times (of a particular process template by state and priority).

If you want to automate things like SLA monitoring or importing in different management systems,
we recommend the BIS Remote (REST) API Interfaces instead - as well as the extension points for BIS
Monitoring, BIS Message Tracking or SEEBURGER Information Layer.

Processes by States
SQL Statement
select count(*), cstate from tbpinstances group by cstate;
Result

Explanation 18546 = processes are running (state = 1)

229312 = processes finished successfully(state = 5)
38 = processes are suspended (state = 8)
4 = processes have failed (state = 7)


For information on all process states, check article ID [20180904-0054] on the
SEEBURGER Knowledgebase ([Link]

Monitoring, Reporting and Maintenance 38

 Diagnostic Database Queries

Number of JMS Messages (Sorted by Queue Name)

SQL statement
select count(*), cdestination from tjmsmessages group by cdestination;
Result

Explanation
Queue MCorrelatorReceiver contains 14420 JMS-messages.

Processes by Exec. Time (Sorted by Type, State and Prio.)

SQL statement
SELECT cprocessqname, cpriority, cstate, MIN(clastaction-cstarttime), MAX(clastaction-cstarttime),
AVG(clastaction-cstarttime), count(1) FROM tbpinstances GROUP BY cprocessqname, cpriority, cstate
ORDER BY cpriority DESC;
Result

Explanation

Monitoring, Reporting and Maintenance 39

 Diagnostic Database Queries

CPROCESSQNAME = type of process (template)

CPRIO = priority
CSTATE = status of a process
MIN/MAX/AVG = minimum, maximum, average process execution time
COUNT = number of records

Monitoring, Reporting and Maintenance 40

 Log Files

Examples of Errors and Exceptions

The following chapter provides a collection of examples of different kinds of error messages and
exceptions.

Log Files
Not Enough Tablespace

[Link]: ORA-01691: unable to extend lob segment

BIS6_HP.SYS_LOB0000020764C00004$$ by 8192 in tablespace SEELOB8
at
[Link](DatabaseErro
[Link])
at
[Link]([Link])
at
[Link]([Link])
at [Link]([Link])
at
[Link].T4CPreparedStatement.doOall8(T4CPreparedStat
[Link])
at
[Link](T4CPrepa
[Link])
at
[Link](OracleS
[Link])
at
[Link](Orac
[Link])
at
[Link](Oracle
[Link])

OutOfMemory

@LGW800 2007-04-09T11:24:04.031+0200
[Link] JBoss Thread-
420851 LOCALHOST
ERROR SID=SESSION_1176110644015166 LOCATION [OFTP-TCP,
SESSION_1176110644015166, null] >> ERROR TYPE [,
COMPONENT_ERROR, not
retry able, fatal]] >> DESCRIPTION [Error occurred while trying
to start
session [SESSION_1176110644015166]: [Link]:
unable to create

Monitoring, Reporting and Maintenance 41

 Process Inspector

new native thread >> null] [SESSION_1176110644015166]

[Link]: unable to create new native thread
at [Link](Native Method)
at
[Link]
able([Link])
at
[Link](Session
[Link])
at
[Link](O
[Link])
at
[Link]
onnection([Link])
at
[Link]
ection([Link])
at
[Link]
tion([Link])
at
[Link](TCPCo
[Link])
at [Link]([Link])

Process Inspector
The Process Inspector provides detailed information of running processes and activities.

Monitoring, Reporting and Maintenance 42

 Process Inspector

1. Open the front-end.

2. Select: Monitoring (1) | All Processes (stored by state or time) (2), and double-click on a process of
dialog all processes (3).
3. A dialog process inspector will open (4).
4. Afterwards, click the right-mouse-button in the activity bar (5) and select Time (6) from the
context menu.
5. A new column will appear showing the execution times by activity (7).
6. Variable values of a particular activity state are also available (8).

 Maximum duration (by time) gives an indication for long running activities.

TPM-Lookup: Missing Entity

Monitoring, Reporting and Maintenance 43

 Process Inspector

It’s possible to jump into the process script, to the location where the process failed, by clicking the
Show fault button.

BIS Inspector

• For abortive processes, a fault view dialog is available.

• A fault can have multiple Exceptions – they can be inspected in the fault view dialog.
• The fault view displays four tabs which show the following content – not each tab is used at any
fault type.
◦ Reason/solution
◦ Fault variable
◦ Exception cause
◦ Stack trace

Monitoring, Reporting and Maintenance 44

 Process Inspector

TPM Entry Cannot be Found

Fault name: {[Link]

Fault variable: Variable TpLookupFault with following attributes:

• errorInfo: [Link]: No entity found for requestTP with

◦ (MessageType=N/A)
◦ (fileMask=[Link])
◦ (folderID=229e93d0-866f-11d9-bb0f-03e60a0a034d)
◦ (DocType=Inhouse)
◦ (MessageVersion=N/A)
◦ (EDISender=N/A)
◦ (EDIReceiver=N/A)
• errorName: Components framework failed.
• XPath: No Pointer for XPath

Monitoring, Reporting and Maintenance 45

 Process Inspector

Monitoring, Reporting and Maintenance 46

 Process Inspector

XPath: Missing Field

A reference to a field in a source variable is missing.

Exception-Class:

[Link]

Exception-Message:

[Link]:
No pointer for xpath:
/hotfolder:eventData/hotfolder:params/hotfolder:param[2]/hotfol
der:value

Exception-Reason:

Cause: No pointer for xpath:

/hotfolder:eventData/hotfolder:params/hotfolder:param[2]/hotfol

Monitoring, Reporting and Maintenance 47

 Process Inspector

der:value

XPath: Invalid System Variable

A system variable referenced by a seeutil:getSystemVariable() call does not exist.

Exception-Class:

[Link]

Exception-Message:

[Link]:

Cannot invoke public static [Link]

[Link]
ariable([Link])
throws [Link];

Could not find variable TEST in profile BISAS_DEFAULT_PROFILE

Exception-Reason:

Cause: Cannot invoke public static [Link]

[Link]
ariable([Link])
throws [Link];
Could not find variable TEST in profile BISAS_DEFAULT_PROFILE

XPath: Invalid Syntax

A call to bpws:getVariableData() without parameters.

Exception-class:

[Link]

Exception-message:

[Link]: Undefined function:

bpws:getVariableData

Exception-reason:

Monitoring, Reporting and Maintenance 48

 Process Inspector

Cause: Undefined function: bpws:getVariableData

XML-Schema: Missing Element

Missing attachment when calling Store-Service.

Fault name:

{[Link]

Fault variable:

[Link]: Unexpected element

{[Link]
tion
at
[Link]
[Link]
([Link])
at
[Link]
[Link]
([Link])
at
[Link]
[Link]
([Link])
at
[Link]
[Link]
([Link])
at
[Link]
[Link]
([Link])
at
[Link]
pl$[Link]([Link])
at
[Link]
[Link]([Link])
at [Link](Unknown
Source)
at
[Link](I
[Link])
at
[Link]([Link]:

Monitoring, Reporting and Maintenance 49

 Process Inspector

177)
at
[Link]([Link]:
214)
at
[Link]([Link]:
183)
at
[Link]([Link]:
214)
at
[Link]([Link]:
183)
at
[Link]([Link]:
72)

StoreService: Missing Access Rights

Error when trying to access the file system.

Fault name:

{[Link]

Fault variable:

[Link]:
F:\temp\test\[Link] (Zugriff verweigert)
at [Link](Native Method)
at [Link].<init>([Link])
at [Link].<init>([Link])
at
[Link]([Link]
:724)
at
[Link](StoreService.j
ava:460)
at
[Link](StoreService.j
ava:161)
at [Link].invoke0(Native Method)
at
[Link](NativeMethodAccesso
[Link])
at
[Link](DelegatingMetho
[Link])
at [Link]([Link])

Monitoring, Reporting and Maintenance 50

 DT Monitor

at
[Link].WSIFOperation_Java.executeReques
tResponseOperation(Unknown
Source)
at
[Link](Serv
[Link])
at
[Link](Inv
[Link])
at
[Link]
ct([Link])
at
[Link](Exe
[Link])
at
[Link](Executi
[Link])

DT Monitor
Example Adapter Faults
• Adapter faults are visible in Front-end, in the folder Monitoring | Adapter-Monitor | Adapter-Errors

In the Details view, the following information is visible:

• (Varying) Waring flag (no concrete failure)

• Fault type
• Fault description
• Stack trace
• Nodes: Information about the adapter instance
• Session: Information about the aborted communication session
• General system information

Monitoring, Reporting and Maintenance 51

 DT Monitor

Monitoring, Reporting and Maintenance 52

 DT Monitor

Monitoring, Reporting and Maintenance 53

 DT Monitor

Monitoring, Reporting and Maintenance 54

 DT Monitor

Mail Adapter

Invalid Mail Address

[Link]: Method:"SendTask::sendMail()"
Error:"Can not SEND message."
Caused by:"[Link]: Invalid Addresses;

nested exception is:

class [Link]: 553 Undefined Mailbox : domain string is NULL.

No Connection to Mail Server

[Link] : Method:"SendTask::connect()" Error:"Can not get

transport." Caused by:"[Link]: Could not connect to SMTP host: localhost,
port: 25;

nested exception is:

[Link]: Connection refused: connect

SAP Adapter

Monitoring, Reporting and Maintenance 55

 DT Monitor

Connection to Gateway Broke Down

JCO_ERROR_SERVER_STARTUP:

partner not reached (host [Link], service 3300):

Stacktrace:

[Link]: JCO_ERROR_SERVER_STARTUP: partner not

reached (host [Link], service 3300)

[Link]$[Link]
rror([Link])

[Link]$[Link]
eptionOccurred([Link])

[Link]([Link])
[Link]$[Link]([Link]) [Link]$[Link]([Link])
[Link]([Link])

Caused by: [Link]$Exception: (129) JCO_ERROR_SERVER_STARTUP: Server startup

failed at Sat Oct 04 22:01:37 CEST 2008.

This is caused by either:

a) Erroneous server settings,
b) the back-end system has been shutdown,
c) Network problems. Will try next startup in 1 second.

Connect to SAP gateway failed

Connect_PM TPNAME=[Link], GWHOST=gwserver, GWSERV=3300

ERROR partner not reached (host [Link], service 3300)

TIME Sat Oct 04 22:01:37 2008

RELEASE 640

COMPONENT NI (network interface)

VERSION 37

RC -10

MODULE nixxi_r.cpp

LINE 8724

DETAIL NiPConnect2

SYSTEM CALL SiPeekPendConn

ERRNO 10061

ERRNO TEXT WSAECONNREFUSED: Connection refused

COUNTER 2

Monitoring, Reporting and Maintenance 56

 DT Monitor

[Link]$[Link]([Link]) [Link]$[Link]([Link])
[Link]([Link])

Connection to SAP Gateway Was Interrupted

Description:

Communication problem with the R/3 system detected.

RFC_ERROR_CANCELLED: gateway shutdown

Stacktrace:

[Link]: RFC_ERROR_CANCELLED: gateway

shutdown

[Link]([Link])
[Link]([Link])
[Link]$[Link]
rror([Link])
[Link]$[Link]
eptionOccurred([Link])
[Link]([Link])
[Link]$[Link]([Link])
[Link]$[Link]([Link])
[Link]([Link])

Caused by:
[Link]: RFC_ERROR_CANCELLED: gateway shutdown
[Link]$[Link]
rror([Link])
[Link]$[Link]
eptionOccurred([Link])
[Link]([Link])
[Link]$[Link]([Link])
[Link]$[Link]([Link])
[Link]([Link])

Caused by:
[Link]$Exception: (109) RFC_ERROR_CANCELLED: CPIC-CALL: 'SAP_CMACCPTP'

ERROR gateway shutdown

TIME Wed Oct 01 19:02:23 2008

RELEASE 640

COMPONENT CPIC (TCP/IP)

VERSION 3

RC 731

MODULE r3cpic.c

LINE 9278

COUNTER 1

Monitoring, Reporting and Maintenance 57

 DT Monitor

[Link]$[Link](Native Method)
[Link]$[Link]([Link])
[Link]$[Link]([Link])
[Link]$[Link]([Link])
[Link]([Link])

FTP-Client

Login Fault:

[Link]:

Unexpected reply:

530 User ftp-user01@[Link] cannot log in.

Could not Establish a Connection:

Description:

[Link]:

Error while reading reply:

Stacktrace:

[Link]: Error in session: 32ed5b80-8978-

11dd-9574-498a0ab60724

[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])

Caused by: [Link]: Error while

connection to remote host

[Link]([Link])
[Link]([Link])
[Link]([Link])

Unexpected Reply

[Link]:

Unexpected reply:

421 Terminating connection.:

[Link]: Error in session: 3c24a530-d3b8-

11dc-8bb9-a2fac08a72a4

[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])

Monitoring, Reporting and Maintenance 58

 DT Monitor

Caused by: [Link]: Error while

connection to remote host

[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])

Caused by: [Link]: Unexpected reply:

421 Terminating connection.

Reading Reply

[Link]:

Error while reading reply:

[Link]: Error in session: 3c24a530-d3b8-

11dc-8bb9-a2fac08a72a4
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
Caused by: [Link]: Error while
connection to remote host
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
Caused by: [Link]: Error while
opening control socket
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
[Link]([Link])
Caused by: [Link]: Error while reading
reply

Monitoring, Reporting and Maintenance 59

 DT Monitor

FTP-Controller

Wrong Password

User <Administrator> authentication failed for listener <OLD b2bgw/gbspcom listener> in LS 300.

No Partner Bound to a Listener

No addresses are assigned to listener <FTP listener BIS comm node LS 000>.

User not Available or not Assigned

FTP-User <Administrator> is not found, or is not active for listener <FTP listener BIS comm node LS
500> and LS 500.

AS2-Client

No Connection

403 Forbidden

Incomplete Configuration

AS2 Adapter failure

HTTPConnection missing. Check your AS2 outbound relation settings.

[Link]([Link])

Connection Refused

[Link]:

Connection refused:

connect:

AS2-Controller

Unknown Partner

403 - Partner with as2id TEST_ID not found.

OFTP Adapter

No Transfer Configuration

Failed to update session configuration.

No valid transfer configuration found in master data.

(Check assignment of own transfer address with ODETTE code O0001110101010 and partner transfer
address with ODETTE code O000111111111111 in TCP connection Test-Connection (ID: 02254c11-
2654-11dd-adc2-269da9fea959).) [SESSION_122276408822822833, initiator: Partner1 (ID:

Monitoring, Reporting and Maintenance 60

 DT Monitor

O0000000101010), responder: Own (ID: O000111111111111), connection: Test-Connection (ID:

02254c11-2654-11dd-adc2-269da9fea959)].

2008/09/30 - 10:41:28:509: ERROR [, CONFIGURATION_ERROR, not fatal, not retry-able] >>

DESCRIPTION: Unable to process incoming report (EERP) for transfer
IN_IBX00002515200809301041250001_ O0000000101010_O094200005561032698000TCP. Failed to
update session configuration. No valid transfer configuration found in master data. >>
DETAIL: Check assignment of own transfer address with ODETTE code O0001110101010 and partner
transfer address with ODETTE code O000111111111111 in TCP connection Test-Connection (ID:
02254c11-2654-11dd-adc2-269da9fea959). >> LOCATION [OFTP-TCP,
SESSION_122276408822822833, null,
[Link] ([Link],
356)]

[Link]([Link]
6)
[Link]([Link])
[Link]([Link])

No Connection Bound to the Listener

Configuration of inbound session failed (Incoming connection: remote socket address:

[Link]:3919, local socket address: [Link]:3305 [TCP listener OFTP listener BIS comm
node LS 100 (ID: 8858f990-2c93-11dd-be64-f8ebc08a7136): [Link]:3305].

No connection configuration found for calling partner (ODETTE code: O000111111111111).

(Either no valid connection is defined or listener [TCP listener OFTP listener BIS (ID: 8858f990-2c93-
11dd-be64-f8ebc08a7136): [Link]:3305] is not assigned.) [SESSION_122344464019074493].

Wrong Partner Configuration

Processing of task with orderId 108f9342-8faa-11dd-b58d-eb54c08a7136 and taskId 108f9342-8faa-

11dd-b58d-eb54c08a7136 failed.

Transfer OUT_ORDERS00023923200810011314340001_O094200005561032698000ESK_
O000111111111111 refused by partner Partner 1 (ID: 4bd6e5e1-8f85-11dd-b58d-eb54c08a7136).

(Reason: Invalid origin (Reason code: 3)) [SESSION_122285967126248971, initiator: Own -

O000111111111111 (ID: O000111111111111), responder: Partner1 (ID: O0001110101010),
connection: Test-Connection (ID: 6e051781-8f86-11dd-b58d-eb54c08a7136)].

Processing of task with orderId 108f9342-8faa-11dd-b58d-eb54c08a7136 and taskId 108f9342-8faa-

11dd-b58d-eb54c08a7136 failed.

Transfer OUT_ORDERS00023923200810011314340001_ O0001110101010_ O000111111111111

refused by partner Partner1 - O000111111111111 (ID: 4bd6e5e1-8f85-11dd-b58d-eb54c08a7136).

(Reason: Invalid origin (Reason code: 3)) [SESSION_122285967126248971, initiator: Own - (ID:
O000111111111111), responder: Partner1 (ID: O0001110101010), connection: Test-Connection (ID:
6e051781-8f86-11dd-b58d-eb54c08a7136)].

Communication Fault

Processing of task with orderId d2430bc0-9504-11dd-b58d-eb54c08a7136 and taskId d2430bc0-

9504-11dd-b58d-eb54c08a7136 failed.

Monitoring, Reporting and Maintenance 61

 Further Classification of Communication Faults

Unable to open connection to partner Partner1 (socket address: [Link]:3305).

Network disconnected.

Session aborted by local entity.

null [SESSION_122344838706934771, initiator: Own (ID: O0001110101010), responder: Partner1 (ID:

O000111111111111), connection: Test-Connection (ID: 02254c11-2654-11dd-adc2-269da9fea959)].

2008/10/08 - 08:46:27:261: ERROR [, COMMUNICATION_ERROR, not fatal, is retry-able] >>

DESCRIPTION: Processing of task with orderId d2430bc0-9504-11dd-b58d-eb54c08a7136 and taskId
d2430bc0-9504-11dd-b58d-eb54c08a7136 failed. Unable to open connection to partner Partner1
(socket address: [Link]:3305). Network disconnected. Session aborted by local entity. null
>> LOCATION [OFTP-TCP, SESSION_122344838706934771, d2430bc0-9504-11dd-b58d-
eb54c08a7136, [Link]
([Link], 145)]

[Link]([Link])

There is already an Open Session with a Partner

Unable to register session for incoming connection (remote socket address: [Link]:52983,
local socket address: [Link]:3305 [TCP listener OLD OFTP listener (ID: d8be27a0-2ce0-11dd-
9dc7-cb5fc08a7136): [Link]:3305]).

(A session for connection Test-Connection (ID: 8402b010-eaae-11dc-8bb9-a2fac08a72a4) already

exists.) [SESSION_122340972839773333, initiator: Partner1 (ID: O000111111111111), responder: Own
(ID: O0001110101010), connection: Test-Connection (ID: 8402b010-eaae-11dc-8bb9-a2fac08a72a4)].

Protocol Violation of a Partner

Protocol violation: invalid state transition.

(State: WFCD, event: SFID) [SESSION_122277613392045374, initiator: Partner1 (ID:

O000111111111111), responder: Own (ID: O0001110101010), connection: Test-Connection (ID:
8bdb7570-28b5-11dd-a467-8d2bc08a72ac)].

Further Classification of Communication Faults

• Installation Problems
◦ Unlimited Strength Policy
• Configuration Problems
◦ General Resource Configuration Problems
◦ MSN Resource Configuration Problems
◦ Encoding Problems
◦ UserConfig_*.xml Problems
◦ Worklist Handler Configuration Problems
◦ Certificate Handling Problems
▪ Exchange existing SSL certificate Problems
▪ Certificate Cipher Problems

Monitoring, Reporting and Maintenance 62

 Database

• Connectivity Problems
◦ Network Problems
◦ Firewall Problems
◦ ISDN Router Problems
◦ Authentication Problems
◦ Certificate Problems
◦ HTTP Exception List
◦ HTTP Status Codes
• Transfer Session Problems
◦ Duplicate Check Problems
◦ Timeout Problems
◦ Initiate Problems
◦ Recovery Problems
◦ Max Retry Reached Situation
• System Critical Problems
◦ OutOfMemory in BIS Instance
◦ Blocked Adapter
◦ Blocked Session
◦ Blocked Reservations
◦ Blocked and Corrupt Orders
• Performance Problems
◦ Throughput Problems

Database
Database Connection Interrupted
[Link] JB11:45:24,73 SchedulerDataReload
{Default}@Web-Service Controller LOCALHOST ERROR
{[Link]=[Link]}{Module=Web-Service
Controller}{} Failed to load Scheduler tasks on profile <Default>!

[Link]: Could not load the children of node

"/com/seeburger/scheduler/persistence/db/SchedulerTaskEngine".

at
[Link]([Link]
32)

at [Link]([Link])

at
[Link]([Link])

at
[Link]([Link]
11)

Monitoring, Reporting and Maintenance 63

 Database

at
[Link]([Link]
88)

at
[Link]([Link]
79)

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

Caused by: [Link]: The connection to "SeeAS" database has been lost. URL:
jdbc:jtds:sqlserver://testserver:1433/SeeASDB-BIS;sendStringParametersAsUnicode=false

at
[Link]([Link]
2)

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

at [Link]([Link])

at
[Link]([Link]
23)

i. 9 more

Thread: JMS SessionPool Worker-17 : priority:5, demon:true, threadId:261, threadState:RUNNABLE,

lockName:null

[Link].socketRead0(Native Method)

[Link]([Link])

[Link](Unknown Source)

[Link](Unknown Source)

[Link](Unknown Source)

[Link](Unknown Source)

[Link](Unknown Source)

Monitoring, Reporting and Maintenance 64

 Database

[Link](Unknown Source)

<more>

[Link](MInitiatorReceiverEJB.j
ava:272)

[Link](Unknown Source)

[Link]([Link])

[Link]([Link])

[Link]([Link])

<more>

[Link]: <ERROR> <[:validateState]

Error transitioning from state DeathPath to DeathPath state!>

arg=(DeathPath,DeathPath)

ctx=([Link]='[Link]')

at [Link]([Link])

at
[Link](AbstractBPELStat
[Link])

at
[Link]([Link]
a:90)

at [Link]([Link])

at
[Link]([Link]
:145)

at
[Link]([Link]
)

at
[Link]([Link]
)

at [Link]([Link])

at
[Link](BusinessProcessImp
[Link])

at
[Link](BusinessProcessImp
[Link])

Monitoring, Reporting and Maintenance 65

 Database

at [Link]([Link])

at
[Link]([Link])

<more>

Monitoring, Reporting and Maintenance 66