0% found this document useful (0 votes)
7 views7 pages

Thread Monitor YAML Configuration Guide

The document discusses the Thread Monitor tool which automatically collects thread dumps and javacores at configured intervals. It can be triggered by high thread usage and other events. The tool is configured via a YAML file and collects diagnostic information to help troubleshoot performance issues.

Uploaded by

raghunandhan.cv
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
7 views7 pages

Thread Monitor YAML Configuration Guide

The document discusses the Thread Monitor tool which automatically collects thread dumps and javacores at configured intervals. It can be triggered by high thread usage and other events. The tool is configured via a YAML file and collects diagnostic information to help troubleshoot performance issues.

Uploaded by

raghunandhan.cv
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Thread Monitor - Yaml configuration file

The Thead Monitor tool gathers thread dumps and Javacores at a configured interval, and during
events such as high WebContainer/Default_Executor pool thread usage. Yaml configuration file
must exist with name /SETUP/support/thread_monitor.yaml, or under the location specified with
the THREAD_MONITOR_CFG environment variable.

The Yaml file is loaded during startup, and scanned for changes every minute. If the Yaml file is
not readable during startup, no further attempts are made to load it during runtime.

By default, Thread Monitor is enabled.

Yaml file configurations:

The following is a list of various Yaml file configurations:

 enabled: main switch. Uses THREAD_MONITOR_ENABLED environment variable. Defaults to


false enabled: "${THREAD_MONITOR_ENABLED:-false}"
 format: threaddump or javacore. Threaddumps are less costly format: threaddump.
 directory: directory to which the threaddumps are written, and where archiving occurs.
 It needs to match the directory used with -Xdump that configures where Javacores are
written directory: "${DUMP_DIR}".
 minCollectionInterval: To ensure threaddumps/javacores are not collected too frequently.
Useful when triggers are enabled. Value is in seconds. It is not recommended to allow
intervals shorter than 15 seconds. Use 0 to disable. minCollectionInterval: 15

 archiving:
o delayedZipNum: when doing realtime debugging, it is useful to read dumps as

they get created.


o This setting delays zipping the most recent N dumps so they remain available to

review with 'vi'. delayedZipNum: 10


o zipSize: number of dumps to include per zip file zipSize: 30
o zipMaxAgeMins: deletes zips whose lastmodified time is older than the

configured time in minutes. 0 to disable zipMaxAgeMins: 2880


o zipMaxCount: only keep the last N zip files. 0 to disable zipMaxCount: 96

 threaddumps:
include: used to filter the threads that are saved to the threaddump file. Only thread
names that start with the specified strings will be logged include:

o WebContainer
o Default Executor
o main
o pool
o redisson

 triggers:
o interval: enables periodic collection of dumps. Value is in seconds. Interval:
30

o webContainer: triggers dumps based on WebContainer/Default_Executor thread

pool usage. The pool usage is evaluated when the monitor API triggers. When
using prometheus, the default scrape interval is 15 seconds.
o inUseLargerThan: trigger a dump when the number of active (not idle) threads is

larger than the configured value inUseLargerThan: 21


o format: use a format different than the default. When the number of threads

available in the thread pool is low, it can be preferred to collect Javacores instead
of Threaddumps as they have more information format: javacore.

 Thread monitoring tools


Javacores, and thread stacks in particular, are key debugging tools. Comparing stacks
across threads, and for different points in time, can help narrow down the cause of
performance slowdowns.

Thread monitoring tools


Javacores, and thread stacks in particular, are key debugging tools. Comparing stacks across
threads, and for different points in time, can help narrow down the cause of performance
slowdowns.

Timing is a major factor when diagnosing issues using Javacores and thread dumps. Thread
dumps must be collected at the exact time when the server is experiencing performance issues.
Some events are sporadic and short in duration, which makes the collection difficult. Often, by
the time a problem is detected and a manual dump is collected, the problem has passed.

Automatic thread dump collection

The Thread Monitor tool automates thread dump collection. It offers options for the format,
frequency and triggering of dumps. It integrates with the metrics framework to enable automatic
dump generation when certain thresholds, such as number of threads in use, are reached. With
this configuration, the Thread Monitor can automatically trigger the collection of dumps during
spikes or slowdowns.
The trigger, which is based on the number of threads in use, requires metrics to be enabled
in [Link].

The Thread Monitor tool is configured with the following YAML file which is available in the
Transaction Server, QueryApp, Solr Search and CRS containers. The configurations include
format (i.e. Thread dump or Javacore), archiving, and threshold triggers.

For details of each configuration option, see the YAML file:

/SETUP/support/thread_monitor.yaml

The Thread Monitor is enabled by default since HCL Commerce 9.1.9.

It is configured to collect dumps in "thread dump" format (a HCL custom format), when the
number of threads in use in the Default Executor/ Web Container pool exceeds a threshold value
of 20. It will collect a maximum of 10 dumps per hour. These configurations can be updated to
better match the scenario you are debugging.

Changes to the configuration in thread_monitor.yaml are automatically loaded at runtime. A


container restart is not required.

When a dump is collected due to the [Link] threshold trigger, the


following warning will be printed to the logs:

[4/12/22 16:20:43:937 GMT] 00000124 WebContainerI W


[Link]
evaluateWebContainerUsage( final int inUse ) WebContainer threads in use 21
is higher than threshold of 20. Generating threaddump...

The Thread Monitor can also be enabled and disabled from vault by configuring the environment
variable as follows:

${TENANT}/${ENVIRONMENT}/${ENVTYPE}/threadMonitorEnable/${component_name}:
value=true

ThreadAnalyzer utility

The Thread Monitor tool generates Javacores or thread dumps in the directory specified by
the DUMP_DIR environment variable. The ThreadAnalyzer utility can be used to facilitate the
analysis. This is particularly helpful when dozens or hundreds of dumps are created. Manual
review can be time consuming.

The tool is available in the containers that enable the Thread Monitor. It can be found at the
following path: /SETUP/support/[Link].

The usage is as follows:

java -jar [Link]


Use -y when running inside a container
usage: java -jar [Link] [options]
-c,--config <arg> Optional path to a Yaml configuration file
-d,--directory <arg> Directory or file where the dumps are found. If a
file is specified, only that dump is analyzed. If
the file is a Zip, all the dumps within the zips
are parsed. If the location is a directory, all
dumps and zips within the directory will be parsed
(this does not include subfolders). When not set,
the tool attempts to determine the directory to
use automatically. If a directory is not
discovered, it defaults to the current directory.
-f,--follow If enabled, the tool will continue to monitor for
new threaddumps. This option is ignored if the
parameter is a file, not a directory. When -f is
set, -t is ignored.
-h,--help Prints this help.
-r,--report <arg> Comma separated list of reports. Defaults to 'Base
Web Activity'.
-s,--since <arg> Optional flag to limit the number of threadumps
Defaults to 3 hours. See 'time format'.
-t,--to <arg> Optional flag to limit the number of threadumps.
Defaults to now. See 'time format'.
-x,--extract-config Prints default config in stdout. This option is
used to customize the config, that can be then
specified with the -c option.
-y,--yes Required to acknowledge the utility can have an
impact when used within the container.

Time format: N[s=seconds, m=minutes, h=hours, d=days] OR yyyy-MM-dd HH:mm


OR epoch (e.g. 1650375222829) OR [Link][ss]

Available reports:
- Top Web Frames (tw): Reports the stacks most frequently found in threads
- Base Web Activity (ba): Summary of total base activities
- Web Activity (wa): Lists the discovered activity for each tread
Note: WARNING: When executed within the container, this utility takes CPU and memory
resources. The amount will vary depending on the number of dumps processed. For production
environments, it is recommended to use this utility outside the container. Using -Xmx to limit
memory usage is also highly recommended.

ThreadAnalyzer activity logic

The ThreadAnalyzer tool generates reports such as "Base Web Activity" and "Web Activity " that
rely on logic to "discover" key activites from a stack. The configuration is generally based on
applying RegEx patterns to stack frames, and to look for certain known classes and methods.

The following example attaches DB2 activity to a stack with frames that include the
package [Link]:
- activity: DB2
startsWith: [Link]
consumePrevious: true

The default configuration can be reviewed and extended.

 Use the -x argument to print the default configuration to the screen. The YAML output can
be redirected to a file for editing:

java -jar [Link] -x -y

 The customized configuration can be applied using the -c argument:

java -jar [Link] -c custom_config.yaml

ThreadAnalyzer reports

Base Web Activity report

The "Base Web Activity" report summarizes activity for each thread dump or Javacore. It
includes the time of the dump, the file name, the total number of non-idle threads, and the base
activities found. This report is useful to get a high level perspective of problem areas and to
identify timeframes, which typically show the most busy threads.

An example of the Base Web Activity report is as follows:


2021/10/26 13:10:12 - [Link]: 3 -
SocketRead[DB2]:3
2021/10/26 13:10:42 - [Link]: 10 -
SocketRead[DB2]:10
2021/10/26 13:11:12 - [Link]: 12 -
SocketRead[DB2]:7 CMD(RetrieveCatalogEntryTask):3 DB2[DBQuery]:1
CMD(CheckCatalogEntryEntitlementBySearch):1
2021/10/26 13:11:27 - [Link]: 15 -
CMD(RetrieveCatalogEntryTask):5 CMD(RetrieveContentTask):4 Web:4
CMD(InsertMoreMarketingContentAttachmentReferenceData):2
2021/10/26 13:11:42 - [Link]: 5 - Web:2
Commit:1 CMD(RetrieveContentTask):1 SocketRead[DB2]:1
2021/10/26 13:12:12 - [Link]: 2 -
DB2[DBQuery]:3
2021/10/26 13:13:27 - [Link]: 2 - Web:1
SocketRead[DB2]:1
2021/10/26 13:13:42 - [Link]: 1 -
SocketRead[DB2]:1
2021/10/26 13:14:12 - [Link]: 1 -
CMD(RetrieveCatalogEntryTask):1

Web Activity report


The "Web Activity" report shows details at the thread level for each dump found, including the
thread state and its activities.
2021/10/26 13:11:12 - [Link]:
WebContainer : 0 RUNNABLE [SocketRead[DB2], DBUpdate, Web]
WebContainer : 3 RUNNABLE [SocketRead[DB2], DBUpdate, Web]
WebContainer : 9 RUNNABLE [CMD(RetrieveCatalogEntryTask),
CMD(FilterCatalogEntryTask), Web]
WebContainer : 10 RUNNABLE [SocketRead[DB2], DBUpdate, Web]
WebContainer : 11 RUNNABLE [SocketRead[DB2], DBQuery,
CMD(RetrieveContentTask), CMD(FilterContentTask), Web]
WebContainer : 13 RUNNABLE [SocketRead[DB2], DBUpdate, Web]
WebContainer : 16 RUNNABLE [CMD(RetrieveCatalogEntryTask),
CMD(FilterCatalogEntryTask), Web]
WebContainer : 18 RUNNABLE [DB2[DBQuery],
CMD(RetrieveContentTask), CMD(FilterContentTask), Web]
WebContainer : 19 RUNNABLE [CMD(RetrieveCatalogEntryTask),
CMD(FilterCatalogEntryTask), Web]
WebContainer : 20 RUNNABLE [SocketRead[DB2], DBQuery,
CMD(RetrieveContentTask), CMD(FilterContentTask), Web]
WebContainer : 21 RUNNABLE [SocketRead[DB2], DBQuery,
CMD(RetrieveCatalogEntryTask), CMD(FilterCatalogEntryTask), Web]
WebContainer : 22 RUNNABLE
[CMD(CheckCatalogEntryEntitlementBySearch),
CMD(CatalogFilterGetContractUnitPrice), CMD(ComposePriceForCatalogEntry),
CMD(RetrieveCatalogEntryTask), CMD(FilterCatalogEntryTask), Web]

Top Web Frames report

The "Top Web Frames" report lists the most frequent stack frames for all the dumps found.

87 threaddumps, 98 total threads analyzed


278 (284%)
[Link]([Link]
:167)
244 (249%)
[Link](MeasuredCacheableC
[Link])
228 (233%)
[Link]([Link]
a:55)
128 (131%)
[Link]
[Link](AbstractBusinessObjectDocumentFacade
[Link])
128 (131%)
[Link]
[Link]([Link])
128 (131%)
[Link]
[Link]([Link])
128 (131%)
[Link]
[Link](AbstractBusinessObjectDocumentFacadeClient.j
ava:529)
125 (128%)
[Link].e
xecuteCommand([Link])
125 (128%)
[Link].p
rocessBusinessObjectDocument([Link])
117 (119%)
[Link]
mmand([Link])
109 (111%)
[Link]
[Link]([Link])
96 (98%)
[Link]
[Link]([Link])

Percentages in excess of 100% mean the stack was found multiple times on a single thread.

The report allows for configurations (using --extract-config and --config) as follows:

report:
...
topWebFrames:
minPercentage: 5
excluded:
- [Link].
- [Link].
- javax.
- java.
- [Link].
- [Link].

Common questions

Powered by AI

The YAML configuration file plays a crucial role in the adaptability of the Thread Monitor tool by allowing administrators to define and adjust various operational parameters dynamically without needing a system restart. This includes settings for enabling the tool, specifying dump formats, setting thresholds for triggers, and managing dump collection and archiving policies . By editing the configuration file, administrators can swiftly adapt the tool's functionality to address different performance scenarios or system environments. This flexibility is particularly advantageous in dynamic or evolving production environments where conditions and monitoring requirements frequently change .

The ThreadAnalyzer utility supports performance troubleshooting by generating detailed reports that focus on different aspects of application behavior. The 'Base Web Activity' report provides a summary of activities during each dump or Javacore, helping pinpoint busy timeframes and potential bottlenecks . The 'Web Activity' report gives a thread-level breakdown of activities, states, and stack traces, aiding in the identification of problematic threads and operations . Additionally, the 'Top Web Frames' report highlights the most frequent stack frames, which can be indicative of common paths leading to performance issues. These reports, derived from systematically analyzing dumps, empower administrators to make informed decisions about performance optimizations .

The 'thread_monitor.yaml' file significantly influences the operation and efficiency of thread monitoring through various configuration settings. Enabling or disabling the Thread Monitor, specifying the format of collected data (such as threaddump or javacore), setting the directory for storing dumps, defining collection intervals, and establishing archiving criteria all contribute to tailoring the monitoring process to specific needs. For example, setting a minimum collection interval prevents excessive data generation, and defining trigger conditions, like 'inUseLargerThan', ensures that dumps are collected precisely during high load periods, increasing efficiency in diagnosing issues . By allowing dynamic reconfiguration without requiring a server restart, it enhances operational flexibility .

Configuring 'minCollectionInterval' and 'trigger-based collection' greatly impacts the effectiveness of the Thread Monitor tool by optimizing when and how often thread dumps are collected. 'minCollectionInterval' prevents the frequent generation of dumps in quick succession, conserving system resources and avoiding unnecessary data redundancy, with the default advised to be not below 15 seconds . Trigger-based collection, which relies on thresholds such as the 'inUseLargerThan' setting, ensures that dumps are captured during significant events, such as high thread usage, allowing for actionable insights directly related to performance anomalies . These settings combined allow for both efficient resource management and the timely collection of meaningful diagnostic data.

The Thread Monitor tool uses configurable mechanisms like archiving and collection intervals to prevent high resource consumption. The 'minCollectionInterval' setting avoids frequent collection of dumps, recommended to be no less than 15 seconds, to reduce excessive resource use . Archiving settings such as 'zipSize', 'zipMaxAgeMins', and 'zipMaxCount' manage the retention and storage of dumps, ensuring that old or excessive files are not consuming storage resources unnecessarily. These configurations allow administrators to balance between timely dump collection and resource preservation . The automatic triggering of dumps during high load scenarios is controlled by specific thresholds that prevent overuse and make monitoring more efficient .

The primary purpose of the Thread Monitor tool in Java applications is to automate the collection of thread dumps and Javacores to diagnose performance issues such as slowdowns and spikes in thread usage. This tool is configured using a YAML file which can specify formats such as 'threaddump' or 'javacore', archiving options, and threshold triggers based on thread pool usage. The tool is enabled by default in HCL Commerce 9.1.9 and can automatically trigger thread dumps when the number of active threads exceeds a configured threshold, such as 20 active threads . The YAML configuration file, located at /SETUP/support/thread_monitor.yaml, allows customization of settings like enabling or disabling the tool, specifying the directory for dumps, and setting intervals or triggers for dump collection .

The ThreadAnalyzer utility enhances thread dump analysis by providing tools to automate and facilitate the review of numerous dumps, which can be time-consuming. Key features include generating reports like 'Base Web Activity' and 'Web Activity,' using RegEx patterns to identify key activities within stack frames, and supporting configurations that allow customization of the analysis process. The utility can print default configurations for user customization and can process dumps in directories or zip files . It also provides detailed thread-level analysis and common stack frame frequency reports, helping identify bottlenecks or frequent issues in the system .

Not setting appropriate values for 'zipMaxAgeMins' and 'zipMaxCount' in the YAML configuration can lead to inefficient storage management and potential system administration challenges. If 'zipMaxAgeMins' is too large or disabled, expired dumps remain stored longer than necessary, consuming significant storage resources and potentially leading to performance degradation due to disk space exhaustion . Similarly, if 'zipMaxCount' is set too high, it results in the retention of excessive zip files, further amplifying storage waste. Conversely, overly restrictive settings may prematurely delete useful dumps, hindering effective performance diagnostics. Therefore, balancing these values based on system capacity and diagnostic needs is critical .

Automated thread dump collection improves troubleshooting efficiency significantly compared to manual methods by ensuring timing precision and reducing the delay between problem detection and data collection. Automated dumps are triggered based on real-time thresholds like high thread activity levels, capturing critical information at the exact moments of performance decline . This reduces the likelihood of missing transient issues that often resolve before a manual intervention can occur. Additionally, automation standardizes the collection process, minimizes human error, and frees up administrative resources to focus on analysis rather than data acquisition . This consistency and timeliness in data collection maximally support effective troubleshooting.

The 'include' configuration option enhances data collection specificity by allowing administrators to filter which threads are captured in a threaddump based on their names. This setting ensures that only relevant threads, such as those prefixed with 'WebContainer', 'Default Executor', or 'main', are included in the dumps . By filtering out unnecessary or less critical threads, this configuration reduces data volume and increases the relevance of the collected information, thereby making analysis more focused and efficient. This targeted approach is particularly beneficial in environments with high thread diversity or activity where processing all threads might be resource-intensive .

You might also like