SUCCESS WORKBOOK
Fine tune your
ServiceNow
platform with regular
performance
administration
Developed by the ServiceNow Best Practice
Center of Excellence
March 2022
SUCCESS WORKBOOK
What is ServiceNow maintenance and
performance tuning?
While we monitor your instance hardware in our data centers, every
organization makes configurations to their environments that can
affect them individually. If left completely unattended, small issues
can build up over time and can cause performance problems,
difficulty upgrading to new releases, or issues when deploying
additional applications. This workbook will provide guidance on how to
keep your instances fine tuned. Steps included are:
• Check for configuration that is not best practice
• Review your log data for errors and warnings
• Maintain your tables for peak performance
Why should I invest in topic?
W ith effective maintenance, you can improv e both the upgradeability and
scalability of your Serv iceNow instance as well as ensure that you don’t hav e
performance issues related to your configurations.
What will this workbook help me do?
Serv iceNow instances can be like cars—two people can own the same model,
year, and make of a v ehicle, but depending on how they use, maintain, and
customize it, they could hav e two entirely different experiences with the car.
Perform the tasks in all fiv e of the following steps as they are explained in the
workbook in order to keep your instance running optimally.
Step 1 – Daily instance maintenance
Step 2 – W eekly instance performance
Step 3 – Monthly instance performance
Step 4 – Quarterly instance performance
Step 5 – Keep your instance continually improv ing
How to use this Success Workbook
This workbook will guide you through the steps to get started with ServiceNow
gov ernance.
Start by reading through the W orkbook Checklist and rev iew the steps and
tasks to get started with ServiceNow gov ernance. From the checklist page, you
can either proceed through the workbook page by page or nav igate only to
the sections that you need using the hyperlinks.
Each section includes “check your progress” questions that you can answer to
test whether you need to complete the tasks listed on that page or if you’v e
already done so and should mov e forward. Hyperlinks are included throughout
the workbook so you can nav igate back and forth between the checklist
page and different sections.
2
SUCCESS WORKBOOK
Workbook Checklist: Fine tune your ServiceNow
platform with regular performance
administration
Step 1: Daily instance maintenance
Rev iew the System Diagnostics homepage
Rev iew the prev ious day’s slow transactions
Consider you use cases
Step 2: Weekly instance maintenance
Rev iew your scheduled jobs
Check for repeated errors in the error log
Look for excessiv e logging
Find log files ov er 1 GB
Find slow-running running jobs
Find long-running jobs
Trend your top 20 transactions
Step 3: Monthly instance maintenance
Monitor your table growth rates
Clean your tables
Clean your tables
Step 4: Quarterly Instance performance
Check for coding that could impact your ability to upgrade
Step 5: Keep your instance continually improving
Find and fix issues with your end to end response time
3
SUCCESS WORKBOOK
Step 1: Daily Instance Maintenance
What’s slowing you down might be simple to find. Check the three
following areas to see if your transactions or data pulls are issues.
Check your progress:
❑ Do you hav e a daily routine for monitoring your instances’ health?
❑ Do you check for errors in your systems diagnostics?
❑ Do you receiv e total response times for transactions?
❑ Do you monitor users who are pulling large data sets and rev iew their
business need?
If you answered “yes” to the questions abov e, proceed to the next step. If not,
complete these action steps to complete this step:
1. Review the System Diagnostics homepage
The System Diagnostics page tracks some high-lev el statistics for each of
the nodes (JVMs) in your instance. (See Figure 1.)
W hen you rev iew this information, don’t worry if the total number of JVM
Classes differs between nodes. This metric is showing the number of classes
that hav e been loaded and subsequently unloaded on each JVM.
Depending on the activ ities users performed on each node, you might
notice a legitimate disparity in what has been called since that JVM was
last started.
a) Go to the System Diagnostics homepage.
b) Rev iew the v alues on this page. You’ll see v alues either in real time at
the point the page is rendered or as cumulativ e counts (such as the
transactions and error v alues) since the node was last started (see the
JVM UP time).
c) Track this information in a spreadsheet or a table in your instance.
Include the uptime, number of errors since the last restart, the number
of transactions performed, and the number of logged-in users for each
node. W hile the Now Platform® does hav e built -in performance
graphs that show this information, they’re rendered on a per-node
basis. If you spot an uncharacteristic jump in these numbers, it can be
a good indicator there is an underlying performance issue you need to
identify and address.
4
SUCCESS WORKBOOK
Figure 1: System Diagnostics homepage
2. Review the previous day’s slow transactions
The Serv iceNow system logs module prov ides a v ariety of logs that you can use to
troubleshoot and debug transactions and ev ents that take place within the instance. The
transaction logs in the system logs table records all browser activ ity for an instance. See
image below:
Figure 2: Access transaction logs through system logs table
5
SUCCESS WORKBOOK
By rev iewing all users’ transaction information, you can see which transactions are taking more than
a reasonable amount of time. Before you start, ensure you hav e the Client Transaction Timings
plugin enabled to capture all the data. Also note, depending on the size of your instance, this table
can be huge and may time out before the results load, so ensure you load it with
'sysparm_filter_only=true' then specify a filter to start looking at results.
In the list of transactions, you can v iew the total response times along with:
• A breakdown of the composite parts – This includes the time spent
rendering in the browser, time spent on the serv er processing the
transaction, and calculated time spent in the network.
• The details of which node processed the request
• The IP address of the host making the request
• The user making the request
• W hen the transaction occurred
• The session ID – Since this is also captured, it’s possible to rev iew the
application logs to dissect ev ery action a user has performed in their session.
Look for:
• If there is a particular time of day when transactions execute slowly
• If these transactions are all being processed by the same node – This
suggests one or more transactions or background jobs are consuming large
quantities of memory.
• If the transaction response times are poor across all nodes – This typically
signifies the database was working harder than usual, impacting all
transactions.
You might notice that the top 10 slowest transactions were all issued by a single
user and are incident lists. If that’s the case, you can rev iew the user’s settings
or impersonate that user and try to recreate the issue.
You may also want to filter transactions by URL to analyze the slow transactions.
Additionally, reporting on aggregate response times can be a powerful way to
track how ov erall instance performance is changing with time.
For more detailed instructions on how to work with the transaction logs, see our
product documentation site.
3. Consider your use cases
How much data do your users truly need to rev iew in a single screen? If
you identify that your list transactions are slow, find out how much data
your users are requesting. W hen a user selects “Show 100 rows per page”
on a list, this sets a user preference. From that point forward, ev ery time
that user runs a list v iew, it will include 100 rows. This includes related and
embedded lists on forms as well as the list v iews where the user set the
preference.
6
SUCCESS WORKBOOK
Figure 3: User chooses to show 100 rows per page
If you hav e a sufficient number of users who are requesting high numbers of rows, you could experience
a platform wide performance degradation because those row counts are making high demands on the
JVM memory required to render the lists.
This becomes problematic when a table with many reference fields must render a list. The platform has to
build the relationships for all the reference fields for all the rows displayed on screen.
For most serv ice env ironments, agents can’t practically use more than 20–30 rows at a time. If the page
load is fast, you can make a good case for “paging” to the next chunk of results rather than scrolling
down. Consider to use the ‘[Link].per_page’ property to define the items per page drop down options
that are v isible for users.
Figure 4 [Link].per_page property
Consider remov ing any options for more than 50 rows at a time from the Now Platform. For
further details, v isit our knowledge base for an article called “Good practices to improv e
instance performance through Rowcount, Related Lists and Dashboards.”
Return to workbook checklist
7
SUCCESS WORKBOOK
Step 2: Weekly Instance performance
If routine tasks have become a problem, finding errors, warnings, large log files,
and slow jobs will help you get those tasks running smoothly.
Check your progress:
❑ Do your scheduled jobs run smoothly?
❑ Do you regularly inv estigate repeated errors and warning in your logs?
❑ Do you hav e a process to monitor users logging excessiv ely and large log
files?
❑ Do you regularly check for slow running jobs?
If you answered “yes” to each question abov e, proceed to the next step. If not, complete
these action tasks:
1. Review your scheduled jobs
By rev iewing your scheduled job activ ity, you can help ensure that
background activ ities, such as scheduled reports, discov ery sensors, and
other routine tasks, run smoothly. Check for anything that's running for
more than an hour (3,60,000 ms)
a) Nav igate to System Logs > Transactions (Background)
b) Apply a filter with the following conditions (see Figure 3):
i. Created > on > This week
ii. URL > starts with > JOB
iii. Response time > Greater than > 360000
NOTE: The response may take sev eral minutes to return. If you don’t
return any results for an hour, try the same steps again with a
more stringent v alue such as a half hour (18000 ms) Of course,
some scheduled jobs are going to take a long time because they
hav e a lot of work to process. Due to how the transaction log
tables are stored and rotated in the database, it is not possible to
use the “group by” function in the list v iew. Because of this, you
may find it easier to do you trend analysis by exporting the result
set to Excel.
c) If you see a job that has executed multiple times for a long duration,
drill down into what the problem is. The most common culprits are glide
record queries, which request information from large tables with un-
indexed “where” clauses or sorts/groups. These are often found inside
of scripted transform maps and sometimes inside of script includes or
business rules.
8
SUCCESS WORKBOOK
Figure 5: Filter showing all job transactions created in the current week that took
m ore than 360,000 m s to com plete
2. Configure scheduled jobs to use “Burst” scheduler workers
To insulate against backed up scheduler worker queues, set the Priority
field on the sys_trigger entry for the scheduled job to 25. This ensures that
the core jobs—ev ent processors, SMTP sender, POP reader, and SMS
sender—get triggered in a timely fashion. Should all the scheduler workers
be busy with other jobs, an “important” job, which is more than 60 seconds
past due, will spawn a “Burst” scheduler worker and execute in parallel to
the core eight schedulers on the node.
Practitioner insight: Using “Burst” scheduler worker is good insulation,
but don’t use it as an excuse to av oid addressing the root causes of
the other long-running or high-v olume scheduled jobs.
3. Check for repeated errors in the error log
a) Nav igate to the System Log.
b) Select Errors.
c) Look for actionable errors as well as frequency within the warning messages.
d) Look for an increased v olume in the number of errors by checking the total number in
the top right corner of the screen.
e) If you see a message like
[Link].sys_script_include_5daa9bf593233100fa71b33e867ffb9b_scrip
t_2555.call(sys_script_include_5daa9bf593233100fa71b33e867ffb9b.script, you can
discov er more about the error by examining the script_include record with that sys_id.
f) Look for repeated errors in the warnings log
I. Nav igate to the System Log.
II. Select Warnings.
III. Look for actionable warnings as well as frequency.
IV. Based on the warnings you see, you may be able to search through a sys_script for
the text output.
9
SUCCESS WORKBOOK
4. Look for excessive logging
Next, look for unusually large log files. This is a relatively crude—but
surprisingly accurate—way to spot potential problems that warrant closer
attention.
a) Nav igate to Utilities > Node Log File Download.
b) Apply a Name starts with local filter. This will show you all the
application logs for the node your session is activ e on.
c) Note that the most recent fiv e days of log files are unzipped, and the
remaining files are zipped. The size v alue is measured in KBs. If you
notice that one day is significantly larger than the others, or that there
is a progressive increase in file size, you may need to inv estigate
further.
Practitioner insight: The application logs all transactions and
associated parameters, so if the number of users has ramped up or
a new piece of functionality has gone liv e, the log files will naturally
increase.
5. Find log files over 1GB
Log files ov er 1 GTB may suggest possible frequent errors or logging issues that you need to
fix.
a) First, look for a significant spike in log file size.
b) Note: This may indicate that the [Link] or [Link] statements ,which were used in sub
production testing, hav e not been remov ed. Unnecessary logging makes the tables
bulky, which slows maintenance activ ates, like backups, and makes searching the
syslog table slow and cumbersome. If that’s the case, try to remov e the [Link] and/or
[Link] statements (unless you need them) and complete steps 1-4 again.
c) Find the log files that are ov er 1 GB.
Figure 6: A log file over 1 GB
10
SUCCESS WORKBOOK
6. Find slow running jobs
a) Apply a Nav igate to the System Scheduler.
b) Select Slow Job Log.
c) View the job details in the URL and Response time columns.
d) Check the SQL time column for the time the job has been in the
database.
e) Check the Business rule time column for the amount of time the job
has been in logic (execution).
f) Right-click the Response time column heading and select Sort (z to a).
g) Rev iew the Response time, SQL time, and Business rule time to look for
suspiciously long run times.
Figure 7: Exam ple of a Slow Job log
7. Find long running jogs
a) Nav igate to User Administration.
b) Select Active Transactions.
c) If there is a background job running, it will show in the User column. Check the Age
column to see how long it’s been running.
d) To kill a job that’s been running for too long or seems to be completely stuck, right-click
the User name and select Kill. Only kill a job if you are ABSOLUTELY sure it is a stuck or
illegitimate long running job.
e) A confirmation message will appear at the top of the list.
11
SUCCESS WORKBOOK
Figure 8: Right-click m enu for killing a stuck job
8. Trend your top 20 transactions
Create a spreadsheet to trend your top 20 transactions. These may constitute the 20 most
executed transactions in a giv en week. Or you may choose to track the most business -
critical transactions (like incident or catalog transactions). Or it may be helpful to trend a
mixture of these. Keep tracking data week after week .
Refer to this knowledge base article on the Now Support Portal for adv ice on how to
inv estigate the performance of indiv idual transactions.
Return to workbook checklist
12
SUCCESS WORKBOOK
Step 3: Monthly Instance performance
Don’t let slow queries or transient data bog down your performance.
Practice these monthly reviews to avoid those issues.
There are broadly two types of data stored in your ServiceNow
instance:
• Persistent data that you want to retain, such as a task or user info
• Transient data that needs to be cleared after a given time frame,
such as log information or staging data for imports or integrations
It’s normal to see persistent data growth over time. But when you see
increased table sizes along with a decreased response time, you may
have list definitions or glide record queries that need to be refactored
or indexed so you can accommodate the data growth.
If there’s an increase in response times for end users and an increase in
execution time for maintenance tasks such as cloning, backup, and
restore, it probably means your data sets are growing. If that’s the
case, it’s important to monitor your table growth on a monthly basis. If
you created a spreadsheet to track your top 20 request response
times, you can extend it to track the number of rows rendering in your
tables, as well as your slow queries, so you can track their improvement
over time.
Check your progress:
❑ Do you regularly track table growth?
❑ Do you purge or archiv e data that is no longer needed in the activ e
system?
❑ Do you rev iew slow queries that might indicate a table that needs to be
indexed?
If you answered “yes” to each question abov e, proceed to the next step. If not, complete
these action tasks:
1. Monitor your table growth rates
W hen you check your table growth, you’ll look for two things:
a) Dramatic changes in size from month to month
b) The total number of records in your tables
Follow these steps:
a) Nav igate to System Definition.
b) Select Tables.
c) Your list of tables appears on the screen.
d) Filter the table information typing u_ in the box. This will
show you all the user-created tables .
e) To see the total number of records in a table, type <name
of table>.list in the nav igation field at the top left of the
screen.
13
SUCCESS WORKBOOK
Figure 9: A table list showing a total of four tables
f) Look for a dramatic change between the prev ious month and this
month. If you see an increase, you may need to inv estigate why the
spike occurred.
g) Check the total records.
If the total number of records in your tables is ov er 50K, you may
need to complete one of these tasks or a combination of them:
• Index the fields used in filters or other queries – To determine if
you need to index fields, rev iew the Slow Queries log. If you find
slow queries, contact ServiceNow for support about indexing.
• Set up table rotation with from ServiceNow technical support –
For more information, read the next section and our product
page on table rotation.
• Extend large tables to store more data or as a part of a table
rotation process – For more information, read our product page
explaining table extension for data retention.
• Clean tables by purging rows – For more information, read the
next section.
Practitioner insight: To see the total size of your database and the
10 biggest tables, use the Database Footprint service catalog item
on the Now Support portal.
2. Clean your tables
To list the data you may want to purge (such as incidents older than one
year):
a) Type [Link] in the left nav igation field, or type
[Link] [Link]/ incident_list.do into your
browser’s address bar.
b) A count of the records will display.
c) You may want to amend your show x records preference to 10 or 20 to
speed up the list rendering time.
14
SUCCESS WORKBOOK
Practitioner insight: If you discov er that you need to purge more
than one million records, submit a ticket on the Now Support portal
to handle the purge. Purging them yourself could cause an outage
or other issues.
3. Review the slow queries log
The Slow Queries log aggregates the data for similar queries.
The platform records any SQL statement where the total
execution time exceeds fiv e seconds.
Not ev ery slow query is a concern—you can expect to hav e a
few. The slow queries on user created tables, those that begin
with “u_,” are the ones that are cause for concern. Dramatic
changes in size from month to month could indicate an issue.
a) Nav igate to System Diagnostics > Stats > Slow Queries to
see your Slow Queries log. The platform records any SQL
statement that takes more than 100ms to complete. The
Slow Queries log groups these transactions into similar
patterns, prov iding you with an example set of parameters
Figure 10: A Slow Queries log
15
SUCCESS WORKBOOK
b) Type u_ in the Example text box
Figure 11: A Slow Query log showing the Exam ple text box.
The slow query log records the queries’ patterns since the beginning of time (or
since the last time sys_query_pattern was truncated). You may find the results
more meaningful by applying a filter to show only patterns that were first
sighted in the last month and that occurred more than 100 times.
If you click through to an indiv idual query pattern record, you’ll see an
example URL where the query was generated, the first and last sighting, the
number of executions, and the av erage execution time.
The stack trace of the thread executing the query also displays. From here:
a) Cross-reference which element on the screen requested the information.
b) Once you know this, you can rev iew the gauge or list that made the call
and v erify whether it would benefit from refactoring or supporting with an
index. Many times, you can significantly reduce the execution time by
simply adding active=1 to a query. This will only query and return activ e
records, thus reducing the number of records included in the query.
Return to workbook checklist
16
SUCCESS WORKBOOK
Step 4: Quarterly Instance performance
Running a quarterly rev iew of your upgrade history can tell you a lot about
your instance performance as you can see issues/performance trending ov er
time.
Check your progress:
❑ Hav e you checked your skipped, inserted, updated, and deleted records
for unusual behav ior such as odd growth patterns, mass deletes, etc.?
❑ Do you monitor changes or customizations to ServiceNow out of the box
object?
If you answered “yes” to each question abov e, proceed to the next step. If
not, complete these action steps:
1. Check your instance for any configurations that could impact your
upgrades
a) Nav igate to System Diagnostics.
b) Select Upgrade History.
c) A table showing your upgrade history appears. Under the Upgrade
started column, look for the last upgrade and select it.
d) W ithin the upgrade record, check the Skipped Changes to Review tab
to see what was skipped, updated, inserted, or deleted.
Figure 12: An upgrade record showing the Skipped Changes to Review tab
e) Rev iew the name of the table and the sys_id of the record
to see exactly what was skipped and where it was skipped
from.
f) Check the type of object that was skipped. If the upgrade
skipped an out-of-the-box object, consider rev erting that
object so it is updated with ev ery upgrade.
Return to workbook checklist
17
SUCCESS WORKBOOK
Step 5: Continually improve your instance
So far, all the activ ities mentioned in this workbook contribute to continuous
improv ement. Check out the tasks in this stage if you’re experiencing:
• Poor list response times
• Poor form load and submit response times
• Poor module response times
Check your progress:
❑ Hav e you checked form load response time?
❑ Hav e you checked form submit time?
❑ Hav e you monitored modules response times?
If you answered “yes” to the questions abov e, you hav e completed the steps
outlined in the workbook. If not, complete these actions:
1. Find out if low-response, form load, form submit, and module response
times are keeping your instance down.
a) Nav igate to your used forms and select Incident.
b) Select Create New.
Figure 13: A new incident record with the end-to-end response tim e bar (bottom right corner)
Return to workbook checklist
18
SUCCESS WORKBOOK
c) W hen the form opens, at the bottom right you will see the
details of the end-to-end response time displayed by a
colorful bar (see Figure 11), including:
• Response time (ms) – The total time between clicking
Create New and seeing the form load
• Network – The total time spent ov er the wire
• Server – The total time spent processing the request on
the serv er
• Browser – The total time the browser spent rendering
the form, including running the client -side script
d) Click any of the elements of the end-to-end response time
to see its details. (See Figure 14.)
Figure 14: An incident record showing end-to-end response time details
In the pop-up box, rev iew how much time each section, script,
field, etc., took to load. This information helps you identify any
potential bad scripts or bottlenecks in your load times so you can
address them.
W hen you hav e slow form load times, the most common causes
are:
• Related/embedded lists – To fix this, either look for a bad query
or filter or the number of rows you’re requesting.
• A high number of AJAX calls – You may want to consolidate
these into fewer round trips.
• An inefficient client-side script – Av oid synchronous AJAX calls
or DOM manipulation.
For additional assistance, check out:
Troubleshooting performance
Client transaction timings
Performance and performance debugging
Return to workbook checklist
© 2021 ServiceNow , I nc. All right s res erved. ServiceNow, the ServiceNow logo, Now, Now Platform, and ot her ServiceNow marks are t rademarks and/or
regis t ered t rademarks of ServiceNow , I nc. in the United St ates and/or ot her count ries . Other company and product names may be t rademarks of t he
res pect ive companies w ith which they are as s ociat ed.
19