0% found this document useful (0 votes)

123 views247 pages

BW MaintenanceGuide R210

Uploaded by

Francoj DA

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

123 views247 pages

BW MaintenanceGuide R210

Uploaded by

Francoj DA

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

Maintenance Guide

Release 21.0
Document Version 1

9737 Washingtonian Boulevard, Suite 350

Gaithersburg, MD 20878
Tel +1 301.977.9440

[Link]
BroadWorks® Guide

Copyright Notice
Copyright© 2014 BroadSoft, Inc.
All rights reserved.
Any technical documentation that is made available by BroadSoft, Inc. is proprietary and
confidential and is considered the copyrighted work of BroadSoft, Inc.
This publication is for distribution under BroadSoft non-disclosure agreement only. No
part of this publication may be duplicated without the express written permission of
BroadSoft, Inc., 9737 Washingtonian Boulevard, Suite 350, Gaithersburg, MD 20878.
BroadSoft reserves the right to make changes without prior notice.
Trademarks
Any product names mentioned in this document may be trademarks or registered
trademarks of BroadSoft or their respective companies and are hereby acknowledged.
This document is printed in the United States of America.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Release Version Reason for Change Date Author

14.0 1 Updated document for re-branding. March 15, 2006 Robb Surridge

14.0 1 Reformatted styles for re-branding. April 24, 2006 Roberta Boyle

14.0 1 Edited changes. June 9, 2006 Patricia Renaud

14.0 1 Added material to section 9.1 Automatic June 28, 2006 Robb Surridge
Backups for EV 31273, added [Link]
file to backup list; added paragraph to
section 9.1.3 Restore Auto-Backup.
INFO corrected to UPDATE for EV 31765
for bwAuditAbnormalCall-Termination in
section [Link] SIP/MGCP Traps.
Added recommendation for
check_dbpages.pl script in section 10.5.4
Update TimesTen Database Paging
Information for EV 33301.
Corrected examples in section 6.1.2 View
System Alarms for EVs 33243 and 33195.
Updated section 14 Install BroadWorks
License Files for EV 32926.
Corrected [Link] example in section
10.3 Database Import for EV 34367.

14.0 1 Updated document with Release 14.0 July 20, 2006 Mario Goupil
content.

14.0 1 Added index. July 21, 2006 Andrea Fitzwilliam

14.0 1 Edited document. August 24, 2006 Patricia Renaud

14.0 2 Added disaster recovery strategies, and August 29, 2006 Robb Surridge
minor textual updates to correct EV 34678.

14.0 2 Added CommonPersistency input channel September 11, 2006 Maxime Larose
parameters.

14.0 2 Edited changes. October 17, 2006 Patricia Renaud

14.0 3 Added new section for EV 37043, (section March 15, 2007 Roberta Boyle
20.28 Software Manager Utilities).
Changed note for EV 37037 in section
8.1.1 Display Full Details about Active
Software Versions and added text about
-showPatch option in section 20.5
bwshowver.
Added warning in section 20.23 stopbw for
EV 39877.

14.0 3 Edited changes. March 16, 2007 Patricia Renaud

14.0 4 Modified document based on Release 14 May 11, 2007 Tony Pilote
config-network changes.

14.0 4 Edited changes. May 21, 2007 Patricia Renaud

14.0 5 Added BEA WebLogic information. June 12, 2007 Martin Bernier

14.0 5 Edited changes and published document. June 20, 2007 Andrea Fitzwilliam

14.0 6 Corrected log file names for BEA for June 26, 2007 Martin Bernier
EV 49826.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

14.0 6 Added section on removing peer from July 3, 2007 Roberta Boyle
redundant system for EV 50350.

14.0 6 Edited changes and published document. July 6, 2007 Andrea Fitzwilliam

14.0 7 Updated section 9.1.3 Restore Auto- July 12, 2007 Martin Bernier
Backup to indicate that the backup restore
must be executed as root when done in the
system root directory for EV 50841.

14.0 7 Updated section 20.9 config-network to July 27, 2007 Patricia Renaud
add a note for HTTP Alias CLI command
for EV 38723.

14.0 7 Edited changes and published document. July 27,2007 Patricia Renaud

14.0 8 Updated section 5.3.3 Enable TimesTen August 30, 2007 Roberta Boyle
Logs for EV 49372.

14.0 9 Added missing CMS-related information in September 18, 2007 Tony Pilote
section 4 BroadWorks Processes.

14.0 9 Edited changes and published document. September 18, 2007 Patricia Renaud

14.0 10 Added a procedure for importing a October 1, 2007 Tony Pilote

database from a server with a different OS.

14.0 10 Updated section 20.9 config-network for October 9, 2007 Roberta Boyle
EV 50203.

14.0 10 Added section 21.2 Adjust BroadWorks October 9, 2007 Roberta Boyle
DST (Daylight Savings Time) for
EV 50504.

14.0 10 Updated section 14 Install BroadWorks October 16, 2007 Roberta Boyle
License Files for EV 54430.

14.0 10 Changed DevDebug to Debug in section October 16, 2007 Roberta Boyle
5.1.1 Log Severity for EV 53781.

14.0 10 Edited changes and published document. October 29, 2007 Andrea Fitzwilliam

14.0 11 Updated BEA Java GC logs information. February 8, 2008 Martin Bernier

14.0 11 Edited changes and published document. February 12, 2008 Andrea Fitzwilliam

14.0 12 Added section 9.1.1 Configure Basic February 13, 2008 Roberta Boyle
Maintenance Tasks for EV 59029.

14.0 12 Updated section 5.1.7 Media Server for February 20, 2008 Roberta Boyle
EV 59375.

14.0 12 Modified section 9 Scheduled Maintenance March 6, 2008 Jean-Francois

Tasks to add CDS and EMS daily DbMaint Dignard
task; also added section 20.12
[Link] for EV 53221.

14.0 12 Created section Optimization TimesTen March 13, 2008 Roberta Boyle
Checkpoint for EV 59930.

14.0 12 Edited changes and published document. March 24, 2008 Andrea Fitzwilliam

14.0 13 Created section 10.5.7 TimesTen April 2, 2008 Roberta Boyle

Connection Optimization for EV 60233.

14.0 13 Added section [Link] Xtended Services April 4, 2008 Martin Bernier
Platform Replacement.

14.0 13 Edited changes and published document. April 16, 2008 Andrea Fitzwilliam

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

14.0 14 Added a description of all the input May 1, 2008 Tony Pilote
channels used by BroadWorks.

14.0 14 Edited changes and published document. May 2, 2008 Andrea Fitzwilliam

14.0 15 Added Profile Server impacts. May 27, 2008 Yves Racine

14.0 15 Edited changes and published document. May 2, 2008 Andrea Fitzwilliam

15.0 1 Updated document for Release 15.0. July 9, 2008 Mario Goupil

15.0 1 Edited changes since Release 14.0, July 16, 2008 Andrea Fitzwilliam
removed references to CMS, and
published document.

15.0 2 Added configuration of CCR allowed proxy October 7, 2008 Frederic Lelievre
URLs in config-network tool and changed
Apache/Tomcat log files locations.

15.0 2 Modified [Link] section to October 9, 2008 Joël Collin

highlight its replacement utility, frepctl.

15.0 2 Added section 17 Change BCCT Listening October 31, 2008 Yves Racine
Ports for EV 62498.

15.0 2 Edited changes and published document. November 21, 2008 Andrea Fitzwilliam

15.0 3 Updated section 11.5.2 Element November 28, 2008 Goska Auerbach
Management System for EV 65145.

15.0 3 Added section 20.14.1 [Link] on December 2, 2008 Goska Auerbach

Element Management System for
EV 81954.

15.0 3 Updated section 12 Disaster Recovery January 14, 2009 Roberta Boyle
Strategy for EV 87423.

15.0 3 Added section 11.1.3 TimesTen January 14, 2009 Frederic Lelievre
Replication Port Number.

15.0 3 Edited changes and published document. February 5, 2009 Andrea Fitzwilliam

15.0 4 Updated section 11.5.1 Application Server, March 12, 2009 Patrick Nadeau
Network Server for EV 48848.

15.0 4 Edited changes and published document. March 23, 2009 Andrea Fitzwilliam

15.0 5 Updated server recovery procedure to April 16, 2009 Mark Kushnir
include information on file synchronizing.

15.0 5 Edited changes and published document. April 23, 2009 Andrea Fitzwilliam

15.0 6 Made minor change to section 9.1.1 May 15, 2009 Goska Auerbach
Configure Basic Maintenance Tasks for
EV 89729.

16.0 1 In section 11.5.3 Profile Server, removed May 25, 2009 Yves Racine
the manual step of importing data from
peer as this is now automatically done by
the script config-redundancy.

16.0 1 Updated document for Release 16.0. May 25, 2009 Mario Goupil

16.0 1 Edited changes and published document. July 22, 2009 Andrea Fitzwilliam

17.0 1 Updated document for Release 17.0 and October 28, 2009 Roberta Boyle
added section 20 BroadWorks Application
Server Time Zone Support for EV 101825.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

17.0 1 Updated section 10.5.1 Change TimesTen February 9, 2010 Goska Auerbach
Database Size for EV 106025.

17.0 1 Updated document for Release 17.0. March 22, 2010 Mario Goupil

17.0 1 Edited changes and published document. April 12, 2010 Andrea Fitzwilliam

17.0 2 Removed section on Optimization April 27, 2010 Goska Auerbach

TimesTen Checkpoint for EV 110711.

17.0 2 Updated section 12.2.3 BroadWorks April 30, 2010 Goska Auerbach
Server Restoration Procedure for
EV 109374.

17.0 2 Edited changes and published document. May 18, 2010 Andrea Fitzwilliam

17.0 3 Updated section [Link] Healthmon File July 20, 2010 Goska Auerbach
System Check for EV 112653.

17.0 3 Edited changes and published document. July 23, 2010 Andrea Fitzwilliam

17.0 4 Updated section 19.3 BroadWorks SNMP August 4, 2010 Tony Pilote
Trap Monitoring for EV 115577.

17.0 4 Added clarification for maintenance tasks August 9, 2010 Tony Pilote
in section 9 Scheduled Maintenance Tasks
for EV 115892.
Updated sections 11.1.2 TimesTen
Replication Monitoring and 11.2.2 MySQL
Replication Monitoring for EV 112000.

17.0 4 Reviewed changes. September 9, 2010 Patricia Renaud

17.0 4 Added section 9.10 Service License September 10, 2010 David Moriconi
Collect and updated document for
EV 114437.

17.0 4 Made minor editorial changes. September 14, 2010 Goska Auerbach

17.0 4 Updated section 10.5.1 Change TimesTen September 24, 2010 Goska Auerbach
Database Size for EV 118789.

17.0 4 Updated document with Release 17.sp1 September 29, 2010 Frederic Lelievre
information.

17.0 4 Edited changes and published document. November 1, 2010 Andrea Fitzwilliam

17.0 5 Updated section 10.3 Database Import for November 2, 2010 Goska Auerbach
EV 120645.

17.0 5 Updated section 21.2 Adjust BroadWorks December 10, 2010 Mario Goupil
DST (Daylight Savings Time) for
EV 123592.

17.0 5 Edited and published document. January 6, 2011 Patricia Renaud

17.0 6 Updated section 10.5.6 Import TimesTen January 24, 2011 Mario Goupil
Datastores between Different Operating
Systems for EV 114783.

17.0 6 Updated section 21.12 [Link] January 31, 2011 Tony Pilote
for EV 124825.
Updated section [Link] Network Server,
Application Server, and Messaging Server
Replacement for EVs 113264 and 122851.
Updated section 11.5.1 Application Server,
Network Server, and Messaging Server for
EV 116345.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

17.0 6 Edited changes and published document. March 8, 2011 Jessica Boyle

17.0 7 Updated section 7.10.1 Healthmon for March 16, 2011 Hang Tran
EV 129579.

17.0 7 Removed section about recreating the April 19, 2011 Mario Goupil
database by executing the [Link] script
for EV 119909.

17.0 7 Added section 21.3 Change Media Server May 3, 2011 Mario Goupil
Kernel Selection on Linux to add steps to
change kernel selection for
EV 121030.

17.0 7 Updated section 11.5.2 Element May 9, 2011 Mario Goupil

Management System for EV 141355.

17.0 7 Updated section [Link] Collecting Heap June 9,2011 Mark Kushnir
Size Indicator for EV 143495

17.0 7 Updated section 9.1.1 Configure Basic June 14, 2011 Goska Auerbach
Maintenance Tasks for EV 143768.

17.0 7 Edited changes and published document. June 27, 2011 Jessica Boyle

17.0 8 Removed section 21.12 [Link] July 18, 2011 Mario Goupil
and updated section 9.4 Database
Statistics Update for EV 124825.

17.0 8 Edited changes and published document. July 27, 2011 Jessica Boyle

17.0 9 Updated section 16 Change IP Address or August 30, 2011 David Moriconi
Host Name for EV 148587.

17.0 9 Added warning in section 10.5.1 Change September 27, 2011 Mario Goupil
TimesTen Database Size for EV 145227.

17.0 9 Corrected CLI reference from NetServ to September 29, 2011 Mario Goupil
NetworkServer for EV 150714.

17.0 9 Added CDS/CCRS restoration procedure September 29, 2011 François Isabelle
in section [Link] Call Detail Server/Call
Center Reporting Server for EV 125022.

17.0 9 Updated section 12.2.3 BroadWorks September 30, 2011 David Moriconi
Server Restoration Procedure for
EV 144853.

17.0 9 Updated section 20.19 resizeDSN for October 12, 2011 Mario Goupil
EV 151585.

18.0 1 Updated document for Release 18.0. October 16, 2011 Mario Goupil

18.0 1 Updated section 16 Change IP Address, October 19, 2011 Goska Auerbach
Host Name, FQDN Name, Gateway, or
DNS Server for EV 144590.

18.0 1 Edited changes and published document. October 21, 2011 Andrea Fitzwilliam

18.0 2 Updated section [Link] Collecting Heap October 24, 2011 Mario Goupil
Size Indicator for EV 147625.

18.0 2 Edited changes and published document. November 4, 2011 Jessica Boyle

18.0 3 Updated section 10.5.6 Import TimesTen November 8, 2011 Frederic Lelievre
Datastores between Different Operating
Systems to include UID and PWD
parameters.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

18.0 3 Added section [Link] Element November 22, 2011 David Moriconi
Management System to cover the disaster
recovery process for the EMS for
EV 130651.

18.0 3 Updated section 10.5.6 Import TimesTen November 30, 2011 Frederic Lelievre
Datastores between Different Operating
Systems to define UID and PWD
parameters as lowercase.

18.0 3 Updated section [Link] Element January 11, 2012 David Moriconi
Management System for EV 130651.

18.0 3 Updated section 16 Change IP Address, January 16, 2012 David Moriconi
Host Name, FQDN Name, Gateway, or DNS
Server for EV 154921.

18.0 3 Edited changes and published document. January 25, 2012 Jessica Boyle

18.0 4 Updated section 12.2.3 BroadWorks Server February 1, 2012 Goska Auerbach
Restoration Procedure for EV 130651.

18.0 4 Edited changes and published document. February 16, 2012 Jessica Boyle

18.0 5 Updated section 11.5 Add New Peer to March 21,2012 François Isabelle
Redundant System for EV 162349.

18.0 5 Removed references to the Conferencing March 27, 2012 François Isabelle
Server for EV 154846.

18.0 5 Edited changes and published document. April 10, 2012 Jessica Boyle

18.0 6 Updated section [Link] Network Server, April 17, 2012 Mario Goupil
Application Server, and Messaging Server
Replacement to add rsync command for
IpDeviceConfig files for EV 164187.

18.0 6 Edited changes and published document. April 27, 2012 Jessica Boyle

18.0 7 Updated section 11.5 Add New Peer to June 15, 2012 François Isabelle
Redundant System for EV 162349.

18.0 7 Updated section 20.8 config-hostname for July 18, 2012 Mario Goupil
EV 169451.

18.0 7 Updated section [Link] Network Server, August 2, 2012 Goska Auerbach
Application Server, and Messaging Server
Replacement for EV 165887.

18.0 7 Updated section 16 Change IP Address, August 6, 2012 François Isabelle

Host Name, FQDN Name, Gateway, or DNS
Server for EV 153412.

18.0 7 Edited changes and published document. August 24, 2012 Jessica Boyle

18.0 8 Updated section 9.1.1 Configure Basic September 5,2012 François Isabelle
Maintenance Tasks for EV 164026.

18.0 8 Reviewed changes. September 5, 2012 Patricia Renaud

18.0 8 Updated section [Link] Network Server, September 13, 2012 François Isabelle
Application Server, and Messaging Server
Replacement for EV 167728.

18.0 8 Reviewed changes. September 14, 2012 Goska Auerbach

19.0 1 Added section 10.6.3 Regain Disk Space September 18, 2012 Hang Tran
Used by MySQL InnoDB Tablespace Files.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

19.0 1 Updated section 9 Scheduled Maintenance September 28, 2012 Goska Auerbach
Tasks for EV 175534.

19.0 1 Update section 12 Disaster Recovery October 1, 2012 François Isabelle

Strategy for EV 175201.

19.0 1 Updated section 7.9 Restore BroadWorks October 15, 2012 Goska Auerbach
Server to In-service for EV 176540.

19.0 1 Added Service Control Function (SCF) October 21 2012 Mario Goupil
server.

19.0 1 Updated section 12 Disaster Recovery October 24, 2012 David Moriconi
Strategy for EV 168478.

19.0 1 Updated section 9.1.3 Restore Auto-Backup October 25, 2012 David Moriconi
for EV 148461.

19.0 1 Modified section 10.6.3 Regain Disk Space November 2, 2012 David Moriconi
Used by MySQL InnoDB Tablespace Files
for EV 164733.

19.0 1 Edited changes and published document. November 16, 2012 Patricia Renaud

19.0 2 Added [Link] tool definition. November 21, 2012 Frederic Lelievre

19.0 2 Adding new peerctl commands in section November 30, 2012 Frederic Lelievre
20.16 peerctl for EV 113764.

19.0 2 Edited changes and published document. March 1, 2013 Joan Renaud

19.0 3 Updated section 9.2 Automatic Disk June 17, 2013 Jerry Zhang
Cleanup for EV 189686.

19.0 3 Updated section 11.5.1 Application Server, August 25, 2013 Mario Goupil
Network Server for EV 197735.

19.0 3 Updated section 9 Scheduled Maintenance September 18, 2013 Jean-Francois

Tasks to add Service Control Function Dignard
(SCF) Server for EV 201523.

19.0 3 Edited changes and published document. September 20, 2013 Jessica Boyle

19.0 4 Added section 20.29 configdctl for September 24, 2013 Frederic Lelievre
EV 202216.

19.0 4 Edited changes and published document. October 23, 2013 Jessica Boyle

20.0 1 Updated section 8.3 Switch Active Software October 29, 2013 Jerry Zhang
Version for EV 198263. Updated section
[Link] Media Server Replacement for
EV 205360.

20.0 1 Updated sections [Link].1 Real-Time December 4, 2013 Goska Auerbach

Memory Usage and [Link].2 Transmit
RTP for EV 206499.

20.0 1 Edited changes and published document. December 13, 2013 Joan Renaud

20.0 2 Updated document for the WebRTC Server, December 14, 2013 Mario Goupil
Messaging Server, and Sharing Server.
Removed references to the Call Detail
Server.

20.0 2 Edited changes and published document. January 27, 2014 Patricia Renaud

20.0 3 Updated section 20.23 stopbw for January 29, 2014 Frederic Lelievre
EV 191707.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

20.0 3 Edited changes and published document. February 4, 2014 Jessica Boyle

20.0 4 Updated section 20.16 peerctl for March 7, 2014 Mario Goupil
EV 218395.

20.0 4 Updated all references to the Sharing March 26, 2014 Joan Renaud
Server (USS) and the Messaging Server
(UMS).
Edited changes and published document.

20.0 5 Updated section 16.4 Update Application May 3, 2014 Mario Goupil
Server Call Processing Routing Information
for EV 221793.

20.0 5 Edited changes. May 30, 2014 Jessica Boyle

20.0 5 Made minor editorial changes and published July 25, 2014 Joan Renaud
document.

21.0 1 Updated document for Release 21.0. Novermber 4, 2014 Mario Goupil
Added Network Function Manager and
Video Server.

21.0 1 Added section [Link] Execution Server November 18, 2014 Goska Auerbach
Replacement for EV 241916.
Updated the BroadSoft legal notice.

21.0 1 Edited changes. November 18, 2014 Joan Renaud

21.0 1 Rebranded and published document. November 19, 2014 Joan Renaud

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

1 Overview....................................................................................................................................... 18
2 Summary of Changes ................................................................................................................ 19
2.1 Changes for Release 21.0 ........................................................................................................ 19
2.2 Changes for Release 20.0 ........................................................................................................ 19
2.3 Changes for Release 19.0 ........................................................................................................ 19
2.4 Changes for Release 18.0 ........................................................................................................ 20
2.5 Changes for Release 17.0 ........................................................................................................ 21
2.6 Changes for Release 16.0 ........................................................................................................ 23
2.7 Changes for Release 15.0 ........................................................................................................ 23
2.8 Changes for Release 14.0 ........................................................................................................ 24
2.9 Changes from Release 13.0 ..................................................................................................... 25
2.10 Changes from Release 12.0 ..................................................................................................... 26
2.11 Changes from Release 11.1 ..................................................................................................... 26
2.12 Changes from Release 10.0 ..................................................................................................... 26
3 Maintenance Recommendations ............................................................................................. 27
4 BroadWorks Processes............................................................................................................. 28
5 Logs .............................................................................................................................................. 32
5.1 BroadWorks Debug Logs .......................................................................................................... 36
5.1.1 Log Severity ...................................................................................................................... 36
5.1.2 Log Input Channels .......................................................................................................... 36
5.1.3 Input Channel Attributes Channels .................................................................................. 51
5.1.4 Enable or Disable BroadWorks Debug Logging ............................................................. 51
5.1.5 Output Channel attributes ................................................................................................ 52
5.1.6 Log Rotation...................................................................................................................... 52
5.1.7 Sample Debug Configuration File.................................................................................... 53
5.2 BroadWorks UNIX Command Execution Logging .................................................................. 53
5.3 Miscellaneous Logs ................................................................................................................... 54
5.3.1 Linux System Logs ........................................................................................................... 54
5.3.2 Enable TimesTen Logs .................................................................................................... 54
5.3.3 Apache Logs ..................................................................................................................... 54
5.3.4 Tomcat Logs ..................................................................................................................... 54
5.3.5 Oracle Logs ....................................................................................................................... 54
6 Fault and Performance Measurements .................................................................................. 55
6.1 Alarms ........................................................................................................................................ 55
6.1.1 Open System Alarms ....................................................................................................... 55
6.1.2 View System Alarms ........................................................................................................ 56
6.1.3 View and Set CLI Alarm Backlog Size ............................................................................ 57
6.1.4 Enable and Disable Real-Time Alarms Echoing............................................................. 58
6.1.5 Clear Alarms ..................................................................................................................... 59

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 11 OF 247
6.1.6 Close System Alarms ....................................................................................................... 59
6.2 Threshold Alarms ...................................................................................................................... 59
6.2.1 Threshold Attributes ......................................................................................................... 59
6.2.2 Configure Thresholds using CLI ...................................................................................... 60
6.2.3 Configure Thresholds through SNMP ............................................................................. 62
6.2.4 Application Server Thresholds Limitation ........................................................................ 63
6.3 Performance Measurements .................................................................................................... 64
6.3.1 Navigate through MIB Nodes........................................................................................... 64
6.3.2 View Next MIB Node ........................................................................................................ 64
6.3.3 View Current Position in MIB ........................................................................................... 65
6.3.4 View MIB Node Values .................................................................................................... 65
7 Server Management ................................................................................................................... 67
7.1 Server State Management ........................................................................................................ 67
7.1.1 Server State ...................................................................................................................... 67
7.1.2 Application State ............................................................................................................... 67
7.1.3 Get Server and Application States................................................................................... 70
7.2 Application Management .......................................................................................................... 70
7.2.1 Installation ......................................................................................................................... 71
7.2.2 Activation ........................................................................................................................... 72
7.2.3 Deployment ....................................................................................................................... 73
7.2.4 Undeployment................................................................................................................... 73
7.2.5 Deactivation ...................................................................................................................... 75
7.2.6 Uninstallation..................................................................................................................... 75
7.3 Start Server ................................................................................................................................ 76
7.3.1 Start BroadWorks ............................................................................................................. 76
7.3.2 Start BroadWorks Application .......................................................................................... 76
7.4 Stop Server ................................................................................................................................ 77
7.4.1 Stop BroadWorks ............................................................................................................. 77
7.4.2 Stop Server without Locking ............................................................................................ 78
7.4.3 Stop BroadWorks Application .......................................................................................... 78
7.5 Check for Running Processes .................................................................................................. 78
7.6 Lock Network Server, Application Server, or Execution Server.............................................. 79
7.6.1 Force Lock Server Immediately ....................................................................................... 80
7.6.2 Unlock Server ................................................................................................................... 80
7.7 Restart Server ............................................................................................................................ 80
7.8 Take BroadWorks Server Out of Service for Maintenance ..................................................... 80
7.8.1 Non-redundant Systems .................................................................................................. 80
7.8.2 Redundant Systems ......................................................................................................... 81
7.9 Restore BroadWorks Server to In-service ............................................................................... 81
7.9.1 Non-redundant Systems .................................................................................................. 81
7.9.2 Redundant Systems ......................................................................................................... 82
7.10 Server State Monitoring Tools .................................................................................................. 83
7.10.1 Healthmon......................................................................................................................... 83

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 12 OF 247
7.10.2 Tech-support ..................................................................................................................... 85
7.11 Server Maintenance State ........................................................................................................ 85
7.11.1 View Current Maintenance State ..................................................................................... 86
7.11.2 Force Server into Maintenance State .............................................................................. 86
7.11.3 Recover Server in Maintenance State............................................................................. 86
8 Software Version Management ................................................................................................ 87
8.1 Display Active Software Version ............................................................................................... 87
8.1.1 Display Full Details about Active Software Versions ...................................................... 87
8.2 List Installed Software Versions ................................................................................................ 87
8.2.1 List Installed and Active Software Versions .................................................................... 87
8.3 Switch Active Software Version ................................................................................................ 88
9 Scheduled Maintenance Tasks ................................................................................................ 90
9.1 Automatic Backups.................................................................................................................... 92
9.1.1 Mode of Operation ............................................................................................................ 93
9.1.2 Auto-Backup File Naming Convention ............................................................................ 93
9.1.3 Restore Auto-Backup ....................................................................................................... 94
9.2 Automatic Disk Cleanup ............................................................................................................ 96
9.3 Healthmon Keep Alives ............................................................................................................. 96
9.4 Database Statistics Update....................................................................................................... 97
9.5 Registration Audit ...................................................................................................................... 97
9.6 Database Synchronization Check ............................................................................................ 97
9.7 CPU Monitoring Tool ................................................................................................................. 97
9.8 Check DB Pages ....................................................................................................................... 97
9.9 Tech-support .............................................................................................................................. 97
9.10 Service License Collect ............................................................................................................. 98
9.11 Delete Scheduled Tasks ........................................................................................................... 98
10 Database Maintenance .............................................................................................................. 99
10.1 Database Backup ...................................................................................................................... 99
10.1.1 Manual Database Backup................................................................................................ 99
10.1.2 Backup Database for Application Server, Element Management System, Call
Center Reporting Server, Network Server, Network Function Manager, or
Messaging Server........................................................................................................... 100
10.2 Database Restore.................................................................................................................... 100
10.2.1 Restore Database........................................................................................................... 100
10.2.2 Restore Database for Application Server, Element Management System, Call
Center Reporting Server, Network Server, Network Function Manager, or
Messaging Server........................................................................................................... 101
10.2.3 Extract Database Backup from Automatic Backup File................................................ 101
10.3 Database Import ...................................................................................................................... 102
10.3.1 Import Database on Application Server, Element Management System, Network
Server, Network Function Manager, or Messaging Server .......................................... 103
10.4 Database Installation ............................................................................................................... 104
10.4.1 Recreate Application Server, Network Server, or Messaging Server Database ........ 104

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 13 OF 247
10.5 Additional TimesTen Maintenance Tasks .............................................................................. 106
10.5.1 Change TimesTen Database Size ................................................................................ 106
10.5.2 Update TimesTen Database Statistics .......................................................................... 106
10.5.3 Start and Stop TimesTen Database Daemon............................................................... 107
10.5.4 Update TimesTen Database Paging Information ......................................................... 107
10.5.5 Compact the Database .................................................................................................. 107
10.5.6 Import TimesTen Datastores between Different Operating Systems .......................... 108
10.5.7 TimesTen Connection Optimization .............................................................................. 110
10.6 Additional MySQL Maintenance Tasks .................................................................................. 111
10.6.1 Update MySQL Database Statistics .............................................................................. 111
10.6.2 Start and Stop MySQL Database Daemon................................................................... 111
10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace Files ................................. 112
10.7 Additional MariaDB Maintenance Tasks ................................................................................ 112
10.7.1 Start and Stop MariaDB Database Daemon................................................................. 112
11 Redundancy Management ...................................................................................................... 113
11.1 Application Server, Network Server, and Messaging Server Database Replication ........... 113
11.1.1 TimesTen Database Synchronization ........................................................................... 113
11.1.2 TimesTen Replication Monitoring .................................................................................. 115
11.1.3 TimesTen Replication Port Number .............................................................................. 117
11.2 Element Management System Database Replication........................................................... 118
11.2.1 MySQL Database Synchronization ............................................................................... 118
11.2.2 MySQL Replication Monitoring ...................................................................................... 120
11.3 Install Cluster of Servers ......................................................................................................... 121
11.3.1 Element Management System Additional Parameters ................................................ 122
11.4 Network Server and Application Server Synchronization...................................................... 124
11.5 Add New Peer to Redundant System .................................................................................... 124
11.5.1 Application Server, Network Server, and Messaging Server ....................................... 124
11.5.2 Element Management System ...................................................................................... 128
11.5.3 Profile Server .................................................................................................................. 131
11.5.4 Network Function Manager ............................................................................................ 132
11.6 Remove Peer from Redundant System ................................................................................. 133
11.7 Recover Faulty Redundant Installation .................................................................................. 133
11.8 Post-installation Control........................................................................................................... 134
11.9 SSH/RSYNC Configuration .................................................................................................... 134
11.9.1 Create SSH Keys (Completed Automatically During Installation) ............................... 134
11.9.2 Configure SSH Between BroadWorks Servers ............................................................ 134
11.9.3 Use SSH ......................................................................................................................... 137
12 Disaster Recovery Strategy .................................................................................................... 138
12.1 Recover Server OS ................................................................................................................. 138
12.1.1 Hot Standby Servers ...................................................................................................... 139
12.1.2 On-Demand Rebuild....................................................................................................... 139
12.1.3 Incremental Backups ...................................................................................................... 140

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 14 OF 247
12.2 BroadWorks Server Restoration ............................................................................................. 141
12.2.1 BroadWorks System Backup ......................................................................................... 141
12.2.2 Licensing ......................................................................................................................... 142
12.2.3 BroadWorks Server Restoration Procedure ................................................................. 142
13 Optional Remember Password Cookie for Administrators .............................................. 154
13.1 Modify [Link] File ................................................................................................................ 154
13.1.1 Disable Cookie Support ................................................................................................. 154
13.1.2 Enable Cookie Support .................................................................................................. 154
14 Install BroadWorks License Files .......................................................................................... 155
15 Uninstall Obsolete BroadWorks Versions ........................................................................... 156
16 Change IP Address, Host Name, FQDN Name, Gateway, or DNS Server ...................... 157
16.1 Configure IP Networking Information at UNIX Level ............................................................. 161
16.2 Update BroadWorks Network Configuration .......................................................................... 162
16.2.1 Restart the configuration agent ...................................................................................... 162
16.2.2 HTTP Server Network Configuration ............................................................................. 162
16.3 Configure Redundancy Routing ............................................................................................. 163
16.4 Update Application Server Call Processing Routing Information ......................................... 163
16.5 Update Application Server IP Address on Network Server................................................... 164
16.6 Update Network Server IP Address on Application Server ................................................... 165
16.7 Update Network Server Routing Alias List ............................................................................. 165
16.8 Update Media Server IP Address on Network Server ........................................................... 166
16.9 Update Media Server IP Address on Application Server ...................................................... 167
16.10 Update Application Server and Network Server Address on OCS....................................... 167
16.11 Update Xtended Services Platform IP Address on Network Server..................................... 168
16.12 Update Network Server or Application Server Address on Xtended Services Platform ..... 169
16.13 Update EMS Redundancy Configuration ............................................................................... 169
16.14 Import EMS Database ............................................................................................................. 170
17 Change BCCT Listening Ports ............................................................................................... 171
18 BroadWorks System Monitoring ........................................................................................... 172
18.1 Platform Resource Monitoring – Key Resource Indicators ................................................... 172
18.1.1 File System Monitoring ................................................................................................... 172
18.1.2 CPU Usage Monitoring .................................................................................................. 173
18.1.3 Memory Usage/Paging Monitoring ................................................................................ 175
18.1.4 Disk Activity ..................................................................................................................... 175
18.1.5 Thread Activity ................................................................................................................ 176
18.1.6 Application Server Execution Server Heap Monitoring ................................................ 177
18.1.7 Database Size Monitoring .............................................................................................. 178
18.2 BroadWorks Performance Measurements Monitoring .......................................................... 178
18.2.1 Media Server................................................................................................................... 179
18.2.2 Network Server ............................................................................................................... 186
18.2.3 Application Server, Execution Server, and Profile Server ............................................ 190
18.3 BroadWorks SNMP Trap Monitoring ...................................................................................... 208

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 15 OF 247
18.3.1 Common BroadWorks Server Traps ............................................................................. 209
18.3.2 Server Process Problems .............................................................................................. 209
18.3.3 Software Error Traps ...................................................................................................... 210
18.3.4 Media Server Traps ........................................................................................................ 210
18.3.5 Network Server Traps .................................................................................................... 211
18.3.6 Application Server, Execution Server, and Profile Server Traps ................................. 211
18.3.7 Application Server Execution Server Process Problems ............................................. 214
18.3.8 Overload Control Monitoring .......................................................................................... 214
18.3.9 Device Operational State Monitoring ............................................................................. 215
18.4 Miscellaneous Monitoring........................................................................................................ 215
18.4.1 BroadWorks System Sanity ........................................................................................... 215
18.4.2 Redundancy Monitoring ................................................................................................. 217
18.4.3 Application Server Migrated User Monitoring ............................................................... 218
18.4.4 Database Synchronization Monitoring........................................................................... 219
19 BroadWorks Application Server Time Zone Support ........................................................ 220
20 BroadWorks Scripts and Utilities .......................................................................................... 221
20.1 [Link]............................................................................................................................. 221
20.2 bwCPUMon.............................................................................................................................. 221
20.3 bwListPortInUse....................................................................................................................... 221
20.4 [Link] ............................................................................................................................ 222
20.5 bwshowver ............................................................................................................................... 222
20.6 [Link] ................................................................................................................... 222
20.7 check_dbpages.pl.................................................................................................................... 223
20.8 config-hostname ...................................................................................................................... 224
20.9 config-redundancy ................................................................................................................... 224
20.10 config-ssh ................................................................................................................................. 224
20.11 [Link]................................................................................................................................ 225
20.12 [Link]............................................................................................................................. 225
20.13 healthmon ................................................................................................................................ 225
20.14 [Link] ............................................................................................................................... 226
20.14.1 [Link] on Element Management System......................................................... 226
20.15 [Link]....................................................................................................................... 227
20.16 peerctl ....................................................................................................................................... 228
20.17 peercmd ................................................................................................................................... 229
20.18 repctl ......................................................................................................................................... 229
20.19 resizeDSN ................................................................................................................................ 230
20.20 showrun.................................................................................................................................... 230
20.21 startbw ...................................................................................................................................... 231
20.22 frepctl ........................................................................................................................................ 231
20.23 stopbw ...................................................................................................................................... 232
20.24 synchcheck_basic.pl ............................................................................................................... 233
20.25 synchcheck_full.pl.................................................................................................................... 233

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 16 OF 247
20.26 ttStatus ..................................................................................................................................... 233
20.27 [Link] ............................................................................................................... 234
20.28 Software Manager Utilities ...................................................................................................... 235
20.28.1 [Link] ............................................................................................... 235
20.28.2 [Link]................................................................................................ 235
20.29 configdctl .................................................................................................................................. 235
20.30 snmpdctl ................................................................................................................................... 235
20.31 lmdctl ........................................................................................................................................ 236
21 Miscellaneous Maintenance Tasks ....................................................................................... 237
21.1 Add Memory to BroadWorks Server ...................................................................................... 237
21.2 Adjust BroadWorks Daylight Savings Time ........................................................................... 237
21.2.1 Description ...................................................................................................................... 237
21.2.2 Service Impact ................................................................................................................ 238
21.2.3 Solution ........................................................................................................................... 238
21.2.4 Instructions ...................................................................................................................... 238
21.3 Change Media Server Kernel Selection on Linux .................................................................. 239
Index ................................................................................................................................................... 241

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Routine maintenance of the system includes procedures for all BroadWorks servers. For
questions about these procedures, contact support@[Link].
For maintenance and care of Sun equipment, see the Sun reference manuals. For
partner devices, see the manufacturer’s documentation.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

This section describes the changes to this document for each release and document
version.

2.1 Changes for Release 21.0

This section describes the changes to this document for each document version.
Release 21.0, Version 1
This version of the document includes the following changes:
 Updated document with Release 21.0 changes.
 Added section [Link] Execution Server Replacement for EV 241916.

2.2 Changes for Release 20.0

This section describes the changes to this document for each document version.
Release 20.0, Version 5
This version of the document includes the following change:
 Updated section 16.4 Update Application Server Call Processing Routing Information
for EV 221793.
Release 20.0, Version 4
This version of the document includes the following changes:
 Updated section 21.16 peerctl for EV 218395.
 Updated all references to Sharing Server (USS) and Messaging Server (UMS).
Release 20.0, Version 3
This version of the document includes the following change:
 Updated section 21.23 stopbw for EV 191707.
Release 20.0, Version 2
This version of the document includes the following changes:
 Updated the document for the WebRTC Server (WRS), Messaging Server (UMS),
and Sharing Server (USS).
 Removed references to the Call Detail Server (CDS).

Release 20.0, Version 1

This version of the document includes the following changes:
 Updated section 8.3 Switch Active Software Version for EV 198263.
 Updated section [Link] Media Server Replacement for EV 205360.
 Updated sections [Link].1 Real-Time Memory Usage and [Link].2 Transmit RTP
for EV 206499.

2.3 Changes for Release 19.0

This section describes the changes to this document for each document version.
Release 19.0, Version 4
This version of the document includes the following change:

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 19 OF 247
 Added section 21.29 configdctl for EV 202216.
Release 19.0, Version 3
This version of the document includes the following changes:
 Updated section 9.2 Automatic Disk Cleanup for EV 189686.
 Updated section 11.5.1 Application Server, Network Server, and Messaging Server
for EV 197735.
Release 19.0, Version 2
This version of the document includes the following changes:
 Added [Link] tool definition.
 Updated section 21.16 peerctl: added new peerctl commands for EV 113764.
Release 19.0, Version 1
This version of the document includes the following changes:
 Added section 10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace Files
for EV 164733.
 Updated section 9 Scheduled Maintenance Tasks for EV 175534.
 Update section 12 Disaster Recovery Strategy for EVs 175201 and 168478.
 Updated section 7.9 Restore BroadWorks Server to In-service for EV 176540.
 Added Service Control Function (SCF) Server.
 Updated section 9.1.3 Restore Auto-Backup for EV 148461.
 Modified section 10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace
Files for EV 164733.

2.4 Changes for Release 18.0

This section describes the changes to this document for each document version.
Release 18.0, Version 8
This version of the document includes the following changes:
 Updated section 9.1.1 Configure Basic Maintenance Tasks for EV 164026. The
changes were subsequently moved to section 9.2 Automatic Disk Cleanup.
 Updated section [Link] Network Server, Application Server, and Messaging Server
Replacement for EV 167728.
Release 18.0, Version 7
This version of the document includes the following changes:
 Updated section 11.5 Add New Peer to Redundant System for EV 162349.
 Updated section 21.8 config-hostname for EV 169451.
 Updated section [Link] Network Server, Application Server, and Messaging Server
Replacement for EV 165887.
 Updated section 16 Change IP Address, Host Name, FQDN Name, Gateway, or DNS
Server for EV 153412.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 20 OF 247
Release 18.0, Version 6
This version of the document includes the following change:
 Updated section [Link] Network Server, Application Server, and Messaging Server
Replacement to add rsync command for IpDeviceConfig files for EV 164187.
Release 18.0, Version 5
This version of the document includes the following changes:
 Updated section 11.5 Add New Peer to Redundant System for EV 162349.
 Removed references to the Conferencing Server for EV 154846.
Release 18.0, Version 4
This version of the document includes the following change:
 Updated section 12.2.3 BroadWorks Server Restoration Procedure for EV 130651.
Release 18.0, Version 3
This version of the document includes the following changes:
 Updated section 10.5.6 Import TimesTen Datastores between Different Operating
Systems to include UID and PWD parameters.
 Added section [Link] Element Management System Replacement to cover the
disaster recovery process for the EMS for EV 130651.
 Updated section 16 Change IP Address, Host Name, FQDN Name, Gateway, or DNS
Server for EV 154921.
Release 18.0, Version 2
This version of the document includes the following change:
 Updated section [Link] Collecting Heap Size Indicator for EV 147625.
Release 18.0, Version 1
This version of the document includes the following changes:
 Updated document for Release 18.0.
 Updated section 16 Change IP Address, Host Name, FQDN Name, Gateway, or DNS
Server for EV 144590.

2.5 Changes for Release 17.0

This section describes the changes to this document for each document version.
Release 17.0, Version 9
This version of the document includes the following changes:
 Updated section 21.19 resizeDSN for EV 151585.
 Updated section 16 Change IP Address or Host Name for EV 148587.
 Updated section 10.5.1 Change TimesTen Database Size for EV 145227.
 Corrected CLI references from NetServ to NetworkServer for EV 150714.
 Added CDS/CCRS restoration procedure in section [Link] Call Detail Server/Call
Center Reporting Server for EV 125022.
 Updated section 12.2.3 BroadWorks Server Restoration Procedure for EV 144853.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 21 OF 247
Release 17.0, Version 8
This version of the document includes the following changes:
 Removed section 21.12 [Link] and updated section 9.4 Database
Statistics Update for EV 124825.
Release 17.0, Version 7
This version of the document includes the following changes:
 Updated section 7.10.1 Healthmon for EV 129579.
 Removed section about recreating the database by executing the _nms.sh script for
EV 119909.
 Added section 22.3 Change Media Server Kernel Selection on Linux for EV 121030.
 Updated section 11.5.2 Element Management System for EV 141355.
 Updated section [Link] Collecting Heap Size Indicator for EV 143495.
 Updated section 9.1.1 Configure Basic Maintenance Tasks for EV 143768.
Release 17.0, Version 6
This version of the document includes the following changes:
 Updated section 10.5.6 Import TimesTen Datastores between Different Operating
Systems for EV 114783.
 Updated section 21.12 [Link] to indicate that the task may cause
provisioning to fail while running for EV 124825.
 Updated section [Link] Network Server, Application Server, and Messaging Server
Replacement for EVs 113264 and 122851.
 Fixed the procedure related to resynchronizing a Media Server in section [Link]
Network Server, Application Server, and Messaging Server Replacement for
EV 122851.
 Updated section [Link] Network Server, Application Server, and Messaging Server
Replacement with information about the localization backup information included in
the auto-backup file for EV 113264.
 Updated section 11.5.1 Application Server, Network Server for EV 116345.
Release 17.0, Version 5
This version of the document includes the following changes:
 Updated section 10.3 Database Import for EV 120645.
 Updated section 22.2 Adjust BroadWorks DST (Daylight Savings Time) for
EV 123592.
Release 17.0, Version 4
This version of the document includes the following changes:
 Updated document with Release 17.sp1 information.
 Updated section 19.3 BroadWorks SNMP Trap Monitoring for EV 115577.
 Updated section 9 Scheduled Maintenance Tasks for EV 115892.
 Updated sections 11.1.2 TimesTen Replication Monitoring and 11.2.2 MySQL
Replication Monitoring for EV 112000.
 Added section 9.10 Service License Collect and updated section 9 Scheduled
Maintenance Tasks for EV 114437.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 22 OF 247
 Updated section 10.5.1 Change TimesTen Database Size for EV 118789.
Release 17.0, Version 3
This version of the document includes the following change:
 Updated section [Link] Healthmon File System Check for EV 112653.
Release 17.0, Version 2
This version of the document includes the following changes:
 Removed section on Optimization TimesTen Checkpoint for EV 110711.
 Updated section 12.2.3 BroadWorks Server Restoration Procedure for EV 109374.
Release 17.0, Version 1
This version of the document includes the following changes:
 Updated document for Release 17.0.
 Updated document for Release 17.0 and added section 20 BroadWorks Application
Server Time Zone Support for EV 101825.
 Updated section 10.5.1 Change TimesTen Database Size for EV 106025.

2.6 Changes for Release 16.0

Release 16.0, Version 1
This is a maintenance version of this document. This version includes the following
changes:
 In section 11.5.3 Profile Server, the manual step of importing data from peer was
removed as this is now automatically done by the script config-redundancy.
 Made minor updates.

2.7 Changes for Release 15.0

This section describes the changes to this document for each release and document
version.
Release 15.0, Version 6
This version of the document includes the following change:
 Made minor change to section 9.1.1 Configure Basic Maintenance Tasks for
EV 89729.
Release 15.0, Version 5
This version of the document includes the following change:
 Updated server recovery procedure to include information on file synchronizing.
Release 15.0, Version 4
This is a maintenance version of this document. This version of the document includes
the following change:
 Updated section 11.5.1 Application Server, Network Server for EV 48848.
Release 15.0, Version 3
This is a maintenance version of this document. This version of the document includes
the following changes:
 Updated section 11.5.2 Element Management System for EV 65145.
 Added section 21.14.1 [Link] on Element Management System for EV 81954.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 23 OF 247
 Updated section 12 Disaster Recovery Strategy for EV 87423.
 Added section 11.1.3 TimesTen Replication Port Number.
Release 15.0, Version 2
This is a maintenance version of this document. This version of the document includes
the following changes:
 Added section 17 Change BCCT Listening Ports for EV 62498.
Release 15.0, Version 1
This is a maintenance version of this document. This version of the document includes
the following changes:
 Made minor updates.
 Removed references to CMS.

2.8 Changes for Release 14.0

Release 14.0, Version 15 is a maintenance version of this document. The following
change was made:
 Added the Profile Server information.
Release 14.0, Version 14 is a maintenance version of this document. The following
change was made:
 Added a description of all the input channels that are used in the BroadWorks
[Link] files.
Release 14.0, Version 13 is a maintenance version of this document. The following
changes were made:
 Added section 10.5.7 TimesTen Connection Optimization for EV 60233.
 Added section [Link] Xtended Services Platform Replacement.
Release 14.0, Version 12 is a maintenance version of this document. The following
changes were made:
 Added section 9.1.1 Configure Basic Maintenance Tasks for EV 59029.
 Updated section 5.1.7 Media Server for EV 59375. Note that this section has
subsequently been deleted from this document.
 Modified section 9 Scheduled Maintenance Tasks to add CDS and EMS daily
DbMaint task; also added section 21.12 [Link] for EV 53221.
 Created section on Optimization TimesTen Checkpoint for EV 59930.
Release 14.0, Version 11 is a maintenance version of this document. The following
change was made:
 Updated BEA Java GC logs information.
Release 14.0, Version 10 is a maintenance version of this document. The following
change was made:
 A procedure describing the process for importing TimesTen datastores between
different operating systems was added.
Release 14.0, Version 9 is a maintenance version of this document. The following
changes were made:
 Clarified the list of processes on the CMS.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 24 OF 247
 Added logging information for the CMS.
Release 14.0, Version 7 is a maintenance version of this document. The following
changes were made:
 Updated section 20.9 config-network to add a note for HTTP Alias CLI for EV 38723.
 Updated section 9.1.3 Restore Auto-Backup to indicate that the backup restore must
be executed as root when done in the system root directory for EV 50841.
Release 14.0, Version 6 is a maintenance version of this document. The following change
was made:
 Corrected log file named for 24-hour use for EV 49826.
Release 14.0, Version 5 is a maintenance version of this document. The following
changes were made to provide information concerning the use of BEA WebLogic with
BroadWorks.
 Provided new information on new and modified log files.
 Added licensing information for BEA.
 Added information on network configuration.
Release 14.0, Version 4 is a maintenance version of this document. The following
changes were made to the document:
 Config-network: This tool was migrated to a Web Server configuration-only tool.
The documentation was updated accordingly for the command, as well as for the
procedure used to change the IP address for a server.
Previous changes to the document include the following:
 Added section 20.28 Software Manager Utilities for EV 37043.
 Changed note in section 8.1.1 Display Full Details about Active Software Version and
added text about -showPatch option in section 21.5 bwshowver for EV 37037.
 Added warning to section 21.23 stopbw for EV 39877.
 Added disaster recovery strategies, and minor textual updates to correct EV 34678.
 Added CommonPersistency input channel parameters.

2.9 Changes from Release 13.0

Release 13.0, Version 5 includes a new section listing all the key processes running on
BroadWorks.
Release 13.0, Version 3 is a maintenance version of this document. The following
changes were made to the document:
 Change IP Address: Updated procedure to change an IP address for a BroadWorks
server.
 ASDump: Clarified that the administrator must enter a HostingNe when running the
command. Specified a temporary location to store the output of the ASDump.
 Auto Backup: Updated procedure to show how to extract files from a backup image.
 Logging: Updated the BroadWorks debugging section. Added basic rules for
increasing the total number of log files for the Application Server Provisioning Server
(PS) and Execution Server (XS).
 Maintenance Tasks: Introduced procedure to add more memory for a BroadWorks
server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 25 OF 247
 TimesTen: Added a procedure to perform a consolidation of the store in use for the
database by compacting it.

2.10 Changes from Release 12.0

The following sections were added or modified since Release 12.0:
 Logging: Added a section about the BroadWorks [Link] file. Updated the
BroadWorks logging section.
 Linux: Generalized Solaris information and added Linux-specific sections when
required.
 Maintenance Tools: Added a description for: check_dbpages.pl
 Servers Address Management: Added a procedure to change the address of the
CDS, EMS, and Web Server.
 EMS Redundancy: Existing methods and procedures were updated and new ones
were added to cover for the EMS redundancy.
 Auto Backups: Added the P option to the tar xvf command so original full path is
restored on Linux.
 System Monitoring: Added sections describing which performance measurements to
monitor.
 BroadWorks Processes: The list of all BroadWorks processes per server type.

2.11 Changes from Release 11.1

The following sections were added or modified since Release 11.1:
 Maintenance Tasks: Added definitions for three new maintenance tasks for the tech-
support tool.
 [Link]: New command was added.
 Database: Added section “Updating Database Paging Information” to the document.

2.12 Changes from Release 10.0

The following sections were added or modified since Release 10.0:
 Maintenance Recommendations: This section was added for Release 11.0.
 New tools as follows:
− Added description for syncheck_basic.pl, resizeDSN, and bwCPUMon.
− Renamed [Link] to syncheck_full.pl.
 Maintenance Tasks: Added definitions for three new maintenance tasks:
bwCPUMon, regAudit, and syncheck.
 Server Management: A description for the server maintenance state was added to
the document.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

An operator should always use the following rules to maintain a BroadWorks server:
 Always use maintenance scripts and procedures provided by BroadSoft.
 Always refer to BroadWorks support personnel before installing any third-party
software that has not been distributed with the BroadWorks application.
 When scripts are used to maintain a server, you should always keep the /tmp
directory as empty as possible because the /tmp is shared with the /swap partition on
the server.
− Filling up the /tmp partition could cause a server’s performance to significantly
decline and may even cause it to stop functioning.
− It is recommended to use a directory other than /tmp to put temporary files and to
erase them at the end of a maintenance task.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

All BroadWorks servers are composed of many processes. These processes can be
associated with the following categories:
 Application – The application processes are those that provide the BroadWorks
functionality, for example, the ExecutionAndProvisioning process on the Application
Server.
 Platform – The platform processes are those that provide external integration or
addition.
The following table provides a list of applications and their corresponding processes
installed with BroadWorks and it indicates which processes go with the different types
of servers. Note that you can find the expansion of abbreviations for server names in the
Abbreviations for Server Names table, which follows.

Application Process Servers

ExecutionAnd execution AS
Provisioning
provisioning

remotexla

WebContainer apache AS, NS, EMS, Xsp, PS,

UMS, USS, NFM
tomcat

Open Client Server OCS AS, EMS, Xsp

FlashPolicy flashPolicy AS, Xsp

AccessMediationServer AMSTurnServer AMS

AMSSignalingGateway AMS

keepalived AMS

LogServer LogServer MS, AMS, SCF, UMS,

USS, WRS, UVS

EMSBackEnd emsBackEnd EMS

MediaStreaming msfe MS

NSExecutionAndProvisioning nsExecution NS

nsProvisioning

remotexla

DeviceManagementTFTP DMTftp Xsp

RatingFunction RatingFunction Xsp

UC-Connect UC-Connect Xsp

Provisioning provisioning PS

subscriberAgent

Execution Dataless execution XS

subscriberAgent

DbManagement DbManagement DBS

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

DBSObserver dbsObserver PS

CCReportingDBManagement ccReportingDB PS
Management

EnhancedCallLogsDB enhancedCallLogsDB PS
Management Management

SCF Processor scfProcessor SCF

SS7ProtocolStack emSS7CAP SCF

emSS7MAP

emSS7Platform

ss7CAP

ss7CAPBridge

ss7MAP

ss7MAPBridge

ss7OAMBridge

ss7Platform

IMP dbConnector UMS

imp

muc

uss uss USS

WebRTCGateway gwcore WRS

ipcfe

stun

NodeController nodeController NFM

LicenseController licenseController

VideoStreaming uvsfe UVS

Configuration Agent BroadWorks All

(core platform application) Configuration Agent

License Manager BroadWorks License Manager All

(core platform application)

SNMP Agent BroadWorks All

(core platform application) SNMP Agent

Software Manager BroadWorks All

(core platform application) Software Manager

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 29 OF 247
Abbreviations for Server Names
The abbreviations for server names used in this guide are listed in the following table.
Abbreviation Server Name

AMS Access Mediation Server

AS Application Server

DBS Database Server

EMS Element Management System

MS Media Server

NFM Network Function Manager

NS Network Server

PS Profile Server

SCF Service Control Function Server

UMS Messaging Server

USS Sharing Server

UVS Video Server

WRS WebRTC Server

XS Execution Server

Xsp Xtended Services Platform

All application processes are monitored through the BroadWorks process monitor. If the
monitored process dies, the process monitor automatically restarts it. The healthmon
monitoring task validates that all deployed applications have all their processes running
and notifies the operator through an alarm if one of the processes listed above is not
present for a specific server. To view the list of BroadWorks application processes
running on a server, an operator can issue the showrun command from a UNIX prompt.
Example:
bwadmin@[Link]$ showrun
Currently running BroadWorks Application processes:
OCS process monitor (pid=12577)
OCS (pid=12656)
apache process monitor (pid=12499)
apache (pid=12644)
execution process monitor (pid=12426)
execution (pid=12957)
flashPolicy process monitor (pid=12547)
flashPolicy (pid=12585)
provisioning process monitor (pid=12453)
provisioning (pid=12992)
remotexla process monitor (pid=12404)
remotexla (pid=12430)
tomcat process monitor (pid=12476)
tomcat (pid=12513)

Currently running BroadWorks Platform processes:

BroadWorks Configuration Agent process monitor (pid=8656)

BroadWorks Configuration Agent (pid=8686)
BroadWorks Software Manager (pid=11894)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BroadWorks supports full logging capability. Certain logs are system-controlled, whereas
others are on all the time, and other logs such as debug logging, can be modified to
provide more information when required. BroadWorks application-related log files that
reside under /var/broadworks/logs include the following:

Process/ Server Log File Location

Web Application Type

execution AS, XS Process Monitor Logs: appserver/[Link]

Debug Logs: appserver/[Link]
Stdout Logs: appserver/[Link]

provisioning AS, PS AS:

Process Monitor Logs: appserver/[Link]
Debug Logs: appserver/[Link]
Stdout Logs: appserver/[Link]
Audit Debug Logs: appserver/[Link]
PS:
Process Monitor Logs: provisioning/[Link]
Debug Logs: provisioning/[Link]
Stdout Logs: provisioning/[Link]
Audit Debug Logs: provisioning/[Link]

Remotexla AS, NS Process Monitor Logs: remotexla/[Link]

Stdout Logs: remotexla/[Link]

CommPilot (webApp) AS Debug Logs: appserver/[Link]

Xsp For the Xsp, the logs are under /var/broadworks/logs/xsp.

DeviceManagement AS Debug Logs: appserver/[Link]-

Files (webApp) [Link]

JWSFiles (webApp) AS Debug Logs: appserver/[Link]

MediaFiles (webApp) AS Debug Logs: appserver/[Link]

OCIFiles (webApp) AS Debug Logs: appserver/[Link]

Xsp For the Xsp, the logs are under /var/broadworks/logs/xsp.

OCIOverSoap AS Debug Logs: appserver/[Link]

(webApp) Xsp For the Xsp, the logs are under /var/broadworks/logs/xsp.

apache AS, NS, Process Monitor Logs: apache/[Link]

EMS, Xsp, Stdout Logs: apache/[Link]
PS, UMS

tomcat AS, NS, Process Monitor Logs: tomcat/[Link]

EMS, Xsp, Stdout Logs: tomcat/[Link]
PS, UMS

OCS AS, Xsp, Process Monitor Logs: openclientserver/[Link]

EMS Debug Logs: openclientserver/[Link]
Stdout Logs: appserver/[Link]

flashPolicy AS Process Monitor Logs: flashPolicy/[Link]

Xsp Stdout Logs: flashPolicy/[Link]

AMSTurnServer AMS Process Monitor Logs: ams/[Link]

Debug Logs: ams/[Link]
Stdout Logs: ams/[Link]

AMSSignalingGate AMS Process Monitor Logs: ams/[Link]

way Debug Logs: ams/[Link]
Stdout Logs: ams/[Link]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

keepalived AMS Process Monitor Logs: keepalived/[Link]

Stdout Logs: keepalived/[Link]

LogServer AMS, MS, Process Monitor Logs: logserver/[Link]

UMS, Debug Logs: logserver/[Link]
USS, WRS Stdout Logs: logserver/[Link]

emsBackEnd EMS Process Monitor Logs: emsBackEnd/[Link]

Debug Logs: emsBackEnd/[Link]
Stdout Logs: emsBackEnd/[Link]
The /var/broadworks/logs/emsBackEnd directory also contains a set of
detailed debug logs the functionality provided by the EMS.

emsFrontEnd EMS Debug Logs: emsFrontEnd/[Link]

(webApp) The /var/broadworks/logs/emsFrontEnd directory also contains a set of
detailed debug logs the functionality provided by the EMS.

LogRepository EMS, PS Debug Logs: logrepos/[Link]

(webApp)

Msfe MS Process Monitor Logs: mediaserver0/[Link]

Debug Logs: mediaserver0/[Link]
Stdout Logs: mediaserver0/[Link]

FileTransfer (webApp) NFM Debug Logs: fileTransfer/[Link]

LicenseController NFM Process Monitor Logs:

licenseController/[Link]
Debug Logs: licenseController/[Link]-
[Link]
Stdout Logs: licenseController/[Link]

NFInterface (webApp) NFM Debug Logs: nfInterface/[Link]

NFMPortal (webApp) NFM Debug Logs: nfmPortal/[Link]

NodeController NFM Process Monitor Logs: nodeController/[Link]

Debug Logs: nodeController/[Link]
Stdout Logs: nodeController/[Link]

NodeInterface NFM Debug Logs: nodeInterface/[Link]

(webApp)

nsExecution NS Process Monitor Logs: routingserver/[Link]

Debug Logs: routingserver/[Link]
Stdout Logs: routingserver/[Link]

nsProvisioning NS Process Monitor Logs: routingserver/[Link]

Debug Logs: routingserver/[Link]
Stdout Logs: routingserver/[Link]

NSPortal (webApp) NS Debug Logs: nsportal/[Link]

DMTftp Xsp Process Monitor Logs: tftp/[Link]

Debug Logs: tftp/[Link]
Stdout Logs: tftp/[Link]

RatingFunction Xsp Process Monitor Logs: RatingFunction/ [Link]

Debug Logs: RatingFunction/[Link]
Stdout Logs: RatingFunction/[Link]

UC-Connect Xsp Process Monitor Logs: UC-Connect/[Link]

Debug Logs: UC-Connect/[Link]
Stdout Logs: UC-Connect/[Link]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

subscriberAgent XS, PS Process Monitor Logs: subscriber/subscriberAgentcontainerpmd [Link]

Debug Logs: subscriber/[Link]
Stdout Logs: subscriber/subscriberOutput [Link]

DbManagement DBS Process Monitor Logs: dbm/DbManagementcontainerpmd [Link]

Debug Logs: dbm/[Link]
Stdout Logs: dbm/DBMLogOutput [Link]

dbsObserver PS Process Monitor Logs: dbsObserver/dbsObservercontainerpmd [Link]

Debug Logs: dbsObserver/[Link]
Stdout Logs: dbsObserver/dbsObserverOutput [Link]

CCReporting PS Debug Logs: profileserver/[Link]

Repository (webApp)

CCReporting PS Debug Logs: profileserver/[Link]

(webApp)

ccReportingDB PS Process Monitor Logs:

Management profileserver/[Link]
Debug Logs: profileserver/[Link]-
[Link]
Stdout Logs: profileserver/[Link]

enhancedCallLogsDB PS Process Monitor Logs:

Management profileserver/[Link]
Debug Logs: profileserver/[Link]
Stdout Logs: profileserver/[Link]

scfProcessor SCF Process Monitor Logs: scf/[Link]

Debug Logs: scf/SCFProcessLog [Link]
Stdout Logs: scf/[Link]

emSS7CAP SCF Process Monitor Logs:

emSS7CAP/[Link]
Stdout Logs: emSS7CAP/emSS7CAP [Link]

emSS7MAP SCF Process Monitor Logs:

emSS7MAP/[Link]
Debug Logs:
emSS7MAP/SPL_trace_app_map_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: emSS7MAP/emSS7MAP [Link]

emSS7Platform SCF Process Monitor Logs:

emSS7Platform/[Link]
Debug Logs:
emSS7Platform/SPL_trace_ss7p_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Debug Logs: emSS7Platform/ss7p_events_xxxxx_MMM_dd_hh.[Link]
Debug Logs: emSS7Platform/LOGS_EM-xxxxx-MMM_dd_hh.[Link]
Stdout Logs: emSS7Platform/[Link]

ss7Platform SCF Process Monitor Logs: ss7Platform/[Link]

Debug Logs:
ss7Platform/SPL_trace_ss7p_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Debug Logs: ss7Platform/LOGS_FEP_Inst_x-xxxxx-MMM_dd_hh.[Link]
Stdout Logs: ss7Platform/[Link]

ss7CAP SCF Process Monitor Logs: ss7CAP/[Link]

Debug Logs:
ss7CAP/SPL_trace_ss7p_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: ss7CAP/[Link]

ss7MAP SCF Process Monitor Logs: ss7MAP/[Link]

Debug Logs:
ss7MAP/SPL_trace_app_map_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: ss7MAP/[Link]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

ss7CAPBridge SCF Process Monitor Logs: ss7Bridge/ [Link]

Debug Logs: ss7Bridge/ [Link]
Stdout Logs: ss7Bridge/[Link]

ss7MAPBridge SCF Process Monitor Logs: ss7Bridge/ [Link]

Debug Logs: ss7Bridge/ [Link]
Debug Logs:
ss7Bridge/SPL_trace_app_map_yyyy_mm_dd_hh_mm_ss_xxxx_0.log
Stdout Logs: ss7Bridge/[Link]

ss7OAMBridge SCF Process Monitor Logs: ss7Bridge/[Link]

Debug Logs: ss7Bridge/[Link]
Stdout Logs: ss7Bridge/[Link]

dbConnector UMS Process Monitor Logs: ums/[Link]

Debug Logs: ums/[Link]
Stdout Logs: ums/[Link]

imp UMS Process Monitor Logs: ums/[Link]

Debug Logs: ums/[Link]
Stdout Logs: ums/[Link]

muc UMS Process Monitor Logs: ums/[Link]

Debug Logs: ums/[Link]
Stdout Logs: ums/[Link]

provisioningAdapter UMS Debug Logs: ums/[Link]

uss USS Process Monitor Logs: uss/[Link]

Debug Logs: uss/[Link]
Stdout Logs: uss/[Link]

uvsfe UVS Process Monitor Logs: videoserver/[Link]

Debug Logs: videoserver/[Link]
Stdout Logs: videoserver/[Link]

gwcore WRS Process Monitor Logs: webrtcserver/[Link]

Debug Logs: webrtcserver/[Link]
Stdout Logs: webrtcserver/[Link]

ipcfe WRS Process Monitor Logs: webrtcserver/[Link]

Debug Logs: webrtcserver/[Link]
Stdout Logs: webrtcserver/[Link]

BroadWorks All Process Monitor Logs: config/[Link]

Configuration Agent Debug Logs: config/[Link]
Stdout Logs: config/[Link]

BroadWorks License All Process Monitor Logs: lmd/[Link]

Manager Debug Logs: lmd/[Link]
Stdout Logs: lmd/[Link]

BroadWorks SNMP All Process Monitor Logs: snmp/[Link]

Agent Debug Logs: snmp/[Link]
Stdout Logs: snmp/[Link]

BroadWorks Software All Process Monitor Logs: swmanager/[Link]

Manager Debug Logs: swmanager/swmanagerActivities [Link]
The /var/broadworks/logs/swmanager directory also contains patching
activity logs.

The following table lists logs related to clustered server configuration and maintenance
activities. Only applicable servers are listed.
Event Type Log File Location

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

File replication-related rsync/ rsync/ rsync/ rsync/ rsync/ rsync/

events frepXX log [Link] [Link] [Link] [Link] [Link]

For all servers, the server state management-related logs are located in the
maintenance/[Link] file.
BroadWorks database activity log files in the /var/broadworks/ directory include those
listed in the following table (note that only applicable servers are listed).
Log File Location
Event Type
AS NS UMS

Database Backup [Link] [Link] [Link]

Database Destroy [Link] [Link] [Link]

Database Import [Link] [Link] [Link]

Database Restore [Link] [Link] [Link]

NOTE: Processes that may destroy the TimesTen database include restoring a database,
importing from a peer server, and upgrading.

5.1 BroadWorks Debug Logs

Debug logs leave a trace of BroadWorks events in chronological order. Debug logs
provide a trace of all internal and external events related to all calls and sessions. By
default, logging is set at the highest severity level (Debug), with only the Session Initiation
Protocol (SIP) and Media Gateway Control Protocol (MGCP) input channels active.

WARNING: Increasing the logging level affects server performance. Therefore, turn on input
channels with caution. By default, only the protocol input channels are on.

5.1.1 Log Severity

The severity of the logs is set on the input channels. Set either all or individual input
channels. The following example shows the severity level settings for the CallP input
channel:
 Warning – General warning messages
 Info – General informational messages
 Notice – Anomalous but gracefully handled behavior
 FieldDebug – Highest level of debugging to be used on a production system
 Debug – Highest level of debugging – all messages
The severity should be left at the default FieldDebug level.

5.1.2 Log Input Channels

BroadWorks sends logs to a number of input channels, each related to a particular sub-
system. For example, the CommonPersistency channel relates to the TimesTen
database. Enable a channel to send logs to a file. Disable a channel to ignore logs.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 36 OF 247
When troubleshooting, enable channels to better understand what may be the cause of a
failure. On all BroadWorks servers, the logging is controlled through the CLI. Each
application has a logging context under its application context. For instance, the
Execution (XS) logging is controlled under the AS_CLI/Applications/ExecutionAnd
Provisioning/XS/Logging> CLI level.
Within the CLI, each application context is listed under the application context. From each
application context, a logging context is available (where applicable) to configure the
logging for the application.
To enable logs for a specific input channel, change the Enabled property from “false” to
“true”.
However, one component still makes use of an eXtensible Markup Language (XML) log
configuration file, which is the following:
 Software Manager: The file /usr/local/broadworks/swmanager/conf/[Link]
controls the log settings.

WARNING: Be very careful when editing the logging configuration on the Application Server or
Network Server.

[Link] Accounting Input Channel

Channel Name(s) Accounting
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel that can be used to dump all accounting events in
to the log file.
Special None
Configuration

[Link] Apache Commons Logging Input Channels

Channel Name(s) HttpClient, DefaultCommonsLogging, Soap
Applies To ExecutionAndProvisioning (PS) application, Provisioning
application
Description Input channel that can be used to dump the logs generated
when using the Apache HttpClient. The Apache HttpClient is
used with the external file system integration in the Application
Server Provisioning Server process.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 37 OF 247
[Link] Audit Log Input Channel
Channel Name(s) Audit
Applies To ExecutionAndProvisioning (PS and XS) application,
Provisioning application
Description Input channel that can be used to:
 For the Application Server, dump the OCI events to the
AuditLog file.
 For the Network Server, dump the OCI and Network
Server legacy audit logs to the AuditLog file.
Special The Audit input channel has the following additional controls:
Configuration
 GetRequests – Indicates whether OCI get requests are
logged.
 AccountInfo – Indicates whether the user ID and
authorization level of the account executing the request is
included.
 Verbose – Indicates whether the xml string of the OCI
request is written.

[Link] Authentication Input Channel

Channel Name(s) Authentication
Applies To USS application
Description Input channel used to report logs related to authentication
activities for the USS application on the Sharing Server.
Special None
Configuration

[Link] BroadWorks Common Communication Transport Input Channels

Channel Name(s) BroadsoftCommonCommunicationTransport,
BroadsoftCommonCommunicationTransportKeepAlive
Applies To ExecutionAndProvisioning (PS and XS) application,
NSExecutionAndProvisioning (NSPS and NSXS) application,
OpenClientServer application, [Link],
NSPortal application, [Link], ExecutionDataless
application, Provisioning application
Description The BCCT transport mechanism is used to transport various
protocols for BroadWorks inter-process communications. The
BroadsoftCommonCommunicationTransport channel can be
enabled to view the messages exchanged between servers.
The BroadsoftCommonCommunicationTransportKeepAlive can
be enabled to view the BCCT Keep Alive messages exchanged
between servers.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 38 OF 247
[Link] Call Center Reporting Input Channel
Channel Name(s) CCRChannel, DatabaseChannel, CommunicationChannel
Applies To CallCenterReporting application
Description Input channel used to log internal Call Center Reporting (CCR)
processing debugging information. The DatabaseChannel can
be used to view all access to the database, while the
CommunicationChannel captures all inter-process
communication events for the CCR.
Special None
Configuration

[Link] CallP Input Channel

Channel Name(s) CallP
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used on the Application Server to output internal
call processing events.
Special None
Configuration

[Link] CallP Application Protocol Input Channel

Channel Name(s) CAP
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used to output all incoming and outgoing CAP
messages.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 39 OF 247
[Link] CommonPersistency Input Channel
Channel Name(s) CommonPersistency
Applies To ExecutionAndProvisioning (PS and XS) application,
NSExecutionAndProvisioning (NSPS and NSXS) application,
Provisioning application, ExecutionDataless application
Description BroadWorks database abstraction layer-specific logs. The logs
exposed by this channel represent internal information captured
by the abstraction layer.
Special The CommonPersistency input channel has additional
Configuration parameters, which are used to monitor the database
subsystem. These parameters are omitted by default and
should only be added when requested or authorized by
BroadSoft personnel.
The parameters are as follows:
 UpdateTimeThreshold – Specifies a time in milliseconds.
If a database update takes longer than the specified
threshold, a log and SNMP trap are output.
 QueryTimeThreshold – Specifies a time in milliseconds. If
a database query takes longer than the specified
threshold, a log and SNMP trap are output.
 UpdateRowsThreshold – Specifies a number of rows. If a
database update modifies more than this number of rows,
a log, and SNMP trap are output.
 QueryRowsThreshold – Specifies a number of rows. If a
database query returns, more than this number of rows,
then a log and an SNMP trap are output.

WARNING: Setting these parameters with incorrect values can lead to severe performance
degradations. Do NOT modify these settings unless requested by BroadSoft personnel.

[Link] BWCommunicationUtility Input Channel

Channel Name(s) BWCommunicationUtility
Applies To FileRepos application
Description Input channel used to report logs related to the
BWCommunicationUtility running in Tomcat on the Profile
Server.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 40 OF 247
[Link] DbConnector Input Channel
Channel Name(s) DbConnector
Applies To IMP application
Description Input channel used to report logs related to the dbConnector
process running in the IMP application on the Messaging
Server.
Special None
Configuration

[Link] Event Notification Input Channel

Channel Name(s) EventNotification
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used to output logs related to the Application
Server event notification framework. The event notification
framework is used for subscription-based services, for example,
Busy Lamp Field and Automatic Callback.
Special None
Configuration

[Link] File Repository Input Channel

Channel Name(s) FileRepos
Applies To FileRepos-based web applications.
Description Input channel used to report logs related to the file repository
web applications running in Tomcat on BroadWorks servers.
Special None
Configuration

[Link] File System Input Channel

Channel Name(s) FileSystem
Applies To Configuration done through BroadWorks Configuration Agent
Description Input channel used to report logs related to the external file
system integration.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 41 OF 247
[Link] Generic Input Channel
Channel Name(s) Generic
Applies To All applications, [Link], [Link]
Description Input channel used to report miscellaneous logs from the
process.
Special None
Configuration

[Link] IMP Input Channel

Channel Name(s) IMP
Applies To IMP application
Description Input channel used to report logs related to the imp process
running in the IMP application on the Messaging Server.
Special None
Configuration

[Link] License Controller Input Channel

Channel Name(s) LicenseController
Applies To LicenseController application
Description Input channel used to output all incoming and
outgoing License Controller (Network Function
Manager application) messages.
Special None
Configuration

[Link] LocationAPI Input Channel

Channel Name(s) LocationAPI
Applies To NSPortal application
Description Input channel used to output all incoming and
outgoing LocationAPI (Network Server Web Server
application) messages.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 42 OF 247
[Link] Media Gateway Controller Protocol Input Channel
Channel Name(s) MGCP
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used to output all incoming and outgoing MGCP
messages.
Special None
Configuration

[Link] IMP Input Channel

Channel Name(s) MUC
Applies To MUC application
Description Input channel used to report logs related to the muc process
running in the IMP application on the Messaging Server.
Special None
Configuration

[Link] Name Service Input Channel

Channel Name(s) NameService
Applies To ExecutionAndProvisioning (XS) application,
NSExecutionAndProvisioning (NSXS) application,
ExecutionDataless application
Description Input channel used to output all naming service lookup
messages along with additional internal debugging information.
Special None
Configuration

[Link] Network Resource Service Input Channel

Channel Name(s) NRSLog
Applies To ExecutionAndProvisioning (XS) application,
NSExecutionAndProvisioning (NSXS) application,
ExecutionDataless application
Description Input channel used to report Network Resource Service (NRS)-
related logs. NRS is the underlying framework that is used by
Application Server Redundancy (ASR), the BroadWorks
Redundancy Protocol used to exchange migration information
from the Application Server to the Network Server and from the
Application Server to the Application Server. Enabling the
channel allows the ability to view incoming and outgoing ASR
messages.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 43 OF 247
[Link] Network Function Interface Input Channel
Channel Name(s) NFInterface
Applies To NFInterface application
Description Input channel used to log Network Function Interface web
application debugging information.
Special None
Configuration

[Link] Network Function Manager Portal Input Channel

Channel Name(s) NFMPortal
Applies To NFMPortal application
Description Input channel used to log Network Function Manager Portal
web application debugging information.
Special None
Configuration

[Link] Node Controller Input Channel

Channel Name(s) NodeController
Applies To NodeController application
Description Input channel used to log Node Controller debugging
information.
Special None
Configuration

[Link] Node Interface Input Channel

Channel Name(s) NodeInterface
Applies To NodeInterface application
Description Input channel used to log Node Interface debugging
information.
Special None
Configuration

[Link] Open Client Server Input Channel

Channel Name(s) OpenClientServer
Applies To OpenClientServer application
Description Input channel used to log internal Open Client Server process
debugging information.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 44 OF 247
[Link] Overload Control Validation Input Channel
Channel Name(s) Overload
Applies To ExecutionAndProvisioning (XS) application,
NSExecutionAndProvisioning (NSXS) application,
ExecutionDataless application
Description Input channel used to report SIP and MGCP-related overload
control events.
Special None
Configuration

[Link] Provisioning Adapter Input Channel

Channel Name(s) PAdapter
Applies To PAdapter web application
Description Input channel used to report logs related to the Provisioning
Adapter web application running in tomcat on the Messaging
Server.
Special None
Configuration

[Link] Patch Tool Input Channels

Channel Name(s) PatchTool, PatchToolBcctTraffic, PatchToolPersistence
Applies To [Link]
Description Input channel used to log Software Manager/patching
debugging information.
Special None
Configuration

[Link] PortalAPI Input Channel

Channel Name(s) PortalAPI
Applies To NSPortal application
Description Input channel that can be used to dump all incoming and
outgoing PortalAPI messages.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 45 OF 247
[Link] Provisioning Validation Input Channel
Channel Name(s) ProvisioningValidation
Applies To ExecutionAndProvisioning (XS and PS) application,
Provisioning application
Description Input channel used by the Application Server Provisioning
Server (PS) to the Execution Server (XS) Provisioning
Validation Protocol. It can be used to dump all provisioning
validation-related messages to the log files.
Special None
Configuration

[Link] Room Control Input Channel

Channel Name(s) RoomControl
Applies To USS application
Description Input channel used to report logs related to Room Control
activities for the USS application on the Sharing Server.
Special None
Configuration

[Link] Room Manager Input Channel

Channel Name(s) RoomManager
Applies To USS application
Description Input channel used to report logs related to Room Manager
activities for the USS application on the Sharing Server.
Special None
Configuration

[Link] Service Control Proxy Input Channel

Channel Name(s) ServiceControlProxy
Applies To NSPortal application
Description Input channel used to output all incoming and outgoing Service
Control Proxy (Network Server Web Server application)
messages.
Special None
Configuration

[Link] Session Initiated Protocol Input Channel

Channel Name(s) SIP
Applies To ExecutionAndProvisioning (XS) application,
NSExecutionAndProvisioning (NSXS) application,
ExecutionDataless application, WebRTCGatewayapplication.
Description Input channel used to output all incoming and outgoing SIP
messages.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

[Link] Session Initiated Protocol Media Input Channel

Channel Name(s) SIPMedia
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used to output all incoming and outgoing SIP
messages exchanged between the Application Server and the
Media Server.
Special None
Configuration

[Link] Sharing Control Input Channel

Channel Name(s) SharingControl
Applies To USS application
Description Input channel used to report logs related to Sharing Control
activities for the USS application on the Sharing Server.
Special None
Configuration

[Link] Sharing Data Input Channel

Channel Name(s) SharingData
Applies To USS application
Description Input channel used to report logs related to Sharing Data
activities for the USS application on the Sharing Server.
Special None
Configuration

[Link] Short Message Peer-to-Peer Input Channel

Channel Name(s) SMPP
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used to output all incoming and outgoing Short
Message Peer-to-Peer Protocol (SMPP)-related messages.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 47 OF 247
[Link] Simple Management Application Protocol Input Channel
Channel Name(s) SMAP
Applies To ExecutionAndProvisioning (XS and PS) application,
NSExecutionAndProvisioning (NSXS and NSPS) application,
OpenClientServer application, [Link],
NSPortal application, ExecutionDataless application,
Provisioning application
Description Input channel used to capture the alarm and performance
measurement (PM)-related information exchanged between the
BroadWorks processes and the SNMP agent.
Special None
Configuration

[Link] Simple Message Desk Interface Input Channel

Channel Name(s) SMDI
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description Input channel used to output all incoming and outgoing
simplified message desk interface (SMDI) messages.
Special None
Configuration

[Link] Simple Network Management Protocol Input Channel

Channel Name(s) SNMP
Applies To [Link]
Description Input channel used to capture SNMP agent internal logs.
Special None
Configuration

[Link] ServiceOS Input Channel

Channel Name(s) ServiceOS
Applies To ExecutionAndProvisioning (XS) application,
NSExecutionAndProvisioning (NSXS) application,
ExecutionDataless application
Description The ServiceOS is an internal framework commonly used by the
Application Server Execution Server and Network Server
Execution Server. Logs on this channel represent internal
framework events.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 48 OF 247
[Link] Sh Interface Input Channel
Channel Name(s) ShInterface
Applies To ExecutionAndProvisioning (PS) application, Provisioning
application
Description Input channel that can be used to dump all events related to the
Sh (IMS mode only) interface integration.
Special None
Configuration

[Link] SynchAPI Input Channel

Channel Name(s) SynchAPI
Applies To NSExecutionAndProvisioning (PS) application, Provisioning
application
Description Input channel that can be used to dump all incoming and
outgoing SyncAPI messages.
Special None
Configuration

[Link] Timer Input Channel

Channel Name(s) Timer
Applies To ExecutionAndProvisioning (XS) application, ExecutionDataless
application
Description The Application Server Execution Server uses internal timers.
Logs related to timers waking up and processing problems are
reported through this channel.
Special None
Configuration

[Link] Subscriber Subsystem Input Channel

Channel Name(s) SubscriberSubsystem
Applies To ExecutionDataless application, Provisioning application
Description This input channel is used by the Subscriber Agent.
Special None
Configuration

[Link] SCF Processor Input Channel

Channel Name(s) SCFProcessor
Applies To SCFProcessor application
Description This input channel is used by the SCF processor.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 49 OF 247
[Link] SS7 CAP Input Channel
Channel Name(s) SS7Cap
Applies To SCFProcessor application, SS7ProtocolStack application
Description This input channel is used by the SS7 CAP.
Special None
Configuration

[Link] SS7 MAP Input Channel

Channel Name(s) SS7Map
Applies To SCFProcessor application, SS7ProtocolStack application
Description This input channel is used by the SS7 MAP.
Special None
Configuration

[Link] SS7 OAM Input Channel

Channel Name(s) SS7oam
Applies To SS7ProtocolStack application
Description This input channel is used by the SS7 OAM.
Special None
Configuration

[Link] SSH Client Input Channel

Channel Name(s) SshClient
Applies To NodeController application
Description This input channel is used by the NodeController application to
log SSH client-related debug information.
Special None
Configuration

[Link] Web Socket Input Channel

Channel Name(s) webSocket
Applies To WebRTCGateway application
Description Input channel used to report logs related to Web Socket
activities for the WebRTCGateway application on the WebRTC
Server.
Special None
Configuration

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 50 OF 247
[Link] XML IPC Input Channel
Channel Name(s) XML-IPC
Applies To WebRTCGateway application
Description Input channel used to report logs related to interprocess
communication (IPC) activities between the processes of the
WebRTCGateway application on the WebRTC Server.
Special None
Configuration

5.1.3 Input Channel Attributes Channels

The attributes for the input channels are configured through the CLI under each
application logging context. The following Profile Server example shows the logging level
commands and sublevels.
PS_CLI/Applications/BroadworksFileRepos/Logging> ?
0) get : show Logging-related attributes
1) set : modify Logging-related attributes
2) InputChannels : go to level InputChannels
3) OutputChannels : go to level OutputChannels

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re
(repeat)
PS_CLI/Applications/BroadworksFileRepos/Logging> InputChannels

PS_CLI/Applications/BroadworksFileRepos/Logging/InputChannels> ?
0) get : show InputChannels-related attributes
1) set : modify InputChannels-related attributes
2) clear : clear InputChannels-related attributes

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re
(repeat)

PS_CLI/Applications/BroadworksFileRepos/Logging/InputChannels> get
Name Enabled Severity
===========================================
BWCommunicationUtility true
FileRepos true
Generic true

3 entries found.

5.1.4 Enable or Disable BroadWorks Debug Logging

The debug logging is enabled and disabled from the CLI. Under the application context,
the enabled attribute can be changed and the change takes effect automatically.
NS_CLI/Applications/NSExecutionAndProvisioning/XS/Logging/InputChannels>
get
Name Enabled Severity
===================================================================
Generic true Debug
Debug false
ServiceOS true
NRSLog true
CommonPersistency true Info
BroadsoftCommonCommunicationTransport true Info

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 51 OF 247
BroadsoftCommonCommunicationTransportKeepAlive false Info
CallP true
SMAP false
Sip true
Timer true
Overload true

12 entries found.

For the SNMP agent, the script /user/local/broadworks/bw_base/bin/bwdebugreload must

be used to reload any changes to the debug configuration file.

5.1.5 Output Channel attributes

The default amount of disk space used by the logs for the Application Server is 7 GB (1
GB for Provisioning Server logs, 6 GB for Execution Server logs). By changing the
OutputChannel “FileSizeMegs” and “NumberOfFiles”, you can control the amount of disk
space used by the Provisioning and Execution Server log files.

Attribute Definition

Name “File” to indicate to the logging framework to use the file output channel.

Directory Indicates where the log files are stored.

File Prefix The string that the log file starts with.

The size the file should grow to before it is closed and a new file is created.
File Size
The size is set in MB.

NumberOfFiles The amount of files to store before the oldest file is deleted.

Enabled Turns on the output channel.

It is recommended to augment the number of log files when the disk size permits it. This
makes sure that log files are kept for a longer period of time, which allows better
troubleshooting when required. The formula for defining the total number of Execution
Server and Provisioning Server logs is as follows:
 The Provisioning Server should never be more than 20% of the Execution Server
logs. The Execution Server generates many more logs per minute, so it reaches it
maximum much more quickly than the Provisioning Server does.
 The total of Execution Server and Provisioning Server logs should never be more than
33% of the total partition size on which the logs are residing. This makes sure that
other applications have available disk space for their logs and the logs would not
conflict with the Application Server database.

5.1.6 Log Rotation

WARNING: If usage of the partition in which /var/broadworks/logs/appserver grows to 100%,

the Application Server crashes. It is important that disk usage be monitored to make sure this
does not occur.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 52 OF 247
There are log files for each day of the month such as, Aspmd, Snaspmd, Snmpas,
[Link], XSOutput##.log, and PSOutput##.log. The day of the month is appended
to the file name, and these files are stored until they are overwritten the next month. There
are also Application Server and Provisioning Server logs that are rotated by size and are
deleted when the configured number of files to store is reached. Delete unnecessary old
files (files not currently being written to).

5.1.7 Sample Debug Configuration File

All debug logging configurations are done through the CLI under the application level (for
example, AS_CLI/Application/CommPilot/Logging). However, there is still an exception;
the Software Manager uses a debug configuration file. Following is an example for the
SNMP agent.

<Log Enabled="True" Priority="4">
<InputChannels Severity="Debug">
<InputChannel Name="SWManager" Enabled="True" Severity="Debug"/>
<InputChannel Name="PatchToolBcctTraffic" Enabled="True"/>
<InputChannel Name="Generic" Enabled="True" Severity="Debug"/>
<InputChannel Name="BroadsoftCommonCommunicationTransport"
Enabled="True" Severity="Debug"/>
<InputChannel Name="BroadsoftCommonCommunicationTransportKeepAlive"
Enabled="False" Severity="Info"/>
<InputChannel Name="PatchToolPersistence" Enabled="True"/>
<InputChannel Name="SmapInputChannel" Enabled="True"
Severity="Debug"/>
</InputChannels>

5.2 BroadWorks UNIX Command Execution Logging

Every time a BroadWorks UNIX command is run, a syslog is generated and stored in a file
under /var/adm/[Link]. A log contains information such as who ran the command,
when it was executed, and the list of arguments that were passed to the command.
bwadmin@mtl64lin02$ cat [Link] | head -10
Jun 15 09:25:14 mtllinuxmisc04 BROADWORKS[3723]: uid=0(root) gid=0(root) groups=
0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel) COMMAND=./bwuseradd --s
hell=/bin/ksh --role=BWORKS --passwd=bwadmin bwadmin
Jun 15 09:30:39 mtllinuxmisc04 BROADWORKS[3045]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/switch
-[Link]
Jun 15 09:30:42 mtllinuxmisc04 BROADWORKS[3117]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/dbsetu
[Link] AS_Rel_13.0_1.193
Jun 15 09:30:42 mtllinuxmisc04 BROADWORKS[3140]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/MoExte
[Link] -getMaintState
Jun 15 09:30:42 mtllinuxmisc04 BROADWORKS[3156]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/timest
[Link] -dsn appserver load
Jun 15 09:30:43 mtllinuxmisc04 BROADWORKS[3330]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/create
[Link]
Jun 15 09:30:50 mtllinuxmisc04 BROADWORKS[3395]: uid=502(bworks) gid=501(bwadmin

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 53 OF 247
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/AS_Rel_13.0_1.193/bin/loadba
[Link]
Jun 15 09:31:05 mtllinuxmisc04 BROADWORKS[3555]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/bw_base/bin/[Link]

Jun 15 09:31:05 mtllinuxmisc04 BROADWORKS[3558]: uid=502(bworks) gid=501(bwadmin

) groups=501(bwadmin) COMMAND=/usr/local/broadworks/bw_base/bin/[Link] /u
sr/local/broadworks/bw_base/bin/[Link] [Link] bwAutoBackup.s
h
Jun 15 09:31:05 mtllinuxmisc04 BROADWORKS[3561]: uid=502(bworks) gid=501(bwadmin
) groups=501(bwadmin) COMMAND=/usr/local/broadworks/bw_base/bin/[Link]
-getMaintState

5.3 Miscellaneous Logs

5.3.1 Linux System Logs

The /var/log/messages file contains all platform hardware, operating system, and kernel
logs, which are helpful to debug platform-related issues.

5.3.2 Enable TimesTen Logs

The TimesTen logs are enabled by default and they are controlled by the
/var/TimesTen/<ttVersions>/[Link] file.
TimesTen outputs logs through syslog to /var/log/messages on Linux. By default, all
[Link] level of severity TimesTen logs is directed to the messages file. This generates
a large number of non-error-related logs. Reduce the level of logging activity by
commenting on the following non-error channel from the /etc/[Link] file. Example:
[Link] /var/adm/messages

To apply the changes, reset the syslogd daemon (/etc/rc2.d/ S74syslog script).

5.3.3 Apache Logs

The Apache error logs are rotated daily and are stored in /var/broadworks/logs/apache.
The name of each log is in the format error_log.[Link].

5.3.4 Tomcat Logs

The Tomcat logs are rotated daily and are stored in /var/broadworks/logs/tomcat. The
name of each log is in the format of [Link], where DD is the day of the
month.

5.3.5 Oracle Logs

The Oracle database running on the Database Server exposes various logs files, each
associated with a functional component of the database. The description of the relevant
log files is documented in the BroadWorks Database Server Configuration Guide.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

The BroadWorks Performance Measurements Interface Specification Guide and the

BroadWorks Fault and Alarm Interface Specification Guide contain detailed information
about faults and performance measurements for the system.
Simple Network Management Protocol (SNMP) v2c/v3 traps and get are supported.

6.1 Alarms
The Network Server, Application Server, Provisioning Server, Execution Server, and
Media Server provide alarms. The complete list of alarms can be found in the
BroadWorks Fault and Alarm Interface Specification Guide.
BroadWorks generated faults can be one of the following severities:
Alarm Level Action Necessary

Informational None

Low Check the affected system area; this alarm typically indicates a problem with
connectivity, and the redundancy of the system may be in jeopardy.

Medium Check the affected system area; this alarm typically indicates a problem with
connectivity, and the redundancy of the system may be in jeopardy.

High Contact BroadSoft immediately.

Critical Contact BroadSoft immediately.

6.1.1 Open System Alarms

Use this command to open a connection with the Simple Network Management Protocol
(SNMP) agent to obtain system alarms and event logs generated by BroadWorks.

[Link] Open Connection for Alarms

From the CLI Monitoring/Alarm level, type open [port]. Valid values include:
Field Type Valid Values Description
[port] Integer 1000 to 65535 Optional – The port used to
communicate with the SNMP agent.
(Port 30002 is recommended and is
the default value.)

Example:
Open
Using port 30002...
...Done

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 55 OF 247
6.1.2 View System Alarms
Use this command to get and search for system alarms in the local CLI backlog. Use the
command alone to display all alarms in the local CLI backlog. Use the command with a
specified variable to display alarms that match the criteria.

[Link] Get and Search for System Alarms

From the CLI Monitoring/Alarm level, type get to display all alarms, or type
get [<attribute> <relation> <value>] [<back>] to specify which alarms to
display. Valid values include:
Variable Field Attribute Valid Values Description
<attribute> type Choice Notification The trap is for informational
purposes.
Alarm The trap is reporting an alarm.
Software Error The trap is reporting a software
error.
TrapName String 0 to 255 The trap name.
characters
Problem String 0 to 255 The text indicating details of the
characters problem.
Subcomp String 0 to 255 Software component within the
characters BroadWorks entity, reporting the
alarm: SIP, database, and so on.
Recommend String 0 to 255 The recommended action to take in
action characters response to this trap.
State Choice Off Alarm is off.
On Alarm is on.
Time String 0 to 255 The date and time at which the
characters alarm was generated.
Comp String 0 to 255 BroadWorks entity reporting the
characters alarm: Application Server, Media
Server, or Network Server.
Sysname String 0 to 255 The host name of the system
characters running the BroadWorks software.
TrapSeverity Choice Informational Warning – No immediate problem.
Low Minor problem – Possible service
impact.
Medium Minor problem – Minor service
impact.
High Major problem – Major service
impact.
Critical Emergency – System outage.
Id Integer 0 to A sequentially generated number
2147483647 that can be used to uniquely identify
the alarm.
<relation> Choice <= Less than or equal to
== Equal to
>= Greater than or equal to
HAS Contains
<back> numAlarms Integer 1 to 10000 The number of alarms to move

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Type
Get

Result (sample provided)

Wed Mar 31 12:30:20 EDT 2010
idnum = 2375
trapName = bwSystemHealthReport
severity = Informational
trapType = Notification
component = Applicationserver
subcomponent = Processmonitor
systemHostName = [Link]
problemText:
No abnormal condition detected.
recommendedActions:

None.

Type
get TrapSeverity <= Medium

Result (sample provided)

Thu Apr 01 07:29:03 EDT 2010
idnum = 2667
trapName = bwSipTcpSocketError
severity = Low
trapType = Notification
component = Applicationserver
subcomponent = Sip
systemHostName = [Link]
problemText:
SIP TCP connection associated with [Link]:61620 was released
or unable to be established.
recommendedActions:

1 alarm(s) found

6.1.3 View and Set CLI Alarm Backlog Size

The SNMP agent backlog size determines how many alarms are stored on disk and thus
the maximum number of alarms that can be retrieved by CLI clients. When the actual
number of historical alarms overflows the size parameter, the oldest alarms are deleted.
Having a “localsize” more than “size” is allowed but it yields the same result as having
“localsize” equal to “size”. In other words, “localsize” is limited by “size”.
The Logging to File option is used to persist alarms across SNMP agent restarts.
The recommended backlog size is “1000” alarms.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 57 OF 247
[Link] View Current CLI Alarm Backlog Size
From the CLI Monitoring/Alarm level, type showconfig. The result displays (a sample is
provided).
Alarms-Related Configuration
Local backlog size (nb. alarms) = 1000
SNMP agent logging to file = enabled
SNMP agent backlog size = 1000

[Link] Set CLI Alarm Backlog Size

1) Make sure you are at the Monitoring/Alarm> level.
2) Enter:
set <attribute> ↵
… where:
Variable Field Type Valid Values Description
<attribute> localsize Integer 1 to 5000 Enter an integer, 1 to
5000.
The local alarms
backlog size (in number
of alarms).
size Integer 1 to 5000 Enter an integer, 1 to
50000.
The SNMP agent alarms
backlog size (in number
of alarms).
logging Choice disabled Logging to disk is off.
enabled Logging to disk is on.

6.1.4 Enable and Disable Real-Time Alarms Echoing

Use this command to enable or disable real-time alarm echoing. Alarms appear at the CLI
as they occur when the command is enabled.

[Link] Enable Real-time Alarms Echoing

From the CLI Monitoring/Alarm level, type show on. The result displays.
...Done

NOTE: When alarms occur, they appear in the window from which this command was invoked.
To avoid interrupting other CLI commands, open a separate window.

[Link] Disable Real-time Alarms Echoing

From the CLI Monitoring/Alarm level, type show off. The result displays (a sample is
provided).
...Done

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 58 OF 247
6.1.5 Clear Alarms
Use this command to clear alarms in the CLI alarm buffer. From the CLI Monitoring/Alarm
level, type clear.
Example
clear

Result
...Done

6.1.6 Close System Alarms

Use this command to close the connection with the SNMP agent, to obtain system alarms
and event logs generated by BroadWorks.

[Link] Close Connection for Alarms

From the CLI Monitoring/Alarm level, type close. The result displays (sample provided).
...Done

6.2 Threshold Alarms

BroadWorks supports configurable threshold alarms. Thresholds provide a mechanism to
monitor the state of a BroadWorks server. With thresholds configured, a BroadWorks
server automatically sends an SNMP trap when an operator-specified limit is reached
against a specific BroadWorks counter or gauge. Thresholds can be assigned against
any counter, gauge, or column defined in a BroadWorks application MIB (applies to all
server types).
Thresholds are configurable through the CLI or via SNMP.

6.2.1 Threshold Attributes

There are two kinds of thresholds: counter thresholds and gauge thresholds.
The attributes of counter thresholds include:
 Supports many thresholds per counter.
 Supports activating and deactivating thresholds for a counter.
 A notification alarm is generated each time a counter crosses the current comparison
level of a threshold.
Each threshold level has the following attributes:
− Current comparison level – This level establishes when an alarm is generated.
It is not configurable by the service provider.
− Initial comparison level – This level provides the basis of comparison. Reset
the counter to 0 to compare to this level.
− Offset level – This level defines the increment added to the current comparison
value whenever the counter crosses the current comparison value. The offset
may be 0, meaning the threshold alarm is raised at most once, unless the counter
is reset.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 59 OF 247
− Severity level – This level defines the severity level included in each notification
alarm generated by this threshold.

NOTE: When the counter rolls over its maximum value, the current comparison level is also
rolled over, to continue generating the alarms at every “offset” increment.

The attributes of gauge thresholds include:

 Support for many thresholds per gauge.
 Support for activating and deactivating thresholds for a gauge.
 To avoid repeatedly generating alarms around a particular threshold level, each
threshold level has a NotifyHigh level and a NotifyLow level. The following attributes
are defined for each threshold:
− NotifyHigh comparison level – This gauge value establishes when an alarm is
generated for a gauge value in a rising direction. When subsequent similar
gauge values cross the NotifyHigh level, they do not generate an alarm until the
gauge has reached the corresponding NotifyLow level.
− NotifyLow comparison level – This gauge value establishes when an alarm is
generated for a gauge value in a falling direction. When subsequent similar
gauge values cross the NotifyLow level, they do not generate an alarm until the
gauge has reached the corresponding NotifyHigh level.

6.2.2 Configure Thresholds using CLI

[Link] Add New Counter Threshold

From the CLI Monitoring/Threshold level, type add counter “Sync API auth
failures” syncNbAuthorizationFailures 1 1 high active. The following
response appears (sample provided).
...Done

NS_CLI/Monitoring/Threshold> get counter

Description Name Initial Offset
Current
Severity Status
Sync API auth failures syncNbAuthorizationFailures 1 1
0
informational active

In this example, “Synch API auth failures” is the description that appears in the Problem
Text field of the threshold alarm and “syncNbAuthorizationFailures” is the counter
monitored by the threshold mechanism. The alarm raises with each new synch API
authorization failure (initial value = 1, offset value = 1).

[Link] Add New Gauge Threshold

From the CLI Monitoring/Threshold level, type add gauge “Unassigned numbers”
systemNbUnassignedDNs 100 200 informational active. The following
response appears (sample provided).
...Done

NS_CLI/Monitoring/Threshold> get gauge

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 60 OF 247
Description Name Low High Severity
Status
Unassigned numbers systemNbUnassignedDNs 100 200 informational
active

In this example, “Unassigned numbers” is the description that appears in the Problem Text
field of the threshold alarm and “systemNbUnassignedDNs” is the value of the threshold
gauge. The alarm is raised whenever the number of unassigned numbers goes outside of
the range (100, 200).

[Link] Delete Threshold

From the CLI Monitoring/Threshold level, type:
delete counter syncNbAuthorizationFailures

or
delete gauge systemNbUnassignedDNs

The following response appears.

...Done

NOTE: If there are multiple thresholds for the same counter/gauge, then a list of thresholds
appear with the corresponding indexes. Use the Delete command (again) with the appropriate
index as an additional parameter on the command line.

[Link] Activate and Deactivate a Threshold

From the CLI Monitoring/Threshold level, type:
set counter syncNbAuthorizationFailures status inactive,

or
set gauge systemNbUnassignedDNs status inactive

The following response appears.

...Done

NOTE: If there are multiple thresholds for the same counter/gauge, then a list of thresholds
appear with the corresponding indexes. Use the Set command (again) with the appropriate
index as an additional parameter on the command line. For example, set counter
syncNbAuthorizationFailures status inactive rowIndex 0.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 61 OF 247
[Link] Invalid Thresholds
Thresholds that are in an active state contain valid parameters. Thresholds that are in an
invalid (4) state contain invalid parameters. Check the status of a threshold. You set it to
the active state to validate it.
Possible causes for a threshold being invalid include:
 The counter or gauge specified does not exist.
 The offset value of a counter threshold is not strictly positive (>0).
 The NotifyLow of a gauge threshold is higher or equal than its NotifyHigh.

6.2.3 Configure Thresholds through SNMP

The threshold SNMP MIB is described in the [Link] file distributed
with BroadWorks.
The following threshold tables are located in [Link].
[Link]:
 [Link]
 [Link]
The following sections explain how to create, delete, and control the thresholds. For more
information on the meaning of each column in the counter or gauge threshold tables, see
the BroadSoft Managed Objects Functional Specification (see your BroadSoft
representative).

[Link] Create New Threshold

To create a new threshold, create a new row in the threshold table. Set the Control
column of the row to “create (3)”. The Control column is
bwCounterThresholdControl for the counter threshold table and
bwGaugeThresholdControl for the gauge threshold table. The row is specified using
the Index column (bwCounterThresholdIndex or bwGaugeThresholdIndex,
respectively). The index is valid. The Control column takes the “inactive (0)” value. The
threshold is inactive by default. You can set other columns before you activate the
threshold.

[Link] Delete Threshold

To delete a threshold, type a “delete (2)” value in the Control column. Once a row is
deleted, it is not accessible.

[Link] Activate or Deactivate Threshold

To activate a threshold, type the “active (1)” value in the Control column. To deactivate a
threshold without deleting it, type the “inactive (0)” value in the Control column.

[Link] Column Descriptions

Description Column – Whatever is in the description column is appended to the alarms
generated by the threshold. Use the description column to assess why an alarm was
generated. Example descriptions include:
 Abnormally high number of MGCP restarts
 Database checkpoints failed too many times

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 62 OF 247
Name Column – Use the name column to identify the counter or gauge to be monitored
by the threshold.

NOTE: This is not the name of the threshold.

For instance, the following names align with the examples provided in the example
included in the previous description column:
 MGCPStatsRestartInProgressIns
 ttNbFailedCheckpoints

6.2.4 Application Server Thresholds Limitation

The Application Server is internally divided in two processes: the Execution Server (XS)
and the Provisioning Server (PS). Some counters and gauges are located in the XS,
whereas others are located in the PS. Under normal circumstances, the Application
Server determines to which process the counter or gauge is connected. It ensures the
right process generates the alarm when the threshold is reached.
A few counters and gauges are defined in both the Execution Server and Provisioning
Server. These counters and gauges may have different names in the Management
Information Base (MIB), but their internal names are the same in both processes. The
threshold sub-system cannot determine on which server they are located.
This has two results:
 A valid Provisioning Server counter or gauge does not cause the threshold to become
“invalid” when activated and therefore, does not generate threshold alarms.
 A valid Execution Server counter or gauge generates threshold alarms for both the
Provisioning Server and Execution Server without discrimination. For example, the
Execution Server generates a threshold alarm and the current value of the threshold
is updated. Subsequently, the Provisioning Server generates a threshold alarm
based on the new value. The current value continues to be updated without
intervention.
An example of a counter is ttNbBackdoorUpdates or psTtNbBackdoorUpdates. In this
case:
 If you create a threshold on psTtNbBackdoorUpdates, it does not have an effect.
The threshold can be “active”.
 If you create a threshold on ttNbBackdoorUpdates, it has the effect described above.
The name of the Provisioning Server version differs from the Execution Server version
only by the prefix.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 63 OF 247
6.3 Performance Measurements
Performance measurements are organized in a tree structure. Commands are provided
to navigate, view the next, view the current position of, and view the values for MIB nodes.

NOTE: The local host address ([Link]) must be included in the SNMP Agent’s access list.

6.3.1 Navigate through MIB Nodes

Use this command to navigate through the MIB nodes in a manner similar to the standard
cd (change directory) command. Use the slash (/) parameter to navigate back to root.
Use the (..) parameter to go back one level.

[Link] Navigate MIB Nodes

From the CLI Monitoring/PM/Execution or from Monitoring/PM/Provisioning (for the
Application Server) level, type cd [<path>]. Valid values include:
Variable Field Type Valid Values Description
<path> String 0 to 1023 characters The path of the desired
node.

[Link] Examples
Type:
cd /system

Response (sample provided)

executionServer/systemModule

6.3.2 View Next MIB Node

Use this command to view the next MIB node in a manner similar to the standard ls (list)
command.
From the CLI Monitoring/PM/Execution or from Monitoring/PM/Provisioning (for the
Application Server) level, type ls [<option>] [<path>]. Valid values include:
Variable Field Type Valid Values Description
<option> Choice -r Shows the tree structure of the
MIB nodes.
-n Shows the name of the nodes
containing values.
<path> String 0 to 1023 characters Shows the path to the node you
want to view.

[Link] Examples
Type:
AS_CLI/Monitoring/PM/Execution> ls –r

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 64 OF 247
Response (sample provided)
mgcpModule
mgcpStats
imsModule
imsStats
databaseModule
databaseStats
timesTen
xsRemoteXla
callpModule
callpStats
smtpModule
smtpStats
sipModule
sipStats
congestionManagement
services
accountCodes
anonymousCallRejection

6.3.3 View Current Position in MIB

Use this command to view the current position in the MIB in a manner similar to the
standard pwd (present working directory) command.
From the CLI Monitoring/PM/Execution or from Monitoring/PM/Provisioning (for the
Application Server) level, type:
pwd

Response (sample provided)

executionServer

6.3.4 View MIB Node Values

Use this command to view MIB node values. MIB node values can be viewed in the
following ways:
 All the performance measurements throughout the MIB nodes tree structure
 The performance measurements related to the current node
 The performance measurements of a specified path
From the CLI Monitoring/PM level, type get [<option>] [<path>] [<tableKey>].
Valid values include:
Variable Field Type Valid Values Description
<option> Choice -r Displays all performance
measurements for the current tree
structure of the MIB nodes.
-f Displays a full description of
performance measurements.
<path> String 0 to 1023 Displays the path of the node for
characters which you want to view performance
measurements.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 65 OF 247
Variable Field Type Valid Values Description
<tableKey> String 0 to 127 characters Used to specify a row key when the
node is displayed as a table.

[Link] Examples
Type:
Get

Response (sample provided)

-------------------------------------------------------------------------
executionServer/callpModule/callpStats/
-------------------------------------------------------------------------
*bwCallpNetworkOriginationAttempts 4
*bwCallpNetworkTerminationAttempts 5
*bwCallpNetworkTerminationsAnswered 4
*bwCallpUserOriginationAttempts 6
*bwCallpUserTerminationAttempts 9
*bwCallpUserTerminationsAnswered 7
*bwCallpEmergencyCallAttempts 0
*bwCallpEmergencyCallAlarms 0

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

This chapter describes server management for the BroadWorks servers.

7.1 Server State Management

7.1.1 Server State

BroadWorks servers support an administrative state.
The server administrative states include:
 Locked – No traffic is allowed through the server.
 Unlocked – Traffic is allowed through the server.
The server is operating normally when it is in the Unlocked administrative state.
BroadWorks servers are in a Locked state while they are rebooting and they transition to
the Unlocked state when they are fully initialized.

7.1.2 Application State

Each BroadWorks application implements its own state. The application state is
presented as an administrative state and an effective state. Both states can be viewed
from the CLI Maintenance/ManagedObject level using the “get broadworks full” command.
Only the administrative state can be controlled by an operator; the effective state simply
reflects the actual state of the application. The administrative state is controlled using the
lock and unlock commands at the CLI Maintenance/ManagedObject level. As the
commands suggest, the possible administrative states are:
 Locked
 Unlocked
When an application administrative state is set to one of these states, it means that the
application tries to transition to that state (assuming the application is running).
On the other hand, the effective state reflects the actual state of the application and can
have the following values.
 Locked: The application is locked. Depending on the application’s implementation, it
may mean that it provides limited to no functionality at all.
 Locking: The application is in the process of being locked. This is a transitional state.
 Unlocked: The application is unlocked; all its functionality is available.
 Unlocking: The application is in the process of being unlocked. This is a transitional
state.
 Stopping: The application is in the process of being stopped. This is a transitional
state.
 Stopped: The application is stopped; none of its functionality is available.
 InTrouble: The application is not functional; one or more (but not all) of its containers
is dead. The application must be restarted. The application also enters this state
when one or more (but not all) of its containers is restarted by the process monitor.
 Dead: The application is not functional; all of its containers are dead. The application
must be restarted.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 67 OF 247
Following is an example of the get broadworks full command, where four
BroadWorks applications are deployed but only two are running.
AS_CLI/Maintenance/ManagedObjects> get broadworks full
BroadWorks Managed Objects
==========================

* Server:
Identity..............: AS
Version...............: Rel_17.0_1.446
Administrative State..: Unlocked

* Applications:
Name Version Deployed Administrative State Effective
State
===================================================================================
====
ExecutionAndProvisioning 17.0_1.446 true Unlocked
Unlocked
FlashPolicy 17.0_1.446 true Unlocked
Stopped
OpenClientServer 17.0_1.446 true Unlocked
Stopped
WebContainer 17.0_1.446 true Unlocked
Unlocked

4 entries found.

* Hosted Applications:
Name Version Context Path Deployed
================================================================
CommPilot 17.0_1.446 /CommPilot true
DeviceManagementFiles 17.0_1.446 /DeviceManagement true
JWSFiles 17.0_1.446 /FileRepos true
MediaFiles 17.0_1.446 /media true
OCIFiles 17.0_1.446 /ocifiles true

5 entries found.

Currently running BroadWorks Application processes:

apache process monitor (pid=25162)

apache (pid=25253)
execution process monitor (pid=25090)
execution (pid=25504)
provisioning process monitor (pid=25117)
provisioning (pid=25540)
remotexla process monitor (pid=25068)
remotexla (pid=25105)
tomcat process monitor (pid=25140)
tomcat (pid=25189)

Currently running BroadWorks Platform processes:

BroadWorks Configuration Agent process monitor (pid=24820)

BroadWorks Configuration Agent (pid=24850)
BroadWorks Software Manager (pid=19703)
BroadWorks SNMP Agent process monitor (pid=23596)
BroadWorks SNMP Agent (pid=23626)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 68 OF 247
The server administrative state, as shown at the top of the above output as Unlocked, also
has an impact on the application’s effective state. For instance, if an operator locks the
server, all application effective states transition to Locked regardless of the application’s
administrative state. This is because the Locked server’s administrative state supersedes
the application’s administrative state. However, when the server is in Unlocked state, an
operator can freely change an application administrative state and the application effective
state transitions to the requested state. Following is an example where the server
administrative state is set to “Locked” and the application’s administrative state is
“Unlocked” but the effective state of the running application is “Locked”.
AS_CLI/Maintenance/ManagedObjects> get broadworks full
BroadWorks Managed Objects
==========================

* Server:
Identity..............: AS
Version...............: Rel_17.0_1.446
Administrative State..: Locked

* Applications:
Name Version Deployed Administrative State Effective
State
===================================================================================
====
ExecutionAndProvisioning 17.0_1.446 true Unlocked
Locked
FlashPolicy 17.0_1.446 false Unlocked
Stopped
OpenClientServer 17.0_1.446 false Unlocked
Stopped
WebContainer 17.0_1.446 true Unlocked
Locked

4 entries found.

5 entries found.

Currently running BroadWorks Application processes:

apache process monitor (pid=25162)

Currently running BroadWorks Platform processes:

BroadWorks Configuration Agent process monitor (pid=24820)

BroadWorks Configuration Agent (pid=24850)
BroadWorks Software Manager (pid=19703)
BroadWorks SNMP Agent process monitor (pid=23596)
BroadWorks SNMP Agent (pid=23626)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: The CLI can be used to query the server and application states without the server
running and without logging in to the CLI.

From the CLI Maintenance/ManagedObjects level, type get broadworks. The result
displays (sample provided).
BroadWorks Managed Objects
==========================

* Server:
Identity..............: AS
Version...............: Rel_17.0_1.430
Administrative State..: Unlocked

* Applications:
Name Version Deployed Administrative State Effective State
=======================================================================================
ExecutionAndProvisioning 17.0_1.430 true Unlocked Unlocked
FlashPolicy 17.0_1.430 true Unlocked Unlocked
OpenClientServer 17.0_1.430 true Unlocked Unlocked
WebContainer 17.0_1.430 true Unlocked Unlocked

4 entries found.

* Hosted Applications:
Name Version Context Path Deployed
================================================================
CommPilot 17.0_1.430 /CommPilot true
DeviceManagementFiles 17.0_1.430 /DeviceManagement true
JWSFiles 17.0_1.430 /FileRepos true
MediaFiles 17.0_1.430 /media true
OCIFiles 17.0_1.430 /ocifiles true

5 entries found.

7.2 Application Management

Operators can install (on the Xtended Services Platform only), activate, and deploy
applications on BroadWorks servers. The servers provide Application Management
functionality through the CLI, under the Maintenance/ManagedObjects level. Applications
can be installed, uninstalled, activated, deactivated, deployed, and undeployed. All
operations are CLI-driven. No manual operation is required.
Applications must be packaged using two possible formats, the standard web application
archive (.war) format, and the BroadWorks Application format (.bwar). The web
application must follow specific guidelines documented in the BroadWorks Xtended
Services Platform Configuration Guide so that they can be installed on the Xtended
Services Platform. The BroadWorks applications are created and released by BroadSoft.
The full life cycle of an application is:
Installation  Activation  Deployment  Undeployment  Deactivation  Uninstallation

Individual CLI operations are described in the following subsections. Unless otherwise
mentioned, no restart is necessary for any operation. All changes are effective
immediately.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: Installation of applications can only be performed on the Xtended Services Platform.

The first step in the application life cycle is the installation. For the installation of
applications that are not prepackaged with the Xtended Services Platform, the application
file (.war or .bwar) must be copied to a local temporary directory on the server, for
example, /tmp. Using the CLI, in the Maintenance/ManagedObjects level, invoke the
install command, providing the full path name of the application file. Note that the name of
the file is irrelevant to the Xtended Services Platform. All the descriptive and operational
data used by the Xtended Services Platform are inside the file. By default, prepackaged
applications are located in the /usr/local/broadworks/apps directory.
An application is made unique by its name and its version, both of which are defined in the
internal deployment descriptor (in the .war/.bwar). For web application, the deployment
descriptor is the [Link] file. On the other hand, the BroadWorks application deployment
descriptor is in the [Link] file. The Xtended Services Platform enforces
uniqueness of applications and does not allow installation of two applications with the
same name/version.
Following is an example of an installation of an application.
XSP_CLI/Maintenance/ManagedObjects> install application /tmp/Xsi-
Actions_5.[Link]
Broadworks SW Manager installing Xsi-Actions_5.[Link]...
- Performing basic validation
. Valid application structure
. Name : Xsi-Actions
. Description: BroadWorks Xtended Services Interface
. Version : 5.0
...Done

XSP_CLI/Maintenance/ManagedObjects>

During the installation process, the application file is validated and, if valid, it is internally
copied and installed. Once the application is installed, the file used to install the
application is no longer required and can be deleted.
After being installed, either the application can be activated to make it configurable or it
can be uninstalled.

NOTE: Applications with an automatic upgrade mode are installed by default when the
BroadWorks server is installed. Such applications cannot be uninstalled.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 71 OF 247
7.2.2 Activation
To be able to configure the application, it must be activated. Activation is performed
through the CLI with the activate command. The following examples illustrate the
activation of applications.
AS_CLI/Maintenance/ManagedObjects> activate ?
<type>, Choice = {application}
application:
<name>, String {1 to 80 characters}
<version>, String {1 to 80 characters}
[<contextPath>, String {1 to 80 characters}]

AS_CLI/Maintenance/ManagedObjects> activate application OpenClientServer

17.0_1.446
BroadWorks SW Manager activating...OpenClientServer version 17.0_1.446
...Done

AS_CLI/Maintenance/ManagedObjects> activate application OCIOverSoap

17.0_1.446 /ociSOAP
BroadWorks SW Manager activating...OCIOverSoap version 17.0_1.446
...Done

When activating an application, the name and version are mandatory. The contextPath
parameter is only required when activating a web application. It must not be specified
when activating a BroadWorks application.
Multiple instances of the same web application can be activated using different context
paths. However, a single activation of a BroadWorks application is allowed.
Once activated, applications can be configured. Configuration is done through the CLI
from the application-specific context. This context can be found under the application
context as illustrated in the following example.
AS_CLI/Applications> ?
0) CommPilot : go to level CommPilot
1) DeviceManagementFiles : go to level DeviceManagementFiles
2) ExecutionAndProvisioning : go to level ExecutionAndProvisioning
3) JWSFiles : go to level JWSFiles
4) MediaFiles : go to level MediaFiles
5) OCIFiles : go to level OCIFiles
6) OCIOverSoap : go to level OCIOverSoap
7) OpenClientServer : go to level OpenClientServer

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re
(repeat)

AS_CLI/Applications/CommPilot> ?
0) CCRSProxyUrls : go to level CCRSProxyUrls
1) GeneralSettings : go to level GeneralSettings
2) Logging : go to level Logging
3) RelativeUrl : go to level RelativeUrl
4) WebBranding : go to level WebBranding

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re
(repeat)

As shown in the example, all the activated applications are listed as subcontext. Note
however, that not all applications support configuration.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 72 OF 247
7.2.3 Deployment
Applications activated on the server can be selectively deployed using the CLI deploy
command. Following are examples of application deployments.
XSP_CLI/Maintenance/ManagedObjects> deploy ?
<type>, Choice = {application}
application:
<nameOrContextPath>, String {1 to 255 characters}

AS_CLI/Maintenance/ManagedObjects> deploy application /ociSOAP

BroadWorks SW Manager deploying /ociSOAP...
...Done

AS_CLI/Maintenance/ManagedObjects> deploy application FlashPolicy

BroadWorks SW Manager deploying FlashPolicy...
...Done

AS_CLI/Maintenance/ManagedObjects>

As shown in the previous example, deploying a web application and a BroadWorks

application requires different parameters. A web application is deployed using the context
path specified during the activation. On the other hand, deploying a BroadWorks
application is done using the application name.
The following subsections present the deployment differences between web and
BroadWorks application.

7.2.4 Undeployment
Applications can be undeployed when they are no longer required. Web applications can
be undeployed without restriction; however, before being undeployed, BroadWorks
applications must be stopped. Undeploying an application is done with the command
undeploy invoked from the CLI level Maintenance/ManagedObjects. For web
applications, the context path must be specified and for BroadWorks applications, the
application name must be used. Following are some examples.
AS_CLI/Maintenance/ManagedObjects> undeploy application FlashPolicy
BroadWorks SW Manager un-deploying FlashPolicy...
...Done

AS_CLI/Maintenance/ManagedObjects> undeploy application /ociSOAP

BroadWorks SW Manager un-deploying /ociSOAP...
...Done

AS_CLI/Maintenance/ManagedObjects> get broadworks full

BroadWorks Managed Objects
==========================

* Server:
Identity..............: AS
Version...............: Rel_17.0_1.446
Administrative State..: Unlocked

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 73 OF 247
FlashPolicy 17.0_1.446 false Unlocked
Stopped
OpenClientServer 17.0_1.446 false Unlocked
Stopped
WebContainer 17.0_1.446 true Unlocked
Unlocked

4 entries found.

6 entries found.

Currently running BroadWorks Application processes:

apache process monitor (pid=25162)

Currently running BroadWorks Platform processes:

BroadWorks Configuration Agent process monitor (pid=24820)

BroadWorks Configuration Agent (pid=24850)
BroadWorks Software Manager (pid=19703)
BroadWorks SNMP Agent process monitor (pid=23596)
BroadWorks SNMP Agent (pid=23626)

AS_CLI/Maintenance/ManagedObjects>

Once a web application is undeployed, no instance of this application’s context path is

running within the WebContainer. To reuse the context path, the web application must be
deactivated.
When a BroadWorks application is undeployed, it can no longer be started.
Applications can still be configured once undeployed.

NOTE: Undeploying the BroadWorks Web Container application affects all web applications.
The web applications are no longer running.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 74 OF 247
7.2.5 Deactivation
Undeployed applications can be deactivated using the deactivate CLI command.
Deactivated applications can no longer be configured. As such, their associated CLI level
under the application context is removed. For web applications, the context path is
released and can be reused. Following are some examples of application deactivation.
Note that deactivating a BroadWorks application is done using the application name,
whereas deactivating a web application is done using the context path.
AS_CLI/Maintenance/ManagedObjects> deactivate application
OpenClientServer
BroadWorks SW Manager deactivating...OpenClientServer version 17.0_1.446
...Done

AS_CLI/Maintenance/ManagedObjects> deactivate application /ociSOAP

BroadWorks SW Manager deactivating...OCIOverSoap version 17.0_1.446
...Done
AS_CLI/Maintenance/ManagedObjects>

7.2.6 Uninstallation

NOTE: Only the Xtended Services Platform supports uninstallation of applications.

Installed applications that are no longer required can be uninstalled. This removes all files
related to this application from the server. Only applications in Installed state can be
uninstalled. Furthermore, only applications having their upgrade mode set to “Manual”
can be uninstalled. Those with an automatic upgrade mode cannot be uninstalled as they
are part of the server configuration and they are upgraded with the platform. An
application is uninstalled using the uninstall command form the CLI, in the
Maintenance/ManagedObjects level. The uninstall command requires an application and
a version. Following is an example of an uninstallation of a web application.
XSP_CLI/Maintenance/ManagedObjects> uninstall application Xsi-Actions 5.0
Broadworks SW Manager un-installing Xsi-Actions version 5.0...
...Done

XSP_CLI/Maintenance/ManagedObjects>

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE 1: It is not necessary to log in to the CLI. The server automatically goes to the Unlocked
administrative state after starting.

NOTE 2: The server can also be started from the UNIX prompt as bwadmin using the startbw
command.

On the Application Server and Network Server, a check of the database replication lagging
state for the cluster occurs at startup. For a description of the database replication lagging
state, see section 11.1.1 TimesTen Database Synchronization.
If a database replication lagging condition exists for any peer in the cluster, a warning is
printed.

7.3.1 Start BroadWorks

From the CLI Maintenance/ManagedObjects level, type start. The following results
appear.
AS_CLI/Maintenance/ManagedObjects> start
--> Starting BroadWorks <--

Broadcast message from bworks (Thu Apr 1 15:35:34 2010):

===== BROADWORKS CONTROL --- START INITIATED =====

Starting [done]
AS_CLI/Maintenance/ManagedObjects>

7.3.2 Start BroadWorks Application

It is possible to start a single application using the start application
<application> command.
AS_CLI/Maintenance/ManagedObjects> start application FlashPolicy
--> Starting application FlashPolicy <--

Broadcast message from bworks (Tue Apr 6 07:46:27 2010):

===== BROADWORKS CONTROL --- START INITIATED (FlashPolicy) =====

Starting [done]

AS_CLI/Maintenance/ManagedObjects>

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

WARNING: Stopping a server results in that server refusing any new incoming requests. This
puts the server in an administrative Disabled state.

The BroadWorks server can be stopped at either the UNIX prompt or CLI.

NOTE: It is not necessary to log in to the CLI. The server cannot be stopped from the CLI
unless a traffic lock has first been performed.

7.4.1 Stop BroadWorks

1) From the CLI Maintenance/ManagedObjects level, type lock. The server
transitions from the Unlocked to Locking state before finally entering the Locked state.
2) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
that the server administrative state is Locked.

NOTE: A server transitions to the locked state once all active sessions are terminated. Use the
lock force command form the CLI Maintenance/ManagedObjects level to force the
administration state to Locked.

3) From the CLI Maintenance/ManagedObjects level, type stop. The results display
(sample provided).
AS_CLI/Maintenance/ManagedObjects> stop

+++ WARNING +++ WARNING +++ WARNING +++

Stopping the server and/or application will cause downtime for the
targeted component.
Continue?

Please confirm (Yes, Y, No, N): y

--> Stopping BroadWorks <--

Broadcast message from bworks (Thu Apr 1 15:34:16 2010):

===== BROADWORKS CONTROL --- STOP INITIATED =====

Stopping [done]

AS_CLI/Maintenance/ManagedObjects>

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 77 OF 247
7.4.2 Stop Server without Locking
Specify the force flag to terminate all active sessions.
From the CLI Maintenance/ManagedObjects level, type stop force. The results
display (sample provided).
AS_CLI/Maintenance/ManagedObjects> stop force
--> Stopping BroadWorks <--

Broadcast message from bworks (Thu Apr 1 15:36:49 2010):

===== BROADWORKS CONTROL --- STOP INITIATED =====

Stopping [done]

AS_CLI/Maintenance/ManagedObjects>

7.4.3 Stop BroadWorks Application

A single application can be stopped (with and without locking) from the CLI using the
command stop [force] application <application>.
AS_CLI/Maintenance/ManagedObjects> stop force application FlashPolicy
--> Stopping application FlashPolicy <--

Broadcast message from bworks (Tue Apr 6 07:50:26 2010):

===== BROADWORKS CONTROL --- STOP INITIATED (FlashPolicy) =====

Stopping [done]

AS_CLI/Maintenance/ManagedObjects>

7.5 Check for Running Processes

Use this command to see all BroadWorks-related running processes.
From the CLI Maintenance/ManagedObjects level, type get broadworks full. The
following results appear (sample provided).
AS_CLI/Maintenance/ManagedObjects> get broadworks full
BroadWorks Managed Objects
==========================

* Server:
Identity..............: AS
Version...............: Rel_17.0_1.440
Administrative State..: Unlocked

* Applications:
Name Version Deployed Administrative State Effective State
=======================================================================================
ExecutionAndProvisioning 17.0_1.440 true Unlocked Unlocked
FlashPolicy 17.0_1.440 false Unlocked Stopped
OpenClientServer 17.0_1.440 false Unlocked Stopped
WebContainer 17.0_1.440 true Unlocked Unlocked

4 entries found.

* Hosted Applications:
Name Version Context Path Deployed
================================================================
CommPilot 17.0_1.440 /CommPilot true
DeviceManagementFiles 17.0_1.440 /DeviceManagement true

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 78 OF 247
JWSFiles 17.0_1.440 /FileRepos true
MediaFiles 17.0_1.440 /media true
OCIFiles 17.0_1.440 /ocifiles true

5 entries found.

Currently running BroadWorks Application processes:

apache process monitor (pid=3137)

apache (pid=3234)
execution process monitor (pid=3062)
execution (pid=3464)
provisioning process monitor (pid=3092)
provisioning (pid=3499)
remotexla process monitor (pid=3039)
remotexla (pid=3069)
tomcat process monitor (pid=3115)
tomcat (pid=3174)

Currently running BroadWorks Platform processes:

BroadWorks Configuration Agent process monitor (pid=32483)

BroadWorks Configuration Agent (pid=32513)
BroadWorks Software Manager (pid=29966)
BroadWorks SNMP Agent process monitor (pid=31263)
BroadWorks SNMP Agent (pid=31293)

AS_CLI/Maintenance/ManagedObjects>

NOTE: From the UNIX prompt, type the showrun command to view BroadWorks-related
processes.

7.6 Lock Network Server, Application Server, or Execution Server

Use this command to put the server in the traffic Locked state to make sure that new
sessions are not accepted, and the server waits until all active sessions are cleared before
a server restarts or stops.

WARNING: Locking a server results in that server refusing any new incoming requests. This
takes the server operationally out-of-service.

NOTE: Configuration changes can be made while the server is in the Locked state (unless the
database is locked using a separate procedure, if applicable).

From the CLI Maintenance/ManagedObjects level, type lock. The following response
appears (sample provided).
+++ WARNING +++ WARNING +++ WARNING +++
This command will lock the server. Note that this action could cause
downtime. Continue?

Please confirm (Yes, Y, No, N): y

…Done

The server is in the Locking administrative state while there are sessions in progress. No
new sessions can be started though. The server becomes Locked when the last session
terminates.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 79 OF 247
The server administrative state can be verified at the Maintenance/ManagedObjects level
by typing get broadworks.

7.6.1 Force Lock Server Immediately

From the CLI Maintenance/ManagedObjects level, type lock force. The results
display (sample provided).

…Done

The server administrative state can be verified using the get broadworks command.

7.6.2 Unlock Server

Use this command to unlock the server.
From the CLI Maintenance/ManagedObjects level, type unlock. The results display
(sample provided).
…Done

7.7 Restart Server

It is recommended to restart a server only when in the Locked administrative state. If an
immediate restart is necessary, the “force” flag can be used. The server automatically
goes to the Unlocked administrative state after restarting.

WARNING: Restarting a server results in that server refusing any new incoming requests until
the server restarts.

7.8 Take BroadWorks Server Out of Service for Maintenance

At times, it is necessary to take a BroadWorks server out of service to perform server
maintenance or to power down the server.

7.8.1 Non-redundant Systems

Use this procedure to take a non-redundant server out of service:
1) From the CLI Maintenance/ManagedObjects level, type lock. The server
transitions from the Unlocked to Locking state before it enters the Locked state.
2) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
the server is in the Locked state.

NOTE: A server transitions to the Locked state only once all active sessions are terminated.
Use the lock force command from the CLI Maintenance/ManagedObjects level to force the
administration state to go to Locked.

3) From the CLI Maintenance/ManagedObjects level, type stop. The server is now
prepared to power down. If a power down is required, the following step should be
followed to ensure a graceful shutdown of the server.
4) Log in (or su -) as root.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 80 OF 247
5) At the UNIX prompt, type # sync.
6) At the UNIX prompt, type # shutdown –y –g0 –i5. The server powers down.

7.8.2 Redundant Systems

Use the following procedure to take a redundant server out-of-service. The procedure
applies to any cluster peer (for example, primary or secondary Application Server). The
remaining peer handles all traffic.
1) From the CLI Maintenance/ManagedObjects level, type lock. This ensures the
remaining in-service cluster peer handles all new calls. The server transitions from
the Unlocked to Locking state before it enters the Locked state.
2) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
the server is in the administration Locked state.

NOTE: Terminate all active sessions to allow the server to make the transition to the Locked
state. Use the lock force command from the CLI Maintenance/ManagedObjects level to force
the administration state to Locked.

3) From the CLI System/Peering level, type lock. This ensures changes are not made
to the database for the server under maintenance. The TimesTen database
transitions from the Unlocked to Locking state before it enters the Locked state.
4) From the CLI System/Peering/Peers level, type get to verify the server database
state is Locked.
5) From the CLI Maintenance/ManagedObjects level, type stop.
6) From the UNIX prompt, type $ repctl stop.
7) The server is now prepared to power down. If a power down is required, the following
step should be followed to ensure a graceful shutdown of the server.
8) Log in (or su -) as root.
9) At the UNIX prompt, type # sync.
10) At the UNIX prompt, type # shutdown –y –g0 –i5.

7.9 Restore BroadWorks Server to In-service

It is important to properly return servers to in-service.

7.9.1 Non-redundant Systems

Use the following procedure to restore a non-redundant server back to in-service:
1) Power up the server. The BroadWorks application starts automatically.
2) Log in as bwadmin.
3) From the CLI Maintenance/ManagedObjects level, type get broadworks full.
4) Verify that the administration state is Unlocked.

NOTE: If the server was locked prior to being shut down, its state is still Locked after the power-
up as the administration state is persisted. Unlock the server using the unlock command.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 81 OF 247
5) From the CLI Maintenance/Tools level, type healthmon. Report severity displays
“Notification”, indicating that all BroadWorks subcomponents are up and running
properly.

7.9.2 Redundant Systems

Use the following procedure to return a member of a redundant server cluster to
in-service. This procedure applies to any cluster peer (including the primary or secondary
Application Server).
1) Power up the server. The BroadWorks application starts up automatically.
2) Log in as bwadmin.
3) From the CLI Maintenance/ManagedObjects level, type get broadworks full.
4) Verify the administration state is Unlocked.

NOTE: If the server was locked prior to being shut down, its state is still Locked after the power-
up as the administration state is persisted. Unlock the server using the unlock command.

5) From the CLI System/Peering/Peers level, type get to verify the server database
state is Locked.
6) From the CLI Maintenance/Tools level, type healthmon. Assuming normal server
conditions, the report severity displays “Notification”, indicating that all BroadWorks
subcomponents are up and running properly.
7) From the CLI Maintenance/ManagedObjects level, type lock to make sure that new
calls are handled by the remaining in service cluster peer. The server transitions from
the Unlocked to Locking state before finally entering the Locked state.
8) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
that the server administration state is Locked.

9) From the CLI Maintenance/ManagedObjects level, type stop.

10) From the UNIX prompt, type $ repctl stop to stop TimesTen replication.

11) From the CLI Maintenance/Tools level, type importdb <remote server
hostaddress> to import the TimesTen database from the peer server.

NOTE: Because the BroadWorks application has been stopped, the CLI login command fails.
The Maintenance/Tools CLI level is accessible without CLI login. The importdb <remote server
hostaddress> parameter must be the same as the UNIX host name of the active server.

12) From the UNIX prompt, to start TimesTen replication, type $ repctl start.

13) From the UNIX prompt, to unlock the TimesTen database, type $ peerctl
unlock.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: The CLI login command fails. The Maintenance/ManagedObjects CLI level is
accessible without CLI login. The server can also be restarted from the UNIX prompt with the
startbw command.

15) From the CLI Maintenance/ManagedObjects level, type get broadworks full
Verify the administration state is Unlocked and the operational state is Enabled.
16) From the CLI System/Peering/Peers level, type get to verify that the server database
state is Unlocked.
17) From the CLI Maintenance/Tools level, type healthmon to verify the report severity
displays “Notification”, indicating that all BroadWorks subcomponents are up and
running properly.

7.10 Server State Monitoring Tools

7.10.1 Healthmon
The healthmon tool is available on all BroadWorks servers. You use this tool to report the
list of processes in trouble, if any, and monitor key system components. For the
Application Server and Network Server, healthmon also monitors the state of the database
and redundancy. Database out-of-sync condition detection, database replication lagging
detection, and database notification lagging detection are part of the healthmon command.

NOTE: The tool can be used in two distinct modes: Report to console or report over SNMP. In
the latter case, an SNMP trap is sent to the list of managers configured on the server. The
severity of the trap varies based on the state of the system. A trap is sent when no failure
conditions are detected. (Trap severity is NOTIFICATION.) This mechanism can be invoked
through a UNIX cron job.

MTLAS01$ healthmon –h

Usage : healthmon <-h|-v|-s|-l> [-p] [-d]

Description : This tool is used to monitor and report the status of a
BroadWorks
server. The report can either be generated locally at
the console
or sent over the network through an SNMP TRAP.
Note that a TRAP is still sent if no problems are
detected (keepalive).
Options
-h : print this usage note
-v : print the version of the script
-s : Generate a report and send the result through an SNMP
trap.
-l : Generate a local to screen.
-p : Test only the disk partitions status.
-d : Generate a detailed trace of all the tests executed.
This option is ignored in case of an SNMP report.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 83 OF 247
Some of the healthmon parameters can be manually configured by editing the
/bw/local/broadworks/bw_base/conf/[Link] file. The settings stored in this
file are preserved during an upgrade. The following table describes the supported
parameters in [Link].
Parameter Description Default Value

WarningPartitionUsage If the occupancy on any disk partition in percentage reaches 80

this threshold value, a low severity trap is generated and the
action specified by parameter WarningPartitionActions is
performed.

WarningPartition Action performed if the occupancy on any disk partition run_autocleanup,

Actions reaches the value specified by parameter minimal_logging
WarningPartitionUsage.

HighPartitionUsage If the occupancy on any disk partition in percentage reaches 90

this threshold value, a high severity trap is generated and
the action specified by parameter HighPartitionActions is
performed.

HighPartitionActions Action performed if the occupancy on any disk partition run_autocleanup,

reaches the value specified by the parameter no_logging
HighPartitionUsage.

CriticalPartitionUsage If the occupancy on any disk partition in percentage reaches 95

this threshold value, a critical severity trap is generated and
the action specified by parameter CriticalPartitionActions is
performed.

CriticalPartitionActions Action performed if the occupancy on any disk partition run_autocleanup,

reaches the value specified by parameter no_logging
CriticalPartitionUsage.

StopBwPartionUsage If the occupancy on any disk partition in percentage (except 98

/persistentLogs partition) reaches this threshold value, a
critical severity trap is generated and the action specified by
parameter StopBwPartitionActions is performed.

StopBwPartionActions Action performed if the occupancy on any disk partition run_autocleanup,

(except /persistentLogs partition) reaches the value stopbw
specified by parameter StopBwPartitionUsage.
The default value is stopping BroadWorks application to
avoid possible database corruption.

WarningPermUsage If the TimesTen permanent memory area in use reaches this 80

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.

HighPermUsage If the TimesTen permanent memory area in use reaches this 90

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.

CriticalPermUsage If the TimesTen permanent memory area in use reaches this 95

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.

WarningTempUsage If the TimesTen temporary memory area in use reaches this 80

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.

HighTempUsage If the TimesTen temporary memory area in use reaches this 90

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.

CriticalTempUsage If the TimesTen temporary memory area in use reaches this 95

threshold value (in percentage), a warning message along
with corresponding recommendation is displayed.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

StopBwPartionUsage./ If the occupancy on /persistentLogs partition in percentage 98

persistentLogs reaches this threshold value, a critical severity trap is
generated and the action specified by the parameter
StopBwPartionActions./persistentLogs performed.

StopBwPartionActions./ Action performed if the occupancy on /persistentLogs stop_replication

persistentLogs partition reaches the value specified by the parameter
StopBwPartionUsage./persistentLogs.
The default value is stopping replication.

CallLogsMaxUsageOn Threshold value in percentage. If the directory for offline 5

Partition CDR file queuing (/var/broadworks/billing/rf/) takes more
than this value of the available space on the partition where
it is mounted, file queuing is disabled.

Depending on the server type, different actions could be set for the Actions parameters in
[Link]. The following table lists the supported actions.
Action Supported Servers

minimal_logging Application Server, Network Server, Execution Server

no_logging Application Server, Network Server, Execution Server

no_billing Application Server

run_autocleanup All servers

stopbw All servers

stop_replication All servers

7.10.2 Tech-support
The tech-support tool is available on all BroadWorks servers. You use this tool to report
the server software version, BroadWorks and UNIX processes status, Internet Protocol
(IP) configuration, redundancy status, and other diagnostics.
Log in as bwadmin to access this tool. The –help option provides the command format
information.

Usage: tech-support [-help|-day <day of month>|-dailyDump]

<no option> : data dump with today's sar and gc output - recommended
-help : print this usage note
-day <day of month> : data dump with specified day's sar and gc output
-dailyDump : performs a tech-support and puts the output in
/var/broadworks/logs/monitoring
-perf : dump performance information only

7.11 Server Maintenance State

During the application of some maintenance tasks, the server goes into an internal
maintenance state. This state is used to prevent major concurrent maintenance tasks
from being run. For example, it prevents an operator from accidentally starting a server
while a new software version is being activated. The following maintenance tasks cause
the server to go into a maintenance state:
 New software version activation (upgrade)
 Patching

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

7.11.1 View Current Maintenance State

An operator can view whether or not a server is in maintenance state by running the
following command.
[Link] –getMaintState

MTLAS01$ [Link] –getMaintState

– RUNNING

7.11.2 Force Server into Maintenance State

An operator can manually force a server to go into a maintenance state by using the
following command as bwadmin. Note that this does not stop the application if it is
currently running.
[Link] –setMaintState 9

7.11.3 Recover Server in Maintenance State

A server could remain in the maintenance state due to the improper termination of a
maintenance task. This causes most maintenance tasks to fail, for example, healthmon.
If there is no ongoing maintenance and the server has remained in the maintenance state,
the state can be changed by executing the following command.
[Link] –setMaintState 7

This moves the server back into a running state.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

This section provides procedures to manage software versions. In this section all
examples provided are for the Network Server; however they also apply to BroadWorks
servers.

NOTE: The procedure to upgrade the server with a new software version is described in the
Installation and Upgrade Guide for that particular server. The final step of an upgrade procedure
can be performed from the Maintenance/ManagedObjects CLI level. Then the new version can
be activated or a previous version can be re-activated should there be a problem with the new
version.

8.1 Display Active Software Version

8.1.1 Display Full Details about Active Software Versions

From the CLI Maintenance/ManagedObjects level, type get versions current. The
following response appears (sample provided).
NS_CLI/Maintenance/ManagedObjects> get versions current
NS version Rel_17.0_1.444

Built Sat Apr 3 00:21:03 EDT 2010

- BASE revision 139841
- NS revision 139841

Patching Info:
Active Patches: 0

NOTE: BroadWorks software version information can also be viewed from the UNIX prompt as
bwadmin using the bwshowver command. For a list of all applied patches, the bwshowver
–showPatch option can be used.

8.2 List Installed Software Versions

8.2.1 List Installed and Active Software Versions

From the CLI Maintenance/ManagedObjects level, type get versions all. The
following response appears (sample provided).
NS_CLI/Maintenance/ManagedObjects> get versions all
Identity Version Install Date Status
===============================================
NS 15.0_1.290 Apr 3, 2010 Installed
NS 17.0_1.444 Apr 3, 2010 Active

2 entries found.

* Applications:
Name Version Install Date Upgrade Mode
Description Status
================================================================================
================================
NSExecutionAndProvisioning 17.0_1.444 Apr 3, 2010 Automatic Provides
translations and routing. Active

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 87 OF 247
NSPortal 17.0_1.444 Apr 3, 2010 Automatic
NSPortal web portal Active
WebContainer 17.0_1.444 Apr 3, 2010 Automatic
Web Container Active

3 entries found.

* Third Party Software:

Third Party Version Status
======================================
java xs_java_base active
java jdk1.6.0_06 installed
java jdk1.6.0_14 active
apache 2.2.6 installed
apache 2.2.10b active
timesten [Link].0 installed
timesten [Link].0 active
tomcat 6.0.18b active
tomcat 6.0.14 installed

9 entries found.

8.3 Switch Active Software Version

The action of switching the active software version results in an upgrade or a rollback.
This action is performed using the set command in the Maintenance/ManagedObjects
CLI level.
BroadSoft suggests that you lock the server before restarting and switching the software
version.

NOTE: Restarting a server results in that server refusing any new incoming requests until the
server restarts.

1) From the CLI Maintenance/ManagedObjects level, type lock to (traffic) lock the
server. This ensures the remaining service handles all new calls. The server
transitions from the Unlocked to Locking state before finally entering the Locked state.
2) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
the server administration state is Locked.

3) From CLI Maintenance/ManagedObjects level, type set

activeSoftwareVersion server NS 17.0_1.444. The following response
appears (sample provided).
NS_CLI/Maintenance/ManagedObjects> set activeSoftwareVersion server NS
17.0_1.444

+++ WARNING +++ WARNING +++ WARNING +++

This command will change the active software version of NS to
17.0_1.444. NOTE that this action will cause downtime.
Continue?

Please confirm (Yes, Y, No, N):

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 88 OF 247
NOTE: Additional logs for the server restart and software version switch are written to file
/var/broadworks/logs/maintenance/setactiveserver_XXXXXXXXXXXX.log.

4) Exit the CLI after the activation of the new BroadWorks software version. This is
mandatory before issuing new CLI commands.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

There are certain tasks that can be scheduled on all BroadWorks servers. These tasks
are listed in the following tables.
Applicable To
Schedule
Access Application Database Element Execution Media
Tasks Mediation Server Server Management Server Server
Server System

AutoCleanup Weekly Weekly Weekly Weekly Weekly Weekly

Healthmon 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes

Backup Weekly Weekly Weekly Weekly Weekly Weekly

DbMaint N/A Weekly N/A Weekly N/A N/A

RegAudit N/A Optional N/A N/A N/A N/A

DbSyncCheck Not applicable Daily N/A N/A N/A N/A

CpuMon 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes

Tech-support Daily Daily Daily Daily Daily Daily

Check_dbpages N/A Daily N/A N/A N/A N/A

Service License N/A N/A N/A N/A

N/A Optional
Collect

File Collector Optional Optional N/A Optional Optional N/A

Applicable To
Schedule Network Network Profile Service Messaging Sharing Video
Tasks Server Function Server Control Server Server Server
Manager Function
Server

AutoCleanup Weekly Weekly Weekly Weekly Weekly Weekly Weekly

Healthmon 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes 30 minutes

Backup Weekly Weekly Weekly Weekly Weekly Weekly Weekly

DbMaint Weekly N/A N/A N/A Weekly N/A N/A

RegAudit N/A N/A N/A N/A N/A N/A N/A

DbSyncCheck Daily N/A N/A N/A Daily N/A N/A

CpuMon 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes 5 minutes

Tech-support Daily Daily Daily Daily Daily Daily Daily

Check_dbpages Daily N/A N/A N/A Daily N/A N/A

Service License N/A N/A N/A N/A N/A N/A N/A

Collect

File Collector Optional Optional Optional Optional Optional Optional Optional

Applicable To
Schedule
WebRTC Xtended
Tasks Server Services
Platform

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Healthmon 30 minutes 30 minutes

Backup Weekly Weekly

DbMaint N/A N/A

RegAudit N/A N/A

DbSyncCheck N/A N/A

CpuMon 5 minutes 5 minutes

Tech-support Daily Daily

Check_dbpages N/A N/A

Service License N/A N/A

Collect

File Collector Optional Optional

You can schedule tasks to run on a specific day of the month, a specific day of the week,
or every day at some specific hour or frequency.

AS_CLI/Maintenance/Scheduler> h add
The ADD command is used to add a new scheduled task.

A task can be configured to be executed on a minute, day or date basis.

Any given task can have multiple scheduled entries.
Following are examples:
"taskx minute 2" means execute taskx every 2 minutes
"taskx day monday 0 30" means execute taskx every monday at 0h30
"taskx date 15 2 15" means execute taskx on the 15 of each month at
2h15
======================================================================
add
<tasks>, Choice = { dbMaint, healthmon, backup, autoCleanup,
regAudit, dbSyncCheck, cpuMon, tech-support, check_dbpages,
serviceLicenseCollect, fileCollector }
<frequency>, Choice = {minute, daily, day, date}
minute:
<value>, Choice = {1, 2, 3, 4, 5, 6, 10, 12, 15, 30, 60, 120,
180…1440}
daily:
<hour>, Integer {0 to 23}
<minute>, Integer {0 to 59}
day:
<day>, Choice = {monday, tuesday, wednesday, thursday, friday,
saturday, sunday}
<hour>, Integer {0 to 23}
<minute>, Integer {0 to 59}
date:
<date>, Integer {1 to 28}
<hour>, Integer {0 to 23}
<minute>, Integer {0 to 59}

WARNING: Tasks should not be scheduled to run at the same time. Running all tasks at the
same time may affect server performance.

NOTE 1: Tasks should set to run at off peak hours.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 91 OF 247
NOTE 2: To minimize the impact of running concurrent scheduled tasks, it is recommended to
use minutes that are different between tasks, and those that do not match the hour (when
applicable).

9.1 Automatic Backups

You use an automatic backup to schedule the backup of the TimesTen database and
other relevant file system directories. The backups are listed in the following tables.
Server

Description Access Application Database Element Execution Media

Mediation Server Server Management Server Server
Server System

Database X X X

/var/broadworks/
X
userfiles

/usr/local/broad
works/bw_base/ X X X X X X
conf/*

/var/TimesTen/
X
[Link]*

Web
applications

Server

Network Network Profile Service Messaging Sharing Video

Description
Server Function Server Control Server Server Server
Manager Function
Server

Database X X X

/var/broadworks/
userfiles

/usr/local/broad
works/bw_base/ X X X X X X X
conf/*

/var/TimesTen/
X X
[Link]*

Web applications X

Server

Description WebRTC Xtended

Server Services
Platform

Database

/var/broadworks/
userfiles

/usr/local/broad
works/bw_base/ X X
conf/*

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Description WebRTC Xtended

Server Services
Platform

/var/TimesTen/
[Link]*

Web applications X

9.1.1 Mode of Operation

Execute backups as a UNIX cron job. Schedules can be controlled either with the
[Link] script or from the CLI. The content of what is to be backed up on each
server is defined in /usr/local/broadworks/bw_base/conf/[Link].
At the scheduled time, the server creates a backup of all relevant information, compresses
it into one file, and places it in the following directory.
/usr/local/broadworks/bw_base/persistent/backups

The script deletes all old backup files. Backups should be transferred off the server for
safekeeping.

NOTE 1: Schedule backups to run on off-peak hours.

NOTE 2: Stagger backups across cluster peer members so that they do not run at the same
time on the primary and secondary Application Servers.

To configure automatic backups to run once a day at 3:30 A.M., from the CLI
Maintenance/Scheduler level, type add backup daily 3 30.

9.1.2 Auto-Backup File Naming Convention

An automatic backup file is a gzipped tar file called AutoBackupFileName, which is a
long name formatted as follows:
<YYYYdddHHmmSS>-<YYYYMMDD>-<random integer>-<hostname>-<hostid>

… where:
YYYY is the year
ddd is the day of year
HH is the hour of day
mm is the minute
SS is the second
MM is the month
DD is the day of month

Similarly, the database backup in the automatic backup file is called

DatabaseBackupFileName, which is a long name formatted as follows.
<YYYYdddHHmmSS>-<YYYYMMDD>-<random integer>-<hostname>-<hostid>-database

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 93 OF 247
… where:
YYYY is the year
ddd is the day of year
HH is the hour of day
mm is the minute
SS is the second
MM is the month
DD is the day of month

9.1.3 Restore Auto-Backup

All files stored in the backup image are stored using relative paths. You can restore all
files or specific files using the following procedure.
Backup files only contain user data and general configuration files, not the entire contents
of the BroadWorks installation directory. If the server host to be restored no longer has
BroadWorks installed, you must first re-install the appropriate BroadWorks components,
and re-apply all BroadWorks patches that were in place when the backup was made.
Continue with the following procedure to restore the backup files.
Because the backup .tar file stores the relative paths of all included files, you need to
extract the archive (restore the server) from an appropriate directory. If you want to
restore files to the directory in which they were originally located when the backup was
made, perform the restore from the system root directory (/). Otherwise, the backup files
are simply extracted relative to the directory from which you run the tar xvf command.
Note that the restore operation should be executed as root when performed from the
system root directory (/).
1) The image must first be uncompressed using the gzip utility from the unzip prompt:
$ gunzip <backupName>.[Link]
2) The content of the backup image can be viewed using the tar utility:
$ tar tvf <backupName>.tar
3) Stop BroadWorks, the SNMP Agent, and the Configuration Agent:
$ stopbw
$ snmpdctl stop
$ lmdctl stop
$ configdctl stop
4) Switch to the root account when restoring from the system root directory (/).
5) A specific file can be extracted by executing the tar command with the x option and by
specifying a specific file name (long name as stored in the backup):
6) $ tar xvf <backupName>.tar <fileToExtract>
Start the Configuration Agent, the SNMP Agent, and BroadWorks:
$ configdctl start
$ lmdctl start
$ snmpdctl start
$ startbw

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

bwadmin@MTLNS01$ gunzip 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

[Link]

bwadmin@MTLNS01$ ls
2006010082501-20060110-5294-NS_Rel_14.0_1.[Link]

bwadmin@MTLNS01$ tar tvf 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

[Link]

bwadmin@MTLNS01$ tar tvf 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

[Link] | grep persistent
-rw-rw-r-- 118/100 7804344 Jan 10 08:25 2006
usr/local/broadworks/bw_base/persistent/backups/2006010082501-20060110-
11469-MTLNS01-80fcc530-database

bwadmin@MTLNS01$ stopbw

bwadmin@MTLNS01$ snmpdctl stop

bwadmin@MTLNS01$ lmdctl stop

bwadmin@MTLNS01$ configdctl stop

bwadmin@MTLNS01$ tar xvf 2006010082501-20060110-5294-NS_Rel_14.0_1.240-

[Link]
usr/local/broadworks/bw_base/persistent/backups/2006010082501-20060110-
11469-MTLNS01-80fcc530-database
x usr/local/broadworks/bw_base/persistent/backups/2006010082501-20060110-
11469-MTLNS01-80fcc530-database, 7804344 bytes, 15243 tape blocks

bwadmin@MTLNS01$ ls
2006010082501-20060110-5294-NS_Rel_14.0_1.[Link]
usr/

bwadmin@MTLNS01$ ls -R usr
usr:
local/

usr/local:
broadworks/

usr/local/broadworks:
bw_base/

usr/local/broadworks/bw_base:
persistent/

usr/local/broadworks/bw_base/persistent:
backups/

usr/local/broadworks/bw_base/persistent/backups:
2006010082501-20060110-11469-MTLNS01-80fcc530-database

bwadmin@MTLNS01$ configdctl start

bwadmin@MTLNS01$ lmdctl start

bwadmin@MTLNS01$ snmpdctl start

bwadmin@MTLNS01$ startbw

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 95 OF 247
9.2 Automatic Disk Cleanup
Use automatic disk cleanup to schedule the automatic cleanup of specific directories on
the server to help manage disk space usage.

Directories Cleaned Up Action

/tmp All files older than two (2) days are deleted.

/var/tomcat/logs All files older than thirty (30) days are deleted.

/var/broadworks/logs/* All files older than thirty (30) days are deleted.
All files older than two (2) days are compressed.

NOTE: For the directory /var/broadworks/logs/*, the action applies to all directories under
/var/broadworks/logs.

Backups are executed as a UNIX cron job. Control the schedule with the
[Link] script or from the CLI.
For example, to set the cleanup to run once a week on Sunday at 4:30 A.M., from the CLI
Maintenance/Scheduler level, type add autoCleanup day sunday 4 30.
When the AutoCleanup task is run on a Database Server, the following files for Oracle
logs, traces, and alerts under the /u01/app/oracle/diag directory structure are compressed
(in addition to the log files under the /var/broadworks/logs directory):
 Audit Dump
 Database Instance traces
 ASM Instance traces
 TNS Listener traces
 TNS Listener alerts
 SCAN Listener traces
 SCAN Listener alerts

9.3 Healthmon Keep Alives

The healthmon tool is available on all BroadWorks servers. You use this tool to report the
list of processes in trouble, if any, and monitor key system components. For the
Application Server and Network Server, healthmon also monitors the state of the database
and redundancy. Database out-of-sync condition detection, database replication lagging
detection, and database notification lagging detection are part of the healthmon command.
Configure healthmon to run periodically with the SNMP Keep Alive functionality. The
severity of the trap is based on the state of the system. If failure conditions are not
detected, trap severity is a Notification. Invoke this mechanism through a UNIX cron job.
To configure healthmon to run every 15 minutes, from the CLI Maintenance/Scheduler
level, type add healthmon minute 15.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 96 OF 247
9.4 Database Statistics Update
The [Link] script is configured to run once a day at 2:15 A.M. This script
optimizes database performance and it should not be disabled. The time the script runs
can be altered to off-peak network hours.

WARNING 1: If you disable the [Link] task from the scheduler, the database may
exhibit performance issues.

WARNING 2: When the [Link] script is running, it may prevent provisioning from
working. Make sure the script is scheduled to run outside of the busy hours.

WARNING 3: The time it takes to execute the [Link] depends on the size of the
database and the hardware on which the application works. The execution can take a few
seconds up to a few minutes.

WARNING 4: This task also runs the check_dbpages task; therefore, both tasks should not be
scheduled at the same time.

9.5 Registration Audit

The RegAudit task is only available on the Application Server. It is responsible for auditing
the SIP registrations in the BroadWorks database. It automatically cleans up SIP
registrations, which have expired for more than a pre-defined period (configured through
the CLI extensionTimeInSeconds option under AS_CLI/System/ Registration).

9.6 Database Synchronization Check

The DbSyncCheck task is available on the Application Server and Network Server. It is
responsible for running the syncheck_basic.pl tool, which provides a lightweight
mechanism to determine if two databases are synchronized. For more details, see
section 11.1 Application Server, Network Server, and Messaging Server Database
Replication.

9.7 CPU Monitoring Tool

The CpuMon tool is available on all BroadWorks servers. CpuMon is responsible for
monitoring CPU idle percentage, and memory usage. It automatically generates an
SNMP trap with a severity level dependent on the configurable idle time threshold that has
been surpassed. This script uses either SAR (if installed) or vmstat over a one-minute
period.

9.8 Check DB Pages

The check_dbpages.pl tool is available on the Application Server and the Network Server.
This tool calculates the page size, detects pages that have exceeded their capacity, and
increases the capacity of the tables when needed. This ensures that provisioning
activities are not impacted by performance degradation. It should be configured to run
daily.

9.9 Tech-support
Run the tech-support maintenance task on a daily basis. The output of the tech-support
daily task is stored in the /var/broadworks/logs/monitoring directory.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 97 OF 247
9.10 Service License Collect
The ServiceLicenseCollect task gathers information about the current licensing and actual
usage of the various BroadWorks services. This information is stored in the:
/var/broadworks/serviceLicenseReporting/serviceLicenseCollect_(host)_(date).zip file.
This file is interpreted by the Service License Report Tool installed by default on the
Application Server.

9.11 Delete Scheduled Tasks

Scheduled tasks cannot be modified. To change the time that a task is configured to run,
delete the task, and then add a new task with the new time.
To delete a scheduled task:
1) From the CLI Maintenance/Scheduler level, type get. The following response
appears (sample provided).

Id Name Date Day Hour Minute

===========================================================
1 healthmon - - - 15
2 dbMaint - sunday 2 33
3 backup - - 4 30

2) Type del <task ID>. The following response appears (sample provided).
...Done

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

BroadWorks uses TimesTen on the Application Server, Network Server, and Messaging
Server, MySQL on the Element Management System, Oracle on the Database Server,
and MariaDB on the Network Function Manager.
All databases are delivered with Structured Query Language (SQL) clients and a set of
maintenance tools. In general, BroadSoft recommends the use of the commands
described in this document. All database-related issues or questions should be directed to
the BroadSoft Support line.
Maintenance procedures for Oracle are slightly different from those used for TimesTen,
MariaDB and MySQL. For more information on Database Server maintenance
procedures, see the BroadWorks Database Server Configuration Guide.

10.1 Database Backup

Backups can be performed with no impact to the system (for example, it is not required to
stop the BroadWorks application).

10.1.1 Manual Database Backup

1) Create a directory where the backups are stored.
Example:
$ mkdir /var/dbbackup/Rel19.0/

2) To run the backup, type $ [Link] <dsn> <file>. DSN is the datastore
name (for example, NetworkServer, AppServer, ElementManagementSystem, NFM)
and file is the file name for the backup (including the path).
Example:
Hostname$ [Link] NetworkServer /var/dbbackup/Rel19.0/[Link]

The database can also be manually backed up from the CLI Maintenance/Tools level.
The DSN (datastore name) is not required.
NS_CLI/Maintenance/Tools> ?
0) backupdb : Backups the content of the BroadWorks TimesTen datastore
1) healthmon : Reports the status of the system
2) importdb : Imports the content of a remote TimesTen database
3) restoredb : Restores the content of the BroadWorks TimesTen datastore
4) tech-support : Captures the status of the system for troubleshooting
5) upgradeCheck : Performs pre-upgrade validation

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re (repeat)

NS_CLI/Maintenance/Tools> backupdb /var/broadworks/backup/[Link]

[Link] version 15.0.0
->Validating specified parameters... [done]
->Backing up dsn NetworkServer to /var/broadworks/backup/[Link]... [
in progress]
->Ensuring database is loaded in memory... [done]
->Backing up... [done]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 99 OF 247
10.1.2 Backup Database for Application Server, Element Management System, Call
Center Reporting Server, Network Server, Network Function Manager, or
Messaging Server
1) Create a backup directory.
2) From the UNIX prompt (or from the CLI Maintenance/Tools level), type
[Link].
3) Transfer the backup to some other storage facility for safekeeping.

10.2 Database Restore

WARNING 1: This procedure overwrites the Application Server, Element Management System,
Network Server, Network Function Manager, or Messaging Server database with the backed-up
version. The existing data is lost.

WARNING 2: The server shall be stopped (stopbw) prior to attempt to restore a backup.

To restore the database, an existing backup must be present on the server. The restore
fails if there are processes attached to the database. Such processes can include
BroadWorks, replication daemons, and TimesTen (ttIsql) or MySQL database client tools.

10.2.1 Restore Database

Type Hostname$ [Link] <dsn> <file>. DSN is the datastore name and
where the file is the file name of the backup (including the path). The database can also
be restored from the CLI Maintenance/Tools level. DSN (datastore name) is not required.
NS_CLI/Maintenance/Tools> ?
0) backupdb : Backups the content of the BroadWorks TimesTen datastore
1) healthmon : Reports the status of the system
2) importdb : Imports the content of a remote TimesTen database
3) restoredb : Restores the content of the BroadWorks TimesTen datastore
4) tech-support : Captures the status of the system for troubleshooting
5) upgradeCheck : Performs pre-upgrade validation

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re (repeat)

NS_CLI/Maintenance/Tools> restoredb /var/broadworks/backup/[Link]

[Link] version 15.0.0

WARNING: ALL information in data store NetworkServer will be lost.

->Restoring dsn NetworkServer from /var/broadworks/backup/[Link]... [

in progress]
->Unloading database from memory... [done]
->Destroying store... [done]
->Restoring database from backup file... [done]
->Loading database in memory... [done]
->Updating statistics... [done]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 100 OF 247
10.2.2 Restore Database for Application Server, Element Management System, Call
Center Reporting Server, Network Server, Network Function Manager, or
Messaging Server
1) From the CLI Maintenance/ManagedObjects level, type lock. This ensures that the
remaining service handles new calls. The server transitions from the Unlocked to
Locking state before it enters the Locked state.
2) From the CLI Maintenance/ManagedObjects level, type get broadworks to verify
the server administration state is Locked.

NOTE: Terminate all active sessions to allow the server to make the transition to the Locked
state. Use the lock force command form the CLI Maintenance/ManagedObjects level to force
the administration state to Locked.

3) From the UNIX prompt, type $ stopbw to stop the BroadWorks server.
4) If the server is part of a redundant cluster, stop the replication (from the UNIX prompt,
type $ repctl stop).
5) Restore the desired database (using the [Link] command from the UNIX
prompt or from the CLI Maintenance/Tools level).
Example:
$ [Link] NetworkServer /var/dbbackup/Rel10.0/[Link]

6) If the server is part of a redundant cluster, restart replication (from the UNIX prompt,
type $ repctl start).
7) Restart the BroadWorks Server (from the UNIX prompt, type $ startbw).

NOTE: For a redundant cluster, restoring a database on just one peer results in a database out-
of-sync condition. A database import is required on the other peer to make sure a return to
database in-sync state.

10.2.3 Extract Database Backup from Automatic Backup File

1) Go to the directory with the automatic backup files. By default, this directory is
/usr/local/broadworks/bw_base/persistent/backups. The following steps assume this
is the directory where the automatic backup files are stored.
Identify the automatic backup file to extract the database backup. An automatic
backup file is a gzipped tar file called AutoBackupFileName, which is a long name
formatted as described in section 9.1.2 Auto-Backup File Naming Convention.
2) Unzip the automatic backup file. From the UNIX prompt, type $ gunzip
[Link].
3) The resulting tar file contains all the files that were automatically backed up. If -
database backup was enabled, the tar file contains the database backup file. The tar
file can also contain other files that had to be backed up. To identify the database
backup file from all the other files in the tar, type the following from the UNIX prompt.
$ tar tvf [Link]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 101 OF 247
If present, the database backup file (ending with -database) is the last file in the list.
Its full path name would be as follows.
$ /usr/local/broadworks/bw_base/persistent/backups/DatabaseBackupFileName

4) If you want to restore the backup files to their original directories, change your working
directory to the system root directory (/). Otherwise, the backup files are simply
extracted relative to the directory from which you run the tar xvf command. Type:
$ cd /

5) To extract the database backup file from the automatic backup tar file, from the UNIX
prompt, type:
$ tar xvfP [Link]
/usr/local/broadworks/bw_base/persistent/backups/DatabaseBackupFileName

6) The database backup should now be in the directory, ready to be restored by

following the instructions in section 10.2 Database Restore.

10.3 Database Import

WARNING: This procedure overwrites the Application Server, Element Management System,
Network Server, Network Function Manager, or Messaging Server database with the backed-up
version. The existing data is lost.

A database can be imported from a cluster peer member. This can be used to make sure
that the databases across all peers are in-sync.
The import fails if there are processes attached to the database. Such processes can
include BroadWorks, replication daemons, and TimesTen (ttIsql) or MySQL client
tools. The import requires that the TimesTen or MySQL replication daemon is running on
the peer server.
The following command can be used to import a database.
$ [Link] [-noWarning] <local dsn> <remote host address> <remote dsn>

… where the local/remote DSN is the datastore name (AppServer,

ElementManagementSystem, NetworkServer, NFM, or UMS) and the remote host
address is the address of the peer server from which you are importing (as set in the
peerctl list).
For example, if you are importing from as1 to as2 and the peerctl list is as follows.
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
as1/[Link] primary, unlocked
as2/[Link] unlocked

...the remote host name should be set to “[Link]”. It is always the peer’s peerctl
address.

NOTE: The loghost must be associated with this peer in the /etc/hosts file.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 102 OF 247
The database can also be restored from the CLI Maintenance/Tools level. DSN
(datastore name) is not required.
NS_CLI/Maintenance/Tools> ?
0) backupdb : Backups the content of the BroadWorks TimesTen datastore
1) healthmon : Reports the status of the system
2) importdb : Imports the content of a remote TimesTen database
3) restoredb : Restores the content of the BroadWorks TimesTen datastore
4) tech-support : Captures the status of the system for troubleshooting
5) upgradeCheck : Performs pre-upgrade validation

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re (repeat)

NS_CLI/Maintenance/Tools> importdb mtl64lin17

-> Making safety backup <-
Backing up dsn NetworkServer to /var/broadworks/backup/NS_Rel_14.0_1.538.0607
[Link]... [done]
-> Importing data from primary server <-
Retrieving localhost in remote replication schema... [[Link]
[Link]]
Performing checkpoint on remote database... [done]
Unloading database from memory... [done]
Destroying database NetworkServer... [done]
Importing the remote database... [done]

10.3.1 Import Database on Application Server, Element Management System, Network

Server, Network Function Manager, or Messaging Server
1) Traffic lock the server (from the CLI, go to Maintenance/ManagedObjects and type
lock). This makes sure that all new calls are handled by the remaining in service
cluster peer.
2) The server transitions from the Unlocked to Locking state before finally entering the
Locked state. Before proceeding, from the CLI Maintenance/ManagedObjects level,
verify that the server administrative state is Locked by typing get broadworks.

3) Stop the BroadWorks Server (from the UNIX prompt type $ stopbw).
4) Stop the replication, if the server is part of a redundant cluster (from the UNIX prompt
type $ repctl stop).
5) Import the database from the cluster peer (using the [Link] command from the
UNIX prompt or from the CLI Maintenance/Tools level). Example:
$ [Link] NetworkServer [Link] NetworkServer

6) Restart the replication, if the server is part of a redundant cluster (from the UNIX
prompt, type $ repctl start).
7) Restart the BroadWorks server (from the UNIX prompt, type $ startbw).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

DANGER: This procedure destroys the existing Application Server, Element Management
System, Network Server, or Messaging Server database and installs a fresh database. This
command should NEVER be run on a production server unless under the direct supervision of
BroadSoft Support.

NOTE: For a redundant cluster, restoring a database on just one peer results in a database
out-of-sync condition. A database import is required on the other peer to ensure a return to
database in-sync state.

10.4.1 Recreate Application Server, Network Server, or Messaging Server Database

It is possible to force an installation or reinstallation of the TimesTen datastore. The first
option is to force an installation when an upgrade was chosen during a BroadWorks
installation. This destroys the current datastore and it installs a fresh, empty data schema,
which can be useful in a lab environment.
 Network Server: Hostname$ [Link] NS_Rel_18.0_1.xxx –install
 Application Server: Hostname$ [Link] AS_Rel_18.0_1.xxx –install
 Messaging Server: Hostname$ [Link] UMS_Rel_20.0_1.xxx –install
If you only need to revert to the datastore as it was right after the installation, or if it was not
installed, you would issue the following commands:
 Network Server: Hostname$ [Link] NS_Rel_16.0_1.xxx –install
 Application Server: Hostname$ [Link] AS_Rel_16.0_1.xxx –install
 Messaging Server: Hostname$ [Link] UMS_Rel_20.0_1.xxx –install
A fresh install fails when there are processes attached to the database. Such processes
can include BroadWorks, replication daemons, and TimesTen tools, such as ttIsql.
bwadmin@ns1$ [Link] NS_Rel_16.0_1.500 -install

---- BroadWorks database setup utility version 16.0 ----

=> Gathering data <=

in progress [....

********************************************************************
*
WARNING - WARNING - WARNING - WARNING - WARNING - WARNING

This command can only be run when the server is in Maintenance

state.

The [Link] utility is meant to be invoked by internal

BroadWorks commands. It should be used with caution since it could
cause a loss of all data contained in the server database.

Please refer to BroadWorks personnel before directly invoking that

that command.

********************************************************************
*

........]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 104 OF 247
=> Checking for perm size usage <=
Perm Size Usage = 6% (limit is at 55%)
=> Checking for available disk space <=
Perm = 128 Temp = 42
Space required [16 Mb]
Checking on /bw... [done]
=> Creating bw_base link... [done]
=> Settings <=
Target BroadWorks version [NS_Rel_16.0_1.500]
Data store name [networkserver]
TimesTen version [[Link].0]
Force installation [true]
Installation type [fresh install]
Redundant System [false]
Current maintenance... [NS_Rel_16.0_1.500]
=> Installing the latest data schema <=
Creating tables... [done]
Loading basic data... [done]
=> Now configuring replication <=
Adding ns1/ns1... [done]
Setting primary peer (ns1)... [done]
Setting replication port (17888)... [done]
=> Setting IMS mode options <=
setting IMS mode to false... [done]
=> Setting platform mode options <=
=> Checking crontab for DB periodic maintenance <=
Crontab entry already present
=> Verifying database paging <=
=> Ensuring database is loaded in memory <=
Current maintenance... [NS_Rel_16.0_1.500]
Setting ram policy to always [done]
==== BroadWorks database setup [DONE] ====

[Link] Reinstall Database on Network Server, Application Server, or Messaging Server

1) Stop the BroadWorks Server (from the UNIX prompt type $ stopbw).
2) Stop TimesTen replication if the server is part of a redundant cluster (from the UNIX
prompt, type $ repctl stop).
3) Either reinstall the database, a fresh install or revert (using the [Link]
command from the UNIX prompt). For example,
$ ./[Link] NS_Rel_16.0_1.500 –install.
4) On the Network Server only, reinstall the desired dial plans (from the UNIX prompt,
type [Link] <NS version>).
5) If the server is part of a redundant cluster, then restart TimesTen replication (from the
UNIX prompt, type $ repctl start).
6) Restart the BroadWorks server (from the UNIX prompt, type $ startbw).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

10.5.1 Change TimesTen Database Size

WARNING 1: This procedure can be used to increase or decrease the database size and
should only be run with the assistance of BroadSoft Support personnel.

WARNING 2: During this procedure, the servers are locked, preventing provisioning changes
and device registration.

An operator shall use the resizeDSN tool packaged with the BroadWorks server to change
the database size. This following procedure provides the steps required to change the
database store name (DSN) size on an Application Server, a Network Server, or a
Messaging Server cluster. Always start the procedure on the primary server and then
execute it on the secondary servers. The procedure must be executed on one server at a
time.
The following step must ONLY be run on the primary server:
1) Lock all servers in the cluster ($peerctl lock all).
The remainder of the procedure must be executed on all servers in the cluster:
2) Stop BroadWorks on the server ($stopbw).
3) Stop replication on the server ($repctl stop).
4) Unload the database from memory ($[Link] unload).
5) Log in as root ($su).
6) Go to the BroadWorks bin directory (#cd
/usr/local/broadworks/bw_base/bin).
7) Run the resizeDSN command (#./resizeDSN).
8) Come back as bwadmin (#exit).
9) Restart replication on the server ($repctl start).
10) Unlock the server ($peerctl unlock).

11) Restart BroadWorks on the server ($startbw).

10.5.2 Update TimesTen Database Statistics

NOTE: This procedure automatically runs weekly and should not run manually unless under the
supervision of BroadSoft Support personnel. Running this procedure can affect performance.

To set the database to update statistics:

Run /usr/local/broadworks/bw_base/[Link].

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

WARNING: The TimesTen database is started automatically upon server power up or reboot.
The database should not be manually stopped unless under direct supervision from BroadSoft
Support.

The TimesTen database daemon is controlled through the following command.

/etc/init.d/tt_daemon[stop | start]

The database cannot be stopped if processes are attached to the database. Such
processes can include BroadWorks, replication daemons, and TimesTen tools like
ttIsql.
To stop the TimesTen database daemon, run the following commands:
1) Stop the BroadWorks server (from the UNIX prompt type $ stopbw).
2) Stop TimesTen replication if server is part of a redundant cluster (from the UNIX
prompt type $ repctl stop).
3) Log in (su -) as root and run the following commands:
# cd /etc/init.d/
# ./tt daemon stop

10.5.4 Update TimesTen Database Paging Information

The database paging information must be re-calculated and re-optimized occasionally.
When the paging is not properly set on a server, major performance degradations may
occur.
The healthmon command periodically runs the check_dbpages.pl script using the detect
option to verify that the paging information is properly set. Should the check_dbpages.pl
script report an issue, an operator may have to run the same command with the modify
option to perform an optimization. The check_dbpages.pl script modifies only the
Application Server on the current host; therefore, it should be run against each redundant
Application Server that reports the same issue.

NOTE: It is recommended to run the check_dbpages.pl with the modify option during hours of
low traffic volume, as it temporarily locks the database.

10.5.5 Compact the Database

Over time (after months of operation), it is possible that data in the database becomes
fragmented causing the PERM size in use to be more than it should. This can also have
an impact on the performance of the database. When the datastore in use becomes close
to the limit of the DSN, it is either recommended to try to compress it (increasing the
datastore may not be possible because there is restriction on the DSN size base on the
available RAM on a server). The following is the procedure for performing the
consolidation of the TimesTen database. Note that the primary server must always be
done first.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 107 OF 247
[Link] Compact the Primary Server
1) Stop BroadWorks ($stopbw).
2) Lock all peers ($peerctl lock all).
3) Stop replication on all peers ($peercmd “repctl stop”).
4) Back up the database ($[Link]).
5) Restore the database ($[Link]).
6) Restart replication ($repctl start).
7) Unlock the primary server ($peerctl unlock).
8) Restart BroadWorks ($startbw).

[Link] Compact Backup Servers

1) Stop BroadWorks ($stopbw).
2) Import the database from the primary server ($[Link]).
3) Restart replication ($repctl start).
4) Unlock the server ($peerctl unlock).
5) Restart BroadWorks ($startbw).

10.5.6 Import TimesTen Datastores between Different Operating Systems

This section provides a procedure to migrate the Application Server and Network Server
database from Linux to Solaris or Solaris to Linux. Since the images obtained through
[Link] are not compatible between the Solaris and Linux operating systems, one
way to bypass the problem is by using a remote connection of the source database from
the destination datastore.
SOURCE_NODE = This is the box you are copying from.
TARGET_NODE = This is the box you are copying to.

NOTE 1: This procedure assumes that BroadWorks is installed on the TARGET_NODE. It is

important that both the SOURCE_NODE and the TARGET_NODE are at the exact same
release and patch level.

NOTE 2: There is no way to take a backup image obtained through [Link] and restore it
directly on a server that has an operating system different from that of the server from which the
backup was obtained.

1) Stop BroadWorks where TARGET_NODE is running (from the UNIX prompt type:
stopbw).
2) Stop BroadWorks on the SOURCE_NODE (from the UNIX prompt type: stopbw).
3) Stop the replication on the SOURCE_NODE (from the UNIX prompt type: repctl
stop).
4) From the TARGET_NODE, execute the following command.
ttMigrateCS -c -v0 –connStr "TTC_Server=<SOURCE_NODE>;
TTC_Server_DSN=<DSN>; TCP_Port=<PORT>;UID=bw;PWD=bw"

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Where
SOURCE_NODE is the address of the source node
DSN is AppServer or NetworkServer depending on the BroadWorks
server type
PORT depends on the TT version, 16003 (can be obtained from
ttStatus)

5) Restore the backup on the TARGET_NODE.

[Link] <DSN> /var/broadworks/backup/remote_db_backup.ttm –migrate

6) Change the host name in the database using peerctl (from the UNIX prompt type
peerctl set).
7) In case the SOURCE_NODE was in a redundant configuration, from ttIsql on the
TARGET_NODE, drop the previous replication schema (from ttIsql, do drop
replication [Link];).
Once the procedure is completed, it is recommended to perform the following steps on the
SOURCE_NODE:
1) Import the database from its peer (from the UNIX prompt: [Link]).
2) Start the replication (from the UNIX prompt: repctl start).
3) Start BroadWorks (from the UNIX prompt: startbw).
Following is a sample execution of the procedure described in this sub-section.
bwadmin@mtlsoldev1$ ttMigrateCS -c -v0 -connStr "TTC_Server=mtl64lin16;
TTC_Server_DSN=AppServer; TCP_Port=16003"
/var/broadworks/backup/remote_db_backup.ttm
bwadmin@mtlsoldev1$ [Link] AppServer
/var/broadworks/backup/remote_db_backup.ttm -migrate
[Link] version 14.2.0

WARNING: ALL information in data store AppServer will be lost.

->Restoring from backup with migrate option...
Press ctrl-c now to abort or <enter> to continue...

->Restoring dsn AppServer from

/var/broadworks/backup/remote_db_backup.ttm... [in progress]
->Unloading database from memory... [done]
->Destroying store... [done]
->Restoring database from backup file... [done]
->Loading database in memory... [done]
->Updating statistics... [done]
bwadmin@mtlsoldev1$ peerctl ls
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
---------
[Link]/[Link]
primary,unlocked
[Link]/[Link]
unlocked
bwadmin@mtlsoldev1$ peerctl del [Link]
peerbwadmin@mtlsoldev1$ peerctl set [Link]
mtlsoldev1 mtlsoldev1
bwadmin@mtlsoldev1$ peerctl ls
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
---------

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

10.5.7 TimesTen Connection Optimization

NOTE: The following change is required on boxes with more than 32 CPUs, in other words, on
T2000 boxes. It should be implemented on these boxes across the board.

On certain servers that are large, the default number of connections provided to TimesTen
is not enough. This information can be retrieved by running the tech-support script and
looking at the TimesTen Server information and status section in the following example.
This section provides information about TimesTen including the number of connections.
Example:
TimesTen status report as of Sun Oct 16 10:05:20 2011

Daemon pid 1842 port 11284 instance [Link].4

In the example, TimesTen is using 48 connections. The default number of connections

provided to TimesTen is 256. If you reach that limit, you can increase the number of
connections by performing the following steps:
1) Edit the /var/TimesTen/[Link] file and change all occurrences of
“Connections=256” to a higher value. For example, you can set a value of “300”.
2) To make this change effective, you must perform the following:
− stopbw
− repctl stop
− [Link] unload
− [Link] load
− repctl start
− startbw

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

10.6 Additional MySQL Maintenance Tasks

10.6.1 Update MySQL Database Statistics

NOTE: This procedure automatically runs every night at 2:15 A.M. and should not be run
manually unless under the supervision of BroadSoft Support personnel. Running this procedure
can affect performance.

To set the database to update statistics:

Run /usr/local/broadworks/bw_base/[Link].

10.6.2 Start and Stop MySQL Database Daemon

WARNING: The MySQL database is started automatically upon server power up or reboot. The
database should not be manually stopped unless under direct supervision from BroadSoft
Support.

The mySQL database daemon is controlled through the following command.

/etc/init.d/mysql_daemon [stop | start | restart]

The database cannot be stopped if processes are attached to the database. Such
processes can include BroadWorks, replication daemons, and MySQL tools such as
mysql.
To stop the MySQL database daemon, run the following commands:
1) Stop the BroadWorks server (from the UNIX prompt type $ stopbw).
2) Stop mySQL replication if server is part of a redundant cluster (from the UNIX prompt
type $ repctl stop).
3) Log in (su -) as root and run the following commands.
# cd /etc/init.d/
# ./mysql_daemon stop

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 111 OF 247
10.6.3 Regain Disk Space Used by MySQL InnoDB Tablespace Files
InnoDB stores data in a tablespace. By default on BroadWorks servers, data (data +
indexes) for each table are stored in a separate tablespace file. The tablespace file is
automatically extended as needed when the table grows (entries are inserted into the
table). When data is removed from the table (entries are deleted), the tablespace file size
does not decrease. This is because InnoDB cannot return space to the file system.
Although, it makes sure to use the freed-up space for further inserts. Perform the
following steps to reclaim space after the tables are purged:
1) Back up the database. For detailed steps, see section 10.1 Database Backup.
2) Restore the database. For detailed steps, see section 10.2 Database Restore.

NOTE: For a redundant cluster, restoring a database on just one peer results in a database out-
of-sync condition. A database import is required on the other peer to ensure a return to
database in-sync state.

10.7 Additional MariaDB Maintenance Tasks

10.7.1 Start and Stop MariaDB Database Daemon

WARNING: The MariaDB database is started automatically upon server power up or reboot.
The database should not be manually stopped unless the application connected to it is also
stopped (stopbw).

The MariaDB database daemon is controlled through the following command.

mdbctl [bootstrap | restart | start | status | stop]

To stop the MariaDB database daemon, run the following commands:

1) Stop the BroadWorks server (stopbw).
2) Stop MariaDB.
bwadmin@nfm$ mdbctl stop

The boostrap option is used when starting the dsatabase of the first Network Fucntion Manager
in the lcuster. This Network Function Manager is therefore acting as the source server for all
other peers to be booted.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Redundancy management for the Database Server is described in the BroadWorks

Database Server Configuration Guide. For more information on database redundancy
management, see this guide.

11.1 Application Server, Network Server, and Messaging Server Database

Replication

11.1.1 TimesTen Database Synchronization

The TimesTen replication daemon makes sure that database information across cluster
peers (Application Server, Network Server, and Messaging Server clusters) remains
synchronized. It starts automatically on a fresh install or upgrade and runs independently
of the BroadWorks application (that is, a stopbw does not stop replication). It is important
that replication be running on all cluster peers at all times.

WARNING: Stopping TimesTen Replication can result in cluster member databases becoming
out-of-sync. Replication should only be stopped when indicated in a BroadWorks maintenance
procedure or under the supervision of BroadSoft Support personnel.

In addition to an out-of-sync condition, failure to run replication on all cluster peers at all
times can possibly lead to longer re-synchronization times once replication is re-enabled
later. The Application Server, Network Server, and Messaging Server track three states
related to the database:
 Replication State – For each peer in the cluster, the state of whether file replication
and database replication are running is tracked.
 Database Replication Lagging State – The database retains transactions (and their
associated log files) if they have not been successfully replicated to all remote peers.
If a peer is shut down (entire server shut down, including replication) for maintenance
for an extended period and replication is not turned off on the operating server, a high
number of transaction files are retained by replication on the operating server. This
can cause excessive disk space usage and result in a long time for databases to re-
synchronize when the previously shut down server is re-instated. For each peer in the
cluster, the number of database transaction log files retained by replication is tracked.
Also tracked are whether an excessive number of transaction log files have been
retained by replication and whether an import resynchronizes the databases faster
than replication.
This threshold is defined at 90% of 360 transaction files (that is, 324 files).
 Database Notification Lagging State – The database retains transactions (and their
associated log files) if associated notifications have not been delivered to the
application. If database notifications have not been delivered to the application, or if
there is an excess of database notifications, a high number of transaction log files are
retained by database notification. This can cause excessive disk space usage,
resulting in the application cache not being synchronized with the database, and
points to a possible defect in the notification delivery mechanism. For each peer in
the cluster, the number of database transaction log files retained by notification is
tracked. Also tracked is whether an excessive number of transaction log files have
been retained by notification.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 113 OF 247
This threshold is defined at 20 * number of central processing units (CPUs). For
example, the threshold for one processor system is 20 transaction log files. The
threshold for a four-processor system is 80 transaction log files.
The healthmon command with the –d option outputs details for the three states. The
summary (true/false status) for the three states is output as part of invoking repctl status.
Warnings for replication lagging and notification lagging exceeding threshold are
generated by repctl stop, healthmon (without the -d option), and BroadWorks startup.
Replication can be stopped or started from the CLI System/Peering level or from the
UNIX prompt using the repctl command.
To stop the TimesTen replication daemon:
From the UNIX prompt, type $ repctl stop. The following response appears (sample
provided).
Locking local database. This may take a few minutes. . .
Database already locked
Pushing transactions held by replication to remote peers
Keeping the database locked
Shutting down Redundancy/Replication
Stopping file replication (this may take up to 600 seconds, please be
patient)
Database Redundancy:
Stopping Database Redundancy... [done]
Dropping Replication Schema... [done]

NOTE: As seen in the sample output above, repctl stop pushes transactions held by replication
to all peers in the cluster. If the transactions held by replication could not be pushed to all peers
in the cluster, the following warning is generated.

WARNING: Outstanding transactions could not be replicated to all remote peers. Databases
are out-of-synch. Run repctl status for more details on replication status. If required, import the
database from the appropriate cluster member to all other cluster members to synchronize the
system.

To start the TimesTen replication daemon:

From the UNIX prompt, type $ repctl start. The following response appears
(sample provided).
Checking replication status on peer servers:
-> Replication is clear for start up
Starting Replication:
Database Replication:
Performing Checkpoint... [done]
Creating Replication Schema... [done]
Starting Database Redundancy... [done]

NOTE 1: The replication daemon can be bounced (stopped/started) in one step using the
repctl restart option.

NOTE 2: repctl start checks the replication lagging state on all servers in the cluster.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 114 OF 247
If replication is found to be lagging on any server in the cluster, the following message is
displayed:

WARNING: Replication is lagging on one of the servers in the cluster. Invoke repctl status for
more details. Use database import to synchronize with the server exhibiting replication lagging.

11.1.2 TimesTen Replication Monitoring

Replication status and database notification status can be checked using the UNIX prompt
command repctl status. It is critical to the BroadWorks Redundancy solution that the
replication daemon is running at all times. Replication status can be checked using the
UNIX prompt command repctl status.
To view the TimesTen replication daemon state:
From the UNIX prompt, type $ repctl status. The following response appears
(sample provided).
Redundancy/Replication Status
-----------------------------
File Replication pid(s) = 1436
Datastore name = AppServer
Replication Agent Policy : always

Replication State
MTLAS04: (filerep: started) (database: started)
MTLAS01: (filerep: started) (database: started)

Database Replication Lagging State

MTLAS04: ok (1 files held)(earliest LSN 40/6688160)
MTLAS01: ok (1 files held)(earliest LSN 42/702072)

Database Notification Lagging State

MTLAS04: ok (1 files held)
(ExecutionServer earliest LSN 40/6676296)
(ProvisioningServer earliest LSN 40/6700176)
MTLAS01: ok (1 files held)
(ExecutionServer earliest LSN 42/618728)
(ProvisioningServer earliest LSN 42/618728)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 115 OF 247
The state of database replication lagging for each peer in the cluster is output under the
title “Database Replication Lagging State” as shown in the example above either as “ok” or
“LAGGING !”. For each peer in the cluster, the canonical host name is output followed by
a status, indicating whether replication is lagging at that peer. The number of database
transaction log files held by replication is output, followed by the TimesTen log sequence
number (that is, LSN) of the earliest transaction held by the TimesTen replication daemon.
The LSN is of the format filenum/transactionNum. The filenum corresponds to the
transaction log file number for transaction logs in
/usr/local/broadworks/bw_base/persistent. The transactionNum is an internal transaction
number.
The state of database notification lagging for each peer in the cluster is output under the
title “Database Notification Lagging State” as shown in the example above. For each peer
in the cluster, the canonical host name is output followed by a status indicating whether
database notification is lagging at that peer. The number of database transaction log files
held by database notification is output followed by the TimesTen log sequence number of
the earliest transaction held by the TimesTen replication daemon. On the Application
Server, there are two entries related to the number of transaction log files held
(ExecutionServer and ProvisioningServer as shown in the output above) for each peer.
On the Network Server, there are other entries for each peer.
To obtain more information about database replication lagging and database notification
lagging, use healthmon -l -d. See the following description of such details.
The healthmon script conducts a full two-way database replication test. A replication
failure results in the following error message.
--------------------------------
System Health Report Page
BroadWorks Server Name: as2
Date and time : Fri Jun 6 13:44:19 EDT 2003
Report severity : CRITICAL
Server type : AppServer
--------------------------------
Replication: a remote server is not receiving replication updates.
Databases may be out-of-synch.

--------------------------------

Recommendations
---------------

Replication must be started on all remote servers (repctl start). If

databases are out-of-synch, they must be re-synchronized first (with the
[Link] tool). Please refer to the BroadWorks Maintenance Guide for
detailed procedures.

--------------------------------

Healthmon also checks the database replication lagging state and database notification
lagging state. Lagging results in the following recommendations.
--------------------------------
System Health Report Page
BroadWorks Server Name: as2
Date and time : Fri Jun 6 13:44:19 EDT 2003
Report severity : CRITICAL
Server type : AppServer
--------------------------------

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 116 OF 247
Recommendations
---------------
Replication is severely lagging on one of the servers in the cluster. If
the system is being operated in standalone mode, turn off replication.
Otherwise, import the database from the server exhibiting the lagging to
ensure prompt data synchronization.

Database XLA is severely lagging on one of the servers in the cluster.

If the files held by database notification continue to grow and the LSN’s
do not advance, contact BroadSoft support prior to restarting the
affected server.
--------------------------------

NOTE: Healthmon should be set up as a scheduled task running at least once an hour.

WARNING: If the repctl status indicates replication is not running, Healthmon indicates that a
remote server is not receiving replication updates, or repctl status/healthmon indicates that
database replication lagging exists at one of the peers in the cluster and the databases need to
be resynchronized. Database resynchronization is accomplished using the Import Database
procedure. In general, the import is run on the secondary server.

11.1.3 TimesTen Replication Port Number

It is possible to specify a port to be used by the TimesTen daemon for replication. By
default, the port number is assigned randomly.

NOTE: A change to a replication port only takes effect after the replication is bounced.
Therefore, changing the port MUST be done during a maintenance window.

To change a replication port, an operator must bounce replication. As such, to prevent

any data de-synchronization, it is important to adhere to the following rules and steps:
1) Change the replication port at the primary server in the cluster. Note that when you
change the replication port, it is automatically replicated to all replication peers.
NS_CLI/System/Peering> get
replicationPort = random

NS_CLI/System/Peering> set replicationPort port 19999

...Done

NS_CLI/System/Peering> get
replicationPort = 19999

2) From the primary server, lock all servers for provisioning.

bwadmin@MTLNS01$ peerctl lock all

3) From the primary server, stop replication on all peers.

bwadmin@MTLNS01$ peercmd repctl stop

4) On the primary server, restart replication.

bwadmin@MTLNS01$ repctl start

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 117 OF 247
5) Unlock the provisioning “function” on the primary server.
bwadmin@MTLNS01$ peerctl unlock mtlns01

On the secondary server for the Application Server or each peer (one at a time) on the
Network Server, and Messaging Server, perform the steps according to the following
commands (6 through 10):
6) Stop BroadWorks.
bwadmin@MTLNS02$ stopbw

7) Import the database.

bwadmin@MTLNS02$ [Link] networkserver mtlns01 networkserver

8) Restart the replication.

bwadmin@MTLNS02$ repctl start

9) Unlock the provisioning “function”.

bwadmin@MTLNS02$ peerctl unlock mtlns02

10) Restart BroadWorks.

bwadmin@MTLNS02$ startbw

It is also possible to view the currently used replication port using bwListPortInUse
script as follows.
bwadmin@MTLNS01$ bwListPortInUse
Usage Type Port Internal
----------------------------------------------------------------
SNMP agent UDP 8001 N
SNMP agent trap server TCP 30002 Y
MO daemon TCP 7120 Y
MO root daemon TCP 7121 Y
Patch Tool TCP 2223 N
Patch Tool (Perl) TCP 2224 Y
TimesTen Main Daemon TCP 16003 N
Replication port TCP 19999 N
Naming service port TCP 1050 Y
SIP listening port UDP 5060 N
MSS listening port UDP 5677 N
ASR listening port UDP 5090 N

11.2 Element Management System Database Replication

11.2.1 MySQL Database Synchronization

The version MySQL packaged with BroadWorks supports for one-way replication. One
server acts as the master, while one or more other servers act as slaves. The master
server writes updates to its binary log files, and maintains an index of the files to keep
track of log rotation. These logs serve as a record of updates to be sent to slave servers.
When a slave server connects to the master server, it informs the master of its last
position within the logs since the last successfully propagated update. The slave catches
up on any updates that have occurred since then, and then blocks and waits for the
master to notify it of new updates.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 118 OF 247
Note that BroadWorks uses twice one-way master-slave replication to provide two-way
replication functionality. This implies that data written on any of the servers in a pair of
servers is replicated to its peer server.
The MySQL replication ensures that database information across cluster peers (Element
Management System pair) remains synchronized. It starts automatically on a fresh install
or upgrade, and runs independently of the BroadWorks application (that is, a stopbw
does not stop replication). It is important that replication be running on both servers of a
pair at all times.

WARNING: Stopping MySQL replication can result in cluster member databases becoming out-
of-sync. Replication should only be stopped when indicated in a BroadWorks maintenance
procedure or under the supervision of BroadSoft Support personnel.

In addition to an out-of-sync condition, failure to run replication on all cluster peers at all
times can possibly lead to longer re-synchronization times when replication is re-enabled
later. The Element Management System tracks the states of the master and slave
connection to the database. Under normal operation, the state is set to “Waiting for
master to send event”.
Replication can be stopped or started from the CLI System/Peering level or from the
UNIX prompt, using the repctl command.
To stop the MySQL replication daemon:
From the UNIX prompt, type $ repctl stop. The following response appears (sample
provided).
bwadmin@[Link]$ repctl stop
Shutting down Redundancy/Replication
File Replication:
Stopping File Replication... [done]
Database Redundancy:
Stopping Database Redundancy... [done]

To start the MySQL replication daemon:

From the UNIX prompt, type $ repctl start. The following response appears
(sample provided).
bwadmin@[Link]$ repctl start
Checking replication status on peer servers... [done]
[Link]: Checking replication status... Started.
Versions must match.
[Link]: Checking version... Versions matched (EMS
version Rel_16.0_1.483).
-> Replication is clear for start up
Starting Replication:
Starting File Replication... [done]
Database Replication:
Starting Database Redundancy... [done]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 119 OF 247
NOTE: The replication daemon can be bounced (stopped/started) in single step using the repctl
restart option.

11.2.2 MySQL Replication Monitoring

Replication status can be checked using the UNIX prompt command repctl status. It is
critical to the BroadWorks Redundancy solution that the replication daemon is running at
all times.
To view the MySQL replication daemon state:
From the UNIX prompt, type $ repctl status. The following response appears
(sample provided).
bwadmin@[Link]$ repctl status

Redundancy/Replication Status
-----------------------------
File Replication pid(s) = 10336
Datastore name = WebNmsDB
Replication Agent Policy : always

Replication State
[Link]: (filerep: true)(database: true)
[Link]: (filerep: true)(database: true)

Database Replication Lagging State

[Link]: (false) Master/Slave position:
3993266/3993266 (delta: 0)
[Link]: (false) Master/Slave position:
4418021/4418021 (delta: 0)

The number after the File Replication PID field is the RSYNC PID number. If no number is
present, file replication is not running on the local server. The Replication Agent Policy
field provides information on the state of mySql Replication on the local server. A value of
“always” indicates the replication is running. A value of “never” indicates the replication is
not running.
The state of database replication and file replication for all peers in the cluster is output
under the title “Replication State” as shown in the example above. The entries are
prefixed by the canonical host name of each peer.
The state of database replication lagging for each peer in the cluster is output under the
title “Database Replication Lagging State” as shown in the example above either as
“false”, that is, no lagging, or “true”. For each peer in the cluster, the canonical host name
is output followed by a status, indicating whether replication is lagging at that peer. The
number of database transaction log files held by replication is output, followed by the
mySql log sequence number (that is, LSN) of the earliest transaction held by the mySql
replication daemon. The LSN is of the format filenum/transactionNum.
 The filenum corresponds to the transaction log file number for transaction logs in
/usr/local/broadworks/bw_base/persistent.
 The transactionNum is an internal transaction number.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 120 OF 247
11.3 Install Cluster of Servers
In a redundant installation, the installation process prompts the operator for basic cluster
information:
 Number of peers
 For each peer:
− Primary or not – The concept of primary “installation server” applies to the
Application Server, Element Management System, Network Server, and
Messaging Server. There is always one and only one primary server in a cluster.
The concept of a primary server for Call Processing applies only to the
Application Server.
− Hostname – The canonical host name of the server.
− Address (signaling address) – A routable address used by the data replication
processes. It can be hostname, Fully Qualified Domain Name (FQDN), or IP
address. Hostname is recommended.
The information in the following installation example is gathered on the primary server.
…
Will this server be a member of a redundant cluster? (y/n) [n]?y

Is this server the primary peer (the first one installed) (y/n) [y]?

This server is the primary peer. Its installation is a bit more involved
and requires you to enter the list of all peers in the cluster. These
other peers (secondaries) will use this information to configure
themselves.

You must enter a list of hostname/address pairs identifying each peer.

The hostnames should be the servers’ canonical hostname

(to know the canonical hostname of a server, type ‘hostname’
at the command prompt). The hostname for the primary server
is retrieved automatically.

The addresses must be routable and will be used when configuring

replication. They can be hostnames, decimal dotted IP addresses,
or fully qualified domain names.

Note that it is possible to add a server to an existing cluster

without re-installing the existing servers. Please consult the
documentation.

What is the ADDRESS of THIS server? [MTLNS04]

Redundant server settings:
Current server hostname is MTLNS04.
Primary server hostname is MTLNS04.
Currently configured peer(s) (list of hostname/address):
MTLNS04/MTLNS04 (primary)
Replication Port: 17888

Enter an additional peer hostname (q to quit, c to clear the list):

MTLNS01
Enter peer address: MTLNS01

Enter an additional peer hostname (q to quit, c to clear the list): q

Please provide a TimesTen replication port [17888]:

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Do you accept these settings (y/n) [n]?y

NOTE: The individual FQDN for this server, [Link], was

automatically taken from the /etc/hosts file.

When run on the secondary server, the installation process automatically retrieves the
cluster information from the primary server. The only input required is the routable
address of the primary server.
…
Will this server be a member of a redundant cluster? (y/n) [n]?y

Is this server the primary peer (the first one installed) (y/n) [y]?n

This server is a secondary server. All that is needed to configure

redundancy is the routable address of the primary server.

What is the routable address of the PRIMARY server? [] MTLNS04

…

Do you accept these settings (y/n) [n]?y

Secure Shell (SSH) is installed on all Network Servers, Application Servers, and
Messaging Servers to support the need for automated replication control to issue remote
commands from one server to the other in a secure manner. During installation, keys are
automatically exchanged between the peers.

11.3.1 Element Management System Additional Parameters

When installing a redundant Element Management System, the administrator is prompted
for additional parameters:
 Heartbeat Interval – Time interval between heartbeat events on the active server (the
heartbeat event is written into the database and replicated to the backup server
database).
 Failover Interval – Time interval between two readings of the heartbeat event on the
backup server. In case the heartbeat event is not set in the database, the backup
servers increment a counter of consecutive heartbeat failure events.
 Retry Count – Number of heartbeat failure events before the backup server initiates a
failover.
Example: In case the failover time is set to 30 seconds, and the retry count is set to
“2”, it takes up to 60 seconds for the backup server, before taking over the failed
active server.

NOTE: The data received by the backup server (SNMP traps) during the failed heartbeat
interval is kept and committed into the database once the backup server takes over the active
server. This implies that there should not be any data lost for as long as the network elements
send their data to both Element Management System servers simultaneously.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 122 OF 247
Following is a sample output for the installation of an Element Management System.
...
Do you want to configure redundancy? (y/n) [y]?y

Will this server be a member of a redundant cluster? (y/n) [y]?

Is this server the primary peer (the first one installed) (y/n) [y]?

You must enter a list of hostname/address pairs identifying each peer.

The hostnames should be the servers' canonical hostname

(to know the canonical hostname of a server, type 'hostname'
at the command prompt). The hostname for the primary server
is retrieved automatically.

The addresses must be routable and will be used when configuring

replication. They can be hostnames, decimal dotted IP addresses,
or fully qualified domain names.

Note that it is possible to add a server to an existing cluster

without re-installing the existing servers. Please consult the
documentation.

What is the ADDRESS of THIS server? [mtlems01]

Data redundancy server settings:
Current server hostname is mtlems01.
Primary server hostname is mtlems01.
Currently configured peer(s) (list of hostname/address):
mtlems01/mtlems01 (primary)
Active server heart beat interval: 20.
Standby server fail over interval: 30.
Standby server retry counts: 2.

Enter an additional peer hostname (q to quit, c to clear the list):

mtlems02
Enter peer address: mtlems02

Enter an additional peer hostname (q to quit, c to clear the list): q

What is the heart beat interval (in seconds)

when running as the active server? [20]

Note: the fail over interval must be

longer than the heartbeat interval.
What is the fail over interval (in seconds)
when running as the standby server? [30]

What is the number of retries before activating

the fail over when running as the standby server? [2]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 123 OF 247
11.4 Network Server and Application Server Synchronization
Application Server group and user add/delete transactions are automatically pushed to the
Network Server. It is possible that the Application Server and Network Server are out of
synchronization, for example, if auto-synchronization is disabled or it is not functioning.
The Application Server and Network Server can be re-synchronized by manually dumping
the Application Server users and groups, transferring the file to the Network Server, and
loading the file on the Network Server.
The Network Server overlays the information on top of existing information.
To re-synchronize the Application Server and Network Server:
1) Create a temporary directory and set full permissions. Example:
$ mkdir /export/home/bwadmin/asdumpOutput
$ chmod 777 /export/home/bwadmin/asdumpOutput

NOTE: The CLI runs as the bworks user, which cannot write to all directories on the server.
Creating and using a temporary directory to store the ASDump information is then
recommended.

2) From the Application Server CLI System/Util/ASDump level, type:

dump /export/home/bwadmin/asdumpOutput/[Link]

3) Transfer the [Link] file to the Network Server bwadmin home directory. The file
should be transferred as text.
4) From the Network Server CLI System/Util/ASUpload level, type:
upload <hostingNeName> /export/home/bwadmin/[Link]
/var/broadworks/[Link]

… where <hostingNeName> is the name of the HostingNe representing the

Application Server on which the ASDump was performed.

11.5 Add New Peer to Redundant System

11.5.1 Application Server, Network Server, and Messaging Server

To add an additional peer to an already redundant system, the BroadWorks Application
Server, Network Server, and Messaging Server must adhere to the following procedure.

NOTE 1: Make sure all servers are of the same release.

NOTE 2: Make sure all servers are at the same patch level.

1) Install the new server following the instructions for the installation of a stand-alone
server provided by BroadSoft in the BroadWorks Software Management Guide.

NOTE 1: Make sure that all servers from the cluster and the newly installed server have SSH
keys configured properly. For more information, see section 11.9.2 Configure SSH Between
BroadWorks Servers.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 124 OF 247
NOTE 2: BroadWorks must not be running on the newly installed server (run stopbw).

2) On one of the existing cluster servers, run peercmd peerctl add <hostname>
<ipaddress> where <hostname> is the stand-alone server host name.

NOTE: This procedure can also be used to configure two stand-alone servers as one single
cluster. Take into account that one of the stand-alone servers is referenced in this guide as the
cluster server while the other is designated as the new stand-alone node.

3) Verify that the new stand-alone node has been added correctly with peerctl ls.
4) On the new stand-alone node, run peerctl add <hostname> <ipaddress> where
<hostname> is the primary server host name.
5) On the new stand-alone node, run peerctl setPrimary <hostname> where
<hostname> is the primary server host name.
6) Verify that the primary site is properly identified on the new stand-alone with peerctl
ls.
7) Run peercmd peerctl lock to lock every peer database.

NOTE: This only needs to run on a single node, since it automatically reaches other nodes.

8) Bounce replication with repctl restart on each server of the cluster only.
9) Run peercmd peerctl unlock to unlock every peer database.
10) On the new stand-alone node, import the database by running [Link]
<LocalDSN> <RemotePeer> <RemoteDSN> and press Enter when prompted.
11) Restart the replication on the newly added node with command repctl restart.

12) For the Network Server, update the Network Server routing alias list by adding the
newly added peer.
13) Start BroadWorks on the newly added node with command startbw.

Following is a sample of this procedure.

NOTE: In the following example, [Link] (primary) and

[Link] are in the same cluster whereas, mtl64lin15 is the stand-alone
server to be added to the cluster. The host name used (some use FQDN, while others do not) is
related to what the host name command returns locally for each server.

bwadmin@[Link]$ peerctl add mtl64lin15 mtl64lin15

bwadmin@[Link]$ peerctl ls
HOSTNAME/ADDRESS State
--------------------------------------------------------------------------------
[Link]/[Link] unlocked
[Link]/[Link] primary,unlocked
mtl64lin15/mtl64lin15 unlocked

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 125 OF 247
bwadmin@[Link]$ peercmd peerctl lock
==== BroadWorks Network Command Spawning Tool version 1.2 ====

=> Executing 'peerctl lock' locally <=

=> Spawning 'peerctl lock' to [Link] <=

=> Spawning 'peerctl lock' to mtl64lin15 <=

==== BroadWorks Network Command Spawning Tool [done] ====

bwadmin@[Link]$ repctl restart

Locking local database. This may take a few minutes. . .

Database already locked
Pushing transactions held by replication to remote peers
Keeping the database locked
Shutting down Redundancy/Replication
Stopping file replication (this may take up to 600 seconds, please be patient)
Database Redundancy:
Stopping Database Redundancy... [done]
Dropping Replication Schema... [done]
Checking replication status on peer servers:
[Link]: Checking replication status... Started.
[Link]: Checking version... Versions match (NS version
Rel_16.0_1.376)
[Link]: Checking patch level... synchronized.
mtl64lin15: Checking replication status... Stopped.
=> Replication is clear for start up
Starting Replication:
Database Replication:
Performing Checkpoint... [done]
Creating Replication Schema... [done]
Starting Database Redundancy... [done]

bwadmin@[Link]$ repctl restart

Locking local database. This may take a few minutes. . .

bwadmin@mtl64lin15$ [Link] appserver [Link] appserver

-> Making safety backup <-
Backing up dsn appserver to
/var/broadworks/backup/AS_Rel_16.0_1.[Link]... [done]

WARNING: ALL information in data store appserver will be lost.

Press Ctrl-C now to abort or <Enter> to continue...
Continuing.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 126 OF 247
-> Importing data from primary server <-
Retrieving localhost in remote replication schema... [mtl64lin15]
Performing checkpoint on remote database... [done]
Unloading database from memory... [done]
Destroying database appserver... [done]
Importing the remote database... [done]
Loading database in memory... [done]
Updating statistics... [done]

bwadmin@mtl64lin15$ repctl restart

Shutting down Redundancy/Replication

Stopping file replication (this may take up to 600 seconds, please be patient)
Database Redundancy:
Stopping Database Redundancy... [done]
Dropping Replication Schema... [done]
Checking replication status on peer servers:
[Link]: Checking replication status... Started.
[Link]: Checking version... Versions match (NS version
Rel_16.0_1.376)
[Link]: Checking patch level... synchronized.
[Link]: Checking replication status... Started.
[Link]: Checking version... Versions match (NS version
Rel_16.0_1.376)
[Link]: Checking patch level... synchronized.
=> Replication is clear for start up
Starting Replication:
Database Replication:
Performing Checkpoint... [done]
Creating Replication Schema... [done]
Starting Database Redundancy... [done]

14) On the newly added server, edit the [Link] file to modify the
REDPRIMARYHOSTNAME to indicate the new primary server host name:
− Delete the line containing REDPRIMARY, since this server is not the primary
server.
− Indicate that the server is now redundant by adding line
REDUNDANTSERVER=true if not already present.
− Delete the host names of the redundant peers on the REDPEERS line. The list
of host names is maintained in the primary server.
mtl64lin15$ vi /var/broadworks/[Link]
“[Link]” 8 lines, 196 characters
BROADWORKS_USERNAME=bwadmin
SERVER_TYPE=NetworkServer
USESSL=false
REDHOSTNAME=mtl64lin15
REDPRIMARYHOSTNAME=[Link] -- the redundant primary hostname
is modified
REDUNDANTSERVER=true -- this line replaces REDPRIMARY line
REDPEERS={} -- hostnames of redundant peers are deleted
~
“[Link]” 8 lines, 212 characters

15) On the primary server, edit the [Link] file to modify the REDPEERS to add
the new cluster member.
[Link]$ vi /var/broadworks/[Link]
“[Link]” 8 lines, 196 characters
BROADWORKS_USERNAME=bwadmin
SERVER_TYPE=NetworkServer
USESSL=false

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 127 OF 247
REDHOSTNAME=[Link]
REDPRIMARYHOSTNAME=[Link]
REDUNDANTSERVER=true
REDPRIMARY=true
REDPEERS={mtl64lin15->mtl64lin15,[Link]-
>[Link],[Link]-
>[Link],} -- hostnames of redundant peers
~
“[Link]” 8 lines, 212 characters

The system is now redundant. You can re-use this procedure to add any number of
additional stand-alone servers to a cluster.

11.5.2 Element Management System

To add an additional peer to a non-redundant system, use the BroadWorks Element
Management System setup-redundancy script. The setup-redundancy script queries for
the list of new peers, it adds them to BroadWorks, it configures SSH, and then it begins
replication on all original servers so that the new node is ready to import the DSN. All
functionality is built into the script, which prompts the user for the necessary information.

NOTE: Make sure that all servers are of the same release.

1) Following the instructions for the installation of a secondary server provided by

BroadSoft in the BroadWorks Software Management Guide, install the new peer.
Make sure to properly set the primary server name while configuring redundancy.

NOTE: Do not log in to the new peer as bwadmin until step 3, which follows.

2) On the primary server, run setup-redundancy and add a new peer when
prompted. Following is a sample of this procedure.
bwadmin@shoule [/usr/local/broadworks/bw_base/bin]$ ./setup-redundancy
Checking current redundancy set-up

You are about to set-up ems redundancy for Rel_15.0_1.200

Do you wish to continue? (y/n) [y]? y

Setting up ems redundancy for Rel_15.0_1.200

... Loading the configuration properties for the ElementManagementSystem

--- BroadWorks Installation Data Entry ---

Data redundancy server settings:
Current server hostname is shoule.
Primary server hostname is shoule.
Currently configured peer(s) (list of hostname/address):
shoule/shoule (primary)

Do you want to configure redundancy? (y/n) [y]?y

Will this server be a member of a redundant cluster? (y/n) [y]?y

Is this server the primary peer (the first one installed) (y/n) [y]?y

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 128 OF 247
This server is the primary peer. Its installation is a bit more involved
and requires you to enter the list of all peers in the cluster. These
other peers (secondaries) will use this information to configure
themselves.

You must enter a list of hostname/address pairs identifying each peer.

The hostnames should be the servers' canonical hostname

(to know the canonical hostname of a server, type 'hostname'
at the command prompt). The hostname for the primary server
is retrieved automatically.

The addresses must be routable and will be used when configuring

replication. They can be hostnames, decimal dotted IP addresses,
or fully qualified domain names.

Note that it is possible to add a server to an existing cluster

without re-installing the existing servers. Please consult the
documentation.

What is the ADDRESS of THIS server? [shoule]

Data redundancy server settings:
Current server hostname is shoule.
Primary server hostname is shoule.
Currently configured peer(s) (list of hostname/address):
shoule/shoule (primary)
Active server heart beat interval: 20.
Standby server fail over interval: 30.
Standby server retry counts: 2.

Enter an additional peer hostname (q to quit, c to clear the list):

tier3mtl
Enter peer address: [Link]

Enter an additional peer hostname (q to quit, c to clear the list): q

What is the heart beat interval (in seconds)

when running as the active server? [20]

Note: the fail over interval must be

longer than the heartbeat interval.
What is the fail over interval (in seconds)
when running as the standby server? [30]

What is the number of retries before activating

the fail over when running as the standby server? [2]

===================================================
Basic settings:

Data redundancy server settings:

Current server hostname is shoule.
Primary server hostname is shoule.
Currently configured peer(s) (list of hostname/address):
shoule/shoule (primary)
tier3mtl/[Link]
Active server heart beat interval: 20.
Standby server fail over interval: 30.
Standby server retry counts: 2.

===================================================
Type Ctrl-c to abort installation

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Re-spawning as root...
INFO: configuring redundancy from a BroadWorks upgrade to
EMS_Rel_15.0_1.200
INFO: running on primary
INFO: Retrieving data from /var/broadworks/[Link]
INFO: updating [Link] configuration file
Killing mysqld with pid 12402
Wait for mysqld to exit. STOPPING server from pid file
/usr/local/mysql/mysql_base/var/[Link]
081128 17:16:22 mysqld ended

done
nohup: redirecting stderr to stdout
Starting mysqld daemon with databases from
/usr/local/mysql/mysql_base/var

INFO: PRIMARY_HOSTNAME is shoule

INFO: PRIMARY_ADDRESS is shoule
INFO: SECONDARY_HOSTNAME is tier3mtl
INFO: SECONDARY_ADDRESS is [Link]

INFO: using config-ssh for bwadmin to [Link]

INFO: Enforcing password for sql user on shoule
INFO: Backing up initial database
SUDO_USER is bwadmin

Backing up ems db...

Backing up from directory /usr/local/mysql/mysql_base/var/WebNmsDB

Backing up to /var/broadworks/db
INFO: preserving old backup

Ems back up completed

INFO: Generating sql privilege statements for peer [Link]

INFO: Running sql redundancy script on shoule
INFO: Generating sed files for configuration changes on shoule
INFO: generating redundancy parameter data for bw auto config
INFO: adding peer (tier3mtl) to /etc/hosts
INFO: configuring
/usr/local/broadworks/EMS_Rel_15_0_1.200/public_html/conf/database_param
[Link]
INFO: configuring
/usr/local/broadworks/EMS_Rel_15.0_1.200/public_html/conf/[Link]
INFO:
/usr/local/broadworks/EMS_Rel_15.0_1.200/public_html/conf/serverparamete
[Link] already configured
INFO: configuring
/usr/local/broadworks/EMS_Rel_15.0_1.200/conf/[Link]
bwadmin@[Link]'s password:
bwadmin@[Link]'s password:

Done.

3) Log in as bwadmin and perform the post installation on the new peer.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 130 OF 247
11.5.3 Profile Server
To add an additional peer to a system (whether or not redundant), the BroadWorks Profile
Server uses the config-redundancy script. The config-redundancy script requests the list
of new peers, adds them to BroadWorks, configures SSH, and begins file replication. All
functionality is built into the script, which prompts the user for necessary information.

NOTE: Make sure all servers are of the same release.

1) Following the instructions for the installation of an additional server provided by

BroadSoft in the BroadWorks Software Management Guide, install the new peer.
Make sure to properly set the primary server name while configuring redundancy.
2) On all servers, run config-redundancy and add new peers when prompted. Following
is a sample of this procedure.
bwadmin@PS01$ ./config-redundancy

-> Data gathering <-

--------------------------
Redundant server settings:
Non-Redundant system.
--------------------------

Enter an additional peer hostname (q to quit, c to clear the list):

PS02
Enter peer address: PS02

Enter an additional peer hostname (q to quit, c to clear the list): q

-> Confirmation <-

This is the New redundancy configuration:

--------------------------
Redundant server settings:
Currently configured peer(s) (list of hostname/address):
PS02/PS02
--------------------------

Do you want to keep the following configuration? (y/n) [y]?y

-> Adding new peer <-

Adding PS02/PS02...

-> Configure Secure Shell <-

----------------------------------------------
---- SSH CONFIGURATION TOOL version 2.2.15 ----
-> Setting default settings <-
Setting 'StrictHostKeyChecking no'
-> DNS Sanity test <-
[###############]
[...............]
-> Peer reachability test <-
[###]
[...]
-> Creating SSH keys <-
Creating keys for bwadmin@PS02...

Creating keys for bwadmin@PS01...

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Sharing keys with bwadmin@PS02...

Cleaning public keys for bwadmin@PS02...
Pushing local public keys...
Pulling remote public keys...
Sharing keys with bwadmin@PS02... [done]

-> Fully meshing SSH peers <-

-> Recursing with bwadmin@PS02 <-

Pushing config-ssh script to bwadmin@PS02...
Launching config-ssh on bwadmin@PS02...

-> Setting default settings <-

Setting 'StrictHostKeyChecking no'
-> DNS Sanity test <-
[###############]
[...............]
-> Peer reachability test <-
[###]
[...]
-> Keying SSH <-
Preparing bwadmin@[Link] for keying...

Sharing keys with bwadmin@PS01...

Cleaning public keys for bwadmin@PS01...
Pushing local public keys...
Pulling remote public keys...
Sharing keys with bwadmin@PS01... [done]

-> Testing ssh configuration <-

Testing bwadmin@PS02... [done]

---- SSH CONFIGURATION TOOL completed ----

---- config-redundancy [done] ----

3) On the new server, import the file repository files from the existing peer. As the
bwadmin user, run the following command on the new server.
$ rsync -vazurSHl <peerServer>:<file repository root>/. <file repository
root>/.

11.5.4 Network Function Manager

This procedure details the steps required to add an additional peer to an NFM cluster.
1) On the new sever (nfm3), configure SSH between the new peer and the existing
peers (nfm1 and nfm2) using the following command.
bwadmin@nfm3$ config config-ssh –createKeysSilent bwadmin@nfm1
bwadmin@nfm2

2) Add the new peer to the peer list of existing peers.

On nmf3 (then new peer):
NFM_CLI/System/Peering/Peers> add nfm3 nfm3

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 132 OF 247
NFM_CLI/System/Peering/Peers> add nfm1 nfm1
NFM_CLI/System/Peering/Peers> add nfm2 nfm2

On nfm1:
NFM_CLI/System/Peering/Peers> add nfm3 nfm3

On nfm2:
NFM_CLI/System/Peering/Peers> add nfm3 nfm3

3) Restart BroadWorks and the database on each peer, one after the other.
bwadmin@nfm1$ stopbw
bwadmin@nfm1$ mdbctl restart
bwadmin@nfm1$ startbw

11.6 Remove Peer from Redundant System

This procedure applies to the Network Server and Messaging Server.

NOTE: For the Application Server, there are only two nodes; removing one leaves you with a
non-redundant configuration.

To remove a peer from a redundant system, type the following commands:

 peerctl del
 repctl restart – on each node in the cluster

11.7 Recover Faulty Redundant Installation

If the primary server is installed and activated before installing the redundant server, the
SSH configuration is incorrect. Such a faulty installation may have three symptoms
depending on the base system:
 SSH is started and bwadmin does not exist.
 SSH is started and bwadmin exists, but SSH is not configured.
In both of these cases, the following message appears on the screen after starting
replication.
The authenticity of host ‘redsrv1 ([Link])’ cannot be established.
RSA key fingerprint is 96:1a:9a:a7:cb:6e:95:26:68:70:cc:6e:39:b0:90:9f.
Are you sure, you want to continue connecting (yes/no)? Please type ‘yes’
or ‘no’.

 SSH is not started (installed or not).

In this case, the following message appears.
Secure connection to redsrv2 refused. unexpected EOF in read_timeout

To correct the situation, the following commands should be entered at the prompt on the
primary server.
$ repctl stop
$ config-ssh –createKeys <redsrv1> <redsrv2> …
$ repctl start

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 133 OF 247
11.8 Post-installation Control
Peer information can be accessed via the server CLI. An operator can also lock the local
database or the database of a peer through the CLI to prevent modifications during a
maintenance window.
NS_CLI/System/Peering> ?
0) get : show replication parameters-related attributes
1) set : modify replication parameters-related attributes
2) lock : to lock the database
3) start : to start system replication
4) status : to obtain status of system replication
5) stop : to stop system replication
6) unlock : to unlock the database
7) Peers : go to level Peers

h (help), e (exit), q (quit), r (read), w (write), t (tree),

c (config), cd (cd), a (alias), hi (history), p (pause), re
(repeat)

NS_CLI/System/Peering> peer

NS_CLI/System/Peering/Peers> get
HostName Address State
=========================================================================
[Link] [Link] unlocked
[Link] [Link] unlocked

2 entries found.

11.9 SSH/RSYNC Configuration

SSH is used as the remote shell invocation method as well as for secure file
synchronization. To ensure trouble-free operation, the following steps should be taken.

11.9.1 Create SSH Keys (Completed Automatically During Installation)

 The SSH keys are created automatically during installation. A passphrase can be
used or not used. The recommended solution is not to use any pass phrase and to
rely on the overall network security, to ensure a proper usage of public keys.
However, the safest solution is to set up a complex pass phrase that must be used to
access the server. Keys can be regenerated at any time using the following three
commands.
$ /usr/local/bin/ssh-keygen –t rsa1 –f /export/home/bwadmin/.ssh/identity
$ /usr/local/bin/ssh-keygen –t dsa –f /export/home/bwadmin/.ssh/id_dsa
$ /usr/local/bin/ssh-keygen –t rsa –f /export/home/bwadmin/.ssh/id_rsa

 For more information on how to use a pass phrase, see the OPENSSH
documentation available from [Link].

11.9.2 Configure SSH Between BroadWorks Servers

[Link] Automated Configuration

SSH keys can be shared automatically, and the known_host list can be filled between
cluster members with the config-ssh utility.
[Link]$ config-ssh –h
Usage:/usr/local/broadworks/bw_base/bin/config-ssh [-createKeys] peer1

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

SSH configuration utility is used to create ssh keys for every peer
within a cluster. The utility then propagates the ssh keys to all peers
to facilitate access between each other. Any UNIX system that has
ksh, ssh and rsync installed can use this utility script.

Option
------
-createKeys is used to create ssh keys for every peer

[Link]$ config-ssh [Link]

----------------------------------------------
---- SSH CONFIGURATION TOOL version 1.14 ----
-> DNS Sanity test <-
[###############]
[...............]
-> Peer reachability test <-
[###]
[...]
-> Keying SSH <-
Preparing bwadmin@[Link] for keying...

Sharing keys with bwadmin@[Link]...

Pulling remote keys from bwadmin@[Link]...
Adding bwadmin@[Link] new keys to
bwadmin@[Link] authorised_keys...
Sharing keys with bwadmin@[Link]... [done]

Sharing keys with bwadmin@[Link]...

Pulling remote keys from bwadmin@[Link]...
Adding bwadmin@[Link] new keys to
bwadmin@[Link] authorised_keys...
Pushing public keys unto bwadmin@[Link]...
Adding bwadmin@[Link] new keys to
bwadmin@[Link] authorised_keys...
Sharing keys with bwadmin@[Link]... [done]

-> Fully meshing SSH peers <-

-> Recursing with bwadmin@solns <-

Pushing config-ssh script to bwadmin@ [Link]...
Launching config-ssh on bwadmin@ [Link]...
-> DNS Sanity test <-
[###############]
[...............]
-> Peer reachability test <-
[###]
[...]
-> Keying SSH <-
Preparing bwadmin@[Link] for keying...

Sharing keys with bwadmin@[Link]...

Pulling remote keys from bwadmin@[Link]...
Adding bwadmin@[Link] new keys to
bwadmin@[Link] authorised_keys...
Sharing keys with bwadmin@[Link]... [done]

Sharing keys with bwadmin@[Link]...

Pulling remote keys from bwadmin@[Link]...
Adding bwadmin@[Link] new keys to
bwadmin@[Link] authorised_keys...

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 135 OF 247
Pushing public keys unto bwadmin@[Link]...
Adding bwadmin@[Link] new keys to
bwadmin@[Link] authorised_keys...
Sharing keys with bwadmin@[Link]... [done]

---- SSH CONFIGURATION TOOL completed ----

[Link] Manual Configuration

[Link].1 Share SSH Keys

To access the server without a pass phrase in SSH, public keys must be in
~/.ssh/authorized_keys or ~/.ssh/authorized_keys2 of the remote server.
To locate the public keys in the proper directory:
#On each servers participating in a cluster (including the current
hosts):
$ cat $HOME/.ssh/[Link] > $HOME/.ssh/authorized_keys.`hostname`
$ cat $HOME/.ssh/id_rsa.pub > $HOME/.ssh/authorized_keys2.`hostname`
$ cat $HOME/.ssh/id_dsa.pub >> $HOME/.ssh/authorized_keys2.`hostname`

#for each other server (than the current)

$ ftp $SERVER
cd .ssh
bin
put authorized_keys.`hostname` (replace `hostname` with the real value)
put authorized_keys2.`hostname`(replace `hostname` with the real value)

#then, on each server

$ cat $HOME/.ssh/authorized_keys.* >> $HOME/.ssh/authorized_keys
$ cat $HOME/.ssh/authorized_keys2.* >> $HOME/.ssh/authorized_keys2

NOTE: Pass phrases are like passwords and should be changed on a periodic basis.
Whenever pass phrases change, public identities need to be redistributed for SSH to continue
working.

[Link].2 Known_hosts List

Fill the known hosts list by connecting to each redundant server using SSH, which also
includes the current hosts. SSH then asks if you want to add the new server to the list of
known hosts.
AS1$ ssh bwadmin@[Link]
The authenticity of host ‘[Link] ([Link])’ cannot be
established.
RSA key fingerprint is 76:9a:9e:06:12:06:4a:84:cc:fc:33:8b:c3:db:24:87.
Are you sure you want to continue connecting (yes/no)?yes
Warning: Permanently added ‘[Link]’ (RSA) to the list of known
hosts.
AS1$

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 136 OF 247
11.9.3 Use SSH
To access another server in the cluster, by issuing ssh bwadmin@serverName you get
access to the server’s serverName.
Example:
$ ssh bwadmin@MTLAS-4
Last login: Wed Feb 6 10:32:25 2002 from denali
Sun Microsystems Inc. SunOS 5.8 Generic February 2000
Sun Microsystems Inc. SunOS 5.8 Generic February 2000
###### # #
# # ##### #### ## ##### # # # #### ##### # # ####
# # # # # # # # # # # # # # # # # # # #
###### # # # # # # # # # # # # # # # #### ####
# # ##### # # ###### # # # # # # # ##### # # #
# # # # # # # # # # # # # # # # # # # # #
###### # # #### # # ##### ## ## #### # # # # ####
BroadWorks and its documentation are protected by copyright law and international
treaties. Unauthorized reproduction or distribution of this software, or any part
thereof, may result in severe civil and criminal penalties, and will be prosecuted
to the maximum extent possible under the law.
MTLAS-4$

If you issue the SSH bwadmin@serverName command, the command is run on the
server serverName.
Example:
$ ssh bwadmin@as4 /usr/local/broadworks/bw_base/bin/repctl status
Redundancy/Replication Status
File Replication pid =
Datastore name = ApplicationServer
Replication Agent Policy = manual
Replication Manually Started = false

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Disaster recovery, defined as the ability to return servers and applications to their working
states following unexpected physical damage or system failures causing data loss, is an
issue that encompasses the high-level design and maintenance of entire back-end
infrastructures. As such, it is outside the direct scope of BroadWorks maintenance and
best considered in close collaboration with your IT department (and in light of best
practices within the industry). However, certain general approaches and guidelines can
be considered with respect to BroadWorks servers.
Disaster recovery from the BroadWorks perspective entails two stages, first creating a
replacement server platform with matching operating system (OS) and BroadWorks
software level, and then recreating the user information and configuration data of the
BroadWorks component and reintegrating the server into the BroadWorks network.
Replacement servers must have the same settings as the servers they replace,
incorporating all the following elements:
 OS image
 OS patches
 UNIX/LINUX host name
 IP address(es) and configuration
 /etc/hosts file
 /etc/hostname.* files
 DNS configuration
 NTP configuration
 Security configuration (for example, JASS)
 BroadWorks release
 BroadWorks patch level
While the platform type of the server is not required to remain the same, it is
recommended that the new server have the same hardware footprint (CPUs, RAM).
Application Servers, Network Servers, and Messaging Servers do require that the new
server have the same OS type as the old server (Solaris or Linux).
If servers have multiple IP interfaces, then the replacement servers must also have the
same interfaces available and they must be defined.

12.1 Recover Server OS

Selecting a method for server restoration depends on a number of factors, such as
available capital investments for backup servers and organizational standards for
acceptable recovery times. The following is an overview of common approaches that you
can use successfully with BroadWorks.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 138 OF 247
12.1.1 Hot Standby Servers
In this scenario, a hot standby server is available for each BroadWorks server type. The
hot standby servers are running the same BroadWorks releases and are at the same
BroadWorks patch levels as their live counterparts (that is, the standby servers are
patched whenever the live servers are patched). Generally, the Network Server,
Application Server, and the Messaging Server standbys are installed as non-redundant
servers, and have blank databases. This helps to speed up reintegration into the
BroadWorks production network.
When an existing server must be replaced, the administrator reconfigures the standby as
follows:
 Configures the IP address to match the server to be replaced.
 Configures the host name to match the server to be replaced.
 Sets the local peer information to match the host name.
For example, if the standby was previously configured as “backup”, then the peerctl ls
command returns “backup” as local peer name.
bwadmin@ne1$ peerctl ls
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
backup/backup primary,unlocked

In the preceding example, the following commands need to be run on the

replacement server to set the peers properly.
bwadmin@ne1$ peerctl set backup ne1 ne1

 Moves the standby to the same subnet as the server to be replaced.

 Powers up the server and connects it to the switch.
Once these tasks are completed, the administrator can recreate the BroadWorks
application data and reintegrate the server back into the BroadWorks production network
as described in section 12.2.3 BroadWorks Server Restoration Procedure.

12.1.2 On-Demand Rebuild

As an alternative to maintaining standby copies of all BroadWorks servers, replacement
servers can be rebuilt on demand with the OS and BroadWorks components required to
emulate the failed server.
OS reinstallation can be done manually. However, for speed and consistency it is
recommended that you configure a Linux KickStart server, ensuring that the replacement
server is built to the same specification as the live servers (that is, the same OS patch
level). This approach also ensures that any customer-specific packages are loaded at the
same time.
Some providers opt to have a single OS pre-built standby server, which can be used to
replace any server in the BroadWorks network. Ideally, this single backup server platform
has a hardware footprint equal to the "largest" BroadWorks server. This single OS pre-
built standby server saves the three to four hours that it can take to reinstall and
reconfigure the operating system on the replacement server.
Some providers opt to flash archive each production server, so that the JumpStart or
KickStart process automatically restores the OS with the exact same configuration as the
existing server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 139 OF 247
Regardless of how the replacement server is built, it needs to be configured at the OS
level with the same information as the server to be replaced. For example:
 Hostname
 IP address
 /etc/hosts file
 /etc/hostname.* files
 Default router
 DNS configurations
 NTP configurations
Once the server is rebuilt with the required OS level and configuration, the corresponding
BroadWorks application can be installed on the replacement server, and any required
BroadWorks patches can be applied to bring the replacement to the same software
release as the production server.
For Application Servers, Network Servers, and Messaging Server s, it is typically easier to
install the component as a non-redundant server. In general, however, the configuration
information entered as part of the fresh install on the replacement server is not important,
as it is overwritten or reconfigured when the server is reintegrated into the live network.
Fresh installation and patching of a BroadWorks component should not take more than 30
to 60 minutes.
For providers that have opted to flash archive each production server, the flash archive
automatically restores the BroadWorks application level that was present at the time the
archive was made. Complications can arise if the version of the BroadWorks component
in the archive does not match the version of the BroadWorks component to be replaced,
as an upgrade to the production version would be required. It is therefore recommended
that the server be flash archived at least after every major BroadWorks release upgrade.
This requires only incremental BroadWorks patch applications to get the replacement
server to the same state as the production server.
Once the BroadWorks application on the replacement server is at the same level as that
on the production server, the administrator can recreate the BroadWorks application data
and reintegrate the server back into the BroadWorks production network as described in
section 12.2.3 BroadWorks Server Restoration Procedure.

12.1.3 Incremental Backups

In this scenario, the provider uses a third-party backup utility to periodically back up all
BroadWorks servers in the production network. There are a number of different
approaches within this scenario depending on the specific backup application used, but
the result from a BroadWorks perspective is the same in all cases. The replacement
server rebuilt from the full and incremental backups must end up with the same software
level as the server it replaces, both for the OS and for the BroadWorks component.
After restoring a server in this scenario, administrators still have to reintegrate the
replacement server into the BroadWorks network following the procedure outlined in
section 12.2.3 BroadWorks Server Restoration Procedure. However, not all the steps
apply in all cases. If the rebuilt server is actually more recent than the last BroadWorks
system backup, the instructions for restoring the BroadWorks system backup do not need
to be carried out.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 140 OF 247
12.2 BroadWorks Server Restoration
As stated previously, replacing a BroadWorks server requires that the replacement server
be at the same OS and BroadWorks level as the server to be replaced. Once that is
accomplished, the replacement server can be repopulated with user and application data if
required, and reintegrated into the production network.
The reintegration procedure consists of a number of manual configuration steps, and uses
the contents of the BroadWorks system backup to configure the replacement server with
the most recent information backed up from the server to be replaced. If the server to be
replaced has a live peer in the network with up-to-date information (for example, database,
user file synchronization), then it is obtained from the peer instead of the backup file. This
applies only to the Application Server, Network Server, and Messaging Server.
The amount of time required to fully restore a BroadWorks server is the sum of the time
required to:
1) Create a replacement server at the proper OS and BroadWorks level.
2) Mount and power the server in the proper location.
3) Run the BroadWorks restoration procedure described (as follows):
The first two variables are outside the scope of BroadWorks maintenance, and depend on
the OS restoration scheme in use. The time required for the BroadWorks Server
Restoration procedure depends on both the server component and the size of the
customer base. For example, a Media Server restoration takes only minutes, while an
Application Server restoration is bound to the time it takes to import data from the
secondary server, which can take as long as one hour on a larger cluster. In general, the
BroadWorks Server Restoration procedure should not take more than two hours on any
server type for any customer base or database size.

NOTE: On the BroadWorks Database Server, the restoration of the centralized database can be
significantly longer, depending on size of the database and the speed of the I/O subsystem. It is
recommended to evaluate the restoration time in a lab environment to determine realistic time
requirements.

12.2.1 BroadWorks System Backup

The system backup created by the [Link] script (run manually or as a
scheduled task) only contains BroadWorks items that change over time, such as:
 Directories that contain server-specific configuration files and user files (for example,
greetings)
 A backup of the database, if applicable for that server type
For a list of what is backed up on each server, see the
/usr/local/broadworks/bw_base/conf/[Link] file.
The items are tarred up, then gzipped into a single archive file and placed in the directory
specified in [Link]. By default, this directory is
/usr/local/broadworks/bw_base/persistent/backups.
When you unzip and untar the backup file in the / directory, all included directories and
files present on the local host are overwritten with the content of the backup, and a copy of
the backed-up database is placed in the default backup directory. BroadWorks
recommends scheduling the BroadWorks system backup to run on a daily basis. These
backups must be periodically transferred to another location to make sure that in the event
of a server loss, the backups are available.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 141 OF 247
NOTE: Starting with Release 18.0, Web Server (HTTP) certificates and keys are now backed up
by the system backup and they do not have to be manually backed up when they are changed
by the service provider.

12.2.2 Licensing
To start the BroadWorks application on the replacement server, you require a license zip
file that contains the hostid of the replacement server. To obtain new license files, contact
BroadSoft. If you use pre-configured recovery servers (either hot-standby, or pre-
configured OS standby, described above), you may be able to get the host IDs of the
standby servers added to existing license files, to avoid having to change license files in
the event of a swap. For more information, contact your BroadSoft Project Manager.

12.2.3 BroadWorks Server Restoration Procedure

Use the following procedures to replace existing BroadWorks servers. These procedures
assume that the replacement server has been built out to the same BroadWorks release
and patch level as the server it is replacing.

[Link] Media Server Replacement

1) Stop the BroadWorks application, the SNMP agent, the License Manager and the
Configuration Agent on the replacement server.
bwadmin@host$ stopbw
bwadmin@host$ snmpdctl stop
bwadmin@host$ lmdctl stop
bwadmin@host$ configdctl stop

2) Copy the last BroadWorks system backup created from the server to be replaced, to
any directory on the replacement server. As the bwadmin user, run the following
commands.
bwadmin@host$ gunzip <backupfile_name.gz>
bwadmin@host$ tar xvf <backupfile_name>

This restores the last backup of the production server's configuration information to
the replacement server. Because the backup .tar file stores the relative paths of all
included files, you need to extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you run the tar xvf command.
Note that the restore operation should be executed as root when performed from the
system root directory (/).
3) Make sure that valid [Link] file is installed on the server ([Link]
<path/to/[Link]>).
4) Start the Configuration Agent, the License Manager, and the SNMP agent.
bwadmin@host$ configdctl start
bwadmin@host$ lmdctl start
bwadmin@host$ snmpdctl start

5) If applications were manually deployed or undeployed on the server to be replaced,

deploy or undeploy them again on the replacement server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

[Link] Network Server, Application Server, and Messaging Server Replacement

Throughout this example, the new replacement server is referred to as “ne1”, while the
existing peer is called “ne2”.
1) On the remaining peer in the cluster, set the local peer as primary (if not already set),
and stop replication. This prevents any attempt to synchronize data to/from the
replacement server during the duration of the recovery procedure.
bwadmin@ne2$ peerctl setPrimary ne2
bwadmin@ne2$ peercmd peerctl lock
bwadmin@ne2$ peercmd repctl stop

2) If required, install BroadWorks on the replacement server and patch it to the level of
the active peer.
3) Stop the BroadWorks application, replication, the SNMP agent, the License Manager,
and the Configuration Agent on the replacement server if it is running.
bwadmin@ne1$ stopbw
bwadmin@ne1$ repctl stop
bwadmin@ne1$ snmpdctl stop
bwadmin@ne1$ lmdctl stop
bwadmin@ne1$ configdctl stop

4) Re-mesh the SSH keys. On all peers, cleanup the .ssh directory by running the
following.
bwadmin@ne1$ rm -Rf /export/home/bwadmin/.ssh
bwadmin@ne2$ rm -Rf /export/home/bwadmin/.ssh

5) As bwadmin, from either server, run the following.

bwadmin@ne1$ config-ssh -createKeys <ne1> <ne2> <other_peer_server>

<other_peer_server> can be the host name of the peer if the /etc/hosts file on the
current host is configured properly with the peer host name.
6) Reconfigure database replication peers.
If any peer server is still in service, as bwadmin, run the peerctl ls command on the
existing peer server host to get a list of peers. Following is an example output of
peerctl ls from the existing server.
bwadmin@ne2$ peerctl ls
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
ne1/ne1 unlocked
ne2/ne2 primary,locked

On the replacement server, you must make sure that the peer list is the same as it is
on the in-service peer.
In case of a hot standby replacement server, the local peer name in the replication
configuration data must have been altered to the local host name, as indicated in
section 12.1.1 Hot Standby Servers.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 143 OF 247
7) Given the above examples, the following commands would be run on the replacement
server to set the peers properly.
bwadmin@ne1$ peerctl add ne2 ne2
bwadmin@ne1$ peerctl setPrimary ne2

8) Verify that the information is consistent across all peers, with the exception of the state
on the new peer, which indicates “unlocked” while it shows “locked” on all other peers.
bwadmin@ne1$ peercmd peerctl ls
==== BroadWorks Network Command Spawning Tool version 1.2 ====

=> Executing 'peerctl ls' locally <=

HOSTNAME/ADDRESS State
------------------------------------------------------------------
ne1/ne1 unlocked
ne2/ne2 primary,unlocked

=> Spawning 'peerctl ls' to ne2 <=

HOSTNAME/ADDRESS State
------------------------------------------------------------------
ne1/ne1 unlocked
ne2/ne2 primary,locked

==== BroadWorks Network Command Spawning Tool [done] ====

9) Copy the last BroadWorks system backup created from the server to be replaced to
any directory on the replacement server. If you want to restore files to the directory in
which they were originally located when the backup was made, perform the restore
from the system root directory (/). As the root user, run the following commands.
root@ne1$ gunzip <backupfile_name.gz>
root@ne1$ tar xvf <backupfile_name>

On the Application Server, the localization information backup is stored in a file called
[Link]. To re-import the localization information, an operator must
use the [Link] –import [Link] command. For more
information, see the BroadWorks Application Server CommPilot Portal Customization
and Localization Guide.
10) On the Network Server, the only files that are replicated across peer servers are the
web branding files. Any web-branding file that was changed on the existing/running
Network Servers since the system backup is automatically resynchronized to the new
server within a short period (less than 30 minutes).
On the Application Server, with reference to files that may have changed since the
system backup, some replication-required file types are automatically resynchronized
once the new server is brought back on line within a short period (less than 30
minutes). These files include items such as web branding files and Device
Management files. For a full list of the automatically resynchronized directories, see
file /usr/local/broadworks/bw_base/conf/[Link]. Other user/group-related
replication-required files such as custom voice mail greeting, device configuration, and
Auto Attendant announcements are not synchronized automatically to the new server
once it comes back on line, as they are only synchronized on modification. To make
sure that these files are resynchronized properly, the following commands should be
executed from the source (existing server, “ne2” in this example) to manually trigger
an rsync to the destination (new server, “ne1” in this example), making sure to replace
the <Destination-Server-Hostname> with the appropriate host name.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 144 OF 247
bwadmin@ne2$ /usr/bin/rsync -vazSHle '/usr/bin/ssh -o batchmode=yes'
/var/broadworks/userfiles/customMediaFiles <Destination-Server-
Hostname>:/var/broadworks/userfiles

bwadmin@ne2$ /usr/bin/rsync -vazSHle '/usr/bin/ssh -o batchmode=yes'

/var/broadworks/IpDeviceConfig <Destination-Server-
Hostname>:/var/broadworks/

11) If the Application Server had been localized, web localization should be automatically
preserved from the system backup; however, any tone or announcement
customization is not part of the system backup and must be recreated. These can be
copied from the existing server. Depending on how localization was performed, these
custom files can be in a number of locations as follows:
− /var/broadworks/AS_Rel<xxxxxx>/sysprompts
− /var/broadworks/userfiles/<country_locale>
− /var/broadworks/userfiles/tones
12) Start the Configuration Agent, the License Manager, and the SNMP agent by running
the following command.
bwadmin@ne1$ configdctl start
bwadmin@ne1$ lmdctl start
bwadmin@ne1$ snmpdctl start

13) Make sure that valid [Link] file is installed on the server (install-licen [Link]
/path/to/[Link]).
14) Restart the BroadWorks replication on the already active peer(s).
bwadmin@ne2$ repctl restart

15) Import the database from the existing peer server. As the bwadmin user, run the
following command on the replacement server.
bwadmin@ne1$ [Link] [appserver <peer_ADDRESS> appserver]

or
bwadmin@ne1$ [Link] [networkserver <peer_ADDRESS> networkserver]

If the peer server is not in service, restore the database that was included in the last
BroadWorks system backup archive.
The parameters to [Link] are optional; if [Link] is called without any
arguments, it guesses the parameters and prompts for confirmation.
16) As bwadmin, run the following commands.
bwadmin@ne1$ repctl start
bwadmin@ne1$ synchcheck_basic.pl -a

Note that repctl start also starts file replication, which makes sure that any user files
that have changed since the last BroadWorks system backup (for example, greetings)
are replicated to the replacement server within the first 10 minutes of operation.
The syncheck_basic.pl is used to detect discrepancies between the databases on the
peer. If an error is reported, follow the recommendation to re-sync the database. Any
attempt to use the replacement peer in a redundant configuration depends on
properly synchronized databases.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 145 OF 247
17) Unlock the BroadWorks database on every peer.
bwadmin@ne2$ peercmd peerctl unlock

18) If applications were manually deployed or undeployed on the server to be replaced,

deploy or undeploy them again on the replacement server.
19) As bwadmin, run the following command.
bwadmin@ne1$ startbw

20) If the server replaced was originally the primary server, set it back as primary.
bwadmin@ne1$ peerctl setPrimary ne1

21) Verify that the information is consistent across all peers, with every peer unlocked.
bwadmin@ne1$ peercmd peerctl ls

[Link] Xtended Services Platform Replacement

1) Stop the BroadWorks application, the SNMP agent, the License Manager, and the
Configuration Agent on the replacement server.
bwadmin@host$ stopbw
bwadmin@host$ snmpdctl stop
bwadmin@host$ lmdctl stop
bwadmin@host$ configdctl stop

This restores the last backup of the production server’s configuration information to
the replacement server. Because the backup .tar file stores the relative paths of all
included files, you must extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you run the tar xvf command.
3) Make sure that valid [Link] file is installed on the server (install –[Link]
/path/to/[Link]).
4) Start the Configuration Agent, the License Manager, and the SNMP agent.
bwadmin@host$ configdctl start
bwadmin@host$ lmdctl start
bwadmin@host$ snmpdctl start

5) If applications were manually deployed or undeployed on the server to be replaced,

deploy or undeploy them again on the replacement server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

[Link] Profile Server Replacement

1) Stop the BroadWorks application, replication, the SNMP agent, the License Manager,
and the Configuration Agent on the replacement server if it is running.
bwadmin@host$ stopbw
bwadmin@host$ repctl stop
bwadmin@host$ snmpdctl stop
bwadmin@host$ lmdctl stop
bwadmin@host$ configctl stop

2) Re-mesh the SSH keys. On all peer servers, as root, run the following.
bwadmin@host$ rm -Rf /export/home/bwadmin/.ssh

As bwadmin, from either server, run the following.

bwadmin@host$ config-ssh -createKeys <other_peer_server>

<other_peer_server> can be the host name of the peer if the /etc/hosts file on the
current host is configured properly with the peer host name.
3) Configure the replication peers on the new replacement server.
If the peer server is still in service, as bwadmin, run the peerctl ls command on
the existing peer server host to get a list of peers. Following is an example of
peerctl ls from the existing server.
bwadmin@host$ peerctl ls
HOSTNAME/ADDRESS

On the replacement server, you must make sure that the peer list is the same as it is
on the in-service peer.
For example, if the server to be replaced has the host name ps1, the existing peer list
on the existing server should have one entry present set to “ps1”. However, that entry
may have a different host name defined, in which case you have to set it correctly.
Following is an example of the replacement server peerctl ls output.
bwadmin@host$ peerctl ls
HOSTNAME/ADDRESS State
-----------------------------------------------------------------------
backup/backup primary,unlocked

Given the above examples, the following commands would be run on the replacement
server to set the peers properly.
bwadmin@host$ peerctl add ps2 ps2

4) Copy the last BroadWorks system backup created from the server to be replaced to
any directory on the replacement server. As the root user, run the following
commands.
bwadmin@host$ gunzip <backupfile_name.gz>
bwadmin@host$ tar xvf <backupfile_name>

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 147 OF 247
This restores the last backup of the production server’s configuration information to
the replacement server. Since the backup .tar file stores the relative paths of all
included files, you must extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, then perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you ran the tar xvf command.

NOTE: HTTPS certificates and keys are not backed up by the system backup. Information
stored in /var/broadworks/ssl/certs /var/broadworks/ssl/private should be backed up and stored
remotely whenever they are changed by the provider. The HTTPS certificate and keys must be
manually restored on the new server.

5) Start the Configuration Agent, the License Maanger, and the SNMP agent by running
the following commands.
bwadmin@host$ configdctl start
bwadmin@host$ lmdctl start
bwadmin@host$ snmpdctl start

6) Make sure that valid [Link] file is installed on the server ([Link]
/path/to/[Link]).
7) Import the file repository files from the existing peer server, since they are not included
in the system backup. As the bwadmin user, run the following command on the
replacement server.
bwadmin@host$ rsync -vazurSHl <peerServer>:<file repository root>/. <file
repository root>/.

8) If applications were manually deployed or undeployed on the server to be replaced,

deploy or undeploy them again on the replacement server.
9) As the bwadmin user, run the following commands.
bwadmin@host$ repctl start
bwadmin@host$ startbw

[Link] Database Server Replacement

For information on the Database Server replacement procedure, see the BroadWorks
Database Configuration Guide.

[Link] Element Management System Replacement

NOTE: The replacement server must be installed as a stand-alone server before proceeding
with the following steps.

1) Stop BroadWorks on the replacement server.

bwadmin@host$ stopbw

2) Re-mesh the SSH keys. On all peer servers, as root, run the following.
bwadmin@host$ rm -Rf /export/home/bwadmin/.ssh

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 148 OF 247
3) As bwadmin, from either server, run the following.
bwadmin@host$ config-ssh -createKeys <bwadmin@other_peer_server>

<other_peer_server> can be the host name of the peer if the /etc/hosts file on the
current host is configured properly with the peer host name.
4) Configure the replication peers on the new replacement server.
If the peer server is still in service, as bwadmin, run the peerctl ls command on
the existing peer server host to get a list of peers. Following is an example of a
peerctl ls from the existing server.
bwadmin@host$ peerctl ls
PEER Role Status
========================================
ems01 PRIMARY NOTRUNNING
ems02 SECONDARY ACTIVE

On the replacement server, you must make sure that the peer list is the same as it is
on the in-service peer.
For example, if the server to be replaced was the primary server, you must configure
the replacement server as primary with the same peer name. To configure the
redundancy, run the following command on the replacement server as bwadmin.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/setup-redundancy

If the new replacement server is the primary server, you need to reset the database
replication configuration on both servers by running the following command as
bwadmin. If the replacement server is the secondary server, the database replication
information is already reset.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/[Link]

5) Stop the SNMP agent, the SNMP agent, and the Configuration Agent on the
replacement server.
bwadmin@host$ snmpdctl stop
bwadmin@host$ lmdctl stop
bwadmin@host$ configdctl stop

6) Copy the last BroadWorks system backup created from the server to be replaced to
any directory on the replacement server. As the root user, run the following
commands.
bwadmin@host$ gunzip <backupfile_name.gz>
bwadmin@host$ tar xvf <backupfile_name>

This restores the last backup of the production server's configuration information to
the replacement server. Since the backup .tar file stores the relative paths of all
included files, you must extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you ran the tar xvf command.
7) Start the Configuration Agent, the License Manager, and the SNMP agent.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 149 OF 247
bwadmin@host$ configdctl start
bwadmin@host$ lmdctl start
bwadmin@host$ snmpdctl start

8) Import the database on the new replacement server. As bwadmin, run the following
command.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/[Link] <other server
hostname>

9) If the server to be replaced has LDAP enabled and was the primary server, restore
the LDAP database on the replacement server. The LDAP database was backed up
with the last BroadWorks system backup and it is located under
/var/broadworks/backup. As root, run the following command.
bwadmin@host$ /usr/local/broadworks/bw_base/bin/[Link]
<BackupFile>

10) Make sure that valid [Link] is installed on the server (install-license
/path/to/[Link]).
11) If applications were manually deployed or undeployed on the server to be replaced,
deploy or undeploy them again on the replacement server.
12) Start the replication on both servers and start BroadWorks on the replacement server.
bwadmin@host$ repctl start
bwadmin@host$ startbw

[Link] Execution Server Replacement

1) Stop the BroadWorks application and the Configuration Agent on the replacement
server.
bwadmin@host$ stopbw
bwadmin@host$ configdctl stop

2) Copy the last BroadWorks system backup created from the server to be replaced to
any directory on the replacement server. As the bwadmin user, run the following
commands.
bwadmin@host$ gunzip <backupfile_name.gz>
bwadmin@host$ tar xvf <backupfile_name>

This restores the last backup of the production server’s configuration information to
the replacement server. Because the backup .tar file stores the relative paths of all
included files, you must extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you run the tar xvf command.
3) Make sure that valid [Link] file is installed on the server (install -[Link]
/path/to/[Link]).
4) Start the Configuration Agent.
bwadmin@host$ configdctl start

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 150 OF 247
5) If applications were manually deployed or undeployed on the server to be replaced,
deploy or undeploy them again on the replacement server.
6) Start BroadWorks.
bwadmin@host$ startbw

[Link] Network Function Manager Replacement

2) Re-mesh the SSH keys. On all peer servers, as root, run the following.
host$ rm -Rf /export/home/bwadmin/.ssh

As bwadmin, from either server, run the following.

bwadmin@host$ config-ssh -createKeys <other_peer_servers>

<other_peer_servers> can be the host name of the peer if the /etc/hosts file on the
current host is configured properly with the peer host name.
3) Configure the replication peers on the new replacement server.
If the peer server is still in service, as bwadmin, run the peerctl ls command on the
existing peer server host to get a list of peers. Following is an example of peerctl ls
from the existing server.
bwadmin@host$ peerctl ls
HOSTNAME/ADDRESS

On the replacement server, you must make sure that the peer list is the same as it is
on the in-service peer.
For example, if the server to be replaced has the host name nfm1, the existing peer
list on the existing server should have one entry present set to “nfm1”. However, that
entry may have a different host name defined, in which case you have to set it
correctly.
Following is an example of the replacement server peerctl ls output.
bwadmin@host$ peerctl ls
HostName Net Address
==========================
nfm1 nfm1

Given the previous examples, the following commands would be run on the
replacement server to set the peers properly.
bwadmin@host$ peerctl add nfm2 nfm2
bwadmin@host$ peerctl add nfm3 nfm3

4) Copy the last BroadWorks system backup created from the server to be replaced to
any directory on the replacement server. As the root user, run the following
commands.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

This restores the last backup of the production server’s configuration information to
the replacement server. Since the backup .tar file stores the relative paths of all
included files, you must extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, then perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you ran the tar xvf command.
5) Start the Configuration Agent, the License Manager, and the SNMP agent by running
the following commands.
bwadmin@host$ configdctl start
bwadmin@host$ lmdctl start
bwadmin@host$ snmpdctl start

6) Make sure that a valid [Link] file is installed on the server ([Link]
/path/to/[Link]).
7) If applications were manually deployed or undeployed on the server to be replaced,
deploy or undeploy them again on the replacement server.
8) As the bwadmin user, run the following commands.
bwadmin@host$ repctl start
bwadmin@host$ startbw

[Link] Video Server Replacement

This restores the last backup of the production server’s configuration information to
the replacement server. Because the backup .tar file stores the relative paths of all
included files, you must extract the archive (restore the server) from an appropriate
directory. If you want to restore files to the directory in which they were originally
located when the backup was made, perform the restore from the system root
directory (/). Otherwise, the backup files are simply extracted relative to the directory
from which you run the tar xvf command.
3) Make sure that valid [Link] file is installed on the server ([Link]
/path/to/[Link]).
4) Start the Configuration Agent, the License Manager, and the SNMP agent.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 152 OF 247
bwadmin@host$ configdctl start
bwadmin@host$ lmdctl start
bwadmin@host$ snmpdctl start

5) If applications were manually deployed or undeployed on the server to be replaced,

deploy or undeploy them again on the replacement server.
6) Start BroadWorks.
bwadmin@host$ startbw

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

System administrators can determine if the “Remember Password” function, in which a

cookie is set on the user’s PC, is available. The system provider can remove the
Remember Password check box from the login web page and refuse any existing user ID
Password cookies.

13.1 Modify [Link] File

The optional cookie parameter is modified manually through the CLI. Since this parameter
is not replicated, the change needs to be propagated manually to all Application Servers
and Network Servers in a cluster. The setting is preserved during an upgrade.

13.1.1 Disable Cookie Support

1) Log in to the BroadWorks server using a telnet console.
2) Enter the CLI (bwcli) and set userCookieSupportEnabled to “false”.
NS_CLI/Applications/NSPortal/GeneralSettings> get
sslMode = on
loginAuthorizationLevel = all
userCookieSupportEnabled = true

NS_CLI/Applications/NSPortal/GeneralSettings> set
userCookieSupportEnabled false
*** Warning: The WebContainer application needs to be restarted for the
changes
to take effect ***

3) Restart the WebContainer application (restartbw WebContainer).

13.1.2 Enable Cookie Support

1) Log in to the BroadWorks server using a telnet console.
2) Enter the CLI (bwcli) and set userCookieSupportEnabled to “true”.
NS_CLI/Applications/NSPortal/GeneralSettings> get
sslMode = on
loginAuthorizationLevel = all
userCookieSupportEnabled = false

NS_CLI/Applications/NSPortal/GeneralSettings> set
userCookieSupportEnabled true
*** Warning: The WebContainer application needs to be restarted for the
changes
to take effect ***

3) Restart the WebContainer application (restartbw WebContainer).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

There are few ways to install a BroadWorks license file. In each case, the license file is
installed in the license store managed by the License Manager. The installation of the
license file in the license store is done automatically following any of these methods:
 Installed during BroadWorks installation though the installation prompt.
Do you wish to configure a license file (zip file path)? y
What is the license file path? /export/home/bwadmin/[Link]

 Installed using the [Link] script.

/usr/local/broadworks/<BW Version>/bin/[Link]
export/home/bwadmin/[Link]

 Installed through the CLI.

CLI/System/Licensing/LicenseManager/LicenseStore> import
/export/home/bwadmin/[Link]
Done.

Note that your BroadWorks license includes software from other companies. For details,
consult your license agreement from BroadSoft.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

As BroadWorks servers are upgraded, old BroadWorks versions are not deleted but they
remain on the server. BroadSoft recommends that after an upgrade, the current active
version and the last version should be kept on the server (at a minimum). Older
BroadWorks versions can be uninstalled using the following procedure.
To uninstall an obsolete server version (for example, NS_Rel_19.0_1.131 on the Network
Server):
1) Log in (or su -) to root.
2) Change the directory (cd) to
/var/broadworks/<current_release>/uninstall and run the uninstall-
[Link] script.
3) Type # /[Link] -r. The “-r” option directs the script to
remove the corresponding /var/broadworks release directory. The following response
appears (sample provided).
#./[Link] -r
Select which release to remove (0 to stop the process)

0 - none
1 - NS_Rel_11.0_1.131
2 - NS_Rel_12.0_1.411
3 - NS_Rel_13.0_1.362
4 - NS_Rel_14.0_1.606

Enter selection [0] ?1

You are about to remove broadworks release NS_Rel_11.0_1.131

Do you want to continue? (y/n) [n]?y

Un-Installing BW Server, version NS_Rel_11.0_1.131...

Removing /var/broadworks/NS_Rel_11.0_1.131

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

This section describes the maintenance steps to be performed on a BroadWorks server

when changing an IP address, host name, FQDN name, gateway, or DNS server.
Following are the steps, on a per-server basis, that must be executed when changing a
BroadWorks server IP address, host name, or FQDN name, gateway, or DNS server:
Application Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Re-configure redundancy routing, if required.
5) Update the Application Server call processing routing information.
6) Update the Application Server address information on the Network Server, if required.
7) Update the Application Server address information on the Open Client Server (OCS)
application, and Xtended Services Platform, if required.
8) Re-run the EMS auto-discover process, if applicable.
9) Start BroadWorks.
Element Management System:
1) Stop BroadWorks.
1) Stop replication.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Update EMS redundancy configuration, if applicable.
5) Start replication.
6) Import the EMS database, if applicable.
7) Start BroadWorks.
Media Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) If deployed with the Network Server Media Server Selection policy:
4) Update the Media Server address on the Network Server, if required.
5) If NOT deployed with the Network Server Media Server Selection policy:
Update the Media Server address on the Application Server, if required,
6) Re-run the EMS auto-discover process, if applicable.
7) Start BroadWorks.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 157 OF 247
Network Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Re-configure redundancy routing, if required.
5) Update the Network Server routing alias list, if required.
6) Re-run the EMS auto-discover process, if applicable.
7) Update the Network Server address information on the Application Server, Open
Client Server application, and Xtended Services Platform, if required.
8) Start BroadWorks.
Xtended Services Platform:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Update the Application Server or Profile Server access lists (OCI and ExtAuth) with
the Xsp address, if required.
5) Rerun the EMS auto-discover process, if applicable.
6) Start BroadWorks.
Profile Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the Profile Server HTTP Server configuration, creating new HTTP servers for
the new IP and deleting old ones. This is done from the
PS_CLI/Interface/Http/HttpServer CLI level. For more information, see the
BroadWorks Profile Server Configuration Guide.
4) Update the Profile Server HTTP Alias with the new host name as required. This is
done from the PS_CLI/Interface/Http/HttpAlias CLI level. For more information, see
the BroadWorks Profile Server Configuration Guide.
5) Update the Profile Server address information on the Xtended Services Platform, if
required.
6) Rerun the EMS auto-discover process, if applicable.
7) Start BroadWorks.
Execution Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the Execution Server address information on the Network Server, if required.
4) Rerun the EMS auto-discover process, if applicable.
5) Start BroadWorks.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 158 OF 247
Messaging Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Rerun the EMS auto-discover process, if applicable.
5) Start BroadWorks.
Sharing Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Rerun the EMS auto-discover process, if applicable.
5) Start BroadWorks.
Video Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Rerun the EMS auto-discover process, if applicable.
5) Start BroadWorks.
WebRTC Server:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Rerun the EMS auto-discover process, if applicable.
5) Start BroadWorks.
Network Function Manager:
1) Stop BroadWorks.
2) Configure the IP networking information at the UNIX level.
3) Update the BroadWorks network configuration.
4) Start BroadWorks.
Database Server:

NOTE 1: IP address/host name configuration on the Database Server depends on the

deployment model and as a result, requires extra care. Changing the Database Server host
name is only possible in single instance deployments; changing the host names of a locally
redundant cluster configuration is not supported.

NOTE 2: In a geographically redundant configuration, the host name change must be done
while the site status is “standby” and while the DBSObserver is shut. Host name modification of
the primary site is not supported.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 159 OF 247
To change the Database Server host name, the following sequence must be executed.
1) On the Profile Server, stop the DBSObserver (if configured).
bwadmin@ps$ stopbw DBSObserver

2) On the Database Server, create a hostname alias using the new host name. This can
be done by adding a CNAME record to your DNS or by adding the new host name to
the host name line in /etc/hosts, as shown in the following example.
# Do not remove the following line, or various programs
# that require network functionality will fail.
[Link] [Link] localhost

[Link] [Link] [Link]

NOTE: This alias must be available to all sites in the redundant configuration and must be
defined before the old name; otherwise, during reconfiguration the old name is still used.

3) While keeping BroadWorks running, run the following command and follow the
instructions it provides.
bwadmin@oldname$ sitectl config hostname=[Link]
Checking parameters... [DONE]

WARNING: Reconfiguring the hostname on the Database Server requires some

preparation steps.
Refer to the BroadWorks Maintenance Guide for details.

Do you still want to proceed? (y/n) [y]?y

Checking hostname... [DONE]
Checking host reachability... [DONE]
Checking host alias... [DONE]
Checking ssh meshing... [DONE]
Checking oracle env... [DONE]
Checking deployment type... [SI]
Checking redundancy... [DONE]
Checking database readiness ... [DONE]
Getting ASM disk groups... [DONE]
Getting ASM spfile... [DONE]
Getting primary db name... [DONE]
Getting current listener port number... [DONE]
Updating database listener... [DONE]
Updating listener host config... [DONE]
Updating database TNS ... [DONE]
Cleaning up database TNS on [Link]... [DONE]
Deconfiguring Oracle Grid... [DONE]
The new host name must now be activated.
Please login as 'root' and set the new system hostname using 'hostname
[Link]'.
During this time, the host must be accessible using both
[Link] and [Link].
Once done, press <enter> to continue.

Configuring Oracle Grid for new hostname... [DONE]

Adding basic Oracle Grid services... [DONE]
Registering database 'bwCentralizedDb1' with Oracle Grid... [DONE]
Adding service 'bwCentralizedDb' for database... [DONE]
Disabling fast_start failover... [DONE]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 160 OF 247
Changing protection mode... [DONE]
Disabling data guard db... [DONE]
Update data guard db host... [DONE]
Re-enabling data guard db... [DONE]
Restoring protection mode ... [DONE]
Re-enabling fast_start failover... [DONE]
The database is now configured to use the new host name
'[Link] .
You must now restart BroadWorks, refer to the BroadWorks Maintenance
Guide for details.

4) Stop BroadWorks.
bwadmin@newname$ stopbw

5) Drop the old host name from the name service configuration, keeping only the new
name.
# Do not remove the following line, or various programs
# that require network functionality will fail.
[Link] [Link] localhost

[Link] [Link] [Link]

6) Start BroadWorks.
bwadmin@newname$ startbw

7) Update the Database Server address information on the Profile Server and Execution
Server, if required.
8) Start the DBSObserver, if applicable.
9) Manually add the Database Server to the EMS configuration.

16.1 Configure IP Networking Information at UNIX Level

NOTE: Read the introduction section before executing the steps described in this section, as
other actions may be required.

The following steps are required:

1) Log in as or su to root.
2) Change the following files: /etc/hosts (IP address on line that includes loghost).
If you are modifying IP addresses for dual homing, IP multipathing, or redundant
networking, those IP addresses must also be changed here.
If changing the host name, these files must also be modified.
/etc/nodename

/etc/[Link]#, where hme# is the network interface device. This may be

dmfe# or qfe# on some systems, depending on the Ethernet hardware available.
There are multiple files for the system configured to use multiple interfaces (for
example, hostname, hme0, hostname.hme1, and so on).
/etc/net/ticlts/hosts
/etc/net/ticots/hosts

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

16.2 Update BroadWorks Network Configuration

16.2.1 Restart the configuration agent

Prior to update the HTTP server network configuration, you must restart the configuration
agent.
MTLNS04$ configdctl restart
Restarting configd...Setting system info values
Patching configuration...
[ok]
[ok]

16.2.2 HTTP Server Network Configuration

The HTTP network configuration on a BroadWorks server is all done through the CLI. The
contexts under CLI/Interface/Http are used for such configuration. The httpServer level
allows the addition of interfaces and ports on which the HTTP server is listening.
Following is an example.
NS_CLI/Interface/Http/HttpServer> get
Interface Port Name Secure Cluster
FQDN
=========================================================================
================
[Link] 443 [Link] true
[Link]
[Link] 80 [Link] false
[Link]

2 entries found.

The httpAlias level allows the definition of host alias for a given interface.
NS_CLI/Interface/Http/HttpAlias> get
Interface Alias Cluster FQDN
========================================================
[Link] [Link]

1 entry found.

Finally, the HttpBinding level allows the specification as to which web application is bind on
which interface. This functionality can be used to restrict the availability of a web
application on a specific interface. By default, httpBinding is empty, allowing any web
application to run on any interface/port combinations.
NS_CLI/Interface/Http/HttpBinding> get
Interface Port Application Name
=======================================
[Link] 80 NSPortal

1 entry found.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

The Application Server, Network Server, and Messaging Server redundancy routing
configuration have to be modified when changing a server host name or IP address. This
is done from the UNIX prompt, by executing the peerctl command.
It is important to execute the commands in the following order:
1) Stop replication on all servers of the cluster using Repctl stop.
2) Execute the peerctl command on all servers of the cluster.
The following is an example of the peerctl command.
MTLNS04$ peerctl
Usage:
(When the hostname is optional, omitting it means the current peer)
[-dsn <dsn>] ls
[-dsn <dsn>] add <hostname> <address> <repType(0=master;1=slave)>
[-dsn <dsn>] set <originalhostname> <hostname> <address>
<repType(0=master;1=slave)>
[-dsn <dsn>] del <hostname>
[-dsn <dsn>] lock [<hostname>|all]
[-dsn <dsn>] unlock [<hostname>|all]
[-dsn <dsn>] setPrimary [<hostname>]
[-dsn <dsn>] waitAllLocked <timeout in minutes (typically 12)>
NOTE: Locking a server can be almost immediate or may take up to 10+
minutes
depending on server activity.

MTLNS04$ peerctl set mtlnsold mtlnsnew [Link] repType 0

3) Restart replication on all servers of the cluster, using Repctl start.

4) Note that, for an Application Server, it may be required to re-import that database on
the secondary server, in cases such as when the replication is stopped for an
extended period.

16.4 Update Application Server Call Processing Routing Information

Some start-up parameters may have to be modified for the SIP and MGCP signaling to
function properly after changing the address of an Application Server. These parameters
can be modified (if present) under the System/StarupParm level.
Parameter Description

publicIPAddress This is the default IP address used by the Application Server for the
SIP stack and other applications. On a server with a single NIC, this is
the only parameter configured.

privateIPAddress On multi-homed server, this is used to specify the IP address to use on

the access side.

[Link]. On multi-homed server, this is used to specify the IP (interface) to use

accessinterfaceviahost on the access side. This parameter takes precedence over the
privateIPAddress parameter for SIP signaling.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

[Link]. On multi-homed server, used to specify the IP (interface) to use on the

networkinterfaceviahost network side. This parameter takes precedence over the
publicIPAddress parameter for SIP signaling.

NOTE: It is important to validate and change (if required) the parameters listed above whenever
an IP address or host name is changed for an Application Server.

16.5 Update Application Server IP Address on Network Server

NOTE: Read the introduction section before executing the steps described in this section, as
other actions may be required.

The Application Server IP address could be configured on the Network Server. After
making changes to the IP address on the Application Server, the Network Server may also
need to be updated. If the Application Server address on the Network Server is an FQDN
or an alias, this step can be skipped.
1) Log in to the Network Server command line interface.
2) Go to the NS_CLI/System/Device/HostingNE/Address> level.
3) Add the new IP address of the Application Server.
4) Delete the old IP address of the Application Server.
The following is an example of updating the Application Server IP address on the
Network Server.
NS_CLI/System/Device/HostingNE/Address> g
Retrieving data… Please wait…
HostingNe NodeID Address type cost weight port
transport
=======================================================================
==
AS1 0 [Link] DualRouting 1 99 5060
tcp
1 entry found.
NS_CLI/System/Device/ResourceNE/Address> add AS1 0 [Link]
DualRouting 1 99 5060 tcp
…Done

NS_CLI/System/Device/ResourceNE/Address> delete AS1 0 [Link]

…Done

NS_CLI/System/Device/HostingNE/Address> g
Retrieving data… Please wait…
HostingNe NodeID Address type cost weight port
transport
=======================================================================
==
AS1 0 [Link] DualRouting 1 99 5060
tcp

1 entry found.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

The Network Server IP address can be configured on the Application Server. After
making changes to the IP address on the Network Server, the Application Server may also
need to be updated. If the Network Server address on the Network Server is an FQDN or
an alias, this step can be skipped.
1) Log in to the Application Server command line interface.
2) Go to the AS_CLI/System/Device/NetworkServers/Routing level.
3) Add the new IP address of the Network Server.
4) Delete the old IP address of the Network Server.
The following is an example of updating the Application Server IP address on the
Network Server.
AS_CLI/System/Device/NetworkServers/Routing> get
Net Address Port Transport Poll OpState Description
=========================================================================
[Link] 5060 UDP false enabled MTLNS01-DNSlong
1 entry found.
AS_CLI/System/Device/NetworkServers/Routing> add [Link] tcp
description “MTLNS04-DNSlong” port 5060
…Done

AS_CLI/System/Device/NetworkServers/Routing> delete [Link]

…Done

Net Address Port Transport Poll OpState Description

=========================================================================
[Link] 5060 UDP false enabled MTLNS01-DNSlong
1 entry found.

16.7 Update Network Server Routing Alias List

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

The Network Server aliases may need to be updated when modifying a Network Server IP
address.
1) Log in to the Network Server command line interface.
2) Go to the NS_CLI/System/Alias level.
3) Add the new address or domain to the Network Server alias list.
4) Delete the old IP address or domain from the Network Server alias list.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 165 OF 247
The following is an example of updating the Network Server alias IP address on the
Network Server.
NS_CLI/System/Alias> get
[Link]
[Link]
NS_CLI/System/Alias> add [Link]
…Done

NS_CLI/System/Alias> delete [Link]

…Done

NS_CLI/System/Alias> get
[Link]
[Link]

16.8 Update Media Server IP Address on Network Server

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

A Network Server for Media Server Selection can also use the Media Server IP address.
After making changes to the IP address on the Media Server, the Network Server needs
to be updated as well.
1) Log in to the Network Server command line interface.
2) Go to NS_CLI/System/Device/ResourceNE/Address level.
3) Add the new IP address of the Media Server.
4) Delete the old IP address of the Media Server.
The following is an example of updating the Media Server IP address on the Network
Server.
NS_CLI/System/Device/ResourceNE/Address> g
Retrieving data… Please wait…
Resource NE Address Cost Weight Port Transport
MS1 [Link] 1 10 5679 tcp

NS_CLI/System/Device/ResourceNE/Address> add MS1 [Link] 1 10 5679

tcp
…Done

NS_CLI/System/Device/ResourceNE/Address> delete MS1 [Link]

…Done

NS_CLI/System/Device/ResourceNE/Address> g
Retrieving data… Please wait…
Resource NE Address Cost Weight Port Transport
MS1 [Link] 1 10 5679 tcp

1 entry found.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

The Media Server IP address is most likely being used by an Application Server to stream
media. After changing the IP address on the Media Server, the Application Server needs
to be updated as well.
1) Log in to the Application Server command line interface.
2) Go to the AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice
level.
3) Add the new IP address of the Media Server.
4) Delete the old IP address of the Media Server.
The following is an example of updating the Media Server IP address on the
Application Server.
AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice> get
Net Address Port Transport Description
===========================================
[Link] 5060 TCP
1 entry found.

AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice>add
[Link] tcp port 5060
…Done

AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice>delet
e [Link]
…Done

AS_CLI/System/CallP/Routing/MediaServerSelection/MediaServerDevice> get
Net Address Port Transport Description
===========================================
[Link] 5060 TCP
1 entry found.

16.10 Update Application Server and Network Server Address on OCS

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

When updating the Application Server and/or Network Server IP address, the
Communication Utility IP address on the Application Server, Xtended Services Platform,
and/or EMS (if applicable) must be updated.
1) Log in to the BroadWorks command line interface.
2) Go to the System/CommunicationUtility/DefaultSettings level.
3) Depending on the mode, change the IP address of either the primary or secondary
Application Server or the Network Server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 167 OF 247
The following is an example of an update of an Application Server IP address for the
Communication Utility.
AS_CLI/System/CommunicationUtility/DefaultSettings> get
AS:
asPrimaryAddress=localhost
asSecondaryAddress=
asOCIPort=2220
asOCISecurePort=2320
asOCICPort=2221
asOCICSecurePort=2321
provisionOnSecondary=false
reconnectionTimerSecs=30
responseTimeoutSecs=10
globalTransactionLimit=100
userTransactionLimit=5
transactionLimitPeriodSecs=1
useSecureBCCT=false
allowExtAuthAgent=false

AS_CLI/System/CommunicationUtility/DefaultSettings> set mode AS

[Link] asSecondaryAddress [Link]

*** Warning: BroadWorks needs to be restarted for the changes to take

effect ***

16.11 Update Xtended Services Platform IP Address on Network Server

NOTE: Read the introduction section prior to executing the steps described in this section, since
other actions may be required.

The Xtended Services Platform address must be properly configured for the Location
Server function on the Network Server to work. After making changes to the IP address
on the Xtended Services Platform, the Network Server must be updated as well.
1) Log in to the Network Server command line interface.
2) Go to the NS_CLI/System/Device/XSPServerFarm/Address level.
3) Add the new IP address of the Xtended Services Platform.
4) Delete the old IP address of the Xtended Services Platform.
The following is an example of an update of the Xtended Services Platform IP
address on the Network Server.
NS_CLI/System/Device/XSPServerFarm/Address> get
XSPServerFarm NodeID Address type cost weight
============================================================
WSF 0 [Link] Access 1 50
WSF 0 mtlws01 Alias - -
WSF 1 [Link] Access 1 50
WSF 1 mtlws02 Alias - -

4 entries found.

NS_CLI/System/Device/XSPServerFarm/Address> add WSF 1 [Link]

Access 1 50
...Done

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NS_CLI/System/Device/XSPServerFarm/Address> get
XSPServerFarm NodeID Address type cost weight
============================================================
WSF 0 [Link] Access 1 50
WSF 0 mtlws01 Alias - -
WSF 1 [Link] Access 1 50
WSF 1 mtlws02 Alias - -

4 entries found.

16.12 Update Network Server or Application Server Address on Xtended Services

Platform

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

The BwCommunicationUtility component on the Xtended Services Platform requires

knowledge of the Network Server or Application Server depending on the system setup.
For more information, see the BroadWorks Xtended Services Platform Configuration
Guide.

16.13 Update EMS Redundancy Configuration

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

On a redundant EMS system, redundancy information such as the peer host name and
address need to be modified to represent the new environment. Run the update-
[Link] script from the UNIX prompt on all peers.
./[Link]
Usage: [Link] oldValue newValue

This script is used when the hostname or the IP address has changed for
one peer.

Parameters
-------
oldValue - The previous value.
newValue - The new value.

Example:
./ [Link] vmdavid01 vmdavid02
Updating local peer address.........[Ok]
Updating peer hostname..............[Ok]
Updating peer address...............[Ok]
Updating db replication info........[Already up-to-date]
Updating privileges.................[Ok]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: Read the introduction section prior to executing the steps described in this section, as
other actions may be required.

On the redundant EMS system, the database must be synchronized on all peers to make
sure that the replication is not lagging. If an IP address has changed, the database may
no longer be synchronized.
Import the database on the peer that has changed their IP or host name (using the
[Link] command from the UNIX prompt).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

This section describes the maintenance steps to be performed on a BroadWorks server

when the BroadWorks Common Communication Transport (BCCT) listening ports
XSListeningPort, PSListeningPort, XSSecureListeningPort, and PSSecureListeningPort
are changed on the Application Server or on the Network Server.
The BCCT listening ports are configurable from the AS_CLI/
/Interface/CommonCommunicationTransport level or from the NS_CLI/
/Interface/CommonCommunicationTransport level.
It is strongly recommended to change the BCCT listening ports of all Application Servers
and Network Servers to use the same values for the BCCT listening ports.
Following are the steps that must be executed after the BCCT listening ports are changed,
on a per-server basis:
Element Management System:
1) Update the BCCT port number used by the cut-through session functionality to the
new value of “PSListeningPort” or “PSSecureListeningPort” for each server being
modified. This is done from the EMS Web Client. (For more information, see the
BroadWorks Element Management System User Guide Part 2.)
2) Update the BCCT port number used by the Auto-Discovery functionality to the new
value of “PSListeningPort” or “PSSecureListeningPort”. This is done from the EMS
Web Client. (For more information, see the BroadWorks Element Management
System Administration Guide, Part 1.)
Network Server:
1) Exit all CLIs.
2) Restart BroadWorks.
Xtended Services Platform:
1) Update the Xsp communication utility default settings and change the value of the
asOCIPort and asOCISecurePort corresponding to the new value of
“PSListeningPort” and “PSSecureListeningPort” respectively. This is done from the
XSP_CLI/ /System/CommunicationUtility/DefaultSettings CLI level. For more
information, see the BroadWorks Xtended Services Platform Configuration Guide.
2) Restart BroadWorks.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Proper BroadWorks system monitoring is comprised of three distinct tasks:

 Platform Resource Monitoring – Proactively monitoring key platform resource
indicators to make sure they are operating within acceptable ranges.
 Performance Measurement Monitoring – Proactively polling key SNMP gauges
and counters to gather information on a server’s health.
 SNMP Trap Monitoring – Reactively responding to traps generated by the
BroadWorks servers.
This section outlines the system monitoring that must be performed to make sure proper
BroadWorks operations.

18.1 Platform Resource Monitoring – Key Resource Indicators

The health of BroadWorks relies on the health of the servers that it runs on. The servers
must be monitored to make sure they are operating correctly. The following key resource
indicators should be monitored:
 File system (disk usage)
 CPU usage
 Memory usage/Paging
 Disk activity
 Execution Server heap usage (Application Server and Network Server)
 Database size (Application Server and Network Server)
As of Release 14.sp1, the BroadWorks Element Management System (EMS)
automatically collects key resource indicators for all BroadWorks nodes.

NOTE: All servers should be running with key resource indicators that are within acceptable
ranges. Acceptable ranges for each of the key resource indicators listed above are defined in
the BroadWorks System Engineering Guide.

18.1.1 File System Monitoring

It is critical that the server file system does not reach 100% utilization; otherwise, the
application ceases to function. In general, disk partition should be running under 75%
utilization.

[Link] Collecting File System Indicator

File system use can be obtained through the following means.

[Link].1 Command Line

File system usage can be viewed manually from the UNIX/LINUX prompt using the df –k
command. The Capacity field indicates the percentage of partition in use.
IHApp$ df –k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 1987399 1208093 719685 63% /
/proc 0 0 0 0% /proc

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 172 OF 247
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 1395576 16 1395560 1% /var/run
swap 1395616 56 1395560 1% /tmp
/dev/md/dsk/d13 13376379 6977170 6265446 53% /bw

[Link].2 SNMP
On Linux systems, file system usage can also be monitored through SNMP by polling the
[Link] MIB file register dskPercent.

[Link] Healthmon File System Check

The healthmon script automatically monitors the /, /bw, and /tmp partitions (if present),
every time it runs and generates the following traps severities:
 Low severity trap at 80% occupancy on any partition
 High severity trap at 90% occupancy on any partition
 Critical severity trap at 95% occupancy on any partition
 Critical severity trap and a stopbw at 98% occupancy on any partition
The healthmon script stops the BroadWorks application at 98% partition occupancy to
avoid possible database corruption.

18.1.2 CPU Usage Monitoring

System CPU usage must be monitored to make sure proper BroadWorks operation. High
CPU usage bottlenecks the application and can lead to outages.
CPU usage is generally broken down into the following run time percentages:
 User – The percentage of time the CPU is spending on user processes, such as
applications, shell scripts, or interacting with the user
 System – The percentage of time the CPU is spending executing kernel tasks
 I/O Wait – The percentage of time the CPU is waiting for input or output from a block
device, such as a disk
 Idle – The percentage of time the CPU is not doing anything useful
For Linux systems, CPU busy occupancy is defined as the sum of CPU User time and
CPU System time. I/O Wait time, although possibly indicative of an I/O issue, does not
contribute to CPU busy and it should be considered as “Idle” time. Certain monitoring
utilities already consider this and include I/O Wait in the “Idle” value.

[Link] Collecting CPU Usage Indicator

Busy CPU percentage (User Time + System Time) can be collected through various
means. It is recommended that averaged CPU usage be gathered as opposed to
instantaneous CPU usage values. BroadSoft recommends a five-minute averaging
window.
CPU usage can also be monitored through SNMP on Linux by polling UCD-SNMP-
[Link], but this is not the recommended approach since it reports an instantaneous CPU
usage ready that may be misleading.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 173 OF 247
The recommended methodology for tracking CPU usage is through either System Activity
Reporter (SAR) or vmstat utilities with a five-minute average.

[Link].1 System Activity Reporter

The System Activity Reporter (SAR) tool provides historical information on system activity
such as CPU usage, paging, and swap space usage. SAR should be activated on all
servers with a five-minute collection window. This provides an excellent mechanism to
retrieve CPU averages for servers. For information on SAR activation, consult the
BroadWorks Software Management Guide.
To view CPU average usage activity, from the UNIX prompt type sar.
Example:
as1$ sar

SunOS IHApp 5.8 Generic_108528-19 sun4u 08/08/03

00:00:01 %usr %sys %wio %idle

00:05:00 25 5 12 57
00:10:00 27 5 15 53
00:15:01 22 4 11 62
00:20:00 27 6 12 55
……..
11:45:00 28 6 10 56
11:50:00 34 7 9 50
11:55:01 35 7 10 48
12:00:01 29 7 10 54
12:05:01 29 7 17 47

Average 27 5 14 53

Total CPU Busy percentage for any given five-minute interval would be equal to %usr +
%sys.
Historical SAR results are stored in /var/log/sa on Linux systems. Historic files can be
viewed using the sar –f <filename> option.

[Link].2 vmstat
The vmstat utility can be used to manually gather averaged CPU information.
To collect five-minute averaged CPU usage, from the UNIX/LINUX prompt type vmstat
300 <count>. The <count> parameter would be the number collections to perform.
Example:
as1$ vmstat 300 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us sy id
0 0 0 3351880 162248 30 160 23 3 4 0 1 1 0 5 1 420 506 738 3 2 95
0 0 0 3489880 255752 51 322 0 1 1 0 0 1 0 3 1 411 2242 1454 3 2 95
0 0 0 3490504 255328 31 197 0 1 1 0 0 1 0 3 1 367 1845 1392 2 1 97
0 0 0 3494720 257776 15 88 0 1 1 0 0 1 0 3 1 377 1584 1408 2 1 97
0 0 0 3492504 257352 66 436 1 2 2 0 0 2 0 4 2 395 2637 1478 5 2 92

Average CPU Busy percentage for each five-minute interval would be equal to us + sy.
The first reading returned by vmstat should to be discarded.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 174 OF 247
[Link] bwCPUMon script CPU Monitoring
The bwCPUMon scheduled task uses SAR the monitor average CPU usage. The
bwCPUMon script should be enabled to run every six minutes and generates the following
severity traps when the SAR five-minute %idle average surpasses the following
thresholds:
 MINUMUM_CPU_IDLE_TIME_MEDIUM=30
 MINUMUM_CPU_IDLE_TIME_HIGH=20
 MINUMUM_CPU_IDLE_TIME_CRITICAL=10

18.1.3 Memory Usage/Paging Monitoring

A lack of platform memory can slow the BroadWorks application down and increases CPU
usage. A server that has exhausted physical memory starts swapping. Swapping refers
to moving processes in and out of physical memory when the system is extremely short of
physical memory needed for use by the processes. Swapping places a heavy load on
CPU, disk I/0 subsystems, and Java. Ideally, the BroadWorks server should not be
swapping.
The easiest way to monitor memory swapping is to monitor server paging activity. Ideally,
there should be no paging activity.

[Link] Collecting Paging Indicator

The SAR utility provides an indication of paging activity on Linux system.

[Link].1 Linux – SAR Paging

To view five-minute averaged paging activity data, from the LINUX prompt enter sar -W.
The pswpin/s parameter provides an indication of average paging activity, which ideally
should be zero.
Example:
bwadmin@serv1$ sar -W
Linux [Link] (serv1) 01/31/2007

12:00:01 AM pswpin/s pswpout/s

12:05:01 AM 0.00 0.00
12:10:01 AM 0.00 0.00
12:15:01 AM 0.00 0.00
12:20:01 AM 0.00 0.00
12:25:01 AM 0.00 0.00
12:30:01 AM 0.00 0.00

[Link] bwCPUMon Script Paging Monitoring

The bwCPUMon scheduled task uses SAR to monitor average paging activity. On Linux
systems, the script generates a Critical severity trap when the SAR five-minute pswpin/s
average is two or greater for last three five-minute windows.

18.1.4 Disk Activity

Although BroadWorks applications tend not to be I/O blocking, excessive disk activity can
be an indication of too much I/O traffic, such as writing to disk. Excessive writing to disk
can be caused by a number of correctable activities such as logging levels set too high for
a server. It is important to monitor disk activity to make sure that disks are not
bottlenecked.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 175 OF 247
[Link] Collecting Disk Activity Indicator
The iostat utility provides an indication of I/O activity on Linux systems.

[Link].1 Linux – iostat

To view five-minute averaged disk activity data, from the LINUX prompt enter iostat –
dx 300 2. The svctm (average service time of the disk) and %util (percentage of time
the disk is busy) parameters provides an indication of disk activity. The first reading
should be discarded.
Example:
bwadmin@serv1$ iostat -dx 300 2
Linux [Link] (serv1) 02/01/2007

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s

avgrq-sz avgqu-sz await svctm %util
sda 0.00 14.23 0.01 3.64 0.44 142.94 0.22 71.47
39.29 0.02 4.63 0.52 0.19
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
57.70 0.00 3.42 3.42 0.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s

avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.28 0.00 0.31 0.00 4.67 0.00 2.33
15.22 0.00 1.01 0.66 0.02
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00

18.1.5 Thread Activity

Whenever the server processors are overstressed, the run queue can exceed the number
of CPUs, which can lead to performance bottlenecks. It is important to monitor thread
activity to make sure that processes are not bottlenecked. The number of threads waiting
in the run queue versus available CPUs can be measured to see if bottlenecks are
occurring.

[Link] Collecting Thread Activity Indicator

To view five-minute averaged thread activity, from the UNIX or LINUX prompt enter
vmstat 300 2. The r (number of threads waiting in the run queue) and b (number of
threads blocked due to I/O blocking) parameters provide an indication of thread activity.
The first reading should be discarded.
Example:
as1$ vmstat 300 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us sy id
0 0 0 3351880 162248 30 160 23 3 4 0 1 1 0 5 1 420 506 738 3 2 95
0 0 0 3489880 255752 51 322 0 1 1 0 0 1 0 3 1 411 2242 1454 3 2 95
0 0 0 3490504 255328 31 197 0 1 1 0 0 1 0 3 1 367 1845 1392 2 1 97
0 0 0 3494720 257776 15 88 0 1 1 0 0 1 0 3 1 377 1584 1408 2 1 97
0 0 0 3492504 257352 66 436 1 2 2 0 0 2 0 4 2 395 2637 1478 5 2 92

The number of CPUs on Linux servers can be obtained using the cat /proc/cpuinfo
command.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 176 OF 247
18.1.6 Application Server Execution Server Heap Monitoring
The Application Server Execution Server (XS) process is assigned a fixed amount of heap
memory at startup. The heap usage grows over time until Java cleans up the heap with a
full garbage collection. The Application Server Execution Server (XS) heap is cleaned up
whenever the heap reaches 68% of maximum allocated heap.
The heap size after the full collection represents the actual heap required to support the
users and active calls at the time. It is important that the post clean up heap does not
grow to 68% of the maximum allocated heap or the heap ends up in a full collection loop,
which severely affects server performance.
The post Java garbage collection heap size needs to be monitored daily and should
always be <= 60% of the allocated heap. As the heap increases towards 60%, it may be
possible to increase the maximum allocated heap by adding more memory to the server.

[Link] Collecting Heap Size Indicator

The following information should be collected from the primary Application Server since it
should be handling all traffic. Heap garbage collection (GC) logging information is stored
in /var/broadworks/logs/appserver/[Link] files. The GC line AFTER any line
containing “CMS-concurrent-reset:” indicates the heap size after a full garbage collection.
For example, the GC line after CMS-concurrent-reset …
225328.399: [GC 225328.399: [ParNew: 24448K->0K(24512K), 0.1030736 secs]
853962K->834988K(1572800K), 0.1033765 secs]
225328.707: [CMS-concurrent-sweep: 6.553/7.059 secs]
225328.707: [CMS-concurrent-reset-start]
225328.914: [CMS-concurrent-reset: 0.207/0.207 secs]
225329.721: [GC 225329.722: [ParNew: 24448K->0K(24512K), 0.0880368 secs]
815126K->796196K(1572800K), 0.0888862 secs]

… indicates that 796196K is the heap size after a full GC collection and 1572800K is the
maximum heap size. After this full collection, 796196/1572800 or 51% of the 1.5 GB heap
is in use.
To obtain the maximum post-collection heap size for a given day, the [Link]
files should be analyzed. The post full collection heap sizes throughout the day can be
obtained using a simple awk statement such as:
$ cat [Link] |awk '$2=="[CMS-concurrent-reset:"{getline;print}'
>> [Link]

You can now obtain the maximum post-collection heap size from the [Link]
file. For example, if [Link] contains the following.
bwadmin@serv1$ sar –W
63741.292: [GC 163741.292: [ParNew: 24448K->0K(24512K), 0.0691374 secs]
608826K->588221K(1572800K), 0.0693588 secs]
164113.614: [GC 164113.615: [ParNew: 24448K->0K(24512K), 0.0709264 secs]
606385K->585812K(1572800K), 0.0711760 secs]
164490.619: [GC 164490.619: [ParNew: 24448K->0K(24512K), 0.0677227 secs]
608213K->587808K(1572800K), 0.0679470 secs]
164862.511: [GC 164862.511: [ParNew: 24448K->0K(24512K), 0.0831934 secs]
606974K->586070K(1572800K), 0.0834866 secs]
165226.308: [GC 165226.308: [ParNew: 24448K->0K(24512K), 0.2493387 secs]
606804K->586216K(1572800K), 0.2496168 secs]
165603.667: [GC 165603.667: [ParNew: 24448K->0K(24512K), 0.0679746 secs]
610465K->589446K(1572800K), 0.0682067 secs]
165971.975: [GC 165971.975: [ParNew: 24448K->0K(24512K), 0.0610591 secs]
599254K->578091K(1572800K), 0.0613286 secs]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 177 OF 247
166367.700: [GC 166367.701: [ParNew: 24448K->0K(24512K), 0.0701297 secs]
594655K->574355K(1572800K), 0.0703674 secs]

… 589446K is the maximum post-collection heap size for that day and 1572800K is the
configured maximum heap size.

[Link] bwCPUMon Script Heap Usage Monitoring

The cpuMon scheduled task monitors overall Execution Server heap usage to make sure
the heap is not growing out of control. The cpuMon task generates a Critical severity trap
when the percentage of heap used goes above 85%. If you receive
bwMemoryLimitReached traps, contact BroadSoft Support, as this may be an indication
of a memory leak or a heap size that is too small.

18.1.7 Database Size Monitoring

The Application Server and Network Server TimesTen database is assigned a fixed
permanent size allocation at installation. The database grows as the system scales. It is
important that the database permanent size in use does not grow beyond 90% of the
maximum allocated permanent size. As the database permanent size increases toward
90%, it may be possible to increase the maximum allocated permanent size depending on
platform memory.

NOTE: Database size monitoring for the Database Server (Oracle database) is done through
File System Monitoring.

[Link] Collecting Database Size Indicator

On both the Application Server and the Network Server, the database allocated
permanent size and permanent size in use can be obtained using the TimesTen ttIsql
utility dssize command.
Example:
bwadmin@IHApp$ ttIsql
Command> dssize;

PERM_ALLOCATED_SIZE: 141312
PERM_IN_USE_SIZE: 69966
PERM_IN_USE_HIGH_WATER: 70412
TEMP_ALLOCATED_SIZE: 43008
TEMP_IN_USE_SIZE: 12693
TEMP_IN_USE_HIGH_WATER: 16276

This database has 69966/141312=49% of permanent size in use.

[Link] healthmon Script Database Size Monitoring

The healthmon scheduled task monitors database permanent size usage to make sure
the system provider receives an indication when the database size passes the 80%
threshold. If possible, the database should be resized once that threshold is reached.

18.2 BroadWorks Performance Measurements Monitoring

BroadWorks provides a number of SNMP counters and gauges that can be polled to
provide useful information. All performance measurements are accessible via the CLI or
through external polling. Counters can be cleared via the SNMP set command.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 178 OF 247
It is important to note that in certain cases, excessive polling can affect server
performance. For example, one should NOT poll all Application Server performance
measurements every minute. Different MIBs should be polled at different intervals to best
balance granularity, requirements, and performance. In addition, polling intervals should
be set so that they do NOT run exactly on the hour, half hour, or quarter hour to avoid
interfering with internal application scheduled tasks.
For more information on how to configure SNMP and for a complete list of BroadWorks
SNMP MIBs, see the BroadWorks Performance Measurements Interface Specification
Guide.
The following sections describe best practices for monitoring performance measurements.
Similar to platform resources, there are performance measurements that are considered
key performance indicators. Key performance indicators are identified. A minimum
number of providers should be collecting all key performance indicators. A summary of all
key performance indicators is also provided in the BroadWorks System Engineering
Guide.
A table is provided at the end of each section outlining what should be monitored. The
table provides the suggested interval (Int) frequency [daily (D), hourly (H), 30 minutes (T),
15 minutes (F)], and the type (Type) of measurement. Trending data (T) is not indicative
of a problem, but rather valuable information from a tracking perspective. Performance
data (P) is information that can be collected and measured or compared against defined
thresholds, indicating a server may be encountering performance issues.

18.2.1 Media Server

The Media Server provides a number of SNMP counters and gauges that can be used to
monitor performance. These MIBs can be polled from an external management system or
via the CLI.

[Link] Media Server Usage

The current number of Media Server ports in use can be monitored (msPortsInUse) as
well as any port allocation failures (msNoPortAvailableErrors) due to licensing violations.
The total number of interactive voice response (IVR) and three-way conferencing sessions
since the last clearing of the counters is provided by the msIvrSessionCount and
msConferenceCount counters.
Example:
MS_CLI/Monitoring/PM/MediaServer> get mcp/msPortsInUse
msPortsInUse 16

MS_CLI/Monitoring/PM/MediaServer> get mcp/msNoPortAvailableErrors

*msNoPortAvailableErrors 2

MS_CLI/Monitoring/PM/MediaServer> get mcp/ivr/msIvrSessionCount

*msIvrSessionCount 199170

MS_CLI/Monitoring/PM/MediaServer> get mcp/conferencing/msConferenceCount

*msConferenceCount 0

msNoPortAvailableErrors – Every instance of msNoPortAvailableErrors means that the

Media Server refused a session due to licensing. The Application Server routes to the
next Media Server in the Media Server list (if one is provided).
Note that:
 Ideally, msNoPortAvailableErrors should be zero.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 179 OF 247
 If you observe instances of msNoPortAvailableErrors on all Media Servers, it probably
means that calls requiring Media Server resources are being dropped.
 Constant high values of msNoPortAvailableErrors on all Media Servers generally
means that additional Media Servers are required to meet network traffic
requirements.
 If you observe instances of msNoPortAvailableErrors on one Media Server, it
probably means that the Media Server Selection (MSS) policy on the Network Server
is returning one Media Server disproportionately. In this case, the MSS algorithm
should be revisited.
msPortsInUse (Key Performance Indicator) – This gauge can be used to monitor actual
Media Server port usage and is also an excellent way to monitor Media Server system
capacity (that is, it can be used to determine if additional Media Servers are required in the
network). It can also be used to determine busy hour Media Server bottlenecks. The
msPortsInUse gauge is a weighted representation of port usage. For example, a G.729
call would result in 1.7 ports in use. msPortsInUse is rounded up after all ports in use are
calculated.
msIvrSessionCount and msConferenceCount – These counters provide a view of a
Media Server session over time and they can be used to plot daily Media Server usage.
You can poll these once an hour to derive Media Server busy-hours and help better
understand overall Media Server usage.
Performance Int Node Type Threshold
Measurement

msNoPortAvailableErrors H MS P Ideally zero, unless Media Server blocking

is acceptable.
The blocking percentage equals
[msNoPortAvailableErrors /
(msIvrSessionCount +
msConferenceCount)] * 100.

msPortsInUse F MS P Yes, low severity at 80%, medium severity

at 90% of total port usage.

msIvrSessionCount H MS T No, trending data only.

msConferenceCount H MS T No, trending data only.

[Link] Voice Messaging SMTP Send

Voice messaging recordings are relayed to an SMTP server for subsequent delivery to a
mail server. You can monitor voice mail send problems by reviewing the
msPrimarySmtpErrors and msSecondarySmtpErrors counters. Note that an SMTP error
results in a SNMP trap as well.
Example:
MS_CLI/Monitoring/PM/MediaServer> get smtp/primary/msPrimaryEmailSent
*msPrimaryEmailSent 32206

MS_CLI/Monitoring/PM/MediaServer> get smtp/primary/msPrimarySmtpErrors

*msPrimarySmtpErrors 0

MS_CLI/Monitoring/PM/MediaServer> get smtp/secondary/msSecondaryEmailSent

*msSecondaryEmailSent 0

MS_CLI/Monitoring/PM/MediaServer> get
smtp/secondary/msSecondarySmtpErrors

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

msPrimaryEmailSent and msSecondaryEmailSent – These counters provide an

historic view of SMTP voice mail forwarding. Monitoring these counters provides a view of
SMTP traffic that is related to Media Servers.
msPrimarySmtpErrors and msSecondarySmtpErrors – These counters are pegged
whenever the Media Server times out or receives a server error from the SMTP server.
Ideally, these counters should be zero.
If the primary SMTP server does not respond or returns a server error, the Media Server
tries the secondary SMTP server and pegs the msPrimarySmtpErrors counter. If the
secondary SMTP server does not respond or returns an error, msSecondarySmtpErrors is
pegged and the recorded voice mail is lost. The Media Server does not perform any local
queuing, so it is critical that there is always an available SMTP server.
The Media Server SMTP timeout is controlled by the MS_CLI/System> parameter
smtpTimeout and it is set to “30 seconds” by default. You may want to consider
increasing this if you are experiencing problems.
SMTP servers are passed to the Media Server by the Application Server and are under
AS_CLI/Interface/Mail>. For redundancy purposes, both primarySMTPServer and
secondarySMTPServer should be defined.
Performance Int Node Type Threshold
Measurement

msPrimaryEmailSent H MS T No, trending data only.

msPrimarySmtpErrors H MS T No, trending data only.

msPrimarySmtpErrors H MS T No, trending data only

[Link] IVR Memory

The Media Server requires memory for media playing and recording. The IVR memory
maximum is statically defined under MS_CLI/Service/IVR> parameter memorySize. If the
Media Server cannot allocate memory, the session fails. The SNMP trap
msMemAllocFailure is generated each time the Media Server terminates a session due to
lack of IVR memory.

[Link].1 Real-Time Memory Usage

Real-time memory usage can be monitored using the msIvrAudioMemoryInUse and
msIvrFreeAudioMemory gauges.

MS_CLI/Monitoring/PM/MediaServer> get mcp/ivr/msIvrFreeAudioMemory

msIvrFreeAudioMemory 500705896

MS_CLI/Monitoring/PM/MediaServer> get mcp/ivr/msIvrAudioMemoryInUse

msIvrAudioMemoryInUse 8467648

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 181 OF 247
msIvrAudioMemoryInUse and msIvrFreeAudioMemory – These gauges can be used
to monitor actual Media Server IVR memory usage for proactive memory monitoring. At
any given time, they provide a snapshot of memory usage. As such, the readings
provided can vary dramatically depending on the polling interval. The shorter the interval,
the more accurate the memory usage information is, although it is not a good idea to poll
too frequently. For example, every minute would probably be too low a granularity, while
every 15 minutes would be good enough for the purposes of these MIBs.
Note that:
 From a more proactive perspective, receiving any msMemAllocFailure trap is an
indication that IVR memory should be increased.
 If active msIvrAudioMemoryInUse/(msIvrAudioMemoryInUse +
msIvrFreeAudioMemory) average memory in use, during busy hour, is more than
75%, it would be best to increase the IVR memory.
 Default memorySize is set to 120 MB. Generally, 50% of the available physical
memory should be allocated to the IVR process at initial installation. This allows the
server to better handle expected traffic without having to tweak memorySize.
 The maximum amount of IVR memory that can be allocated on a server is 2 GB.
 Resizing the MS_CLI/Service/IVR parameter memorySize requires a Media Server
stop and start.
Performance Int Node Type Threshold
Measurement

msIvrAudioMemoryInUse F MS P Yes. If average memory in use

/(msIvrAudioMemoryInUse during busy hour is more than 75%
+ msIvrFreeAudioMemory) (low alarm), or more than 85%
(high alarm).

[Link] RTP Statistics

The Media Server reports Real-Time Transport Protocol (RTP) system-wide statistics.
These statistics should be viewed as informational only and not be taken as a definitive
indication that there is a quality of service (QoS) problem.

[Link].1 Receive RTP

The Media Server provides a number of performance measurements related to RTP
received by the Media Server.
MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpPacketsExpected
msRtpPacketsExpected 197273595

MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpPacketsReceived

msRtpPacketsReceived 196818032

MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpOutOfOrder

msRtpOutOfOrder 203444

MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpBadPayload

msRtpBadPayload 11669

MS_CLI/Monitoring/PM/MediaServer> get rtp/receive/msRtpSsrc

msRtpSsrc 13004

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 182 OF 247
msRtpPacketsReceived/msRtpPacketsExpected – These counters provide a general
view on received packet loss. Note that since msRtpPacketsExpected is reported by the
remote party via Real-Time Control Protocol (RTCP), it is only as accurate as the device
sending it and should be used as only one indicator of a potential problem.
The msRtpPacketsReceived/msRtpPacketsExpected ratio can be used to help identify
QoS issues related to the Media Server. The acceptable ratio depends on the network
and codec selection, but in general, a receive loss of less than 2.5% is considered
acceptable.
A good practice would be to start monitoring the received loss ratio once a day and if high
results are obtained, reduce the granularity to identify if the overall result is evenly
distributed or a result of loss spikes.
msRtpOutOfOrder – This counter provides an indication of how many RTP packets are
received out of sequence. A high percentage of out-of-order packets versus
msRtpPacketsReceived may be an indication of a routing problem. However, the Media
Server automatically reorders the packets so there should be no impact on performance.
msRtpBadPayload – This counter reports the number of RTP packets received with an
unknown payload (not 0 for 711, 2 for 726, 18 for 729, or rfc2833 dynamic payload).
msRtpSsrc – This counter reports the number of packets received with a bad
synchronization source (ssrc). The packet ssrc must be the same for all RTP packets in a
stream. A high number of any of these with respect to msRtpPacketsReceived could
indicate an issue with the far-end device.
msRtpReceivedPacketJitter – Disregard this counter as it provides no value.

[Link].2 Transmit RTP

The Media Server provides a number of performance measurements related to RTP
transmitted by the Media Server.
MS_CLI/Monitoring/PM/MediaServer> get
rtp/transmit/msRtpCumulativePacketsLost
msRtpCumulativePacketsLost 2738437

MS_CLI/Monitoring/PM/MediaServer> get rtp/transmit/msRtpPacketsSent

msRtpPacketsSent 195136008

MS_CLI/Monitoring/PM/MediaServer> get rtp/transmit/msRtpFramesSkipped

msRtpFramesSkipped 11079

msRtpCumulativePacketsLost/msRtpPacketsSent – These counters provide a general

view on transmit packet loss. Note that since msRtpCumulativePacketsLost is reported by
the remote party via RTCP, it is only as accurate as the device sending it and should be
used as only one indicator of a problem.
The msRtpCumulativePacketsLost/msRtpPacketsSent ratio can be used to help identify
QoS issues related to the Media Server. The acceptable ratio depends on the network
and codec selection, but in general, a transmit loss of less than 2.5% is considered
acceptable.
A good practice would be to start monitoring the received loss ratio once a day and if high
results are obtained, reduce the granularity to identify if the overall result is evenly
distributed or a result of loss spikes.
msRtpFramesSkipped – This counter indicates how many frames were dropped by the
Media Server on the transmit side due to a resource issue (for example, CPU). A value of
less than a five-unit increase over a ten-second period is considered acceptable. A higher
value than that affects voice quality.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 183 OF 247
msRtpFramesSkipped should be monitored periodically. A good practice would be to poll
the delta in msRtpFramesSkipped every 15 minutes and divide by 90. If the value is more
than five, investigation is required.
Performance Measurement Int Node Type Threshold

msRtpCumulativePacketsLost H MS P Yes. The acceptable ratio

/msRtpPacketsSent depends on the network and
codec selection, but in general, a
transmit loss of less than 2.5% is
considered acceptable.

msRtpPacketsReceived/msRt H MS P Yes. The acceptable ratio

pPacketsExpected depends on the network and
codec selection, but in general, a
transmit loss of less than 2.5% is
considered acceptable.

msRtpFramesSkipped F MS P Yes. Represents RTP frames not

sent due to resource issues. Poll
msRtpFramesSkipped every 15
minutes and divide by 90. If the
result (over four polling intervals) is
more than five (low alarm), or if the
result is more than 10 (high alarm).

[Link] SIP Statistics

The Media Server uses SIP as the call control protocol. SIP protocol statistics can be
used to identify signaling issues.
MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsInviteIns
*msSipStatsInviteIns 265162

MS_CLI/Monitoring/PM/MediaServer> get
sip/msSipStatsInvite200OKRetransmitsOuts
*msSipStatsInvite200OKRetransmitsOuts 1027

MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsInviteIns

*msSipStatsInviteIns 265249

MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsByeIns

*msSipStatsByeIns 262823

MS_CLI/Monitoring/PM/MediaServer> get sip/msSipStatsInfoIns

*msSipStatsInfoIns 668622

MS_CLI/Monitoring/PM/MediaServer> get
sip/msSipStatsRequestRetransmittedIns
*msSipStatsRequestRetransmittedIns 43622

msSipStatsInvite200OKRetransmitsOuts/msSipStatsInviteIns – These counters

provide an indication of what percentage of 200 OK responses required re-transmission.
By default (under CLI level int/sip>) the Media Server re-transmits the 200 OK if it does not
receive a response within 200 milliseconds. Even with a quick re-transmission timer, the
200 OK re-transmission ratio should be under 5%.
msSipStatsRequestRetransmittedIns/(msSipStatsInviteIns+msSipStatsByeIns +
msSipStatsInfoIns) – These counters provide an indication of re-transmissions requests
from an Application Server. The Application Server request retry ratio should be under
5%.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

msSipStatsInvite200OKRetr H MS P Yes. These counters provide an

ansmitsOuts/msSipStatsInvit indication of re-transmissions of
eIns 200 OK responses from the
Media Server. The ratio should
be less than 5%.

msSipStatsRequestRetrans H MS P Yes. These counters provide an

mittedIns/(msSipStatsInviteI indication of re-transmissions
ns + requests from the Application
msSipStatsByeIns+msSipSt Server. The Application Server
atsInfoIns) request retry ratio should be less
than 5%.

[Link] MSCML Statistics

Media Server Control Markup Language (MSCML) is the protocol extension to SIP used
to exchange Media Server directives such as play file and record between an Application
Server and Media Server. There are three commands:
 PlayCollect – Play a .wav file and collect digits. Used for IVR prompt functions such
as voice messaging.
 Play – Play a .wav file without any digit collection. Used to play treatments or
announcements.
 PlayRecord – Play a .wav file (for example, a user’s greeting) and record an incoming
voice message.
The Media Server provides MSCML statistics that can be monitored.
MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLPlayCollectFailed
*msMSCMLPlayCollectFailed 144

MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLPlayFailed

*msMSCMLPlayFailed 3

MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLRecordFailed

*msMSCMLRecordFailed 297

MS_CLI/Monitoring/PM/MediaServer> get mscml/msMSCMLSendMailFailed

*msMSCMLSendMailFailed 10695

msMSCMLPlayFailed, msMSCMLRecordFailed, and msMSCMLPlayCollectFailed –

These counters are pegged in response to a failure condition with respect to the Play,
PlayCollect, or PlayRecord commands. In general, each time these are pegged it means
that a file could not be played and the session with the Media Server has failed.
Note that:
 Ideally, all these should be zero.
 These failures are all related to .wav file retrieval, parsing, or playout and there is a
corresponding trap indicating the problem (for example, HTTP get/parsing error).
msMSCMLSendMailFailed – This counter is pegged each time a Media Server receives
an error from an SMTP/POP server that an e-mail could not be delivered. It means that
the SMTP server was reachable, but the Media Server received an error code back.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 185 OF 247
This error occurs in cases where the SMTP server attempts to forward an e-mail to the
mail server and returns an error code back to the Media Server. A common example
would be an invalid e-mail address assigned to the user’s unified voice messaging
account or an invalid e-mail address on a carbon copy or notification request. There is a
corresponding trap indicating the problem.
Performance Int Node Type Threshold
Measurement

msMSCMLPlayFailed , D MS T No, trending data only since a

msMSCMLRecordFailed, trap was generated.
msMSCMLPlayCollectFailed

msMSCMLSendMailFailed D MS T No, trending data only since a

trap was generated.

18.2.2 Network Server

The Network Server provides a number of SNMP counters and gauges that can be used
to monitor performance. These MIBs can be polled from an external management system
or via the CLI.

[Link] Internal Queue Statistics (Key Performance Indicator)

In general, SIP-signaling delays provide a view on any delays that are affecting call
processing. The Network Server provides Execution Server process internal queue
statistics as well.
NS_CLI/Monitoring/PM/NetworkServer> pwd
broadworks/nsExecutionServer/system/internalStats/

NS_CLI/Monitoring/PM/NetworkServer> get bwNSSystemInternalQueueTable

bwNSSystemInternalQueueTable:

(1) bwNSSystemInternalQueueIndex
(2) bwNSSystemInternalQueueName
(3) bwNSSystemInternalQueueSize
(4) bwNSSystemInternalQueueTimeAvg
(5) bwNSSystemInternalQueueTimeMin
(6) bwNSSystemInternalQueueTimeMax
(7) bwNSSystemInternalQueueTimeMaxTimestampMSB
(8) bwNSSystemInternalQueueLengthCurrent
(9) bwNSSystemInternalQueueLengthAvg
(10) bwNSSystemInternalQueueLengthMax
(11) bwNSSystemInternalQueueLengthMaxTimestampMSB
(12) bwNSSystemInternalQueueLengthMaxTimestampLSB
(13) bwNSSystemInternalQueueTimeMaxTimestampLSB

(1)
(2) (3) (4) (5) (6) (7) (8) (9) (10)
(11) (12) (13)

2 SIP
EncodeQ 93398 1680 0 217 854448300 0 466 4
773098086 0 2256983835
4 SIP
DecodeQ 103692 275 0 183 773075987 0 6 4
773458450 0 2257346653
5 SipRedirectSessionManager (inputAdapter/rtQueryManager) thread
#0 29036 1271 0 1537 1303331261 0 1 3
773781135 0

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 186 OF 247
Although all queue statistics serve a purpose, the following have a direct impact on call
processing and should be monitored.
SipRedirectSessionManager – These queue statistics refer to the Network Server call
processing thread. This thread processes the incoming INVITEs and provides a
redirection response. A delay in this queue would translate to a delay in
inbound/outbound PSTN calls.
SIP EncodeQ – This queue statistic is related to the encoding and building of outgoing
SIP messages. A delay in this queue results in outgoing messages being backed up.
SIP DecodeQ – This queue statistic is related to the decoding of incoming SIP messages.
A delay in this queue results in incoming messages being backed up.
The two parameters to monitor for each queue are as follows:
bwSystemInternalQueueTimeAvg – The average queue holding time. Note that this
value is in 1/1000 milliseconds, that is, divide by 1000 for the value in milliseconds. It is
recommended that it should be polled and reset once every 15 minutes. High average
queue delays during server busy hour could be an indication of a performance problem.
bwSystemInternalQueueTimeMax – The queue delay high-water mark. It should be
polled and reset once every 15 minutes. Consistent hourly high-water marks (for
example, four polling intervals) of more than one second or any single high-water mark of
more than five seconds should be reported to BroadSoft Support.

Performance Measurement Int Node Type Threshold

bwSystemInternalQueueTimeAvg F NS P Yes. For acceptable queue time

ranges, see BroadWorks System
Engineering Guide.

bwSystemInternalQueueTimeMax F NS P Yes. For acceptable queue time

ranges, see BroadWorks System
Engineering Guide.

[Link] System Call Processing Statistics

The Network Server acts as a SIP redirect server and as such is stateless. The Network
Server receives INVITEs, processes them, and returns a redirection response (“302
Moved Temporarily” in a successful transaction). Network Server call processing capacity
can be calculated using the BroadWorks System Capacity Planner, which is an Excel
spreadsheet. The basic variable used to describe Network Server capacity is:
calls per seconds (CPS)/transactions per second (TPS).
There are call processing performance measurements that allow a carrier to monitor
Network Server performance. They are also used to perform basic validation on the call
model used in the BroadWorks System Capacity Planner.

[Link].1 Transaction Processing

System-wide transaction processing statistics are available to help monitor system
performance.
NS_CLI/Monitoring/PM/NetworkServer> pwd
broadworks/nsExecutionServer/processing/

NS_CLI/Monitoring/PM/NetworkServer> get bwNSCallpCallsPerSecond

bwNSCallpCallsPerSecond 3

NS_CLI/Monitoring/PM/NetworkServer> pwd

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NS_CLI/Monitoring/PM/NetworkServer> get bwNSSipStatsInviteIns

*bwNSSipStatsInviteIns 14396

bwNSCallpCallsPerSecond (Key Performance Indicator) – This gauge measures the

number of transactions per second that the Network Server is currently handling. It uses
the time stamps of (up to) the last 100 INVITEs received.
Network Servers are licensed based on the number of transactions per second within a
certain time window. When the capacity is exceeded, the Network Server returns a “503
Service Unavailable”. Although traps are generated when licensing capacity is reached,
this gauge can also be used to trend TPS growth.
SIP Messages Per Second (Key Performance Indicator) – The number of SIP
messages per second (MPS) can be used to gauge if a server’s call patterns have
changed. SIP MPS can be calculated using the following counters:
SIP_MPS = Delta [bwNSSipStatsTcpIns + bwNSSipStatsTcpOuts +
bwNSSipStatsUdpIns + bwNSSipStatsUdpOuts] / Polling_Interval_In _Seconds
bwNSSipStatsInviteIns – This counter measures the number of INVITEs processed by
the Network Server.
This counter should be polled hourly and can be used to trend transactions per hour,
which would be the equivalent of busy hour call attempts (BHCA) on an Application
Server.

Performance Measurement Int Node Type Threshold

bwNSCallpCallsPerSecond F NS T No, trending data only.

SIP_MPS H NS T No, trending data only.

bwNSSipStatsInviteIns H NS T No, trending data only.

[Link].2 Processing Treatments

The Network Server keeps track of calls that result in a treatment. These internal
treatment messages map back to SIP responses (either a 4XX or 5XX). The list of
system-defined treatments and corresponding SIP error messages can be found under
CLI level NS_CLI/System/CallP/Treatment>.
Example:
NS_CLI/Monitoring/PM/NetworkServer> pwd
broadworks/nsExecutionServer/processing/

NS_CLI/Monitoring/PM/NetworkServer> get errorStatTable

errorStatTable:

(1) errStatID
(2) errStatName
(3) errStatNbOccurences

(1) (2) (3)

1 invld 73
2 ndmch 16
3 untmt 0
4 undge 0

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 188 OF 247
5 usrnf 260
6 nrixc 0
7 norte 0
8 dftmt 0
9 carjc 0
10 innpa 9
11 incac 0
12 invdd 0
13 inddd 0
14 inidd 0
15 inldd 0
16 tmpun 0
17 bdreq 0
18 frbdn 0
19 reqto 0
20 nserr 0
21 noimp 0
22 bdgwy 0
23 srvun 0
24 srvto 0
25 verns 0
26 invod 0
27 unadn 8
28 unkgw 15
29 sbovf 0
30 carbk 0

This table can (optionally) be polled once a day for trending purposes.
Not all treatment codes report a problem. For example, invld is pegged when a user dials
an invalid number and therefore it is not representative of a problem, while unkgw would
indicate a problem, since it means that the requesting device is not known to the Network
Server. As such, only occurrences of following treatments must be investigated:
 bdgwy – bad gateway
 bdreq – not acceptable
 frbdn – Network Server refuses to process the request
 noimp – functionality not supported
 nserr – Network Server error
 reqto – Network Server request timeout
 sbovf – service bus re-enter overflow
 srvto – Network Server timeout
 srvun – Network Server unavailable
 unkgw – unknown gateway
 verns – SIP version not supported
Performance Measurement Int Node Type Threshold

errStatNbOccurences D NS T No, trending data only.

This information is useful because
some internal treatments are an
indication of a problem (for example,
unkgw).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 189 OF 247
[Link].3 Request per Device
The Network Server keeps track of requests received from each device.
Example:
NS_CLI/Monitoring/PM/NetworkServer> pwd
broadworks/nsExecutionServer/processing/

NS_CLI/Monitoring/PM/NetworkServer> get neStatTable

neStatTable:

(1) neStatID
(2) neStatName
(3) neStatNbSIPRequests
(4) neStatNbSIPRequestsFailures
(5) neStatNbMSSRequests
(6) neStatNbMSSRequestsFailures

(1) (2) (3) (4) (5) (6)

1 [Link] 0 0 0 0
3 [Link] 0 0 0 0
4 [Link] 4513 7 0 0
5 [Link] 1268 0 0 0
6 [Link] 0 0 0 0
7 [Link] 270 0 0 0

This table can (optionally) be polled once a day for trending purposes.
neStatNbSIPRequests – This counter allows monitoring the amount of traffic the Network
Server is receiving for all known nodes. If useDomainForSubscriberAddress is set to
“true” on the Application Server, the Network Server tracks INVITEs per user domain, not
per Application Server identity.
Performance Measurement Int Node Type Threshold

neStatNbSIPRequests D NS T No, trending data only.

This information is useful to identify
where Network Server requests are
coming from.

18.2.3 Application Server, Execution Server, and Profile Server

The Application Server provides a number of SNMP counters and gauges that can be
used to monitor performance. These MIBs can be polled from an external management
system or via the CLI.
When using the IMS mode, Provisioning and Execution activities are split between the
Profile Server and the Execution Server respectively. Counters and gauges are also valid
on the specified server, when indicated.

[Link] Internal Queue Statistics – Key Performance Indicator

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 190 OF 247
In general, SIP signaling delays provide a view on any delays that are affecting call
processing. The Application Server does provide Execution Server process internal
queue statistics as well. Even though these delays should be covered in the SIP signaling
delays, monitoring Execution Server internal queue statistics is recommended.
AS_CLI/Monitoring/PM/ApplicationServer> get
-------------------------------------------------------------------------
broadworks/executionServer/systemModule/internalStats/
-------------------------------------------------------------------------
*bySystemInternalQueueResets 0
bwSystemInternalQueueTable:

(1) bwSystemInternalQueueIndex
(2) bwSystemInternalQueueName
(3) bwSystemInternalQueueSize
(4) bwSystemInternalQueueTimeAvg
(5) bwSystemInternalQueueTimeMin
(6) bwSystemInternalQueueTimeMax
(7) bwSystemInternalQueueTimeMaxTimestampMSB
(8) bwSystemInternalQueueLengthCurrent
(9) bwSystemInternalQueueLengthAvg
(10) bwSystemInternalQueueLengthMax
(11) bwSystemInternalQueueLengthMaxTimestampMSB
(12) bwSystemInternalQueueLengthMaxTimestampLSB
(13) bwSystemInternalQueueTimeMaxTimestampLSB

(1) (2) (3) (4) (5)

(6)
(7) (8) (9) (10) (11) (12) (13)
1 CallP ThreadedDBAccessQueue 21 24285 9
103
268 0 0 0 268 2255816335 2257044192
2 Call Half Input Adapter 0 108 95731 0
866
268 0 1685 11 268 2253911749 2253912586
3 Call Half Input Adapter 1 210 47838 0
916
268 0 658 11 268 2253911780 2257044446
4 Registration Input Adapter 799 3702 0
308
268 0 14 3 268 2258153986 2253914110

Although all queue statistics serve a purpose, the following primary queues have a direct
impact on call processing and they should be monitored:
Call Half Adapters – These queue statistics refer to the Execution Server call half thread
queues. Each Application Server has at least two call half adapter threads. A delay in
one thread corresponds to a delay in the call that that adapter was handling. If adapters
are delayed, then messaging queues until the delay ends.
SIP EncodeQ – This queue statistic is related to the encoding and building of outgoing
SIP messages. A delay in this queue results in outgoing messages backing up.
SIP DecodeQ – This queue statistic is related to the decoding of incoming SIP messages.
A delay in this queue results in incoming messages backing up.
MGCP EncodeQ – This queue statistic is related to the encoding and building of outgoing
MGCP messages. A delay in this queue results in outgoing messages backing up.
MGCP DecodeQ – This queue statistic is related to the decoding of incoming MGCP
messages. A delay in this queue results in incoming messages backing up.
The two parameters to monitor for each queue are as follows:

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 191 OF 247
bwSystemInternalQueueTimeAvg – The average queue holding time. Note that this
value is in 1/1000 of a millisecond, that is, divide by 1000 to get the value in milliseconds.
This should be polled and reset once every 15 minutes. An average queue delay during a
server busy hour of more than 200 milliseconds would be an indication that this server has
reached server capacity.
bwSystemInternalQueueTimeMax – This is the queue delay high-water mark. For the
call half adapter queues, a high-water mark delay of 2.5 seconds or greater results in the
generation of a bwThreadDelayDetected trap.
This should be polled and reset once every 15 minutes.

[Link] System Call Processing Statistics

Application Server call processing capacity can be calculated using the BroadWorks
System Capacity Planner, which is an Excel spreadsheet. The basic variables used to
describe Application Server capacity are as follows:
 Number of users
 Busy hour call attempts
 Calls per second
 Simultaneous active calls
There are a number of call processing performance measurements available, which allow
a carrier to monitor Application Server performance and they perform a basic validation on
the call model used in the BroadWorks System Capacity Planner.

[Link].1 Number of Users

The number of provisioned users can be polled.
Example
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/systemModule/systemStats/

AS_CLI/Monitoring/PM/ApplicationServer> get bwNumberOfUsers

bwNumberOfUsers 24739

bwNumberOfUsers – This gauge provides an indication of the number of provisioned

users on the Application Server.
Performance Measurement Int Node Type Threshold

bwNumberOfUsers D AS T No, trending data only.

[Link].2 Call Originations/Terminations

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

System-wide call processing statistics are available to help monitor system performance.
Example:
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/callpModule/callpStats/

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 192 OF 247
AS_CLI/Monitoring/PM/ApplicationServer> get
bwCallpNetworkOriginationAttempts
*bwCallpNetworkOriginationAttempts 2365

AS_CLI/Monitoring/PM/ApplicationServer> get
bwCallpUserOriginationAttempts
*bwCallpUserOriginationAttempts 3008

AS_CLI/Monitoring/PM/ApplicationServer> get
bwCallpUserTerminationsAnswered
*bwCallpUserTerminationsAnswered 2542

AS_CLI/Monitoring/PM/ApplicationServer> get
bwCallpNetworkOriginationAttempts
*bwCallpNetworkOriginationAttempts 2379

AS_CLI/Monitoring/PM/ApplicationServer> get
bwCallpNetworkTerminationAttempts
*bwCallpNetworkTerminationAttempts 3454

AS_CLI/Monitoring/PM/ApplicationServer> get
bwCallpNetworkTerminationsAnswered
*bwCallpNetworkTerminationsAnswered 1971

bwCallpUserOriginationAttempts – This counter represents the number of BroadWorks

user origination attempts.
bwCallpNetworkOriginationAttempts – This counter represents the number of incoming
calls to BroadWorks users from network devices (for example, PSTN gateway or
softswitch).
bwCallpUserTerminationAttempts – This counter represents the number of calls that
terminated to a BroadWorks user.
bwCallpNetworkTerminationAttempts – This counter represents the number of calls
that terminated to a network device (for example, PSTN gateway or softswitch).
bwCallpUserTerminationsAnswered – This counter represents the number of calls to
BroadWorks users that are answered.
bwCallpNetworkTerminationsAnswered – This counter represents the number of calls
to network devices that are answered (for example, PSTN calls).
Although all of the above counters provide information, the basic monitoring statistic that is
derived is the busy hour call attempts (BHCA).
NbCallAttempts – This is the sum or all access-side and network-side originations that
can be used to calculate the overall number of call attempts the system is handling,
where:
NbCallAttempts = bwCallpUserOriginationAttempts + bwCallpNetworkOriginationAttempt
Polling these counters hourly would provide the BHCA values for the entire day and allow
a carrier to validate the BHCA assumption used in the BroadWorks System Capacity
Planner.
Performance Measurement Int Node Type Threshold

bwCallpNetworkOriginationAttempts F AS T No, trending data only.

bwCallpUserOriginationAttempts F AS T No, trending data only.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

bwCallpNetworkTerminationAnswered H AS T No, trending data only.

bwCallpUserTerminationAnswered H AS T No, trending data only.

bwCallpUserTerminationAttempts H AS T No, trending data only.

bwCallpNetworkTerminationAttempts H AS T No, trending data only.

NbCallAttempts = F AS T No, trending data only.

[bwCallpUserOriginationAttempts + Multiply by four to calculate the
bwCallpNetworkOriginationAttempt] BHCA for the server.

[Link].3 Current/Active Call Statistics

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

Two current or active call gauges can be monitored.

Example:
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/callpModule/callpStats/

AS_CLI/Monitoring/PM/ApplicationServer> get bwCallpCallsPerSecond

bwCallpCallsPerSecond 5

AS_CLI/Monitoring/PM/ApplicationServer> get bwCallpActiveCalls

bwCallpActiveCalls 234

bwCallpCallsPerSecond – This gauge measures the rate at which network origination

attempts and user origination attempts occur. It uses the time stamps of up to the last 100
network origination attempts and user origination attempts to calculate the rate.
Polling this gauge every quarter hour can also identify any call processing spikes. A
threshold alarm can be created to send a trap when CPS hits a certain threshold.
Overload controls can be enabled to help mitigate any attack.
bwCallpActiveCalls – This gauge indicates the total number of active sessions. An
active session is defined as an active originating session.
A list of all users involved in active calls can be generated by service provider or enterprise
name, using the Application Server CLI level as follows.
AS_CLI/System/Util/Diag> list svcProviderId <SP/ENT_name>

Performance Measurement Int Node Type Threshold

bwCallpCallsPerSecond F AS T No, trending data only.

bwCallpActiveCalls F AS T No, trending data only.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: This section is valid for the Profile Server as well (Provisioning application).

Release 13.0 introduced the open client interface (OCI) for unified provisioning. All
provisioning transactions initiated regardless of source (web, CLI, or external third party
OCI client) go through the OCI. There are performance measurements that can be
monitored to track provisioning performance.
Example:
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/provisioningServer/psOciModule/psOciStats/

AS_CLI/Monitoring/PM/ApplicationServer> get psOciStatsRequestResponseTime

psOciStatsRequestResponseTime 114

AS_CLI/Monitoring/PM/ApplicationServer> get
psOciStatsMaxRequestResponseTime
*psOciStatsMaxRequestResponseTime 6685

OCI Messages Per Second (Key Performance Indicator) – OCI MPS rate can be used
to identify changes in provisioning traffic.
OCI_MPS= Delta [psOciStatsNbUpdateRequests + psOciStatsNbQueryRequests +
psOciStatsNbAuthorizationRequests] / Polling_Interval_In_Seconds
psOciStatsRequestResponseTime – This gauge indicates the average time (in
milliseconds based on a rolling average of the last 100 samples) it takes to process an
OCI request.
Polling this gauge hourly provides an indication of OCI performance.
The expected average response time depends on the platform; generally, it should be less
than 300 milliseconds under normal operating conditions. Continuous average response
times of more than 500 milliseconds should be reported to BroadSoft Support.
psOciStatsMaxRequestResponseTime – This gauge provides the high-water mark for
OCI response delay.
This gauge can be polled hourly and reset, to provide a view of the maximum delay the
OCI interface has experienced within that polling window.
It is normal that at times the maximum delay (certainly on a small system configuration
such as a V120) is in the range of seconds. Repeated maximum delays polled in the tens
of seconds should be reported to BroadSoft Support.
You can poll psOciStatsMaxRequestName to get the actual request type that caused the
high-water mark delay.

Performance Int Node Type Threshold

Measurement

OCI_MPS H AS T No, trending data only.

psOciStatsRequestRespons H AS P Yes. If the result is more than 300

eTime milliseconds (low alarm) or if more
than 500 milliseconds (high alarm).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

psOciStatsMaxRequestResp H AS P Yes. When the result is more than

onseTime five seconds for four polling intervals
(high alarm).

[Link] Messaging Statistics

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

There are SIP-related and MGCP performance measurements that can be used to
monitor system performance.

[Link].1 SIP/MGCP Messaging Rates

Normal call originations and terminations monitored by BHCA are not the only type of
messaging that impact system capacity.
SIP Messages Per Second (Key Performance Indicator) – SIP MPS rate can be used
to identify changes in SIP traffic patterns. SIP MPS can be calculated using the following:
SIP_MPS= Delta [bwSipStatsTcpIns + bwSipStatsTcpOuts + bwSipStatsUdpIns +
bwSipStatsUdpOuts] / Polling_Interval_In _Seconds
MGCP Messages Per Second (Key Performance Indicator) – MGCP MPS rate can be
used to identify changes in MGCP traffic patterns. SIP MPS can be calculated using the
following:
MGCP_MPS= Delta [bwMGCPStatsMGCPCommandOuts +
bwMGCPStatsMGCPCommandIns + MGCPStatsMGCPResponseOuts +
bwMGCPStatsMGCPResponseOuts] / Polling_Interval_In _Seconds
Performance Int Node Type Threshold
Measurement

SIP_MPS H AS T No, trending data only.

MGCP_MPS H AS T No, trending data only.

[Link].2 Ancillary SIP Messaging

Normal call originations and terminations monitored by BHCA are not the only type of
messaging that impact system capacity. A system with a high number of ancillary SIP
messages impacts performance (for example, high CPU or call processing delays) without
any calls in progress. SIP REGISTER, SUBSCRIBE/NOTIFY, and OPTIONS message
rates must be monitored so that the system is not flooded.

AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/sipModule/sipStats/

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsRegisterIns

*bwSipStatsRegisterIns 109926

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsOptionsIns

*bwSipStatsOptionsIns 12424

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsNotifyIns

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsNotifyOuts

*bwSipStatsNotifyOuts 574471

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsSubscribeIns

*bwSipStatsSubscribeIns 210491

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsSubscribeOuts

*bwSipStatsSubscribeOuts 77

[Link].2.1 SIP Registration Traffic

Excessive SIP registration traffic can have a serious impact on server performance. The
BroadWorks System Capacity Planner considers this by having the SIP registration
interval and SIP authentication as input variables.
Generally, a SIP device’s registration interval towards the Application Server should be 60
minutes or greater, which would mean no more than four REGISTER messages per user
per hour. The four REGISTER messages per user, per hour is derived from the standard
model of a user with SIP authentication enabled and a device registering before the 60
minute period expires. If registrations must be more frequent for NAT traversal, a session
border controller should be used to front the high registration traffic.
The following performance measurement can be used to monitor registration traffic:
bwSipStatsRegisterIns – This counter reflects the total number of REGISTER requests
received by BroadWorks and it can be used to monitor registration traffic.
Monitoring this counter every 15 minutes can help identify “bursty” registration traffic. The
results should be relatively the same throughout the day and should only increase if the
number of SIP registering devices increase.
Generally, each quarter hour reading of this counter should be within 20% of the average
15-minute result.
Excessive “bursty” registration traffic patterns should be investigated to identify the root
cause.
The hourly result of bwSipStatsRegisterIns can be used to see if the registration traffic is in
line with the expected registration traffic. A high value of registrations per user has a
scaling impact on a server.
Ideally, registrations per SIP registering user in any given hour
(bwSipStatsRegisterIn/bwNumberOfUsers) should equal to four or less. In general, a
value of more than 10 should be investigated.
The hourly result of bwSipStatsRegisterIns can be used to estimate the impact of
registration traffic in a BHCA equivalency fashion. In general, from a messaging
perspective, a registration and response is equivalent to 2/16 of a call.
For example, an hourly bwSipStatsRegisterIns of 200 KB would be equivalent to (200 KB
* 2/16) ~25 KB BHCA.
[Link].2.2 SIP OPTIONS Traffic
The SIP OPTIONS method is used by some session border controllers to “ping”
BroadWorks Application Servers. The following performance measurement can be used
to monitor OPTIONS traffic”
bwSipStatsOptionsIns/ bwSipStatsOptionsOuts – This counter reflects the total
number of OPTIONS requests received by BroadWorks and can be used to monitor
OPTIONS traffic.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 197 OF 247
Ideally, SIP OPTIONS traffic from session border controllers should be no lower than one
message every 45 seconds. This allows the OPTIONS session to terminate. OPTIONS
pinging at a rate below 40 seconds may result in the session remaining active and a
bwAbnormalCallTermination trap when the session audit runs.
The hourly result of bwSipStatsOptionsIns + bwSipStatsOptionsOuts can be used to
estimate the impact of OPTIONS traffic in a BHCA equivalency fashion. In general, from a
messaging perspective, a registration and response is equivalent to 2/16 of a call.
For example, an hourly OPTIONS traffic of 100 KB would be equivalent to (100 KB * 2/16)
~12 KB BHCA.
[Link].2.3 SIP SUBSCRIBE/NOTIFY Traffic
SIP SUBSCRIBE/NOTIFY messages are used between the Application Server and many
SIP devices to support advanced call control (auto-answer, hold event reporting, off-hook
event reporting). These messages can affect overall performance and they should be
monitored.
bwSipStatsSubscribeIns/bwSipStatsSubscribeOuts – These counters reflect the total
number of SUBSCRIBE traffic in/out of the Application Server.
bwSipStatsNotifyIns/bwSipStatsNotifyOuts – These counters reflect the total number
of NOTIFY traffic in/out of the Application Server.
Monitoring the counters every hour can help identify SUBSCRIBE/NOTIFY traffic impact
on overall performance.
The hourly result of bwSipSubscribeIns + bwSipSubscribeOuts can be used to estimate
the impact of SUBSCRIBE traffic in a BHCA equivalency fashion. In general, from a
messaging perspective, a SUBSCRIBE and response is equivalent to 2/16 of a call.
For example, an hourly bwSipSubscribeIns + bwSipSubscribeOuts of 100 KB would be
equivalent to (100 KB * 2/16) ~12 KB BHCA.
The hourly result of bwSipNotifyIns + bwSipNotifyOuts can be used to estimate the impact
of NOTIFY traffic in a BHCA equivalency fashion. In general, from a messaging
perspective, a SUBSCRIBE and response is equivalent to 2/16 of a call.
For example, an hourly bwSipSubscribeIns + bwSipSubscribeOuts of 100 KB would be
equivalent to (100 KB * 2/16) ~12 KB BHCA.

Performance Int Node Type Threshold

Measurement

bwSipStatsRegisterIns H AS T Ideally, registrations per SIP registering

user in any given hour
(bwSipStatsRegisterIn/bwNumberOfUsers)
should be equal to four or less.
The hourly result of any of these ancillary
message PMs can be used to estimate the
impact of registration traffic in a BHCA
equivalency fashion. In general, from a
messaging perspective, a registration and
response is equivalent to 2/16 of a call.

[bwSipStatsOptionsIns + H AS T No, trending data only.

bwSipStatsOptionsOuts]

[bwSipStatsSubscribeIns+b H AS T No, trending data only.

wSipStatsSubscribeOuts]

[bwSipStatsNotifyIns+bwSi H AS T No, trending data only.

pStatsNotifyOuts]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 198 OF 247
[Link].3 SIP Signaling Delays
There are call processing delay-related performance measurements that can be used to
monitor overall performance.
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/sipModule/sipStats/

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsSetupSignalDelay

bwSipStatsSetupSignalDelay 13

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsMaxSetupSignalDelay

*bwSipStatsMaxSetupSignalDelay 594

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsAnswerSignalDelay

bwSipStatsAnswerSignalDelay 17

AS_CLI/Monitoring/PM/ApplicationServer> get bwSipStatsMaxAnswerSignalDelay

*bwSipStatsMaxAnswerSignalDelay 255

bwSipStatsSetupSignalDelay – This gauge is applicable to SIP-originated calls. It

measures the time (in milliseconds, based on a rolling average of up to 100 samples) it
takes between the receipt of an INVITE message for the origination of a new call and the
transmission of an INVITE (SIP terminator) to the primary device of the original called
party (in the case of intra-group call) or to the network element of the original called party
(in the case of a call to the PSTN). Delays incurred by a query to an external element (for
example -dip to the Network Server, CNAME query) are part of the measurement. This
measurement is intended to reflect the elapsed delay between receipt of the call setup
signal from the caller and transmission of the call setup signal to the called party.
This gauge should be polled every 15 minutes to get a view of the average SIP setup
delays. The average delay should be no more than 200 milliseconds during busy hour.
Averages higher than 200 milliseconds during busy hour are an indication that the server
is performing above capacity.
bwSipStatsMaxSetupSignalDelay – This gauge measures the longest SIP setup-signal
delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of high-water SIP
setup-signal delay. It is not abnormal to see a spike in SIP setup-signal delay. An
occasional reading of one or two seconds is not indicative of a problem. Constant high-
water marks of more than two seconds should be reported to BroadSoft Support.
bwSipStatsAnswerSignalDelay – This gauge measures the time (in milliseconds, based
on a rolling average of up to 100 samples) between the receipt of a 200 OK message
indicating answer and the transmission of a 200 OK indicating answer to the originator.
This measurement is intended to reflect internal queuing, scheduling, and processing
delays. This gauge is applicable only to the SIP-originated calls for which the setup-signal
delay is measured. Answer signal delay is not measured when the call has been
forwarded, redirected, or when the call has encountered route advancing.
This gauge should be polled every 15 minutes to get a view of the average SIP answer
delays. The average delay should be no more than 200 milliseconds during busy hour.
Averages higher than 200 milliseconds during busy hour are an indication that the server
is performing above capacity.
bwSipStatsMaxAnswerSignalDelay – This measures the longest SIP answer-signal
delay sampled since the system was started.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 199 OF 247
This gauge should be polled and reset every 15 minutes to get a view of high-water SIP
answer-signal delay. It is not abnormal to see a spike in SIP answer-signal delay. An
occasional reading of one or two seconds is not indicative of a problem. Constant high-
water marks of more than two seconds should be reported to BroadSoft Support.
Performance Measurement Int Node Type Threshold

bwSipStatsSetupSignalDelay F AS P Yes. If the result is more than

200 milliseconds (medium alarm)
or if more than 500 milliseconds
(high alarm).

bwSipStatsMaxSetupSignalDelay F AS P Yes. If the result is more than

2.5 seconds for four polling
intervals (high alarm).

bwSipStatsAnswerSignalDelay F AS P Yes. If the result is more than

200 milliseconds (medium alarm)
or if more than 500 milliseconds
(high alarm).

bwSipStatsMaxAnswerSignalDelay F AS P Yes. If the result is more than

2.5 seconds for four polling
intervals (high alarm).

[Link].4 MGCP Signaling Delays

There are call processing delay-related performance measurements that can be used to
monitor MGCP overall performance.
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/mgcpModule/mgcpStats/
AS_CLI/Monitoring/PM/ApplicationServer> get bwMGCPStatsDialToneDelay
*bwMGCPStatsDialToneDelay 2

AS_CLI/Monitoring/PM/ApplicationServer> get
bwMGCPStatsMaxDialToneDelay
*bwMGCPStatsMaxDialToneDelay 578

AS_CLI/Monitoring/PM/ApplicationServer> get
bwMGCPStatsSetupSignalDelay
*bwMGCPStatsSetupSignalDelay 89

AS_CLI/Monitoring/PM/ApplicationServer> get
bwMGCPStatsMaxSetupSignalDelay
*bwMGCPStatsMaxSetupSignalDelay 4276

AS_CLI/Monitoring/PM/ApplicationServer> get
bwMGCPStatsAnswerSignalDelay
bwMGCPStatsAnswerSignalDelay 17

AS_CLI/Monitoring/PM/ApplicationServer> get
bwMGCPStatsMaxAnswerSignalDelay
*bwMGCPStatsMaxAnswerSignalDelay 255

bwMGCPStatsDialToneDelay – This gauge measures the time (in milliseconds, based

on a rolling average of up to 100 samples) between receipt of NTFY indicating off-hook
and transmission of RQNT indicating dial tone and request for digits.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 200 OF 247
This gauge should be polled and reset every 15 minutes to get a view of the average
MGCP setup delays. The average delay should be no more than 50 milliseconds during
busy hour. Averages higher than 50 milliseconds during busy hour are an indication that
the server is performing above capacity.
bwMGCPStatsMaxDialToneDelay – This measures the longest MGCP dial-tone delay
sampled since the system was started or since this measurement was cleared.
This gauge should be polled and reset every 15 minutes to get a view of the high-water
SIP setup signal delay. It is not abnormal to see a spike in a MGCP setup-signal delay.
An occasional reading of one or two seconds is not indicative of a problem. Constant
high-water marks of more than two seconds should be reported to BroadSoft Support.
bwMGCPStatsSetupSignalDelay – This gauge is applicable to MGCP-originated calls.
It measures the time (in milliseconds, based on a rolling average of up to100 samples) it
takes between the receipt of a NTFY message with digits dialed for the origination of a
new call and the transmission of an INVITE (SIP terminator), RQNT (MGCP terminator
with in-band ringback), or CRCX (MGCP terminator without in-band ringback) to the
primary device of the original called party (for an intra-group call) or to the network element
of the original called party (for a call to the PSTN). Delays incurred by a dip to the Network
Server are removed from the measurement. This measurement is intended to reflect
internal queuing, scheduling, and processing delays.
This gauge should be polled and reset every 15 minutes to get a view of the average
MGCP setup delays. The average delay should be no more than 200 milliseconds during
busy hour. Averages higher than 200 milliseconds during busy hour are an indication that
the server is performing above capacity.
bwMGCPStatsMaxSetupSignalDelay – This gauge measures the longest MGCP setup-
signal delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of a high-water
MGCP setup-signal delay. It is not abnormal to see a spike in a MGCP setup-signal
delay. An occasional reading of one or two seconds is not indicative of a problem.
Constant high-water marks of more than two seconds should be reported to BroadSoft
Support.
bwMGCPStatsAnswerSignalDelay – This gauge measures the time (in milliseconds,
based on a rolling average of up to 100 samples) between the receipt of a 200 OK
message indicating answer or a NTFY off-hook, indicating answer and the transmission of
a MDCX indicating answer to the originator. This measurement is intended to reflect
internal queuing, scheduling, and processing delays. This gauge is applicable to only the
MGCP-originated calls for which the setup-signal delay is measured. Answer signal delay
is not measured when the call has been forwarded, redirected, or when the call has
encountered route advancing.
This gauge should be polled every 15 minutes to get a view of the average MGCP answer
delays. The average delay should be no more than 200 milliseconds during busy hour.
Averages higher than 200 milliseconds during busy hour are an indication that the server
is performing above capacity.
bwMGCPStatsMaxAnswerSignalDelay – This measures the longest MGCP answer-
signal delay sampled since the system was started.
This gauge should be polled and reset every 15 minutes to get a view of high-water
MGCP answer delay. It is not abnormal to see spike in MGCP answer delay. An
occasional reading of one or two seconds is not indicative of a problem. Constant high-
water marks of more than two seconds should be reported to BroadSoft Support.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

bwMGCPStatsSetupSignal F AS P Yes. If the result is more than 200

Delay milliseconds (medium alarm) or if
more than 500 milliseconds (high
alarm).

bwMGCPStatsMaxSetupSignal F AS P Yes. If the result is more than 2.5

Delay seconds for four polling intervals
(high alarm).

bwMGCPStatsAnswerSignalDe F AS P Yes. If the result is more than 200

lay milliseconds (medium alarm) or if
more than 500 milliseconds (high
alarm).

bwMGCPStatsMaxAnswerSign F AS P Yes. If the result is more than 2.5

alDelay seconds for four polling intervals
(high alarm).

bwMGCPStatsDialToneDelay F AS P Yes. If the result is more than 50

milliseconds (medium alarm) or if
more than 100 milliseconds (high
alarm).

[Link].5 SIP Network Element Retry Percentage

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

The Application Server keeps track of SIP retry percentages to network-side nodes such
as Media Servers, Network Servers, other Application Servers, and PSTN interconnection
nodes (softswitch or network gateways).
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/sipModule/sipStats/

AS_CLI/Monitoring/PM/ApplicationServer> get
bwSipStatsMsgRetryToNeTable
bwSipStatsMsgRetryToNeTable:

(1) bwSipStatsMsgRetryToNeID
(2) bwSipStatsMsgRetryToNeAddr
(3) bwSipStatsMsgRetryToNePercentage

(1) (2) (3)

1 [Link] 0
2 [Link] 51
3 [Link] 2
4 [Link] 60
5 [Link] 0
6 [Link] 52
7 [Link] 0
8 [Link] 0
9 [Link] 0
10 [Link] 0
11 [Link] 8
12 [Link] 0
13 [Link] 0
14 [Link] 0

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 202 OF 247
This table should be polled daily to get a view of retry percentages to various nodes. A
retry percentage more than two percent to any network-side node should be investigated.
Session border controllers show up in the list as well. The retry percentage to these
nodes should be ignored since they include retries to access devices.
Performance Measurement Int Node Type Threshold

bwSipStatsMsgRetryToNe D AS P Yes. If the result is more than

Percentage 3% (medium alarm) or if the
result more than 5% (high
alarm).

[Link].6 SIP Message Responses

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

The Application Server tracks responses to all SIP methods received and sent. This can
be useful in “trending” responses to monitor for changes in call processing behavior (that
is, a sudden increase in a specific negative response).
Example
AS_CLI/Monitoring/PM/ApplicationServer> pwd
broadworks/executionServer/sipModule/sipStats/

AS_CLI/Monitoring/PM/ApplicationServer> get
bwSipStatsInviteResponsesTable
bwSipStatsInviteResponsesTable:

(1) bwSipStatsInviteResponseCodeValue
(2) bwSipStatsInviteResponseIns
(3) bwSipStatsInviteResponseOuts

(1) (2) (3)

100 11554 6254
180 2754 3573
181 0 0
182 0 0
183 4386 2208
200 13185 5886
300 0 0
301 0 0
302 9466 0
305 0 0
380 0 0
400 2 14
401 0 0
402 0 0
403 0 0
404 180 0
405 0 0
406 10 0
407 1 0
408 1 40
409 0 0
410 3 0
411 0 0
413 0 0

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 203 OF 247
414 0 0
415 11 0
420 0 0
480 167 36
481 24 6
482 0 0
483 0 0
484 100 0
485 0 0
486 95 44
487 3534 715
488 6 0
500 49 0
501 0 0
502 0 0
503 26 0
504 34 0
505 0 0
600 18 95
603 35 0
604 13 51
606 4 2

Although “trending” all messages may be worthwhile, the two SIP methods tables that
should be monitored are bwSipStatsRegisterResponsesTable and
bwSipStatsInviteResponsesTable.
These tables should be polled daily to trend request/response messaging.
For INVITE in/out, sudden increases in “5XX Server Failures” and “6XX Global Failures”
should be investigated. 5XX Server Failures result in traps as well.
For REGISTER in, sudden increases in 404 User Not Found, 401 Unauthorized, and 482
Loop Detected responses should be investigated. Each 401 Unauthorized also results in
a trap.
Performance Measurement Int Node Type Threshold

bwSipStatsInviteResponseIns/ D AS T No, trending data only.

bwSipStatsInviteResponseOuts Trending responses to INVITEs
IN/OUT and REGISTERs IN can
help identify problems (that is,
many 503s coming in, 404
responses to registers).

[Link] Provisioning Server BCCT Queue Statistics

NOTE: This section is valid for the Profile Server as well (Provisioning application).

Release 13.0 introduced the BroadWorks Common Control Transport (BCCT) protocol.
BCCT is used to transport OCS to Application Server OSS/OCI messages, Application
Server to Network Server NSSYNC messages, third-party OCI provision systems, and
some internal functions such as protocol monitoring. BCCT queue statistics can optionally
be monitored.
Example:
AS_CLI/Monitoring/PM/ApplicationServer> get -r

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 204 OF 247
-------------------------------------------------------------------------
-----
broadworks/provisioningServer/psSystemModule/psInternalStats/
-------------------------------------------------------------------------
-----
*psSystemInternalQueueResets 0
psSystemInternalQueueTable:

(1) psSystemInternalQueueIndex
(2) psSystemInternalQueueName
(3) psSystemInternalQueueSize
(4) psSystemInternalQueueTimeAvg
(5) psSystemInternalQueueTimeMin
(6) psSystemInternalQueueTimeMax
(7) psSystemInternalQueueTimeMaxTimestampMSB
(8) psSystemInternalQueueLengthCurrent
(9) psSystemInternalQueueLengthAvg
(10) psSystemInternalQueueLengthMax
(11) psSystemInternalQueueLengthMaxTimestampMSB
(12) psSystemInternalQueueLengthMaxTimestampLSB
(13) psSystemInternalQueueTimeMaxTimestampLSB

(1) (2) (3) (4) (5) (6) (7)

(8)
(9) (10) (11) (12) (13)
1 OCIReporting Output Adapter 643 2979 0 158 268
0
0 0 268 2254028188 2258305114

psSystemInternalQueueTimeAvg – This BCCTIncomingMessageQueue gauge

provides the average time a message is in the BCCT queue. Note that this value is in
1/1000 of a millisecond, that is, divide by 1000 for the value in milliseconds.
Polling this gauge hourly provides a view on the average time a BCCT message remains
in the BCCT queue prior to processing.
The average BCCT queue time should be less than 100 milliseconds.
psSystemInternalQueueTimeMax – This BCCTIncomingMessageQueue gauge
provides the high-water mark time (in milliseconds) a message is in the BCCT queue.
Polling and resetting this gauge hourly can provide a view of the maximum time a
message was in the BCCT queue.
It is normal that at times the maximum delay (certainly on a small system configuration
such as a V120) is in the range of seconds. Repeated high maximum delays polled
should be reported to BroadSoft Support.
Performance Measurement Int Node Type Threshold

psSystemInternalQueueTimeAvg H AS P Yes. If the result was more than

300 milliseconds (low alarm) or
if more than 500 milliseconds
(high alarm).

psSystemInternalQueueTimeMax H AS P Yes. If the result is more than

five seconds for four polling
intervals (high alarm).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

The Application Server uses defined SMTP servers to send e-mail for features such as
Call Notify and Voice Messaging Notification. The Application Server tries the primary
SMTP server; if that server does not respond within 30 seconds, the Application Server
tries the secondary server.
AS_CLI/Monitoring/PM/ApplicationServer> get -r
--------------------------------------------------
broadworks/executionServer/smtpModule/smtpStats/
--------------------------------------------------
*bwSMTPTotalPrimaryEmailSendAttempts 559
*bwSMTPTotalSecondaryEmailSendAttempts 0
*bwSMTPPrimaryUnSuccessfulEmailSendAttempts 0
*bwSMTPSecondaryUnSuccessfulEmailSendAttempts 0

bwSMTPTotalPrimaryEmailSendAttempts/bwSMTPPrimaryUnSuccessfulEmailSen
dAttempts – These counters are pegged each time the Application Server successfully
sends an e-mail to the SMTP server.
Polling this counter hourly provides a view of Application Server to SMTP server traffic.
Each pegging of bwSMTPPrimaryUnSuccessfulEmailSendAttempts means the primary
SMTP server could not be reached. If this counter is being pegged regularly, the carrier
should investigate why the primary SMTP server did not reply.
bwSMTPPrimaryUnSuccessfulEmailSendAttempts/bwSMTPSecondaryUnSuccessf
ulEmailSendAttempts – These counters are pegged each time the Application Server
times out to the SMTP server or the SMTP server returns an error.
A bwSMTPConnectivityFailure trap is generated each time either of these counters is
pegged.
Every pegging of an unsuccessful e-mail send counter means that the SMTP server could
not be reached or returned an error code. If this counter is pegged regularly, the carrier
should investigate.
Depending on the setup of the SMTP server, unsuccessful e-mail send counters may be
pegged if an invalid e-mail address is set for a user.
Performance Measurement Int Node Type Threshold

bwSMTPTotalPrimaryEmailSendAt H AS T No, trending data only.

tempts

bwSMTPTotalSecondaryEmailSen H AS T No, trending data only.

dAttempts

bwSMTPPrimaryUnSuccessfulEm H AS T No, trending data only.

ailSendAttempts

bwSMTPSecondaryUnSuccessfulE H AS T No, trending data only.

mailSendAttempts

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

When using BroadWorks Voice Messaging, the Application Server is responsible for
connecting to the user’s mail server and downloading messages that are played back via
the voice portal. Each time a user logs in to the voice portal, the Application Server
connects as a mail client to the user’s mail account using the provisioned credentials.
Voice messaging retrieval statistics can be monitored.
Example:

AS_CLI/Monitoring/PM/ApplicationServer> get -r
-------------------------------------------------------------------------
-----
broadworks/executionServer/imsModule/imsStats/
-------------------------------------------------------------------------
-----
*bwIMSSuccessfulConnectionAttempts 285
*bwIMSUnsuccessfulConnectionAttempts 4
*bwIMSSuccessfulDownLoadAttempts 350
*bwIMSUnsuccessfulDownLoadAttempts 0

bwIMSUnsuccessfulConnectionAttempts – This counter is pegged when the

Application Server is unable to connect to a user’s mail server account as a mail client.
A bwIMSConnectionFailure trap is generated for every connection failure.
The unsuccessful connection is generally related to improper voice messaging
configuration for that user (for example, incorrect user ID or password). A protocol monitor
capture of the user’s attempt to retrieve messages should identify the problem.
When attempting to listen to the voice mail via the voice portal, the user hears the
announcement “This operation cannot be completed at this time”.
This counter should be polled periodically (for example, once a day) to track connection
failures. Ideally, the result should be “0”.
bwIMSUnsuccessfulDownLoadAttempts – This counter is pegged when the
Application Server is unable to download a message from a user’s mail server account.
A bwIMSRetrieveMessageError trap is generated for every retrieval failure.
This counter should be polled periodically (for example, once a day) to track connection
failures. Ideally, the result should be “0”.
If there are a high number of these failures daily, the carrier should investigate using the
generated trap time stamp as a reference to trace back into the Application Server
Execution Server logs.
Performance Measurement Int Node Type Threshold

bwIMSSuccessfulConnection H AS T No, trending data only.

Attempts

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

bwIMSUnsuccessfulConnecti H AS T No, trending data only since an alarm

onAttempts is generated.
Even a peg of one means that
someone has incorrectly configured
voice mail. No alarm should be raised,
but ideally, this number should be
zero.

bwIMSSuccessfulDownLoadA H AS T No, trending data only.

ttempts

bwIMSUnsuccessfulDownLoa H AS T No, trending data only since an alarm

dAttempts is generated.
Pegging in this case means that the
Application Server attempted to
download what was considered a valid
e-mail and it failed.

[Link] Database Usage

Application Server Provisioning Server and Execution Server database update and query
usage statistics can be “trended”.

Broadworks/executionServer/databaseModule/databaseStats/timesTen/
-------------------------------
*bwXSUpdateCount 3568
*bwXSQueryCount 43165

broadworks/provisioningServer/psDatabaseModule/psDatabaseStats/psTimesTen
--------------------------------
*bwPSUpdateCount 6702
*bwPSQueryCount 21992

bwXSUpdateCount/bwPSUpdateCount – These counters represent the number of

database updates initiated by the Provisioning and Execution Server processes.
XSQueryCount/bwPSQueryCount – These counters represent the number of database
queries initiated by the Provisioning and Execution Server processes.
Performance Measurement Int Node Type Threshold

bwXSUpdateCount/bwPSUpdateCount H AS T No, trending data only.

bwXSQueryCount/bwPSQueryCount H AS T No, trending data only.

18.3 BroadWorks SNMP Trap Monitoring

BroadWorks supports SNMPv2 and SNMPv3. Traps generated by all servers, should be
collected, and monitored using the service provider’s network management station (NMS).
For information on how to configure SNMP as well as a list of all BroadWorks traps, see
the BroadWorks Fault and Alarm Interface Specification along with each server’s specific
fault MIB.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 208 OF 247
There are hundreds of BroadWorks traps. Not all traps are indicative of a problem.
BroadWorks traps range in severity from Informational to Critical. Trap severity is based
on the impact on the system operations as a whole and should be dealt with according to
severity. Trap severities are a generalized view of impact on the system. Low and
medium severity traps should not be disregarded unless the impact is understood.
Although all traps provide value, as a best practice, the following sections provide a view
of the minimum subset of traps that should be monitored.

18.3.1 Common BroadWorks Server Traps

These traps are generated as part of the server scheduled tasks.
bwSystemHealthReport:
 Sent when healthmon task has been activated.

 Healthmon checks all BroadWorks component and disk space.

 Should be scheduled to run every 30 minutes.
 A trap is sent every 30 minutes. Notification severity indicates normal server
operation.
bwCPUIdleTimeLimitReached and bwMemoryLimitReached:
 Sent when the CPUMon task has been activated.
 CPUMon monitors server CPU usage, memory, and paging.
 Should be scheduled to run every six minutes.
 A trap is only sent if any of the monitored values surpass the thresholds defined in the
CPUMon script.
 Threshold levels can be modified by changing the bwCPUMon script.
bwDatabaseSyncReport:
 Sent when the synchcheck_basic.pl task has been activated on Application Servers,
Network Servers, and Messaging Servers.
 The synchcheck_basic.pl does a quick non-service impacting comparison of the
number of table rows in two peer databases.
 Should be scheduled to run once a night (off-peak hours) on the secondary server.
Do not activate on the primary server.
 A trap is only sent whenever the script executes. It is notification severity when
databases are the same and critical severity when a discrepancy is detected.

NOTE: Before escalating this trap to BroadSoft Support, manually run synchcheck_basic.pl
again. It is possible that during the snapshot test, not all replicated information was pushed, or
that tables were in flux.

18.3.2 Server Process Problems

The following critical severity traps are generated on all servers in the event of
BroadWorks process death. The process should be restarted automatically by the
process monitor, but any occurrence of these traps must be escalated to BroadSoft
Support to make sure the root cause of the process death is understood.
 bwPMProvisioningServerDeath
 bwPMExecutionServerDeath

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 209 OF 247
 bwPMNSProvisioningServerDeath
 bwPMNSExecutionServerDeath
 bwPMMediaServerDeath
 bwPMremotexlaDeath

18.3.3 Software Error Traps

The following traps are generated in the event of an internal software error. They do not
necessarily affect the system, but they are an indication of a software problem. These
traps must be escalated to BroadSoft Support to make sure the root cause of the software
error is understood.
 bwGeneralSoftwareError
 msSoftwareError

18.3.4 Media Server Traps

Although the Media Server generates a host of traps, the following traps affect the service
and they must be investigated.
msSmtpFailure
 Media Server could not relay the recorded voice messaging e-mail through the
requested SMTP server.
 Usually indicative of a problem on the SMTP server.
msHTTPConnectionFailure, msHTTPReceiveFailure, msHTTPFileNotFound, and
msInvalidAudioFile
 Media Server downloads .wav files (greetings, announcements, or voice mail) for
playback from the Application Server.
 Any of these traps indicate a failure with the download or file in question.
 General result is that the end user does not hear the greeting, announcement, or
voice mail (call failure).
msMemAllocFailure
 Media Server IVR process does not have enough memory.

 Every instance of this trap means that the Media Server could not allocate memory for
the call and the call fails.
 IVR memory can be increased under CLI level MS_CLI/Service/IVR, parameter
memorySize.
msNoPortAvailableErrors
 This trap occurs every time a request comes to the Media Server and it cannot
process it due to licensing limits.
 The Application Server should route advance to the next Media Server.
 From Release 12 and higher, the number of ports a Media Server can support is
calculated based on hardware type. The different port numbers are defined in file
/usr/local/broadworks/bw_base/conf/hardware_cap.csv. If the hardware type is not
known, the Media Server calculates the maximum number of ports based on number
of CPUs and CPU speed.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 210 OF 247
18.3.5 Network Server Traps
The Network Server does not generate many traps under normal operating conditions.
However, the following traps affect the service and they must be investigated.
bwNSSynchUpdateFailureError
 The Network Server could not process a synchronization request from an Application
Server (for example, to add or delete an enterprise, group, or user).
 The reason text should provide a clear explanation as to why the synchronization
failed. A synchronization failure results in a failure on the Application Server for that
specific task.
 One of the main causes for synchronization errors once an Application Server and
Network Server have been synchronized is attempting to assign a directory number
(DN) to an Application Server user that does not meet the valid NDC format for that
country code.
On the Network Server, you can check your specific country code rules under
NS_CLI/System/CallP/CountryCodes/NDC>.
bwLicenseThreshold
 This trap is generated when a Network Server reaches 80% of the license file defined
transactions per time period value.
 Once a Network Server hits the maximum transactions in the defined time period
window, it returns a SIP “503 Service Unavailable” to all SIP requests.
 This condition can be the result of a routing loop between the PSTN and BroadWorks.
For example, the PSTN considers that a number is owned by BroadWorks and
BroadWorks considers that the number is owned by the PSTN and the routingNE has
an off-net policy that causes the call to route back to the PSTN.
bwNSCallGotTreatment
 This informational severity trap is not necessarily indicative of a problem, but it could
be used as a flag whenever the Network Server hits a specific treatment.
 On the Network Server under NS_CLI/System/CallP/Treatment>, you can enable
which treatments result in a trap. By default, all are enabled.
 You may want to disable all traps and then enable only ones for treatments for which
you are most interested. For example, “unkgw” means the Network Server received a
SIP request from a device it does not recognize.

18.3.6 Application Server, Execution Server, and Profile Server Traps

Although the Application Server generates a host of traps, as a best practice, the following
traps should be monitored.
When using the IMS mode, Provisioning and Execution processes are divided between
the Profile Server and the Execution Server. Traps are also valid on the specified server,
when indicated.

[Link] POP3/IMAP Traps

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 211 OF 247
The Application Server must be able to reach defined SMTP server(s) to support features
such as Call Notify and Call Center Statistics Reporting. If none of the defined SMTP
servers can be reached, a bwSMTPConnectivityFailure trap is generated.
The Application Server Voice Messaging service uses a standard POP3/IMAP protocol to
store user’s voice mail. The Application Server must be connected to the user’s mail
server account to retrieve messages and manage the mailbox. Any one of the following
traps would indicate a problem related to Application Server Voice Messaging service for a
user:
 bwIMSConnectionFailure
 bwIMSRetrieveMessageError
 bwIMSRetrieveMailBoxInfoError
 bwIMSUpdateMailBoxError
 bwIMSCloseConnectionError

[Link] Routing Traps

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

bwMediaServerNotResponding
This trap is generated when a Media Server fails to respond to an Application Server
request in the allotted time. On the initial INVITE to the Media Server, the Application
Server uses the re-transmission parameters defined under the CLI level
AS_CLI/System/CallP/Routing/MediaServerSelection> MediaServerTimerLength.
By default, this timer is set to 1500 milliseconds. In general, a Media Server should
respond back within 1.5 seconds. Occurrences of this trap must be investigated to see if
the timer needs to be increased for a specific network.
bwMssNetworkServerNotResponding and bwMssNoResponse
The bwMssNetworkServerNotResponding trap, which relates to Application Server MSS
requests to the Network Server, is generated if a Network Server does not respond within
the allotted time. The MSS timer is controlled under the CLI level
AS_CLI/System/CallP/Routing/MediaServerSelection> mssRouteTimerLength.
By default, this timer is set to 800 milliseconds. In general, a Network Server should
respond back within 800 seconds. Occurrences of this trap need to be investigated to see
if the timer needs to be increased for a specific network.
From an operational perspective, a bwMssNetworkServerNotResponding trap is not
service affecting in that the Application Server route advances to the next Network Server.
On the other hand, a bwMssNoResponse trap means that no Network Server returned a
respond and the call failed.

[Link] Application Server/Network Server Synchronization Traps

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

These traps are generated on the Application Server in the event of a failure to push
user/group additions/deletions to the Network Server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 212 OF 247
bwNSSynchronizationConnectivityFailure
The Application Server could not open a TCP socket (or the established socket failed) to
the Network Server. An occasional occurrence of this trap is not an issue. The
Application Server re-opens the socket to the NetworkServer. However, continuous
occurrence of this trap indicates a connection problem and it must be escalated to
BroadSoft Support.
bwNSSynchronizationFailure
This trap is generated on the Application Server when an attempt to synchronize a
group/user addition/deletion failed. The trap contains detailed information about the
failure. There should be a corresponding trap on the Network Server as well.

[Link] SIP/MGCP Traps

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

These traps are generated on a per-call basis by the Application Server. Most are
classified as low severity, but a carrier should understand why they are occurring.
bwAuditAbnormalCallTermination
The Application Server audits (send a re-INVITE or UPDATE) every active call. If the
endpoint fails to respond, this trap is generated and the CDR release cause indicates an
AbnormalCallTermination reason. Session audit is controlled at the following CLI levels:
 AS_CLI/System/CallP/SessionAudit> active = true
 AS_CLI/Interface/SIP> sessionAuditAllowed = true
A session audit should NOT be disabled, as this can lead to a hanging session. The
default session audit interval is 10 minutes for every call. The interval is controlled with the
following CLI parameter (in seconds):
 AS_CLI/System/CallP/SessionAudit> interval = 600
If you are experiencing a large number of these traps, it can indicate an interoperability
issue with a specific device type.
bwSipServiceUnavailableReceived
This trap is generated in the event of an endpoint returning a 5xx SIP response. An
endpoint returning a 5xx response (for example, 500 or 503) generally means that the
device could not handle the call and depending on when the 5xx is received either results
in a call failure or the Application Server route advancing to the next contact.
This trap must be investigated by the carrier as it is indicative of a potential issue in the
network (for example, a device problem or no circuits were available on the PSTN
interconnect).
bwSipMaxRetriesExceeded
This trap indicates that the Application Server tried to send a SIP message to a device and
the device did not respond within the allotted time allowed in SIP. This can be considered
normal when a device is no longer available (for example, a SIP phone is unplugged).
When there are a large number of these traps, it must be investigated by the carrier.
bwMGCPUnrecognisedDomainName and bwSipUnrecognisedDomainName
These traps are generated when the Application Server needs to send a message to a
contact that is not an IP address and requires address resolution (for example, DNS,
/etc/hosts) and that the address resolution failed.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 213 OF 247
These traps can be serious, in that the call processing thread handling that specific call
hangs until the resolution is complete. If the DNS is slow in responding with the failed
lookup, the call processing thread can hang for many seconds.
Upon receiving this trap, the carrier should take action to make sure that the unknown
domain is addressed by either proper address resolution or device reconfiguration.
bwMGCPReTransmissionTimeout
This trap indicates that the Application Server tried to send an MGCP message to a device
and the device did not respond. Given that, MGCP integrated access devices (IADs) are
generally static devices and always online, this trap needs to be investigated by the
carrier.
bwSipAuthenticationFailure
This trap is generated when a SIP user has authentication enabled and the authentication
process fails. This is indicative of devices trying to register without the proper credentials.
These should be investigated by the carrier to determine the source device.

18.3.7 Application Server Execution Server Process Problems

NOTE: This section is valid for the Execution Server as well (ExecutionDataless application).

The following traps are generated on the Application Server whenever a serious problem
occurs with one of the call processing internal threads. The thread should be restarted
automatically, but any occurrence of these traps needs to be escalated to BroadSoft
Support to make sure the root cause of the thread death is understood.
 bwForcedExitDueToHungThread
 bwCallPThreadAutoRestart
Less serious than a thread death, a call processing thread can be delayed. A delay
can occur for various reasons. The following trap is generated whenever one of the
existing call processing threads is delayed for more than 2.5 seconds.
 bwThreadDelayDetected
Any occurrence of this trap needs to be escalated to BroadSoft Support to make sure
the root cause of the thread death is understood.

18.3.8 Overload Control Monitoring

The Application Server and Network Server support call processing overload controls.
They transition from one zone to another (green, yellow, or red) as call processing delays
increase. Once a server hits the red zone, an overload action occurs (either drops a
message or responds with a “302 Temporarily Moved” response, redirecting the user to
the secondary server). The bwOverloadZoneTransition trap is generated on a zone
transition.
Overload control parameters are configurable using the CLI levels as follows:
 AS_CLI/System/CallP/OverloadControls for the Application Server
 NS_CLI/System/ OverloadControls for the Network Server

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 214 OF 247
18.3.9 Device Operational State Monitoring
Device polling (Application Server/Network Server/Media Server/PGW) can be enabled on
the Application Server and Network Server. When activated, the device monitoring
periodically pings the device with a SIP OPTIONS method to see if the device is available.
Servers poll devices every five minutes (default setting) to establish the operational state.
The following traps are generated (depending on the device type) on a device failure to
respond to SIP ping:
 bwNetworkDeviceIsFailed
 bwAccessDeviceIsFailed
Once a device starts to respond to device pings again, the same alarm is sent out with a
status of “clear”.
The online/offline status provided by this feature does not take the device out-of-service
from a routing perspective. The BroadWorks server does not stop sending or receiving
messages from failed devices, but it moves them to the end of any route list. For example,
if a network gateway is not responding to pinging, it is returned as the last contact
regardless of the weighting results returned by call processing.
Polling can be enabled on the Network Server at the following CLI levels by setting the poll
parameter to “true”:
 NS_CLI/System/Device/HostingNE
 NS_CLI/System/Device/RoutingNE
 NS_CLI/System/Device/ResourceNE
Polling can be enabled on the Application server at the following CLI levels:
 AS_CLI/System/Device/NetworkServers
 AS_CLI/System/Device/Monitor/AccessDevice
A best practice is to enable routingNE, hostingNE, and resourceNE device polling on the
Network Server and NetworkServers device polling should be enabled on the Application
Server. Access device polling should not be enabled on dynamically registering devices,
such as SIP phones, since the user can take the device offline by unplugging it. Access
device monitoring should be restricted to static devices, such as IADs or trunking
gateways.

18.4 Miscellaneous Monitoring

The section outlines various miscellaneous items that should be monitored on
BroadWorks servers.

18.4.1 BroadWorks System Sanity

The health of the BroadWorks application should be monitored using the healthmon
command available on all BroadWorks servers. Healthmon can be configured as a
scheduled task that runs at the designated interval and provides a health check of a server
by sending an SNMP trap. It is recommended that the system provider configure
healthmon to run at 30-minute intervals.
Healthmon monitors the following items:
 File system size (disk usage)
 BroadWorks server processes
 SNMP subagents

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 215 OF 247
 On the Application Server, Network Server, and Messaging Server:
− Third-party processes (TimesTen, Tomcat, Apache, RSYNC, and tnameserver)
− Database replication (full two-way replication testing)
− Database replication lagging (for more information, see section 11.1 Application
Server, Network Server, and Messaging Server Database Replication and 11.1.2
TimesTen Replication Monitoring)
− Database notification lagging (for more information, see section 11.1 Application
Server, Network Server, and Messaging Server Database Replication and 11.1.2
TimesTen Replication Monitoring)
− Peer-to-peer SSH connectivity
The healthmon script Keep Alive trap can be configured using the command line
interface.
AS_CLI/Maintenance/Scheduler> add healthmon minute 30
...Done

AS_CLI/Maintenance/Scheduler> get
Id Name Date Day Hour Minute
===========================================================
2 healthmon - - - 30

2 entries found.

Under usual operating conditions, healthmon sends a SNMP trap with severity equal to
Informational. If healthmon detects a problem, the trap severity reflects the severity of the
problem and provides recommendation text on how to proceed.
Following is a sample output when replication is not running.
--------------------------------
System Health Report Page
BroadWorks Server Name: as1
Date and time : Thu Aug 7 09:56:41 EDT 2003
Report severity : CRITICAL
Server type : AppServer
--------------------------------
Replication: a remote server is not receiving replication updates.
Databases may be out-of-synch.
--------------------------------

Recommendations
---------------
Replication must be started on all remote servers (repctl start). If
databases are out-of-synch, they must be re-synchronized first (with the
[Link] tool). Please refer to the BroadWorks Maintenance Guide for
detailed procedures.
--------------------------------

Following is a sample output when database replication lagging and notification lagging
are detected.
--------------------------------
System Health Report Page
BroadWorks Server Name: as2
Date and time : Fri Jun 6 13:44:19 EDT 2003
Report severity : CRITICAL
Server type : AppServer
--------------------------------

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 216 OF 247
Recommendations
---------------
Replication is severely lagging on one of the servers in the cluster. If
the system is being operated in standalone mode, turn off replication.
Otherwise, import the database from the server exhibiting the lagging to
ensure prompt data synchronization.

Database XLA is severely lagging on one of the servers in the cluster.

If the files held by database notification continue to grow and the LSN’s
do not advance, contact BroadSoft support prior to restarting the
affected server.
--------------------------------

Healthmon, when run with the -d option, provides details on database replication lagging
and database notification lagging. For more information, see the healthmon description in
section 11.1.2 TimesTen Replication Monitoring.
The healthmon script can also be run manually from the UNIX prompt or from the
command line interface under AS_CLI/Maintenance/Tools. Healthmon should be
manually run on all servers after any maintenance task has been performed.

18.4.2 Redundancy Monitoring

The BroadWorks Application Server, Network Server, and Messaging Server redundancy
solution relies on TimesTen database replication to make sure the cluster members’
databases are synchronized. It is critical that replication is not stopped during normal
operations. Stopping TimesTen replication can result in a database out-of-sync condition.
Replication status is automatically monitored by healthmon and problems are reported as
part of the healthmon SNMP trap generation. Replication status can be manually checked
from the UNIX prompt using the repctl status command.
Example:
bwadmin@MTLAS04$ repctl status

Redundancy/Replication Status
-----------------------------
File Replication pid(s) = 29846
Datastore name = AppServer
Replication Agent Policy : always

Replication State
MTLAS04: (filerep: started) (database: started)
MTLAS01: (filerep: started) (database: started)

Database Replication Lagging State

MTLAS04: ok (1 files held)(earliest LSN 47/575216)
MTLAS01: ok (1 files held)(earliest LSN 49/123752)

Database Notification Lagging State

MTLAS04: ok (1 files held)
(ExecutionServer earliest LSN 47/552496)
(ProvisioningServer earliest LSN 47/565064)
MTLAS01: ok (1 files held)
(ExecutionServer earliest LSN 49/121472)
(ProvisioningServer earliest LSN 49/89312)

A Replication Agent policy set to “always” indicates that the replication process is running.
A Replication Agent policy set to “never” indicates that the replication process is not
running. The File Replication pid represents the process ID used by the RSYNC process.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 217 OF 247
Database replication can be manually tested on peer servers by adding data to one server
and ensuring it is present on the other, and then deleting it from the peer, ensuring the
delete operation is replicated back to the original server. For example, on the primary
server from CLI level System/Alias, add an entry called test, verify that it is replicated to
the secondary server, and then on the secondary server, delete the system alias test and
make sure it is deleted from the primary.
File replication can be manually tested on the Application Server by creating a test file
called [Link] on the primary server in /var/broadworks/userfiles/VM and ensuring that it
is replicated to the secondary server within 10 minutes. You can then add a test file on the
secondary server in /var/broadworks/userfiles/VM and verify that it replicates back to the
primary within 10 minutes. Note that you need to use a peercmd rm to delete the files
from both servers at the same time.
For more information on replication state, database replication lagging state, and database
notification lagging state data, see sections 11.1 Application Server, Network Server, and
Messaging Server Database Replication and 11.1.2 TimesTen Replication Monitoring.

18.4.3 Application Server Migrated User Monitoring

Under usual operating conditions, all users should be hosted on the primary Application
Server. However, it is possible that a user temporarily rolls over (migrates) to the
secondary server (for example, for an intermittent IP problem). The user rolls back to the
primary on the next origination or rolls back when the secondary Application Server
Redundancy (ASR) timer expires. This is normal and does not create a problem for a
redundancy system.
However, it is not usual to have a large number of users migrated to the secondary server
while the primary server is up and running, or to have users go between the primary and
secondary servers. This can be an indication of an improper DNS configuration or an IP
routing issue affecting a number of users. It is important to identify and correct the cause
of these unnecessary migrations.
The Application Server provides the nbOfMigratedUsers SNMP gauge to track the
number of migrated users. This is polled by the system provider’s Network Management
System (NMS).

AS_CLI/Monitoring/PM/ApplicationServer> get
-----------------------------------------------------------------
broadworks/executionServer/redundancyModule/redundancyStats/
------------------------------------------------------------------
nbOfMigratedUsers 2

The system provider should set up threshold alarms to provide SNMP traps when the
number of migrated users crosses a certain threshold. Two threshold alarms should be
configured, one for low severity and the other for medium severity. The low severity trap
provides an indicator for an acceptable amount of migrations. The medium severity trap
provides an indicator that a large number of users have been migrated and should be
investigated.

AS_CLI/Monitoring/Threshold> add gauge "5 users migrated. Verify that

users rollback to primary AS" nbOfMigratedUsers 0 5 low active

AS_CLI/Monitoring/Threshold> add gauge "50 users migrated. This is an

abnormal condition and should be investigated" nbOfMigratedUsers 6 50
medium active
...Done
AS_CLI/Monitoring/Threshold> get gauge

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 218 OF 247
Index Description
Name Low High Severity Status
===================================================================
1 5 users migrated. Verify that users rollback to primary AS
nbOfMigratedUsers 0 5 low active
2 50 users migrated. This is an abnormal condition and
should be investigated nbOfMigratedUsers 6 50 medium
active

Depending on network reliability, the high and low notification values may need to be
adjusted based on the service provider’s network.
Once a migration problem has been identified, the actual migrated users can be identified
using the CLI under the System/Redundancy/MigratedUsers level. This information can
be helpful to identify the root cause of the problem.
AS_CLI/System/Redundancy/MigratedUsers> get
User ID
======================
fred@[Link]
mkushnir@[Link]

18.4.4 Database Synchronization Monitoring

The Application Server, Network Server, and Messaging Server peer databases are kept
synchronized using TimesTen replication. TimesTen replication is not a Database
mirroring application, but instead pushes transactions from one server to the other. If the
replication daemon is stopped on either peer, the peer databases may not be
synchronized.
Database synchronization can be verified manually by running the
synchcheck_basic.pl script on the secondary server using the –a option.
bwadmin@bsns2$ synchcheck_basic.pl -a
Report:
synchcheck_basic.pl started Fri Jul 29 10:31:35 EDT 2005
Performing basic comparison between datastore NetworkServer on bsns2
(local) and NetworkServer on ns1
Databases appear in sync
Recommended Actions:
None

The synchcheck_basic.pl performs a row count comparison of every database table. It is

not service affecting and can be run at any time. If discrepancies are discovered, it
indicates the TimesTen table in issue.
The dbSyncCheck is scheduled task that should be configured on the secondary server to
run every night. It performs a database comparison and sends a critical severity
bwDatabaseSyncReport trap if any discrepancies are discovered.

NOTE: The synchcheck_full.pl script performs a row-by-row comparison of the peer

databases. This script severely affects performance in that it locks both peer databases during
the compare. In addition, synchcheck_full.pl requires twice the amount of the database size
memory free on the server running the script, since the peer database is imported and compared
in memory. It is not recommended to run synchcheck_full.pl on a production server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

The valid time zones supported by the BroadWorks Application Servers are stored in the
following configuration files:
 [Link]
 [Link]
[Link] contains a list of valid time zones. Each line has a time zone ID
display name and a supported flag. If the supported flag is set to “Y”, then the time zone
can be used by users and groups. All time zones used in BroadWorks should be included
in the [Link] file. A system default time zone can be configured for the
Application Server that is running BroadWorks. The default time zone is also a supported
time zone even if it is not included in the [Link] file.
[Link] contains a list of labels for those supported time zones in the
[Link] file. Each line contains two values; the first one is the time zone
display name, which has to match the display name value in the [Link] file.
The second value is the label used to display the time zones. For more information on
how to localize this file for multiple languages, refer to the BroadWorks Web Portal
Customization and Localization Guide.
The list of supported time zones are returned by the
SystemTimeZoneGetListRequest OCI-P command, and can be viewed on the web
portal as a drop-down list wherever the time zone can be assigned. This includes the
Group Profile, User Profile, and Call Center Profile pages.
When a time zone is assigned to a user or group for example, it is validated against the list
of supported time zones as mentioned above. However, after a time zone is assigned, it
can become invalid after one (but not limited to) of the following changes:
 The time zone is removed from the [Link] file or not “supported” any
longer.
 The system default time zone is not in the [Link] and has been
changed.
 The time zone name is changed in java following a system/java upgrade.
If a group has an invalid time assigned, the system default time zone is returned as part of
the group profile, when requested. If a user (real user or virtual user such as call center or
hunt group) has an invalid time zone, the group time zone – or the system default time
zone if the group time zone is invalid – is returned as part of the user profile.
When adding or modifying a user or a group, only the valid time zones can be assigned or
modified.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

20.1 [Link]
Usage – [Link] <dsn> <file>

NOTE: A database backup can also be initiated from the CLI /Maintenance/Tools level using the
backupdb command.

Description – This script backs up the content of the BroadWorks TimesTen datastore to
the specified directory. Database backups can be initiated with the application running.

20.2 bwCPUMon
Usage – bwCPUMon [-h]
Description – CPU monitoring script (for CPU idle percentage, and memory usage) that
can be used to automatically generate a trap with severity level depending on the
configured idle time threshold that has been surpassed. This script should be set up via
the CLI as a configurable maintenance task at five-minute intervals.
The generated BroadWorks traps are:
 bwCPUIdleTimeLimitReached (multiple thresholds)
 bwMemoryLimitReached (one threshold)

20.3 bwListPortInUse
Usage – bwListPortInUse
Description – This tool provides a list of all ports transmission control protocol (TCP) and
user datagram protocol (UDP) configured to be used by BroadWorks.
Sample of bwListPortInUse:
Usage Type Port Internal
-------------------------------------------------------------------
SNMP agent UDP 8001 N
SNMP agent trap server TCP 30002 Y
MO daemon TCP 7120 Y
MO root daemon TCP 7121 Y
SW Manager TCP 2223 N
SW Manager (Perl) TCP 2224 Y
TimesTen Main Daemon TCP 17003 N
XLA Server TCP 2048 Y
Replication port TCP 17888 N
ASR interface UDP 5090 N
CAP Server TCP 2206 N
MGCP interface UDP 2727 N
SMDI server UDP 11234 N
SIP listening port UDP 5060 N

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

NOTE: A database restore can also be initiated from the CLI /Maintenance/Tools level using the
restoredb command.

Description – This script restores the content of the BroadWorks TimesTen datastore
from the specified directory. The content must have previously been backed up with
[Link].
There can be no connection to the database when restoring a database (the application
must be stopped and replication must also be stopped). Logs for the database restore
can be found in /var/broadworks/[Link].

20.5 bwshowver
Usage – bwshowver
Description – This script is used to view the active BroadWorks release and applied
patches. For a list of applied patches, use the bwshowver –showPatch option.
Sample of bwshowver:
bwadmin@[Link]$ bwshowver
AS version Rel_17.0_1.430

Built Fri Mar 26 09:15:30 EDT 2010

- BASE revision 139067
- AS revision 139067

Patching Info:
Active Patches: 0

20.6 [Link]
Usage – [Link] [-h]
Description – This command should be used to validate that the MoDaemon and
MoDaemonRoot are properly configured and operational on the target server.
Sample of [Link]:
bwadmin@MTLAS04$ [Link]
************************************************************
* BroadWorks [Link] *
* *
* Version: 1.0 *
************************************************************

Validating data

Validating configuration
Validating bwMoDaemon
... Found port value: 7120

Validating bwMoDaemonRoot
... Found port value: 7121

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Validating bwMoDaemonRoot on host:localhost and port:7121

Done

20.7 check_dbpages.pl
Usage – check_dbpages.pl <dsn> <detect|modify>
Description – This utility is used to optimize the paging in the database. The detect
option can be used to check if any table is either undersized or oversized, and the modify
command is used to perform the optimization.

NOTE: This script should be executed during a maintenance window as it causes the database
to be temporarily locked.

bwadmin@mtl64lin02$ check_dbpages.pl networkserver detect

bwadmin@mtl64lin02$ check_dbpages.pl networkserver modify

************WARNING************

This script should only be run during a maintenance window!

This script will cause the database to lock temporarily.
Do you want to continue (y/n)? y

Type ? or "help" for help, type "exit" to quit ttIsql.
All commands must end with a semicolon character.

connect "dsn=networkserver";
Connection successful:
DSN=NetworkServer;UID=bwadmin;DataStore=/usr/local/broadw
orks/bw_base/persistent/NetworkServer;AutoCreate=0;LogFileSize=8;DurableC
ommits=
0;WaitForConnect=0;DRIVER=/usr/local/TimesTen/tt_base/lib/[Link];SMPO
ptLevel
=1;LogBuffSize=8192;Authenticate=0;PermSize=128;TempSize=42;
(Default setting AutoCommit=1)

run /tmp/tmp_file.sql;

exit;
Disconnecting...
Done.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 223 OF 247
20.8 config-hostname
Usage – config-hostname
Description – This utility is used whenever a host name must be changed for a
BroadWorks server. This utility calls the configure-network script.
Sample of config-hostname:
---- Hostname Configurator version 1.0 ----

Data Gathering 
What is the new hostname? [as1]^C
as1$ config-hostname ?
---- Hostname Configurator version 1.0 ----

Data Gathering 
What is the new hostname? [as1]
What is the new address? [[Link]]
Changing system hostname 
Please enter root password:
Setting new hostname to as1… [done]
now processing /etc/hosts… [done]
now processing /etc/nodename… [done]
now processing /etc/net/ticlts/hosts… [done]
now processing /etc/net/ticots/hosts… [done]
now processing /etc/net/ticotsord/hosts… [done]
now processing /etc/hostname.dmfe0… [done]

You may have to manually update the HTTP interfaces

(CLI/Interface/Http)

==> BroadWorks configuration <==

Configuring new BroadWorks hostname... [done]
Restart replication on all servers of the cluster using repctl
restart.

---- Hostname Configurator completed! ----

20.9 config-redundancy
Usage – config-redundancy
Description – This utility is used to add a peer to a cluster. It calls the config-ssh utility.

20.10 config-ssh
Usage –./config-ssh [-createKeys | -remote] peer1 peer2 …
Description – This is an SSH configuration utility that is used to create keys, propagate
them to other peers, and ensure connectivity between the peers.
Options:
 createKeys allows SSH keys to be created for each peer
 remote requests that peers do not iterate down to other peers
peer1 peer2 … are the FQDN (equal to host name) of all peers.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

WARNING: The dbsetup script is wrapped by other BroadWorks scripts and should not be run
unless requested by BroadSoft Support.

Description – This script encompasses all aspects of database setup. It may perform
one or more of the following steps depending on the desired configuration:
 Backs up the datastore to upgrade.
 Installs a fresh datastore, meaning that tables are created and basic data fill is added
(using the -install option).
 Populates the list of redundant peers.
 Locks all redundant servers before performing an upgrade.
 Unlocks the current server once an install or upgrade is complete.
 Upgrades a previous datastore (the backup version is restored in the new location
and patches are applied).
It uses directly or indirectly the following scripts:
 [Link]
 [Link]
 peerctl
 [Link]
 [Link]
 [Link]

20.12 [Link]
Usage – [Link] <dsn>
Where – dsn - Datastore name (either CallCenterRDB, radius, or WebNmsDB)
Description – This tool provides the MySQL database size of the given datastore name.
It is available for the Element Management System.

20.13 healthmon
Usage – healthmon <-h|-v|-s|-l|-c> [-p] [-d] [5,10,15,20,30,60,120,720,1440,remove,show]
Options – Options are as follows:
 -h: print this usage note
 -v: print the version of the script
 -s: generate a report and send the result through an SNMP trap
 -l: generate a local to screen
 -p: test only the disk partitions status
 -d: generate a detailed trace of all the tests executed
This option is ignored for an SNMP report.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 225 OF 247
 -c: schedule the script to run in background with the -s option
You must specify a valid scheduling timeslot:
− Every 5, 10, 15, 20, or 30 minutes
− Every hour (60), every other hour (120), twice a day, (720) or daily
 Keyword “remove” removes the healthmon scheduling entry
 Keyword “show” shows the current schedule
Description – This tool is used to monitor and report the status of a BroadWorks server.
The report can be generated locally at the console or sent over the network through an
SNMP trap. Note that a trap is still sent even when no problems are detected (keep alive).

20.14 [Link]
Usage – [Link] <local dsn> <remote hostname> <remote dsn>
Where:
 local dsn – Local datastore name (either AppServer or NetworkServer)
 remote host name – Remote server UNIX host name
 remote dsn – Remote datastore name (either AppServer or NetworkServer)

NOTE: A database import can also be initiated from the CLI Maintenance/Tools level using the
importdb command.

Description – This script imports the content of a remote database in the specified
datastore. The remote server must have its replication enabled. Logs for the database
import can be found in /var/broadworks/[Link].

20.14.1 [Link] on Element Management System

On the Element Management System server, the [Link] command is defined
differently compared to how it is on other BroadWorks servers.
Usage – [Link] <remote hostname>
Where:
 remote host name – Remote server UNIX host name
Example:
bwadmin@ems2$ [Link] ems1
Re-spawning as root...

-> Disabling peer database replication <-

[done]

-> Making safety backup <-

Backing up ems local db to
/var/broadworks/backup/EMS_Rel_14.sp3_1.[Link]...
[done]

WARNING: ALL information in ems db will be lost.

Press ctrl-c now to abort or <enter> to continue...
Continuing.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 226 OF 247
-> Backing-up data on primary server <-
[done]
-> Importing data from primary server <-
[done]
-> Destroying ems database <-
[done]
-> Extracting data to ems database <-
[done]
-> Re-setting local redundancy data <-
[done]
-> Re-setting peer redundancy data <-
[done]
WARNING: You should re-enable local database replication.
-> Re-enabling peer database replication <-
[done]

[Execution Completed]

bwadmin@ems2$

20.15 [Link]
Usage – ./[Link] <NS version> [--run-once]
Description – This script is used to install new dial plans on the Network Server. Dial
plan files can be created offline and loaded by placing the new dial plan in the
/usr/local/broadworks/bw_base/persistent/setup/dataoptional directory and updating the
[Link] with the new entry.
Sample of $ ./[Link] NS_Rel_8.0_1.43:

==> Dial plan installation <==

The following dial plans are already installed:

CC Name
--------------
1 NADP
1 NADP_E164
1 NADPCAC

Optional dial plan list:

CC FILENAME Description
------------------------------------------------
[1] NS.CC_DIAL_PLANS "North American dial plan"
[61] NS.AUSTRALIA_DIAL_PLANS "Australian dial plan"
[65] NS.SINGAPORE_DIAL_PLANS "Singapore dial plan"
[81] NS.JAPAN_DIAL_PLANS "Japan dial plan"

Which dial plan would you like to install (enter number only)? 65

Now installing NS.SINGAPORE_DIAL_PLANS...

==> Dial plan installation completed! <==

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 227 OF 247
20.16 peerctl
Usage – peerctl [command]
Where the command must be one of the following:
 ls
 add <hostname> <address> <repType(0=master;1=slave)>
 set <hostname> <address> <repType(0=master;1=slave)>
 setReplicationAddress <hostname> <replication routing address>
 clearReplicationAddress <hostname>
 showReplicationRouting
 del <hostname>
 lock [<hostname>|all]
 unlock [<hostname>|all]
 setPrimary [<hostname>]
 waitAllLocked <timeout in minutes (typically 12)>
Where host name is optional (omitting it means the current peer).
Description – This script is used to manage cluster peers. Its primary use is to list all
cluster peers, add/delete peers from a cluster, and lock/unlock the TimesTen database. It
can also be used to specify a different interface for TimesTen replication than the
configured peer address.
The following example illustrates how to set an address of the replication interface.
bwadmin@mtlsol25$ peerctl setReplicationAddress MTLSOL-25 [Link]

Replication traffic for peer 'MTLSOL-25' will be routed via address

'[Link]'
BroadWorks replication must be restarted by using the 'repctl' command
before this change will take effect.

To ensure that replication can function properly, you must generate ssh
keys for
the new routing address '[Link]' by using the 'config-ssh -
createKeys' utility

As indicated, ssh keys, for the new interface must be generated as follows.
bwadmin@mtlsol25$ config-ssh -createKeys bwadmin@<new-replication-
address-on-Prim> bwadmin@<new-replication-address-on-Secondary>

If you want to clear the routing address and put back the initial settings, execute the
following command.
bwadmin@mtlsol25$ peerctl clearReplicationAddress MTLSOL-25

WARNING: When the cluster is locked through the command ‘peerctl lock all’, the device
registrations and provisioning changes are prevented.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 228 OF 247
20.17 peercmd
Usage – peercmd <command to spawn across BroadWorks peers>
Description – This utility spawns a given command to all peers as listed by “peerctl ls”. It
can be used to simultaneously launch an action or gather the status on a BroadWorks
cluster. Complex commands can be created with escaped control characters.
Example: $ peercmd bwshowver\;showrun\&repctl status

20.18 repctl
Usage – /usr/local/broadworks/bw_base/bin/repctl {start|stop|restart|status}
Description – This script is used to start, stop, and get the status of file and database
replication. This script should always be used to start and stop replication as it performs a
cleanup. SSH must be properly configured before starting replication. Failing to configure
SSH leads to broken file replication. A lock file is created in /var/broadworks/lock/repctl to
prevent multiple instances from running simultaneously.
Sample of status when replication is started:
bwadmin@[Link]$ repctl status

Redundancy/Replication Status
-----------------------------
File Replication pid(s) = 4011
Datastore name = NetworkServer
Replication Agent Policy : always

Replication State
[Link]: (filerep: started) (database: started)
[Link]: (filerep: started) (database: started)

Database Replication Lagging State

[Link]: ok (1 files held)(earliest LSN 4/5321312)
[Link]: ok (1 files held)(earliest LSN 3/6284464)

Database Notification Lagging State

[Link]: ok (1 files held)
(NetworkServerExecutionServer earliest LSN 4/5323240)
(NetworkServerProvisioningSer earliest LSN 4/5325672)
(CAPProxyServrUtility earliest LSN 4/5319664)
[Link]: ok (1 files held)
(NetworkServerExecutionServer earliest LSN 3/8249064)
(NetworkServerProvisioningSer earliest LSN 3/8250240)
(CAPProxyServrUtility earliest LSN 3/8247736)

Sample of status when replication is not running:

bwadmin@[Link]$ repctl status

Redundancy/Replication Status
-----------------------------
File Replication pid(s) =
Datastore name = NetworkServer
Replication Agent Policy : manual
Replication Manually Started : False

Replication State
[Link]: (filerep: stopped) (database: stopped)
[Link]: (filerep: stopped) (database: stopped)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 229 OF 247
Database Replication Lagging State
[Link]: ok (0 files held)
[Link]: ok (0 files held)

Database Notification Lagging State

[Link]: ok (1 files held)
(NetworkServerExecutionServer earliest LSN 4/5963440)
(NetworkServerProvisioningSer earliest LSN 4/5964880)
(CAPProxyServrUtility earliest LSN 4/5966056)
[Link]: ok (2 files held)
(NetworkServerExecutionServer earliest LSN 3/8249064)
(NetworkServerProvisioningSer earliest LSN 3/8250240)
(CAPProxyServrUtility earliest LSN 3/8247736)

20.19 resizeDSN
Usage – resizeDSN

WARNING: The resizeDSN tool requires the application to be stopped. This tool should be
used with caution and only used following an explicit acknowledgement from BroadSoft Support
personnel.

Description – This command can be used to resize the Application Server, Network
Server, or Messaging Server DSN. This tool can only be run as root.

NOTE: For information on database sizing, see the BroadWorks System Engineering Guide.

20.20 showrun
Usage – showrun
Description – This script is used to view all active BroadWorks processes. An empty list
means the application is not running.
Example of showrun:
bwadmin@MTLAS01$ showrun

Currently running (BroadWorks related) processes:

Execution Server process monitor (pid=2329)

Execution Server (pid=2367)
Provisioning Server process monitor (pid=2344)
Provisioning Server (pid=2387)
Open Client Server process monitor (pid=2391)
Open Client Server (pid=2429)

BroadWorks RemoteXla Server process monitor (pid=2290)

BroadWorks RemoteXla Server (pid=2322)
BroadWorks SNMP Agent process monitor (pid=2276)
BroadWorks SNMP Agent (pid=2280)

tnameserver (pid=2296)

Tomcat process monitor (pid=2372)

Tomcat (pid=2422)
Apache (pid=2438)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 230 OF 247
20.21 startbw
Usage – startbw
Description – This script is used to start the BroadWorks application. As part of the
startup sequence, this script also checks if database replication is lagging on any peers in
the cluster.
Example of startbw:
bwadmin@[Link]$ startbw
BroadWorks control script version 17.0
Starting BroadWorks...

Broadcast message from bworks (Mon Apr 5 08:13:05 2010):

===== BROADWORKS CONTROL --- START INITIATED =====

Starting remotexla...
Starting nsExecution...
Starting nsProvisioning...
Starting tomcat...
Starting apache...

20.22 frepctl
Usage – frepctl <-start|-stop|-restart> [-daemon] [-once] [peer list] [-i, --interval timeInSec,
default 600] [-v, --verbose] [-h, --help]

WARNING: The [Link] file should only be modified after consulting BroadSoft Support
personnel. Adding new directories to file replication may affect system performance. This script
is called from the repctl script and under normal conditions, should not be run manually.

Description – This command is used to start RSYNC file replication. This command
starts ssh-agent, holds SSH passphrase, and iterates over the list of configured peers to
synchronize configured files. Files or directories can be added or removed from file
synchronization in /usr/local/broadworks/bw_base/conf/[Link]. In [Link], comments
are supported (comments start with a #) and entries can be a specific file or a directory.
For a directory, you must add a “.” to clearly specify that it is the content that is replicated.
Logs reside in /var/broadworks/logs/rsync/[Link], where XX is the current day. A lock
file is created in /var/broadworks/lock/[Link] to prevent multiple instances from running
simultaneously.
Example conf/[Link]:
#File replication list
#[2001-22-13 Basic file replication configuration]
/bw/local/broadworks/bw_base/public_html/.
/bw/local/broadworks/bw_base/conf/[Link]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 231 OF 247
20.23 stopbw
Usage – stopbw [--force]
Description – This script is used to gracefully stop the BroadWorks application. The --
force option to force BroadWorks to stop even if a service disruption will occur (for
redundant servers only).

WARNING 1: The following error can occur:

BroadWorks cannot be stopped at this time…

This occurs on redundant servers if a disruption of service is expected after BroadWorks is

stopped (peer is not available for servicing). This check can be bypassed by using the –force
option.

WARNING 2: The following error can occur:

***FAILURE***
***Some processes did not terminate gracefully
***Killing -9 (may result in broken pipes)…

This occurs because some processes take more time to shut down. If the showrun command
does not give any running process after the stopbw, everything is fine.

Example stopbw:
bwadmin@[Link]$ stopbw
BroadWorks control script version 17.0
Stopping BroadWorks...

Broadcast message from bworks (Mon Apr 5 08:12:02 2010):

===== BROADWORKS CONTROL --- STOP INITIATED =====

Stopping tomcat...
Stopping apache...
Stopping nsExecution...
Stopping nsProvisioning...
Waiting for core processes to terminate.........
... Done.
Stopping remotexla...
Waiting for core processes to terminate.......
... Done.

Currently running BroadWorks Application processes:

Currently running BroadWorks Platform processes:

BroadWorks Configuration Agent process monitor (pid=5984)

BroadWorks Configuration Agent (pid=6012)
BroadWorks Software Manager (pid=26161)
BroadWorks SNMP Agent process monitor (pid=31868)
BroadWorks SNMP Agent (pid=31896)

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 232 OF 247
20.24 synchcheck_basic.pl
Usage – synchcheck_basic.pl [-s] <-a | <local dsn> <remote hostname> <remote dsn>
Description – This script is used for redundant Application and Network Servers, to make
sure that the specified local database is identical to remote databases.
This command is used to determine if two databases are synchronized. It performs a
basic synchronization sanity test, which validates that peer databases are synchronized. It
checks that the number of rows in database tables is the same on peer servers, and
provides a list of suspect tables on mismatches.

20.25 synchcheck_full.pl
Usage – synchcheck_full.pl

WARNING: This script can take ten minutes or longer and keeps servers locked while running.

Description – This script for redundant Application and Network Servers ensures that the
specified local database is identical to the remote databases.
All local and remote servers are locked while the script runs, so it should only be run in
periods of low system usage. No changes to the databases take effect while the script is
running.

20.26 ttStatus
Usage – ttStatus
Description – This tool provides information on the TimesTen databases in the system.
Connections to the database indicate that the database is in use. These connections are
related to the TimesTen replication daemon, the BroadWorks application, or a manual
connection via ttIsql.
Example ttStatus:
bwadmin@[Link]$ ttStatus
TimesTen status report as of Sun Oct 16 10:23:15 2011

Daemon pid 1842 port 11284 instance [Link].4

TimesTen server pid 1850 started on port 11285
------------------------------------------------------------------------
Data store /bw/broadworks/persistent/AppServer
There are 48 connections to the data store
Shared Memory KEY 0x10036160 ID 557061
Type PID Context Connection Name ConnID
Process 3087 0x000000001821aa90 remotexla 5
Process 3087 0x0000000018323440 remotexla 6
Process 3583 0x00000000468846d0 Execution Server 7
Process 3583 0x000000004698eb50 Execution Server 11
Process 3583 0x00000000469f81d0 Execution Server 12
Process 3583 0x0000000046a6e450 Execution Server 13
Process 3583 0x0000000046ad5040 Execution Server 14
Process 3583 0x0000000046b3fa50 Execution Server 15
Process 3583 0x0000000046be03a0 Execution Server 10
Process 3583 0x0000000046d96c30 Execution Server 18
Process 3583 0x0000000046eb2800 Execution Server 19
Process 3583 0x000000004710a1b0 Execution Server 20
Process 3583 0x0000000047175190 Execution Server 23
Process 3583 0x0000000047225e10 Execution Server 24

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 233 OF 247
Process 3583 0x00000000474ae430 Execution Server 34
Process 3583 0x00002aaab73185e0 Execution Server 33
Process 3583 0x00002aaab737cdb0 Execution Server 29
Process 3583 0x00002aaab75e85b0 Execution Server 31
Process 3583 0x00002aaab76a4bd0 Execution Server 32
Process 3583 0x00002aaab7c20d00 Execution Server 36
Process 3583 0x00002aaab7cbad70 Execution Server 30
Process 3583 0x00002aaab7e4d260 Execution Server 38
Process 3583 0x00002aaab84b8b70 Execution Server 27
Process 3583 0x00002aaab8703420 Execution Server 25
Process 3583 0x00002aaab8b78290 Execution Server 28
Process 3583 0x00002aaabc2cc340 Execution Server 26
Process 3583 0x00002aaabc45d840 Execution Server 8
Process 3583 0x00002aaabc6cd340 Execution Server 35
Process 3611 0x0000000045957cf0 Provisioning Server 22
Process 3611 0x00000000459a2170 Provisioning Server 21
Process 3611 0x00002aaab42abaa0 Provisioning Server 16
Process 3611 0x00002aaab443aef0 Provisioning Server 9
Process 3611 0x00002aaab455f620 Provisioning Server 17
Replication 19582 0x000000001d0e9a60 LOGFORCE 4
Replication 19582 0x000000001d109e70 REPHOLD 2
Replication 19582 0x000000001d15d4c0 REPLISTENER 3
Replication 19582 0x000000001d1b0b10 TRANSMITTER 37
Replication 19582 0x000000001d2285b0 RECEIVER 1
Subdaemon 1848 0x00000000152d2b50 Manager 2032
Subdaemon 1848 0x000000001533e750 Flusher 2034
Subdaemon 1848 0x0000000015391fc0 Monitor 2035
Subdaemon 1848 0x00000000153e5830 Deadlock Detector 2036
Subdaemon 1848 0x00000000154390a0 Checkpoint 2037
Subdaemon 1848 0x000000001548c910 Aging 2038
Subdaemon 1848 0x00000000154e0180 Log Marker 2039
Subdaemon 1848 0x00000000155339f0 AsyncMV 2040
Subdaemon 1848 0x0000000015587260 HistGC 2041
Subdaemon 1848 0x00002aaaac042b70 Rollback 2033
RAM residence policy: Always
Replication policy : Always
Replication agent is running.
Cache Agent policy : Manual
------------------------------------------------------------------------
Accessible by group bwadmin
End of report

20.27 [Link]
Usage – [Link]
Description – This tool is used to modify the system configuration with the BroadSoft
hardening recommendations, as defined in section 4 of the BroadSoft System
Configuration Guide.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

20.28.1 [Link]
Usage – [Link] [-h | -help | <filename>]
Description – This script is used to back up the Software Manager database. The
backup file is stored in /usr/local/broadworks/swmanager/dbbackup. This script is located
in /usr/local/broadworks/swmanager/sbin.

NOTE: Backing up the patch tool database can cause a loss of information and it should be
done only when the patch tool is stopped.

20.28.2 [Link]
Usage – [Link] [<filename>]
Description – This script is used to restore a Software Manager database file created by
the [Link] script. The backup file includes the Software Manager database
and it should only be used with the same Software Manager revision that was used to
back it up. The [Link] script includes the Software Manager revision
number in the name of the backup file. This script is located in
/usr/local/broadworks/swmanager/sbin.

20.29 configdctl
Usage – configdctl {start | stop | stopnow | restart | status}
Description – This script is used to control the configuration agent daemon.
This command is used to start, stop, restart, and get the status of the configuration agent
daemon.

NOTE: The configuration agent must not be stopped while BroadWorks is running. This can
lead to unexpected behavior.

20.30 snmpdctl
Usage – lmddctl {start | stop | restart | status}
Description – This script is used to control the SNMP agent daemon.
This command is used to start, stop, restart, and get the status of the SNMP agent
daemon.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 235 OF 247
20.31 lmdctl
Usage – lmddctl {start | stop | restart | status}
Description – This script is used to control the license manager daemon.
This command is used to start, stop, restart, and get the status of the configuration agent
daemon.

NOTE: The License Manager must not be stopped while BroadWorks is running. This can lead
to loss of service.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

21.1 Add Memory to BroadWorks Server

At installation time, BroadWorks optimizes its configuration based on server
characteristics. For instance, memory usage of the different processes is calculated and
validated against the available amount of RAM. To take advantage of new memory
installed on a server, an administrator must run the [Link] command as
follows:
1) Stop BroadWorks.
2) Run the [Link] tool from a UNIX prompt.
3) Restart BroadWorks.
For the procedure to stop and restart BroadWorks, see section 7 Server Management.
Example:
bwadmin@[Link]$ [Link]

Optimizing server parameters...

Optimization completed.
Please restart BroadWorks to take advantage of the optimization.

21.2 Adjust BroadWorks Daylight Savings Time

21.2.1 Description
In 2005 and 2006, Daylight Saving Time (DST) began for most of the United States at
2 A.M. on the first Sunday of April. Then time reverted to standard time at 2 A.M. on the
last Sunday of October.
United States and Canada are extending Daylight Saving Time from 2007, and other
regions and countries in North America may follow. The new start date is the second
Sunday in March (currently the first Sunday in April) and the new end date is the first
Sunday in November (currently the last Sunday in October).
Year DST Begins 2 a.m. DST Ends 2 a.m.
First Sunday in April) (Last Sunday in October)
==== ========================== ===============
2005 April 3 October 30
2006 April 2 October 29

NOTE: The DST changes to the start and end date began March 2007.

Year DST Begins 2 a.m. DST Ends 2 a.m.

(Second Sunday in March) (First Sunday in November)
==== ========================== =============================
2007 March 11 November 4
2008 March 9 November 2
2009 March 8 November 1
2010 March 14 November 7
2011 March 13 November 6
2012 March 11 November 4

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

The new start and stop dates were set in the Energy Policy Act of 2005 in the United
States. In addition, the majority of the Canadian provinces have announced that they are
changing the dates for the DST to align with the U.S. Only the province of Saskatchewan,
which does not observe DST, as well as Nunavut Territory are not changing. Note that
Western Australia has introduced DST for a three-year trial period that started December
3, 2006 and ended March 25, 2007.

21.2.2 Service Impact

DST of current users on Application Servers is “out-of-phase” with the new DST as
described by the Energy Policy Act of 2005. This impacts time-oriented services, call log
date/time stamps, log message header time stamps, and so on. This alert needs to be
implemented prior to March 11, 2007.
Although not essential for the Xtended Services Platform, Network Server, and Media
Server since there is no time zone used on these servers, it is still recommended to
proceed with the following instructions. This is because the XSlogs, [Link], and the
snmp traps have time stamps associated with them, which means that logs and traps may
not be synchronized with other servers.

21.2.3 Solution
Linux:
Red Hat Enterprise Linux 4 – tzdata-2005m-1.EL4 is required. Connect to Red Hat Linux
and download this via the up2date command. For more information, read
[Link]
After using the up2date command, users must set the /bin/nice setuid bit: # chmod ug+s
/bin/nice. After running the chmod command as root, restart BroadWorks (reference Alert
222646).
BroadWorks (on Linux):
Release 11.1 – All servers: Execute the tzupdater tool.
Release 12.0 – All servers: Execute the tzupdater tool.
Release 13.0 – Application Server only: Upgrade the JDK to JDK1.5.0_07 from 1.5.0_03.
After the upgrade, execute the tzupdater tool.
Release 13.0 – All servers: Execute the tzupdater tool.
Release 14.0 – All servers: Execute the tzupdater tool.

21.2.4 Instructions
tzupdater Tool

NOTE: The official Sun “read me” is [Link]

Download the tool from [Link] (JDK US DST Timezone
Update Tool – 1.0.1).

Perform the following in a maintenance window as bwadmin, on each BroadWorks server

node:

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 238 OF 247
1) Make sure that peer BroadWorks servers are updated before moving on to another
BroadWorks server type.
2) Download the tool to each BroadWorks server.
3) Create directory /export/home/bwadmin/DSTChange.
4) Move the update tool archive into this directory.
5) Extract the archive within this directory.
6) Lock BroadWorks from the command line interface (CLI). Wait until the server is
locked.
7) When the server is locked, exit the CLI and stop BroadWorks.
8) Navigate to the /usr/local/java/java_base/bin directory.
9) Execute /java -jar <full path of location>/[Link] –u.
10) Start BroadWorks.

21.3 Change Media Server Kernel Selection on Linux

In some cases, it may be required to select a previously installed kernel to recover from
one of the following conditions:
 The newer kernel installed during a Media Server upgrade causes kernel panic.
 The kernel was upgraded to a more recent kernel than the one included in
BroadWorks packages and no longer has the real-time extensions.
To recover from one of these conditions, these steps must be followed:
1) From the server’s console, reboot the faulty server.
2) While the server boots, the GNU GRUB window appears. There is a five-second
count down. Press the ESCAPE key (ESC).
3) Using the down arrow, navigate to the previous kernel. (It should be “Red Hat
Enterprise Linux WS ([Link])”).
4) Press ENTER.
5) Once the server has booted, log in as root.
6) As root, remove the faulty kernel, for example, as root run “rpm –e kernel-smp-2.6.9-
[Link]”.
7) As root, edit /boot/grub/[Link] and verify that the default is set to the zero-based
index of the working BroadWorks kernel.
Example:
# [Link] generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this
file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /[Link]
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,0)/grub/[Link]

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 239 OF 247
hiddenmenu
title Red Hat Enterprise Linux WS ([Link])
root (hd0,0)
kernel /[Link] ro
root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /[Link]
title Red Hat Enterprise Linux WS ([Link])
root (hd0,0)
kernel /[Link] ro root=/dev/VolGroup00/LogVol00
rhgb quiet
initrd /[Link]

8) Reboot the server.

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Activation, Application management, 72 Application Server Execution Server, Profile Server

Active software versions, 87 traps, SNMP trap monitoring, 211
Display full details, 87 Application Server migrated user monitoring, 218
Software version management, 88 Application Server, Execution Server, Profile Server
Add BroadWorks performance measurements
Memory to BroadWorks server, maintenance monitoring, 190
tasks, 238 Application Server, Network Server, Messaging
New peer to redundant system
Server
Application Server, Network Server,
Add new peer to redundant system, 124
Messaging Server, 124
Database replication
Element Management System, 128
Redundancy management, 113
Network Function Manager, 132
Profile Server, 131 Application state, 70
Redundancy management, 124 Server management, 67
Additional MariaDB maintenance tasks Attributes, threshold alarms, 59
Database, 112 Automatic backups
Start and stop MariaDB database daemon, 112 File naming conventions, 93
Additional MySQL maintenance tasks Mode of operation, 93
Database, 111 Restoring auto-backup, 94
Start and stop MySQL database daemon, 111 Scheduled maintenance tasks, 92
Update MySQL database statistics, 111 Automatic disk cleanup, scheduled maintenance
Additional TimesTen maintenance tasks tasks, 96
Change TimesTen database size, 106 Backup
Compact database, 107 Database, 100
Database, 106 Database maintenance, 99
Import TimesTen datastores between operating Manual database, 99
systems, 108 [Link], 236
Start and stop database daemon, 107 BCCT listening ports, changing, 171
TimesTen connection optimization, 110
BroadWorks Application Server, time zone
Update TimesTen database paging
information, 107 support, 221
Update TimesTen database statistics, 106 BroadWorks license files, installing, 155
Alarms BroadWorks performance measurements monitoring
Clear, 59 Application Server, Execution Server, Profile
Close system, 59 Server, 190
Enable and disable real-time alarms Media Server, 179
echoing, 58 Network Server, 186
Fault and performance measurements, 55 BroadWorks server restoration, 141
Open system, 55 BroadWorks system backup, 141
View and set CLI alarm backlog size, 57 License, 142
View system, 56 Procedure, 142
Apache logs, 54 BroadWorks system backup, 141
Application management, 70 BroadWorks system monitoring, 172
Activation, 72 Application Server migrated user, 218
Deactivation, 75 Database synchronization, 219
Deployment, 73 Performance measurements, 178
Installation, 71 Platform, 172
Undeployment, 73 Redundancy monitoring, 217
Uninstallation, 75 SNMP trap, 208
Application Server and Network Server System sanity, 215
Database replication [Link], scripts and utilities, 222
TimesTen database synchronization, 113 bwCPUMon, scripts and utilities, 222
TimesTen replication monitoring, 115 bwListPortInUse, scripts and utilities, 222
TimesTen replication port number, 117 [Link], scripts and utilities, 223
Application Server Execution Server bwshowver, scripts and utilities, 223
Heap monitoring, platform, 176, 177
[Link], scripts and utilities, 223
Process problems, SNMP trap monitoring, 214

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 241 OF 247
Change Configure
BCCT listening ports, 171 IP networking information at UNIX level,
HTTP server network configuration, updating changing IP address or host name, 161
BroadWorks network configuration, 162 Redundant routing, changing IP address or host
TimesTen database size, additional TimesTen name, 163
maintenance tasks, 106 SSH between servers, SSH/RSYNC
Change IP address or host name configuration, 134
Configure IP networking information at UNIX Threshold alarms, CLI, 60
level, 161 Thresholds, 62
Configure redundant routing, 163 Cookie password, 154
Import EMS database, 170 CPU
Update Application Server and Network Server Monitoring tool, scheduled maintenance
address on OCS, 167 tasks, 97
Update Application Server call processing Usage, platform monitoring, 173
routing information, 163 Create SSH keys, SSH/RSYNC configuration, 134
Update Application Server IP address on
Database backup
Network Server, 164
Application Server, Element Management
Update EMS redundancy configuration, 169
System, Call Center Reporting Server,
Update Media Server IP address on Application
Network Server, Messaging Server,
Server, 167
Network Function Manager, 100
Update Media Server IP address on Network
Maintenance, 99
Server, 166
Manual, 99
Update network configuration, 162
Update Network Server IP address on Database import
Application Server, 165 Application Server, Element Management
Update Network Server or Application Server System, Network Server, Messaging
address on Xtended Services Server, 103
Platform, 169 Maintenance, 102
Update Network Server routing alias list, 165 Database installation, maintenance, 104
Update Xtended Services Platform IP address Database maintenance, 99
on Network Server, 168 Additional MariaDB maintenance tasks, 112
Change IP address, host name, FQDN name, Additional MySQL maintenance tasks, 111
gateway, or DNS server, 157 Additional TimesTen tasks, 106
Backup, 99
Changes
Import, 102
History, 19
Installation, 104
Release 10.0, 26
Recreate Application Server, Network Server,
Release 11.1, 26
or Messaging Server database, 104
Release 12.0, 26
Restore, 100
Release 13.0, 25
Release 14.0, 24 Database restore, 100
Release 15.0, 23 Application Server, Element Management
Release 16.0, 23 System, Call Center Reporting Server,
Release 17.0, 21 Network Server, Messaging Server,
Release 18.0, 20 Network Function Manager, 101
Release 19.0, 19 Extract database backup from automatic
backup file, 101
Changing Media Server Kernel Selection on Linux,
Maintenance, 100
maintenance tasks, 240
Database scheduled tasks, maintenance tasks, 98
Check for running process, server management, 78
Database size monitoring, platform, 178
check_dbpages.pl, scripts and utilities, 224
Database statistics, scheduled maintenance
Clear, alarms, 59
tasks, 97
CLI alarm backlog size, 57
Database synchronization
Close system alarms, 59 Check, scheduled maintenance tasks, 97
Common BroadWorks server traps, SNMP trap Monitoring, 219
monitoring, 209 [Link], scripts and utilities, 226
Compact database, additional TimesTen Deactivation, Application management, 75
maintenance tasks, 107 Debugging
config-hostname, scripts and utilities, 225 Enable or disable debug logging, 51
config-redundancy, scripts and utilities, 225 Input channel attributes, 51
config-ssh, scripts and utilities, 225 Log input channels, 36

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 242 OF 247
Log rotation, 52 TimesTen datastores between operating
Log severity, 36 systems
Output channel attributes, 52 Additional TimesTen maintenance
Sample debug configuration file, 53 tasks, 108
Deployment, Application management, 73 [Link]
Description, DST adjustments, 238 Element Management System, 227
Device operational state monitoring, SNMP trap, 215 Scripts and utilities, 227
Disable cookie support, modifying [Link] file, 154 Input channel attributes, debugging, 51
Disable debug logging, 51 Install BroadWorks license files, 155
Disaster recovery strategy, 138 Install cluster of servers
BroadWorks server restoration, 141 Element Management System additional
Recover server OS, 138 parameters, 122
Redundancy management, 121
Disk activity, platform, 175
Installation
Disk cleanup, automatic, 96
Application management, 71
Display active software versions, software version Database, 104
management, 87 [Link], scripts and utilities, 228
DST adjustments, 238 Installed software versions, 87
Description, 238 List installed and active, 87
Instructions, 239
Instructions, DST adjustments, 239
Service impact, 239
Solution, 239 IP address, host name, FQDN name, gateway, or
Element Management System DNS server, changing, 157
Add new peer to redundant system, 128 License, 142
Additional parameters, installing cluster of License files, installing BroadWorks, 155
servers, 122 Limitation for Application Server thresholds, 63
Database replication Linux system logs, 54
MySQL database synchronization, 118
List installed software versions, software version
MySQL replication monitoring, 120
Redundancy management, 118 management, 87
[Link], 227 Lock Network Server or Application Server
Enable and disable real-time alarms echoing, 58 Force lock server, 80
Enable cookie support, modifying [Link] file, 154 Lock Network Server, Application Server, or
Enable debug logging, 51 Execution Server
Server management, 79
Extract database backup from automatic backup file,
Log input channels, debugging, 36
restore, 101
Log rotation, debugging, 52
Fault and performance measurements
Alarms, 55 Log severity, debugging, 36
Performance, 64 Logs, 32
Threshold alarms, 59 Apache, 54
File system monitoring, platform, 172 Debugging, 36
Enabling TimesTen, 54
Force
Linux system, 54
Lock server immediately, lock Network Server
Miscellaneous, 54
or Application Server, 80
Oracle logs, 54
Server into maintenance state, server
Tomcat logs, 54
maintenance state, 86
UNIX command execution logging, 53
frepctl, scripts and utilities, 232
Maintenance
Get server state, 70 Database, 99
[Link], scripts and utilities, 226 Recommendations, 27
Healthmon Maintenance tasks
Keep alives, scheduled maintenance tasks, 96 Add memory to BroadWorks server, 238
Scripts and utilities, 226 BroadWorks DST adjustments, 238
Server state monitoring tools, 83 Change Media Server Kernel Selection on
History, changes, 19 Linux, 240
Hot standby servers, 139 Miscellaneous, 238
Import Scheduled, 90
Database, 102, 103 Management
EMS database, changing IP address or host Servers, 67
name, 170 Software version, 87

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 243 OF 247
Managing redundancy, 113 Monitoring BroadWorks system, 178
Manual database backup, 99 Navigate through MIB nodes, 64
View current position in MIB, 65
MariaDB
View MIB node values, 65
Database daemon, 112
View next MIB node, 64
Maintenance tasks, 112
Platform monitoring
Measurements, fault, and performance, 55
Application Server Execution Server
Media Server heap, 176, 177
BroadWorks performance measurements BroadWorks system, 172
monitoring, 179 CPU usage, 173
Traps, SNMP trap monitoring, 210 Database size monitoring, 178
Memory usage/paging monitoring, platform, 175 Disk activity, 175
MIB node values, 65 File system, 172
MIB nodes, 64 Memory usage/paging, 175
Miscellaneous monitoring, 215 Thread activity, 176
Mode of operation, automatic backups, 93 Post-installation control, Redundancy
Modify [Link] file management, 134
Disable cookie support, 154 Procedure, BroadWorks server restoration, 142
Enable cookie support, 154 Processes, 28
Remember password cookie for Profile Server, adding new peer to redundant
administrators, 154 system, 131
MySQL Real-time alarms echoing, 58
Database daemon, 111
Recommendations, maintenance, 27
Database statistics, 111
Database synchronization, 118 Recover
Maintenance tasks, 111 Faulty redundant installation, Redundancy
Replication monitoring, 120 management, 133
Server in maintenance state, 86
Naming conventions, automatic backups, 93
Recover server OS, 138
Navigate through MIB nodes, performance
Hot standby servers, 139
measurements, 64 Incremental backups, 140
Network Function Manager, adding new peer to On-demand rebuild, 139
redundant system, 132 Recreate
Network Server Application Server, Network Server, or
BroadWorks performance measurements Messaging Server database, 104
monitoring, 186 Redundancy management, 113
Traps, SNMP trap monitoring, 211 Add new peer to redundant system, 124
Network Server and Application Server Application Server Network Server, Messaging
synchronization, Redundancy Server database replication, 113
management, 124 Element Management System, database
replication, 118
Non-redundant systems
Install cluster of servers, 121
Restore server to in-service, 81
Network Server and Application Server
Take server out-of-service for maintenance, 80
synchronization, 124
Obsolete BroadWorks versions, uninstalling, 156 Post-installation control, 134
On-demand rebuild, 139 Recover faulty redundant installation, 133
Open system alarms, 55 Remove peer, 133
Optional remember password cookie for SSH/RSYNC configuration, 134
administrators, 154 Redundancy monitoring, BroadWorks system, 217
Oracle logs, 54 Redundant routing, 163
Out-of-service for maintenance, 80 Redundant systems
Output channel attributes, debugging, 52 Remove peer, 133
Overload control monitoring, SNMP trap, 214 Restore server to in-service, 82
Take server out-of-service for maintenance, 81
Password cookie, 154
Registration audit, scheduled maintenance tasks, 97
Peer, remove from redundant system, 133
Release 10.0, changes, 26
peercmd, scripts and utilities, 230
Release 11.1, changes, 26
peerctl, scripts and utilities, 229
Release 12.0, changes, 26
Performance measurements
Fault and performance, 64 Release 13.0, changes, 25

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 244 OF 247
Release 14.0, changes, 24 stopbw, 233
Release 15.0, changes, 23 synchcheck_basic.pl, 234
synchcheck_full.pl, 234
Release 16.0, changes, 23
ttStatus, 234
Release 17.0, changes, 21
Server maintenance state
Release 18.0, changes, 20 Force server into maintenance state, 86
Release 19.0, changes, 19 Recover server in maintenance state, 86
Remember password cookie for administrators, 154 Server management, 85
Modify [Link] file, 154 View current maintenance state, 86
repctl, scripts and utilities, 230 Server management, 67
resizeDSN, scripts and utilities, 231 Application, 70
Restart configuration agent, updating BroadWorks Application state, 67, 70
network configuration, 162 Check for running process, 78
Get server, 70
Restart server, server management, 80 Lock Network Server, Application Server,
Restore Execution Server, 79
Auto-backup, automatic backups, 94 Restart server, 80
Database, 100, 101 Restore server to in-service, 81
Database maintenance, 100 Server maintenance state, 85
Restore server to in-service Server state, 67
Non-redundant systems, 81 Server state monitoring tools, 83
Redundant systems, 82 Start server, 76
Server management, 81 Stop server, 77
[Link], 236 Take BroadWorks server out-of-service for
Running process, 78 maintenance, 80
Sample debug configuration file, debugging, 53 Server process problems, SNMP trap
Scheduled maintenance tasks, 90 monitoring, 209
Automatic backups, 92 Server state
Automatic disk cleanup, 96 Get, 70
CPU monitoring tool, 97 Monitoring tools
Database scheduled tasks, 98 Healthmon, 83
Database statistics, 97 Server management, 83
Database synchronization check, 97 Tech-support, 85
Healthmon keep alives, 96 Server management, 67
Registration audit, 97 Service impact, DST adjustments, 239
Service license collect, 98 Service license collect, scheduled maintenance
Tech-support, 97 tasks, 98
Scripts and utilities, 222 showrun, scripts and utilities, 231
[Link], 222
SNMP trap monitoring
bwCPUMon, 222
Application Server Execution Server process
bwListPortInUse, 222
[Link], 223 problems, 214
Application Server, Execution Server, Profile
bwshowver, 223
Server traps, 211
[Link], 223
BroadWorks system, 208
check_dbpages.pl, 224
config-hostname, 225 Common BroadWorks server traps, 209
config-redundancy, 225 Device operational state, 215
Media Server traps, 210
config-ssh, 225
Network Server traps, 211
[Link], 226
frepctl, 232 Overload control, 214
[Link], 226 Server process problems, 209
Software error traps, 210
Healthmon, 226
[Link], 227 Software error traps, SNMP trap monitoring, 210
[Link], 228 Software Manager utilities, 236
peercmd, 230 [Link], 236
peerctl, 229 [Link], 236
repctl, 230 Software version management, 87
resizeDSN, 231 Active software versions, 88
showrun, 231 Display active software versions, 87
Software Manager, utilities, 236 List installed software versions, 87
startbw, 232 Solution, DST adjustments, 239

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 245 OF 247
SSH keys, creating, 134 Database statistics, 106
SSH, using, 137 Database synchronization, Application Server
and Network Server database
SSH/RSYNC configuration
replication, 113
Configure SSH between servers, 134
Logs, 54
Create SSH keys, 134
Replication monitoring, Application Server and
Redundancy management, 134
Network Server database replication, 115
SSH, 137
Replication port number, Application Server and
Start Network Server database replication, 117
BroadWorks, 76
Tomcat logs, 54
BroadWorks application, 76
ttStatus, scripts and utilities, 234
Start and stop
Database daemon, additional TimesTen Undeployment, Application management, 73
maintenance tasks, 107 Uninstallation, Application management, 75
MariaDB database daemon, additional MariaDB Uninstalling obsolete BroadWorks versions, 156
maintenance tasks, 112 UNIX command execution logging, 53
MySQL database daemon, additional MySQL Unlock Network Server or Application Server, 80
maintenance tasks, 111
Update
Start server Application Server and Network Server address
Server management, 76 on OCS, change IP address or host name,
Start BroadWorks, 76 167
Start BroadWorks application, 76 Application Server IP address on Network
startbw, scripts and utilities, 232 Server
Statistics, TimesTen database, 106 Change IP address or host name, 164
Stop Media Server IP address on Network Server,
BroadWorks, 77 changing IP address or host name, 166
BroadWorks application, 78 Network Server IP address on Application
Server without locking, 78 Server, changing IP address or host
Stop server name, 165
Server management, 77 Network Server or Application Server address
Stop BroadWorks, 77 on Xtended Services Platform, changing IP
Stop BroadWorks application, 78 address or host name, 169
Without locking, 78 Network Server routing alias list, changing IP
address or host name, 165
stopbw, scripts and utilities, 233
TimesTen database paging information
Strategy, disaster recovery, 138 Additional TimesTen maintenance
synchcheck_basic.pl, scripts and utilities, 234 tasks, 107
synchcheck_full.pl, scripts and utilities, 234 TimesTen database statistics, additional
System monitoring, 172 TimesTen maintenance tasks, 106
Miscellaneous, 215 Xtended Services Platform IP address on
System sanity, BroadWorks system monitoring, 215 Network Server, changing IP address or
Take server out-of-service for maintenance host name, 168
Non-redundant systems, 80 Update Application Server call processing routing
Redundant systems, 81 information, 163
Server management, 80 Update BroadWorks network configuration
Tech-support Change IP address or host name, 162
Scheduled maintenance tasks, 97 HTTP Server network configuration, 162
Server state monitoring tools, 85 Restart configuration agent, 162
Thread activity, platform, 176 Update EMS redundancy configuration, changing IP
Threshold alarms address or host name, 169
Attributes, 59 Update Media Server IP address on Application
Configure through SNMP, 62 Server, changing IP address or host name, 167
Configure using CLI, 60 View
Fault and performance measurements, 59 Current maintenance state
Limitation for Application Server thresholds, 63 Server maintenance state, 86
Time zone support, BroadWorks Application Server, Current position in MIB, performance
221 measurements, 65
TimesTen MIB node values, performance
Connection optimization, additional TimesTen measurements, 65
maintenance, 110

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

©
2014 BROADSOFT, INC. PAGE 246 OF 247
Next MIB node, performance View and set CLI alarm backlog size, 57
measurements, 64 [Link] file, 154
System alarms, 56

BROADWORKS MAINTENANCE GUIDE 03-BD5005-00

Maintenance Guide
Release 21.0
Document Version 1
9737 Washingtonian Boulevard, Suite 350
Gaithersburg, MD 20878

BroadWorks® Guide
Copyright Notice
Copyright© 2014 BroadSoft, Inc.
All rights reserved.
Any technical documentation t

Document Revision History
Release
Version
Reason for Change
Date
Author
14.0
1
Updated document for re-branding.

Release
Version
Reason for Change
Date
Author
14.0
6
Added section on removing peer from
redundant system for EV

Release
Version
Reason for Change
Date
Author
14.0
14
Added a description of all the input
channels used by Broad

Release
Version
Reason for Change
Date
Author
17.0
1
Updated section 10.5.1 Change TimesTen
Database Size for EV

Release
Version
Reason for Change
Date
Author
17.0
6
Edited changes and published document.
March 8, 2011
Jessic

Release
Version
Reason for Change
Date
Author
18.0
3
Added section 12.2.3.6 Element
Management System to cover th

Release
Version
Reason for Change
Date
Author
19.0
1
Updated section 9 Scheduled Maintenance
Tasks for EV 175534.

Release
Version
Reason for Change
Date
Author
20.0
3
Edited changes and published document.
February 4, 2014
Jes

BW MaintenanceGuide R200
No ratings yet
BW MaintenanceGuide R200
261 pages
BW ASEnterpriseWebInterfaceAdminGuide R210
No ratings yet
BW ASEnterpriseWebInterfaceAdminGuide R210
350 pages
BW SoftwareManagementGuide
No ratings yet
BW SoftwareManagementGuide
225 pages
BW ASEnterpriseWebInterfaceAdminGuide R220
No ratings yet
BW ASEnterpriseWebInterfaceAdminGuide R220
421 pages
BW ASGroupWebInterfaceAdminGuidePart1 R210
No ratings yet
BW ASGroupWebInterfaceAdminGuidePart1 R210
314 pages
BW ASGroupWebInterfaceAdminGuidePart2 R230
No ratings yet
BW ASGroupWebInterfaceAdminGuidePart2 R230
519 pages
BW SystemEngineeringGuide
No ratings yet
BW SystemEngineeringGuide
95 pages
Broadworks Provisioning Guide
100% (1)
Broadworks Provisioning Guide
125 pages
BW SIPAccessSideExtensionsInterfaceSpec
No ratings yet
BW SIPAccessSideExtensionsInterfaceSpec
139 pages
Double Take5.3
No ratings yet
Double Take5.3
628 pages
Synchrony EndPoint Activator Guide
No ratings yet
Synchrony EndPoint Activator Guide
692 pages
Broadworks Deployment Studio: User Guide
No ratings yet
Broadworks Deployment Studio: User Guide
42 pages
Network UPS Tools User Manual
No ratings yet
Network UPS Tools User Manual
144 pages
BW SystemEngineeringGuide
No ratings yet
BW SystemEngineeringGuide
76 pages
Large Scale Linux on zSeries Guide
No ratings yet
Large Scale Linux on zSeries Guide
314 pages
vSphere 5.0 Upgrade Guide
No ratings yet
vSphere 5.0 Upgrade Guide
176 pages
Vsphere Esxi Vcenter Server 60 Installation Setup Guide
No ratings yet
Vsphere Esxi Vcenter Server 60 Installation Setup Guide
272 pages
Administeringad ch1
No ratings yet
Administeringad ch1
24 pages
Netiq Identity Manager: Driver For Active Directory Implementation Guide
No ratings yet
Netiq Identity Manager: Driver For Active Directory Implementation Guide
122 pages
Vsphere Esxi Vcenter Server 602 Installation Setup Guide PDF
No ratings yet
Vsphere Esxi Vcenter Server 602 Installation Setup Guide PDF
298 pages
Double-Take Availability 6.0 User Guide
No ratings yet
Double-Take Availability 6.0 User Guide
879 pages
VBrick Enterprise Media System Guide
No ratings yet
VBrick Enterprise Media System Guide
50 pages
VEMS Mystro Portal Server Admin Guide
No ratings yet
VEMS Mystro Portal Server Admin Guide
236 pages
WebSphere Process Server Runtime Guide
No ratings yet
WebSphere Process Server Runtime Guide
256 pages
Sun VirtualBox User Manual 3.0.8
No ratings yet
Sun VirtualBox User Manual 3.0.8
267 pages
Ad - Idm, Netiq, Identity Manager
No ratings yet
Ad - Idm, Netiq, Identity Manager
130 pages
WebSphere Application Server Admin Guide
No ratings yet
WebSphere Application Server Admin Guide
2,319 pages
V14.2 Vmware Installation Guide
No ratings yet
V14.2 Vmware Installation Guide
62 pages
VMW White Paper VSPHR Whats New 6 5
No ratings yet
VMW White Paper VSPHR Whats New 6 5
34 pages
Vsphere Esxi Vcenter Server 65 Upgrade Guide
No ratings yet
Vsphere Esxi Vcenter Server 65 Upgrade Guide
298 pages
Imperva DAM Upgrade Instructions
No ratings yet
Imperva DAM Upgrade Instructions
75 pages
55246AC Micrsoft SQL 2016 Always On and High Avilability Build Guide Rev 02-10-18
No ratings yet
55246AC Micrsoft SQL 2016 Always On and High Avilability Build Guide Rev 02-10-18
9 pages
Marvell Fibre Channel Cna Adapters Qlogic Series Vmware User Guide
No ratings yet
Marvell Fibre Channel Cna Adapters Qlogic Series Vmware User Guide
76 pages
02 User Directory Plugin
No ratings yet
02 User Directory Plugin
124 pages
NetBackup Troubleshoot Guide
0% (1)
NetBackup Troubleshoot Guide
608 pages
VMW White Paper VSPHR Whats New 6 5
No ratings yet
VMW White Paper VSPHR Whats New 6 5
36 pages
BASE24-eps 3.0 Installation Guide
100% (1)
BASE24-eps 3.0 Installation Guide
109 pages
NetVizura 4.4.0 User Guide
No ratings yet
NetVizura 4.4.0 User Guide
239 pages
Vsphere Update Manager
No ratings yet
Vsphere Update Manager
192 pages
NethServer Documentation
No ratings yet
NethServer Documentation
71 pages
Ibm HTTP Server - sg245132
No ratings yet
Ibm HTTP Server - sg245132
260 pages
DSfW Best Practices Guide
No ratings yet
DSfW Best Practices Guide
24 pages
FreeNAS 11.2
No ratings yet
FreeNAS 11.2
370 pages
4.0.x Inst Upgr 2
No ratings yet
4.0.x Inst Upgr 2
730 pages
OpenScape Enterprise Express V9, Staging Guide, Installation Guide, Issue 20.
100% (1)
OpenScape Enterprise Express V9, Staging Guide, Installation Guide, Issue 20.
151 pages
Integration Server Admin Guide
No ratings yet
Integration Server Admin Guide
440 pages
Firebird 5 Quick Start Guide
No ratings yet
Firebird 5 Quick Start Guide
49 pages
Coldfusion 10 Admin PDF
No ratings yet
Coldfusion 10 Admin PDF
162 pages
Firebird 3 Installation and Setup Guide
No ratings yet
Firebird 3 Installation and Setup Guide
55 pages
Exam Ref 70-247 - Configuring and Deploying A Private Cloud
100% (1)
Exam Ref 70-247 - Configuring and Deploying A Private Cloud
345 pages
HP Data Protector Integration Guide
No ratings yet
HP Data Protector Integration Guide
296 pages
BW ASMode IMSSolutionGuide R230
No ratings yet
BW ASMode IMSSolutionGuide R230
84 pages
IBM AIX Enhancements and Modernization
No ratings yet
IBM AIX Enhancements and Modernization
236 pages
FreeNAS 11.2 U3
No ratings yet
FreeNAS 11.2 U3
371 pages
WPG 6.1 Administrator Guide
No ratings yet
WPG 6.1 Administrator Guide
234 pages
M BISFileExchange Installadminguide en
No ratings yet
M BISFileExchange Installadminguide en
253 pages
Marvell Fibre Channel User Guide for VMware
No ratings yet
Marvell Fibre Channel User Guide for VMware
78 pages
BW XspCLIAdminGuide R210
No ratings yet
BW XspCLIAdminGuide R210
533 pages
BW SubscriberExportGuide R220
No ratings yet
BW SubscriberExportGuide R220
46 pages
Net-Net 4000 S-CX6.4.0 Rev1.01 HDR Resource Guide
No ratings yet
Net-Net 4000 S-CX6.4.0 Rev1.01 HDR Resource Guide
186 pages
BW ASCLIAdminGuide R210
No ratings yet
BW ASCLIAdminGuide R210
1,006 pages
Data Scientist & Software Engineer Resume
No ratings yet
Data Scientist & Software Engineer Resume
2 pages
Data Warehouse Basics for Beginners
No ratings yet
Data Warehouse Basics for Beginners
13 pages
Waterfall and RAD Models Explained
No ratings yet
Waterfall and RAD Models Explained
17 pages
OpenERP Technical Memento v0.6.4
No ratings yet
OpenERP Technical Memento v0.6.4
7 pages
Hostel Management System Project Overview
No ratings yet
Hostel Management System Project Overview
4 pages
Tourism Management System Overview
No ratings yet
Tourism Management System Overview
17 pages
Database Design for Bank Operations
No ratings yet
Database Design for Bank Operations
2 pages
Spring Boot CRUD API with MySQL Guide
No ratings yet
Spring Boot CRUD API with MySQL Guide
15 pages
Common DCL Commands Explained
No ratings yet
Common DCL Commands Explained
7 pages
Power BI Interview Questions Guide
No ratings yet
Power BI Interview Questions Guide
14 pages
SQL INNER JOIN Error Explanation
No ratings yet
SQL INNER JOIN Error Explanation
2 pages
Database System Concepts Overview
No ratings yet
Database System Concepts Overview
4 pages
PHP MySQL User Login System Guide
No ratings yet
PHP MySQL User Login System Guide
16 pages
MuleSoft Developer Expertise Summary
No ratings yet
MuleSoft Developer Expertise Summary
4 pages
USC ISD Database Standards Guide
No ratings yet
USC ISD Database Standards Guide
10 pages
Abaqus Tutorial: Consistent Units Guide
0% (1)
Abaqus Tutorial: Consistent Units Guide
15 pages
IBPS PO Exam Model Question Paper
No ratings yet
IBPS PO Exam Model Question Paper
32 pages
Azure Migration Checklist Guide
No ratings yet
Azure Migration Checklist Guide
5 pages
Database Management for ICT Technicians
No ratings yet
Database Management for ICT Technicians
3 pages
IXP300 IXP400 Installers Guide v1.00
No ratings yet
IXP300 IXP400 Installers Guide v1.00
308 pages
Zcart Database Table Structures
No ratings yet
Zcart Database Table Structures
239 pages
Hybrid ML Model for Skincare Solutions
No ratings yet
Hybrid ML Model for Skincare Solutions
37 pages
Full-Stack Fitness Tracker App
No ratings yet
Full-Stack Fitness Tracker App
8 pages
Client/Server Architecture Overview
No ratings yet
Client/Server Architecture Overview
49 pages
Understanding Datomic Architecture
No ratings yet
Understanding Datomic Architecture
5 pages
User Interface for Soil Conservation Planning
No ratings yet
User Interface for Soil Conservation Planning
69 pages
SAP Basis Administrator Overview and Practices
No ratings yet
SAP Basis Administrator Overview and Practices
10 pages
Practical Data Modeling Exercises
No ratings yet
Practical Data Modeling Exercises
9 pages
Face Detection and Recognition System
No ratings yet
Face Detection and Recognition System
37 pages
Guide Installation SSMS
No ratings yet
Guide Installation SSMS
16 pages