NEC Express servers Management tools
[Link] Senior Manager Pre-sales Server & Storage solutions
NEC Corporation 2009
NEC Confidential
NEC Management framework
MasterScope is a series of Integrated System Management software products which simplify day-to-day operations and control whole IT infrastucture
Corporate Management
Unified Management
IT Service Level Management Invariant Analyzer
MISSION CRITICAL OPERATIONS
Operations Management
Job Management JobCenter Software Deployment Deployment Manager Platform Optimization SigmaSystemCenter Backup Networker/ NetVault/ NetBackUp
System Management
System Management System Manager Operations Navigator Network Management Network Manager Storage Management StorageManager App Management Application Navigator
Server & Storage Management (Hardware)
ESMPRO Manager Storage Manager
NEC Confidential
NEC Management framework
Target layer Monitoring points
Monitor performance information of middleware soft (DB, AP server, Web server etc). Monitor service level of certain protocols Monitor logs from OS and AP and detect abnormal situation Check the running status by monitoring up down status of Windows service and process Monitor threshold of OS performance (remaining CPU, memory, disk etc). Server alive monitoring by ICPM and SNMP trap monitoring Detects HW failure HDD, fan,.), component status change Remote Control Storage mgmt tool for running status and config mgmt. Network mgmt MasterScope Network Manager Server mgmt MasterScope SystemManager
Product lineup
AP performance monitoring
Application mgmt MasterScope Application Navigator
Log monitoring Service, Process monitoring Resource monitoring
Integrated mgmt MasterScope MCO
NW monitoring
Today Focus HW monitoring Storage monitoring
Server HW mgmt ESMPRO Manager
Storage mgmt MasterScope StorageManager
NEC Confidential
ESMPRO Manager version 5
NEC Confidential
ESMPRO5 Server Management Overview
At a glance
Management solution for NEC Express5800 servers Bundled with Express5800 servers (Express Builder CD-ROM) Centralized and distributed management of NEC Express servers
Management Features
Real-time monitoring Hardware components status
Software information (Firmware/OS)
Alert management (Hardware failure/status change) Performance/usage monitoring Power control & monitoring Remote control operations (Power, Logs, Batch, ..) Integration with other frameworks (HP Open-View, Microsoft MOM, Nagios ..)
5
NEC Confidential
Server Infrastucture monitoring
WEB-based Manager
ESMPRO Manager
Internet Intranet
NEC Express5800 Servers
ESMPRO agents (SNMP)
NEC Confidential
Server hardware monitoring
An alert is sent to the manager
ESMPRO agents (SNMP)
Internet Intranet
ESMPRO Manager
Notification of any hardware failure & component status change
!
Hardware failure Component status change
Power supply, Fans failure Disk failure Network failure
Running Warning Error
NEC Confidential
Event/failure notification & escalation
!
e-mail with source of error(s)
Internet Intranet
Server in error
ESMPRO Manager
Send an e-mail to the administrator
Running
Warning Error Notification of hardware failure & component status change
NEC Confidential
Discovering Server Infrastucture
Discovery
ESMPRO agents (SNMP)
Detection & organization
ESMPRO Manager
Internet Intranet
NEC Express5800 Servers
NEC Confidential
Server detection and registration in the organization
Manual or automatic detection
Choose the method to detect the servers : IP range or network mask
10
NEC Confidential
Server organization (Root = Infrastucture)
Server tree organization
Servers are organized in groups which reflects the organization
Add group(s) Detect server(s) inside a given group
11
NEC Confidential
Monitoring server health status
NEC Express5800 Servers
ESMPRO agents (SNMP)
Internet Intranet
ESMPRO Manager
Hardware components health status
Power supply, Fans Disk Network Mother boards 2+2 Power Configuration (Hot Plug)
12
NEC Confidential
Likely to be the most important to monitor
NEC Express5800 Servers
ESMPRO Manager
Internet Intranet
More prone to failures .
Power supply Hard disk drive
Much less problems with
Mother board PCI boards
Power supply, Fans Disk Network Mother boards 2+2 Power Configuration (Hot Plug)
Very critical component
Memory Dimms But highly reliable ECC + optional redundancy on top
NEC Confidential
13
Server health status
Running Warning Error
All server components
Main server information
14
NEC Confidential
Deeper dive in
Detailled view of server components Expand the tree
15
NEC Confidential
Alert viewer
Alerts/Events from the monitored servers
Server(s) name and IP adress of the monitored server(s) 16
NEC Confidential
Alert detail (SNMP trap)
An e-mail is sent Take the corrective action
Trap error: One of power supply has failed
17
NEC Confidential
Alert forwarding to an SMTP mailer
Adress recipient
Alerts/Events forwarding to SMTP e-mail
SMTP mailer
NEC Confidential
18
Remote Control operations
Remote control operations
Power control Batch Server update (Bios/Firmware) Running time .
Test
Internet Intranet
Functionnal server groups
ESMPRO Manager
Production Development
Can be executed at server group or individual server level
19
NEC Confidential
Remote control operations
Remote operations
20
NEC Confidential
Power measurements at server level
Power
Power consumption for a given server
Measurement period
21
NEC Confidential
Power measurements at server group level
Server Power consumption
Power consumption for a group of servers
Cumulated power consomption of all servers in a group
22
NEC Confidential
Reduce Power consumption (ECO)
Reduce/adjust Powerconsumption at server level
Different levels Power reduction by lowering CPU frequency
23
NEC Confidential
Reduce Power consumption (ECO)
Choose the group Choose the server(s) inside the group
24
NEC Confidential
Remote Power Control at server/group level
Choose the group Choose the server(s) inside the group
Power control OS shutdown Reset operations for a given server or a group of servers
25
NEC Confidential
Remote batch at server/group level
Choose the batch to execute and scedule it
26
NEC Confidential
Manage the software package version
Bios 1.21 Bios 1.27
ESMPRO Manager
Internet Intranet
Firm xxx Firm yyy Package 1
In typical organisation
Difficulties to maintain software package homogeneity Most of the time software packages repository are not updated regularly Or merely do not exist Updates are manual
Package 2 NEC Express5800 Servers
Software control and versionning is non accurate
Requirement for a fully automated solution
NEC Confidential
27
Software update with NEC Express update
Central Repository (NEC Web site)
ESMPRO Manager
Internet Intranet
Download update modules automatically from a unique central repository (NEC)
BIOS, Firmware, .
NEC Express5800 Servers
Notify the servers on which updates are required
Specify the update required state with special icon
Instruct Starting Remote Update
Easy operation from ESMPRO console Multiple update modules to multiple servers
Remotely confirm update completion
Display the completion result with special icon
28
NEC Confidential
NEC Express Update
Download software packages from this NEC web site
29
NEC Confidential
NEC Express Update (package dowload)
Select the software package(s) to install on a given server or a group of server
30
NEC Confidential
NEC Express Update (package dowload)
Current software package on a given server
31
NEC Confidential
In-Band server management
In-band management ESMPRO agent (SNMP)
Lan Redundant Lan (teaming) Lan ports Internet Intranet
ESMPRO Manager
In case of network failure Transparent failover NO communication loss
Network connection down notification 32
NEC Confidential
Take the corrective action
Out-of-Band server management
In-band management ESMPRO agent (SNMP)
Lan Redundant Lan (teaming) Lan ports Internet Intranet
ESMPRO Manager
Dedicated Lan
Lan management port
Complete network failure
Need another way for getting access
33
Out-of-Band management
Login to NEC Express Scope engine
NEC Confidential
Out-of-Band server management
Login to NEC ExpressScope engine
NEC Confidential
Out-of-band management with NEC ExpressScope
Expressscope V2:
BMC designed by NEC and compliant with IPMI 2.0 standard. Out-of-band management mode. Dedicated management port with web server. Real time sensor reading SEL log can be saved into a file. Remote Power Control. Server monitoring (TC, Voltage, Fan speed, Chassis Intrusion,
Consumption) PSU Module, HDD Status, Power
Remote media : including write access on USB key. Remote KVM as standard : faster, ready to use. BMC network configuration by SETUP Alerts by SNMP traps or SMTP mail. CLI using TELNET / SSH with extended commands.
35
NEC Confidential
Expressscope V2
PCIe(x8) PCIe(x8)
Low profile riser
Intel Xeon Processor
Intel Xeon Processor
CPU 2 LGA 771 sockets
8.5-10.6GB/s 1066/1333MHz Front Side Bus
PCIe(x8) 4GB/s PCIe(x8) 4GB/s
5.3GB/s 5.3GB/s 5.3GB/s
Intel 5000P (Blackford)
5.3GB/s
Memory
PCIe(x8) 4GB/s
x4 ESI
MAC 2ch
12x DDR2-667 FBD (Up to 48GB) ECC, Chipkill, Spare, Mirror 2-4Way Interleaved
GbE PHY(2ch)
(82563EB)
2x LAN (for General Use) I/OAT (Standard)
Full height riser
PCIe(x8) 4GB/s Intel 6321 (ESB2E) PCI-X133 1.06GB/s
PCIe(x8)
1x ATA100 6x USB2.0 (2x Front, 2x Rear, 2x Internal*) 5x SATA(1.5G or 3G bps) (No Repeater)
4x SATA for for HDD & 1x SATA for tape drive LPC
PCIe(x1) 2x USB
TPM1.2
64bit/133MHz PCI-X (3.3V)
BIOS Flash
BMC
1x IPMB 1x VGA 2x PS/2 2x Serial Ports
(1x Rear, 1x internal)
Video
ServerEngines Server Management Controller2 (Pilot2) MAC 1ch
DRAM for BMC & VRAM SPI Flash RMII
for BMC 1x LAN (for Management)
10/100 PHY(1ch)
36
NEC Confidential
Ethernet Management Port
Dedicated management port
IP address policy:
1. 2. 3. Default IP address is [Link] (DHCP disabled) Can be forced to IP address Can be changed to DHCP
Web server policy:
1. 2. 3. Disabled by default Can be enabled for HTTP Can be enabled for HTTPS
37
NEC Confidential
Pre requisites / Log In
OS and Browser: XP SP2 with IE6/7 or Firefox 2.0 Vista with IE7 or Firefox 2.0 RedHat 4 with Firefox 2.0 JRE: 1.6 from SUN Microsystems or +
Connection URL: [Link] http: better performance https: better security
Default account (must be changed for security):
Log in : administrator Password: <none>
38
NEC Confidential
Main Screen: the Control Panel Function
39
NEC Confidential
Power Control Menu
These power ON/OFF controls are hardware commands send to BMC
40
NEC Confidential
IPMI
MC Information => BMC Firmware version Backup IPMI => [Link] file System Environment => Real time sensor reading (T, V, speed, PSU, Watts)
41
NEC Confidential
Sensor Real Time reading
42
NEC Confidential
Remote Devices
43
NEC Confidential
Remote Devices
Licence
Licence is always bundled with server but still has to be entered into BMC (log out/log in required) Be careful on similar character of license (1 and l, )
44
NEC Confidential
Remote Devices:
Remote KVM 1/2
Remote KVM allows to redirect video, keyboard and mouse. Mouse cursor synchronization is auto under Windows (see User Guide for maximum resolution)
45
NEC Confidential
Remote Devices:
Remote KVM 2/2
In case of key map problem, a virtual keyboard can be used (EN/FR)
46
NEC Confidential
Remote Devices:
Remote Media
Following device/image can be used for remote media: Floppy drive CD/DVD drive USB Key (read/write) ISO image of Floppy/CD/DVD Device is detected as USB device on server => Refer to User Guide for limitation on Remote Media On 100MB LAN, Remote Media Transfert is = to 1.6x CDROM speed
47
NEC Confidential
Configuration (Network/Web/Alerts/CLI)
Network=> Web Server=> Alert=> Miscellaneous=>
LAN settings security settings of Web SMTP alerts telnet/SSH settings
48
NEC Confidential
Configuration (Users)
-Maximum of three users connected simultaneously
- Three users levels: Administrator: view IPMI info, view/change settings, access to remote KVM/Media Operator: view IPMI info, view settings only and no remote KVM/Media User: view IPMI info only
49
NEC Confidential
CLI by Telnet/SSH
Once CLi enabled, NEC BMC specific commands can be used to control server remotely (See User Guide for details)
50
NEC Confidential
Integration with other Management framework
HP-Open View (OVO & NNM) Microsoft MOM Nagios
NEC Confidential
Integration with Nagios
Windows computer with ESMPRO Manager
Nagios management server Net-SNMP(snmptrapd) SNMPTT
Nagios
redirect trap to Nagios
Output the result.
Nagios periodically check the command file.
trap
Nagios external command file
NEC Express with ESMPRO Agent
NEC Corporation 2009 NEC Confidential
Requirements for Nagios integration
Linux, RHEL Server release 5.2 for this document Nagios, version 3.0.6 for this document
Net-S NMP, providing trap capture tool snmptrapd. Tested version: [Link]
Associated Perl package Net-SNMP. SNMP trap translator (aka SNMPTT), that translates SNMP traps to human readable format, and sends the result to Nagios. Tested version: 1.2 NEC ESMPRO Manager, that receives SNMP traps from ESMPRO Agent, then redirects them to the Nagios management server.
NEC Corporation 2009
NEC Confidential
Microsoft Operation Manager 2007
Select the management pack to import
NEC Corporation 2009
NEC Confidential
Microsoft Operation Manager 2007
NEC Confidential
Microsoft Operation Manager 2007
NEC Confidential
Define policy to receive ESMPRO traps in OVO
Select Action > New > SNMP Interceptor menu from OVO.
Press New to define rule. (see next page)
NEC Corporation 2007
NEC Confidential
Define rule Condition in OVO
Enterprise ID : .[Link].[Link].[Link].2 Generic ID : (6) EnterpriseSpecific Specific ID : 1 - 3
NEC Corporation 2007 NEC Confidential
ESMPRO has 3 kind of traps. You must define policy rule respectively.
Define rule Actions Message in OVO
Severity : select Normal for Specific ID 1 select Minor for Specific ID 2 select Major for Specific ID 3 Message Text : Use $ notation to refer to trap variables.
NEC Corporation 2007 NEC Confidential
Press OK until you see the policy editor main window.(next page)
Define rule respectively
NEC Corporation 2007
NEC Confidential
ESMPRO traps in the OVO active messages window
File system alert detected by ESMPRO
NEC Corporation 2007
NEC Confidential
Server Management Software Comparison
In-Band management
NEC ESMPRO Ver.5
Remote Monitoring (HW State, Log, etc.) Remote Power Control Alert Monitoring Retrieve H/W Information Configuration and OS Yes Yes Yes Yes Comparison = = = =
HP
System Insight Manager
Yes Yes Yes Yes
Command Line I/F, Script I/F
Security (SSL, SSH) Web Server Task Scheduling Scheduled Power Control w/o UPS Automated BIOS/BMC Update, Version Control Integrated RAID Management Virtual LCD Power Consumption Monitoring Power Capping Control Directory Service (LDAP) VM Management Page 62
Yes
Yes Yes Yes Yes Yes (ExpressUpdate) Yes Yes Yes Yes No No (Support with SSC)
=
= = = > = > > > > < <
Yes
Yes Yes Yes No Yes Yes (monitor only) No Yes (Option License) Yes (Option License) Yes Yes (Option License)
NEC Corporation 2010
NEC Confidential
Management Controller Comparison
Out-of-Band management
NEC
EXPRESSSCOPE Engine2
Comparison
HP
iLO2
Remote Monitoring (HW State, Log, etc.)
Remote Power Control Send Alert (SNMP, E-Mail) Retrieve H/W Configuration Monitor HDD failure (agent-less) Remote KVM Remote Media
Yes
Yes Yes Yes Yes Yes (java-base) Yes
=
= > = > < =
Yes
Yes Yes (SNMP only) Yes No Yes (java-base, ActiveX) Yes
Security (SSL, SSH)
Web Server Command Line I/F (SMASH CLP) Remote F/W update Virtual LCD Power Consumption Monitoring Power Capping Control
Yes
Yes Yes Yes (require tftp server) Yes Yes (current value only) Yes
=
= = < > < =
Yes
Yes Yes Yes No Yes Yes
Directory Service (LDAP)
Page 63
No
<
Yes
NEC Corporation 2010
NEC Confidential
ESMPRO operation with CLI
ESMPRO operations can be also done through command line
Power ON
Power ON Power ON
C:\>dscli groupPowerOn GroupName
(Power ON every servers belonging to the same group)
CLI examples
Power Control (Group DC-ON) - dscli groupPowerOn GroupName - dscli powerOff ServerName Status Check (Status of server status LED) - dscli getStatusLamp ServerName [sample results] GREEN_ON (LED is green) AMBER_BLINK (Amber blinking)
64
NEC Corporation 2009
NEC Confidential
Demo for La Caixa De Depositos
ESMPRO Manager 5.x
Windows 2008 Lan Management station
Internet
Paris
Universal Raid Utility
Barcelona
RDP session on management station running ESMPRO Manager
EMPRO agent 4.52
Windows 2008R2 Enterprise
Teaming 192.168.2..202
Lan ports Management port
[Link]
Express R120a-1 with LSI Raid 65
NEC Corporation 2009 NEC Confidential