Network Management Cards, UPS Network Management Card 2, with Environmental Monitoring

Issue
How can I upgrade firmware on multiple APC Network Management Cards at one time or in an automated fashion?

Product Line

• Network Management Card 1 (NMC1) - AP9617, AP9618, AP9619

Devices with an embedded Network Management Card 1 include (but are not limited to): Metered/Switched Rack PDUs (AP78XX, AP79XX), Rack Automatic Transfer Switches (AP77XX, Environmental Monitoring Units (AP9320, AP9340, NetBotz 200)

• Network Management Card 2 (NMC2) - AP9630/AP9630CH, AP9631/AP9631CH, AP9635/AP9635CH

Devices with an embedded Network Management Card 2 include (but are not limited to): 2G Metered/Switched Rack PDUs (AP86XX, AP88XX, AP89XX), Certain Audio/Video Network Management Enabled products.
Environment

• All serial numbers
• All firmware versions, unless otherwise noted

Cause
A user may wish to automate the process of upgrading firmware on Network Management Cards or devices with an embedded card to save time, especially with large quantities of devices.

Resolution

Users can upgrade the firmware on individual Network Management Cards via a couple of different methods, specifically

• FTP
• SCP
• XMODEM
• InfraStruXure Manager (discontinued product)
• StruxureWare DCE/InfraStruXure Central
• Firmware Upgrade Utility (the executable is included with the firmware when downloaded)
• USB (if available)

In some cases, a user may have multiple devices that need to be upgraded. The user can upgrade each device individually. However, this may prove to be time consuming.

Therefore, the user has two options to perform a mass upgrade of firmware.

• The first, InfraStruXure Manager, is a discontinued item that monitors APC network-enabled devices. A feature of the InfraStruXure Manager is the ability to perform mass firmware upgrades. Please reference the User Guide for the InfraStruXure Manager to perform the mass firmware upgrade. InfraStruXure Central/StruxureWare Central/StruxureWare DCE is InfraStruXture Manager's replacement and offers this feature as well. 
• The second, the Firmware Upgrade Utility which is included with all currently downloaded firmware for Network Management Card 1, 2 and 3
•  devices as well as those that embed one of these model cards. You can obtain the latest firmware via the Software / Firmware Download site.
  
  Download the firmware to your local computer. An example of the firmware that you download will appear as “apc_hw02_aos374_rpdu374.exe”
  
  Once downloaded, you can run this file and it will automatically launch one of two executables, depending on which firmware you have downloaded.
  
  NOTE: Certain applications have requirements for a firmware update can be applied. Please refer to the particular application's documentation to ensure a mass upgrade can be performed on the device with no other requirements. For instance, the g2ats application has important requirements outlined in Knowledge Base article FA156178  that must be followed before a standalone or mass upgrade can be done.

Network Management Card 1 devices

Once you execute the firmware file .exe, you'll be prompted to extract the files to a chosen location or directory.

Once extracted, the executable will attempt to run the upgrade. Simply close the command prompt windows that open. Locate the directory that the extracted files were saved to. In this directory, you should see a file named “iplist.txt”. Open this file in a text editor. The contents on the iplist.txt should appear similar to:
 
;SystemIP=168.192.0.1 (the IP address of the device you wish to upgrade)
;SystemUserName=username (the username of the device you with to upgrade)
;SystemPassword=password (the password of the device you with to upgrade)
 
In order to upgrade multiple devices, you must edit this text file. You will have to create login parameters for each device you wish to upgrade. (APC recommends no more than 10 devices at one time).

NOTE: Any lines in the iplist.txt file starting with a semicolon will be ignored. 
 
An example of this file with multiple devices is below. (Actual device parameters may appear differently based on user preferences):
 
SystemIP=192.168.1.1
SystemUserName=apc
SystemPassword=apc

SystemIP=192.168.1.2
SystemUserName=apc
SystemPassword=apc

Once you have entered the login parameters for each device, make sure to save and exit the file.
Next, locate the “upgrd_util.exe” file, also located in the same directory that you extracted the firmware files to. Double-click on the “upgrd_util.exe” file and the mass upgrade will begin.

NOTE: In NMC1 Upgrade Utility v1.2 or higher, a username or password longer than 11 characters will be truncated. Thus, you will need to change the password to 11 characters or less to use upgrd_util.exe. This could be modified using a mass configuration first, outlined in Knowledge Base article FA156117.


A menu should appear with the following options:
1. Use parameters in configuration file.
2. Enter parameters for single upgrade.
3. Exit firmware upgrade tool.


Choose option 1.

The firmware upgrade will now begin. The executable will attempt to upgrade all devices listed in your IP list. If any errors are found, the upgrade will continue on to the next device.

Network Management Card 2 and 3 devices

Locate the directory that the extracted firmware files were saved to and in this directory, open the file named devices.txt manually in a text editor. The contents on the devices.txt should appear similar to:

; Firmware Update Device File for Network Management Card X Devices
; Auto-generated on: 1/31/2022 11:41:07 AM

;[Device]
;Host=0.0.0.0
;Protocol=SCP
;Port=22
;Username=apc
;Password=apc

Remove the ";" symbols from the [Device] line and below, then enter the appropriate configuration for each device that will be upgraded. For example, if two devices will be upgraded with the IP address 192.168.86.245 and 192.168.86.246, it should look like:

[Device]
Host=192.168.86.245
Protocol=SCP
Port=22
Username=apc
Password=apc
Language=

[Device]
Host=192.168.86.246
Protocol=SCP
Port=22
Username=apc
Password=apc
Language=

Once the configuration is complete, make sure to save and exit the file. (APC recommends no more than 10 devices at one time). After launching the NMC upgrade utility, the tool will automatically detect the presence of a devices.txt file, select "Start Update" to proceed:



Once the upgrade is complete, you will be required to close the executable/command prompt window.

If you have any questions on this process or experience trouble, please contact APC Technical Support. Before calling, please have the following information available:

• Firmware that you downloaded.
• Which method are you using?
• Device model and serial number that you are attempting to upgrade
• What type of error do you receive when attempting to upgrade?
• How many devices are you attempting to upgrade?

Issue

RADIUS set up with APC Network Management Card enabled products.

Product Line

• Network Management Card 1 (NMC1) - AP9617, AP9618, AP9619

Devices with an embedded Network Management Card 1 include (but are not limited to): Metered/Switched Rack PDUs (AP78XX, AP79XX), Rack ATS (AP77XX, Environmental Monitoring Units (AP9320, AP9340, Netbotz 200)

• Network Management Card 2 (NMC2) - AP9630/AP9630CH, AP9631/AP9631CH, AP9635/AP9635CH

Devices with an embedded Network Management Card 2 include (but are not limited to): 2G Metered/Switched Rack PDUs (AP84xx, AP86XX, AP88XX, AP89XX), Rack ATS (AP44xx), Certain SmartUPS online (SRT) Certain Audio/Video Network Management Enabled products.

• ​​​​​​​Network Management Card 3 (NMC3) - AP9640/AP9641/AP9643

Devices with an embedded Network Management Card 3 include (but are not limited to): Certain 2G Metered/Switched RackPDU's manufactured after 2021 (AP84xx, AP86xx, AP88xx, AP89xx), APDU9XXX RackPDU's, RackATS (AP44xxA), Certain SmartUPS online (SRT)

• Network Management Card 4 (NMC4) - AP9644

Environment
 

• All serial numbers
• Firmware versions v2.X.X and higher

Cause

Configuration can vary based on the RADIUS server being used. Below is a successful configuration taken from a FreeRADIUS server. Though this configuration worked through testing, APC by Schneider Electric cannot guarantee that this configuration will work on your RADIUS server. Configuring of your RADIUS server should be provided by your RADIUS server administrator.

Note: Further information on using the specific variety of FreeRADIUS (v2.1.10) included in Zentyal Linux 3.3 is also available in knowledge base article ID FA232648.

Resolution

The three files that need to be edited in order for FreeRADIUS to authenticate properly are:

1) clients.conf
2) dictionary
3) users file

Below are examples of entries that should be entered into each file.

1) clients.conf

client xxx.xxx.xxx.xxx (you would enter the IP address or IP address range of devices authenticating through RADIUS)
secret = radius
shortname = apc

2) dictionary

VENDOR APC 318

BEGIN-VENDOR APC

ATTRIBUTE APC-Service-Type 1 integer APC

VALUE APC-Service-Type Admin 1
VALUE APC-Service-Type Device 2
VALUE APC-Service-Type ReadOnly 3

3) users

# APC local radius authentication (working)
apcradius (username) Auth-Type := Local, User-Password == "apcradius"
     APC-Service-Type = 1

Once these entries are made, you should be able to access your APC product via RADIUS authentication.

In some cases, you do need to specify Vendor Specific Attributes. For example, APC part number AP7900 (Switched Rack PDU) uses the same login accounts as listed in the previous example. However, the AP7900 also allows specific outlets to have their own separate accounts/logins. Therefore, you must create additional entries in the event that a user has a login for specific outlets.

1) clients.conf

client xxx.xxx.xxx.xxx {IP address or IP address range of devices using RADIUS authentication}
secret = radius
shortname = ap7900

2) dictionary

VENDOR APC 318

BEGIN-VENDOR APC

ATTRIBUTE APC-Service-Type 1 integer APC
ATTRIBUTE APC-Outlets 2 string APC

VALUE APC-Service-Type Admin 1
VALUE APC-Service-Type Device 2
VALUE APC-Service-Type ReadOnly 3
VALUE APC-Service-Type Outlet 4

3) users

VSAAdmin Auth-Type = Local, Password = "admin"
     APC-Service-Type = Admin

VSADevice Auth-Type = Local, Password = "device"
     APC-Service-Type = Device

VSAReadOnly Auth-Type = Local, Password = "readonly"
     APC-Service-Type = ReadOnly

userA Auth-Type = Local, Password = "apc"
     APC-Service-Type = Outlet, APC-Outlets = "1,3,5"

userB Auth-Type = Local, Password = "apc"
     APC-Service-Type = Outlet, APC-Outlets = "1,2,3,4,5,6,7,8"

UserA will now have access to outlets 1,3,5 and userB will have access to outlets 1,2,3,4,5,6,7,8.

RADIUS and Network Port Sharing with Switched Rack PDU 2G (AP86XX, AP89XX)

Note: See the Security Handbook for APC Network Management Cards for more information on using RADIUS.

For RADIUS users file with VSAs, outlets on guest Rack PDUs can be associated to RADIUS users by using the following method.

# give user access to outlets 1, 2, and 3 on PDU 1,
# outlet 7 on PDU 2, outlets 1 through 6
# on PDU 3, and outlets 1,2,4 through 6, 7 through 10,
# and 20 on PDU 4

newOutletUser Auth-Type = Local, User-Password = "newoutlets"
APC-Service-Type = Outlet,
APC-Outlets = "1[1,2,3];2[7];3[1-6];4[1,2,4-6,7-10,20];"

If you have trouble configuring your RADIUS server, it is advised that you contact your RADIUS server vendor for assistance.

NOTE: To create a "network only" user for supported AOS files, the following line must be added to the dictionary file: VALUE APC-Service-Type NetworkOnly 6
 

Issue:
How to have configure PowerChute Network Shutdown with more that 50 computers connected to one UPS?

Product Line/s:
PowerChute Network Shutdown
AP9617, AP9618, AP9619, AP9630, AP9631, AP9635, AP9640, AP9641

Environment:
All supported OS

Cause:
Installation information

Solution:

Please review attached  App Note #101 - PowerChute Network Shutdown with more than 50 Computers connected to one UPS.pdf

Issue

Unable to connect to a Network Management Card or Network Management Card enabled device via a serial connection.


Product Line


• Web/SNMP Card - AP9606
Devices with an embedded Web/SNMP Card include (but are not limited to): Environmental Monitoring Unit 1 (AP9312TH)

• Network Management Card 1 (NMC1) - AP9617, AP9618, AP9619
Devices with an embedded Network Management Card 1 include (but are not limited to): Metered/Switched Rack PDUs (AP78XX, AP79XX), Rack Automatic Transfer Switches (AP77XX, Environmental Monitoring Units (AP9320, AP9340, Netbotz 200)

 
• Network Management Card 2 (NMC2) - AP9630/AP9631CH, AP9631/AP9631CH, AP9635/AP9635CH
Devices with an embedded Network Management Card 2 include (but are not limited to): 2G Metered/Switched Rack PDUs (AP84XX, AP86XX, AP88XX, AP89XX), Rack Automatic Transfer Switches (AP44XX), Certain Audio/Video Network Management Enabled products, Smart-UPS Online (SRT).

Environment

• All firmware versions
• All serial numbers


Cause

Several issues can cause a problem with a terminal session over a serial connection. See below for some troubleshooting tips.


Resolution

1. Check the physicals – Ensure you are using the correct cable. There are several different communication cables APC manufactures. Check your unit’s user manual for the correct part number, which should also be printed on the cable itself. If you have another cable, try it. You can also confirm the cable works if you have another unit to try it in.

2. Ensure you are using the correct COM port - Some servers have many com ports. Ensure you are using the communication port mapped to the serial port.

3. Ensure you are connecting at the correct settings - APC units can communicate at 2400, 9600, 19200, 38400, and 57600 bps. Please reference the product’s user manual for the default settings. You can also try connecting at each rate. Please keep in mind that the baud rate CAN BE CHANGED by the user, a change to the configuration file, or a managing device like ISX Manager or StruxureWare Central.
   
   APC units can be forced to a baud rate at the username and password prompt. If you are having an issue connecting, try setting the baud rate to a different value:
   
   Username: <username>
   Password: <password><space>-<baud rate>
   
   This example will set the baud rate to 2400 bps, using the default username and password:
   
   Username: apc
   Password: apc -2400
   
   The unit will inform you of the baud rate change. If it does NOT, you might have the incorrect username and password. In which case, reference knowledgebase number FA156075.
    
4. If applicable, check to ensure your USB to Serial Adapter is working properly - Many of these devices require a driver to work properly. Check to make sure this device is recognized correctly by your operating system.

Issue
My UPS Network Management Card or Network Management Card-enabled product is reporting one or more of the following alarms: “System: Warmstart,“ “System: Coldstart,“ “System: Network Interface restarted,“ “System: Network Interface Coldstarted.“

Product Lines

• Web/SNMP Card - AP9606
  Devices with an embedded Web/SNMP Card include (but are not limited to): Environmental Monitoring Unit 1 (AP9312TH)
• Network Management Card 1 (NMC1) - AP9617, AP9618, AP9619
  Devices with an embedded Network Management Card 1 include (but are not limited to): Metered/Switched Rack PDUs (AP78XX, AP79XX), Rack Automatic Transfer Switches (AP77XX, Environmental Monitoring Units (AP9320, AP9340, NetBotz 200)
• Network Management Card 2 (NMC2) - AP9630/AP9631CH, AP9631/AP9631CH, AP9635/AP9635CH
  Devices with an embedded Network Management Card 2 include (but are not limited to): 2G Metered/Switched Rack PDUs (AP84XX, AP86XX, AP88XX, AP89XX), Rack Automatic Transfer Switches (AP44XX), Certain Audio/Video Network Management Enabled products, Smart-UPS Online (SRT).
• Network Management Card 3 (NMC3) – AP9640, AP9641

Environment

• All serial numbers
• All firmware versions
   

NOTE: As of APC Operating System (AOS) v5.1.5 or higher, the Critical alerts of “System: Warmstart” or “System: Coldstart” have now been renamed to an Informational alert of “System: Network Interface restarted” or “System: Network Interface coldstarted,” respectively.

Cause
A “System Coldstart” event indicates that the NMC (Network Management Card) has just been powered and has completed startup.

• Power interruption to the NMC - If a device powering the NMC suffers an interruption of power, the NMC will restart when power is reapplied to that device.

A “System Warmstart” (or restart) event indicates that the NMC has rebooted and completed startup, without losing power. This may be due to any of the conditions mentioned below.

• The default gateway is wrong or the network traffic is too heavy and the gateway can not be reached.

The default gateway can be any valid node's IP on the Network Management Card's network. The NMC implements internal watchdog mechanisms to protect itself from becoming inaccessible over the network. For example, if the NMC does not receive any network traffic for 9.5 minutes (either direct traffic, such as SNMP, or broadcast traffic, such as an Address Resolution Protocol [ARP] request), it assumes that there is a problem with its network interface and restarts to prevent further problems. To ensure that the NMC does not restart if the network is quiet for 9.5 minutes, the NMC attempts to contact the default gateway every 4.5 minutes. If the gateway is present, it responds to the NMC, and that response restarts the 9.5-minute timer. If your application does not require or have a gateway, specify the IP address of a computer that is running on the network most of the time and is on the same subnet. The network traffic of that computer will restart the 9.5-minute timer frequently enough to prevent the NMC from restarting.  

• FTP upload completion - After a new AOS or Application firmware upgrade has been uploaded to the NMC, the NMC will automatically reboot and run the new firmware.
• Logout if settings are changed - Modification of some NMC settings will require a reboot of the NMC.
• Reset Switch - If the Reset button on the front panel of the NMC is pressed, the NMC will reboot immediately.
• Web Interface Reboot request - One of the NMC options for Reboot was selected in the Web UI.
• Network settings have changed - At least one of the TCP/IP settings changed. The system will need to reboot for the settings to take effect.
• Restart SNMP Agent (SNMP v1 or v3) - A request to restart the current SNMP agent service was received. Majority of SNMP configuration changes will require an SNMP agent service restart to apply the change.
• Load and Execute SNMP Agent Service (SNMP v1 or v3) - An internal request to load and execute a new SNMP agent service was received. Enabling or disabling SNMP v1 or v3 will require a restart to apply the change.
• Clear Network and Start SNMP Agent (SNMP v1 or v3) - A request to clear the NMC's network settings and restart the SNMP agent service was received.
• Smart-UPS Output Voltage Change - Some Smart-UPS models allow the output voltage to be changed. This requires a reboot of the NMC firmware.
• Remote Monitoring Service (RMS) communication has been lost (NMC2 only, removed in firmware v6.8.0) - If for a certain period of time, RMS communication is lost, you may see several “System: RMS failed to communicate with the server” events followed by a “Warmstart/Network Interface Restart” message in the NMC2 Event Log. This is normal and the NMC2 reboots in this situation as a failsafe mechanism in case the communication problem is internal to the NMC2. NOTE: On NMC2, it may be helpful to obtain the debug.txt or dump.txt file from the card's file system in the /dbg folder to help validate, if the files are present.
• Detected Firmware Error - An internal firmware error was detected by the NMC and to clear the error, the NMC firmware explicitly reboots itself as a failsafe. NOTE: On NMC2, it may be helpful to obtain the debug.txt or dump.txt file from the card's file system in the /dbg folder to help validate, if the files are present.
• Unrecoverable Firmware Error - An undetected firmware error occurred and the hardware watchdog reboots the NMC to clear the error. NOTE: On NMC2, it may be helpful to obtain the debug.txt or dump.txt file from the card's file system in the /dbg folder to help validate, if the files are present.

Resolution
 
It is recommended to follow the instructions in Knowledge Base article FA156131 and download all available event logs for your product. This includes event.txt, data.txt, and config.ini for NMC1 and NMC2 as well as debug.txt and dump.txt for NMC2 and NMC3.

For NMC1:

• Review the event.txt file to see if any of the causes listed above could be why your Network Management Card has restarted or coldstarted.
• Is this affecting more than one Network Management Card in your environment? This may point to a network traffic issue (too much, too little, or "bad" packets) causing the Management Card to reboot due to the watchdog mechanism outlined above.
• Note the frequency of the events in question. Can you pinpoint it to a certain time/certain set of events before and after?
  • If the warmstarts/network interface restarts are always the same amount of time apart, this may relate to a network traffic issue as well.

• Depending on what you find, try rebooting your card's interface or resetting the card to defaults (after backing up your configuration). See if the issue persists.
• Please contact APC Technical Support at any time for additional options and help to obtain a root cause for a warmstart/restart/coldstart. Also note these types of events don't always necessarily indicate a problem and only affect the Network Management Card's interface. Your UPS or device load is unaffected. NOTE: Please have your log files (event.txt, data.txt, config.ini) available for tech support to review.

 
 For NMC2/NMC3:

• Review the event.txt file to see if any of the causes listed above is why your Network Management Card has restarted or coldstarted. 
• Is this affecting more than one Network Management Card in your environment? This may point to a network traffic issue (too much, too little, or "bad" packets) causing the Management Card to reboot due to the watchdog mechanism outlined above.
• Note the frequency of the events in question. Can you pinpoint it to a certain time/certain set of events before and after?
  • If the warmstarts/network interface restarts are always the same amount of time apart, this may relate to a network traffic issue as well.
• Depending on what you find, consider rebooting your card's interface or resetting the card to defaults (only after backing up your configuration and obtaining the aforementioned log files). See if the issue persists.
• Please contact contact APC support for additional options and help to obtain a root cause for a warmstart/restart/coldstart. Also note these types of events don't always necessarily indicate a problem and only affect the Network Management Card's interface. Your UPS or device load is unaffected. NOTE: Please have your log files available for Technical Support to review - event.txt, debug.txt and dump.txt are most important as these files can easily debug the situation. These two files are not available on NMC1 products.

Issue

RADIUS and LDAP compatibility - Does my APC network enabled device support RADIUS or LDAP?



Product Line
 

• APC branded Network Access Enabled Products

Environment

• RADIUS or LDAP/LDAPS environments

Cause

This article outlines RADIUS and LDAP/LDAPS capability of APC Network Enabled devices. Refer below to review compatibility.


Resolution


Almost all APC Integrated Network Devices can utilize RADIUS (Remote Authentication Dial In User Service) as a centralized authentication mechanism. These devices include the Network Management Card (AP9617, AP9618, AP9619, AP9630, AP9631, AP9635), Environmental Management (AP9319, AP9320, AP9340), Switched/Metered Rack PDU (AP78XX/AP79XX, AP84XX, AP88XX, AP89XX, AP86XX), KVM Switches (KVMXXXX), Automatic Transfer Switch (AP77XX), and Netbotz Rack Access PX (AP9360/AP9361).

APC's RADIUS implementation follows RFC (Request For Comments) standards for RADIUS authentication. LDAP is currently used in our KVM switches (Models AP56XX, KVMXXXX). All other APC devices do not support LDAP, or any other method of authentication except for local device authentication. APC does not certify or warrant, in either express or implied fashion, third party tools and software that will allow integration with APC's implementation of RADIUS.

Some RADIUS servers allow integration with secondary authentication servers (LDAP as one possible example). Implementation examples can be referenced below. Any and all implementation is the sole responsibility of the end user.

Please refer to your software server vendor for information on integrating RADIUS into your security infrastructure.

Microsoft provides information regarding integration into Active Directory/LDAP using IAS and RADIUS in their Technet documentation

Information on other LDAP implementations can be found at

• http://www.openldap.org/

Keep in mind, by default, RADIUS users will have Read-Only access to the APC networked device unless the Service-Type attribute is changed on the RADIUS server to allow administrative access. Also, ensure that your device is upgraded to the latest firmware version available.

Please note that APC does not test and does not support nor recommend the applications listed or any comparable substitute that may be available. All documentation is provided on an "AS IS" basis as this information is the property and responsibility of the organizations which provide it. APC provides no warranties expressed or implied in any of these cases.

 

All APC by Schneider Electric Network Management Card 2 devices with AOS (APC Operating System) > version 5.1.1 or higher are Certified IPv6 Ready (Phase-2/Gold Logo) by the IPv6 forum (http://www.ipv6ready.org). The Logo ID# is 02-C-000439 and was approved on Feb 15, 2010. The Phase 2 Gold Logo verifies optimum compliance with the IPv6 protocol.

UPS Network Management Card 2 SKUs include but are not limited to: AP9630, AP9631, AP9635, AP9630CH, AP9630J, AP9631CH, AP9631J, AP9635, and AP9635CH.

Devices that have a mini embedded Network Management Card 2 or UPSs with a preinstalled Network Management Card 2 are Certified IPv6 Ready (Phase-2/Gold Logo) by the IPv6 forum (http://www.ipv6ready.org), as above. 

Devices with mini embedded Network Management Card 2 include but are not limited to: second generation APC by Schneider Electric Rack Power Distribution Units and select Audio/Video products.

You can check the model and AOS version from the interface of your Network Management Card 2 from the command line interface via telnet, SSH, and local access by issuing the about command. Via the web interface, navigate to the Administration tab->General->About to view the model and AOS version.

Additionally, certain APC by Schneider Electric KVM switches include IPv6 functionality, including AP5610, AP5615, AP5616, KVM2132P, KVM2116P, and KVM1116P.