Data Collection Procedure to Troubleshoot NetScaler Related Issues

This article contains information that you must collect for troubleshooting an issue with the NetScaler appliance.

Overview

You must collect the following information to troubleshoot any issues with the NetScaler appliance:

• NetScaler hardware model – (from FreeBSD, run sysctl -a netscaler).

• NetScaler software version including the build – (from NetScaler CLI, run show version).

• Serial number of the appliance – (from FreeBSD, run sysctl -a netscaler).

• Production setup or new installation.

• Whether an application/service that was working is now broken or the user wants to configure an application/service.

• Network topology information.

• Any change(s) performed on the NetScaler appliance prior to the issue.

• Any change(s) performed on the connected switches, upstream router, or back end server prior to the issue.

• Collect the "ns.conf" file after saving the configuration – (from NetScaler CLI, run save configuration).

  Notes:

  • If the appliance is installed with NetScaler software release 6.x or 7.x, then the configuration and license files are located in the /flash/nsconfig directory.
  • If the appliance is installed with NetScaler software release 8.x or 9.x, then the configuration file is located in the /flash/nsconfig directory and the license file(s) are located in the /flash/nsconfig/license directory.

Additional Information Required for Specific Issues

The following is a list of specific categories of issues for which you must collect additional information:

Unable to Access NetScaler for Management Purposes

High Availability (HA) Issues

Load Balancing Issues

Global Server Load Balancing (GSLB) Issues

Content Switching Issues

Caching Issues

SSL Related Issues

SSL VPN Related Issues

Web Logging Related Issues

Hardware Related Issues

Performance Related Issues

Unable to Access NetScaler for Management Purposes

Command Line Interface Access

• Network tool used for access (PuTTY, TeraTerm Pro).

• Mode of access (SSH or telnet).

• Version, if you use SSH.

GUI and Dashboard Access

• Web browser used.

• Version of Java Runtime Environment (JRE) installed.

• Operating system on the client from which you are accessing the GUI along with the service pack/patch level on the client.

High Availability (HA) Issues

Examples of such issues include:

• HA synchronization failing.

• Both appliances appear as primary or both appliances appear as secondary.

• Continuous change of state between the nodes.

• You run a command on the primary appliance which fails on the secondary appliance (command propagation failure).

The data required to troubleshoot these issues include:

• show node – (from NetScaler CLI on both nodes).

• show interface <interface ID> – (from NetScaler CLI on both nodes).

• "ns.conf" file from both the NetScaler appliances.

• Latest newnslog files from both the NetScaler appliances.

Load Balancing Issues

Examples of such issues include:

• Services flapping up and down.

• Virtual IP address flapping up and down.

• Uneven load balancing.

• Slow response when accessing applications through a virtual IP address.

The data required to troubleshoot these issues include:

• newnslog file.

• show lb vserver <vservername> for issues where virtual IP address is flapping up or down.

• show service <servicename> for issues where a service is flapping up or down.

• nstrace for issues related to packet flow and handling. Refer to Additional Resources section of this article to collect an nstrace.

• If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.

Global Server Load Balancing (GSLB) Issues

Examples of such issues include:

• Network traffic does not reach the intended site.

• Metric Exchange Protocol (MEP) not being formed.

• Remote site not coming up.

• Proximity not working.

The data required to troubleshoot these issues include:

• The "ns.conf" file from all the sites participating in GSLB along with basic information listed in the Overview section for all the sites.

• show ns license output from the NetScaler CLI to verify if you have purchased proximity based GSLB license.

• Latest newnslog files from all the NetScaler appliances.

• show gslb site output from each NetScaler appliance participating in GSLB.

• show gslb runningconfig output from each NetScaler appliance participating in GSLB.

Content Switching Issues

Examples of such issues include:

• Network traffic does not reach the intended Load Balancing virtual server (under Content Switching virtual server).

• Policy does not match properly.

• No content served.

The data required to troubleshoot these issues include:

• The newnslog file.

• If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.

Caching Issues

Examples of such issues include:

• Required content not cached.

• Expired content served.

• Cache expiry causing network traffic surge to the back end.

The data required to troubleshoot these issues include:

• The newnslog file.

• If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.

• stat cache output from the NetScaler CLI.

• date (from FreeBSD shell) output for the NetScaler appliance and back end server.

SSL Related Issues

Examples of such issues include:

• Access to SSL virtual server failing.

• Users receiving certificate related warnings when accessing HTTPS site.

• Unable to bind certificate to SSL virtual server.

The data required to troubleshoot these issues include:

• The newnslog file.

• A copy of the certificate if the issue is related to SSL certificate installation/binding.

• An nstrace for problems related to packet flow and handling. Refer to Additional Resources section of this article to collect an nstrace.

• If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.

• show ssl stats output from the NetScaler CLI.

SSL VPN Related Issues

Examples of such issues include:

• Unable to access the server through the SSL VPN.

• FTP or data transfer through the SSL VPN fails.

• SSL VPN authentication related issues.

The data required to troubleshoot these issues include:

• The "sslvpn.txt" file (for Windows platform) or "mpSSLVpn" and "Netscaler.log" files (for MAC platform) from the client computer if issues are related to the SSL VPN.

• An nstrace for problems related to packet flow and handling. Refer to Additional Resources section of this article to collect an nstrace.

• If the problem site is publicly accessible, then the URL for the site and any logon credentials required for access.

Web Logging Related Issues

Examples of such issues include:

• Web logging does not occur.

• Web logging stops intermittently.

• Required details are not logged.

• Irrelevant IP addresses or domains are logged.

The data required to troubleshoot these issues include:

• Details of the server on which the Web Logging service is installed and the version of the Web Logging client installed.

   >nswl –version   Version: Netscaler Weblogging Application(nswl) NS9.2: Build 47.11,     Date: Aug 10 2010, 21:55:57 (release) [Linux]   Done !!

• The "log.conf" file and debug (file name: nswl.log-ddmmyyyyHHMM) file if the issue is related to Web Logging. These files reside on the Web Logging server in the /etc and /bin folders respectively (if Web Logging is installed as a Windows service, debug files are created in the \WINDOWS\system32 folder).

File name	Remark	Location in NetScaler Software Release 8.x/9.x/10.x
ns.conf	configuration file	/flash/nsconfig
ns.conf.x	older configuration file	/flash/nsconfig
newnslog	main log file (data format)	/var/nslog
newnslog.xx.gz	archived newnslog file	/var/nslog
ns.lic	license file	/flash/nsconfig/license
nstrace.sh	script to collect nstrace	/netscaler
nstcpdump.sh	script to collect tcpdump	/netscaler
nstrace.x	packet trace	/var/nstrace
vmcore.x.gz	core dump during a crash	/var/crash
kernel.x	kernel dump during a crash	/var/crash
process-pid	user process core file	/var/core
savecore.log	core dump log file	/tmp
pitboss.debug	open pipe for debug info	/tmp
aaad.debug	open pipe for debug info	/tmp
ns.log	system syslog file	/var/log
messages	all logged entries	/var/log
auth.log	authentication/authorization	/var/log
dmesg.*	hardware error/boot sequence	/var/nslog

Additional Resources

How to Obtain Performance Statistics and Event Logs from NetScaler Appliance

To record a network packet trace on a NetScaler appliance, complete the following procedure:

1. Run the following command from the shell prompt of the appliance:
   /netscaler/nstrace.sh -sz 0
   Note: The trace file is stored in the /var/nstrace directory with the name nstrace.x, where "x" is a number.

2. Reproduce the problem scenario.

3. Press CTRL+C to end the trace.

You must collect the trace file along with details such as source IP address, destination IP address, and virtual IP address.
Network trace collected can be decrypted using Wireshark tool, Refer to the following links ??
How to Decrypt SSL and TLS Traffic Using Wireshark
Filter Expressions for Wireshark When Using NetScaler Appliance
How to Extract an SSL Certificate from a Network Packet Trace File in Wireshark
NetScaler Packet Trace Format in NetScaler 11.0 Release