Problem Definition
The customer reported that when monitoring the performance of a Web application by using the Topaz Application Performance Monitoring (APM) client. At times, the client observed that the average response time ranged between two to three seconds. However, the expected average response time was half a second.
Environment
The customer was using the following hardware and software components for the network topology:
??鷉G NetScaler MPX 17000 series appliances
??鷉G NetScaler software release 8.0
??鷉G The Topaz APM client installed on the network
??鷉G An appliance on the server side that reorders the packets
Troubleshooting Methodology
The Engineers requested the customer to send the latest newnslog, trace, and configuration files from the appliance when the issue was noticed. Additionally, they simulated the client environment in the laboratory and continued with the analysis of the issue.
Initially, the Engineers suspected the issue to be a result of Surge Queue (SQ) or the HTTP DoS protection feature on the NetScaler appliance. However, analyzing the trace files from the server and client sides, and HTTP error counters helped the Engineers to find the cause of the issue.
Cause
On analyzing the newnslog and configuration files, the Engineers noticed that the size of the HTTP request was approximately 2 KB. Additionally, the Path Maximum Transmission Unit Discovery (PMTUD) feature was disabled on the client. As a result the client request was sent as three segmented packets of 516 bytes.
When the NetScaler appliance receives the request in multiple packets, the appliance accumulates the HTTP header, parses it, and then sends all the packets almost at the same time, indicating a burst. The appliance at the server side reorders the packets. When the server receives the request, it is reordered, which causes a delay for the service to respond with a delay of approximately 500 ms.
If you configure a NetScaler appliance for TCP, the issue is not noticed.
Resolution
After troubleshooting the issue, the Escalation Engineers recommended that the customer implement any of the following resolutions to address the issue:
??鷉G Prevent the server-side reordering of packets. This is an ideal solution. However, there could be some design constraints based on setup of the customer network.
??鷉G Enable PMTUD on the client. This enables the client to send the request with two packets. As a result, there is no reordering of the packets and there is no delay in the response time.