NetStorm Troubleshoot

Topic Covered: Script Manager | NetStorm Core | Jenkins

Unable to open Script Manager

Possible Reasons #1Due to Proxy settings.
Screen7
Steps to DiagnoseCheck the proxy settings of the system and make it sure it is not on or off according to the requirement of client network connection.
Commands to validate / SolutionDisable/Enable the proxy setting of the system according to the requirement.

Possible Reasons #2Soft link of the library not found.
Screen9
Steps to DiagnoseCheck under the

$NS_WDIR/webapps/netstorm/WEB-INF/lib

that all softlink of the libraries should be present.

Commands to validate / SolutionMake the softlink of the library to respective path as per other libraries and restart the tomcat.

Possible Reasons #3Due to old cache present in Java settings.
Steps to Diagnose
Commands to validate / SolutionGo to java settings of the system and clear all the caches.

Possible Reasons #4Jar files missing on the server.
Steps to DiagnoseNeed to check no one has tempered the lib directory under

$NS_WDIR/webapps/netstorm/WEB-INF.

To get $NS_WDIR, type command : echo $NS_WDIR in CLI.

Commands to validate / SolutionWe can re-upgrade the controller with same build.

Possible Reasons #5Java content was not enabled in browser in java console.
Screen8
Steps to DiagnoseIf we are getting following exception:

BadFieldException[ Cache must be enabled for nativelib or installer-desc support,https://10.33.59.176:443/netstorm/lib/jacob_dll.jar]

Commands to validate / SolutionOpen Java Settings, click the Settings button under “Temporary Internet Files” on the General tab.
Make sure the check box “Keep temporary files on my computer” is selected.

Possible Reasons #6Disk space is full on the server.

Screen10

Steps to DiagnoseCheck the exceptions and disk space of the server.
Commands to validate / SolutionCreate some space on the server.

Possible Reasons #7Memory not available in the server.
Steps to DiagnoseCheck the exceptions at catalina.out file of tomcat and available memory of the server.
Commands to validate / SolutionNeed to stop the unwanted process of the server or need to reconfigure the heap memory allocated in site.env.

Possible Reasons #8Old java version present on the system.
Steps to DiagnoseNeed to check the Error logs for compatibility.
Commands to validate / SolutionNeed to update the java version to the latest.

Possible Reasons #9appBase not defined properly in server.xml.
Screen11
Steps to DiagnoseNeed to check the configuration of the tomcat in server.xml
Commands to validate / SolutionCorrect the configuration in sever.xml of the tomcat.


Unable to record script

Possible Reasons #1Host name is not reachable.

Screen12

Steps to DiagnoseWget or telnet the remote server to which we need to make connection.
Commands to validate / SolutionNeed to check the connectivity between the netstorm and website and make it available.

Possible Reasons #2Proxy of the browser is not set.

Screen7

Steps to DiagnoseCheck the proxy settings under browser settings in script manage.
Commands to validate / SolutionThere is a chance that we may require proxy settings in order to record script from client server. Make the correct entry of proxy in script browser settings.

Possible Reasons #3The default script to which the script manager tries to open is not in correct format.
Steps to DiagnoseUsing CLI, need to check the flow of the script.
Commands to validate / SolutionMake the required changes in the flow of the script.


Unable to view script

Possible Reasons #1User is not having the access to view script.
Steps to DiagnoseCheck the accessibility of the user under admin tab in left panel in NetStorm home page:

Admin –> Access Control

Commands to validate / SolutionProvide access to the user for write permission.


Unable to compile script

Possible Reasons #1Unknown page name error due to not defined in flow.
Steps to DiagnoseCheck the script’s registration.spec file.
Commands to validate / SolutionNeed to define the page name in flow.

Possible Reasons #2Netbits missing in /home/cavisson/etc/cav.conf

Screen14

Steps to DiagnoseCheck cav.conf file for netbits.
Commands to validate / SolutionNeed make entry of netbits in cav.conf

Possible Reasons #3Package missing in script.
Steps to DiagnoseCheck the script flow file for imports of packages.
Commands to validate / SolutionNeed to import the respective package in script.


Getting error while testing services through script

Possible Reasons #1Header passed in the script is in wrong format.
Steps to DiagnoseNeed to check the script.
Commands to validate / SolutionNeed to pass the header in correct format in script.


Request not passing in encoded format

Possible Reasons #1NS APIs have not been used to encrypt request.
Steps to DiagnoseVerify the script in script manager and look for NS APIs have been used or not in flow file.
Commands to validate / SolutionIf not used, ‘ns_encode_specific_eval_string()’ can be used for the same to get expected result.


Unable to start test. Core is formed

Possible Reasons #1Buffer size allocated to variable is small compared to size it needs to handle.
Steps to DiagnoseNeed to check the script and validate the variables where size is allocated.

For core check at /home/cavisson/core_files path

Commands to validate / SolutionIncrease the buffer size of the variable declared in script.


Request failure in script body

Possible Reasons #1
Size of request is very large.
Screen6
Steps to DiagnoseCheck the size of the request passed in script.
Commands to validate / SolutionPlease keep the body in JSON file and add the file in script using CAVINCLUDE.


Unable to create scenario

Possible Reasons #1User is not having the access to create scenario.
Steps to DiagnoseCheck the accessibility of the user under admin tab in left panel :

Admin –> Access Control

Commands to validate / SolutionGive access to the user for write permission.


Unable to start test

Possible Reasons #1License error.
Screen13
Steps to DiagnoseError will show, license is not valid.
Commands to validate / SolutionNeed to contact the Cavisson team.

Possible Reasons #2Script issue.
Steps to DiagnoseNeed to compile and check if the script is correct.
Commands to validate / SolutionCorrect the script from script manager and check by compiling the script.

Possible Reasons #3
Steps to DiagnoseData file contains no data.
Commands to validate / SolutionCorrect the script from script manager and check by compiling the script.

Possible Reasons #4Hostname error
Screen11
Steps to DiagnoseNeed to check wether nameserver entry is not at /var/run/dnsmasq/resolve.conf
Commands to validate / SolutionNeed to make nameserver entry at resolv.conf file.

Possible Reasons #5Stack size issue
Steps to DiagnoseCheck if core is forming at /home/cavisson/core_files path and stack size defined in scenario is enough.
Commands to validate / SolutionIncrease the stack size in scenario


Test is taking time to start

Possible Reasons #1Debug mode is ON in scenario
Screen15
Steps to DiagnoseCheck if the debug is ON in scenario.
Commands to validate / SolutionDisable the debug mode using scenario GUI.


Unable to see page dump

Possible Reasons #1Logging is not enabled in scenario.
Screen16
Steps to DiagnoseCheck the scenario
Commands to validate / SolutionNeed to enable the logging in scenario.

Possible Reasons #2PostgreSQL is not running
Screen18
Steps to DiagnoseRun the postgresql and check the status of postgresql. Ex – psql test cavisson
Commands to validate / SolutionStart the postgresql in case it is not running.

Possible Reasons #3pg_bulkload is not installed properly.
Steps to DiagnoseCheck for lilbraries required by pg_bulkload like bison-3.0, flex-2.5.37 and pam_1.1.3
Commands to validate / SolutionNeed to reinstall pg_bulkload. Contact Cavisson team, if further help needed.


Getting SSL handshake failure in test

Possible Reasons #1Correct cipher key is not used.
Screen17
Steps to DiagnoseCheck the scenario configuration.
Commands to validate / SolutionCheck the cipher key used by the server using openssl command and deploy the same in scenario.

Command to check cipher key on server :

openssl s_client -connect server_ip:https_port

Ex – openssl s_client -connect 102.165.33.203:443

Possible Reasons #2Wrong server name used in scenario.
Steps to DiagnoseCheck the scenario configuration.
Commands to validate / SolutionCorrect the host server name used in scenario and rerun the test.

Possible Reasons #3TLS version mismatch between netstorm and host server.
Screen19
Steps to DiagnoseCheck the TLS version of host version and netstorm server.
Commands to validate / SolutionNeed to upgrade or downgrade the TLS version in netstorm server according to host server.


Unable to hit all transactions

Possible Reasons #1There is a chance of page failures.
Steps to DiagnoseCheck failures in http failures under Test metrics.
Commands to validate / SolutionEnable the Continue session on page failure in scenario.


Unable to hit https transactions

Possible Reasons #1Due to Accept-Encoding Header, which the server is not accepting.
Steps to DiagnoseValidate it using Postman request.
Commands to validate / SolutionDisable the Accept-Encoding Header in request.


Unable to route to host servers defined in scenario for more than 4

Possible Reasons #1Unable to make connection with the server
Steps to DiagnoseSERVER_SELECT_MODE is set to 1 in scenario by default. Which limits the use of host servers identical to first server host used.
Commands to validate / SolutionMake the SERVER_SELECT_MODE to 0.


Unable to achieve TPS

Possible Reasons #1G_TRACING is enabled for all logs. Screen16
Steps to DiagnoseCheck the scenario configuration for logging reports.
Commands to validate / SolutionEnable G_TRACING for failed session only.


Possible Reasons #2Amount of http Failures are more.
Screen5
Steps to DiagnoseCheck the http error graph of the test run.
Commands to validate / SolutionDecrease the timeout time in scenario.


User dropped in test

Possible Reasons #1Progress interval is less compared to total number of CPU core used in scenario.
Steps to DiagnoseNeed to check the scenario .
Commands to validate / SolutionIncrease the Progress interval and decrease the number of CPU used.


Test not stopped after defined duration

Possible Reasons #1Test is in paused state
Steps to DiagnoseCheck if someone has paused the test.
Commands to validate / SolutionResume the test and it will automatically stop.

Possible Reasons #2Loop used in scenario, based on random number generated.
Steps to DiagnoseCheck the script for any loop used, which makes the user go on forever execution of transactions.
Commands to validate / SolutionCorrect the script and limit the random to specific value.

Possible Reasons #3Running test with only CPU thread and multiple groups.
Steps to DiagnoseCheck the scenario configuration
Commands to validate / SolutionIncrease the number of CPU threads in scenario or set G_RAMP_DOWN_METHOD ALL 2 0.


Unable to open Jenkins UI

Possible Reasons #1Jenkins is not running on server.
Steps to DiagnoseRun the command : /etc/init.d/jenkins status. It will show status of jenkins whether it is running or not.
Commands to validate / SolutionStart the jenkins process and check Jenkins UI.


Unable to install Jenkins HPI

Possible Reasons #1Jenkins is not properly installed on server.
Steps to DiagnoseRun the command : /etc/init.d/jenkins status. It will show error.
Commands to validate / SolutionReinstall the jenkins on server.


Unable to generate NetStorm Performance Report

Possible Reasons #1netstorm.checkRuleReport.format is set to 0 in $NS_WDIR/webapps/sys/config.ini
Steps to DiagnoseCheck the config.ini file for the keyword value.
Commands to validate / SolutionEnable the netstorm.checkRuleReport.format to 1 and restart the tomcat of the controller.


Unable to update rule in check profile

Possible Reasons #1Steps not followed properly to update check profile.
Steps to DiagnoseNeed to know the exact step followed.
Commands to validate / SolutionSteps needs to be done to update rule.

1.Select rule for update.

2.Change value in rule which needs to be changed.

3.Click Add Rule.

4.Click Save Rule.


Facing error while scheduling task through Jenkins

Possible Reasons #1Mismatch in netstorm build and netstorm jenkins hpi used.
Steps to DiagnoseCompare the netstorm build version and netstorm’s jenkins api used.
Commands to validate / SolutionUse the correct netstorm jenkin api matching the netstorm build.


Not getting report on mail

Possible Reasons #1Sendmail is not running.
Steps to DiagnoseCheck whether the sendmail is running or not using command: /etc/init.d/sendmail status.
Commands to validate / SolutionStart the sendmail, using command: /etc/init.d/sendmail start.

Possible Reasons #2Check profile’s rule not configured properly in testsuite.
Steps to DiagnoseCheck the jenkins testsuite configuration.
Commands to validate / SolutionIn jenkins testsuite configuration, correct the checkprofile rule.