NetOcean Troubleshoot

Unable to Start HPD

Possible Reasons #1Due To SSL certificate Error
Steps to DiagnoseCreate the SSL chain file certificate in cert directory
Commands to validate
$ openssl genrsa -out root.key 2048
$ openssl req -new -x509 -days 365 -key root.key -subj “/C=US/ST=California/L=Santa Clara/O=Cavisson Systems Inc./CN=Cavisson Root CA” -out root.crt$ openssl req -newkey rsa:2048 -nodes -keyout device.key -subj “”/C=US/ST=California/L=Santa Clara
/O=Cavisson Systems Inc./CN=*.vip.local”” -out device.csr$ openssl x509 -req -extfile <(printf “subjectAltName=DNS:nde-dr.vip.local,DNS:nde47.vip.local,DNS:nde48.vip.local”) -days
365 -in device.csr -CA root.crt -CAkey root.key -CAcreateserial -out device.crt$ openssl genrsa -out proxyCA.key 2048$ openssl req -new -x509 -days 365 -key proxyCA.key -subj “/C=US/ST=California/L=Santa
Clara/O=Cavisson Systems Inc./CN=My Custom C”” -out proxyCA.cer$ openssl req -newkey rsa:2048 -nodes -keyout proxy.key -subj “/C=US/ST=California/L=Santa
Clara/O=Cavisson Systems Inc./CN=*” -out proxy.csr$ openssl x509 -req -days 365 -in proxy.csr -CA proxyCA.crt -CAkey proxyCA.key -CAcreateserial -out proxy.pemopenssl x509 -in server.crt -text -noout

SolutionCreate an SSL Certificate by following the SSL Certificate creation
Document Or contact Networking team for help.

Possible Reasons #2Due to Port Issues
Steps to DiagnoseCheck the available ports to use in HPD.
From back end check, sudo netstat -natp to check the availability status
Commands to validatesudo -netstat natp | grep <port>

/etc/init.d/hpd show

/etc/init.d/hpd restart

SolutionOnce we give a free port, the hpd will be started.

Possible Reasons #3Due to HPD license Error
Steps to DiagnoseCheck license is valid or not using command “nsu_show_license -h”.

1) This shows license file is not present.

2) This shows invalid licence/expired license.

Commands to validatensu_show_license -h

./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0

sz and rz

Solution
If licence file is not present, we can create a HPD licence in
m/c 10.10.30.2 and have to deploy it in required m/c.*). To provide license for HPD from local machine 10.10.30.2./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0sz license.nl1 : to export license file on desktop in working directory from root of 10.10.30.2rz license.nl1 : to import license file from desktop to hpd > .license


Unable to activate the Services

Possible Reasons #1Due to wrong configuration in template
Steps to DiagnoseClick on the HPD restart button in GUI (The tick mark in top right corner).
It will show us the reason why HPD is not getting restarted and due to which service, the error is coming.
Commands to validate
SolutionAs per the error, we have to configure the service and its templates.


Facing issues while recording

Possible Reasons #1Getting below message on GUI Port is already running please select another port to record
Steps to DiagnoseCheck the recording with some other available ports.
From back end check, sudo netstat -natp to check the availability status
Commands to validatesudo -netstat natp | grep <port>
SolutionOnce we give a free port, the recording will be started.

Possible Reasons #2Some ports may have blocked in Firewall of the Machine
Steps to Diagnose1- Try to record with some free port

2- If HPD is running then stop hpd and use the HPD port.

Commands to validatesudo -netstat natp | grep <port>
SolutionCo-ordinate with respective team/Networking team to validate Firewall blockage of ports.


Service is getting disappeared from UI

Possible Reasons #1Some corrupted or faulty service may have present in the service list.
Steps to Diagnose1- Go to the hpd/correlation/default/default/services directory

2- Take a backup of all the services to another directory.

3- Clear the current directory

4- Start porting all the services one by one to the blank directory and keep monitoring the service list in UI.

Commands to validatecp, mv

rm -r

tar -zcvf filename.tar.gz file-path

tar -xvf file name

Solution1- You will see one or more services were making the whole service list blank.

2 – Open the service.conf file of the service and validate and reconfigure all the keywords by comparing with other services and save.

3- After proper reconfiguration, the service list will be visible.

Possible Reasons #2Due to the absence of URL keyword in service.conf file in back end
Steps to Diagnose1 – In back end, open the particular service directory

2 – Open service.conf file and check whether the URL is defined or not.

Commands to validatevi, cd

/etc/init.d/hpd restart

SolutionIf the URL is not defined there or properly then we have to define that. Otherwise the service will remain invisible in UI.


Unable to open GUI of NO machine

Possible Reasons #1The shell ‘/etc/init.d/hpd check_after_restart’ getting stuck due to which we are unable to login machine
Steps to DiagnoseCheck the guierror.log and observe whether this shell is getting stuck or not.
Commands to validatecd, tailf, vi, sh
SolutionWe have to comment “check_after_restart_fun” line in the file hpd_control

Possible Reasons #2Postgress may be not running properly.
Steps to Diagnose1- From back end, check whether postgress is properly running or not.
Commands to validatepsql test cavisson

/etc/init.d/postgressql restart

SolutionRestart the postgress and try again login.


Export Services Feature not working

Possible Reasons #1Ownership of /home/cavisson/logs/tools could be root
Steps to Diagnose1- Check the error and permission in gui error log.

2- Then we try the same command in debug mode and we can find that ownership of

/home/cavisson/logs/tools is root.

Commands to validatesh -x

cd, vi

SolutionWe have to change the ownership to cavisson and we will be able to use the export services option.


GUI and Access log Response time mismatch

Possible Reasons #1GUI response time includes network speed
Steps to Diagnose1 – GUI response time always contain network speed+load time and SVC delay If applied.

2- But back end shows the exact time period taken by the service to give response.

Commands to validate
SolutionThis is not an issue and it is a NetOcean Feature.


Unable to Start HPD

Possible Reasons #1Due to Port configuration in HPD.conf
Steps to DiagnoseWe need to check in which port http and https Port is configured then checked ports are in which state (listen,connect)
Commands to validatesudo -netstat natp | grep <port>
SolutionIn keyword HPD_SPORT and HPD_PORT, provide a free and available port and then restart HPD.

Possible Reasons #2, #3, #42- If any service is not properly configured or corrupted then due to that particular enabled service, HPD does not start.

3- Some other reasons like keyword configuration, required file absence.

4- Some services may have got corrupted.

Steps to Diagnose1- First, we have to move core to /home/cavisson/work/hpd.

2- Then open log directory and open hpd_error.log”

Commands to validatecd /home/cavison/work/hpd

vi hpd_error.log

/etc/init.d/hpd restart

Solution1- Check what is the error to start the HPD and have to correct the defect.

2- Restart HPD, with the command /etc/init.d/hpd restart


Access log in not appending in NO

Possible Reasons #1Make sure that application is pointing to NO or not.
Steps to DiagnoseBy enabling HPD TRACE level in hpd.conf file, we are unable to see the service request and response .dat file.
Commands to validate
SolutionIf HPD Trace level is enabled in hpd.conf file, then we have to disable that and have to restart hpd.


NetOcean is taking more time to response

Possible Reasons #1HPD might be running in Debug mode
Steps to DiagnoseRun command /etc/init.d/hpd show

It will show us currently running hpd processes.
Debug mode hpd displays as hpd.debug

Commands to validatelocate hpd_all_debug_off.sh

/etc/init.d/hpd show

sh hpd_all_debug_off.sh

Solution

1 – It may be because of hpd is running in debug mode and also if the debug trace is enabled.So, we have to make sure that hpd is not running in debug mode and also debug trace is disabled.

2- In order to stop the hpd debug mode, we can execute
hpd_all_debug_off.sh . This stops the hpd in debug mode without any change.

3-Then we have to restart the hpd for normal mode.

Possible Reasons #2SVC Time may have applied
Steps to Diagnose1 – Open GUI

2- Go to Services >> Manage Services

3- Open any service

4- Go to Service Delay and check Service Delay

Commands to validate
SolutionResponse will come according to the Service Delay. So, manage the service delay according to that.

Possible Reasons #3Trace Level may have applied
Steps to Diagnose1 – Open GUI

2- Go to Services > Manage Services

3- Open any service

4- Go to Trace level and check

Commands to validate
SolutionTrace level also impacts response time. Hence use it according to the necessity or disable it.


HPD is stopped frequently while load test is running

Possible Reasons #1Core Files may be generating
Steps to Diagnose1 – Go to /home/cavisson/core_files/

2- To read core use gdb /home/cavisson/work/bin/netstorm ‘core’

Commands to validatecd /home/cavisson/core_files/

gdb /home/cavisson/work/bin/netstorm ‘core’

SolutionContact Dev Team to provide a patch


HPD child process continuously goes in Defunct state

Possible Reasons #1If some core files are missing from HPD_core directory, this issue can occur.
Steps to Diagnose1 – First, we have to move core to /home/cavisson/work/HPD_core.

2- We need to check these files are present or not hpd_control,nsu_stop_hpd,
hpd_control and nsu_stop_hpd file.

Commands to validatecd /home/cavisson/work/HPD_core

nsu_server_admin -s -F -D <Path>

Solution1- If pd_control,nsu_stop_hpd,, hpd_control and nsu_stop_hpd files are not present then copy them from other m./c.


Unable to save the services

Possible Reasons #1Due to wrong configuration in Template
Steps to Diagnose

1- Click the Save button.

2- If the service could not be saved, it throws an error message. We need to reconfigure the service template according to that.

3- If the service is getting saved without any error, then during the HPD restart from GUI, it must be giving the error message of activation.
We need to reconfigure that.

Commands to validate
Solution1- We have to reconfigure the service templates according to the errors.

2- Save that service then and restart HPD.


Insufficient IOVector in NetOcean

Possible Reasons #1Due to Keyword Error
Steps to Diagnose

1 – Go to home/cavisson/work/hpd/conf/hpd.conf

2- We need to add a keyword in hpd.conf file.
That is IO_VECTOR_SIZE 250000.

What if does?
3- 2500000 is the normal value that we can use and it will work with the service without any issues.

4 – We can provide value up to 255000

Commands to validatevi home/cavisson/work/hpd/conf/hpd.conf
SolutionAfter using the proper keyword with values, issue will get resolved.


NetOcean response getting spiked up

Possible Reasons #1SVC Time have applied
Steps to Diagnose1 – Open GUI

2- Go to Services >> Manage Services

3- Open any service

4- Go to Service Delay and check Service Delay

Commands to validate
SolutionResponse will come according to the Service Delay. So, manage the service delay according to that.

Possible Reasons #2Someone have restarted hpd during the test run
Steps to DiagnoseObserve the response at the point of spike and talk to user whether they have
restarted the hpd during a ongoing load test.
Commands to validate
SolutionAdvice the user, not to restart hpd during any test as it creates zigzag and spikes in response graph.


Unable to stop HPD debug mode

Possible Reasons #1Multiple HPD Processes running in background
Steps to Diagnose1- In back end, disable HPD Debug keyword in hpd.conf file and restart HPD.

2- Run hpd show to check the hpd mode has changed or not.

Commands to validatelocate hpd_all_debug_off.sh

/etc/init.d/hpd show

hpd_all_debug_off.sh

SolutionIn order to stop the hpd debug mode, we can execute hpd_all_debug_off.sh .
This stops the hpd in debug mode without any change.


Log files are generated even on disabling Debug Log

Possible Reasons #1If HPD_DEBUG_TRACE keyword is enabled, it generates unwanted log files
Steps to Diagnose1- Go to hpd/conf/hpd.conf file

2- Disable HPD_DEBUG_TRACE keyword.

Commands to validate/etc/init.d/hpd restart
SolutionAfter disabling the HPD_DEBUG_TRACE keyword, the debug file will not not generated
Restart HPD after doing changes.


Response time is coming 0 in the Accesslog File

Possible Reasons #1404 response code might be coming
Steps to Diagnose1- In the same accesslog files, check for the response codes. Whether it is 404 or not.
Commands to validatecd hpd/logs/accesslog/
SolutionIf 404 error is coming, we need to validate the service template and have to correct it.


Invalid HPD Licence

Possible Reasons #1- HPD Licence might have expired.

2- HPD Licence might not be present.

Steps to DiagnoseCheck license is valid or not using command “nsu_show_license -h”.

1) This shows license file is not present.

2)This shows invalid licence/expired license.

Commands to validatensu_show_license -h

./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0

sz and rz

Solution

If licence file is not present. We can create a HPD licence in m/c 10.10.30.2 and have to deploy it in required m/c.

*)To provide license for HPD from local machine 10.10.30.2

./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0

sz license.nl1: to export license file on desktop in the working directory from root of 10.10.30.2

rz license.nl1 : to import license file from desktop to hpd > .license


Invalid SSL Certificate

Possible Reasons #1- SSL Certificate validity might have expired.

2- No SSL certificate will be there.

3- Certificate file has removed or modified by someone.

4- SSL Certificate Keyword might not have properly defined.

Steps to Diagnose

1- Go to hpd/conf/hpd.conf file

2- Search for SSL Certificate Keyword.

3- Check whether this Keyword is properly applied or not.

4- Also check the Certificate path is correct or not.

5- Go to the certificate directory path and check whether the certificate is present there or not.

6- If present, then validate the certificate.

Commands to validate

$ openssl genrsa -out root.key 2048

$ openssl req -new -x509 -days 365 -key root.key -subj “/C=US/ST=California/L=Santa Clara/O=Cavisson Systems Inc./CN=Cavisson Root CA” -out root.crt

$ openssl req -newkey rsa:2048 -nodes -keyout device.key -subj “/C=US/ST=California/L=Santa Clara
/O=Cavisson Systems Inc./CN=*.vip.local” -out device.csr

$ openssl x509 -req -extfile <(printf “subjectAltName=DNS:nde-dr.vip.local,DNS:nde47.vip.local,DNS:nde48.vip.local”) -days
365 -in device.csr -CA root.crt -CAkey root.key -CAcreateserial -out device.crt

$ openssl genrsa -out proxyCA.key 2048

$ openssl req -new -x509 -days 365 -key proxyCA.key -subj “/C=US/ST=California/L=Santa
Clara/O=Cavisson Systems Inc./CN=My Custom CA”” -out proxyCA.cer

$ openssl req -newkey rsa:2048 -nodes -keyout proxy.key -subj “/C=US/ST=California/L=Santa
Clara/O=Cavisson Systems Inc./CN=*” -out proxy.csr

$ openssl x509 -req -days 365 -in proxy.csr -CA proxyCA.crt -CAkey proxyCA.key -CAcreateserial -out proxy.pem

openssl x509 -in server.crt -text -noout

SolutionCreate a SSL Certificate by following SSL Certificate creation docoment Or contact Networking team for help.


NetOcean Service Delay time is not working

Possible Reasons #1Make sure that Service delay time applied correctly or not
also we should know the priority of Service delay time
Steps to Diagnose1. Open GUI

2. Go to Services > Manage Services

3. Choose a Service

4. Go inside its template and go to Service Delay

5. Check how SVC delay is applied

Commands to validate
SolutionAlways prefer to use the Template Based Service time because its priority level remains higher than Service based Service time delay.


Required 4xx/5xx server response once in a while

Possible Reasons #1The requirement is to configure a service which will give response as 4xx/5xx once in a while. To configure this we can use random number parameter
Steps to Diagnose

1- Select Services -> Add to create a new service Example – Create first template for success type

2- Add another template for failure type with status as 500 Server error

3- Select Parameter -> Random Number parameter

4- Here Use minimum value as 1 & maximum value as 5 which will update the value on every use.

5- Activate both the services & restart the hpd using the “Activate” button.

6- You can test this service using “Test” button or by using the URL in scenario.

Commands to validate
SolutionWe need to follow these mention steps to configure the requirement.


NetOcean Response is giving 404 in Browser but it is running fine in NO machine.

Possible Reasons #1Parameters used in the service might not be passing in the response body.
Steps to Diagnose

1- Go inside the services.

2- Click on the Service Parameters.

3- Validate all the Service parameters with the response template.

Commands to validate
SolutionWe have to use the strings in the response templates in order to fulfil the service condition.