NetOcean Troubleshoot
|
Unable to Start HPD |
| Possible Reasons #1 | Due To SSL certificate Error |
| Steps to Diagnose | Create the SSL chain file certificate in cert directory |
| Commands to validate |
$ openssl genrsa -out root.key 2048
$ openssl req -new -x509 -days 365 -key root.key -subj “/C=US/ST=California/L=Santa Clara/O=Cavisson Systems Inc./CN=Cavisson Root CA” -out root.crt$ openssl req -newkey rsa:2048 -nodes -keyout device.key -subj “”/C=US/ST=California/L=Santa Clara /O=Cavisson Systems Inc./CN=*.vip.local”” -out device.csr$ openssl x509 -req -extfile <(printf “subjectAltName=DNS:nde-dr.vip.local,DNS:nde47.vip.local,DNS:nde48.vip.local”) -days 365 -in device.csr -CA root.crt -CAkey root.key -CAcreateserial -out device.crt$ openssl genrsa -out proxyCA.key 2048$ openssl req -new -x509 -days 365 -key proxyCA.key -subj “/C=US/ST=California/L=Santa Clara/O=Cavisson Systems Inc./CN=My Custom C”” -out proxyCA.cer$ openssl req -newkey rsa:2048 -nodes -keyout proxy.key -subj “/C=US/ST=California/L=Santa Clara/O=Cavisson Systems Inc./CN=*” -out proxy.csr$ openssl x509 -req -days 365 -in proxy.csr -CA proxyCA.crt -CAkey proxyCA.key -CAcreateserial -out proxy.pemopenssl x509 -in server.crt -text -noout |
| Solution | Create an SSL Certificate by following the SSL Certificate creation Document Or contact Networking team for help. |
| Possible Reasons #2 | Due to Port Issues |
| Steps to Diagnose | Check the available ports to use in HPD. From back end check, sudo netstat -natp to check the availability status |
| Commands to validate | sudo -netstat natp | grep <port>
/etc/init.d/hpd show /etc/init.d/hpd restart |
| Solution | Once we give a free port, the hpd will be started. |
| Possible Reasons #3 | Due to HPD license Error |
| Steps to Diagnose | Check license is valid or not using command “nsu_show_license -h”.
1) This shows license file is not present. 2) This shows invalid licence/expired license. |
| Commands to validate | nsu_show_license -h
./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0 sz and rz |
| Solution |
If licence file is not present, we can create a HPD licence in
m/c 10.10.30.2 and have to deploy it in required m/c.*). To provide license for HPD from local machine 10.10.30.2./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0sz license.nl1 : to export license file on desktop in working directory from root of 10.10.30.2rz license.nl1 : to import license file from desktop to hpd > .license |
|
Unable to activate the Services |
| Possible Reasons #1 | Due to wrong configuration in template |
| Steps to Diagnose | Click on the HPD restart button in GUI (The tick mark in top right corner). It will show us the reason why HPD is not getting restarted and due to which service, the error is coming. |
| Commands to validate | – |
| Solution | As per the error, we have to configure the service and its templates. |
|
Facing issues while recording |
| Possible Reasons #1 | Getting below message on GUI Port is already running please select another port to record |
| Steps to Diagnose | Check the recording with some other available ports. From back end check, sudo netstat -natp to check the availability status |
| Commands to validate | sudo -netstat natp | grep <port> |
| Solution | Once we give a free port, the recording will be started. |
| Possible Reasons #2 | Some ports may have blocked in Firewall of the Machine |
| Steps to Diagnose | 1- Try to record with some free port
2- If HPD is running then stop hpd and use the HPD port. |
| Commands to validate | sudo -netstat natp | grep <port> |
| Solution | Co-ordinate with respective team/Networking team to validate Firewall blockage of ports. |
|
Service is getting disappeared from UI |
| Possible Reasons #1 | Some corrupted or faulty service may have present in the service list. |
| Steps to Diagnose | 1- Go to the hpd/correlation/default/default/services directory
2- Take a backup of all the services to another directory. 3- Clear the current directory 4- Start porting all the services one by one to the blank directory and keep monitoring the service list in UI. |
| Commands to validate | cp, mv
rm -r tar -zcvf filename.tar.gz file-path tar -xvf file name |
| Solution | 1- You will see one or more services were making the whole service list blank.
2 – Open the service.conf file of the service and validate and reconfigure all the keywords by comparing with other services and save. 3- After proper reconfiguration, the service list will be visible. |
| Possible Reasons #2 | Due to the absence of URL keyword in service.conf file in back end |
| Steps to Diagnose | 1 – In back end, open the particular service directory
2 – Open service.conf file and check whether the URL is defined or not. |
| Commands to validate | vi, cd
/etc/init.d/hpd restart |
| Solution | If the URL is not defined there or properly then we have to define that. Otherwise the service will remain invisible in UI. |
|
Unable to open GUI of NO machine |
| Possible Reasons #1 | The shell ‘/etc/init.d/hpd check_after_restart’ getting stuck due to which we are unable to login machine |
| Steps to Diagnose | Check the guierror.log and observe whether this shell is getting stuck or not. |
| Commands to validate | cd, tailf, vi, sh |
| Solution | We have to comment “check_after_restart_fun” line in the file hpd_control |
| Possible Reasons #2 | Postgress may be not running properly. |
| Steps to Diagnose | 1- From back end, check whether postgress is properly running or not. |
| Commands to validate | psql test cavisson
/etc/init.d/postgressql restart |
| Solution | Restart the postgress and try again login. |
|
Export Services Feature not working |
| Possible Reasons #1 | Ownership of /home/cavisson/logs/tools could be root |
| Steps to Diagnose | 1- Check the error and permission in gui error log.
2- Then we try the same command in debug mode and we can find that ownership of /home/cavisson/logs/tools is root. |
| Commands to validate | sh -x
cd, vi |
| Solution | We have to change the ownership to cavisson and we will be able to use the export services option. |
|
GUI and Access log Response time mismatch |
| Possible Reasons #1 | GUI response time includes network speed |
| Steps to Diagnose | 1 – GUI response time always contain network speed+load time and SVC delay If applied.
2- But back end shows the exact time period taken by the service to give response. |
| Commands to validate | — |
| Solution | This is not an issue and it is a NetOcean Feature. |
|
Unable to Start HPD |
| Possible Reasons #1 | Due to Port configuration in HPD.conf |
| Steps to Diagnose | We need to check in which port http and https Port is configured then checked ports are in which state (listen,connect) |
| Commands to validate | sudo -netstat natp | grep <port> |
| Solution | In keyword HPD_SPORT and HPD_PORT, provide a free and available port and then restart HPD. |
| Possible Reasons #2, #3, #4 | 2- If any service is not properly configured or corrupted then due to that particular enabled service, HPD does not start.
3- Some other reasons like keyword configuration, required file absence. 4- Some services may have got corrupted. |
| Steps to Diagnose | 1- First, we have to move core to /home/cavisson/work/hpd.
2- Then open log directory and open hpd_error.log” |
| Commands to validate | cd /home/cavison/work/hpd
vi hpd_error.log /etc/init.d/hpd restart |
| Solution | 1- Check what is the error to start the HPD and have to correct the defect.
2- Restart HPD, with the command /etc/init.d/hpd restart |
|
Access log in not appending in NO |
| Possible Reasons #1 | Make sure that application is pointing to NO or not. |
| Steps to Diagnose | By enabling HPD TRACE level in hpd.conf file, we are unable to see the service request and response .dat file. |
| Commands to validate | — |
| Solution | If HPD Trace level is enabled in hpd.conf file, then we have to disable that and have to restart hpd. |
|
NetOcean is taking more time to response |
| Possible Reasons #1 | HPD might be running in Debug mode |
| Steps to Diagnose | Run command /etc/init.d/hpd show
It will show us currently running hpd processes. |
| Commands to validate | locate hpd_all_debug_off.sh
/etc/init.d/hpd show sh hpd_all_debug_off.sh |
| Solution |
1 – It may be because of hpd is running in debug mode and also if the debug trace is enabled.So, we have to make sure that hpd is not running in debug mode and also debug trace is disabled. 2- In order to stop the hpd debug mode, we can execute 3-Then we have to restart the hpd for normal mode. |
| Possible Reasons #2 | SVC Time may have applied |
| Steps to Diagnose | 1 – Open GUI
2- Go to Services >> Manage Services 3- Open any service 4- Go to Service Delay and check Service Delay |
| Commands to validate | — |
| Solution | Response will come according to the Service Delay. So, manage the service delay according to that. |
| Possible Reasons #3 | Trace Level may have applied |
| Steps to Diagnose | 1 – Open GUI
2- Go to Services > Manage Services 3- Open any service 4- Go to Trace level and check |
| Commands to validate | — |
| Solution | Trace level also impacts response time. Hence use it according to the necessity or disable it. |
|
HPD is stopped frequently while load test is running |
| Possible Reasons #1 | Core Files may be generating |
| Steps to Diagnose | 1 – Go to /home/cavisson/core_files/
2- To read core use gdb /home/cavisson/work/bin/netstorm ‘core’ |
| Commands to validate | cd /home/cavisson/core_files/
gdb /home/cavisson/work/bin/netstorm ‘core’ |
| Solution | Contact Dev Team to provide a patch |
|
HPD child process continuously goes in Defunct state |
| Possible Reasons #1 | If some core files are missing from HPD_core directory, this issue can occur. |
| Steps to Diagnose | 1 – First, we have to move core to /home/cavisson/work/HPD_core.
2- We need to check these files are present or not hpd_control,nsu_stop_hpd, |
| Commands to validate | cd /home/cavisson/work/HPD_core
nsu_server_admin -s -F -D <Path> |
| Solution | 1- If pd_control,nsu_stop_hpd,, hpd_control and nsu_stop_hpd files are not present then copy them from other m./c. |
|
Unable to save the services |
| Possible Reasons #1 | Due to wrong configuration in Template |
| Steps to Diagnose |
1- Click the Save button. 2- If the service could not be saved, it throws an error message. We need to reconfigure the service template according to that. 3- If the service is getting saved without any error, then during the HPD restart from GUI, it must be giving the error message of activation. |
| Commands to validate | — |
| Solution | 1- We have to reconfigure the service templates according to the errors.
2- Save that service then and restart HPD. |
|
Insufficient IOVector in NetOcean |
| Possible Reasons #1 | Due to Keyword Error |
| Steps to Diagnose |
1 – Go to home/cavisson/work/hpd/conf/hpd.conf 2- We need to add a keyword in hpd.conf file. What if does? 4 – We can provide value up to 255000 |
| Commands to validate | vi home/cavisson/work/hpd/conf/hpd.conf |
| Solution | After using the proper keyword with values, issue will get resolved. |
|
NetOcean response getting spiked up |
| Possible Reasons #1 | SVC Time have applied |
| Steps to Diagnose | 1 – Open GUI
2- Go to Services >> Manage Services 3- Open any service 4- Go to Service Delay and check Service Delay |
| Commands to validate | — |
| Solution | Response will come according to the Service Delay. So, manage the service delay according to that. |
| Possible Reasons #2 | Someone have restarted hpd during the test run |
| Steps to Diagnose | Observe the response at the point of spike and talk to user whether they have restarted the hpd during a ongoing load test. |
| Commands to validate | — |
| Solution | Advice the user, not to restart hpd during any test as it creates zigzag and spikes in response graph. |
|
Unable to stop HPD debug mode |
| Possible Reasons #1 | Multiple HPD Processes running in background |
| Steps to Diagnose | 1- In back end, disable HPD Debug keyword in hpd.conf file and restart HPD.
2- Run hpd show to check the hpd mode has changed or not. |
| Commands to validate | locate hpd_all_debug_off.sh
/etc/init.d/hpd show hpd_all_debug_off.sh |
| Solution | In order to stop the hpd debug mode, we can execute hpd_all_debug_off.sh . This stops the hpd in debug mode without any change. |
|
Log files are generated even on disabling Debug Log |
| Possible Reasons #1 | If HPD_DEBUG_TRACE keyword is enabled, it generates unwanted log files |
| Steps to Diagnose | 1- Go to hpd/conf/hpd.conf file
2- Disable HPD_DEBUG_TRACE keyword. |
| Commands to validate | /etc/init.d/hpd restart |
| Solution | After disabling the HPD_DEBUG_TRACE keyword, the debug file will not not generated Restart HPD after doing changes. |
|
Response time is coming 0 in the Accesslog File |
| Possible Reasons #1 | 404 response code might be coming |
| Steps to Diagnose | 1- In the same accesslog files, check for the response codes. Whether it is 404 or not. |
| Commands to validate | cd hpd/logs/accesslog/ |
| Solution | If 404 error is coming, we need to validate the service template and have to correct it. |
|
Invalid HPD Licence |
| Possible Reasons # | 1- HPD Licence might have expired.
2- HPD Licence might not be present. |
| Steps to Diagnose | Check license is valid or not using command “nsu_show_license -h”.
1) This shows license file is not present. 2)This shows invalid licence/expired license. |
| Commands to validate | nsu_show_license -h
./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0 sz and rz |
| Solution |
If licence file is not present. We can create a HPD licence in m/c 10.10.30.2 and have to deploy it in required m/c. *)To provide license for HPD from local machine 10.10.30.2 ./_nsu_generate_hpd_license -I 45.43.30.90:80,81,443 -P 1 -t 1 -s 0 sz license.nl1: to export license file on desktop in the working directory from root of 10.10.30.2 rz license.nl1 : to import license file from desktop to hpd > .license |
|
Invalid SSL Certificate |
| Possible Reasons # | 1- SSL Certificate validity might have expired.
2- No SSL certificate will be there. 3- Certificate file has removed or modified by someone. 4- SSL Certificate Keyword might not have properly defined. |
| Steps to Diagnose |
1- Go to hpd/conf/hpd.conf file 2- Search for SSL Certificate Keyword. 3- Check whether this Keyword is properly applied or not. 4- Also check the Certificate path is correct or not. 5- Go to the certificate directory path and check whether the certificate is present there or not. 6- If present, then validate the certificate. |
| Commands to validate |
$ openssl genrsa -out root.key 2048 $ openssl req -new -x509 -days 365 -key root.key -subj “/C=US/ST=California/L=Santa Clara/O=Cavisson Systems Inc./CN=Cavisson Root CA” -out root.crt $ openssl req -newkey rsa:2048 -nodes -keyout device.key -subj “/C=US/ST=California/L=Santa Clara $ openssl x509 -req -extfile <(printf “subjectAltName=DNS:nde-dr.vip.local,DNS:nde47.vip.local,DNS:nde48.vip.local”) -days $ openssl genrsa -out proxyCA.key 2048 $ openssl req -new -x509 -days 365 -key proxyCA.key -subj “/C=US/ST=California/L=Santa $ openssl req -newkey rsa:2048 -nodes -keyout proxy.key -subj “/C=US/ST=California/L=Santa $ openssl x509 -req -days 365 -in proxy.csr -CA proxyCA.crt -CAkey proxyCA.key -CAcreateserial -out proxy.pem openssl x509 -in server.crt -text -noout |
| Solution | Create a SSL Certificate by following SSL Certificate creation docoment Or contact Networking team for help. |
|
NetOcean Service Delay time is not working |
| Possible Reasons #1 | Make sure that Service delay time applied correctly or not also we should know the priority of Service delay time |
| Steps to Diagnose | 1. Open GUI
2. Go to Services > Manage Services 3. Choose a Service 4. Go inside its template and go to Service Delay 5. Check how SVC delay is applied |
| Commands to validate | — |
| Solution | Always prefer to use the Template Based Service time because its priority level remains higher than Service based Service time delay. |
|
Required 4xx/5xx server response once in a while |
| Possible Reasons #1 | The requirement is to configure a service which will give response as 4xx/5xx once in a while. To configure this we can use random number parameter |
| Steps to Diagnose |
1- Select Services -> Add to create a new service Example – Create first template for success type 2- Add another template for failure type with status as 500 Server error 3- Select Parameter -> Random Number parameter 4- Here Use minimum value as 1 & maximum value as 5 which will update the value on every use. 5- Activate both the services & restart the hpd using the “Activate” button. 6- You can test this service using “Test” button or by using the URL in scenario. |
| Commands to validate | — |
| Solution | We need to follow these mention steps to configure the requirement. |
|
NetOcean Response is giving 404 in Browser but it is running fine in NO machine. |
| Possible Reasons #1 | Parameters used in the service might not be passing in the response body. |
| Steps to Diagnose |
1- Go inside the services. 2- Click on the Service Parameters. 3- Validate all the Service parameters with the response template. |
| Commands to validate | — |
| Solution | We have to use the strings in the response templates in order to fulfil the service condition. |