NetVision Troubleshoot
Unable to capture sessions |
Possible Reasons #1 | Tag is not inserted properly in target application. |
Steps to Diagnose & Command Used |
For Capturing session in NetVision, the first step is we need to check is our Tag (NV_TAG), which is responsible to bring data over NV Server, is inserted or not on Target Application (Application which we are going to monitor). It can be verified by searching nv_bootstrap.js in network panel of developer field of browser. If nv_bootstrap.js or startup tag is not visible then it can be assumed tag not inserted on target application. |
Solution |
Insert tag on target application in below format –
<script src=//NV Machine IP/Domain:HPD_SPORT/nv/Client_ID/nv_bootstrap.js type=””text/javascript””></script>HPD_SPORT can be found in /home/cavisson/controller_name/hpd/conf >> hpd.conf of NV ServerClient ID can be found in /home/cavisson/controller_name/hpd/rum/config >> rum.conf of NV Server” |
Possible Reasons #2 | HPD may not be running |
Steps to Diagnose & Command Used | Command to check hpd is running or not –
For work controller – /etc/init.d/hpd show For other controller other than work – /etc/init.d/hpd_controller_name show |
Solution |
Search for error in hpd_error.0.log in path – /home/cavisson/controller_name/hpd/logs/ Error Possibility: 1) Port binding – Check for Port binding either in Below are the keywords for same –HPD_PORT, HPD_SPORT, rum.conf – Path – /home/cavisson/controller_name/hpd/rum/config/ Below are the keywords for same – 2) Pending sick child – Any process (nr_db_upload,nv_memory_reporter) may not be running. Can be checked using below command – ps -ef | grep nv_memory_reporter – If not running can check error in nv_memory_reporter.trace ps -ef | grep nr_db_upload – If not running can check error in nr_db_upload_error.log in path – 3) Postgres not running – 4) Invalid HPD License – Need to deploy valid License for HPD |
The captured sessions not visible in Completed session window |
Possible Reasons #1 | Make sure tomcat process is making connection with database in order to show the data |
Steps to Diagnose & Command Used |
It can be checked using the command – sudo netstat -natp |grep 5432 – Port of postgres can be viewed in postgresql.conf in path – /etc/postgresql/9.5/main Always make sure that port of postgres must have connection with tomcat/java process to show data of completed sessions. |
Commands to validate / Solution |
If not making connection with tomcat then kill the process in which it is making connection using the command – kill -9 (Process Id) and restart the tomcat. |
Possible Reasons #2 | Postgresql not running |
Steps to Diagnose & Command Used | Check postgres is running or not using below command – ps -ef |grep postgres or /etc/init.d/postgresql status or Pg_isready or we can also check using psql test cavisson |
Solution |
If it is not running check the errors from postgresql logs in path –
/home/cavisson/pgsql/data/pg_logand take action according to logs found.To start PostgresSQL Server, use below command – /etc/init.d/postgresql start” |
Possible Reasons #3 | Data may be coming in Active Session and stored in buffer memory in .CSV format but not able to dump in database, as nr_db_upload process is not running. |
Steps to Diagnose & Command Used | It can be checked using command – ps -ef | grep nr_db_upload |
Solution |
If not running, check error in nr_db_upload_error.log in path –
/home/cavisson/controller_name/hpd/rum/ Error Possibilities 1) Failed to upload to db. Missing data for column 2) PostgresSQL not working 3) Garbage value received with data 4) Table not created to dump data |
Some request of NetVision is getting failed while capturing sessions |
Possible Reasons #1 | Due to Cross Origin Resource Sharing (CORS) Policy |
Steps to Diagnose & Command Used |
NetVision JS Agent has three components: nv_bootstrap.js, cav_nv.js, config.js
Sometimes issue is faced that 3 component request are successfully captured but data capturing requests are getting failed such as – timing, pagedump, useraction, domwatcher, event logger etc.This may be due to CORS Policy which can be be viewed in Console tab of developer tools of browser while capturing a session. |
Commands to validate / Solution |
Check CorsFilter in web.xml file to handle multiple requests.
Path: webapps/netvision/WEB-INF/web.xml All the properties like allowed methods and etc should be mentioned in this file. NV has provided keyword for handling CORS Policy Error Note: We need to restart the HPD Process to reflect the changes. |
NetVision GUI is not opening |
Possible Reasons #1 | Tomcat is not running |
Steps to Diagnose & Command Used | To check if tomcat process running or not –
ps -ef | grep tomcat | grep -w Controller_name |
Solution | To Start tomcat , need to run given command from anywhere or from the working directory.
(i) /etc/init.d/tomcat start (For work Controller) |
Possible Reasons #2 | Tomcat is running but got following exception in catalina.out Bind Exception Caused by: java.net.BindException: Address already in use (Bind failed) at java.net.PlainSocketImpl.socketBind(Native Method) |
Steps to Diagnose & Command Used | Error can be found in catalina.out in path – $NS_WDIR/apps/apache_tomcat-7.0.91/logs/catalina.out |
Solution | To solve this error, we can either kill the service taking that port or can change our web server to run on another port.
The port conflicting with which service can be verified using below command – |
Unable to Login the Product UI |
Possible Reasons #1 | Postgresql not running |
Steps to Diagnose & Command Used | Check postgres is running or not using below command – ps -ef |grep postgres or /etc/init.d/postgresql status or Pg_isready or we can also check using psql test cavisson |
Solution | If it is not running check the errors from postgresql logs in path – /home/cavisson/pgsql/data/pg_logand take action according to logs found.To start PostgresSQL Server, use below command – /etc/init.d/postgresql start |
Possible Reasons #2 | Authentication failure due to Given User is not present in database |
Steps to Diagnose & Command Used | Check in database i.e., access_control psql access_control cavisson; |
Solution | If access to that user is not provide then Create a New User using admin in UI 1) Login with admin user. 2) Click Admin menu located in left panel of UI. 3) Then click on Access control in Admin Menu . 4) Click on ‘+’ icon on User to add a new user. |
Replay feature is not working Properly |
Possible Reasons #1 | CSS Content get disturbed not visible properly in case of HTTPs Target Application |
Steps to Diagnose & Command Used | nv.replay.fetchURLFromRemoteServer = true keyword is responsible to fetch resources of target web application locally on NetVision Server.
It can be checked in file – config.ini in path – /home/cavisson/controller_name/webapps/sys/ |
Solution | If this keyword is not mentioned then mention this keyword in config.ini in path – /home/cavisson/controller_name/webapps/sys/ Keyword – nv.replay.fetchURLFromRemoteServer = true |
Possible Reasons #2 | CSS Content get disturbed and not visible properly in case of HTTP Target Application |
Steps to Diagnose & Command Used | It may be due to unable to access HTTP resources on HTTPS. While playing replay of a session, it can be viewed in console column of developer tools and it will also show the Security Constraints that are blocking those resources. |
Solution | Comment Security constraint in web.xml in path – /home/cavisson/work/apps/apache-tomcat-7.0.91/conf that is blocking to allow Http resources and make below keyword in config.ini in path – /home/cavisson/controller_name/webapps/sys/ Keyword – nv.replay.https=false |
Possible Reasons #3 | Getting mixed content error in console while playing replay. |
Steps to Diagnose & Command Used | While playing replay of a session it can be viewed in console column of developer tools. It may be due to some resource content loading over HTTP while some HTTPs content loading over HTTPs. |
Solution |
Mixed content error arise in that case when our Monitored application and Monitoring Application is served over different security protocols.
If our target application is served over HTTP and our monitoring application (NV Replay Feature) is serving over HTTPS then this problem might be occurred. Solution to this problem is disabling this keyword from via making as false from config.ini(webapps/sys) file. Keyword – nv.replay.https=false and moreover disable the security constraints arises in console of developer tool in browser.” |
Possible Reasons #4 | Pagedump available but showing error message – “Pagedump not available in DB loading” |
Steps to Diagnose & Command Used |
“We can cross check that pagedump for that particular session is actually captured or not by following below steps – 1) Search that session id from smart search option available on top right corner of Session GUI. |
Solution | This problem might have occurred due to wrong handling of timestamp from nvIntermediate.jsp file. Path of nvIntermediate.jsp file: /work/webapps/netvision/reports” |
Not getting Transaction |
Possible Reasons #1 | Check for HPD restart/partition Switching |
Steps to Diagnose & Command Used |
“May be due to restating of HPD and partition switching transaction got missed
How to check partition for a specific date: Path : /work/hpd/rum Command to be executed: Eg, du -sh 20200408* Where Format to be followed for time is YYYYMMDD* Note: Whenever HPD gets restarted then a new partition gets created beyond defined configuration in rum.conf file. Keyword: PARTITION_SETTINGS 1 1 8h (In ideal condition partition switches on each 8 hour interval for a date) Hours can be customised as per our requirement.” |
Solution | Check for any HPD restart or partition switching at the time of issue reported in the machine in which the transaction is captured. Rarely it happens while partition switching data loss may occur. |
Page name not captured showing ‘others’ |
Possible Reasons #1 | Pagename may not be configured |
Steps to Diagnose & Command Used | To verify this please follow below steps: 1) Open NV GUI 2) Click on config button on left side bar >> config 3) ConfigUI will open . Click on Pagename from left toggle button to see pagenames configured. |
Solution |
If pagename configured then see the next possible reason.
If pagename not configured then configure pagename by following below steps :In GUI1) Open NV GUI through Admin User. 2) Click on config button on left side bar >> config 3) ConfigUI will open . Click on Pagename from left toggle button . 4) Click the Add button from bottom of the page. 5) Follow the fields, please follow below for example, Page Name: Login Definition Method: URL pattern Complete Url: Active Url Pattern: kohls.login.jsp (Note: URL Pattern should be a Regex pattern that matches Test URL) Test Url: https://www.kohls.com/myaccount/kohls_login.jsp (Note: It is the URL of Page of the target application which user want to configure in NetVision) Test it if matches then save it if it didn’t match there is some mistakes in the Regex pattern used. In CLI 1) Go to path $HPD_ROOT/real/default/nv/client_id/ 2) Open config.js 3) Add below configuration – CONFIG: { pageUrlValues: [ {s: 12, c: /(kohls.login.jsp)/, t: 1, cu: true} ] } where 12 is page id which can be found by typing by typing a query in postgres, psql test cavisson test=> select * from nvpagemetadatatable_clientid ; c is the pattern which to be same as given in GUI, t is Definition method which is 1 for URL Pattern cu is complete url which to be same as given in GUI . 4) After completing above changes save the file. 5) Then perform runtime changes for HPD by command – hpd_sp_rt_debug” |
Possible Reasons #2 | Pagename configured but not correctly |
Steps to Diagnose & Command Used | To verify configuration is correct/incorrect, please follow steps provided in above section. |
Solution | To verify configuration is correct/incorrect, please follow steps provided in above section. |
Unable To Search Sessions In NVSPA |
Possible Reasons #1 | Data not dumped in database |
Steps to Diagnose & Command Used | Data may be coming in Active Session and stored in buffer memory in .CSV format but due to some error in nr_db_upload process it is not dumped in database. |
Solution |
In CLI : Active session data can be searched by below steps – It will list out the .csv files of a session. If data not available in active sessions then we will have to go to the next possible reason. If data is available in .csv but not in the database then follow the next step. 3) In database, that session can be found by typing a query in postgres. Below is the step for same – 4) If no data in database for that session id then there must be some error in nr_db_upload process for which .csv data is not dumped in database. Then go to that partition in which session id is captured, then open file – nr_db_upload.trace and search for that session id. It will list out the error. |
Possible Reasons #2 | Data is not captured in Active Session |
Steps to Diagnose & Command Used | Search for session in access_log in path $HPD_ROOT/logs/access_log/ For example: if you want to search a session with Session id – 849887050766221462 captured on 09/04/2020. Then type below command – grep -irl 849887050766221462 access_log_http_20200409* If no results found” |
Solution | It is failing due to network failure. test_rum call is not reaching to the NV Node. So these request must be failing either at browser end or LTM end. We have to see Stats of failure request at LTM. |
Session not searchable through client IP |
Possible Reasons #1 | Session through that client ip is not captured (In Active Session) |
Steps to Diagnose & Command Used | Search for client ip in access_log in path $HPD_ROOT/logs/access_log/ For example if you want to search a client ip with client ip – 66.249.79.126 captured on 09/04/2020. Then type below command – grep -irl 66.249.79.126 access_log_http_20200409* If no results found |
Solution | It is failing due to network failure. test_rum call is not reaching to the NV Node. So these request must be failing either at browser end or LTM end. We have to see Stats of failure request at LTM. |
Possible Reasons #2 | Session through that client ip is captured but not dumped in db (In Complete Session) |
Steps to Diagnose & Command Used | Data may be coming in Active Session for that client ip and stored in buffer memory in .CSV format but due to some error in nr_db_upload process it is not dumped in database. |
Solution |
In CLI: Active session data for client ip can be searched by below steps – It will list out the .csv files of a session for that client ip. If data not available in active session then we will have to refer the above possible reason. If data is available in .csv but not in the database then follow the next step 3) In database, that session with that client ip can be found by typing a query in postgres. Below is the step for same – 4) If no data in database for that client ip then there must be some error in nr_db_upload process for which .csv data is not dumped in database. Then go to that partition in which session id is captured , then open file – nr_db_upload.trace and search for that session id. It will list out the error.” |
HTTP Data not captured |
Possible Reasons #1 | Keyword may not be enable |
Steps to Diagnose & Command Used | In cav_nv.js or config.js below keyword to capture HTTP data is disable – Path – $HPD_ROOT/real/default/nv/client_id/AjaxMonitor: { enabled: false }, |
Solution | Enable below keyword in cav_nv.js or config.js to capture HTTP data –
Path – $HPD_ROOT/real/default/nv/client_id/ |
Data not available in Page Performance Overview and Page Performance Detail |
Possible Reasons #1 | Parsers may not be running |
Steps to Diagnose & Command Used |
Parser aggregates raw data available in DB and parse based on various constraints. Below is the command that gives information about all the parsers with last running date and time : If parsers last running date and time not matches the current date and time then we can assume parser is either stopped or stuck for which we have to view errors by following below steps – 1) Go to path – $HPD_ROOT/rum/ nv_aggregate_reporter_NVResourceAggregateTable.trace |
Commands to validate / Solution |
In CLI: We need to restart/start the parsers from that point of time where they are stopped. For Eg, Parser is stopped at Thu Apr 8 20:20:00 CST 1976 Command to convert human-readable time into timestamp format : Path for .lpt files : /home/cavisson/controller_name/hpd/rum/reports (These are hidden files) We have to update the 1st field of .lpt file as ‘0’ and 2nd field of .lpt file with the timestamp value which we have converted from date. Then type command to apply runtime changes – where nvresourceaggregateparser.aprof is parser name. |
No data in dashboard |
Possible Reasons #1 | HPD or Test is not running |
Steps to Diagnose & Command Used |
In NetVision HPD sends data to test. If any of the either process stops working then it will impact dashboard. HPD running or not can be validated using below command – /etc/init.d/hpd show – For work controller /etc/init.d/hpd_ControllerName show – For other controllers Test running or not can be validated using below command – nsu_show_netstorm |
Solution |
If HPD is not running then please follow diagnose steps and solution steps provided above to debug HPD not running issue .
There is a module in HPD which starts HPD. When we start HPD, test automatically gets started. Using below command –/etc/init.d/hpd start – For work controller/etc/init.d/hpd_ControllerName start – For other controllers” |
Possible Reasons #2 | HPD and test are not in sync. So, HPD is unable to send data to test. |
Steps to Diagnose & Command Used | HPD and Test should be sync in ideal condition. Command to check elapsed time of HPD Process: ps -o etime PPID of HPD Command to check elapsed time of nv.conf (Test of NV is run based on this scenario): ps -ef|grep nv.conf ps -o etime PID of nv.conf” |
Solution | If HPD and Test are not in sync then restart the HPD using below command –
/etc/init.d/hpd start – For work controller /etc/init.d/hpd_ControllerName start – For other controllers |
Possible Reasons #3 | No request from target application to NetVision Server as a result, no data are captured in NetVision. |
Steps to Diagnose & Command Used | For this possible reason how to diagnose steps are provided above. Please refer. |
Solution | For this possible diagnose, possible solutions are provided above. Please refer. |
Data is not coming in Business Process and some metrics |
Possible Reasons #1 | Check data is available in the RTG file or not |
Steps to Diagnose & Command Used |
For displaying data in BP and metrics, data needs to be available in RTG File.
So we need to check that data is there or not .You can validate the data through RestApi in below format :https://10.10.30.71:4433/DashboardServer/web/dev/checkGraphData?testRun=223344&startTime=MM/DD/YYYY%20HH:mm:ss&endTime=MM/DD/YYYY%20HH:mm:ss&groupId=10108&graphId=18&vectorName=AppTier%3EQAServer4&mode=0&strict=true&isLongFormat&token=Generatetokenfromgui where, 10.10.30.71:4433 : Machine IP: Http port or Https port |
Solution | If data is available in RTG file, check disk utilization. |
Possible Reasons #2 | Check disk Utilization |
Steps to Diagnose & Command Used |
In GUI, 1) Click on Advanced >> Application health option from Left Side Menu bar after logging into UI. |
Solution |
There is one keyword that defines the threshold value for multiple disks assigned for HPD in rum.conf file
Path of rum.conf file – /home/cavisson/Controller_name/hpd/rum/config/NV_MULTIDISK_OVERLOAD_THRESHOLD 98 97Where 98 is a threshold of disk that can be maximally utilized and in 97 is threshold user will receive a warning message. If Disks were utilized as per defined value in the keyword and due to this reason, HPD stops sending data to monitors and we were unable to see data in metrics. We have to clear the older data from disks and need to restart the HPD using below command – /etc/init.d/hpd restart (For work Controller) After HPD is restarted, now we will be getting proper data in NetVision metrics. |
Transaction by Store is not visible under UX Metric |
Possible Reasons #1 | Keywords may not be enable |
Steps to Diagnose & Command Used | In CLI :
Below Keywords may not be enabled in KeywordDefination.dat File – Keywords – |
Solution | In CLI : Enable below Keywords by setting 7th field as 1 in KeywordDefination.dat File – Path – $NS_WDIR/etc/
Keywords – NV_ENABLE_STATS_UA_MONITOR NV_ENABLE_UA_MONITOR_BY_STORE hpd and tomcat both processes need to restart for reflecting changes in GUI. |