NetVision Troubleshoot

Unable to capture sessions

Possible Reasons #1 Tag is not inserted properly in target application.
Steps to Diagnose & Command Used

For Capturing session in NetVision, the first step is we need to check is our Tag (NV_TAG), which is responsible to bring data over NV Server, is inserted or not on Target Application (Application which we are going to monitor).

It can be verified by searching nv_bootstrap.js in network panel of developer field of browser. If nv_bootstrap.js or startup tag is not visible then it can be assumed tag not inserted on target application.

Solution
Insert tag on target application in below format –
<script src=//NV Machine IP/Domain:HPD_SPORT/nv/Client_ID/nv_bootstrap.js type=””text/javascript””></script>HPD_SPORT can be found in /home/cavisson/controller_name/hpd/conf >> hpd.conf of NV ServerClient ID can be found in /home/cavisson/controller_name/hpd/rum/config >> rum.conf of NV Server”

Possible Reasons #2 HPD may not be running
Steps to Diagnose & Command Used Command to check hpd is running or not –

For work controller – /etc/init.d/hpd show

For other controller other than work – /etc/init.d/hpd_controller_name show

Solution

Search for error in hpd_error.0.log in path – /home/cavisson/controller_name/hpd/logs/ Error Possibility:

1) Port binding – Check for Port binding either in
hpd.conf – Path – /home/cavisson/controller_name/hpd/conf/

Below are the keywords for same –HPD_PORT, HPD_SPORT,

rum.conf – Path – /home/cavisson/controller_name/hpd/rum/config/

Below are the keywords for same –
NV_REPORTING_GUI_PORT, NV_REPORTING_SERVER_PORT”


2) Pending sick child – Any process (nr_db_upload,nv_memory_reporter) may not be running.

Can be checked using below command –

ps -ef | grep nv_memory_reporter – If not running can check error in nv_memory_reporter.trace
in path – /home/cavisson/controller_name/hpd/logs/ .

ps -ef | grep nr_db_upload – If not running can check error in nr_db_upload_error.log in path –
/home/cavisson/controller_name/hpd/rum/”


3) Postgres not running –
/etc/init.d/postgresql status – To check status if postgres running or not. If not running start the PostgresSQL server using the command –
/etc/init.d/postgresql start”


4) Invalid HPD License –

Need to deploy valid License for HPD
To check if license belonging to that machine or not –
Go to path – $HPD_ROOT/.license/
nsu_show_license -h”


The captured sessions not visible in Completed session window

Possible Reasons #1 Make sure tomcat process is making connection with database in order to show the data
Steps to Diagnose & Command Used

It can be checked using the command –

sudo netstat -natp |grep 5432 – Port of postgres can be viewed in postgresql.conf in path – /etc/postgresql/9.5/main

Always make sure that port of postgres must have connection with tomcat/java process to show data of completed sessions.

Commands to validate / Solution

If not making connection with tomcat then kill the process in which it is making connection using the command –

kill -9 (Process Id) and restart the tomcat.

Possible Reasons #2 Postgresql not running
Steps to Diagnose & Command Used Check postgres is running or not using below command –
ps -ef |grep postgres
or /etc/init.d/postgresql status
or Pg_isready
or we can also check using psql test cavisson
Solution
If it is not running check the errors from postgresql logs in path –
/home/cavisson/pgsql/data/pg_logand take action according to logs found.To start PostgresSQL Server, use below command –
/etc/init.d/postgresql start”

Possible Reasons #3 Data may be coming in Active Session and stored in buffer memory in .CSV format but not able to dump in database, as nr_db_upload process is not running.
Steps to Diagnose & Command Used It can be checked using command – ps -ef | grep nr_db_upload
Solution
If not running, check error in nr_db_upload_error.log in path –
/home/cavisson/controller_name/hpd/rum/
Error Possibilities
1) Failed to upload to db. Missing data for column
2) PostgresSQL not working
3) Garbage value received with data
4) Table not created to dump data


Some request of NetVision is getting failed while capturing sessions

Possible Reasons #1 Due to Cross Origin Resource Sharing (CORS) Policy
Steps to Diagnose & Command Used
NetVision JS Agent has three components: nv_bootstrap.js, cav_nv.js, config.js
Sometimes issue is faced that 3 component request are successfully captured but data capturing requests are getting failed such as – timing, pagedump, useraction, domwatcher, event logger etc.This may be due to CORS Policy which can be be viewed in Console tab of developer tools of browser while capturing a session.
Commands to validate / Solution
Check CorsFilter in web.xml file to handle multiple requests.
Path: webapps/netvision/WEB-INF/web.xml
All the properties like allowed methods and etc should be mentioned in this file.


NV has provided keyword for handling CORS Policy Error
Path: $HPD_ROOT/rum/config/rum.confKeyword –
NV_CORS_ORIGIN https://www-cstressmob3-prp.kohlsecommerce.com
Here www-cstressmob3-prp.kohlsecommerce.com is the target application which we are going to monitor.

Note: We need to restart the HPD Process to reflect the changes.
Command to restart HPD :
(i) /etc/init.d/hpd restart (For work Controller)
(ii) /etc/init.d/hpd_ControllerName restart (For other Controller)”


NetVision GUI is not opening

Possible Reasons #1 Tomcat is not running
Steps to Diagnose & Command Used To check if tomcat process running or not –

ps -ef | grep tomcat | grep -w Controller_name

Solution To Start tomcat , need to run given command from anywhere or from the working directory.

(i) /etc/init.d/tomcat start (For work Controller)
(ii) /etc/init.d/tomcat_ControllerName start (For another Controller)

Possible Reasons #2 Tomcat is running but got following exception in catalina.out Bind Exception Caused by: java.net.BindException: Address already in use (Bind failed) at java.net.PlainSocketImpl.socketBind(Native Method)
Steps to Diagnose & Command Used Error can be found in catalina.out in path –
$NS_WDIR/apps/apache_tomcat-7.0.91/logs/catalina.out
Solution To solve this error, we can either kill the service taking that port or can change our web server to run on another port.

The port conflicting with which service can be verified using below command –
netstat -natp | grep -w 8080 (For e.g. 8080 is the binding port)


Unable to Login the Product UI

Possible Reasons #1 Postgresql not running
Steps to Diagnose & Command Used Check postgres is running or not using below command –
ps -ef |grep postgres
or /etc/init.d/postgresql status
or Pg_isready
or we can also check using psql test cavisson
Solution If it is not running check the errors from postgresql logs in path –
/home/cavisson/pgsql/data/pg_logand take action according to logs found.To start PostgresSQL Server, use below command –
/etc/init.d/postgresql start

Possible Reasons #2 Authentication failure due to Given User is not present in database
Steps to Diagnose & Command Used Check in database i.e., access_control
psql access_control cavisson;
Solution If access to that user is not provide then Create a New User using admin in UI
1) Login with admin user.
2) Click Admin menu located in left panel of UI.
3) Then click on Access control in Admin Menu .
4) Click on ‘+’ icon on User to add a new user.


Replay feature is not working Properly

Possible Reasons #1 CSS Content get disturbed not visible properly in case of HTTPs Target Application
Steps to Diagnose & Command Used nv.replay.fetchURLFromRemoteServer = true keyword is responsible to fetch resources of target web application locally on NetVision Server.

It can be checked in file – config.ini in path – /home/cavisson/controller_name/webapps/sys/

Solution If this keyword is not mentioned then mention this keyword in config.ini in path – /home/cavisson/controller_name/webapps/sys/
Keyword –
nv.replay.fetchURLFromRemoteServer = true

Possible Reasons #2 CSS Content get disturbed and not visible properly in case of HTTP Target Application
Steps to Diagnose & Command Used It may be due to unable to access HTTP resources on HTTPS.
While playing replay of a session, it can be viewed in console column of developer tools and it will also show the Security Constraints that are blocking those resources.
Solution Comment Security constraint in web.xml in path – /home/cavisson/work/apps/apache-tomcat-7.0.91/conf that is blocking to allow Http resources and make below keyword in config.ini in path – /home/cavisson/controller_name/webapps/sys/
Keyword –
nv.replay.https=false

Possible Reasons #3 Getting mixed content error in console while playing replay.
Steps to Diagnose & Command Used While playing replay of a session it can be viewed in console column of developer tools. It may be due to some resource content loading over HTTP while some HTTPs content loading over HTTPs.
Solution
Mixed content error arise in that case when our Monitored application and Monitoring Application is served over different security protocols.
If our target application is served over HTTP and our monitoring application (NV Replay Feature) is serving over HTTPS then this problem might be occurred.
Solution to this problem is disabling this keyword from via making as false from config.ini(webapps/sys) file.
Keyword –
nv.replay.https=false
and moreover disable the security constraints arises in console of developer tool in browser.”

Possible Reasons #4 Pagedump available but showing error message – “Pagedump not available in DB loading”
Steps to Diagnose & Command Used

“We can cross check that pagedump for that particular session is actually captured or not by following below steps –

1) Search that session id from smart search option available on top right corner of Session GUI.
2) Open details of that session by double clicking on the session.
3) Open page details tab from the session details window.
4) Double click on a particular page you want to view pagedump captured or not.
5) After above step click on Pagedump tab and check if pagedump captured or not .”

Solution This problem might have occurred due to wrong handling of timestamp from nvIntermediate.jsp file.
Path of nvIntermediate.jsp file:
/work/webapps/netvision/reports”


Not getting Transaction

Possible Reasons #1 Check for HPD restart/partition Switching
Steps to Diagnose & Command Used
“May be due to restating of HPD and partition switching transaction got missed
How to check partition for a specific date:
Path : /work/hpd/rum
Command to be executed: Eg,
du -sh 20200408*
Where Format to be followed for time is YYYYMMDD*
Note: Whenever HPD gets restarted then a new partition gets created beyond defined configuration in rum.conf file.
Keyword:
PARTITION_SETTINGS 1 1 8h (In ideal condition partition switches on each 8 hour interval for a date)
Hours can be customised as per our requirement.”
Solution Check for any HPD restart or partition switching at the time of issue reported in the machine in which the transaction is captured. Rarely it happens while partition switching data loss may occur.


Page name not captured showing ‘others’

Possible Reasons #1 Pagename may not be configured
Steps to Diagnose & Command Used To verify this please follow below steps:
1) Open NV GUI
2) Click on config button on left side bar >> config
3) ConfigUI will open . Click on Pagename from left toggle button to see pagenames configured.
Solution
If pagename configured then see the next possible reason.
If pagename not configured then configure pagename by following below steps :In GUI1) Open NV GUI through Admin User.
2) Click on config button on left side bar >> config
3) ConfigUI will open . Click on Pagename from left toggle button .
4) Click the Add button from bottom of the page.
5) Follow the fields, please follow below for example,
Page Name: Login
Definition Method: URL pattern
Complete Url: Active
Url Pattern: kohls.login.jsp (Note: URL Pattern should be a Regex pattern that matches Test URL)
Test Url: https://www.kohls.com/myaccount/kohls_login.jsp
(Note: It is the URL of Page of the target application which user want to configure in NetVision)
Test it if matches then save it if it didn’t match there is some mistakes in the Regex pattern used.
In CLI
1) Go to path $HPD_ROOT/real/default/nv/client_id/
2) Open config.js
3) Add below configuration –
CONFIG: {
pageUrlValues: [
{s: 12, c: /(kohls.login.jsp)/, t: 1, cu: true} ]
}
where 12 is page id which can be found by typing by typing a query in postgres,
psql test cavisson
test=> select * from nvpagemetadatatable_clientid ;
c is the pattern which to be same as given in GUI, t is Definition method which is 1 for URL Pattern
cu is complete url which to be same as given in GUI .
4) After completing above changes save the file.
5) Then perform runtime changes for HPD by command – hpd_sp_rt_debug”

Possible Reasons #2 Pagename configured but not correctly
Steps to Diagnose & Command Used To verify configuration is correct/incorrect, please follow steps provided in above section.
Solution To verify configuration is correct/incorrect, please follow steps provided in above section.


Unable To Search Sessions In NVSPA

Possible Reasons #1 Data not dumped in database
Steps to Diagnose & Command Used Data may be coming in Active Session and stored in buffer memory in .CSV format but due to some error in nr_db_upload process it is not dumped in database.
Solution

In CLI :

Active session data can be searched by below steps –
1) Go to $HPD_ROOT/rum/
2) For example, if you want to search a session with Session id – 849887050766221462 captured on 09/04/2020. Then type below command –
grep -irl 849887050766221462 20200409*/0*/db/*.csv

It will list out the .csv files of a session. If data not available in active sessions then we will have to go to the next possible reason. If data is available in .csv but not in the database then follow the next step.

3) In database, that session can be found by typing a query in postgres. Below is the step for same –
psql test cavisson
test=> select * from rumsessiontable_clientid where sid= 849887050766221462 ;

4) If no data in database for that session id then there must be some error in nr_db_upload process for which .csv data is not dumped in database. Then go to that partition in which session id is captured, then open file – nr_db_upload.trace and search for that session id. It will list out the error.

Possible Reasons #2 Data is not captured in Active Session
Steps to Diagnose & Command Used Search for session in access_log in path $HPD_ROOT/logs/access_log/
For example: if you want to search a session with Session id – 849887050766221462 captured on 09/04/2020. Then type below command
grep -irl 849887050766221462 access_log_http_20200409* If no results found”
Solution It is failing due to network failure. test_rum call is not reaching to the NV Node. So these request must be failing either at browser end or LTM end. We have to see Stats of failure request at LTM.


Session not searchable through client IP

Possible Reasons #1 Session through that client ip is not captured (In Active Session)
Steps to Diagnose & Command Used Search for client ip in access_log in path $HPD_ROOT/logs/access_log/
For example if you want to search a client ip with client ip – 66.249.79.126
captured on 09/04/2020. Then type below command –
grep -irl 66.249.79.126 access_log_http_20200409* If no results found
Solution It is failing due to network failure. test_rum call is not reaching to the NV Node. So these request must be failing either at browser end or LTM end. We have to see Stats of failure request at LTM.

Possible Reasons #2 Session through that client ip is captured but not dumped in db (In Complete Session)
Steps to Diagnose & Command Used Data may be coming in Active Session for that client ip and stored in buffer memory in .CSV format but due to some error in nr_db_upload process it is not dumped in database.
Solution

In CLI:

Active session data for client ip can be searched by below steps –
1) Go to $HPD_ROOT/rum/
2) For example, if you want to search a session with Client IP – 66.249.79.126 captured on 09/04/2020. Then type below command –
grep -irl 66.249.79.126 20200409*/0*/db/*.csv

It will list out the .csv files of a session for that client ip. If data not available in active session then we will have to refer the above possible reason. If data is available in .csv but not in the database then follow the next step

3) In database, that session with that client ip can be found by typing a query in postgres. Below is the step for same –
psql test cavisson
test=> select * from rumsessiontable_clientid where clientip= 66.249.79.126;

4) If no data in database for that client ip then there must be some error in nr_db_upload process for which .csv data is not dumped in database. Then go to that partition in which session id is captured , then open file – nr_db_upload.trace and search for that session id. It will list out the error.”


HTTP Data not captured

Possible Reasons #1 Keyword may not be enable
Steps to Diagnose & Command Used In cav_nv.js or config.js below keyword to capture HTTP data is disable –
Path – $HPD_ROOT/real/default/nv/client_id/AjaxMonitor: {
enabled: false
},
Solution Enable below keyword in cav_nv.js or config.js to capture HTTP data –

Path – $HPD_ROOT/real/default/nv/client_id/
Keyword –
AjaxMonitor: {
enabled: true
},


Data not available in Page Performance Overview and Page Performance Detail

Possible Reasons #1 Parsers may not be running
Steps to Diagnose & Command Used

Parser aggregates raw data available in DB and parse based on various constraints.

Below is the command that gives information about all the parsers with last running date and time :
get_parser_info

If parsers last running date and time not matches the current date and time then we can assume parser is either stopped or stuck for which we have to view errors by following below steps –

1) Go to path – $HPD_ROOT/rum/
2) Pick the current partition using the command – cat .curPartition
3) Go to the current Partition. Here the logs of different parsers are generated –

nv_aggregate_reporter_NVResourceAggregateTable.trace
nv_aggregate_reporter_NVSessionAggregateTable.trace
nv_aggregate_reporter_NVPageAggregateTable.trace

Commands to validate / Solution

In CLI:

We need to restart/start the parsers from that point of time where they are stopped.

For Eg, Parser is stopped at Thu Apr 8 20:20:00 CST 1976

Command to convert human-readable time into timestamp format :
date -d’Apr 8 20:20:00 CST 1976′ +%s
Now we need to update the timestamp what we got from the above command in .lpt of a respective parser.

Path for .lpt files : /home/cavisson/controller_name/hpd/rum/reports (These are hidden files)

We have to update the 1st field of .lpt file as ‘0’ and 2nd field of .lpt file with the timestamp value which we have converted from date.

Then type command to apply runtime changes –
nv_rtc_aggregate nvresourceaggregateparser.aprof start/restart

where nvresourceaggregateparser.aprof is parser name.


No data in dashboard

Possible Reasons #1 HPD or Test is not running
Steps to Diagnose & Command Used

In NetVision HPD sends data to test. If any of the either process stops working then it will impact dashboard.

HPD running or not can be validated using below command –

/etc/init.d/hpd show – For work controller

/etc/init.d/hpd_ControllerName show – For other controllers

Test running or not can be validated using below command – nsu_show_netstorm

Solution
If HPD is not running then please follow diagnose steps and solution steps provided above to debug HPD not running issue .
There is a module in HPD which starts HPD. When we start HPD, test automatically gets started. Using below command –/etc/init.d/hpd start – For work controller/etc/init.d/hpd_ControllerName start – For other controllers”

Possible Reasons #2 HPD and test are not in sync. So, HPD is unable to send data to test.
Steps to Diagnose & Command Used HPD and Test should be sync in ideal condition.
Command to check elapsed time of HPD Process:
ps -o etime PPID of HPD
Command to check elapsed time of nv.conf (Test of NV is run based on this scenario):
ps -ef|grep nv.conf
ps -o etime PID of nv.conf”
Solution If HPD and Test are not in sync then restart the HPD using below command –

/etc/init.d/hpd start – For work controller

/etc/init.d/hpd_ControllerName start – For other controllers

Possible Reasons #3 No request from target application to NetVision Server as a result, no data are captured in NetVision.
Steps to Diagnose & Command Used For this possible reason how to diagnose steps are provided above. Please refer.
Solution For this possible diagnose, possible solutions are provided above. Please refer.


Data is not coming in Business Process and some metrics

Possible Reasons #1 Check data is available in the RTG file or not
Steps to Diagnose & Command Used
For displaying data in BP and metrics, data needs to be available in RTG File.
So we need to check that data is there or not .You can validate the data through RestApi in below format :https://10.10.30.71:4433/DashboardServer/web/dev/checkGraphData?testRun=223344&startTime=MM/DD/YYYY%20HH:mm:ss&endTime=MM/DD/YYYY%20HH:mm:ss&groupId=10108&graphId=18&vectorName=AppTier%3EQAServer4&mode=0&strict=true&isLongFormat&token=Generatetokenfromgui

where,

10.10.30.71:4433 : Machine IP: Http port or Https port
223344 : Running testrun number in the machine which can be checked through nsu_show_netstorm (CLI)
10108 : groupid
18 : graphid”

Solution If data is available in RTG file, check disk utilization.

Possible Reasons #2 Check disk Utilization
Steps to Diagnose & Command Used

In GUI,

1) Click on Advanced >> Application health option from Left Side Menu bar after logging into UI.
2) In Application health window, see the right bottom corner there is a widget – NetVision Disk Space Utilization (GB) from where we can also identify the disk space utilization of different disks.
or
In CLI,
Command to check disk utilization for NV Components –
nv_df ”

Solution
There is one keyword that defines the threshold value for multiple disks assigned for HPD in rum.conf file
Path of rum.conf file – /home/cavisson/Controller_name/hpd/rum/config/NV_MULTIDISK_OVERLOAD_THRESHOLD 98 97Where 98 is a threshold of disk that can be maximally utilized and in 97 is threshold user will receive a warning message.
If Disks were utilized as per defined value in the keyword and due to this reason, HPD stops sending data to monitors and we were unable to see data in metrics.
We have to clear the older data from disks and need to restart the HPD using below command –

/etc/init.d/hpd restart (For work Controller)
/etc/init.d/hpd_ControllerName restart (For other Controller)

After HPD is restarted, now we will be getting proper data in NetVision metrics.

Transaction by Store is not visible under UX Metric

Possible Reasons #1 Keywords may not be enable
Steps to Diagnose & Command Used In CLI :

Below Keywords may not be enabled in KeywordDefination.dat File –
Path – $NS_WDIR/etc/

Keywords –
NV_ENABLE_STATS_UA_MONITOR
NV_ENABLE_UA_MONITOR_BY_STORE

Solution In CLI : Enable below Keywords by setting 7th field as 1 in KeywordDefination.dat File – Path – $NS_WDIR/etc/

Keywords – NV_ENABLE_STATS_UA_MONITOR NV_ENABLE_UA_MONITOR_BY_STORE

hpd and tomcat both processes need to restart for reflecting changes in GUI.