Monitors Troubleshoot

General configuration mistakes on NS side

Possible Reasons #1 For running any monitor test, we need to do the basic configuration on appliance side.
Steps to Diagnose

1.) Lps port is not defined in scenario.

2.) Lps is running on different port, but in scenario Lps port is different.

3.) Lps port defined is correct in scenario and Lps file, but Lps port is not running.

4.) The IP in server.conf is not same as of /etc/hosts resulting not able to make the connection.

5.) Server must be present in Server.conf for the given topology with status 1(i.e active) and agent field 2 or 3.

6.) JSON keyword should be mentioned in Scenario.

7.) Topology keyword should be mentioned in Scenario.

8.) All configurations should be correct in Json file.

Commands to validate / Solution

LPS_SERVER 127.0.0.1 7896

netstat -natp|grep 7896

netstat -natp|grep 7896
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:7896 0.0.0.0:* LISTEN 19652/nsu_lps

ENABLE_AUTO_JSON_MONITOR 1 monitor.json

HIERARCHICAL_VIEW 1 Topo >

Screen1


General configuration mistakes on server side

Possible Reasons #1 For running any monitor test, we need to do the basic configurations on server side.
Steps to Diagnose

1.) cmon should be running in appliance/server side.

2.) Cmon.env is not present/correct on the server, resulting the un-updated topology (Autoscaling).

3.) Server must be present in Server.conf for the given topology with status 1(i.e active) and agent field 2 or 3 .

4.) Standalone for shell based monitor (run the standalone on the server with the programme name of monitor in bin directory)

5.) Standalone for java based monitor( run the standalone on server with the program name of monitor along with the mandatory options and the type)

6.) After running the standalone, check for error and debug logs on server side.

Screen2

Commands to validate / Solution

“nsu_server_admin -s 10.10.30.74 -c “”ps -ef””|grep cmon

/etc/init.d/cmon [start/show/restart]

CMON_OPTS=””-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=10.10.30.74″”
CAV_MON_AGENT_OPTS=””-p 7891″”
CMON_USER=cavisson
Controller=10.10.30.122:7892
TIER=Cavisson

==Standalone
nsu_server_admin -s 10.208.112.206 -c “”/opt/java/jdk8/bin/java -DCAV_MON_HOME=/opt/cavisson/monitors -cp /opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/jruby.jar:/opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/java-getopt-1.0.9.jar:/opt/cavisson/monitors/custom:/opt/cavisson/monitors/bin/CavMonAgent.jar:/opt/cavisson/monitors/lib/CmonLib.jar:/opt/cavisson/monitors/lib/gcviewer-1.29.jar:/opt/cavisson/monitors/lib/asm-all-4.0.jar:/opt/cavisson/monitors/lib/base64.jar:/opt/cavisson/monitors/lib/json_simple-1.1.jar:/opt/cavisson/monitors/lib/jedis-2.6.0.jar:/opt/cavisson/monitors/lib/mongo-java-driver-3.6.3.jar:/opt/cavisson/monitors/lib/bcprov-jdk15on-156.jar:/opt/cavisson/monitors/lib/bcpkix-jdk15on-1.48.jar:/opt/cavisson/monitors/lib/commons-net-3.0.1.jar:/opt/cavisson/monitors/lib/mysql-connector-java-5.1.37.jar:/opt/cavisson/monitors/lib/log4j-1.2.16.jar:/opt/cavisson/monitors/lib/slf4j-log4j12-1.5.8.jar:/opt/cavisson/monitors/lib/slf4j-api-1.5.8.jar:/opt/cavisson/monitors/lib/commonlib-1.0.jar:/opt/cavisson/monitors/lib/appengine-api-1.0-sdk-1 .9.54.jar:. com.cavisson.monitoring.haproxy.cm_haproxy_stats -U http://127.0.0.1:28080/haproxy?stats -X green-ui-proxy -s BACKEND -D””

==Path of cmon logs
/home/cavisson/monitors/logs/CavMonAgentError.log
/home/cavisson/monitors/logs/CavMonAgentDebug.log”


Access Log Data is not coming

Possible Reasons #1 Check lps is running or not. Check fields of logs.
Steps to Diagnose
1.) Firstly we have to check lps is running or not (Run LPS) and check the lps connectivity
2.) Secondly Logs are appending or not.3.) fields are correct on not (Json Configuration).Change the delimiter from space to tab in the Json.Screen3Screen4Screen5Screen6
Commands to validate
“LPS_SERVER 127.0.0.1 7896
ps -ef|grep lps/etc/init.d/lps [start/show/restart]””%2Fhome%2Fnetstorm%2FMonitor%2FSunil.logSunil.log”” -A CMON_Health>overall -o -F request:4,statusCode:4,respTime:2,respSize:6 -S “”space”” -u sec -h 0 -M 0 -I CMON_Health==log Sample
10.202.193.253 10.202.196.26:7001 – 2017-07-10 16:10:22 32.661 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7″”
10.202.193.252 10.202.196.26:7001 – 2017-07-10 16:10:22 32.693 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)””
10.202.193.252 10.202.196.26:7001 – 2017-07-10 16:10:22 32.424 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0″”
10.202.193.252 10.202.196.26:7001 – 2017-07-10 16:10:22 32.553 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko””
10.202.193.254 10.202.196.26:7001 – 2017-07-10 16:10:22 32.851 GET /cs/sale-event/mens-clothing.jsp 200 361896 “”Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5″”==>Confoguration of JSON
“”%2Fklslogs%2Fweblogic%2FKLS%2Fkls-wcsdelivery-03%2Faccess.log”” -A h121618vaps2018.att.kohls.com -o -F request:8,statusCode:9,respTime:6,respSize:10 -S “”space”” -u sec -h 0 -M 1 -X h121618vaps2018.att.kohls.com”
Solution If LPS is stopped, we have to start the LPS.

Pass the correct field value in the configuration of the JSON file.

Need to give correct logs path for Access Log.


System metrics data is not coming or Tier Servers are missing

Possible Reasons #1 Configuration mistakes in topology.

Field mapping of Tier.conf & Server.conf

Steps to Diagnose

1.) We have to check topology i.e. Tier.conf and Server.conf, all the config of both are correct. Servers are auto scaled properly.

To check whether the server is active or not,  we have to run a standalone of that particular monitor whose data is not coming.

Screen7

Screen8

Commands to validate Standalone for all the Shell base monitor.

nsu_server_admin -s 10.10.30.74 -c “”./cm_cpu_stats””

Solution If data is coming in standalone, there is some monitor configuration, so correct the configuration.


MySql data is not coming

Possible Reasons #1 Port Connectivity Issue.
Steps to Diagnose Check though Process Status command on which port MySQL is configured. Default port for MySQL is 3306
Commands to validate

“nsu_server_admin -s 10.10.30.74 -c “”ps -ef””|grep mysql

root@kohls-ddh-prod:/home/cavisson/work# nsu_server_admin -s 10.184.24.48 -c ‘netstat -natp’ | grep 3306
tcp 0 0 0.0.0.0:33061 0.0.0.0:* LISTEN –
tcp 0 0 10.184.24.48:44164 10.184.24.48:3306 TIME_WAIT –
tcp 0 0 10.184.24.48:43954 10.184.24.48:3306 TIME_WAIT –
tcp 0 0 10.184.24.48:44082 10.184.24.48:3306 TIME_WAIT -”

Solution Need to Set default port 3306 for MySQL and restart the application again.


Possible Reasons #2 Authentication issue: MySQL User and password is wrong or User don’t have the permissions to read else user don’t have admin privilege.

Screen9

Steps to Diagnose Run the standalone for the same

Verify the MySQL DB user/Password and all the configuration.
Verify the users privileges for reading the data

Commands to validate
Dummy Standalone for mysql:-
nsu_server_admin -s 10.207.8.4 -c “”/opt/java/jdk8/bin/java -DCAV_MON_HOME=/opt/cavisson/monitors -cp /opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/jruby.jar:/opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/java-getopt-1.0.9.jar:/opt/cavisson/monitors/custom:/opt/cavisson/monitors/bin/CavMonAgent.jar:/opt/cavisson/monitors/lib/CmonLib.jar:/opt/cavisson/monitors/lib/gcviewer-1.29.jar:/opt/cavisson/monitors/lib/asm-all-4.0.jar:/opt/cavisson/monitors/lib/base64.jar:/opt/cavisson/monitors/lib/json_simple-1.1.jar:/opt/cavisson/monitors/lib/jedis-2.6.0.jar:/opt/cavisson/monitors/lib/mongo-java-driver-3.4.2.jar:/opt/cavisson/monitors/lib/bcprov-jdk15on-156.jar:/opt/cavisson/monitors/lib/bcpkix-jdk15on-1.48.jar:/opt/cavisson/monitors/lib/commons-net-3.0.1.jar:/opt/cavisson/monitors/lib/mysql-connector-java-5.1.37.jar:/opt/cavisson/monitors/lib/log4j-1.2.16.jar:/opt/cavisson/monitors/lib/slf4j-log4j12-1.5.8.jar:/opt/cavisson/monitors/lib/slf4j-api-1.5.8.jar:/opt/cavisson/monitors/lib/commonlib-1.0.jar:. com.cavisson.monitoring.db.mysql.cm_mysql_stats -h 10.207.8.4 -p 3306 -U vpappuser -P cI6NZNHUIlmXcR4Z -t Activity -D””
Connecting to server 10.207.8.4 …
6427266560 2508745984 4889583 3468943 335745536 60599377920 993466 17929099264 7783 103544 103544Screen9
Solution Need to give full access/admin privileges to the mentioned user for MySQL db.


MongoDB data is not coming

Possible Reasons #1 Port Connectivity Issue.
Steps to Diagnose Check though Process Status command on which port MongoDB is configured. Default port for MongoDB is 27017.
Commands to validate

nsu_server_admin -s 10.10.30.74 -c “”ps -ef””|grep mongod

root@kohls-ddh-prod:/home/cavisson/work# nsu_server_admin -s 10.184.24.48 -c ‘netstat -natp’ | grep 27017
tcp 0 0 10.184.24.48:44164 10.184.24.48:27017 TIME_WAIT –
tcp 0 0 10.184.24.48:43954 10.184.24.48:27017 TIME_WAIT –
tcp 0 0 10.184.24.48:44082 10.184.24.48:27017 TIME_WAIT –

Solution Need to Set default port 27017 for MongoDB and restart the application again.


Possible Reasons #2 Authentication issue: MongoDB User and password is wrong or User don’t have the permissions to read the data in database or else user don’t have admin priviledge to access and read to the database.

Screen10

Steps to Diagnose Run the standalone for the same.

Verify the MySQL DB user/Password and Verify the all the configuration .
Verify the users privileges for reading the data

Commands to validate
Dummy Standalone for MongoDB:
root@ecom-pmdm-stress-nd-1:/home/cavisson/PMDM_Stress1# nsu_server_admin -s 10.207.8.217 -c “”””java -DCAV_MON_HOME=/opt/cavisson/monitors -cp /opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/*:/opt/cavisson/monitors/custom:/opt/cavisson/monitors/bin/CavMonAgent.jar:/opt/java/jdk8/lib/tools.jar:. com.cavisson.monitoring.mongodb.cm_mongo_db_monitor_stats -h 127.0.0.1 -p 27017 -U admin -P 2HD+L9eUGYJNXOBVfxyXlKJnqI/QY1w4IOKXC936c2s= -e -A admin -s server “”””
Connecting to server 10.207.8.217 …
0.986 1670 49530 0.986 1497 0 1712 144 0 28009 34986 0 0 1244589771 1845746339 10393841 100390343 3302174 594297 12865 4081614 0 0 0 99186 94869431343.808 5086291 109479416.676 1769453 48896378.326 3536456 16930.477 25572 1231.457 33317194 772760461 128 1 0 127 1640
Solution Need to give full access/admin privileges to the mentioned user for MongoDB.


Data is not coming for Generic DB monitor.

Possible Reasons #1 Entry of monitor is missing in standardmonitor.dat

GDF is not present

Query is not working

Some configuration mistakes in XML file

Steps to Diagnose Check the group ID is unique in created GDF.

Check the entry of Created monitor is present in standardmonitor.file with monitor json file is present or not.

Check in monitor’s json file group-index and column-index are same w.r.t. the output of the query.

Commands to validate Run the query in DB and check the output.
Solution Check monitor entry in standardmonitor.dat

Run the query on DB and check the output of the query.


Apigee data is not coming

Possible Reasons #1 Apigee URL is not working.
Steps to Diagnose Check the URL of Apigee is accessible or not.
Commands to validate

Dummy Standalone for Apigee:

java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_121/lib/tools.jar:. com.cavisson.monitoring.apigee.cm_apigee_stats -t https://kohlslle.login.apigee.com/oauth/token -o kohls-stress -e APIStress3 -u machine_kohls@apigee.com -p Apigee123$ -N TKMA2DR -T Vikash123 -m 10 -b edgecli -P edgeclisecret -a ApigeeTier -S ApigeeServer -D -s deployment
0:ApigeeTier_ApigeeServer_us-east1|50 50
1:ApigeeTier_ApigeeServer_us-central1|50 50

Solution Hit the mentioned URL on browser and check Apigee UI is opening or not.


Possible Reasons #2

Configuration mistakes. Wrong Apigee credentials.

Screen11

Screen12

Screen13

Steps to Diagnose Verify the given password are valid or not by running a standalone.
Commands to validate

Dummy Standalone for Apigee:

java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_121/lib/tools.jar:. com.cavisson.monitoring.apigee.cm_apigee_stats -t https://kohlslle.login.apigee.com/oauth/token -o kohls-stress -e APIStress3 -u machine_kohls@apigee.com -p Apigee123$ -N TKMA2DR -T Vikash123 -m 10 -b edgecli -P edgeclisecret -a ApigeeTier -S ApigeeServer -D -s deployment
0:ApigeeTier_ApigeeServer_us-east1|50 50
1:ApigeeTier_ApigeeServer_us-central1|50 50

Solution Open the apigee URL with the mentioned credentials


Possible Reasons #3 Apigee URL is accessible with Proxy or without proxy
Steps to Diagnose Check the URL of Apigee is accessible with proxy or without.
Commands to validate

Dummy Standalone for Apigee:

java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_121/lib/tools.jar:. com.cavisson.monitoring.apigee.cm_apigee_stats -t https://kohlslle.login.apigee.com/oauth/token -o kohls-stress -e APIStress3 -u machine_kohls@apigee.com -p Apigee123$ -N TKMA2DR -T Vikash123 -m 10 -b edgecli -P edgeclisecret -a ApigeeTier -S ApigeeServer -D -s deployment
0:ApigeeTier_ApigeeServer_us-east1|50 50
1:ApigeeTier_ApigeeServer_us-central1|50 50

Solution Open the apigee URL with the mentioned proxy


Jetty monitors metrics is not getting displayed

Screen14

Possible Reasons #1 Application Jar issue.
Steps to Diagnose 1.) Check all jars are present then need to check whether in mbeans data is present or not .

2.) If you find given mbeans then data will be displayed otherwise no data will be present for jetty monitors.

Commands to validate

Jetty Mbeans are not present on server side. We have dumped mbeans and found below stats

Mbeans required:

org.eclipse.jetty.util.thread:type=queuedthreadpool,id=*
org.eclipse.jetty.server.session:context=*

we can found mbean mbeans through below stand alone :-
cavisson@10:~/work/topology/KOS_Prod$ nsu_server_admin -s 10.208.112.193 -c ‘/opt/java/jdk8/bin/java -DCAV_MON_HOME=/opt/cavisson/monitors -cp /opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/jruby.jar:/opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/java-getopt-1.0.9.jar:/opt/cavisson/monitors/custom:/opt/cavisson/monitors/bin/CavMonAgent.jar:/opt/cavisson/monitors/lib/CmonLib.jar:/opt/cavisson/monitors/lib/gcviewer-1.29.jar:/opt/cavisson/monitors/lib/asm-all-4.0.jar:/opt/cavisson/monitors/lib/base64.jar:/opt/cavisson/monitors/lib/json_simple-1.1.jar:/opt/cavisson/monitors/lib/jedis-2.6.0.jar:/opt/cavisson/monitors/lib/mongo-java-driver-3.6.3.jar:/opt/cavisson/monitors/lib/bcprov-jdk15on-156.jar:/opt/cavisson/monitors/lib/bcpkix-jdk15on-1.48.jar:/opt/cavisson/monitors/lib/commons-net-3.0.1.jar:/opt/cavisson/monitors/lib/mysql-connector-java-5.1.37.jar:/opt/cavisson/monitors/lib/log4j-1.2.16.jar:/opt/cavisson/monitors/lib/slf4j-log4j12-1.5.8.jar:/opt/cavisson/monitors/lib/slf4j-api-1.5.8.jar:/opt/cavisson/monitors/lib/commonlib-1.0.jar:/opt/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:. com.cavisson.monitoring.generic_mbean.cm_mbean_server_stats -h 127.0.0.1 -p 9012 -U null -P null -M true -t all’ |grep -i jetty”

Solution The mandatory jars should be present on the server side and version of Jars should match.


Not getting data of process stats monitor of particular server

Possible Reasons #1 Auto scaling issue.
Steps to Diagnose After debugging the issue we came to know that in topology the last 5th field was zero (0 )which indicates that server is not autoscaled due which data was not showing.

Check the process is running or not on server.

Commands to validate

“Pass the correct data in cmon.env like Tier name and Server Ip.

CMON_OPTS=””-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=10.10.30.74″”
CAV_MON_AGENT_OPTS=””-p 7891″”
CMON_USER=cavisson
Controller=10.10.30.122:7892
TIER=Cavisson”

Solution For autoscaling, cmon.env should be correct and after passing the entry in cmon.env, restart of cmon is mandatory.

After restating the cmon changes will be reflected.


Unable to get data for ServerHealthExtendedStats Monitor

Possible Reasons #1 Wrong configuration in cmon.env. Wrong configuration in Server.conf file for the server
Steps to Diagnose
1.Check the configuration in cmon.env . Wrong entries was done in cmon.env
127.0.0.1:7891
After changing the configuration we are able to get data .1.In this case, first, check topology whether the server status is active or inactive2.Server is auto-scaled with which mode “0/1/2/3”Mode 0– Not scaledMode 1- scaled with bciMode 2:- Scaled with cmonMode 3:- Scaled with Bci and cmon
Commands to validate

“Standalone for serverhealtextended monitor

cavisson@CAV-KOL-GTW-NS-008:~/monitors/bin$ /home/cavisson/apps/jdk1.8.0_144/bin/java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/jruby.jar:/home/cavisson/monitors/lib/java-getopt-1.0.9.jar:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/monitors/lib/CmonLib.jar:/home/cavisson/monitors/lib/gcviewer-1.29.jar:/home/cavisson/monitors/lib/asm-all-4.0.jar:/home/cavisson/monitors/lib/base64.jar:/home/cavisson/monitors/lib/json_simple-1.1.jar:/home/cavisson/monitors/lib/jedis-2.6.0.jar:/home/cavisson/monitors/lib/mongo-java-driver-3.6.3.jar:/home/cavisson/monitors/lib/bcprov-jdk15on-156.jar:/home/cavisson/monitors/lib/bcpkix-jdk15on-1.48.jar:/home/cavisson/monitors/lib/commons-net-3.0.1.jar:/home/cavisson/monitors/lib/mysql-connector-java-5.1.37.jar:/home/cavisson/monitors/lib/log4j-1.2.16.jar:/home/cavisson/monitors/lib/slf4j-log4j12-1.5.8.jar:/home/cavisson/monitors/lib/slf4j-api-1.5.8.jar:/home/cavisson/monitors/lib/commonlib-1.0.jar:. com.cavisson.monitoring.server_health.cm_server_health_stats_v2 -f /home/cavisson/CMFC/mprof/CMFC_Stress/json/CMFC_AR.healthcheck -s server”

Solution Correct the cmon.env and restart cmon for solution.

Severs should be autoscaled through the cmon.


Cmon not responding

Possible Reasons #1 Default cmon port .i.e. 7891 is busy.
Steps to Diagnose Check through netstat command that cmon is running or not
Commands to validate netstat -nao|find 7891
Solution Need to start cmon at 7891 port

Screen23


Possible Reasons #2 cmon.env configuration is wrong
Steps to Diagnose Check cmon.env file ,no configuration error should be there
Commands to validate Configuration file should be like

Controller= <Controller Name>
Tier=<Tier Name>

Eg- >
Controller=10.10.30.74:7892
Tier=Windows_Tier

Solution Need to correct cmon.env configuration
Screen24


Possible Reasons #3 CMON is not running on default port
Steps to Diagnose If cmon is running on 7891 port, netstorm box will not send monitor request to cmon unless that port is mentioned in topology
Commands to validate Either restart cmon on default port or add port entry in topology
Eg–>
66.220.31.153:8001|DESKTOP-IHPSENF|6|6|1|NA|NA|NA|NA|C:\Program Files (x86)\opt\cavisson\wmonitors\|N|Windows|2|03/30/20 05:48:11|1586167806015|1586168572516|1″
Solution Either restart cmon on default port or add port entry in topology
Screen25


Monitor got failed

Possible Reasons #1 Monitor is not present in build
Steps to Diagnose Need to check in perfCounter.xml file monitor is present .
Commands to validate Sample entry of perfCounter.xml file for one monitor:

<vbs name =””wmi_msexchange_transport_storage_rest_api.vbs”” type=””””>
<Category name=””MSExchangeTransport Storage RESTAPI””>
<Counter name=””Storage REST Api Total Messages”” />
<Counter name=””Storage REST Api Recent Messages”” />
<Counter name=””Storage REST Api Total Requests”” />
<Counter name=””Storage REST Api Recent Requests”” />
</Category></vbs>

Solution If entry of monitor is not there we will not get data.We need to upgrade build which contain this feature
Screen22


Possible Reasons #2 Vectors are not present in server
Steps to Diagnose Need to check in perfmon and resource monitor that vector and data is present or not
Commands to validate Need to check in perfmon and resource monitor that vector and data is present or not
Solution Need to check Perfmon and resource monitor data
Screen21


Possible Reasons #3 Cmon is not running on windows server
Steps to Diagnose Check through netstat command that cmon is running or not
Commands to validate netstat -nao|find 7891
Solution Need to start cmon at 7891 port

Screen23

Possible Reasons #4 Cmon is not running on default port
Steps to Diagnose If cmon is running on 7891 port, netstorm box will not send monitor request to cmon unless that port is mentioned in topology.
Commands to validate Either restart cmon on default port or add port entry in topology
Eg–>
66.220.31.153:8001|DESKTOP-IHPSENF|6|6|1|NA|NA|NA|NA|C:\Program Files (x86)\opt\cavisson\wmonitors\|N|Windows|2|03/30/20 05:48:11|1586167806015|1586168572516|1
Solution Either restart cmon on default port or add port entry in topology
Screen25


Large number of vectors are getting added and deleted

Possible Reasons #1 Vectors are getting added and delete in Server frequently
Steps to Diagnose As our environment is dynamic, vector will get added and deleted as per the requirement and load. But in some cases, vector gets deleted and within a sample of few sec/minutes, same vector gets added again (mostly in case of network related monitors). Here every time monitor will delete it.

To avoid this, we have keyword : /maxVecPersistCount: which will not delete vector until given samples.

Commands to validate
We will add this keyword in json file .
{
“”app””: [
{
“”options””: “”ProcessorStats””,
“”lastUpdatedTime””: -1,
“”app-name””: “”default””
}
],
“”lastUpdatedTime””: 1576479094187,
“”std-mon-name””: “”/maxVecPersistCount:10″”,
“”enabled””: true
}
Solution “Need to add keyword maxVecPersistCount in option in json
Screen19


Possible Reasons #2 Client had applied monitor for all process but they do not need all processes
Steps to Diagnose There are few monitors which gives us process-wise data. By default if we apply those monitors, we get data of all the process.
So, to get data for specific process, we provide process names which we want to monitor .
We pass it in json under argument field.
Commands to validate We will add this keyword in json file .
{
“”app””: [
{
“”options””: “”ProcessorStats””,
“”lastUpdatedTime””: -1,
“”app-name””: “”default””
}
],
“”lastUpdatedTime””: 1576479094187,
“”std-mon-name””: “”/args: CavWMon,java /interval:10000 /trnum:$MON_TEST_RUN /parnum:$MON_PARTITION_IDX””,
“”enabled””: true
}
Solution Need to add process in under args option in json
Screen20


kubernetes IP monitor not running

Possible Reasons #1 Wrong token pass ni the token file. In the JSON , there is some configuration mistake.

Screen15

Screen16

Steps to Diagnose 1.) Run the standalone for the IP monitor, if not getting data is in standalone, then follow next step.

2.) Check the token is working or not through the Curl command.

Commands to validate
java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/apps/java/jdk1.8.0_144/lib/tools.jar:. com.cavisson.monitoring.kubernetes.monitor.cm_kubernetes_stats -h 35.193.68.194 -A /home/cavisson/work/mprof/token/kohls.json -s ip CURL COMMAND:- curl -H “Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRhc2hib2FyZC10b2tlbi02OGc0YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJjYX7J0Po-SyTsxIBLqwkYoaaUR7NZH_LNjdSl9g-Qsrb1XiCluz1CFYWBr2Yb4Tt7S0lLMr5FuVxWun4D_F7y6Nl8GpHh-q1IngnbMsHy_n34FqsMUPGtKbOLPtEhfllLy_HsofctjFcCLqnzrQS8NEMa3wpv8JWymDWgyCKGlxE0hxYe6Cid8QPZ1-wWndEcivrAMjgAs40OObp1Y8ctGJoWTi4fZochQ3_mKNiHFhbSDlkfGeXFXqVP859cGB3_m5BnYBajC3g” https://35.202.143.37/api/v1/pods -k
Solution


Data not coming in Kubernetes monitor

Possible Reasons #1 1. Kubernetes IP monitor is not running.

2.Tier-server mapping.

3. Check json Configuration.

Steps to Diagnose 1. Kubernetes IP monitor is not running checked by the above standalone.

2. Tier-server mapping where that cluster present in topology or not with that check autoscaling has been done and in “Server.conf” in last 5th –field and last field is enable or disable.

3. Check the configuration of json(json should me made from front end only), All mandatory options are there or not.

4. App-name is passed correct or not .

5. Check monitor.log, cmon.log, error and debug log some time connection is break or many other reasons or root cause we found in these logs only.

Commands to validate
java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/apps/java/jdk1.8.0_144/lib/tools.jar:. com.cavisson.monitoring.kubernetes.monitor.cm_kubernetes_stats -h 35.193.68.194 -A /home/cavisson/work/mprof/token/kohls.json -s ip
Solution


If user can face the same issue in OUTBOUND mode like data is not coming in Kubernetes Ip monitor or any Kubernetes monitor.

Possible Reasons #1 1. When the test is running in outbound mode firstly we have to check the keyword in scenario is correct or not.

2. Some configuration issue in cmon.env(auto-scaling)

3. NDC port is correct and NDC is running or not.

Steps to Diagnose 1. Open cmon.env and check NDC port is correct or not.

2. If port is correct NDC is running or not.

3. Run nsu_server_admin command.

Commands to validate “Outbound Keyword

ENABLE_OUTBOUND_CONNECTION

NDC_LPS_PORT 7876

ps -ef | grep ‘ndc’

nsu_server_admin -s MachineIP -H hostIP -N ndc port -c ‘options'”

Solution All configuration should be correct. mendatory keywords should be present in the scenario. connectvity with the NDC port.


Redis data is not coming

Screen17Screen18

Possible Reasons #1 Redis application is not running
Steps to Diagnose Check redis process is running or not
Commands to validate nsu_server_admin – s ip -c “ps -ef”|grep redis
Solution Need to Set default port 6379 for redisDB and restart the application again.


Possible Reasons #2 Port passed in json is wrong
Steps to Diagnose Check the redis port
Commands to validate nsu_server_admin – s ip -c “netsta -natp”|grep “redis-port”
Solution Need to give full acccess/adim privilages to the mentioned user for redisDB.


Possible Reasons #3 Redis db password is wrong
Steps to Diagnose Check the query on Redis db with same user. Wheather query is giving output or not
Commands to validate nsu_server_admin -s 127.0.0.1 -c “java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*

:/home/cavisson/monitors/custom

:/home/cavisson/monitors/bin/CavMonAgent.jar

:/home/cavisson/apps/jdk1.8.0_161/lib/tools.jar

:. com.cavisson.monitoring.redis.cm_redis_stats -h 127.0.0.1 -p 6379 -T stats-type”

Solution All the credential should be correct Which was pass in the Json file.

Possible Reasons #4 Server side cmon is not running
Steps to Diagnose Check user privileges.
Commands to validate
Solution


JavaGCJMX data is not coming

Possible Reasons #1 jmx port is not enabled for that java process
Steps to Diagnose Check the connectivity with jmx port using telnet
Commands to validate telnet <host> <port>
Solution Port is accessible from ND-Machine


Possible Reasons #2 jmx remote is not true
Steps to Diagnose Check jmx configuration of the process
Commands to validate JMX port configuration

{“-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=7991 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.10.30.124”}

Solution Need to make the connectivity with jmx port.


Possible Reasons #3 rmi server hostname is not provided
Steps to Diagnose check the pid if javaGCJMX is configured on java process PID


check PID file if monitor is configured on PId file

Commands to validate nsu_server_admin -s 127.0.0.1 -c “ps -ef”|grep


nsu_server_admin -s 127.0.0.1 -c “java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_161/lib/tools.jar:. com.cavisson.monitoring.java.gc_jmx.cm_java_gc_jmx -h 127.0.0.1 -p 7991 -j SunHotSpot”

Solution


Not getting data in DB Query

Possible Reasons #1 1. Authentication issue: db password is different and not in encoded form.

2. Connection closed and retry for particular monitor.

3. csv files dumped in tables but data go to diff column instead of csv column due to different in sequence of data and system time and server time differ from each other.

Steps to Diagnose Check json configuration for that monitors


Check monitor.log


Check in current partition that all csv files are present. eg..mssql_process_done.txtcheck csv files data dumped in table or not.

Commands to validate

path: NS_WDIR/TRNo./Partition/reports/csv


“neu_validate_db_data -n 1211 -p 2020xxxxxx -o ns -l
where n=> TRNO. and p => partitionpath: NS_WDIR/TRNo./Partition/ns_logs/ns_db_upload.copytrace
to check errors for which tables related errors are coming.

nsu_server_admin -s 10.10.220.2 -M “”mssql: -h 10.10.220.2 -p 1433
-A 0 -U hmisnr\\esdsadmin -P bmc@123 -t locks_stats””

Solution csv file should be updated timely.


Need to check CavWMON running on the server side.


Authentication should be correct and password must be encrypted form


Need that all configuration correct and connectivity with port.


Possible Reasons #2 In CavWmon logs, logs are updating for TRNo. or not


In cmon trace log file, monitors request are there or not

Steps to Diagnose “check sever side logs .
check through nsu_server_admin command data is coming or not”
Commands to validate “neu_validate_db_data -n 1211 -p 2020xxxxxx -o ns -l
where n=> TRNO. and p => partitionpath: NS_WDIR/TRNo./Partition/ns_logs/ns_db_upload.copytrace
to check errors for which tables related errors are coming.nsu_server_admin -s 10.10.220.2 -M “”mssql: -h 10.10.220.2 -p 1433
-A 0 -U hmisnr\\esdsadmin -P bmc@123 -t locks_stats””
Solution need to make the connectivity with jmx port.