Monitors Troubleshoot
General configuration mistakes on NS side |
| Possible Reasons #1 | For running any monitor test, we need to do the basic configuration on appliance side. |
| Steps to Diagnose | 1.) Lps port is not defined in scenario. 2.) Lps is running on different port, but in scenario Lps port is different. 3.) Lps port defined is correct in scenario and Lps file, but Lps port is not running. 4.) The IP in server.conf is not same as of /etc/hosts resulting not able to make the connection. 5.) Server must be present in Server.conf for the given topology with status 1(i.e active) and agent field 2 or 3. 6.) JSON keyword should be mentioned in Scenario. 7.) Topology keyword should be mentioned in Scenario. 8.) All configurations should be correct in Json file. |
| Commands to validate / Solution | LPS_SERVER 127.0.0.1 7896 netstat -natp|grep 7896 netstat -natp|grep 7896 ENABLE_AUTO_JSON_MONITOR 1 monitor.json HIERARCHICAL_VIEW 1 Topo >
|
General configuration mistakes on server side |
| Possible Reasons #1 | For running any monitor test, we need to do the basic configurations on server side. |
| Steps to Diagnose | 1.) cmon should be running in appliance/server side. 2.) Cmon.env is not present/correct on the server, resulting the un-updated topology (Autoscaling). 3.) Server must be present in Server.conf for the given topology with status 1(i.e active) and agent field 2 or 3 . 4.) Standalone for shell based monitor (run the standalone on the server with the programme name of monitor in bin directory) 5.) Standalone for java based monitor( run the standalone on server with the program name of monitor along with the mandatory options and the type) 6.) After running the standalone, check for error and debug logs on server side.
|
| Commands to validate / Solution | “nsu_server_admin -s 10.10.30.74 -c “”ps -ef””|grep cmon /etc/init.d/cmon [start/show/restart] CMON_OPTS=””-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=10.10.30.74″” ==Standalone ==Path of cmon logs |
Access Log Data is not coming |
| Possible Reasons #1 | Check lps is running or not. Check fields of logs. |
| Steps to Diagnose | 1.) Firstly we have to check lps is running or not (Run LPS) and check the lps connectivity 2.) Secondly Logs are appending or not.3.) fields are correct on not (Json Configuration).Change the delimiter from space to tab in the Json.     |
| Commands to validate | “LPS_SERVER 127.0.0.1 7896 ps -ef|grep lps/etc/init.d/lps [start/show/restart]””%2Fhome%2Fnetstorm%2FMonitor%2FSunil.logSunil.log”” -A CMON_Health>overall -o -F request:4,statusCode:4,respTime:2,respSize:6 -S “”space”” -u sec -h 0 -M 0 -I CMON_Health==log Sample 10.202.193.253 10.202.196.26:7001 – 2017-07-10 16:10:22 32.661 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7″” 10.202.193.252 10.202.196.26:7001 – 2017-07-10 16:10:22 32.693 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)”” 10.202.193.252 10.202.196.26:7001 – 2017-07-10 16:10:22 32.424 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0″” 10.202.193.252 10.202.196.26:7001 – 2017-07-10 16:10:22 32.553 GET /cs/wcs-internal/tracking_analytics.jsp 200 3847 “”Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko”” 10.202.193.254 10.202.196.26:7001 – 2017-07-10 16:10:22 32.851 GET /cs/sale-event/mens-clothing.jsp 200 361896 “”Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5″”==>Confoguration of JSON “”%2Fklslogs%2Fweblogic%2FKLS%2Fkls-wcsdelivery-03%2Faccess.log”” -A h121618vaps2018.att.kohls.com -o -F request:8,statusCode:9,respTime:6,respSize:10 -S “”space”” -u sec -h 0 -M 1 -X h121618vaps2018.att.kohls.com” |
| Solution | If LPS is stopped, we have to start the LPS. Pass the correct field value in the configuration of the JSON file. Need to give correct logs path for Access Log. |
System metrics data is not coming or Tier Servers are missing |
| Possible Reasons #1 | Configuration mistakes in topology. Field mapping of Tier.conf & Server.conf |
| Steps to Diagnose | 1.) We have to check topology i.e. Tier.conf and Server.conf, all the config of both are correct. Servers are auto scaled properly. To check whether the server is active or not, we have to run a standalone of that particular monitor whose data is not coming.
|
| Commands to validate | Standalone for all the Shell base monitor. nsu_server_admin -s 10.10.30.74 -c “”./cm_cpu_stats”” |
| Solution | If data is coming in standalone, there is some monitor configuration, so correct the configuration. |
MySql data is not coming |
| Possible Reasons #1 | Port Connectivity Issue. |
| Steps to Diagnose | Check though Process Status command on which port MySQL is configured. Default port for MySQL is 3306 |
| Commands to validate | “nsu_server_admin -s 10.10.30.74 -c “”ps -ef””|grep mysql root@kohls-ddh-prod:/home/cavisson/work# nsu_server_admin -s 10.184.24.48 -c ‘netstat -natp’ | grep 3306 |
| Solution | Need to Set default port 3306 for MySQL and restart the application again. |
| Possible Reasons #2 | Authentication issue: MySQL User and password is wrong or User don’t have the permissions to read else user don’t have admin privilege.
|
| Steps to Diagnose | Run the standalone for the same Verify the MySQL DB user/Password and all the configuration. |
| Commands to validate | Dummy Standalone for mysql:- nsu_server_admin -s 10.207.8.4 -c “”/opt/java/jdk8/bin/java -DCAV_MON_HOME=/opt/cavisson/monitors -cp /opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/jruby.jar:/opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/java-getopt-1.0.9.jar:/opt/cavisson/monitors/custom:/opt/cavisson/monitors/bin/CavMonAgent.jar:/opt/cavisson/monitors/lib/CmonLib.jar:/opt/cavisson/monitors/lib/gcviewer-1.29.jar:/opt/cavisson/monitors/lib/asm-all-4.0.jar:/opt/cavisson/monitors/lib/base64.jar:/opt/cavisson/monitors/lib/json_simple-1.1.jar:/opt/cavisson/monitors/lib/jedis-2.6.0.jar:/opt/cavisson/monitors/lib/mongo-java-driver-3.4.2.jar:/opt/cavisson/monitors/lib/bcprov-jdk15on-156.jar:/opt/cavisson/monitors/lib/bcpkix-jdk15on-1.48.jar:/opt/cavisson/monitors/lib/commons-net-3.0.1.jar:/opt/cavisson/monitors/lib/mysql-connector-java-5.1.37.jar:/opt/cavisson/monitors/lib/log4j-1.2.16.jar:/opt/cavisson/monitors/lib/slf4j-log4j12-1.5.8.jar:/opt/cavisson/monitors/lib/slf4j-api-1.5.8.jar:/opt/cavisson/monitors/lib/commonlib-1.0.jar:. com.cavisson.monitoring.db.mysql.cm_mysql_stats -h 10.207.8.4 -p 3306 -U vpappuser -P cI6NZNHUIlmXcR4Z -t Activity -D”” Connecting to server 10.207.8.4 … 6427266560 2508745984 4889583 3468943 335745536 60599377920 993466 17929099264 7783 103544 103544  |
| Solution | Need to give full access/admin privileges to the mentioned user for MySQL db. |
MongoDB data is not coming |
| Possible Reasons #1 | Port Connectivity Issue. |
| Steps to Diagnose | Check though Process Status command on which port MongoDB is configured. Default port for MongoDB is 27017. |
| Commands to validate | nsu_server_admin -s 10.10.30.74 -c “”ps -ef””|grep mongod root@kohls-ddh-prod:/home/cavisson/work# nsu_server_admin -s 10.184.24.48 -c ‘netstat -natp’ | grep 27017 |
| Solution | Need to Set default port 27017 for MongoDB and restart the application again. |
| Possible Reasons #2 | Authentication issue: MongoDB User and password is wrong or User don’t have the permissions to read the data in database or else user don’t have admin priviledge to access and read to the database.
|
| Steps to Diagnose | Run the standalone for the same. Verify the MySQL DB user/Password and Verify the all the configuration . |
| Commands to validate | Dummy Standalone for MongoDB: root@ecom-pmdm-stress-nd-1:/home/cavisson/PMDM_Stress1# nsu_server_admin -s 10.207.8.217 -c “”””java -DCAV_MON_HOME=/opt/cavisson/monitors -cp /opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/jruby.jar:.:/opt/cavisson/monitors/lib/*:/opt/cavisson/monitors/custom:/opt/cavisson/monitors/bin/CavMonAgent.jar:/opt/java/jdk8/lib/tools.jar:. com.cavisson.monitoring.mongodb.cm_mongo_db_monitor_stats -h 127.0.0.1 -p 27017 -U admin -P 2HD+L9eUGYJNXOBVfxyXlKJnqI/QY1w4IOKXC936c2s= -e -A admin -s server “””” Connecting to server 10.207.8.217 … 0.986 1670 49530 0.986 1497 0 1712 144 0 28009 34986 0 0 1244589771 1845746339 10393841 100390343 3302174 594297 12865 4081614 0 0 0 99186 94869431343.808 5086291 109479416.676 1769453 48896378.326 3536456 16930.477 25572 1231.457 33317194 772760461 128 1 0 127 1640 |
| Solution | Need to give full access/admin privileges to the mentioned user for MongoDB. |
Data is not coming for Generic DB monitor. |
| Possible Reasons #1 | Entry of monitor is missing in standardmonitor.dat GDF is not present Query is not working Some configuration mistakes in XML file |
| Steps to Diagnose | Check the group ID is unique in created GDF. Check the entry of Created monitor is present in standardmonitor.file with monitor json file is present or not. Check in monitor’s json file group-index and column-index are same w.r.t. the output of the query. |
| Commands to validate | Run the query in DB and check the output. |
| Solution | Check monitor entry in standardmonitor.dat Run the query on DB and check the output of the query. |
Apigee data is not coming |
| Possible Reasons #1 | Apigee URL is not working. |
| Steps to Diagnose | Check the URL of Apigee is accessible or not. |
| Commands to validate | Dummy Standalone for Apigee: java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_121/lib/tools.jar:. com.cavisson.monitoring.apigee.cm_apigee_stats -t https://kohlslle.login.apigee.com/oauth/token -o kohls-stress -e APIStress3 -u machine_kohls@apigee.com -p Apigee123$ -N TKMA2DR -T Vikash123 -m 10 -b edgecli -P edgeclisecret -a ApigeeTier -S ApigeeServer -D -s deployment |
| Solution | Hit the mentioned URL on browser and check Apigee UI is opening or not. |
| Possible Reasons #2 | Configuration mistakes. Wrong Apigee credentials.
|
| Steps to Diagnose | Verify the given password are valid or not by running a standalone. |
| Commands to validate | Dummy Standalone for Apigee: java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_121/lib/tools.jar:. com.cavisson.monitoring.apigee.cm_apigee_stats -t https://kohlslle.login.apigee.com/oauth/token -o kohls-stress -e APIStress3 -u machine_kohls@apigee.com -p Apigee123$ -N TKMA2DR -T Vikash123 -m 10 -b edgecli -P edgeclisecret -a ApigeeTier -S ApigeeServer -D -s deployment |
| Solution | Open the apigee URL with the mentioned credentials |
| Possible Reasons #3 | Apigee URL is accessible with Proxy or without proxy |
| Steps to Diagnose | Check the URL of Apigee is accessible with proxy or without. |
| Commands to validate | Dummy Standalone for Apigee: java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_121/lib/tools.jar:. com.cavisson.monitoring.apigee.cm_apigee_stats -t https://kohlslle.login.apigee.com/oauth/token -o kohls-stress -e APIStress3 -u machine_kohls@apigee.com -p Apigee123$ -N TKMA2DR -T Vikash123 -m 10 -b edgecli -P edgeclisecret -a ApigeeTier -S ApigeeServer -D -s deployment |
| Solution | Open the apigee URL with the mentioned proxy |
Jetty monitors metrics is not getting displayed
|
| Possible Reasons #1 | Application Jar issue. |
| Steps to Diagnose | 1.) Check all jars are present then need to check whether in mbeans data is present or not . 2.) If you find given mbeans then data will be displayed otherwise no data will be present for jetty monitors. |
| Commands to validate | Jetty Mbeans are not present on server side. We have dumped mbeans and found below stats Mbeans required: org.eclipse.jetty.util.thread:type=queuedthreadpool,id=* we can found mbean mbeans through below stand alone :- |
| Solution | The mandatory jars should be present on the server side and version of Jars should match. |
Not getting data of process stats monitor of particular server |
| Possible Reasons #1 | Auto scaling issue. |
| Steps to Diagnose | After debugging the issue we came to know that in topology the last 5th field was zero (0 )which indicates that server is not autoscaled due which data was not showing. Check the process is running or not on server. |
| Commands to validate | “Pass the correct data in cmon.env like Tier name and Server Ip. CMON_OPTS=””-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=10.10.30.74″” |
| Solution | For autoscaling, cmon.env should be correct and after passing the entry in cmon.env, restart of cmon is mandatory. After restating the cmon changes will be reflected. |
Unable to get data for ServerHealthExtendedStats Monitor |
| Possible Reasons #1 | Wrong configuration in cmon.env. Wrong configuration in Server.conf file for the server |
| Steps to Diagnose | 1.Check the configuration in cmon.env . Wrong entries was done in cmon.env 127.0.0.1:7891 After changing the configuration we are able to get data .1.In this case, first, check topology whether the server status is active or inactive2.Server is auto-scaled with which mode “0/1/2/3”Mode 0– Not scaledMode 1- scaled with bciMode 2:- Scaled with cmonMode 3:- Scaled with Bci and cmon |
| Commands to validate | “Standalone for serverhealtextended monitor cavisson@CAV-KOL-GTW-NS-008:~/monitors/bin$ /home/cavisson/apps/jdk1.8.0_144/bin/java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/jruby.jar:/home/cavisson/monitors/lib/java-getopt-1.0.9.jar:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/monitors/lib/CmonLib.jar:/home/cavisson/monitors/lib/gcviewer-1.29.jar:/home/cavisson/monitors/lib/asm-all-4.0.jar:/home/cavisson/monitors/lib/base64.jar:/home/cavisson/monitors/lib/json_simple-1.1.jar:/home/cavisson/monitors/lib/jedis-2.6.0.jar:/home/cavisson/monitors/lib/mongo-java-driver-3.6.3.jar:/home/cavisson/monitors/lib/bcprov-jdk15on-156.jar:/home/cavisson/monitors/lib/bcpkix-jdk15on-1.48.jar:/home/cavisson/monitors/lib/commons-net-3.0.1.jar:/home/cavisson/monitors/lib/mysql-connector-java-5.1.37.jar:/home/cavisson/monitors/lib/log4j-1.2.16.jar:/home/cavisson/monitors/lib/slf4j-log4j12-1.5.8.jar:/home/cavisson/monitors/lib/slf4j-api-1.5.8.jar:/home/cavisson/monitors/lib/commonlib-1.0.jar:. com.cavisson.monitoring.server_health.cm_server_health_stats_v2 -f /home/cavisson/CMFC/mprof/CMFC_Stress/json/CMFC_AR.healthcheck -s server” |
| Solution | Correct the cmon.env and restart cmon for solution. Severs should be autoscaled through the cmon. |
Cmon not responding |
| Possible Reasons #1 | Default cmon port .i.e. 7891 is busy. |
| Steps to Diagnose | Check through netstat command that cmon is running or not |
| Commands to validate | netstat -nao|find 7891 |
| Solution | Need to start cmon at 7891 port
|
| Possible Reasons #2 | cmon.env configuration is wrong |
| Steps to Diagnose | Check cmon.env file ,no configuration error should be there |
| Commands to validate | Configuration file should be like Controller= <Controller Name> Eg- > |
| Solution | Need to correct cmon.env configuration |
| Possible Reasons #3 | CMON is not running on default port |
| Steps to Diagnose | If cmon is running on 7891 port, netstorm box will not send monitor request to cmon unless that port is mentioned in topology |
| Commands to validate | Either restart cmon on default port or add port entry in topology Eg–> 66.220.31.153:8001|DESKTOP-IHPSENF|6|6|1|NA|NA|NA|NA|C:\Program Files (x86)\opt\cavisson\wmonitors\|N|Windows|2|03/30/20 05:48:11|1586167806015|1586168572516|1″ |
| Solution | Either restart cmon on default port or add port entry in topology |
Monitor got failed |
| Possible Reasons #1 | Monitor is not present in build |
| Steps to Diagnose | Need to check in perfCounter.xml file monitor is present . |
| Commands to validate | Sample entry of perfCounter.xml file for one monitor: <vbs name =””wmi_msexchange_transport_storage_rest_api.vbs”” type=””””> |
| Solution | If entry of monitor is not there we will not get data.We need to upgrade build which contain this feature |
| Possible Reasons #2 | Vectors are not present in server |
| Steps to Diagnose | Need to check in perfmon and resource monitor that vector and data is present or not |
| Commands to validate | Need to check in perfmon and resource monitor that vector and data is present or not |
| Solution | Need to check Perfmon and resource monitor data |
| Possible Reasons #3 | Cmon is not running on windows server |
| Steps to Diagnose | Check through netstat command that cmon is running or not |
| Commands to validate | netstat -nao|find 7891 |
| Solution | Need to start cmon at 7891 port
|
| Possible Reasons #4 | Cmon is not running on default port |
| Steps to Diagnose | If cmon is running on 7891 port, netstorm box will not send monitor request to cmon unless that port is mentioned in topology. |
| Commands to validate | Either restart cmon on default port or add port entry in topology Eg–> 66.220.31.153:8001|DESKTOP-IHPSENF|6|6|1|NA|NA|NA|NA|C:\Program Files (x86)\opt\cavisson\wmonitors\|N|Windows|2|03/30/20 05:48:11|1586167806015|1586168572516|1 |
| Solution | Either restart cmon on default port or add port entry in topology |
Large number of vectors are getting added and deleted |
| Possible Reasons #1 | Vectors are getting added and delete in Server frequently |
| Steps to Diagnose | As our environment is dynamic, vector will get added and deleted as per the requirement and load. But in some cases, vector gets deleted and within a sample of few sec/minutes, same vector gets added again (mostly in case of network related monitors). Here every time monitor will delete it. To avoid this, we have keyword : /maxVecPersistCount: which will not delete vector until given samples. |
| Commands to validate | We will add this keyword in json file . { “”app””: [ { “”options””: “”ProcessorStats””, “”lastUpdatedTime””: -1, “”app-name””: “”default”” } ], “”lastUpdatedTime””: 1576479094187, “”std-mon-name””: “”/maxVecPersistCount:10″”, “”enabled””: true } |
| Solution | “Need to add keyword maxVecPersistCount in option in json |
| Possible Reasons #2 | Client had applied monitor for all process but they do not need all processes |
| Steps to Diagnose | There are few monitors which gives us process-wise data. By default if we apply those monitors, we get data of all the process. So, to get data for specific process, we provide process names which we want to monitor . We pass it in json under argument field. |
| Commands to validate | We will add this keyword in json file . { “”app””: [ { “”options””: “”ProcessorStats””, “”lastUpdatedTime””: -1, “”app-name””: “”default”” } ], “”lastUpdatedTime””: 1576479094187, “”std-mon-name””: “”/args: CavWMon,java /interval:10000 /trnum:$MON_TEST_RUN /parnum:$MON_PARTITION_IDX””, “”enabled””: true } |
| Solution | Need to add process in under args option in json |
kubernetes IP monitor not running |
| Possible Reasons #1 | Wrong token pass ni the token file. In the JSON , there is some configuration mistake.
|
| Steps to Diagnose | 1.) Run the standalone for the IP monitor, if not getting data is in standalone, then follow next step. 2.) Check the token is working or not through the Curl command. |
| Commands to validate | java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/apps/java/jdk1.8.0_144/lib/tools.jar:. com.cavisson.monitoring.kubernetes.monitor.cm_kubernetes_stats -h 35.193.68.194 -A /home/cavisson/work/mprof/token/kohls.json -s ip CURL COMMAND:- curl -H “Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRhc2hib2FyZC10b2tlbi02OGc0YiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJjYX7J0Po-SyTsxIBLqwkYoaaUR7NZH_LNjdSl9g-Qsrb1XiCluz1CFYWBr2Yb4Tt7S0lLMr5FuVxWun4D_F7y6Nl8GpHh-q1IngnbMsHy_n34FqsMUPGtKbOLPtEhfllLy_HsofctjFcCLqnzrQS8NEMa3wpv8JWymDWgyCKGlxE0hxYe6Cid8QPZ1-wWndEcivrAMjgAs40OObp1Y8ctGJoWTi4fZochQ3_mKNiHFhbSDlkfGeXFXqVP859cGB3_m5BnYBajC3g” https://35.202.143.37/api/v1/pods -k |
| Solution | — |
Data not coming in Kubernetes monitor |
| Possible Reasons #1 | 1. Kubernetes IP monitor is not running. 2.Tier-server mapping. 3. Check json Configuration. |
| Steps to Diagnose | 1. Kubernetes IP monitor is not running checked by the above standalone. 2. Tier-server mapping where that cluster present in topology or not with that check autoscaling has been done and in “Server.conf” in last 5th –field and last field is enable or disable. 3. Check the configuration of json(json should me made from front end only), All mandatory options are there or not. 4. App-name is passed correct or not . 5. Check monitor.log, cmon.log, error and debug log some time connection is break or many other reasons or root cause we found in these logs only. |
| Commands to validate | java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/GCPAuth.jar:/home/cavisson/monitors/lib/google-http-client-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-appengine-1.22.0.jar:/home/cavisson/monitors/lib/google-http-client-jackson2-1.22.0.jar:/home/cavisson/monitors/lib/jackson-annotations-2.7.5.jar:/home/cavisson/monitors/lib/jackson-core-2.7.5.jar:/home/cavisson/monitors/lib/jackson-databind-2.7.5.jar:/home/cavisson/monitors/lib/appengine-api-1.0-sdk-1.9.54.jar:/home/cavisson/monitors/lib/sqljdbc41.jar:/usr/share/java/google-collections.jar:/home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/apps/java/jdk1.8.0_144/lib/tools.jar:. com.cavisson.monitoring.kubernetes.monitor.cm_kubernetes_stats -h 35.193.68.194 -A /home/cavisson/work/mprof/token/kohls.json -s ip |
| Solution | — |
If user can face the same issue in OUTBOUND mode like data is not coming in Kubernetes Ip monitor or any Kubernetes monitor. |
| Possible Reasons #1 | 1. When the test is running in outbound mode firstly we have to check the keyword in scenario is correct or not. 2. Some configuration issue in cmon.env(auto-scaling) 3. NDC port is correct and NDC is running or not. |
| Steps to Diagnose | 1. Open cmon.env and check NDC port is correct or not. 2. If port is correct NDC is running or not. 3. Run nsu_server_admin command. |
| Commands to validate | “Outbound Keyword ENABLE_OUTBOUND_CONNECTION NDC_LPS_PORT 7876 ps -ef | grep ‘ndc’ nsu_server_admin -s MachineIP -H hostIP -N ndc port -c ‘options'” |
| Solution | All configuration should be correct. mendatory keywords should be present in the scenario. connectvity with the NDC port. |
Redis data is not coming
|
| Possible Reasons #1 | Redis application is not running |
| Steps to Diagnose | Check redis process is running or not |
| Commands to validate | nsu_server_admin – s ip -c “ps -ef”|grep redis |
| Solution | Need to Set default port 6379 for redisDB and restart the application again. |
| Possible Reasons #2 | Port passed in json is wrong |
| Steps to Diagnose | Check the redis port |
| Commands to validate | nsu_server_admin – s ip -c “netsta -natp”|grep “redis-port” |
| Solution | Need to give full acccess/adim privilages to the mentioned user for redisDB. |
| Possible Reasons #3 | Redis db password is wrong |
| Steps to Diagnose | Check the query on Redis db with same user. Wheather query is giving output or not |
| Commands to validate | nsu_server_admin -s 127.0.0.1 -c “java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/* :/home/cavisson/monitors/custom :/home/cavisson/monitors/bin/CavMonAgent.jar :/home/cavisson/apps/jdk1.8.0_161/lib/tools.jar :. com.cavisson.monitoring.redis.cm_redis_stats -h 127.0.0.1 -p 6379 -T stats-type” |
| Solution | All the credential should be correct Which was pass in the Json file. |
| Possible Reasons #4 | Server side cmon is not running |
| Steps to Diagnose | Check user privileges. |
| Commands to validate | — |
| Solution | — |
JavaGCJMX data is not coming |
| Possible Reasons #1 | jmx port is not enabled for that java process |
| Steps to Diagnose | Check the connectivity with jmx port using telnet |
| Commands to validate | telnet <host> <port> |
| Solution | Port is accessible from ND-Machine |
| Possible Reasons #2 | jmx remote is not true |
| Steps to Diagnose | Check jmx configuration of the process |
| Commands to validate | JMX port configuration {“-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=7991 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.10.30.124”} |
| Solution | Need to make the connectivity with jmx port. |
| Possible Reasons #3 | rmi server hostname is not provided |
| Steps to Diagnose | check the pid if javaGCJMX is configured on java process PID check PID file if monitor is configured on PId file |
| Commands to validate | nsu_server_admin -s 127.0.0.1 -c “ps -ef”|grep nsu_server_admin -s 127.0.0.1 -c “java -DCAV_MON_HOME=/home/cavisson/monitors -cp /home/cavisson/monitors/lib/*:/home/cavisson/monitors/custom:/home/cavisson/monitors/bin/CavMonAgent.jar:/home/cavisson/apps/jdk1.8.0_161/lib/tools.jar:. com.cavisson.monitoring.java.gc_jmx.cm_java_gc_jmx -h 127.0.0.1 -p 7991 -j SunHotSpot” |
| Solution | — |
Not getting data in DB Query |
| Possible Reasons #1 | 1. Authentication issue: db password is different and not in encoded form. 2. Connection closed and retry for particular monitor. 3. csv files dumped in tables but data go to diff column instead of csv column due to different in sequence of data and system time and server time differ from each other. |
| Steps to Diagnose | Check json configuration for that monitors Check monitor.log Check in current partition that all csv files are present. eg..mssql_process_done.txtcheck csv files data dumped in table or not. |
| Commands to validate | path: NS_WDIR/TRNo./Partition/reports/csv “neu_validate_db_data -n 1211 -p 2020xxxxxx -o ns -l nsu_server_admin -s 10.10.220.2 -M “”mssql: -h 10.10.220.2 -p 1433 |
| Solution | csv file should be updated timely. Need to check CavWMON running on the server side. Authentication should be correct and password must be encrypted form Need that all configuration correct and connectivity with port. |
| Possible Reasons #2 | In CavWmon logs, logs are updating for TRNo. or not In cmon trace log file, monitors request are there or not |
| Steps to Diagnose | “check sever side logs . check through nsu_server_admin command data is coming or not” |
| Commands to validate | “neu_validate_db_data -n 1211 -p 2020xxxxxx -o ns -l where n=> TRNO. and p => partitionpath: NS_WDIR/TRNo./Partition/ns_logs/ns_db_upload.copytrace to check errors for which tables related errors are coming.nsu_server_admin -s 10.10.220.2 -M “”mssql: -h 10.10.220.2 -p 1433 -A 0 -U hmisnr\\esdsadmin -P bmc@123 -t locks_stats”” |
| Solution | need to make the connectivity with jmx port. |