Alerts – Working with Left Menu

What are Alerts

Cavisson products have an in-build support of proactive alerting through which users can get notified whenever a KPI (Key Performance Indicator metrics like CPU utilization, request per second, average response time etc.) breaches the threshold configured as part of alert rules. This allows users to get an early notification of performance degradation even before an actual issue happens.

Types of Alerts

There are following types of alerts:

Capacity Alerts

In this type of alert, alert generation is done based on a fixed threshold value. User needs to define a Rule and Condition with threshold value. If condition is met, then the alert is generated.

Alert Properties

Following alert properties are available:

Alert Actions

These refer to the actions associated with an alert, which are executed on a certain trigger. Mainly alert actions are divided in to 3 categories:

  • Notification Actions like sending e-mails or SMS or sending SNMP Traps. Besides this, there are multiple extensions available to send notification to third-party programs like ServiceNow, Cisco Spark, and PagerDuty.
  • Diagnostics actions like taking TCP Dump, Thread Dump, and Heap Dump.
  • Remediation action like running some custom scripts to restart a server/instance upon consuming high CPU/Memory.
Alert Settings

Alert settings are used to configure various parameters required for the configuration of alerts.

Alert Policy

Alert Policies enable a user to take different actions when an alert condition occurs and works as link between alert rules and alert actions. User can configure one or more policies with different actions. Policies are associated with Selected or All alert rules configured.

Alert Maintenance Window

Alert maintenance window configuration is required to disable alert generation at the time of maintenance period or a scheduled down-time.

Alert Rules

Alert rule is the key element where user defines the metric and associated threshold to generate an alert for a specific severity.

Alert History

Alert history is useful in obtaining insights into past-generated alerts. Alert history is also used to understand how severity of specific alerts may have changed over a period of time.

The detailed description is provided in the subsequent sections.

Alert Actions

These refer to the actions associated with an alert, which are executed on a certain trigger. Mainly alert actions are divided in to 3 categories:

  • Notification Actions like sending e-mails or SMS or sending SNMP Traps. Besides this, there are multiple extensions available to send notification to third-party programs like ServiceNow, Cisco Spark, Slack, and PagerDuty.
  • Diagnostics actions like taking TCP Dump, Thread Dump, and Heap Dump.
  • Remediation action like running some custom scripts to restart a server/instance upon consuming high CPU/Memory.

Create an Alert Action

To create alert actions, follow the below mentioned steps:

  1. Go to Alert Actions that is displayed in the Alert window.

  1. Click the Add button to add an alert action. The Add Action window is displayed.

  1. Specify the name of the action. There are various sections under this, such as Notification, Diagnostics, and Remediation.

Notification

  1. Under Notification section, select the notification type either email, SMS, or SNMP and specify the details.

Email

  • Turn on the Send an Email toggle key.
  • Select the ‘Generate Mail using Template’ option if you want to generate the email using the Thymeleaf template. You can easily switch to any mail generation type (Normal/Thymeleaf). Thymeleaf is a modern server-side Java template engine for both web and standalone environments. Thymeleaf Template is the default Template for LRM.

Sample output with Thymeleaf Template

  • In the Email Receiver box, type the valid email ID of the receiver.
  • Specify the mail subject by clicking the Add icon. Clicking the Add icon displays the following variables, from which a user can select any number of variables.

  • Similarly, specify the Pre Text and Post Text. Pre Text is the text that is displayed before the email body, and Post Text is the text that is displayed after the email body.

  • Click the Advanced E-Mail Settings icon for more settings.
    • Turn on the Show charts in E-Mail toggle key. This is turned off by default.
    • Specify the maximum number of charts, type of chart (line, bar or area), and the duration of each graph (in minutes).
    • Specify the criteria to select chart metrics. To do this, a user can choose only one from the following options:
      • Use metrics on which alerts are generated: Turn on this toggle key to use the metrics on which alerts are generated. This is turned on by default.
      • Identify and show more relevant metrics where similar pattern is matched: Turn on this toggle key to apply the pattern match. A user needs to specify a catalogue, which contains the graphs information on which the user can apply the pattern match. To apply pattern match, the user needs one baseline with which the user can compare all the graphs available in catalogue. To get the baseline graph (for which alert is generated), the user needs some criteria and it should have the minimum or maximum value across all the indices of one condition for which the alert is generated. Once the baseline is selected, the user can compare each sample of both the graphs (baseline as well as the catalogue graphs) and filter all the graphs, which satisfy the threshold value given in the action. After that, the user can send all the resultant graphs in mail body along with all the alert information. Below is a sample of such email:

  • Place all the metrics in separate charts: Turn on this toggle key to place all the metrics in separate charts. This is turned off by default, in which case all the metrics are merged together and displayed in a single chart.
  • Click Save to save the advanced email settings.

Now, the user can view charts for the metrics on which alert has been generated, as well as user can also see the metrics, which are matching the same pattern as the alert metric, to find out some clue about the root cause. A user can now provide a maximum of 150 charts in alert email and graph duration can be added up to 1440 minutes. If user tries to provide inputs beyond the specified limits, an alert message is displayed on saving and do not allow to save the values.

Whenever an alert is generated, a tabular report is sent in alert email as a link and an attachment. This helps the user to reduce manual processing time to analyze an alert. The report can also be generated by using a template.

Note:  If a graph has NAN samples and alert is configured for only one value, and then if only one value is obtained, the user can view the graph in the alert chart instead of a blank graph.

SMS

  • Turn on the Send an SMS message toggle key.
  • Type the phone number in SMS Receiver to which the SMS message needs to be sent.

SNMP

  • Turn on the Send SNMP Traps toggle key. The user can also enable or disable the Consolidated notification for a rule.
  • Specify the SNMP server and SNMP port.
  • Specify the SNMP version. There are three versions:
    • v1: On selecting this, specify the community to which the alert is to be sent.
    • v2c: On selecting this, specify the community to which the alert is to be sent.

  • v3: On selecting this, SNMP v3 security level section is displayed. This section contains three options:
    • NO_AUTH_NO_PRIV: Authentication protocol and Privacy protocol not required. On selecting this, specify the username.

    • AUTH_NO_PRIV: Authentication protocol required, but Privacy Protocol not required. On selecting this, specify the username, select the authentication protocol, and provide the authentication password.

    • AUTH_PRIV: Both Authentication protocol and Privacy protocol are required. On selecting this, specify the username, select the authentication protocol, and specify the authentication password. Also, select the privacy protocol, and provide the password.

Send Alert to Extension

Select the extension to which the alert is to be sent. Extensions are groups of packages and classes that are bundled in a single jar and make available to you when the need arises. Examples of extensions are –  Big Panda, Cisco Spark, Microsoft Kaizala, Microsoft Teams, Pager Duty, Service Now, Slack, and other Cavisson Products.

Forward Alerts to a Central Dashboard (Other Cavisson Product)

This feature enables the forwarding of alerts from one Cavisson product to another Cavisson product so that all the alerts are visible at one central dashboard. This is useful in the scenarios where the user wants to see multiple products’ alerts in a single dashboard. For example: In one environment there are multiple Cavisson products configured, let’s say NetVision (NV) and NetDiagnostics (NDE) and there is a requirement of sending all NV alerts automatically to NDE so that all the alerts generated from both the products are visible at one place. This feature enables:

  • Alert integration added for sending alerts to other Cavisson product(s).
  • Sending alerts from multiple products to a single Dashboard (many to one).

Forward Alert to Slack

The user can also send notifications to the Slack tool. Once the Alert Action is configured using the Slack extension to send a notification, the notification begins flowing into the Slack application. The user can see the alert incidents in the Slack application.

Forward Alert to Big Panda

The user can also send notifications to the Big Panda tool. Once the Alert Action is configured using the Big Panda extension to send a notification. The notification begins flowing into the Big Panda application. The user can see the alert incidents in the Big Panda application.

Forward Alert to Microsoft Teams

Microsoft Teams is a unified communication and collaboration platform that combines persistent workplace chat, video meetings, file storage, and application integration. A support is provided to send notification from alert engine to Microsoft Teams using alert extension.

The user can also send notifications to the Microsoft Teams tool. Once the Alert Action is configured using the Microsoft Teams extension to send a notification. The notification begins flowing into the Microsoft application. The user can see the alert incidents in the Microsoft Teams application.

Forward Alert to Microsoft Kaizala

Microsoft Kaizala is a secure messaging and work management software application for collaboration among users inside and outside of organizations, including the ability to send and receive instant messages, coordinate tasks, and submit invoices.

You can also send notifications to the Microsoft Kaizala tool. Once the Alert Action is configured using the Microsoft Kaizala extension to send a notification, the notification begins flowing into the Microsoft Kaizala application. You can see the alert incidents in the Microsoft Kaizala application.

Diagnostics

This section contains sub-sections for configuration of thread dump, heap dump, and TCP dump.

Thread Dump
  1. To take a thread dump, turn on the Take a Thread dump toggle key and specify the number of thread dumps along with the interval. Thread dump is applicable for all applications other than node.js. The user can set a timeout (in minutes) for taking thread dump within the provided field.

Heap Dump
  1. To take a heap dump, turn on the Take a Heap dump toggle key and specify the number of heap dumps along with the intervals. Heap dump is applicable for all types of applications. The user can set a timeout (in minutes) for taking heap dump within the provided field.

CPU Profiling
  1. CPU profile is used to identify how the virtual machine (VM) has been spending its time when executing a method. To enable CPU profiling, turn on the CPU Profiling toggle key and specify the duration.

TCP Dump

8. To take a TCP dump, turn on the Take a TCP dump toggle key and specify – Interface, Max Duration, Size, Number of Packets, and Port. Mention the additional attributes, if required.

Remediation

9. To run a script or an .exe file on problematic nodes, turn on the Run a script or executable on problematic Nodes toggle key and type a script name. To run this on server, turn on the Run on server toggle key.

10. After the configuration, click Save.

Advanced Configuration

This feature is used for sending location in Alert Mail, SNMP Trap, and Cisco Spark. Regex is used to extract locations from Alert vectors.

The user needs to provide pattern to fetch Indices Identifier ($INDICES_ID) and then test the provided indices identifier pattern on a string. After that, click the Test button to identify the results after applying pattern on the provided string.

Specify the indices identifier in the Subject Mail section along with the pre text and post text.

The email is generated in the below format.

When a user applies pattern matching, the Alert Mail, which is generated contains the following information:

  • Pattern Matching Threshold
  • Catalogue Used
  • Maximum number of Graphs

Note: In tier-wise alert rule, for relational operators, such as >=, <=, >, <, = or != then ‘Baseline Value’ column is not displayed in Alert Mail and ‘Severity’ column is not displayed for Tier Wise Alert.

Now, the Product Dashboard URL is sent with preselected time frame in alert message. Therefore, when a user hits a URL that is sent via alert email, SNMP trap, and alert extension, only those graphs are displayed for which the alert is generated. Earlier, this used to open the default favorite and the user was required to plot the graphs (where the alert is generated) manually. This feature saves the user’s time and enables to view the relevant graphs automatically.

URL Sent via Alert Email

Graph Where Alert is Generated

 

Import and Export

The Import and Export features are also provided for Alert Action, which were earlier limited to Alert Rule, Alert Maintenance, and Alert Settings. This feature is used for exporting Alert Action and Policy from one machine and import it in another machine. It reduces the effort of adding same policy and action in machine.

Import: This feature is used to import data (policy and action) from server or the local machine. The changes can be viewed immediately after refreshing the UI.

Export: This feature is used to export data (policy and action) to local or other machines. Then, it can be imported to other machines to see the changes.

Export an Alert Action

A user can export one or more alert actions to server or to a local machine. To export an alert action, follow the below mentioned steps:

  1. Select an alert action and click the Export button.

2. This displays the Export window and prompts the user to export to local or to server.

Export to Server

  1. To export the alert action to server, select the Server option and click OK. This prompts the user for server IP and path.

  • Server IP: Specify the IPV4 type IP address of another machine where to export the alert action.
  • Path: Specify the path of another machine where to export the alert action.

4. Click OK. This exports the data at the specified server location and a confirmation message is displayed.

Export to Local

  1. To export alert action to local, select the Local option and click OK.

6. This exports the file to local machine and a confirmation message is displayed.

Import an Alert Action

A user can import one or more alert actions from server or from a local machine. To import an alert action, follow the below mentioned steps:

  1. Click the Import  button.This prompts the user to import the alert action from server or from a local machine.

Import from Server

  1. Select the From Server option and click Choose.

3.Browse the file from the server and click Select.

4. This displays the added alert action in the Import window.

5. Click OK. A confirmation message is displayed for success import.

Import from Local

6. Select the From Local option and click Choose.

7. Select the file from local machine. The added file is displayed in the Import window.

8. Click Upload. A confirmation message is displayed for successful upload of file.

9. Click OK. A confirmation message is displayed that alert action is imported successfully.

Note: Import and Export operations can also be performed in Alert Policy, Alert Rule, Alert Maintenance, and Alert Settings.

Alert Settings

Alert settings are used to configure various parameters required for the configuration of alerts.

  1. Click the Alert Settings sub-menu within the Alerts menu. The following window is displayed.

Alert Settings window consists of following sections:

Rule Triggering Alert Settings

  • Enable/Disable Alerts: Use the toggle button to enable/disable the capturing of alerts. On disabling, user can remove all active alerts. Once enabled, user can further enable capacity alert. On disabling, the user can remove all active alerts or both. On enabling Alert Policy toggle, user can enable/disable the alert notifications (e-mail, SMS, SNMP trap, and Extension), Diagnostics (thread dump, heap dump, TCP dump, and CPU profiling), and remediation (run script).
  • Enable / disable prediction alerts: Predictive alerting provides predictions of certain events or inputs. It is related to machine learning because the technology is able to learn from the data it is regularly processing and based on its learning, it makes predictions and generate alert. Once alert is generated then alert engine perform action based on policy. Predictions alerts are based on some trends. Use the toggle button to enable / disable prediction alerts. Upon disabling, user can remove all prediction alerts.
  • Enable Alert Policy: Here, the user can enable disable alert notification (E-mail, SMS, SNMP trap, extension), diagnostics (thread dump, heap dump, tcp dump, CPU profiling, and Java flight recorder), and remediation (run script).
  • Enable Maintenance Window: To enable/disable the alert maintenance window, use the toggle button.
  • Enable Alert History Logging: To enable / disable the saving of alert history logs in the log file.
  • Skip number of samples on session restart: Specify the number of samples that need to be skipped upon restart of a session. For that number of samples, alerts will not be generated.
  • Selection of graph time and view by: Here, user can specify the minimum graph time (in minutes) to view the graph from the time when an alert is generated. User can also specify the view of the samples by using the ‘view by’ option. This is applicable when user uses the ‘Show graphs’ option within the active alerts and alert history.

Note: The system saves same configurations across all the DCs in case a user tries to save Alert Settings for ‘All’ or individual DC. If any individual DC is selected, saving the Alert Settings impact only on the selected DC and get the configuration from the selected DC that is reflected in UI. If DC is selected as ‘All’, all the changes done by user is saved on all the configured DCs. However, the configuration that is being displayed in the UI is of Master DC.

Rest API Triggered Alert Settings

To enable/disable Rest API triggered alerts, use the toggle button. User can specify the time after which the system clears all the Rest API triggered alerts. On disabling the toggle, user can remove all active Rest API triggered alerts.

Debug Settings

Here, user can configure the following:

  • Debug log: A debug level is a set of log levels for debug log categories. Value range from 0 to 4
  • Profiling log: Value range from 0 to 4
  • Baseline view format: Either basic or extended. In ‘Basic’ view only average/sum values is displayed. In ‘Extended’ view average/sum with count is displayed.

Email/SMS Settings

This section is used to configure the email/SMS settings. To open this, click the Email/SMS Settings button at the top-right corner of the window. Email/SMS settings can also be configured from the top menu.

   

This displays the Email/SMS Settings window:

Provide the required details related to mail configuration and SMS career. Test the configuration and click the Apply button to apply the settings.

Note:

  • The user can import / export alert settings using the Import / Export buttons.
  • The user can make the changes in the Email Settings from Global Settings in the product Home page.

Alert Maintenance

Alert maintenance window configuration is required to disable alert generation at the time of maintenance period or a scheduled down-time.

Here, user can add a maintenance schedule and can view the applied maintenance schedules. In addition, user can search or delete a maintenance schedule based on the requirements.

Configure Maintenance Schedule

  1. Select first indices level (i.e. tier). Upon selecting ‘All’, maintenance schedule is applicable for all tiers. Upon selecting pattern, user needs to specify a pattern for the tier(s) to which maintenance schedule is to be applied.
  2. Select second indices level (i.e. server). Upon selecting ‘All’, maintenance schedule is applicable for all servers. Upon selecting pattern, user needs to specify a pattern for the server(s) to which maintenance schedule is to be applied.
  3. In the same manner, select the third indices (i.e. instance).

User can view the further elements of each level of indices by clicking the Test button. For example, if user selects pattern as *Cav at first indices, and upon clicking the Test button, the details are displayed based on the selected pattern.

This functionality can be used at any indices level based on the pattern or for specific tier/ server/ instance.

Note: If a user selects single Tier, the server level displays the details for that tier, and if pattern is applied, the indices or server list is displayed based on the applied pattern. When the user selects a tier, only that tier gets selected despite of other tiers starting with the same name.

4. Then, select the schedule type from the following list of options:

  • Every Day of Every Month: This maintenance schedule is meant for every day of every month. Select the starting and ending hour at which the maintenance schedule is applicable for each day of each month.

  • Day of Every Month: This maintenance schedule is meant for a particular day of every month. First, select the day from the list and then select the starting and ending hour at which the maintenance schedule is applicable for the selected day of each month.

  • Last Day of Every Month: This maintenance schedule is meant for last day of every month. Select the starting and ending hour at which the maintenance schedule is applicable for last day of each month.

  • Weekday of Every Month: This maintenance schedule is meant for a particular weekday of every month. First, select the week, then day, and then starting and ending hour at which the maintenance schedule is to be applied.

  • Day of Every Year: This maintenance schedule is meant for a particular day of a month. First, select the month, then day, and then starting and ending hour at which the maintenance schedule is to be applied.

  • Weekday of Every Year: This maintenance schedule is meant for a particular weekday of a year. First, select the week, then weekday, then month, and then the starting and ending hour at which the maintenance schedule is to be applied.

  • Custom: This maintenance schedule is meant for a custom duration. Select the starting date and time and ending date at time at which the maintenance schedule is to be applied.

  1. Provide a description for the maintenance schedule and click Add. The system prompts to apply the maintenance schedule as soon as rule is applied. Click Yes.
  2. This adds the alert maintenance schedule and displays in the Applied Maintenance Schedule section.

Other Operations

  • To view maintenance window history, enable the Show Maintenance Window History toggle button.
  • To use the advance filters, click the  icon.
  • To delete the selected maintenance schedule and make the schedule in-effective, first select a record and click the  icon.
  • To delete the selected maintenance schedule and make the schedule effective, first select a record and click the  icon.
  • To download the report in Word, Excel, and PDF format, select the corresponding icons.

Alert Policy

Alert Policies enable a user to take different actions when an alert condition occurs. User can configure one or more policies with different actions. Policies are attached to Selected or All alert rules configured.

Go to Alert Policy that is displayed in the Alert window.

There are following sections within this:

  • Policy Name: Name of the alert policy
  • When to Trigger: The policy is triggered when generated alert satisfied the given criteria. That means, if the severity of the alert gets changed from Critical to Major or Critical to Minor, then only the policy gets triggered.
  • Enabled: Is the policy enabled / disabled .
  • Action(s) to Trigger: Actions to be executed based on trigger, such as taking heap dump, taking thread dump, alert notification via email etc.

Working with Policy

A user can perform following actions with a policy:

  • Add a policy: To add a policy, click the Add 102 button.
  • Edit a policy: To edit a policy, select the policy and click the Edit 103
     button.
  • Delete a policy: To delete a policy, select the policy and click the Delete 104 button.
  • Copy a policy: To copy a policy, select the policy and click the Copy 105 button
  • Enable a policy: To enable a disabled policy, select the policy and click the Enable 106 button.
  • Disable a policy: To disable an enabled policy, select the policy and click the Disable 107 button.

Add a Policy

  1. Click the Add button on the Alert Policy window. The Add Policy window is displayed.

  1. Type the policy name. To enable it, select the Enable Policy checkbox.
  2. If Applicable Only for Predictive Alert option is enabled, the policy works for only prediction alerts.
  3. AlertMail and SNMP Trap can be sent when Alert is generated using REST API. For that, turn on the Applicable Only for Rest API toggle key. If the user turns it off, policy will be applicable for both – Alerts and Alert through Rest API.
  4. In the Policy Events section, turn on the Enable/Disable all criteria toggle key.
  5. Select the intensity of the alert rule, such as Critical, Major, and Minor for both starting and ending the rule violation.
  6. Specify the Capacity alert rule. To specify alert rules, select Specified Alert Rule(s).
  7. Click Action.

Add Actions to Execute

This has already been covered in Alert Actions section.

Alert Rules

Alert rule is based on the alerts defined if a condition is met. To view the Alert Rule window, go to Alerts menu and click the Rules menu item. The Alert Rule window is displayed.

There are the following columns:

  • Status: Represents whether the Rule is enabled/disabled.
  • Rule Type: Represents whether the rule is configured for tier level or individual indices level.
  • Rule Name: Name of the rule.
  • Condition Expression: Displays the summarized view of the conditions. Upon mouse hover, the detailed condition is displayed.
  • Alert Message: Message to be displayed when an alert is encountered.
  • Alert Description: Description of the alert to understand the cause and other insights.
  • Recommendation: Specify preventive measures for improvement and overcome the reason for alert generation.

There are following actions a user can apply on rules:

  • Enable: To enable a rule
  • Disable: To disable a rule
  • Edit: To edit a rule
  • Add: A add a new rule. The user can also add a prediction rule from here.
  • Delete: To delete a rule
  • Update: To update a rule
  • Copy: To copy a rule
  • Import: To import a rule from server or local
  • Export: To export a rule to another server or local

Creating an Alert Rule

User can create a rule based on the Tier level or individual indices level. For both the levels, there are different configurations. We will describe the configuration for both the levels.

To create an alert rule, follow the below-mentioned steps:

  1. Click the Add button on the Alert Rule This displays Alert Rule Configuration window.

2. Specify the rule name. To enable it, select the Enable Rule check box.

3. Specify the moving window timelines. For moving window advance settings, click the settings  icon. This displays the Moving Window Advanced Settings window.

4. In this window, there are the following options for calculating graph data value:

  • Moving Average: Graph data value for alert is calculated on each new sample generated.
  • Fixed Window Average: Graph data value for alert is calculated for fixed time as specified.
  • Moving Advanced: Graph data value for alert is calculated for last / any stated percentage of samples individually with specified rule condition.

User can also enable checking of logs for a specified duration if alert remains in the same state. Also, initiate action when ‘Alert rule continues in the same state’ flag is enabled in the policy.

  1. To enable alerts for business health rules, select the Business Health Rule checkbox.

Applying Rule at Tier Level

Here, the user can apply the rule at the tier level and alert is generated on the overall results of the tier based on the conditions applied.

  1. First, select the Tier radio button and then select the tier from the drop-down list. User can also apply a pattern for selecting the tiers from the list.
  2. Specify whether to consider all conditions or any of the condition (for the alert generation) by selecting ‘Every’ or ‘Any’ respectively.
  3. Provide the condition name.
  4. Select for which value (such as average, sample count, sum, minimum, and maximum) the alert condition is to be configured.
  5. Then, select the metric group and metric on which the condition is to be applied.
  6. Specify the condition and its threshold value. There could be one of the following conditions:

Comparing with Baseline Data: Absolute value / percentage specified for the alert condition is compared with the selected trend of the Baseline data.

  • Increases from Baseline: If the current value is increased with the specified value / percentage as compared to the baseline data for that particular trend, then alert will be generated. User can also specify the minimum increase from the baseline.
  • Decreases from Baseline: If the current value is decreased with the specified value / percentage as compared to the baseline data for that particular trend, then alert will be generated. User can also specify the maximum increase from the baseline.
  • Changes from Baseline: If the current value is changed (either increased or decreased) with the specified value / percentage as compared to the baseline data for that particular trend, then alert will be generated. User can also specify the minimum change from the baseline.

Comparing with Current Data: Absolute value / percentage specified for the alert condition is compared with the current data.

  • Increases: If the current value is increased with the specified value / percentage, then alert will be generated. User can also specify the minimum increase value / percentage.
  • Decreases: If the current value is decreased with the specified value / percentage, then alert will be generated. User can also specify the minimum decrease value / percentage.
  • Changes: If the current value is changed (either increased or decreased) with the specified value / percentage, then alert will be generated. User can also specify the minimum change value/ percentage.
  • Greater than Equals to: If the current value is greater than and equals to with the specified value, then alert will be generated.
  • Less than Equals to: If the current value is less than and equals to with the specified value, then alert will be generated.

User can also notify alert engine to consider the condition as true when data is not present for a common indices by selecting the Consider as true when data is not present for a common indices check box.

Adding Multiple Conditions

User can add multiple condition and specify whether to consider ‘all’ or ‘any’ condition for the alert generation.

  • To add a new condition, click the Add  icon at the top-right corner. This displays the place holders for the new condition. User can either provide the configuration details manually or just copy the configuration from any / previous condition and edit later.
  • To copy the same configuration from any condition, click the Copy  icon from the condition whose configuration is to be copied and click the Paste  icon corresponding to the condition where the configuration is to be copied.
  • To delete a condition, user can click the Delete  icon.

Alert Severity

In this section, user can configure the alert severity based on the indices affected if the specified conditions are satisfied. The severity can be specified either for:

  • % of Indices: If the condition is met, user can define the severity (such as critical, major, and minor) based on the percentage of affected indices. For example, if 12 indices are affected out of 20 then it means 60% indices are affected. For severity configuration, if we specify critical for 80%, major for 60%, and minor for 40% then major alert will be generated in this case as the affected indices are 60%.
  • Number of Indices: If the condition is met, user can define the severity (such as critical, major, and minor) based on the number of affected indices.
  • Aggregate value of Indices: If the condition is met, user can define the severity (such as critical, major, and minor) based on the aggregate value of indices.
  • Individual Indices: If the condition is met, user can define the severity (such as critical, major, and minor) based on the individual indices.
  1. Specify the Alert message that is to be displayed when the alert is generated.
  2. Specify the Alert description that provides an insight about the generated alert.
  3. Specify the recommendation that is to be followed to overcome from the alert in production environment.
Applying Rule at Indices Level

Here, user needs to specify the condition at indices level corresponding to a severity. User can add multiple conditions for each severity.

  1. Select the Individual Indices level radio button.
  1. Specify the condition for Critical severity with the same process as defined in the Tier level alert configuration. In the same way, specify condition(s) for other severity, such as Major and Minor. User can apply multiple conditions for any severity and can mention whether to consider each or any condition for a particular severity.
  1. For adding a condition, user needs to provide the condition name, value on which the condition is to be applied (average, sample count, sum, minimum, or maximum), metrics group, metrics, and indices.
  1. For indices selection, user is having multiple options that are displayed upon clicking the Select Indices link.

Here user can have following options for indices selection:

  • All: Condition is applied on all indices.

  • Specified: Condition is applied only on the selected indices. Select the indices by using the  button.

  • Advanced: This is an advanced level indices selection. User can also specify pattern for the tier, server, and instances.

  1. Specify the Alert message, alert description, and recommendation.
Using the Value Generated from Graph A as a Threshold for Graph B

An alert is generated on a metric by comparing it with the threshold value of another metric based on some conditions. A new condition ‘Percentage of’ is used to select a threshold graph. This is applicable for Increases, Decreases, Changes, Increases from Baseline, Decreases from Baseline, and Changes from Baseline condition.

Alert Value in Active Alert

Prediction Rule Configuration

Prediction Alert works on an algorithm to detect the trend of any graph and find out the time when it reaches the target value provided by the user. This is an important feature to detect problems in prior and take necessary action regarding that.

The user can configure the prediction alert rule from the prediction alert window. Predictive alerting provides predictions of certain events or inputs. It is related to machine learning because the technology is able to learn from the data it is regularly processing and based on its learning, it makes predictions and generates alerts. Once an alert is generated, then the alert engine performs actions based on policy. Predictive alerting is different from other alerting systems such as rule-based alerting because it adapts and learns from new data sets, which can be varied, and complex in nature. Rule-based alerts, on the other hand, can work only based on preconfigured rules and may not be suitable for real-life scenarios.

To configure a prediction alert rule, go to Add and click the Add Prediction Rule link.

This displays the Prediction Rule Configuration window.

Provide the following details:

  • Prediction alert rule name
  • Moving window
  • Forecasted value
  • Metric Group
  • Metric Name
  • Indices
  • Target value
  • Estimated Time for Critical Severity
  • Estimated Time for Major Severity
  • Estimated Time for Minor Severity

 

Example

In the below configuration: an alert rule ‘Heap Memory is getting full’ is triggered based on last ‘60’ minutes trend when forecasted ‘Average’ value of ‘Java7 Garbage Collection’ group’s ‘JVM Heap used memory’ graph’s Cavisson>NDAppliance>Tomcat indices reaches the target value ‘80’ in estimated time of ‘10’ minutes for Critical alerts, ‘60’ Minutes for major, and ‘120’ minutes for Minor severity.

An Alert message is generated as: Heap Memory is getting full, with the description as: Heap memory is getting full, and recommendation as: Heap memory is getting full. Please check the configuration in site.env file.

Tuning Factor for Prediction Algorithm

Go to  icon to set the tuning factor for the prediction algorithm. This is the interval in which the Tuning Factor is applied to tune the prediction.

Rules Filter

To filter the Rules, go to the Rules Filter drop down at the top-right corner of the Alert Rule window.

Import / Export Alert Rule

User can import / export an alert rule to/from a server or local based on the requirements. To do this, follow the below mentioned steps:

To import a rule, click the Import button. This displays Import Window popup where user is having two options, such as ‘From Server’ and ‘From Local’ to import a rule.

Import Alert Rule from Server

To import an alert rule from server, follow the below mentioned steps:

  1. Select From Server option and click Choose.

This displays external data file manager window. Select the required path / folder from this window where to import the alert rule.

                                                                              

Import Alert Rule from Local

To import an alert rule from local, follow the below mentioned steps:

  1. Select From Local option and click Choose.

2. Select the alert rule file (containing the alert rule configurations) which is to be imported.

Export Alert Rule to Server

To export alert rule to server, follow the below mentioned steps:

  1. Select an alert rule to be exported and click the Server option.

2. Provide the Server IP and Path configurations.

Download Report

The user can download the alert rule report. A user can view both global and custom filters applied in the downloaded report in all alert data tables.

Applied Filter

Downloaded Report

Alert History

Alert history is useful in obtaining insights into past-generated alerts. Alert history is also used to understand how severity of specific alerts may have changed over a period of time. To open the Alert History page, click the Alert History menu item on the Alerts menu. This displays the Alert History window.

This window consists of two panes – left pane and right pane. On the left pane, filters are displayed, and on the right pane, alert details are displayed.

Alert Details

On the right pane, following alert details are displayed – alert type, alert severity, rule type, status, rule name, indices, alert value, a time when an alert is generated, condition expression (on mouse hover it displays the complete condition), and message displayed on alert generation.

Alert Severity

Icon

Meaning

Normal

Major

Critical

Info

 

Note: The user can see an ‘Info Alert’ icon  in ‘Severity’ column of Alert History if alert is generated with severity value as 6.

To view detailed information, double-click the alert. A section for detailed alert information is displayed at the bottom of the window:

Disable this detailed section by double-click the alert again.

Show Graph

To view the graph of a particular alert, select the alert and click the Show Graph button. User can select multiple alerts and get their corresponding graphs displayed on the widget panel.

Row Grouping

This feature enables a user to group alerts based on certain parameter. For this, user needs to click the Apply Row Grouping icon  on the top-right corner of the window.  Post that, user needs to select the grouping parameter from the drop-down list (for example, Rule Name). The alerts are grouped and get displayed on the alert history window.

To remove the row grouping, click the  icon again.

Advance Alert Filters

Time Filter: This section is used for time filter, such as last 10 minutes, 30 minutes, and so on.

Alert Severity: This section is used to filter alerts based on severity, such as new alert, continues alert, upgraded alerts, and downgraded alerts.

Alert Type: This section is used to filter alerts based on alert type, such as capacity.

Alert Rules: This section is used to filter alerts based on alert rules. Alert rules are displayed in a list. User can search for an alert rule using the search icon.

String Filter: This filter is used to filter alerts based a matching rule name, baseline name, or message.

Topology Filter: This filter is based on topology (tier, server, and instance).

Other: This filter is based on other parameters, such as rule changes, baseline changes, alert setting changes, maintenance window changes, and tomcat changes.

Actions on Alert Filters

  • Apply: This applies the selected filter and displays the result accordingly.
  • Reset: This resets the filters and selects only the default filter values.
  • Clear All: This clears all the specified values for the filters.
  • Select All: This selects all the filters.

Global Filter

User can apply global filters (in the form of a string) on the resultant alerts displayed. This is not specific to any column. The filter is applied on all columns of the displayed result.

Hide Filters

To hide all filters from the left pane, click the Hide Filter icon . To get it back, click the same icon again.

Records per page

By default, 20 records per page is displayed. User can change it to 50, 100, or 200.

Columns level Filter

User can also apply column level filter on the generated alerts. To do this, click the Show Filter icon  on the top-right corner of the window. The column level filters get enabled and user can use those filters for filtering the alerts.

Delete Records

To delete a record(s), select the record(s) and click the Delete Record(s) icon  on the top-right corner of the window. The record gets deleted from the alert history.

Download Alert History

User can download the alert history in word, excel, and PDF format by clicking the icons provided at the bottom of the window.

Indices Column in Alert History

On clicking, the value of Indices in the Indices column is displayed in a pop-up with ‘Alert Value’ column.

Display of ‘User Action’ in Rule Changes and Alert Setting Changes

On alert rule and alert setting update, the details of all the fields that have been changed are displayed in alert history in a pop-up window on the click of message field. In alert history, message field of all rule change and setting change rows are clickable (rest are same as it used to be). On clicking the message field text, a pop-up is displayed, showing all the changed fields in a tabular format.

Alert Rule Changes

Alert Setting Changes

Filter Data for Action, Policy Changes

Policy and action changes are dumped in alert history and filters for fetching the policy and action data is added in history left pane. ‘Rule Name’ has been changed to ‘Name’. And policy, action, and baseline name along with their icon is displayed in ‘name’ and ‘alert type’ column respectively.

Alert Integration Framework

The user can now get the alert generated during the running test in the Alert History window.

Alert Integration is used to generate and send alerts to tomcat locally and remotely both which will be shown on the Dashboard through which it will take minimum effort to debug why the test has started failing or got exited.

During the running of the testrun, if any kind of failure occurs from below list then alert will be generated by which user would be able to identify the problem occurring during the testrun easily.

  • Content validation failures alert
  • NVM failures alert
  • NVM Delay alert
  • Generator failures/Discarded alert
  • Generator Delay alert
  • Progress Sample queue stuck alert
  • Script Failures in RBU/NVSM
  • Element not found and iframe failure alert (RBU/NVSM)

Active Alert

Active alert window is similar to Alert History window. This window displays all the active alerts generated. The details displayed here are – alert ID, alert severity, rule type, rule name, alert message, alert time, time ago the alert was generated, indices, alert value, and condition expression. Whenever a new alert is generated, Alert ID is displayed in the Alert History table and Active Alert UI.

Here also, user can perform certain actions:

  • From the Alert Type drop-down list, user can specify for which alert type (capacity) the details is to be displayed.
  • User can also view the alerts based on the severity, such as critical, major, or minor.
  • User can search an alert from the Global Filter box.
  • User can view the graphs of individual alert or for all alerts by selecting them and clicking the Show Graph button.

The time-frame in ‘Show Graph’ and ‘Graph in Alert Email’ are shown highlighted.

Graph

Email

  • User can also navigate to alert history by clicking the Alert History
  • To clear alert(s) from the records, select the row and click the Force Clear

The ‘Alert Value’ field displays ‘Alert Value’, ‘Baseline Value’ or ‘Previous Window Value’ and ‘Load Metric Value’ if present for a generated alert.

Note: Alert feature is also implemented for different test runs for Multi DC. The user can view active alert bell counter and the corresponding records in Active alerts.

Satisfied Indices details are displayed in Active Alert/Alert History/Alert Mail and in Alert History table. As per the previous design, if any alert was configured with multiple conditions, there was a chance that alert got generated at certain level of hierarchy which was common in both the conditions. Due to that, the user was unable to get the exact indices for which alert was generated. Now, complete details are provided for all the satisfied indices along with the partial ones.

For Example:

Active Alert UI

Alert History UI

Note:

The time format of showing alert in ‘time ago’ field in active alert is changed from HH:mm:ss to 30d 10h 20m 5s format.

 

Example: If an alert is generated 5 days back, earlier it was represented like 121:20:10 in time ago field.

Now, it is displayed like 5d 1h 20m 10s where,

  • 5d represents 5days
  • 1h represents 1hour
  • 20m represents 20 minutes

10s represents 10sec

Active Alert Graphs

This section displays graphs of active alerts. To view Active alert graphs, follow the below mentioned steps:

  1. Go to Alerts menu on the left pane and click the Active Alert Graphs menu item. The Active alert graphs are displayed.
  2. User can perform all the operations with alert graphs, such as widget settings, show graph data, add to custom metrics, show graph in tree, view reports, run command, download.

Alert Stats Report

User can view the stats report of the graph data as generated in the active alert graphs. To view the stats report, follow the below mentioned steps:

Go to Alerts menu on the left pane and click the Alert Stats Report menu item. The Alert Stats Report is displayed.

In this report, graph details of alert graphs are displayed, such as graph name, minimum value, maximum value, average value, standard deviation, last value, and number of samples.

On clicking a graph link, the graph is displayed.

User can download a graph in PDF, word, and excel format.

Alert Action History

It contains details of action taken by alerts, such as SNMP Trap Sent / Email Sent / Cisco Spark Chat sent and so on.

To access the Alert Action History section, go to Alert > Alert Action History. This displays the Alert Action History window.

This contains following details:

  • Rule Name
  • Policy Name
  • Action Type
  • Action Time
  • Indices Name
  • Message
  • Description

In SNMPTrap, IDCName field holds the following:

  1. IDCName_SourceIP (Appliance IP) if IDCName already configured.
  2. Source_IP if IDCName is not configured.

Note: The user can download the alert history report in PDF, Excel, and Word formats.