NetStorm Continuous Delivery Configuration

It involves the identification of test cases that needs to be run as a part of continuous delivery followed by configuration settings, which is further divided into two phases.

  1. Configuring Jenkins: This phase includes configuration at third party tool i.e. Jenkins.
  2. Configuring NetStorm: This phase includes configuration at NetStorm (Creation of Test suite and Test Cases).

Phase 1: Configuring Jenkins

For integrating TestSuite with Jenkins, you need to install the cavisson-ns-nd-integration-plugin. This plugin generates CICD reports of the tests executed in NetStorm.

Pre-requisite

Maven should be installed on your system.

Installing Maven

If you have not installed Maven on your system, perform the following steps:

  1. Download Maven from the following location: https://maven.apache.org/download.cgi?Preferred=ftp://ftp.osuosl.org/pub/apache/
  1. Follow installation steps as mentioned in the following file: https://maven.apache.org/install.html

Creating the Netstorm-Jenkins Plugin File

To integrate TestSuite with Jenkins, you need to first create a Netstorm.hpi file. To create this file:

  1. Access the cavisson-ns-nd-integration-plugin download page from the following location:https://github.com/cavissonsystems/cavisson-ns-nd-integration-plugin
  1. To download the zip file:
    • Click the Code tab.
    • Click the Download Zip icon.

3. Unzip the file.

4. Open command prompt and go to path “C:\Users\<Your Username>\ Downloads\cavisson-ns-nd-integration-plugin-master\cavisson-ns-nd-integration-plugin-master”.

For example:

  1. Execute the following commands:
    • mvn versions:set -DnewVersion=4.5.0.26
    • mvn package
  2. Netstorm.hpi file gets created at the following location:

C:\Users\<Your Username>\Downloads\cavisson-ns-nd-integration-plugin-master\cavisson-ns-nd-integration-plugin-master\target

Configuring the Netstorm Plugin in Jenkins

After creating the Netstorm.hpi file, you need to install it in Jenkins.

Perform the following steps:

  1. Login to Jenkins dashboard where settings need to be done.
  2. “Netstorm.hpi” is required to execute the test remotely on NS machine. Hence, check if it is installed or not. If not, need to be installed.
  3. To install “Netstorm.hpi” in the Jenkins Global configuration, go to Manage Jenkins (A) and click on Manage Plugins (B).

2

  1. Click the Advanced button, then browse the “Netstorm.hpi” file from the desktop, and click the Upload button.

3

5. For generating HTML reports, click the Available tab and install the HTML Publisher Plugin.

6. Restart Jenkins using the following commands:

  • /etc/init.d/Jenkins restart (From “CLI”)

or

  • Parameterize the Jenkins URL, for example, 10.10.30.9:8080/restart (From “GUI”)

7. Add a New Job (New item) by clicking the New Item link.

4

8. Enter the item name in the text box, select the Freestyle project radio button, and click OK.

5

9. Under the Add build step drop-down list, select the Execute NetStorm / NetCloud Test option.

10. Enter the URL, Username, Password to connect with the NS machine and enter the project, subproject, and TestSuite name to execute.

7

Enabling Security in Jenkins for not Displaying the External CSS

To enable the security in Jenkins for not displaying the external CSS:

  1. Create a file named groovy in the Jenkins directory.
  2. Copy the following content in the init.groovy file:

System.setProperty(“hudson.model.DirectoryBrowserSupport.CSP”, “default-src * ‘unsafe-inline’ ‘unsafe-eval’; script-src * ‘unsafe-inline’ ‘unsafe-eval’; connect-src * ‘unsafe-inline’; img-src * data: blob: ‘unsafe-inline’; frame-src *; style-src * ‘unsafe-inline’; font-src *”)

  1. Select Manage Jenkins > Manage Nodes and clouds.
  2. Click the Settings icon.
  3. Open the Script Console from the left panel.

6. Copy the following command in the text area: setProperty(“hudson.model.DirectoryBrowserSupport.CSP”, “”)

7. Click Run.

After configuring these settings in Jenkins, you can add a post-build action by selecting NetStorm Performance Publisher, NetDiagnostics Performance Publisher, or NS-ND Integration Performance Publisher.

Configuring the NetStorm Performance Publisher Settings
  1. From the Add post-build action drop-down list, select the NetStorm Performance Publisher option to publish the HTML report.

2. Enter the URL, Username, and Password of the NS machine from where you require the HTML report to publish on Jenkins and click the Save button.

3. Click the Post-build Actions tab and select the Publish HTML reports option:

4. Specify the following configuration:

5. Click Save.

Pulling NetStorm Testsuites/Scenarios from GIT

To pull the latest version of test suites/scenarios from a GIT repository:

  1. Click the Build tab in the NetStorm Performance Publisher window.
  2. Provide details about the NetStorm machine that you want to connect to the GIT repository.
  3. Select the Git Pull check box.
    • If you have already configured the GIT repository with your NetStorm machine, the saved test suites and scenarios will be pulled from the repository and a success message is displayed.

    • If you have not configured a GIT repository, the following error message is displayed:

To configure the GIT repository, perform the following steps:

  1. Click the Build tab in the NetStorm Performance Publisher window.
  2. Provide details about the NetStorm machine that you want to connect to the GIT repository.
  3. Select the Git Configuration option.

4. Specify your GIT configuration settings.

5. Click Test to verify that the settings configured.

6. Click OK. After you have configured GIT, select the Git Pull check box to retrieve the latest test suites and scenarios from the Git repository.

Configuring NetDiagnostics Performance Publisher Settings
  1. From the Add post-build action drop-down list, select the NetDiagnostics Performance Publisher option to publish the HTML report.

2. Enter the URL, Username, and Password of the ND machine along with some other details from where you require the HTML report to publish on Jenkins.

You can also select the time period for completing the NS test while generating an ND report. While selecting the time period, you need the time of duration phase (excluding RampUp and RampDown).  To achieve this, select one of the following time frame options:

  • Whole Scenario: Specifies that the ND report will be generated for the whole NS test run.
  • Absolute Time: Specifies that the ND report will generate for the duration between given absolute Date/Time in “From” and “To” from the whole NS test run. The format of user input field will be MM/dd/yy HH:mm:ss.
  • Elapsed Time: Specifies that the ND report will be generated for the duration between the given Elapsed Time in “From” and “To” from the whole NS test run. The start time of the ND report will be the sum of Start Date/Time of NS test and given time in “Form” Field. The end time of ND report will be the sum of Start Date/Time of NS test and given time in “To” field. The format of user input in field will be HH:mm:ss.
  • Phase: Specifies that the ND report will be generated for the selected phase from the whole NS test run. Phase is given as per the below screenshot.

3. Click the Post-build Actions tab and select the Publish HTML reports option.

4. Specify the following configuration for generating an HTML report:

Configuring NS-ND Integration Performance Publisher Settings

To configure NS-ND Integration Performance Publisher Settings:

  1. From the Add post build action drop-down list, select the NS-ND Integration Performance Publisher option to publish the HTML report.

  1. Enter the NS details and ND details, such as NS / ND HTTP URI, user name, and password. The user also needs to specify the test case pass criteria. The baseline for ND will be the same as configured for NS and the details will be populated automatically.

  1. Specify the configuration displayed in the following figure for generating an HTML report:

4. In the General section, the user can parameterize the project by selecting the This project is parameterized check box and specify the parameter.

5. After performing all the required actions, click the Save button.

6. Click the Build with Parameters link to run the test suite from Jenkins.

Operations in Jenkins

Environment Name: Environment name is the server name of the client, which is present in Server Host of the scenario [10.10.30.3:81 is in this case].

You can change the following fields from Jenkins as well as TestSuite.

  • Number of users
  • Server Host
  • Duration
  • Ramp-up Time
  • Changing the SLAs
  • Test Name
  • AutoScript

Perform the following steps for Jenkins parameters:

  1. Select the This build is parameterized check box.
  2. Under the Add Parameter drop-down list, select the NetStorm Test Parameter option.

12

3. Enter the parameters (according to requirements) and click Save.

13

4. Run the Test. To view the report, click the report link (For example: NetStorm Performance Report).

5. Web Dashboard (For individual Graph): Upon clicking the  link corresponding to a record, the dashboard is displayed.

To view graphs, click the  icon:

6. Web Dashboard (Jenkins Panel Graph View): Users can also view the overall graphs in the Web dashboard by clicking the  icon.

7. Transaction Trend: Earlier NetStorm transaction trend link only has the current test run trend, but now it also shows the Baseline and Previous Trend for a better comparison.

In trend analysis, the below colors are used for differentiation.

  • Current Test Run –> RED
  • Baseline Test Run –> BLUE
  • Previous Test Run –> GREEN

8. NS-ND Integrated Performance Report: User is now able to view the integrated performance report for NS and ND.

Phase 2: Configuring NetStorm

Adding a Test Case

Test Cases provide an automated way to run a Test scenario. A test case uses a single scenario however it may be designed to include multiple test scenario/script with a different set of parameters to achieve a specified target.

Follow the below-mentioned steps for adding a test case:

  1. Enter the URL http://hostname/ProductUI in the address bar of the browser. The NetStorm login window is displayed.
  2. Enter the Username and Password and click the Login button. The product home page appears.
  1. Select Advanced > Automation > Test Cases. The Automation Test Case page is displayed.

  1. Click the Add button to open the Test Case Configuration page.

It is the NetStorm scenario used in this test case. One test case uses a single scenario. We can parameterize keyword values in the selected scenario to our requirement allowing us to create multiple cases for multiple test runs at runtime. Scenario parameterization is done by replacing values with variables beginning with ‘$’. Scripts used in scenario are parameterized with variables beginning with ‘%’. As the script name is present within the scenario we can test with multiple scripts too by parameterizing the script name.

To design a Test Case, following key section are available on NetStorm – Test Case Configuration page:

  • Scenario
  • Check Profile
  • Check Script
  • Iteration
  • Pre Test Script
  • Post Test Script
  • Mail Configuration

Note:

To run a Test Case, Scenario and Check profiles are the mandatory sections while Iteration, Pre Test Script, and Post Test Script sections provide the additional flexibility to add more features to test case.

Scenario

In the Scenario section, you need to add the scenario to be executed, edit the scenario settings, edit the script, and view the scenario details.

Adding a Scenario

To add the required scenario:

  1. Select a project, sub-project, and provide a unique Test Case Name with a description.
  2. Select a scenario from the existing scenarios list:

3. To edit the scenario settings, click the Edit in Scenario GUI icon.

Editing the Script

Click the Script icon to open the Script Manager window and update settings.

Check Profile

Check profile is a group of rules on selected metrics and provides the flexibility to add validation on the threshold values. You can add a new check profile or edit an existing check Profile.

Note: You can also add/view a check profile by selecting Advanced > Automation > Check Profile.

Adding a New Check Profile

To add a new check profile:

  1. Select a scenario from the Scenario section.
  2. Click the Add New Check Profile icon. The Add Check Profile window is displayed:

In the Add Check Profile window, specify the following information:

Profile information

Specify a profile name and description.

Select a Test Run

Select the corresponding Test Run number. It will add all the metric groups and metric names available in that particular Test Run.

To specify advanced settings, click . The Advanced Settings dialog box is displayed:

Select the Include Discontinued Metrics check box to fetch the list of vectors according to the specified time duration. It enables you to select a style and provide a custom partition.

Add Metrics

You can define a rule/metric by specifying the following information:

  • Select a data field value: Percentile, Average, Minimum, or maximum.
  • Specify the indices value to apply the rule for displaying the result in the HTML Report You can select All, Specified, or Pattern indices. Selecting Pattern enables you to enter the pattern in order to select multiple indices at a time. For example, if you provide the “xx” pattern, then all the indices ending with string “xx” will come in the report.

To add a metric, click the Add Metric icon. You can add multiple metrics, as shown highlighted in the following figure:

Specify Operations

Select an operation from the list of operations and provide the value for comparison based on the operation:

  • Define a unique name for the rule and display name.
  • Define conditions by using Arithmetic as well as Logical operators.

Check Script

In the Check Script section, you can compare multiple test runs and define the pass/fail criteria. You can compare the current test case execution with previous test executions. Presently, multiple baseline support is up to 3 test cases.

Compare Tests

For comparing tests:

  1. Select the Baseline checkbox.
  2. Specify a measurement name. The measurement name can be a maximum of 16 characters long.
  3. Select a scenario and test run with which you want to compare the results.

4. Select the Override Checkprofile SLA with option and specify the baseline test. It is optional to select this option.

Pass/Fail Criteria

In the Pass/Fail Criteria section, you can specify any % value to the Critical threshold (default 50%) Warning threshold (default 20%), and Overall Fail Criteria (default 10%) which works on rules defined in Check Profile.

You can specify decimal values in the Critical Threshold, Warning Threshold, and Overall Fail Criteria. Decimal Values are limited to two digits only. Also, you can specify 0 in the Warning Threshold and Overall Fail Criteria field.

Note: The Critical Threshold cannot be zero as the Critical threshold should be greater than Overall Fail Criteria and if you specify Critical Threshold as 0 then the Overall fail criteria will have to be less than 0 which is not possible.

Next, the user can select any of the existing check profiles or a new check profile.

Users can also select the Start Date/Time and Environment Name check box based on the requirements.

To include unexecuted transactions and view success transactions and transaction count in the test report, select the Include Unexecuted Transactions, Show Success Transactions, and Show Transaction Count checkboxes respectively.

Pre Test Script

The Pre Test Script tab enables you to add different actions on selected servers.  You can specify the required configuration in this tab before executing the test cases. Based on the selected action, input fields are displayed. For example, selecting the action “Copy_File” displays the Source and Destination fields. The following actions can be defined in the Pre Test Script tab.

  • Start_Command: Enables you to start a service on server machine. It is also done with and without cavmon agent.
  • Stop_Command: Enables you to stop service on server machine. It is also done with and without cavmon agent.
  • Remove file: Enables you to remove a file on the server machine.
  • Start_service_sudo: Enables you to start a service on the server machine. When running the privileged command with the pseudo prefix, you will be prompted for your regular password before it is executed.
  • Stop_service_sudo: Enables you to stop service on server machine.
  • Upload: Enables you to upload a file on server. It can also be done in two ways – With and without cavmon agent.
  • Download: Enables you to download a file from server. It is done using nsu_import_file.
  • Sleep: Enables you to provide sleep between any two actions. This function suspends execution of the calling thread for specified time.

You can configure the required action in the following ways:

  • Using the GUI mode.
  • Using the Text mode.

To define a pre-test script using the GUI mode:

  1. Select a scenario.
  2. Expand the Pre Test Script tab.
  3. Select an action. Based on the selected action, the related fields are displayed.
  4. Select a server name.

5. Provide details in the required fields.

6. Click Add Action.

7. Click the Save button to save the selected action in Pre Test Script.

To specify the Pre Test Script in Text Area, click View in Text mode and write the required script in the given text box.

You can also edit, delete, or reorder the already added Actions. You can manage the order of pre and post script data by dragging the required action row:

Post Test Script

Using the Post Test Script tab, you can specify the required actions that can be performed after execution of the test case. These actions can be used for cleaning up of the existing settings.

Note: The functionality of Post Test Script is similar to the Pre Test Script tab. The only difference between them is that the Pre Test Script is executed before running the test cases, while the Post Test Script is executed after running the test case.

 

Configuring Mail

You can send TestSuite reports from NetStorm to the selected users. Using the Mail Configuration tab, you can configure the email settings in NetStorm and send reports as an attachment.

Before using the mail feature, you need to ensure that the global Email/SMS settings are specified for the host, port, and sender. To specify the email settings:

  1. Select the Email/SMS Settings option from the drop-down list.

2. Specify a valid SMTP host and port.

3. Select a protocol from the Use Secure Connection drop-down list that will be used for sending mails.

4. If you want to authenticate the sender while sending mail:

  • Specify the sender’s email address in the Send Mail To
  • Enable the Authentication Required checkbox and specify the password.

Note: It is recommended to check the configuration once by sending a test mail through the “Test Configuration”.

To specify the email addresses in NetStorm:

  1. Expand the Mail Configuration tab in the Test Case Configuration.
  2. Specify the required email IDs in the To, Cc, and Bcc fields. On report generation, an email is sent to the mentioned users in To, Cc, and Bcc fields.

Adding a Test Suite

Test Suite is the NetStorm entity to control the execution of tests together with the comparison of actual outcome to the predicted outcome (SLA). It involves automating a manual process that uses a formalized testing process. A TestSuite can be configured to execute a single test case or multiple test cases.

  1. Select Advanced > Automation > Test Suite. The Automation Test Suite screen is displayed.

2. Click the  button to open the Automation Add/Edit Test Suite page.

26

  1. To add a test, select a project, sub-project, and specify a unique Test Suite Name.

  1. Select a test case. User can set action on failure which can be:
    • Continue Test Case
    • Abort Test Case
  2. In the Parameterization Mode, the user can select any of the below options:
    • No Parameterization
    • Parameterization Using Iteration File
  3. Provide Execution count (number of times to run the test case) in the Execution Count Text box.
  4. Click the Add button to add the test case into Test Suite.

8. Click the Save button to save the test suite.