Reports 

Reports is a reporting module available in the Cavisson products. These reports are generated using Templates. User can select reporting format as Word, HTML or Excel. It includes tabular data with or without graphical illustration.

Note: Report Access Permission: If a user is assigned with ‘No Permission’ in capabilities section to access ‘Report’ feature, then on accessing the Reports feature from TestRun / Session UI / Dashboard, the session logs out with a message “You are accessing unauthenticated links hence closing this session”. In addition, the user cannot access the Reports menu in Widgets settings. On having Read-only permissions for Reports, the user can only view the generated reports or template, but cannot create a new one from Report Management UI.

This is categorized into the following sections:

• Metrics
• Detailed
• Ready
• Templates

Metrics

This section is further categorized into following sub-sections:

• All Reports
• Performance Stats Report
• Compare Report
• Excel Report
• Hierarchical Report
• Summary Report

All Reports

This section displays all reports with the details, such as report type, report name, template name, created by, created on and action. The user can edit or delete a report from here. To add a new report, the user is required to click the  icon. The process for report creation is described in the subsequent sections.

Performance Stats Report

Performance Testing is defined as a type of software testing to ensure software applications will perform well under their expected workload. Once the performance test is executed, the user can generate its report by using the Cavisson Reporting feature. For this, first, the user needs to define the configurations for the performance stats report and then needs to generate it. Once the performance stats report is generated, the user can view it for a detailed investigation of the results.

Cavisson performance stats report is broadly categorized into two sections, Performance Report and Metrics Report.

• Performance Report is further categorized into Analysis Summary and Performance Summary. The Analysis Summary report contains details, such as overall analysis summary, executive summary, and statistics summary. The Performance Summary contains details, such as overview of the test, sessions, transactions, pages, network, and errors.
• Metrics Report is further categorized into Metrics Summary and other charts, such as URL, Page, Transaction, Session.

Configuring Performance Stats Report

To define the configurations of a performance stats report and generating it, the user needs to follow the below mentioned steps:

1. Go to Reports > Metrics > Performance Stats Report.

Note: The user can also open the performance stats report by clicking Reports > Metrics > All Reports. Within the All Reports(s) window, the user needs to click the  icon and select Performance Stats as the Report Type.

2. Provide the inputs (as mentioned in the subsequent sections) and click the Generate Report

General Settings

• Report Type: It is the type of report that is to be created. The available options are Stats, Compare, Excel, Hierarchical, Summary, and Performance Stats. Select Performance Stats from the drop-down list.
• View Type: It is the view type of the report. For a Performance Stats Report, the default view type is HTML. Other view type includes – Word, Tabular, and PDF. The PDF view type is only visible for metrics option selected as favorite.
• Report Name: It is the name of the report. Provide a valid report name.
• Report Settings: Here, the user can select the reports that are required to be displayed, such as Analysis Summary, Performance Summary, Metrics Summary, and Metrics Charts.

Preset Options

• Preset: The user can fetch the transactions for the whole test run or for a specified duration, such as last 1/2/4/6/12 hours and so on. User can also apply custom time on selecting the Custom option from the drop-down list.

• Show Discontinued Metrics: This enables a user to get the data of discontinued metrics too while generating a Performance Stats report. Discontinued metrics are those metrics whose data is dropped in between of the test run. These metrics are skipped in the current view of graphs. It is neither displayed on the graph panel nor displayed in the tree structure. All such metrics can be displayed by selecting the Show Discontinued Metrics option.
• Start Date / Time and End Date / Time: On selecting the Custom option, two options are displayed further for the selection of time – Absolute and Elapsed. Absolute time is the exact time. Elapsed time is simply the amount of time that passes from the beginning of an event to its end. In case of Absolute, enter start date, start time, end date and end time. In case of Elapsed, enter start time and end time. The specified date/time should be in range with current session.

‘View By’ Settings

• View By: In the View by option, the user can specify the format of aggregated data (such as hours, minutes, and seconds) based on the selection of Preset option. For example, if user selects ‘Last 15 minutes’ in preset option, then ‘View by’ option contains ‘Seconds’ and ‘Minutes’ only. If user selects ‘Last 4 hours’ in preset option, then ‘View by’ contains ‘Seconds’, ‘Minutes’ and ‘Hours’. On selecting the Auto option, the view by value is automatically considered based on the progress interval defined in scenario settings.

Note: The last sample received in a progress interval is dumped into RTG and displayed to the user.

• Value: Based on the View By selected, the user needs to specify the value for View By option.

Metrics Option

Metrics is the source of capturing the report data.

There are various options available to select the metrics:

• All Metrics: On this selection, all metrics are captured for reporting.
• Selected Metrics: On this selection, the user needs to specify certain other details, such as chart type, graph type, Report set name, group metrics, and graph metrics.
• Templates: On this selection, the user needs to select the template from the drop-down list.

• Favorite: User can generate the report from a favorite also. To do this, select the Using Favorite option and select the favorite from the drop-down list for which the report is to be generated.

• Include Charts: To include charts in the report, select this check box.
• Override Charts: On selecting this, the original charts are replaced with the chart type selected from the drop-down list.

Filter by Value Option

The user can discard the un-desired graphs in the report by using the ‘Filter by Value’ option. To enable the filters, the user first needs to select the Enable Filter check box.

There are following filter options:

• All Non- Zero: Upon enabling this, all non-zero graphs are involved in the report.
• All Zero: Upon selecting this, all zero graphs are involved in the report.
• Advanced: Upon selecting this, user can have some advance level options based on value, such as <, >, <=, >=, Top, Bottom, In between, which helps to achieve the desired output. User can filter graph values by Min, Max, and Avg value of its graph data. User can also use the Include or Exclude options to get the filtered results. Include/exclude options includes/excludes the graphs that lies within the specified values. For example – if we try to find the graphs that has values between 50-100, then on include, system displays such graphs that has values from 50-100 and exclude option skips those results that contains graph values from 50-100 and displays the rest graphs. To enable the filters, user needs to enable the toggle button.

Threshold File Option

• Threshold Template: This is the default template created in csv format to define the threshold values for various fields of the report, such as Min, Max, Avg, standard deviation, and various percentile values. The color-coding of these fields in the report is defined based on the threshold values specified in this file. First, the user needs to select a template from the drop-down list, then download this template, and define the object name and module on which the color-coding is to be applied. Then, based on a field, such as Avg, Min, Max, the user can define the values for minor, major, and critical separated by a space (for example, 400 800 950). The color-coding is applicable in the below sequence.
  • First value is for Minor threshold and is represented by Green
  • Second value is for Major threshold and is represented by Yellow
  • Third value is for Critical threshold and is represented by Red

Here is a sample format of the threshold template:

Note:

• • The user can also use a regular expression (regex) pattern in the transaction name.
  • The user can specify operators, such as >, >=, <, <= for comparing data value with threshold values.
  • The column names in the threshold template should always be the same as mentioned: “Metric, Object, Module, Avg, Min, Max, P50, P80, P90, P95, P99, Operator”
  • An option is provided in scenario to select Threshold Template in auto generated html report after test completion.
• Download: To define the values, the user first needs to download the threshold template.
• Upload: Once the threshold values are specified in the template, the user needs to upload the threshold template.
• Delete: The user can delete a template if this is not required.

The PerformanceDefaultThresholds threshold is the default template.

You can edit and upload this template as per your requirements.

Summary

• Executive Summary (Optional): It is a short document or section of a document, produced for business purposes, that summarizes a longer report or proposal or a group of related reports in such a way that readers can rapidly become acquainted with a large body of material without having to read it all.
• Conclusion (Optional): A conclusion is not merely a summary of the main topics covered or a re-statement of your research problem, but a synthesis of key points and, if applicable, where you recommend new areas for future research.

Once all the details are filled and the user clicks the Generate Report button, the report gets generated in the background and following message is displayed. The user can perform any other activities by the time the report is generated.

The reports, which are getting generated, are displayed with different icons as highlighted.

Once the Performance Stats Report generation is completed, a notification is displayed to the user that the report is ready.

Viewing Performance Stats Report

The Performance Stats report or Analysis report enables a user to have a detailed investigation about the test based on the configuration applied while creating the performance stats report. It also provides a transaction summary where the user can view the summary of all or individual transactions in tabular and graphical format. The color-coding on the fields in the transaction summary report (tabular) is based on the threshold values applied for each category (minor / major / critical). Performance Stats Report can be generated for multiple data centers.

To view the performance stats report, go to Reports > Metrics > Performance Stats Report.

Select a performance stats report from the list. This displays the report based on the ‘View Type’ as HTML, PDF, Word, or Tabular.

View Type – PDF

View Type – Tabular

View Type – Word

View Type – HTML

You can view the detailed performance report with various aspects. This window is divided into two panels. From the left panel, you can select the report type and its corresponding details is displayed on the right.

On the left panel, this window is further categorized into two sections – Performance Report and Metrics Report. The description of these topics is provided in the subsequent sections.

Performance Report

Performance report displays the overall performance of the test. It is further categorized into two sections:

• Analysis Summary
• Performance Summary

Analysis Summary

The Analysis summary is used for analyzing the test results. This displays various type of summary of the test that includes summary of configurations applied for the report generation, executive summary, statistics summary, and conclusion of the performance test.

• Analysis Summary: This section displays the summary of analysis with the below details:
  • Scenario Name
  • Test Name
  • Test Run Number
  • View By
  • Report Time Period
  • Phase Name
  • Test Run Start Date Time
  • Test Run End Date Time
  • Duration (hh:mm:ss)

• Executive Summary: This displays the executive summary provided at the time of specifying inputs for generating the performance stats report.

• Conclusion: This displays the conclusion provided at the time of specifying inputs for generating the performance stats report.

• Statistics Summary: This section displays the statistical summary of the performance stats report with the following details:
  • Max Running Virtual Users
  • URL Average Time (ms)
  • URL Total
  • URL Success
  • URL Failures
  • Average Session/Minute
  • TCP Received Throughput (Kbps)
  • TCP Sent Throughput (Kbps)

Performance Summary

This section is further divided into following sub-sections:

• Overview
• Sessions
• Transactions
• Pages
• Network
• Errors

The details of each is provided in the subsequent topics:

Overview: This section displays the overall summary of the test that includes the sections, such as Load Profile, Test comment, General information, Performance summary, and Network.

• Load Profile: This section displays two sub-sections, the first section contains Scenario type and Schedule type (group name, user profile, script name, users/session rate, percentage, and arrival rate) and second section contains scenario phases (start, ramp-up, stabilization, duration, ramp-down, and session pacing).

Note: In case of FCU, the User(s) column is displayed and in case of FSR, the Session(s) column is displayed.

• Page Think Time: The user can view page think time for all scenario types if added in the scenario.

• Test Comment: This displays the test name and test notes.

• General Information: This displays the test duration, stat time of test, and end time. It also displays certain graphs, such as graphs of concurrent users, request per second, and transaction errors. One graph is displayed as line with all samples and second is for 95^th percentile or which is configured.

• Performance Summary: This displays total count, count per second, count per hour, and count per day for URL, Pages, Transactions, and Sessions.

The description of fields are as follows:

Note: The user can configure the percentile from Dashboard. For this, go to Configurations > Configuration Settings > Dashboard Settings and click the Percentile tab in the Configuration Settings window.

• Network: This displays the total numbers, numbers per second, per hour, and per day of Requests, Bytes sent and Bytes received. This data is generated from Send/Received Throughput (VUser), Total = (Rate * Duration (Sec)) /8.

Sessions: A session is an executed and completed test case. Each Session consists of one or more transactions. Session Summary Includes session wise statistic within the particular test, sessions per sec, Errors, runtime (ms), Runtime Percentiles (ms). The Summary chart section comprises all data across sessions to allow for a quick overview on session problems and performance. In the following, all chart sections below visualize this information:

• Overview: Sessions Response Time, 95th Percentile and Errors.
• Averages: Sessions Response Time, 95th Percentile and Errors.
• Counts/s: number of completed Sessions per second
• Arrival Rate: Arrival rate per hour, 95th Percentile
• Concurrent Users: number of Active Users

The description of fields are as follows:

Within this summary table, there are multiple tabs with different charts.

• Overview: This displays different graphs for Session Time with 95th Percentile, Session Time with all samples, Session Error count, and Session Error % and Slab count chart.

• Average: This displays the 95^th Percentile, Median, and Mean graph of Session time.

• Count / s: This displays the 95^th Percentile and per second Session count.

• Arrival Rate: This displays the current Sessions Per hour and its 95^th

• Concurrent Users: This displays the active Vusers.

Note: The user can see the stats of each session, such as overview, average, count/s, arrival rate, and concurrent users by clicking a transaction name.

Transactions: Transaction is part of a session and usually consists of one or more pages. When testing web applications, an action resembles a page view. The data shown here reflects the time spent on the execution routine of a transaction. Therefore, the transaction’s runtime includes the runtime of a page (an HTTP operation, for instance) and the time required to prepare, send, wait for, and receive the data. The error count defines the number of errors detected for a particular transaction such as timeouts or connection resets.

The Summary chart section comprises all data across Transactions to allow for a quick overview on transaction problems and performance. The tabs in this chart section contain single charts that refer to one transaction.

In the following, all chart sections below visualize this information:

• Overview: Transactions Response Time, 95th Percentile and Errors
• Averages: Transactions Response Time, 95th Percentile and Errors.
• Counts/s: Number of completed transactions per second.

This section displays transaction name, total count, count per second, per hour, per day, total errors, errors %, runtime in millisecond, and runtime percentile in milliseconds.

Within this summary table, there are multiple tabs with different charts.

• Overview: This displays different graphs for Transaction Time with 95^th percentile, Transaction Time with all samples, and Transaction Error count as bar chart.

• Average: This displays 95^th percentile, Median, and Mean graph of Transaction time.

• Count/s: This displays 95^th percentile and per second transaction count.

Pages: This displays Page name, Total Page count, Page count per seconds, per hour, per day, Errors, Runtime in milliseconds, Runtime Percentiles in milliseconds and Runtime Segmentation in milliseconds. This section if further divided into three sub-sections: Overview, Bandwidth, and Network Timing.

• Overview: This displays count, errors, runtime, runtime percentiles (ms), and runtime segmentation (ms).

• Bandwidth: This displays Total, 1/second, 1/hour, 1/day, avg, min, and max of bytes sent and bytes received of each page.

• Network Timing: This section displays DNS Time, Connect Time, Send Time, Server Busy Time, Receive Time, Time to First, and Time to Last.

• Network: This displays Total number, numbers/seconds, numbers/hour, and numbers/day of Requests, Bytes Sent, and Bytes Received. In case of Byte Sent and Bytes Received, Total = (Rate * Duration (Sec)).

This section also displays certain charts, such as requests per second, TCP send throughput, and TCP receive throughput.

• Hosts: This displays host name, count, and percentage.

• HTTP Response Code: This displays HTTP Response codes, count, and percentage.

• Errors: This displays URL Errors, Sessions Errors, Transaction Errors, and Page Errors in a chart for a time period.

• URL Errors

• Sessions Error

• Transaction Errors

•  Page Error

Metrics Report

This section displays the metrics summary report and individual description of each reports (with separate link) based on the configurations. Here is an example of reports created based on certain metrics, such as Generator CPU Utilization (%), Generator Available Memory (MB), Generator Ethernet Throughput, Generator Load Average Over 1 Min, and Generator TCP Stats.

Note: By clicking on an individual report, the user can view its details.

The fields that are displayed are as follows:

• Min: This is the minimum value of graph in a particular metrics.
• Max: This is the maximum value of graph in a particular metrics.
• Avg: This is the average value of graph in a particular metrics.
• Dev: This is the standard deviation value of graph in a particular metrics.
• Last: This is the last value of graph in a particular metrics.
• Samples: This is the total number of samples.

Key Points

The user can navigate to the graphical representation of any individual transaction by clicking the transaction name.

The user can navigate to the table by clicking the Back to table link.

Any failed transaction is represented with Red color in the Errors column.

The user can also generate the performance stats report from Scenario after the test completion. For this, open the Scenario, go to Logs and Reports and provide the inputs in Reports section.

The user can show / hide the left panel by clicking the  button at the top left corner.

The user can search a string by using the Search section. This is applicable in case of Sessions, Transactions, and Pages within the Performance Report section.

Note:

Display of ‘Data’ Value and ‘Percentage’ Value

Data value and percentage value is displayed with legend in case of ‘Pie’ and ‘Donut’ chart.

‘Last’ Option in Report Management Window

A ‘Last’ option is displayed in the drop down list of Formula Type for Pie and Donut chart type while creating Template from Template UI and from Report Management UI using Select Metrics.

Compare/Trend Report

Compare Report compares two or more Test runs or releases. Trend Report shows the trends of configured monitors over a specific time period. Compare/Trend Report gives an insight about how a particular website or application has performed during a particular trend period chosen by user.

Creating a Compare Report

Follow the below mentioned steps for creating a compare report:

1. On the Compare report, click the  icon.

2. Enter the report name.
3. Specify the View Type, such as Tabular/Word/HTML (same as mentioned in the Stats report).
4. Specify the View by
5. Select the template from the drop-down list.
6. Enter the measurement name. Measurement name is a unique name which is assigned for the test run measurement.

User can also set the Preset value and Time format.

7. Then, click the Add button, a new row will be added.

8. Specify the Measurement name, select the test run (for which comparison needs to be done), and specify other options, such as Preset, time format. Click the Add button.

User can add multiple test runs for comparison by following this process (Step 6). Measurement which is selected will include in report. User can update/delete a measurement also.

9. Click the Generate Report button, the report gets generated as displayed.User can add further columns, such as Minimum and Maximum Value and Max sample timestamp in Compare Report.

In Compare Report, a heading named ‘Selection Mode’ is provided with a check box. This appends a check box preceding to each metric name, so that the user can perform further operations, such as view selected graphs, show selected, hide selected and so on.

User can apply filter on the report. User can download the report in PDF, Excel, and Word format.

Creating a Trend Report

Follow the below mentioned steps for creating a trend report:

1. On the Compare report, click the icon.

2. Select the Trend Compare check box.
3. Enter the report name and select a template from the drop-down list.
4. Next, select the project, sub-project, and scenario. Click the Apply The measurement details will be filled automatically.

5. Select the measurement name for compare, and click the Generate Report button. The report gets generated and displayed.

Note: The baseline and original graphs are represented with different colors when a user performs comparison with trend baseline. This helps the user to distinguish between both the graphs.

Output

Transaction Time (Secs) Trend

Tabular View of Trend Report

Trend Report for Data of 95th Percentile

The trend compare report is generated with 95^th Percentile value in case of Avg, Min, and Max data values. Table in the generated report displays all the selected percentile values but chart is formed based on 95^th value. 

For graph type as ‘percentile graph’, the values are 95^th percentile.

Percentile Values:

One Page View

Excel Reports

An Excel template is a report layout that you design in Microsoft Excel for retrieving and formatting your enterprise reporting data in Excel. Excel templates provide a set of special features for mapping data to worksheets and for performing additional processing to control how your data is output to Excel workbooks.

Creating Excel Based Templates

To create a template, create two sheets, one for Formula and another for Actual Report and save them as template. The formula sheet captures the data from graphs, perform mapping with report sheet, and post the data on the reports sheet. User needs to create certain columns and specify the formula for respective columns. A formula is a collection of Graph instances. The report sheet is called when scheduling is performed via portal.

User can change the layout of the Report sheet in the template also, such as color coding, setting labels, setting threshold values and so on based on requirements.

In a template, some headers are optional, such as Page Name, Metrics, Competitor, and Browser while others are mandatory to create, such as DC, Formula, Type, Graph Type, Decimal, Value 1, Value 2…., and so on.

Creating a Formula

User can get a formula either from derived graph feature of dashboard or navigating to the tree structure of the graph manually i.e. tier>server>instance in the left pane of the Dashboard GUI.

Creating a Formula Manually

To create a formula manually, use the following syntax:

• Group ID.Graph ID.[Vector Name] For example – 107.9.[Cavisson>NDAppliance]
• {Group Name}{Graph Name}[Vector Name] For example – {SysStats Linux Extended}{Process Waiting For Run Time}[Cavisson>NDAppliance]

Getting Formula from Derived graph

1. Open the Dashboard GUI, right-click on Custom Metrics and click Add Derived Graph. The Derived graph window is displayed.

2. Select the Group name, Graph name, and Indices.
3. Click Add Graph. The formula is displayed.
4. Copy the same and paste on the Formula sheet under Formula column.

The graphs can also be added via providing Group ID and Graph ID. Format: Group ID.Graph ID.[Vector Name]

Working with Formula Sheet:

1. Mention the User name, current date time, report time period, and Aggregate.
2. Create the column headers as Page name, Metrics, Competitor, Browser, and DC.
3. Next, specify the Formula In this column, specify the Graph path.

Group ID.Graph ID.[Vector Name] or {Group Name}{Graph Name}[Vector Name]

4. Create the Type This stores the type of value required (i.e. Min/Max/Avg and so on). Enter the value based on your requirement.
5. Create the Graph Type Mention ‘Derived’ by default.
6. Create the Decimal It specifies up to what decimal places, result is required.
7. Create the columns in which the graph result is stored. These columns are prefixed with ‘Value’, such as Value1, Value2, Value3, and so on. The Value columns are created based on the specification of Aggregation.

For example, if user wants to see the data of last one week and aggregated it by one day. So, seven value columns need to be created. Each value column will capture the aggregated data of one day.

Specify the column name in sequence. There can be multiple formula sheets, each formula sheet is mapped with one report sheet.

Working with Report Sheet

Report sheet contains the mapped results from formula sheet. Suppose, the data of last one week needs to be captured and it is aggregated by one day. So, seven value columns are created and mapped with the value columns in the report sheet.

Creating an Excel Report

Follow the below mentioned steps for creating an Excel report:

1. On the Excel Report window, click the icon.

2. Enter the report name, specify the preset options, and select the template from the drop-down list. The Override Preset from Template option is used to take time from excel template rather than preset option in report UI. Once override preset option is selected then time applied in the preset of report UI becomes null and void.

To upload a template, click the Upload button. To download a template, click the  icon. To delete a template, click the  icon.

3. To aggregate the entire report in a single column, select the Aggregated over the entire Report Time Period check box. For example, if the user wants to capture the data of the last one week and select this check box, then the aggregated data will be displayed in a single column.

4. If Show Last week’s data day-wise option is selected and user needs graph data for two weeks aggregated by week-wise, then 8 value columns will be created one for a complete week data and rest seven for other week data day-wise.

5. To show reports in either Column or Tab, select the same from Show report in If user selects the column option, then graph data will be captured in columns and if user selects the tab option, the graph data will be displayed in separate sheets/tabs.

6. Click the Generate Report button, the report gets generated and displayed on the Report Management

7. To view the report, click on the report link. The report gets displayed.

User can now generate excel Report for more than two DCs. For this, user needs to use configuration with IP and DC name, such as <IP>: <DC Name>.

Following settings need to be done in $NS_WDIR/webapps/sys/config.ini

netstorm.executivedashboard.ndclient.datacentername = MOSAIC

dashboard.excelReport.ip=65.49.37.719:MosaicAccount,65.49.37.71:MosaicCNC,65.49.37.71:Mosaic_Solr,209.51.175.196:SCS,209.51.175.196:PKY

dashboard.excelReport.port=4431,4431,4431,4469,4470

netstorm.dashboard.KPIClient.UseSecureConnection=1

The excel report can be generated with Last value of the graph. The user can generate this by selecting ‘LastValue’ in ‘Type’ column of excel template.

Note: The default status of ‘Show Discontinued Metrics’ check box is enabled/selected in Excel report. This enables a user to get the data of discontinued metrics too (by default) while generating an Excel report.

Hierarchical Reports

This report is based on the meta data component. There will be multiple drop-downs up to maximum level of hierarchy in test run. For example, if there are maximum 5 levels of meta data in test run, then there will be 5 drop-downs. Each drop-down will be filled by unique list of specific level of meta data component. Also drop-down should be filtered by the previous selection. Next drop-down is enabled only after selecting any value in the current drop-down.

Creating a Hierarchical Report

Follow the below mentioned steps for creating a hierarchical report:

1. On the Hierarchical Report window, click the  icon.

2. Enter the report name and specify the preset options.
3. Select a template along with other options (if required).
4. Select the meta data details (such as Tier > Server > Instance) and click Generate Report The report gets generated and displayed.

Summary Report

Summary report contains the summarized information of the test, such as test summary, transaction details, HTTP status code details, and other important information. Summary report contains some predefined sections and headers which are more relevant.

Generating a Summary Report

Follow the below mentioned steps for generating the summary report:

1. On the Report Management window, right-click on Summary, and then click New Report.

2. Enter the report name and specify the report percentile from the drop-down list.
3. Specify the preset options. User can select multiple duration in summary report.
4. Click the Generate Report button. The report gets generated. Click the report name to view its details

There are following section in the Summary report:

• Test Information: This section contains information on test run, scenario, machine, and so on.
• Test Summary: This section contains the following information:
  • Transaction: Number of transactions in the test run.
  • Session: Number of sessions in the test run.
  • Total Vuser: Describes the total number of users participated in the test run.
  • Page Download Completed Metrics: Describes the number of page download completed after the test run is completed
• Other Information: This section contains information on TCP connections, SSL Sessions, HTTP hit rate, and throughput.
• Detail Transaction Report: This section contains transaction details, such as Min, Max, Avg, success, Failure, and so on.
• HTTP status Codes: This section contains HTTP status code that are occurred in the test run.

Summary Report with multiple selection presets

There’s a provision to generate Summary Report for multiple presets. Example: if user wants to generate Summary Report of all duration phases of Test Run like Duration1, Duration2, and so on, then user needs to select multiple duration from preset drop down list and generate Report.

For example, in below screen, we have selected three durations, such as Duration 3, 4, and 5 of Group-1.

Generated Report: Upon clicking the Generate Report button, the stats for the selected duration is generated and displayed in a report.

Detailed

This section displays some detailed reports, such as Drill down reports. The description of each topic is provided in the subsequent sections.

Drill Down

Here, the user can view the available drill down reports, such as:

• URL
• Session
• Transaction
• Page

URL

This report displays URL summary, such as URL name, number of tries, number of success, number of failures, minimum duration, average duration, maximum duration, median, 80^th percentile, 90^th percentile, 95^th percentile, 99^th percentile, percentage fail, and script count. 

Further Reports in this Section

The user can view further reports from this window by clicking the specified links.

• To view URL instance report for tried attempts, click the Tried link.
• To view URL instance report for successful attempts, click the Success link.
• To view URL instance report for failed attempts, click the Faillink.
• To view URL Component details, click the Averagelink.
• To view URL Session Summary, click the Script Countlink.

Session

This report displays Session summary, such as Script name, number of tries, number of success, number of failures, minimum duration, average duration, maximum duration, median, 80^th percentile, 90^th percentile, 95^th percentile, 99^th percentile, and fail percentage.

Further Reports in this Section

The user can view further reports from this window by clicking the specified links.

• To view the script file in script manager, click the Script Name link.
• To view Session instance report for tried attempts, click the Triedlink.
• To view Session instance report for successful attempts, click the Success link.
• To view Session instance report for failed attempts, click the Fail link.
• To view Session details report, click the Average link.

Transaction

This report displays Transaction summary, such as transaction name, minimum duration, average duration, maximum duration, median duration, 8^th percentile, 90^th percentile, 95^th percentile, 99^th percentile, script count, number of tries, success, fails attempts, and fail percentage.

Further Reports in this Section

The user can view further reports from this window by clicking the specified links.

• To view Transaction details report, click the Average link.
• To view the Transaction session summary, click the Script Count link.
• To view the transaction instance for successful transactions, click the Tried link.
• To view the transaction instance for failed transactions, click the Fail link.

Page

The user can also view the Page Summary report from Drill Down Queries window. This window displays the following page summary details: Page name, number of tries, success, and failures. It also displays minimum, maximum, average, and median duration of the page along with 80^th percentile, 90^th percentile, 95^th percentile, 99^th percentile, fail percentage, and script count.

Other Reports from this Window

• To view the page instance report, click the Tried link.
• To view the page failure report, click the Fail link.
• To view the page component detail report, click the Average link.
• To view the page session summary report, click the Script Count link.

Ready Reports

Ready reports are those reports that do not require any process for generation. The user just need to click the respective report on the Reports menu and corresponding report is displayed:

• Page Detail Report
• Lighthouse Report
• Detailed Report
• Failure Report
• Page Dump Report
• Page Breakdown Report
• Progress Report
• Detailed User
• Scheduler

Page Detail Report

Click the Page Detail menu item on the Report menu. The Page Detail Report is displayed.

Lighthouse Report

On clicking the Light House Report, the following window is displayed:

This window displays following details:

• Page name
• Host name
• Group
• Session Count
• Performance Score (%)
• Accessibility Score (%)
• Best Practice Score (%)
• First Meaningful Paint (sec)
• Time to Interact (sec)
• First CPU Idle (sec)
• Speed index (ms)

On clicking the page name link, a detailed report is displayed.

Failure Report

Click the Failure Report icon on the left pane. The Failure report is displayed.

Page Dump Report

Click the Page Dump Report on the left pane. The Page dump report is displayed.

Page Breakdown Report

Click the Page Breakdown Report menu item to view this report.

Progress Report

Click the Progress Report icon on the left pane. The Progress report is displayed.

Detailed User Report

On clicking this link, page average report is displayed where the user can see the page name, host name, group, session count, DOM content load time in seconds, onload time in seconds, page load time, number of requests, browser cache, bytes received and sent.

Scheduler

User can schedule a report by specifying certain information. Scheduling is performed to generate reports on continuous intervals. For scheduling a report, the user first needs to create a task list.  Once the task is created, it can be scheduled, and the generated reports can be viewed.

In the Reports window, go to Ready Reports and click Scheduler.

This contains the following tabs:

• Scheduler Management: The list of all the tasks is displayed and the user can schedule a task from here.
• Create Task: To create a new task.
• Generated Reports: The list of generated reports is displayed

Creating a Task

To create a task, follow the below mentioned steps:

1. Go to the Create Task tab.

2. Provide the following details:

• Report Type: Type of the report, such as Excel, Performance Stats report, or Alert Digest. Based on the report type, select the format (Word / Excel / PDF). Some of rest fields are common for all report types, but some may vary based on the report type selected.

 Report Type – Excel

Report Type – Performance Stats

Report Type – Alert Digest

• Report Name: Name of the report. Report name should be meaningful for reference.
• Template / Favorite: Choose the template / favorite from the list based on the selected option.

Note:

• The ‘Using Favorite’ option is displayed in the case of ‘Performance Stats Report’ only.
• After template creation, the user needs to upload it to the system. The uploaded templates are displayed in the list.
• To upload a template, click the Upload button and browse the template.
• To download a template, select it from the list and then click the button.
• To delete a template, select it from the list and click the button.
• On selecting HTML option in the Performance Stats Report, a ‘Report Setting’ icon is displayed from where the user can select which sections to be displayed in the report.

• Description: Description of the report.

3. Click Next and provide the Report Parameters. Different parameters are displayed based on the Report type.

For Excel Report

• Select the Preset option from the drop-down list. It is the time period for which the report needs to be generated. For example – Last 15 minutes, Last 8 hours and so on. The date and time are filled automatically.

If user selects the Custom option, then needs to specify the Start date/time and End date/time.

• In the View by option, user can specify the format of aggregated data (such as hours, minutes, seconds) based on the selection of Preset option. For example, if user selects ‘Last 15 minutes’ in preset option, then ‘View by’ option displays ‘Seconds’ and ‘Minutes’ only. If user selects ‘Last 4 hours’ in preset option, then ‘View by’ displays ‘Seconds’, ‘Minutes’ and ‘Hours’.
• Override Preset from Template: Override preset from template option is used to take time from excel template rather than preset option in report UI. Once override preset option is selected then time applied in the preset of report UI becomes null and void.
• To aggregate the entire report in a single column, select the Aggregated over the entire Report check box. For example, if user want to capture the data of last one week and select this check box, then the aggregated data is displayed in a single column.
• The user can also aggregate the data based on the desired format (minutes / hours / day / week / months). For example: To view the graph data for two weeks aggregated by week-wise, then 8 value columns are created one for a complete week data and rest seven for other week data day-wise.
• To show reports in either Column or Tab, select the same from Show report in option. If user selects the column option, then graph data are captured in columns and if user selects the tab option, the graph data is displayed in separate sheets/tabs.
• To view the data of those metrics whose data is dropped in between of the test run, select the Show Discontinued Metrics option.

 For Performance Stats Report

• Preset: The user can fetch the transactions for the whole test run or for a specified duration, such as last 1/2/4/6/12 hours and so on. User can also apply custom time on selecting the Custom option from the drop-down list.
• To view the data of those metrics whose data is dropped in between of the test run, select the Show Discontinued Metrics option.
• Start Date / Time and End Date / Time: If user selects the Custom option, then needs to specify the Start date/time and End date/time.
• View By: In the View by option, the user can specify the format of aggregated data (such as hours, minutes, and seconds) based on the selection of Preset option. For example, if user selects ‘Last 15 minutes’ in preset option, then ‘View by’ option contains ‘Seconds’ and ‘Minutes’ only. If user selects ‘Last 4 hours’ in preset option, then ‘View by’ contains ‘Seconds’, ‘Minutes’ and ‘Hours’. On selecting the Auto option, the view by value is automatically considered based on the progress interval defined in scenario settings.

Value: Based on the View By selected, the user needs to specify the value for View By option. 

• Include Metrics Charts: This option enables the charts in the generated report.
• Add Graph Image: This option enables the images with in the email itself.
• Number of Report Set: It will show the number of report sets user wants to display.

For Alert Digest Report

• Preset: The user can fetch the transactions for the whole test run or for a specified duration, such as last 1/2/4/6/12 hours and so on. User can also apply custom time on selecting the Custom option from the drop-down list.
• Start Date / Time and End Date / Time: If user selects the Custom option, then needs to specify the Start date/time and End date/time.

4. Enter Emailing Information.

• Enter the mail subject.
• Enter recipients of the mail.
• Enter the description in the Body section.

Note:

• • User can pick Report name, Start date, and End date from the Hints section. This automatically fills the specified information in the mail body.
  • A predefined variable is provided in Scheduler Report as ‘GenerationDateTime = $genDateTime’, which is used to pass the Date and Time when the report is generated. This predefined variable can be used in Subject or Body of the mail and is available in Hints section in Scheduler Report window. The date time format is in MM-DD-YYYY HH:mm:ss.

5. On clicking Next, the Summary window is displayed that contains details about task description and mail parameters.

6. Click Finish, the task is displayed in the Schedule Tasks list. From here, the user can schedule a task, update a task, and enable/disable a task.

There are following headers in Schedule Tasks page:

• Report Type: Type of the report. For example – Word / Excel / HTML report
• Description: Description of the task.
• Schedule Time: Schedule time for the task execution
• Expiry Date Time: Schedule expiry time for the task
• Status: Status of the task, either enabled or disabled
• Actions: User can schedule / delete / update / disable / enable the task.

Update a Task

To make changes in the created task, the user can update it by selecting the task and clicking the Update Task button () within the Action section.

Enable / Disable a Task

To enable / disable a task, select the task and click the Enable Task () / Disable Task () buttons respectively. 

Note:

• A task can only be scheduled when it is in Enabled state.

Scheduling a Task

Once a task is created, it needs to be scheduled for the report generation. Follow the below mentioned steps to schedule a task:

1. Click the Schedule Task button () within the Action section. This displays the task scheduler section.

2. There are multiple options displayed on the form of tabs for which the task can be scheduled – Minutes, Hourly, Weekly, and Monthly. Various further options are displayed based on the duration selected.

Minutes

Mention the duration (in minutes) and schedule expiry date. The report is generated after every specified minutes until it reaches the specified schedule expiry date and time.

Hourly

There are two options within this – Hourly and Daily.

• For Hourly, specify the duration (in hours) and offset (if required). The offset here means the interval. At what time, the user wants to generate report by using offset option. For example – if user wants to generate report in every 1 Hour 2 Minutes.

Add OffSet In Preset Time: On selecting this option, the Offset time is added in the preset time, which is not included if the user does not select this option.

For Example: Suppose the user has scheduled a task in which the preset is last 8 hour and has added a scheduled time of every 8 hour with offset value 2 minutes. Now, the task execution time is 08:02, 16:02, and 24:02 and preset time is 00:02:00 to 08:02:00, 08:02:00 to 16:02:00, 16:02:00 to 24:02:00.

On selecting this option, the offset time is added in the preset time and it becomes 00:02:00 to 08:02:00, 08:02:00 to 16:02:00, 16:02:00 to 24:02:00.

On not selecting this option, the offset time is excluded in the preset time and it is not impacted. So, in that case, it becomes 00:00:00 to 08:00:00, 08:00:00 to 16:00:00, and 16:00:00 to 24:00:00. 

• For Daily, specify a time (in HH:MM format) at which the daily report is to be generated.

Weekly

• Select the day from the Week Days options and specify the start time. The report is generated on the specified day of the week and at specified time.

Monthly

• In the ‘Day’ field, mention the day on which the report is to be generated.
• For specified months like every month, every two months, three months or so, mention the value accordingly within the ‘Months’ field.
• Mention the start time at which the report is to be generated based on the day and month selected.

Viewing Generated Report

Once the schedule has been created, the record is displayed in the Scheduler Management section.

Once the report is generated based on the schedule, it is displayed within the Generated Report section.

Click a report to view the details. The report is downloaded in its specified format.

The report is sent to the email as specified. Below is a sample format.

Sample Alert Digest Report Link

Alert Digest Report represents the count of all the alerts, which are generated within the provided time period of a test run based on grouping option i.e. Vectors or Alert Rule Name. Along with this, Alert Digest Report is also supported for Tier-Wise Alert count. The user needs to mention ‘$$Tier$$’ for Group By value in Alert Digest Template.

Sample Alert Digest Report

Scheduler Report for Multi DC

The user can schedule report for multiple DCs at once. Scheduling report for multiple DCs enables the user to get data from multiple data centers in a single report at desired scheduled time.

Templates

A template is a well-defined structure of information gathered from graphs.  User do not need to recreate the file each time. Once a template is created, it can be reused further. Therefore, if user finds creating similar reports repeatedly, it might be a good idea to save one of them as a template. Then, the user does not have to format reports each time to make a new one. Just use the template and start from there.

Layout of Template Management Window

Template Management window consists of two panes – left pane and right pane. Left pane consists of System templates and User templates. Right pane consists of description of the templates.

Left Pane

• System Templates: These are system defined templates. User cannot add/edit/delete such templates, but can view them only.
• User Templates: These are user defined templates. User can add/edit/delete/view such templates and can create a report(s) under them.

Right Pane

• Template Name: Name of the template.
• Report Sets: Number of reports under the template.
• Last Modification Date: Date and time when the template was last modified.
• Owner: Owner of the template.
• Template Description: Description of the template.
• Action: Actions that can be performed on the template, such as adding report ( ), edit the template ( ), and delete the template ( ).

Creating a Template

User can create a template by following the below mentioned steps:

1. On the Report Template Management window, click the icon next to available template. The New Template page is displayed.

2. Enter the following Template details:

• • Template Name: Name of the template.
  • Template Description: Description of the template.

3. Enter the following Report details:

• • Graph Type: Select the graph type, such as Normal graph, percentile graph, or slab count graph from the list. Then, select the chart type based on the selected graph type.
  • Chart Type: Graphs are most important part of any report. They need to be selected in all type of templates. All graphs are time X-axis based graphs.
    • Simple: Simple graphs have graph data on Y axis. Simple graphs have single graph on each panel.
    • Multi: Multi graphs are also called merged graphs. Multi graphs have multiple graphs on Y axis with same X axis.
    • Tile: Tile graphs have multiple simple graphs in tiled form. Tile graphs can have two or more graphs’ graphical view of data at same time in multi panel from top to bottom.
    • Correlated: These graphs show correlation with other graphs.
    • Percentile: A percentile is the value of a variable below which a certain percent of observations fall. So, the 20th percentile is the value (or score) below which 20 percent of the observations may be found. The pth percentile is a value so that roughly p% of the data is smaller and (100-p) % of the data is larger. Percentiles can be computed for ordinal, interval, or ratio data.
    • Slab count: Slab count graph is used to count number of samples in a particular time interval. User can change all graphs to Slab Count Graph. Earlier, user was able to convert only those graphs which have Percentile Data File (PDF) file associated with them.
    • Frequency Distribution: This chart is similar to slab count chart, but instead of bars, system represents the data points via dots.
    • Multi with Layout: This is similar to multi graphs, but instead of displaying all graphs on the same panel, system displays them in separate panels based on the reports. It is applicable for Word report only.
    • Bar: In this case, line chart is converted to Bars.
    • Pie: It is useful in case of multiple graphs. If one graph is converted into Pie, then a simple circle filled with one color is displayed. It is needed to take average value or Last value of graph. If graph type is cumulative then system displays sample value otherwise average value.
    • Area: An area chart displays graphically quantitative data. The area between axis and line are commonly emphasized with colors and textures. Area chart is used to represent cumulative data using numbers or percentages over time. Use the area chart for showing trends over time among related attributes. The area chart is like the plot chart except that the area below the plotted line is filled in with color to indicate volume.
    • Stacked Area: Stacked Area chart means area chart of each graph is stacked on each other.
    • Stacked Bar: In this case, graph is converted to Stacked Bar chart. Stacked bar chart means bar of each graph is stacked on each other. It is useful in case of multiple graphs. If one graph is converted into Stacked Bar, then it looks like normal Bar graph.
    • Dual Axis Area Chart:  In this type of graph, one graph is displayed as area chart and another graph is displayed as line chart. It is also known as multi-line area graph.
    • Donut Chart: Donut chart is represented in the form of a donut. User needs to specify the criteria either last or average.
    • Line Stacked Bar:  If user wants to see combined graphs of a stacked bar and line chart, then Line Stacked chart is used. In Line Stacked chart, first selected graph displays as line graph and other displays as Stacked bar graph.
    • Dual Axis Stacked Bar: This chart provides the data in the widget as one of the data being Dual axis line and another being Stacked bar.

User can select scaling on graphs, such as Multi, Tile, Percentile, Slab count, Frequency distribution, Multi with layout, Bar, Area, Stacked Area, and Stacked Bar.

• Graph Type
  • Normal: If user selects Normal graphs, then graphs are selected as it is from the Graph metrics and user cannot perform arithmetic operations on the graphs.
  • Derived: If user selects Derived graphs, then user can perform arithmetic operations on the graphs
  • Report Set: Enter a valid report set name. Report name must start with alphabet. Maximum length should be 128. Allowed characters are Alpha, numeric, space and special characters (/%()_;:,.-).
• Group Metric(s): Select the Group metrics from the drop-down list. All the available graphs are displayed under Graph metrics .
• Graph Metric(s):
  • In case of Graph type as Normal: Select one or more graphs and click the icon , the selected graphs are added on the right side under Selected Graphs area.
    

To remove the selected graph, first select the graph from the Selected Graphs area and then click the  icon.

In case of Graph type as Derived: Select a graph from the graph metrics and click the icon. Then, add any arithmetic operation (that needs to be performed) and select another graph with the same process.

This process can be repeated and user can perform any arithmetic operation (from the available operations) and create a derived graph.

To clear the selected graphs, click the    icon.

4. Click Add Report, the report gets added.

User can add any number of reports under a template by following the above mentioned process. To Edit a report, click the  icon. To Delete a report, click the  icon.

5. Filter by Value Options: User can discard the un-desired graphs in the report by using the ‘Filter by Value’ option. There are following filter options:

• • All Non- Zero: Upon enabling this, all non-zero graphs are involved in the report.
  • All Zero: Upon selecting this, all zero graphs are involved in the report.
  • Advanced: Upon selecting this, user can have some advance level options based on value, such as <, >, <=, >=, Top, Bottom, In between, which helps to achieve the desired output. User can filter graph values by Min, Max, and Avg value of its graph data. User can also use the Include or Exclude options to get the filtered results. Include/exclude options includes/excludes the graphs that lies within the specified values. For example – if we try to find the graphs that has values between 50-100, then on include, system displays such graphs that has values from 50-100 and exclude option skips those results that contains graph values from 50-100 and displays the rest graphs. To enable the filters, user needs to enable the toggle button.

6. Click the Save button, to save the template. The saved template is displayed on the Report Template Management window.