NetForest – Search

Overview

A user can interactively explore data from the Search page. The user has access to every document in every index that matches the selected index pattern. Users can submit search queries, filter the search results, and view document data. The user can also see the number of documents that match the search query and get field value statistics. If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in a histogram at the top of the page.

Setting Time Filter

The time filter restricts the search results to a specific time period. The User can set a time filter if the index contains time-based events and a time-field is configured for the selected index pattern. By default, the time filter is set to the last 15 minutes. User can use the Time Picker to change the time filter or select a specific time interval or time range in the histogram at the top of the page.

To set a time filter with the Time Picker, follow the below mentioned steps:

  • Click the Time Picker icon 7 on the NetForest toolbar at the top-right corner.
  • To set a quick filter, click one of the shortcut links.

  • To apply time filter, relative to the current time, click Relative and specify the start time as a number of seconds, minutes, or hours, days, months, or years ago.

  • To specify both the start and end times for the time filter, click Absolute and select a start and end date. User can adjust the time by editing the To and From fields.

  • Click the caret at the bottom right corner (as highlighted) to close the Time Picker. This can be applied for other sections of time period too.

To set a time filter from the histogram, do one of the following:

  • Click the bar that represents the time interval to zoom in on,
  • Click and drag to view a specific timespan. Start the selection with the cursor over the background of the chart- the cursor changes to a plus sign on mouse-hover to a valid start point.

To undo the changes, use the browser Back button.

The time range and interval are displayed on the histogram. By default, the interval is set automatically based on the time range. To use a different interval, click the link and select an interval.

Searching Data

For searching data, user first needs to click the New button, then select the index from the drop down list, and pass a query in the query bar to get the search results.

Whenever user enters a query, the result is displayed in the following layout:

Note: Data is gradually displayed as it is fetched from the NFDB. For example, if you search for seven days data, NF starts plotting data for 1st day, 2nd day, 3rd day, and so on until all seven days data is loaded.

This screen is categorized into following sections:

Histogram

Document Table

To view the result in Tabular or in JSON format, click the  icon corresponding to the result and select the Table or JSON tab respectively.

Tabular Format

JSON Format

A link is provided to view the full report (in tabular and JSON format). User can click this link and view the report in expanded form (full screen).

Selection of Fields

User can select a field from the left pane where all and popular fields are listed. To add a field, mouse-hover to that field and click add.

To remove a field from the selected fields, click the remove button on mouse-hover a field.

Searching Data

User can search the indices that match the current index pattern by entering the search criteria in the Query bar. User can perform a simple text search or use the NetForest query syntax.

When user submits a search request, the histogram, Documents table, and Fields list are updated to reflect the search results. The total number of hits (matching documents) is displayed in the toolbar. The Documents table shows the first five hundred hits. By default, the hits are listed in reverse chronological order, with the newest documents displayed first. User can reverse the sort order by clicking the Time column header. User can also sort the table by the values in any indexed field.

To search data, click the New button (at the top), enter the search criteria in the Query bar and press Enter or click Search 19 to submit the request to NetForest.

  • To perform a free text search, simply enter a text string. For example, to search web server logs, enter apache to search all fields for the term apache.
  • To search for a value in a specific field, prefix the value with the name of the field. For example, to find all the entries that contain the value 200 in the status field, enter status:200.
  • To search for a range of values, user can use the bracketed range syntax, [START_VALUE TO END_VALUE]. For example, to find entries that have 4xx status codes, enter status:[400 TO 499].
  • To specify more complex search criteria, user can use the Boolean operators AND, OR, and NOT. For example, to find entries that have 4xx status codes and have an extension of php or html, enter status:[400 TO 499] AND (extension:php OR extension:html).

Saving the Search

Saving a search enables the user to reload and use it as the basis for charts. Saving a search saves both the search query string and the currently selected index pattern.

To save the current search, follow the below mentioned steps:

  1. Click Save on the NetForest toolbar,
  2. Enter a name for the search and click Save.

Note: Whenever a new tier is added, a notification is displayed on the UI about the same

Opening an Existing Search Query

To load a saved search, follow the below mentioned steps:

  1. Click Open on the NetForest toolbar,
  2. Select the search to open.

If the saved search is associated with a different index pattern than is currently selected, opening the saved search also changes the selected index pattern.

Autosuggestion in Search

Whenever user enters a query, functions (commands and/or fields) are displayed as autosuggestions in the query bar. This enables the user to choose from the displayed functions. This saves the time of the user from typing the whole query.

Changing the Indices

On submitting a search request, the indices that match the currently-selected index pattern are searched. The current index pattern is displayed below the toolbar. To change the indices, click the index pattern and select a different index pattern.

Refreshing the Search Results

As more documents are added to the indices on searching, the search results displayed (that are used to display charts) get stale. User can configure a refresh interval to resubmit the searches periodically to retrieve the latest results.

To enable auto refresh:

  1. Click the Time Picker 7 on the NetForest toolbar,
  2. Click Auto refresh,
  3. Specify a refresh interval from the list.

When auto refresh is enabled, the refresh interval is displayed next to the Time Picker, along with a Pause button. To disable auto refresh temporarily, click Pause. If auto refresh is not enabled, user can manually refresh charts by clicking Refresh.

Filtering by Field

User can filter the search results to display only those documents that contain a particular value in a field. User can also create negative filters to exclude documents that contain the specified field value.

User can add field filters from the Fields list or the Documents table. In addition to create positive and negative filters, the Documents table enables the user to filter on whether a field is present. The applied filters are displayed below the Query bar. Negative filters are displayed in red.

To add a filter from the Fields list:

  • Click the name of the field to apply filter on. This displays the top five values for that field.

  • To add a positive filter, click the Positive Filter button 21. This includes only those documents that contain that value in the field.
  • To add a negative filter, click the Negative Filter button 22. This excludes documents that contain that value in the field.

To add a filter from the Documents table:

  1. Expand a document in the Documents table by clicking the Expand 24 button to the left of the document’s table entry.

2. To add a positive filter, click the Positive Filter button 21 to the right of the field name. This includes only those documents that contain that value in the field.

3. To add a negative filter, click the Negative Filter button 22 to the right of the field name. This excludes documents that contain that value in the field.

4. To filter on whether documents contain the field, click the Exists 25 button to the right of the field name. This includes only those documents that contain the field.

Viewing Document Data

On submitting a search query, the 500 most recent documents that match the query are listed in the Documents table. User can configure the number of documents displayed in the table by setting the discover:sampleSize property in Advanced Settings. By default, the table displays the localized version of the time field configured for the selected index pattern and the document _source. User can add fields to the Documents table from the Fields list. User can sort the listed documents by any indexed field that is included in the table.

To view a document’s field data, click the Expand button 24 to the left of the document’s table entry or double click

 

To view the original JSON document (pretty-printed), click the JSON tab. To view the document data as a separate page, click the document link. User can bookmark and share this link to provide direct access to a particular document. To display or hide a field’s column in the Documents table, click the Toggle column in table button. To collapse the document details, click the Collapse button .

Viewing Field Data Statistics

From the Fields list, user can see how many of the documents in the Documents table contain a particular field, what the top 5 values are, and what percentage of documents contain each value. To view field data statistics, click the name of a field in the Fields list.

Sharing Snapshot URL

To load a saved search, follow the below mentioned steps:

  1. Click Share on the NetForest toolbar,
  2. Copy the link for sharing. User can get the short URL using the Short URL link.

Additional Features

There are following additional features in NetForest

  • Flowpath related features
  • Search related features

Flowpath Related Features

  • Show all Logs by Flowpath ID
  • Open Transactions by Flowpath ID
  • Open NetVision Timing Report
  • Open NetVision Replay Report

 

 1.1 User needs to enter FP field in the search box to get the Flowpath related records..

Show all Logs by Flowpath ID: To view all logs by Flowpath ID, click the 29 icon on the result section. The logs corresponding to that Flowpath ID is displayed.

Open Transactions by Flowpath ID: To open transactions by Flowpath ID, click the 31 icon. The NetDiagnostics Enterprise – Flowpath report is displayed.

Open NetVision Timing Report: To open NetVision timing report, click the 33  icon.

Open NetVision Replay Report: To open NetVision replay report, click the 33 icon.

Search Related Features

  • Select any test in the log and click Search popup button to search for log entries with the same text.

  • Clicking on any field in log entry displays all logs filtered with that field value.
  • Alert option navigates the user to the alert section (for adding a new alert rule). This has been described in the Alert section.

VIS Construct

The VIS construct is supported at the NFDB side. With the help of this, you can populate the NF indexed data on the web dashboard in the form of visualization.

Visualizations: This REST is hit by the NFUI and the Web Dashboard returns the list of visualization saved in the NetForest.

REST URL format:

<origin>/unified_vis_data

Example:

https//:10.20.0.74:8000/unified_vis_data

When you hit the above URL directly, NFDB with required parameter and get the visualization response. The required information from the user end is:

jsonObject ={
     gte: 1525113000000,
     lte: 1556648939000,
interval: ‘5m’,
env: ‘prod’,
    query: ‘*|VIS count() by @timestamp[]’,
     indexPattern: ‘*’,
     metricAggregation: [  ],
     bucketAggregation: [ ]
}

Below is the sample of response (visualizations) of this REST, which is returned by NFDB after processing the VIS query:

[“New-Visualization1″,”search1″,”areachart1″,”vip1″,”chart1″,”2″,”New”,”New-Visualization”]

vis_data: This REST needs some parameters to process, which are below:

  • gte/gt: It contains start time of the dashboard window.
  • lte/lt: It contains end time of the dashboard window.
  • interval: Interval for bucketization in NetForest (day/week/month).
  • query: Query with VIS construct to get data from NFDB.
  • env: NetForest environment to get data.
  • indexPattern: From which index data will be retrieved.
  • metricAggregation: Chart metric aggregation ( [{aggType:’count’, field:’resptime’}] )
  • bucketAggregation: Chart bucket aggregation ( [{aggType:’term’, field:clientip}] )

Processing of Request

The handling of request takes place at NFDB with new rest point ‘unified_vis_data’.

REST URL format: <origin>unified_vis_data

Below are the steps of processing of the request and returning of response:

  1. Collect the vis_data provided by user for visualization.
{                               
    gte: 1436251020000,
    lt: 1594103819000,
    interval: '1M',
    env: 'prod',
    timeZone: 'Asia/Kolkata',
    query: '*|VIS count() by @timestamp[]',
    indexPattern: '*',
    metricAggregation: [  ],
    bucketAggregation: [ ]
}
  1. Design standard msearch request body from unified_vis_data.

{“body”:[{“index”:”*”},{“query”:{“bool”:{“must”:[{“query_string”:{“query”:”*”,”analyze_wildcard”:true}},{“range”:{“@timestamp”:{“gte”:1436251020000,”lt”:1594103819000,”format”:”epoch_millis”}}}],”must_not”:[]}},”size”:10,”sort”:[{“@timestamp”:{“order”:”desc”,”unmapped_type”:”boolean”}}],”aggs”:{“2”:{“date_histogram”:{“field”:”@timestamp”,”interval”:”1M”,”min_doc_count”:1,”time_zone”:”Asia/Kolkata”},”aggs”:{}}}}]}

  1. Hit _msearch request and collect the response.
  2. Fetch Aggregation part from msearch response.

aggregations”:{
“2”:{“buckets”:[{“key_as_string”:”2020-07-    01T00:00:00.000+05:30″,”key”:1593541800000,”doc_count”:4694}]}},”status”:200}]
}

  1. Design standard visualization response and return to NFUI/Web Dashboard backend.

[{“visualizationName”:”nf_unified_chart”,”metaData”:[“@timestamp”],”graphID”:50001,”groupID”:50002,”arrTimeStamp”:[1593541800000],”visualizationData”:[{“metricName”:”count”,”vectorName”:”_all”,”totalCount”:””,”data”:[4694],”graphID”:1,”groupID”:50002,”max”:4694,”min”:4694,”avg”:4694,”stdDev”:0,”lastSample”:4694,”sampleCount”:1}]}]

Below is some important field information from the visualization response:

  • visualizationName: Name of saved visualization.
  • metaData: Array of applied bucket aggregation in the chart.
  • chartType: Type of chart (line/bar, etc.).
  • arrTimeStamp: Array of timestamp coming from NFDB in response.
  • visualizationData: Array bucket aggregation data