Security Stats API Guide

Overview

This document describes using the Instart Logic security statistics API to make specific queries against the raw security event data and aggregated security statistics collected by our log processing system and data platform. You can access raw security event data, data on trends, Bot Defense, Custom Security Rules, and Web Application Firewall (WAF) events.

NOTE The WAF API has been deprecated, replaced by the WAF methods in the Security stats API described in this document.

You can find some query examples at the end of this document.

Events

These methods provide access to raw security event data.

NOTE The retention period of the security event log is the previous three days plus whatever time has elapsed in the current day.

GET /stats/security/events/data

This provides raw event log data. You specify a time span and a list of the dimensions you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or by adding an optional filter parameter to the query string.

The query structure is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/<your unique customer name>/v1/stats/security/data?
dimensions=<dimension1,dimension2...>
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

To be valid, each query requires

The additional query parameters filters, sortby, and limit may be provided to further refine your query, but are optional.

All query parameters must be URL-encoded.

Dimensions

To get event data, provide the dimensions parameter with a valid list of dimensions. The list of supported dimensions can be found by querying the API, as described below.

The dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records. For example, event_severity and event_type are two types of data that are recorded in the security events logs.

Starttime

The starttime parameter is the start time of the desired time range of the desired statistics, which corresponds to when the user requests were received as logged in the access logs. It is required and cannot be left empty. It must be specified in standard ISO8601 date/time format, but always in the UTC time zone. Don't specify a time zone designator or the request will be rejected. Start times must be earlier than the end time to be valid.

Endtime

The endtime parameter is the end time of the desired time range of the statistics, which corresponds to when the user requests were received as logged in the access logs. It is required and cannot be left empty. It must be specified in standard ISO8601 date/time format, but always in the UTC time zone (specifying another time zone designator will cause the request to be rejected). End times must be later than the start time to be valid.

Filters (optional)

The optional filters parameter consists of a comma-separated list of filters to be applied to the requested data. A filter is of the form:

<dimension><operator><value>

For example:

filters=property==public

where property is the dimension, == is the operator (equal to) and public is the value. This filter, if added to a query, would get back only the events that that occurred for the property named "public."

Valid operators are == and !=

Multiple individual filters can be combined using the OR(,) or AND(;) operators – but not both – to form a logical filter expression.

NOTE If the special characters "," or ";" need to be treated as literal characters in a filter expression (for example, the comma character occurs in some country strings you might want to filter on), you need to escape them by prepending them with "\"

For example:

filters=property==public;client_browser==Chrome

contains two filters: the first one to allow only results from the property public, and a second one to allow only results that came from requests from the Chrome browser. They are separated by a semicolon, which means AND. So this pair of filters would return only results with an property named "public" AND were requests from Chrome.

Sortby (optional)

The optional sortby parameter contains a list of one or more dimensions and/or metrics over which the resultant statistics data should be sorted. For example:

sortby=event_timestamp_ms

Sorting order is specified by the left to right order of the metrics and dimensions listed. Sorting direction defaults to ascending and can be changed to descending by using a minus sign (-) prefix on the requested field.

One common use case would be if you are troubleshooting an issue that just started, and you want to see the latest events first. You can use

sortby=-event_timestamp_ms

to return results in reverse order.

Limit (optional)

The optional limit parameter is an integer that will restrict the total number of results that are returned by the query. For example, using

limit=25

tells the Stats API server to return the first 25 results.

If unspecified, the limit parameter defaults to returning 10 results.

The limit parameter has no upper limit. To get all events in a particular time span, you can query without the limit parameter, observe the total number of events, then query a second time with limit=<total number of events>.

HEAD /stats/security/events

The HEAD method is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

GET /stats/security/events/dimensions

The dimensions are the kinds of available data. They correspond to the column headings of a table listing event log records. For example, event_severity and event_type are two types of data that are recorded in the security events logs.

WAF topK

This provides data on the top 1,000 WAF events for a variety of data categories (each of which has its own dimensions and metrics) based on the aggregated WAF data. You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or adding an optional filter parameter to the query string.

The query structure for each is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/<your unique customer name>/v1/stats/security/waf/topk/<category>/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time> [&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

where the parameters in square brackets are optional. The possible values for category are listed below:

The descriptions of the parameters are the same as for the WAF trend methods. Each category also has its corresponding dimensions, groupings, and metrics methods.

HEAD /stats/security/waf/topk/<category>

There are corresponding HEAD methods for all the topK categories is identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

Traffic shaping topK

This provides data on the top 1,000 traffic shaping events for a variety of data categories (each of which has its own dimensions and metrics) based on the aggregated traffic shaping data. You specify a time span and a list of the dimensions and metrics you want. You can optionally filter out some of the data you receive by either limiting the number of results shown or adding an optional filter parameter to the query string.

The query structure for each is appended to the API URL as a standard query string, as follows:

https://api.instartlogic.com/<your unique customer name>/v1/stats/security/trafficshaping/topk/<category>/data?
dimensions=<dimension1,dimension2...>
&metrics=hits
&startime=<UTC date/time of the desired beginning time>
&endtime=<UTC date/time of the desired end time>
[&filters=<filter expression 1,...>]
[&sortby=<dimension1,dimension2...>]
[&limit=<number of results desired>]

where the parameters in square brackets are optional. The possible values for category are listed below:

The descriptions of the parameters are the same as for the WAF trend methods. Each category method has its corresponding dimensions, groupings, and metrics methods.

HEAD /stats/security/trafficshaping/topk/<category>

There are corresponding HEAD methods for all the topK categories. These methods are identical to GET except that the server doesn't return a message body in the response, just the same headers. This is useful for testing hypertext links for validity, accessibility, and recent modification.

Examples

The following are some examples of using the security stats API to query for statistics. Line breaks are added for readability.

If you use cURL, execute the command

curl -v -u 'username:password' '<base URL>?<query string>'

If you use Postman or a similar REST client:

  1. Open a new Request window.
  2. In the URL field at the top enter the base URL and the query string.
  3. Select GET from the method list to the left of the URL field.
  4. Below, in the Authorization tab, select Basic Auth from the Type list, and enter your username and password in the fields provided.
  5. Click Send.

With either method you would of course use your own authorization string and company name in place of the placeholders. Also be sure that the starttime and endtime parameters cover a time span that occurred within the last three days if you are querying raw event data from the access logs.

Example 1 – raw events query without optional parameters

The following query requests raw event data logged between 11 November 2017 at midnight (UTC) and 12 November 2017 at midnight (UTC). It is requesting a subset of dimensions: the remote IP address of the client connection, the client browser, the event type, event action, and the domain the request was sent to.

The base URL and query string for this query are

https://api.instartlogic.com/acme/v1/stats/security/events/data?
dimensions=client_connection_remote_ip,client_browser,event_type,
event_action,domain
&starttime=2017-11-11T00:00:00
&endtime=2017-11-12T00:00:00

Note that line breaks were added to the query string for readability.

If successful, the API server responds with a JSON object similar to the following (for brevity only a few rows are shown of the full number actually returned). Note that the timestamp of each event (event_timestamp_ms) is included in the results by default. Also note that limit (how many results to display) is set at the default value of 10 since that parameter was not specified in the request, and that the total number of events found in the specified time span is displayed as the value of the total_results field:

{
    "query": {
        "starttime": "2017-11-11T00:00:00",
        "endtime": "2017-11-12T00:00:00",
        "start": 0,
        "limit": 10,
        "dimensions": [
            "client_connection_remote_ip",
            "client_browser",
            "event_type",
            "event_action",
            "domain"
        ],
        "filters": "customer==acme"
    },
    "columns": [
        {
            "name": "event_timestamp_ms"
        },
        {
            "name": "client_connection_remote_ip"
        },
        {
            "name": "client_browser"
        },
        {
            "name": "event_type"
        },
        {
            "name": "event_action"
        },
        {
            "name": "domain"
        }
    ],
    "rows": [
        [
            "2017-11-11T00:02:20",
            "45.33.61.221",
            "Chrome",
            "CUSTOM",
            "WARN",
            "www.acme.com"
        ],
        [
            "2017-11-11T00:02:40",
            "139.162.210.253",
            "Chrome",
            "CUSTOM",
            "WARN",
            "www.acme.com"
        ],
        [
            "2017-11-11T00:02:33",
            "45.79.179.69",
            "Chrome",
            "CUSTOM",
            "WARN",
            "www.acme.com"
        ],
        ...
        [
            "2017-11-11T00:02:07",
            "139.162.244.180",
            "Chrome",
            "CUSTOM",
            "WARN",
            "www.acme.com"
        ]
    ],
    "total_results": 26584
}

Example 2 – raw events query with filters and limit

The following query requests the client IP address, ASN, and rule ID from raw event data logged between 11 November 2017 at midnight (UTC) and 12 November 2017 at midnight (UTC). A filter is applied so that only those log entries from Singapore (country code SG) are returned, and a limit of 50 is applied.

The base URL and query string for this query are

https://api.instartlogic.com/acme/v1/stats/security/events/data?
dimensions=event_timestamp_ms,client_ip,rule_id,client_asn
&starttime=2017-11-12T00:00:00
&endtime=2017-11-14T24:00:00
&limit=50
&filters=client_country_code==SG

Note that line breaks were added to the query string for readability.

If successful, the API server responds with a JSON object similar to the following (for brevity only a few rows are shown of the full number actually returned):

{
    "query": {
        "starttime": "2017-11-12T00:00:00",
        "endtime": "2017-11-14T24:00:00",
        "start": 0,
        "limit": 50,
        "dimensions": [
            "event_timestamp_ms",
            "client_ip",
            "rule_id",
            "client_asn"
        ],
        "filters": "client_country_code==SG;customer==acme"
    },
    "columns": [
        {
            "name": "event_timestamp_ms"
        },
        {
            "name": "client_ip"
        },
        {
            "name": "rule_id"
        },
        {
            "name": "client_asn"
        }
    ],
    "rows": [
        [
            "2017-11-12T00:02:46",
            "52.76.85.66",
            "D173128",
            16509
        ],
        [
            "2017-11-12T00:10:32",
            "52.76.85.66",
            "D173128",
            16509
        ],
        [
            "2017-11-12T00:06:33",
            "52.76.85.66",
            "D173128",
            16509
        ],
        ...
        [
            "2017-11-12T01:37:18",
            "52.76.85.66",
            "D173128",
            16509
        ],
        [
            "2017-11-12T01:33:19",
            "52.76.85.66",
            "D173128",
            16509
        ]
    ],
    "total_results": 4365
}