Skip to content

PDL Correlation

Correlation Statement

PDL provides several correlation statements for analyzing and aggregating data over time, including:

  • event_count: Calculates the count of events over a given timespan and stores the results under padasAggregation field.
  • value_count: Calculates the count of specific field values over a given timespan and stores the results under padasAggregation field.
  • temporal: Evaluates a list of events over a given timespan based on query expressions and stores the results under padasTemporal field.

Correlation statements have the following generic structure: 

<function-command> [<function-specific-params>] <timespan-param> <group-by-clause> <where-clause>

Some example correlation statements: 

event_count timespan=5m group_by field1 where field3 > 100
value_count(myfield) timespan=30s
temporal(ordered=true) [ padasRule="internal_error" || padasRule="new_network_connection" ] timespan=1m group_by internal_ip, remote_ip

Common Parameters for Functions

All correlation statements evaluate streaming events for a given time window (defined via timespan parameter) and optionally groups them according to selected fields (defined via group_by clause). For counting aggregation/correlation statements it's also possible to limit the results by providing a query expression (defined via where clause).

Argument Order

Correlation statements must start with the one of the available functions, followed by function specific parameters (if any). Common argument order and descriptions are provided in the following table.

Order Keyword Required Description Example
1 timespan Yes Specifies time window to perform aggregated function.
The value should be an integer followed by one of the following identifiers:
s for second(s)
m for minute(s)
h for hour(s)
d for day(s)		 timespan=5m
timespan=1h
2 group_by No Group correlation results according to specified field(s). group_by field1, field2
3 where No Filter events according to specified query expression. where field1 > 100

Event Count

Description

Counts events occurring in the given time frame and stores the result in padasAggregation.eventCount field. The result may also contain count of events for each group specified by group_by separately.

Syntax and Functions

... | event_count <timespan-param> <group-by-clause> <where-clause>

Event Count Examples

The following table provides examples of available functionality based on the following JSON value:

JSON Input Expression Expected Output 
  {"field1":"value1", "field2":123}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":123}

  event_count timespan=1m

  {
    "padasRule":"myrule1",
    "padasAggregation": {
      "result":{},
      "eventCount":4
    }
  }

  {"field1":"value1", "field2":123}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":123}

  event_count timespan=5m group_by field1 where padasAggregation.eventCount > 1

    {
      "padasRule": "myrule2",
      "padasAggregation": {
        "groupBy": {
          "field1": "value1"
        },
        "result": {
          "field1": {
            "value1": 2
          }
        },
        "eventCount": 2
      }
    }

    {
      "padasRule": "myrule2",
      "padasAggregation": {
        "groupBy": {
          "field1": "value2"
        },
        "result": {
          "field1": {
            "value2": 2
          }
        },
        "eventCount": 2
      }
    }

Value Count

Description

Counts number of distinct values in a field defined by fieldName and stores the result in padasAggregation.valueCount field. The result may also contain count of events for each group specified by group_by separately.

Syntax and Functions

... | value_count(<fieldName>) <timespan-param> <group-by-clause> <where-clause>

Field Name parameter: This is the field name where count of distinct values are calculated.

Value Count Examples

The following table provides examples of available functionality based on the following JSON value:

JSON Input Expression Expected Output 
  {"field1":"value1", "field2":123}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":123}

  value_count(field1) timespan=60s

  {
    "padasRule": "myrule6",
    "padasAggregation": {
      "field": "field1",
      "valueCount": 2,
      "result": {
        "field1": {
          "value1": 2,
          "value2": 3
        }
      },
      "eventCount": 5
    }
  }

  {"field1":"value1", "field2":123}
  {"field1":"value1", "field2":123}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":123}
  {"field1":"value3", "field2":123}

  value_count(field1) timespan=60s  group_by field2

    {
      "padasRule": "myrule2",
      "padasAggregation": {
        "groupBy": {
          "field2": 123
        },
        "field": "field1",
        "valueCount": 3,
        "result": {
          "field1": {
            "value1": 2,
            "value2": 2,
            "value3": 1
          }
        },
        "eventCount": 5
      }
    }

Temporal

Description

Temporal correlation statement checks for all the events matching the expression array within the time frame defined. If the boolean value ordered is set to true, then all the events are expected to occur in the given order. The result may also contain count of events for each group specified by group_by separately.

Syntax and Functions

... | temporal(<ordered-param>) [ <expression> || <expression> || ... ] <timespan-param> <group-by-clause> <where-clause>

Ordered parameter: order is assigned either true or false as value (e.g. ordered=true) to specify whether the events are expected to match expression array order. Expression array: The array consists of one or more expressions separated by double-pipe || character (e.g. [ field1="valu*" || field3 < 100 AND field4=false>])

Temporal Examples

The following table provides examples of available functionality based on the following JSON value:

JSON Input Expression Expected Output 
  {"field1":"value1", "field2":99}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":123}
  {"field1":"value2", "field2":125}

  temporal [ field1?="value" || field2 < 100 ] timespan=2m

  {
    "padasRule": "myrule8",
    "padasTemporal": {
      "result": {
        "field1?=\"value\"": [
          {
            "field1": "value1",
            "field2": 99
          },
          {
            "field1": "value1",
            "field2": 124
          },
          {
            "field1": "value2",
            "field2": 123
          },
          {
            "field1": "value2",
            "field2": 125
          },
        ],
        "field2<100": [
          {
            "field1": "value1",
            "field2": 99
          }
        ]
      }
    }
  }

  {"field1":"sometext", "field2":90}
  {"field1":"value1", "field2":124}
  {"field1":"value2", "field2":95}
  {"field1":"value2", "field2":123}

  temporal(ordered=true) [ field1?="value" || field2 < 100 ] timespan=2m group_by field2

    {
      "padasRule": "myrule9",
      "padasTemporal": {
        "groupBy": {
          "field2": 95
        },
        "result": {
          "field2<100": [
            {
              "field1": "value2",
              "field2": 95
            }
          ],
          "field1?=\"value\"": [
            {
              "field1": "value2",
              "field2": 95
            }
          ]
        }
      }
    }