Skip to content

PDL Expressions

Expressions

Expressions are run per streaming event and the following expressions are available in PDL.

Query

Description

The query expression is used to filter data based on a specified condition. If the query matches (evaluates to true), then the event data is returned.

Syntax and Operators

Query syntax consists of field name and value comparisons that may include boolean operators and grouping with paranthesis. Note that all operators are case sensitive.

Following are supported syntax for query expression:

  • NOT <query>
  • <query> AND|OR <query>
  • <fieldName> <comparisonOperator> <fieldvalue>

Comparison Operators

  • IN: Returns true if the field value exists within the provided array. Note that all array values must be one value type (either String or Integer).
  • =: Equals, returns true if the value is an exact match. A single wildcard * is also accepted for string values.
  • !=: Not Equals, returns true if the value does not match.
  • ?=: Contains, checks whether the string value contains the text. For arrays, it checks for the array item.
  • ~=: Regex, checks whether the string value matches the given Regular Expression.
  • >: Greater than, returns true if query comparison value is greater than event field value.
  • <: Less than, returns true if query comparison value is less than event field value.
  • >=: Greater than or equals, returns true if query comparison value is greater than or equals to the event field value.
  • <=: Less than or equals, returns true if query comparison value is less than or equals to the event field value.

Boolean Operators

  • NOT: Negates the result of following (grouped) query
  • AND: Expects both sides of the expression to be true.
  • OR: Expects at least one side of the expression to be true.

Supported JSON Data Types

PDL comparisons work on String, Number, Boolean, and Array JSON value data types. String comparisons MUST be defined in quotes " within PDL query definition. Array comparisons are limited to Equals (=), Not Equals (!=), and Contains (?=) operators.

Examples:

  • PDL query with field1="123" will compare "123" as a String JSON data type.
  • PDL query with field2=123 will compare 123 as a Number JSON data type.
  • PDL query with field3=true will compare true as an Boolean JSON data type.
  • PDL query with field4=[5,6,7] will compare [5,6,7] as an Array JSON data type and expect field4 to be an array as well.

Wildcard Support

PDL supports a single wildcard * with Equals operator (=) for String JSON values. Following are valid PDL query examples with wildcard usage:

field1="val*1"
field1="val*"
field1="*ue1"
field1="*"

Grouped arguments

Sometimes the syntax must display arguments as a group to show that the set of arguments are used together. Parenthesis ( ) are used to group arguments.

For example in this syntax: (field1="val1" OR field2=123) AND field3="value3"

The grouped argument is (field1="val1" OR field2=123) and its results are evaluated as a whole.

Query Examples

The following table provides descriptions and examples of available operators based on the following JSON value:

{
  "field1":"value1",
  "field2":"value2 text2 value2",
  "field3":123,
  "field4":["item1","item2","item3"]
}
Operator/Keyword Example (evaluates to true)
NOT NOT (field1 = "valueXXX")
AND field1="value1" AND field3=123
OR field1="xyz" OR field3=123
IN field3 IN [111, 222, 123, 444]
= field1="value1"
field1="val*"
field1="*"
!= field3 != 456
field4 != ["other1","item3"]
?= field2 ?= "text2"
field4 ?= "item2"
~= field2 ~= "\stext"
field4 ~= "\d\stext\d\s"
> field3 > 100
< field3 < 200
>= field3 >= 123
<= field3 <= 123

Eval

Description

The eval expression is used to modify or compute one or more fields in the data (separated by comma , character). The expression must start with eval keyword.

Syntax and Functions

Eval expression requires the eval keyword followed by a field name and an assignment with = character. Right side of the assignment can be an if function or a calculated field value using literals (string, number) or field names. 

... | eval <fieldName>=<ifFunction>
... | eval <fieldName>=<fieldValue> <arithmeticFunction> <fieldValue>

The following list provides available evaluation expression functionality and operators:

Supported Function and Syntax Description 
  if(<query>, <true evaluationExpression> , <false evaluationExpression>)
If <query> expression matches the event (see query for details), returns the value of <true evaluationExpression>, otherwise the function returns the <false evaluationExpression>. 
  <fieldValue> <evaluationOperator> <fieldValue>
Where <fieldValue> can be a String or Number literal as well as a field name from JSON even data. The following are the list of supported <evaluationOperator> values:
  • +: Addition for numbers and concatenation for string values.
  • -: Substraction for number fields only.
  • *: Multiplication for number fields only.
  • /: Division for number fields only.

Eval Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":"value1",
  "field2":123
}
Expression Expected Output 
  eval myfield=field2 - 3

  {
    "field1":"value1",
    "field2":123,
    "myfield":120
  }

  eval myfield=if(field2 < 150, field1 + "xyz", "N/A"), mytag="sometag"

  {
    "field1":"value1",
    "field2":123,
    "myfield":"value1xyz",
    "mytag":"sometag"
  }

Fields

Description

The fields expression is used to keep or remove fields from the data.

Syntax and Functions

... | fields <fieldsFunction> <fieldName>, ...

Supported functions are:

  • keep: to keep the list of fields only
  • remove: to remove the list of fields and keep the rest

Fields Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":"value1",
  "field2":123,
  "field3": {
    "subfield1": 456,
    "subfield2": "value2"
  }
}
Expression Expected Output 
  fields remove field2, field3.subfield1

  {
    "field1":"value1",
    "field3": {
      "subfield2": "value2"
    }
  }

  fields keep field2, field3.subfield1

  {
    "field2":123,
    "field3": {
      "subfield1": 456
    }
  }

Rename

Description

The rename expression is used to rename one or more fields in the data. This command is useful for giving fields more meaningful names, such as "processId" instead of "pid".

Syntax and Operators

... | rename <fieldName> AS <fieldName>

Operators

  • AS: This operator is used to define the new name of the field.

Rename Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":"value1",
  "field2":123,
  "field3": {
    "subfield1": 456,
    "subfield2": "value2"
  }
}
Expression Expected Output 
  rename field1 AS myField

  {
    "myField":"value1",
    "field2":123,
    "field3": {
      "subfield1": 456,
      "subfield2": "value2"
    }
  }

  rename field2 AS myField2, field3.subfield1 AS mySubfield

  {
    "field1":"value1",
    "myField2":123,
    "mySubfield": 456,
    "field3": {
      "subfield2": "value2"
    }
  }

Flatten

Description

The flatten expression is used to flatten nested JSON event data.

Syntax and Operators

This expression does not have any functions/operators and expects JSON input.

... | flatten

Flatten Examples

The following table provides examples of available functionality with sample JSON value:

JSON Input Expression Expected Output 
  {
    "field1":"value1",
    "field2":123,
    "field3": [4, 5, 6]
  }

  | flatten

  {
    "field1":"value1",
    "field2":123,
    "field3": [4, 5, 6]
  }

  {
    "field1":"value1",
    "field2":123,
    "field3": {
      "subfield1": 456,
      "subfield2": "value2"
    }
  }

  | flatten

  {
    "field1":"value1",
    "field2":123,
    "field3_subfield1": 456,
    "field3_subfield2": "value2"
  }

Lookup

Description

The lookup expression is used to enrich event data. When an event is processed, the lookup expression is used to perform a match in the lookup table, based on a specified field. Once a match is found, relevant information from the lookup table is retrieved and added to the event data.

Syntax and Operators

... | lookup <lookup_id> <lookup_fieldName> OUTPUT <lookup_dest_fieldName>

Arguments

  • lookup_id: It refers to the lookup name.
  • lookup_fieldName: It refers to the field in the lookup table used for matching against the events.
  • lookup_dest_fieldName: Specifies a field in the lookup table that will be copied into the events.

Operators

  • AS: This operator is used to define the new name of the field.

Lookup Examples

The following table provides examples of available functionality based on the following event data and lookup example:

Lookup Example: 

{
        "id": "windows_signatures_substatus",
        "content": [
            {
                "signature_id": 4625,
                "Sub_Status": "0xc0000064",
                "signature": "User name does not exist"
            },
            {
                "signature_id": 4625,
                "Sub_Status": "0xC0000064",
                "signature": "User name does not exist"
            },
            {
                "signature_id": 4625,
                "Sub_Status": "0xc000006a",
                "signature": "User name is correct but the password is wrong"
            }
        ]
    }

Event Data Example: 

{
  "EventTime": "2017/08/25 14:09:12",
  "Hostname": "CIVDCS-ADC1.changeme.com",
  "EventID": 4625,
  "SourceName": "Microsoft-Windows-Security-Auditing",
  "Message": "An account failed to log on.",
  "Category": "Logon",
  "SubjectLogonId": "0x0",
  "TargetUserSid": "S-1-0-0",
  "TargetUserName": "MININT-UP26I95$",
  "TargetDomainName": "changeme",
  "Status": "0xc000006d",
  "FailureReason": "%%2313",
  "Sub_Status": "0xc000006a",
  "LogonType": "3",
  "IpAddress": "172.23.130.64"
}

Expression Expected Output 
  lookup windows_signatures_substatus Sub_Status OUTPUT signature

        {
          "EventTime": "2017/08/25 14:09:12",
          "Hostname": "CIVDCS-ADC1.changeme.com",
          "EventID": 4625,
          "SourceName": "Microsoft-Windows-Security-Auditing",
          "Message": "An account failed to log on.",
          "Category": "Logon",
          "SubjectLogonId": "0x0",
          "TargetUserSid": "S-1-0-0",
          "TargetUserName": "MININT-UP26I95$",
          "TargetDomainName": "changeme",
          "Status": "0xc000006d",
          "FailureReason": "%%2313",
          "Sub_Status": "0xc000006a",
          "LogonType": "3",
          "IpAddress": "172.23.130.64",
          "signature": "User name is correct but the password is wrong"
        }

Rex

Description

This command is used to extract fields using regular expression named groups from a given field in the event data.

Syntax and Operators

... | rex <fieldName> "<regex>"

Arguments

  • fieldName: It refers to the field name where the regular expression will be applied to.
  • regex: This is the named capturing group regular expression, in quotes, to extract new fields.

Rex Examples

The following table provides examples of available functionality based on the following event data:

{
  "field1": {
    "subfield1":"subvalue1",
    "subfield2":"sub value2"
    }, 
    "field2": "value2", 
    "field3": 123
}
Expression Expected Output Explanation 
  rex field2 "(?<field2Text>\w+)(?<field2Number>\d+)"

  {
    "field1": {
      "subfield1":"subvalue1",
      "subfield2":"sub value2"
    }, 
    "field2": "value2", 
    "field3": 123,
    "field2Text": "value",
    "field2Number": 2
  }
Regular expression is applied to field1 value, which is "value2". The regular expression separates text with ending number and puts them in 2 different fields (namely field2Text and field2Number). 
  rex field1.subfield2 "\w+\s(?<field1subfield3>\w+)"

  {
    "field1": {
      "subfield1":"subvalue1",
      "subfield2":"sub value2"
    }, 
    "field2": "value2", 
    "field3": 123,
    "field1subfield3": "value2"
  }
Regular expression is applied to field1.subfield2 nested field value, which is "sub value2". The regular expression skips the first word including the following space character and captures the remaining word as field1subfield3.

Parse CSV

Description

The parse_csv expression is used to properly parse the values of a specific CSV encoded field from the given eventdata. Newly parsed fields are added to the event. Some example values that can be parsed:

  • value1, value2, value3
  • value1, value2, 123 (Last one is parsed as Integer in JSON)
  • "value 1", "value 2", value 3 (Optional quoted values for Strings.)
  • "value 1"| "value 2"| value 3 (Using custom delimiters, e.g. |)

Syntax and Functions

... | parse_csv <fieldName> header=<headerValue> [delimiter=<delimiterValue>]

Arguments

  • fieldName: Field name to parse.
  • headerValue: Must be a comma separated String in quotes (e.g. "myfield1, myfield2, myfield3"). Represents the header values to be used as new field names for the parsed CSV encoded data.
  • delimiterValue: Must be a String in quotes (e.g. ";"). Delimiter value to use for CSV encoded data. Default is ","

Fields Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":
    {
      "subfield1":"subvalue1",
      "subfield2":"sub value2"
    }, 
  "field2": "aaa; 123 ; ccc", 
  "field3": 123,
  "field4": "xxx, 456 , yyy"
}
Expression Expected Output 
  parse_csv field2 header="csvField1, csvField2, csvField3" delimiter=";"

  {
    "field1":
      {
        "subfield1":"subvalue1",
        "subfield2":"sub value2"
      }, 
    "field2": "aaa; 123 ; ccc", 
    "field3": 123,
    "field4": "xxx, 456 , yyy",
    "csvField1": "aaa",
    "csvField2": 123,
    "csvField3": "ccc"
  }

  parse_csv field4 header="csvField1, csvField2, csvField3" 

  {
    "field1":
      {
        "subfield1":"subvalue1",
        "subfield2":"sub value2"
      }, 
    "field2": "aaa; 123 ; ccc", 
    "field3": 123,
    "field4": "xxx, 456 , yyy",
    "csvField1": "xxx",
    "csvField2": 456,
    "csvField3": "yyy"
  }

Parse KV

Description

The parse_kv expression is used to properly parse the values of a specific Key-Value encoded field from the given eventdata. Newly parsed fields are added to the event. Some example values that can be parsed:

  • key1=value1 key2=value2 key3=value3
  • key1=value1 key2 = value2 key3= 123 (Last one is parsed as Integer in JSON)
  • key1=value1 key2="value2" key3=value3 (Optional quoted values for Strings.)
  • key1:value1 key2 :value2 key3:value3 (Using custom delimiters, e.g. :)

Syntax and Functions

... | parse_kv <fieldName> [delimiter=<delimiterValue>]

Arguments

  • fieldName: Field name to parse.
  • delimiterValue: Must be a String in quotes (e.g. ":"). Delimiter value to use for KV encoded data. Default is "="

Fields Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":
    {
      "subfield1":"subvalue1",
      "subfield2":"sub value2"
    }, 
  "field2": "kvField1=aaa kvField2=123 kvField3=ccc", 
  "field3": 123,
  "field4": "kvField1 : xxx kvField2: 456 kvField3:\"yyy\"",
}
Expression Expected Output 
  parse_csv field2

  {
    "field1":
      {
        "subfield1":"subvalue1",
        "subfield2":"sub value2"
      }, 
    "field2": "kvField1=aaa kvField2=123 kvField3=ccc", 
    "field3": 123,
    "field4": "kvField1 : xxx kvField2: 456 kvField3:\"yyy\"",
    "kvField1": "aaa",
    "kvField2": 123,
    "kvField3": "ccc"
  }

  parse_kv field4 delimiter=":"

  {
    "field1":
      {
        "subfield1":"subvalue1",
        "subfield2":"sub value2"
      }, 
    "field2": "kvField1=aaa kvField2=123 kvField3=ccc", 
    "field3": 123,
    "field4": "kvField1 : xxx kvField2: 456 kvField3:\"yyy\"",
    "kvField1": "xxx",
    "kvField2": 456,
    "kvField3": "yyy"
  }