Skip to content

PDL Expressions

Expressions

Expressions are run per streaming event and the following expressions are available in PDL.

  • Query
  • Eval
  • Fields
  • Rename
  • Flatten

Query

Description

The query expression is used to filter data based on a specified condition. If the query matches (evaluates to true), then the event data is returned.

Syntax and Operators

Query syntax consists of field name and value comparisons that may include boolean operators and grouping with paranthesis. Note that all operators are case sensitive.

Following are supported syntax for query expression:

  • NOT <query>
  • <query> AND|OR <query>
  • <fieldName> <comparisonOperator> <fieldvalue>

Comparison Operators

  • IN: Returns true if the field value exists within the provided array. Note that all array values must be one value type (either String or Integer).
  • =: Equals, returns true if the value is an exact match. A single wildcard * is also accepted for string values.
  • !=: Not Equals, returns true if the value does not match.
  • ?=: Contains, checks whether the string value contains the text. For arrays, it checks for the array item.
  • >: Greater than, returns true if query comparison value is greater than event field value.
  • <: Less than, returns true if query comparison value is less than event field value.
  • >=: Greater than or equals, returns true if query comparison value is greater than or equals to the event field value.
  • <=: Less than or equals, returns true if query comparison value is less than or equals to the event field value.

Boolean Operators

  • NOT: Negates the result of following (grouped) query
  • AND: Expects both sides of the expression to be true.
  • OR: Expects at least one side of the expression to be true.

Supported JSON Data Types

PDL comparisons work on String, Number, Boolean, and Array JSON value data types. String comparisons MUST be defined in quotes " within PDL query definition. Array comparisons are limited to Equals (=), Not Equals (!=), and Contains (?=) operators.

Examples:

  • PDL query with field1="123" will compare "123" as a String JSON data type.
  • PDL query with field2=123 will compare 123 as a Number JSON data type.
  • PDL query with field3=true will compare true as an Boolean JSON data type.
  • PDL query with field4=[5,6,7] will compare [5,6,7] as an Array JSON data type and expect field4 to be an array as well.

Wildcard Support

PDL supports a single wildcard * with Equals operator (=) for String JSON values. Following are valid PDL query examples with wildcard usage:

field1="val*1"
field1="val*"
field1="*ue1"
field1="*"

Grouped arguments

Sometimes the syntax must display arguments as a group to show that the set of arguments are used together. Parenthesis ( ) are used to group arguments.

For example in this syntax: (field1="val1" OR field2=123) AND field3="value3"

The grouped argument is (field1="val1" OR field2=123) and its results are evaluated as a whole.

Query Examples

The following table provides descriptions and examples of available operators based on the following JSON value:

{
  "field1":"value1",
  "field2":"value2 text2 value2",
  "field3":123,
  "field4":["item1","item2","item3"]
}
Operator/Keyword Example (evaluates to true)
NOT NOT (field1 = "valueXXX")
AND field1="value1" AND field3=123
OR field1="xyz" OR field3=123
IN field3 IN [111, 222, 123, 444]
= field1="value1"
field1="val*"
field1="*"
!= field3 != 456
field4 != ["other1","item3"]
?= field2 ?= "text2"
field4 ?= "item2"
> field3 > 100
< field3 < 200
>= field3 >= 123
<= field3 <= 123

Eval

Description

The eval expression is used to modify or compute one or more fields in the data (separated by comma , character). The expression must start with eval keyword.

Syntax and Functions

Eval expression requires the eval keyword followed by a field name and an assignment with = character. Right side of the assignment can be an if function or a calculated field value using literals (string, number) or field names. 

... | eval <fieldName>=<ifFunction>
... | eval <fieldName>=<fieldValue> <arithmeticFunction> <fieldValue>

The following list provides available evaluation expression functionality and operators:

Supported Function and Syntax Description 
  if(<query>, <true evaluationExpression> , <false evaluationExpression>)
If <query> expression matches the event (see query for details), returns the value of <true evaluationExpression>, otherwise the function returns the <false evaluationExpression>. 
  <fieldValue> <evaluationOperator> <fieldValue>
Where <fieldValue> can be a String or Number literal as well as a field name from JSON even data. The following are the list of supported <evaluationOperator> values:
  • +: Addition for numbers and concatenation for string values.
  • -: Substraction for number fields only.
  • *: Multiplication for number fields only.
  • /: Division for number fields only.

Eval Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":"value1",
  "field2":123
}
Expression Expected Output 
  eval myfield=field2 - 3

  {
    "field1":"value1",
    "field2":123,
    "myfield":120
  }

  eval myfield=if(field2 < 150, field1 + "xyz", "N/A"), mytag="sometag"

  {
    "field1":"value1",
    "field2":123,
    "myfield":"value1xyz",
    "mytag":"sometag"
  }

Fields

Description

The fields expression is used to keep or remove fields from the data.

Syntax and Functions

... | fields <fieldsFunction> <fieldName>, ...

Supported functions are:

  • keep: to keep the list of fields only
  • remove: to remove the list of fields and keep the rest

Fields Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":"value1",
  "field2":123,
  "field3": {
    "subfield1": 456,
    "subfield2": "value2"
  }
}
Expression Expected Output 
  fields remove field2, field3.subfield1

  {
    "field1":"value1",
    "field3": {
      "subfield2": "value2"
    }
  }

  fields keep field2, field3.subfield1

  {
    "field2":123,
    "field3": {
      "subfield1": 456
    }
  }

Rename

Description

The rename expression is used to rename one or more fields in the data. This command is useful for giving fields more meaningful names, such as "processId" instead of "pid".

Syntax and Operators

... | rename <fieldName> AS <fieldName>

Operators

  • AS: This operator is used to define the new name of the field.

Rename Examples

The following table provides examples of available functionality based on the following JSON value:

{
  "field1":"value1",
  "field2":123,
  "field3": {
    "subfield1": 456,
    "subfield2": "value2"
  }
}
Expression Expected Output 
  rename field1 AS myField

  {
    "myField":"value1",
    "field2":123,
    "field3": {
      "subfield1": 456,
      "subfield2": "value2"
    }
  }

  rename field2 AS myField2, field3.subfield1 AS mySubfield

  {
    "field1":"value1",
    "myField2":123,
    "mySubfield": 456,
    "field3": {
      "subfield2": "value2"
    }
  }

Flatten

Description

The flatten expression is used to flatten nested JSON event data.

Syntax and Operators

This expression does not have any functions/operators and expects JSON input.

... | flatten

Flatten Examples

The following table provides examples of available functionality with sample JSON value:

JSON Input Expression Expected Output 
  {
    "field1":"value1",
    "field2":123,
    "field3": [4, 5, 6]
  }

  | flatten

  {
    "field1":"value1",
    "field2":123,
    "field3": [4, 5, 6]
  }

  {
    "field1":"value1",
    "field2":123,
    "field3": {
      "subfield1": 456,
      "subfield2": "value2"
    }
  }

  | flatten

  {
    "field1":"value1",
    "field2":123,
    "field3_subfield1": 456,
    "field3_subfield2": "value2"
  }