Validation Helper

The validation helper is used to validate CSV files, spreadsheets (e.g., XLSX files), lists, and dictionaries.
​​
​​
The data to be validated is specified in the Data Reference field, which expects a reference just by the ID of the flow step, e.g., spr1.
The user can then choose what action should be taken if a field of a row does not adhere to specified conditions. The possible actions are:
Terminate flow - will directly terminate the entire flow if a field could not pass the specified validation.
Drop invalid rows - will drop all rows in which a field could not pass the specified validation and return the remaining rows inside a dictionary called data, while errors are returned inside a dictionary called errors.
Return errors - will return all the input data inside a dictionary called data, while errors are returned inside a dictionary called errors.
Next, the columns or fields to which the validation should be applied have to be specified; to do so, either write the columns name (if the data comes from a spreadsheet or dictionary) or the column's index (if the data comes from a CSV or list of lists) into the Column field. Afterwards, select the field type for this column or field and a corresponding condition. Finally, you enter the condition value.
Furthermore, one can declare if blank fields should be ignored for the validation or not. This can be repeated for each field or column, which should be validated by adding a new set of fields by clicking the + button.
Example Validation Condition
Examples
Data comes from a CSV file and is hence handled as a list of lists in the flow data handling logic:
[
  [
    123,
    "Michael",
    "United States"
  ],
  [
    124,
    "Maria",
    "Spain"
  ]
]
The references to the columns are made by specifying the column index, i.e., 0 for the first column and 2 for the third column (note: the indexes start at 0).
Validation Helper - CSV
 The resulting output by the validation helper is:
{
  "data": [
    {
      "0": 124,
      "1": "Maria",
      "2": "Spain"
    }
  ],
  "errors": [
    {
      "row": [
        123,
        "Michael",
        "United States"
      ],
      "errors": [
        {
          "2": {
            "United States": [
              "max length is 6"
            ]
          }
        }
      ]
    }
  ]
}
If the same data is provided in the form of an XLSX file, the data would be stores as a list of dictionaries:
[
  {
    "id": 123,
    "name": "Michael",
    "country": "United States"
  },
  {
    "id": 124,
    "name": "Maria",
    "country": "Spain"
  }
]
Almost all the settings in the Validation Helper could stay the same, except the Data Reference and Column, which would, in this example, be filed with id and country. 
The Validation Helper can also work with more complex, nested dictionaries, such as:
{
  "contacts" : [
    {
      "id": 123,
      "name": "Michael",
      "address": {
        "country": "United States",
        "city": "New York"
      }
    },
    {
      "id": 124,
      "name": "Maria",
      "address": {
        "country": "Spain",
        "city": "Madrid"
      }
    }
  ],
  "companies": [
    {
      "id": 200,
      "name": "Tech Company",
      "website": "www.tech.com",
    }
  ]
}
In order to only access and validate the contacts (dictionary), in the above example, the path of the nested dictionary, which should be validated, has to be specified in the Data Reference field, e.g., if the data comes from connector xls1, the reference would be xls1.contacts. In this way, one can even reference to nested fields within the contacts dictionary, such as the field country:
Validation Helper - Nested Dictionary
The resulting output in this case is:JavaScript
{
  "data": [
    {
      "id": 124,
      "name": "Maria",
      "address": {
        "country": "Spain",
        "city": "Madrid"
      }
    }
  ],
  "errors": [
    {
      "id": 123,
      "name": "Michael",
      "address": {
        "country": "United States",
        "city": "New York"
      },
      "errors": {
        "address.country": {
          "United States": [
            "max length is 6"
          ]
        }
      }
    }
  ]
}

Data Warehouse Helper

XML Helper

Last modified 1yr ago

Copy link