Errors

Data validation occurs at many steps throughout the configured data ingestion pipelines that are downstream of all data interfaces

A general data pipeline consists of

• Data receipt/ingestion

• Data schema validation

• Data transformation and operations

• Data sinking into downstream services

Data schema validation

All data ingested undergoes automated data validation against a configured data schema. The schema defines both the shape, data types, required fields, and additional data value validation of the incoming data. Data schemas are highly configurable and applied in any combination to support incoming datasets.

Data schemas are defined leveraging the json-specification.

Below is an example dataset schema for a Person object

{
  "$id": "https://example.com/person.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string",
      "description": "The person's first name."
    },
    "lastName": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years which must be equal to or greater than zero.",
      "type": "integer",
      "minimum": 0
    }
  }
}

For sychronous data events like via the data API, data schema validation happens synchronously and any data validation errors will result in a 400 HTTP error code. All other asynchronous data events, any error state if available can be available via the transactions API

Types of Errors

Below is a summary of possible errors at various data steps

Error Type	Data Step	Description
UnsafeFileError	Validation	Data was scanned for malicious content, was deemed unsafe and will be quarantined. Applicable if flat files are provided via the data interfaces
InvalidSchemaError	Validation	Data fails the configured data schema for that specific dataset. Additional validation error details can be found via the transactions API
OperationError	Transformation	The configured operation/transformation failed to execute
ObjectAlreadyExistsError	Sink	Where applicable and configured, the provided data is a duplicate of another data object in the platform
MissingLookupError	Sink	Where applicable, the provided data references fail to find an association i.e. unable to match an incoming measurement to its Validere source object
LookupCollisionError	Sink	Where applicable, the provided data references results in a non-unique/more than one association and the intended outcome is ambiguous
RequestExceptionError	Sink	An internal error occurred on the Validere server