Data validation: methods, types and best practices

Data validation is the process of checking whether data is correct, complete, consistent and compliant with the rules defined for a given use case. In an industrial environment, this covers sensor data, SCADA and MES systems, historians and operational reports, among others.

Without validation, data may look technically correct and still lead to faulty reports, false alarms, poor decisions or unreliable analytics. Validation should therefore be treated as a permanent element of data management, not a one-off test performed just before a report is published.

What is data validation?

Data validation is the process of checking whether data meets defined requirements for type, range, format, completeness, consistency and fitness for use.

In practice, this means data is compared against agreed rules before it is stored, passed to the next system, or used in a report, an analytical model or a decision-making process. IBM defines data validation as confirming that data is accurate and ready to be used.

A simple example: the “order end date” field cannot be earlier than the “order start date”. An industrial example: a temperature signal from a sensor should not exceed the physically possible range, and a flow value should be consistent with pump operation and system pressure.

Data validation helps answer the following questions:

• Does the value have the correct data type?
• Is it within the permitted range?
• Does the format match the expected pattern?
• Are all required fields populated?
• Is the record a duplicate?
• Is data from different fields or systems mutually consistent?
• Can the data be safely used in a report, an analysis or a decision?

Validation matters especially in industry, because much of the data is generated automatically and at high frequency. A unit error, a time offset, a frozen signal, a missing quality status or a gap in a time series can all affect production calculations, analytics, alarming and operational decisions.

Validated data is one of the foundations of an industrial data platform, because only once quality and context have been checked can it be used successfully in the higher layers of the system.

Data validation vs. data verification

Validation checks whether data complies with defined rules; verification checks whether data matches its source or reality.

The two terms are often confused, but they describe different activities. Precisely draws a similar distinction: validation typically occurs when a record is created or updated, whereas verification may form part of a recurring data quality process.

Aspect	Data validation	Data verification
Core question	Does the data comply with the rules?	Does the data match the source or the actual state?
Typical moment	At entry, integration, import or processing	After migration, import or transfer, or during accuracy checks
Example	Temperature must be a number between -40 and 150°C	The temperature reading in the system matches the source device
Typical data validation techniques	Type, range, format, consistency, completeness, uniqueness	Source-to-target comparison, double entry, checksum, hash
Purpose	Reject, flag or correct data that breaches the rules	Confirm that data has not been distorted or transferred incorrectly
Scope	Fitness of the data for its intended use	Conformity of the data with a reference, document, source or another system

An example from a data migration: validation checks that every record has the required fields, a correct date format and a permitted status code. Verification checks that record counts, checksums and values after migration match the source system.

In practice the two processes should complement each other. Validation does not guarantee that data is true. Verification does not guarantee that data will be usable downstream if it fails to meet quality rules.

Data validation vs. data quality

Data validation is a technique that supports data quality, but it does not replace full data quality management.

Data quality covers a broader set of practices: setting standards, profiling data, managing rules, monitoring issues, cleansing data, assigning ownership and driving continuous improvement. IBM identifies six core dimensions of data quality: accuracy, completeness, consistency, timeliness, validity and uniqueness.

Data quality dimension	Meaning	How validation supports it
Accuracy	Data reflects the actual state	Comparison against a range, a source or a reference value
Completeness	Required data is present	Mandatory-field and data-gap checks
Consistency	Data does not conflict across fields or systems	Cross-field rules, cross-reference checks and cross-system comparisons
Timeliness	Data is available when it is needed	Latency, timestamp and data-freshness checks
Validity	Data conforms to the permitted format or domain	Type, format, range and reference-list rules
Uniqueness	A record does not appear more than once without reason	Duplicate and unique-key checks

Data validation is therefore one of the mechanisms that maintain data quality. Data cleansing is the next step: correcting, completing, merging or removing faulty records. Validation identifies the problem; cleansing removes or corrects it.

Good data quality translates directly into the quality of decisions. In the context of Decision Intelligence, this means a decision system should know not only the value of a data point, but also its status, reliability, source and limitations.

Types of data validation

Types of data validation define the conditions a single value, a record or a data set must meet in order to be considered correct.

Validation type	Description	Example	When to use it
Data type check	Checks whether a value has the expected type: number, text, date or boolean	The “energy consumption” field must be a floating-point number	Forms, imports, APIs and data from source systems
Range check	Checks whether a value falls within permitted limits	Process temperature must be between 60 and 95°C	Measurements, process parameters and quality thresholds
Format check	Checks whether a value matches a pattern	A batch number must follow the format PL-YYYY-NNNN	Codes, identifiers, batch numbers, dates and addresses
Consistency check	Checks dependencies between fields	The end date cannot be earlier than the start date	Process records, work orders, batches and events
Uniqueness check	Checks whether a record or identifier is duplicated	A production order number must be unique	Primary keys, imports, migrations and integrations
Completeness check	Checks whether required fields are empty	A failure record must have a date, an asset, a category and a status	Reporting, audit, quality and ESG data
Length check	Checks the length of a text string or code	A material code must be 10 characters long	Codes, identifiers and text fields
Lookup check	Checks whether a value exists in a reference table	A cost centre code must exist in the organisational reference list	Master data, reference lists, statuses and classifications
Pattern check	Checks a value against a regular expression or an agreed pattern	An equipment code must match the location and asset-type pattern	Integrating data from multiple systems
Cross-check	Compares data across sources or tables	Output reported in MES should match the line counter	OT/IT integration, migration and reporting

The most common questions concern the “four types” or “five types” of validation. In practice, classifications differ between sources, but the basic set usually covers type, range, format, consistency and completeness checks. Industrial environments add checks on time signals, quality statuses, engineering units and relationships between sensors.

Data validation methods

Validation methods can be organised by the level at which the check operates: a single field, a whole record, a data set, a schema, or a comparison between systems.

Validation level	Scope	Typical moment	Example
Field-level validation	A single field or value	At data entry, integration or sensor read	Temperature must be a number and fall within the process range
Record-level validation	Several fields within one record	After a record or event is created	A work order cannot have an end date earlier than its start date
Batch-level validation	A whole data set or import package	During ETL, migration, import or periodic reporting	Record counts and checksums must match the source system
Schema validation	Data structure, types and constraints	For APIs, files, databases and pipelines	A JSON file must conform to the approved schema
Cross-system validation	Data held in multiple systems	After integration or replication	An order in MES must correspond to the order in ERP
Statistical validation	Patterns, deviations and distributions	For process data or large data sets	A value several standard deviations from the usual profile must be flagged

Field-level validation

Field-level validation checks a single value. It is often applied in real time — at data entry, on file import, when an API message is received, or when a source system is read.

Examples:

• a quantity field must be an integer;
• a flow value cannot be negative;
• a date must follow the agreed format;
• a status code must belong to the permitted reference list.

This level of validation is simple but highly effective. It stops many errors before they reach the next system.

Record-level validation

Record-level validation checks the relationships between fields within a single record.

Examples:

• the end date must be later than the start date;
• a “closed” status requires a closing date;
• the number of good units cannot exceed the total number of units;
• a batch with a “released” status must have a quality control result.

In a production environment this level matters particularly for work orders, shift reports, quality events, downtime records, operator forms and maintenance logs.

Batch validation

Batch validation covers an entire file, import, migration or data package processed in a pipeline.

Examples:

• the record count after import must match the record count in the source file;
• the sum of production by shift must equal the daily total;
• the file must not contain duplicate keys;
• the date range in the data package must match the reporting period.

This type of validation is typical of ETL (Extract, Transform, Load) processes and of periodic loads into data warehouses, reports, analytical models and central systems.

Schema validation

Schema validation checks whether data has the expected structure.

It can apply to a database table, a CSV file, a JSON message, an XML document or an API structure. The check covers field names, data types, required columns, constraints, relationships and sometimes permitted values.

Schema validation is important in system integration, because many errors stem not from the value itself but from a change in the data structure — a renamed column, a reordered set of fields in a file, or a new status added without updating the receiving system.

Data validation in industrial environments

Validating industrial data covers signals from sensors, controllers, SCADA and MES systems, historians and operational forms, where errors typically stem from problems with the signal, timing, units, quality statuses and process context.

In industry it is not enough to check that a field contains a number. You need to know whether that number is physically plausible, whether it comes from a reliable source, whether it is current, whether it carries the right unit, and whether it agrees with the other signals describing the same process.

Industrial check type	What it detects	Example
Sensor range check	A value outside the physical or process range	Boiler temperature outside the range permitted for the current operating mode
Spike detection	A sudden, short-lived jump inconsistent with process dynamics	Flow rises by 300% within one second with no change in pump status
Frozen signal detection	A signal that has not moved for too long	A temperature sensor reports an identical value for 12 hours
Timestamp validation	Gaps, duplicates, wrong sequence or inconsistent time zones	Data from several lines is offset by an hour after a clock change
Tag-level validation	An incorrect unit, scale, status or tag description	A pressure signal stored in bar but interpreted as kPa
Cross-sensor validation	Inconsistency between related signals	The pump is running, but flow and pressure do not respond as expected
Alarm validation	False, repeating or inconsistent alarms	A low-level alarm is raised despite a stable sensor reading
Historian completeness check	Missing data, interpolation and compression artefacts	A gap in historical data is wrongly read as a stable process
OPC UA quality code check	The quality status of a value: Good, Bad, Uncertain	A value flagged as Uncertain should not feed a predictive model unflagged

Sensor data validation

Sensor data requires both technical and process validation.

Technical checks confirm that the signal is available, has the correct type, falls within range and carries a current timestamp. Process checks confirm that the signal is consistent with what is known about the installation, its operating mode and the related measurements.

For example, negative flow may be an error in one process but a legitimate state in another where the installation runs in both directions. Validation rules should therefore never be copied across without accounting for equipment context, operating mode and technology.

Sensor data validation is a precondition for reliable predictive maintenance models, because a model built on frozen, offset or incorrectly scaled signals can produce false indications.

Validating historian and time-series data

A data historian stores process data over time, often at high frequency and with compression mechanisms in place. Validating data both before it is written and after it is read helps avoid misreading gaps, interpolation and artefacts.

Checks should cover, among other things:

• continuity of timestamps;
• duplicate samples;
• chronological order;
• time-zone consistency;
• the quality status of each sample;
• the proportion of interpolated data;
• changes in sampling frequency;
• the effect of compression on averages, sums and extremes.

Data should be validated before it is written to the historian and again when it is used in reports or analyses. OPC UA represents the quality of a value through a StatusCode, whose severity can indicate a Good, Bad or Uncertain state.

Validation in OT/IT integration

In OT/IT integration, validation ensures that data keeps its meaning as it moves between control, manufacturing execution, reporting and business systems.

Typical problems include:

• the same asset carrying a different name in SCADA and in MES;
• different units in the source and target systems;
• a time offset between the OT and IT environments;
• a missing quality status on data passed into a report;
• a product code that exists in ERP but not in MES;
• sensor data assigned to the wrong piece of equipment.

Our article on OT/IT integration describes the flow of data between operational and information technology in more detail. From a validation standpoint, the key point is that data must not lose its unit, timestamp, source identifier or quality status along the way.

Validating production and ESG data

Validation also determines the credibility of production indicators such as OEE, good-unit counts, downtime and the quality of shift reports. Detailed production metrics are a separate topic, but their results depend on the correctness of the input data described in our article on OEE.

Energy and environmental data likewise requires checks on completeness, range, units and assignment to the correct period or site. In ESG reporting, errors in meters, time gaps and conversion factors can distort reported emissions, energy consumption and environmental indicators. This area is covered further under ESG Data Management.

The data validation process: how to implement data validation

A data validation process should cover the identification of critical data, the definition of rules, automated execution of checks, exception handling, monitoring and periodic rule review.

The practical flow looks as follows:

Input data → validation rules → pass/fail/warning result → action → monitoring → rule review

1. Identify critical data

Not all data needs the same level of control. Start by identifying the data that affects reports, decisions, compliance, safety, quality, settlements or analytical models.

Examples of critical data:

• production counters;
• scrap and defect data;
• operating and downtime hours;
• energy measurements;
• sensor data feeding predictive models;
• batch and material identifiers;
• quality control results;
• data required for regulatory reporting.

2. Define validation rules

Rules should be unambiguous, testable and linked to a named data owner.

Example rules:

• temperature_value must be a floating-point number;
• pressure_value must fall between 0 and 16 bar;
• end_time must be later than start_time;
• asset_id must exist in the asset reference list;
• quality_status may only take the values Good, Bad or Uncertain;
• the number of good units cannot exceed the total number of units.

Rules can be expressed declaratively — in YAML, JSON or SQL, for example — or programmatically in application code. In larger environments it is worth using a validation rules engine that supports versioning, change history, alerts and exceptions.

3. Decide how errors are handled

Not every breach should lead to data being rejected automatically.

Typical responses include:

• rejecting the record;
• flagging the record;
• routing it to quarantine;
• raising a warning;
• automatic correction under an approved rule;
• escalation to the data owner;
• storing the data with a reduced quality status.

For industrial data it is often more important to retain a value with the appropriate quality status than to delete it. Deleting data creates a gap that may later be misread as the absence of an event.

4. Monitor rule effectiveness

Validation rules should be monitored in the same way as any other element of the data process.

Useful measures include:

• the number of errors per rule;
• the number of errors per data source;
• the share of data rejected, flagged and corrected;
• exception handling time;
• the number of false alarms;
• the impact of errors on reports and processes;
• the most frequently breached rules.

Monitoring makes it possible to tell whether the problem lies with the data source, the integration, the rule configuration or the way users work.

Data validation best practices

Best practice in data validation centres on catching errors as close to the source as possible, automating checks and clearly assigning ownership of the rules.

Do	Don’t
Validate data as close as possible to the point where it is created	Don’t wait until the final report to run checks
Define rules together with the process owners	Don’t build rules on purely technical grounds, without business context
Document the rules, the exceptions and the owners	Don’t leave validation logic buried in undocumented code
Automate repeatable checks	Don’t rely on manual checks for critical validation
Version your validation rules	Don’t change thresholds and reference lists without an audit trail
Distinguish between an error, a warning and missing data	Don’t treat every breach the same way
Preserve the data quality status	Don’t delete problematic data without recording why
Monitor the volume of exceptions and false alarms	Don’t assume a rule set once will stay valid forever
Test rules against historical data	Don’t roll out new rules without assessing the impact on existing processes
Connect validation to data governance	Don’t push the entire responsibility onto the IT team

The single most important principle: validation should be designed together with the process. Energy metering needs one kind of logic, maintenance data another, quality reports another again, and operator form data something different still.

In industrial organisations, a phased rollout works well: start with critical data, move on to data used frequently in reports, and only then extend to a wider range of sources.

Data validation tools and platforms

Data validation tooling spans database constraints, ETL platforms, data quality systems, data observability tools, iPaaS solutions, custom scripts and industrial data platforms.

Vendor	Area	Key capability
IBM	Data integration and quality	IBM InfoSphere Information Server covers data integration, quality and validation
Teradata	Analytics and large data sets	Teradata Vantage supports data validation and quality control in analytical environments
TIBCO	Middleware and data quality	TIBCO Data Quality supports profiling, validation and cleansing
Precisely	Data integrity	The Precisely Data Integrity Suite covers profiling, validation and quality monitoring
Semarchy	Master data management and data quality	Semarchy xDM combines master data management with validation and data quality
Flatfile	Data import and onboarding	Flatfile supports validation of data imported from customers and partners
SnapLogic	iPaaS and integration pipelines	SnapLogic enables data validation within integration flows
Informatica	Enterprise data quality	Informatica Data Quality handles validation rules, profiling and quality monitoring
Atlan	Data governance and data catalogue	Atlan links metadata management with quality and governance processes
Monte Carlo	Data observability	Monte Carlo monitors data pipelines and detects quality anomalies
Smart RDM	Industrial and operational data	Smart RDM can support industrial data validation, exception workflows and operational context

The right tool depends on where the data originates, how quickly you need to react, the data volume, whether real-time operation is required, who owns the rules, and whether a validation result needs to trigger a workflow.

A Manufacturing Data Platform can centralise validated industrial data, while Manufacturing Analytics puts that data to work in analyses, reports and models. Validation does not replace analytics, but it determines whether the results of that analytics can be trusted.

FAQ

What is data validation?

Data validation is the process of checking whether data complies with agreed rules on type, range, format, completeness, consistency and fitness for use.

In practice, it means checking data before it is stored, processed, analysed or used to support a decision.

What are the four types of data validation?

The four types most commonly cited are:

• data type check;
• range check;
• format check;
• consistency check.

In practice, organisations also apply completeness, uniqueness, length, lookup, pattern and cross-system checks.

What are the five types of validation check?

Five common validation checks are:

• data type check;
• range check;
• format check;
• length check;
• lookup or reference check.

In industrial environments it is worth extending this set with checks on timestamps, quality statuses, engineering units, data gaps and consistency between sensors.

What is an example of data validation?

A simple example: a system checks whether an email address follows a valid format.

An industrial example: a system checks that a pressure reading falls within the permitted range, carries a current timestamp and a correct quality status, and is consistent with pump operation and flow in the installation.

Is data validation hard to learn?

The basic rules are easy to learn, because they concern data type, format, range and completeness.

The difficulty grows in industrial environments, where rules must account for process context, equipment operating mode, units, latency, signal dynamics, historical data and dependencies between systems.

How does data validation differ from data verification?

Validation checks whether data complies with agreed rules. Verification checks whether data matches its source or the actual state.

For example, validation confirms that a “temperature” field holds a number within the permitted range. Verification confirms that this value agrees with the source device or another reference system.

How does data validation differ from data quality?

Data validation is a data control technique. Data quality is a broader discipline covering standards, governance, profiling, cleansing, accountability, monitoring and improvement.

Validation helps maintain data quality, but on its own it does not resolve every issue around ownership, definitions, lineage and use.

What are the most important data validation rules?

The most important rules cover data type, range, format, length, completeness, uniqueness, reference values, relationships between fields and consistency across systems.

Industry adds rules for quality statuses, units, timestamps, data gaps, frozen signals, value spikes and consistency between related measurements.

How does data validation work in manufacturing?

In manufacturing, validation checks data from systems such as SCADA, MES, historians, sensors, operator forms and quality systems.

Examples include detecting out-of-range values, frozen signals, gaps in time-series data, incorrect units, duplicate orders, inconsistent production counters and invalid quality statuses.

How do you implement data validation?

Start by identifying critical data, then define the rules, decide how errors are handled, deploy automated checks, monitor exceptions and review the rules periodically.

A good approach is to begin with data that affects reports, decisions, compliance, quality, production and analytical models.

Sources and further reading

1. IBM — materials on data validation, data quality dimensions and data integrity testing.
2. Precisely — materials on the difference between data validation and data verification.
3. OPC Foundation — the OPC UA specification covering StatusCode and data value quality.
4. DAMA / DAMA UK — materials on the dimensions of data quality.
5. GOV.UK — Government Data Quality Framework.
6. Teradata — materials on data validation, processes and tools.
7. Semarchy — materials on validation techniques and data quality.
8. Flatfile — materials on data validation in import and onboarding.
9. SnapLogic — materials on data validation in integration pipelines.