CSV and Excel in Smart RDM: how to regain control of knowledge hidden in files

Mapping, validation, and automated ingestion of CSV data into Smart RDM

In many industrial companies, the most important operational knowledge does not reside only in central IT systems. Very often, it is stored in CSV and Excel files: laboratory exports, maintenance summaries, shift reports, local production applications, measurement device data, or spreadsheets prepared manually by employees.

The file itself is not the problem. The problem is that files quickly become an informal database for the organization. They circulate by email, sit on network drives, accumulate multiple versions, are corrected manually, and often only one person knows where they came from and what the individual columns actually mean. As everyone knows, the truly “irreplaceable” solutions are often VBA-based files that also contain knowledge embedded by individual employees.

In this model, knowledge exists, but it is difficult to manage. An organization may have data on failures, quality, energy consumption, process parameters, or machine operation, while still being unable to confirm whether the data is complete, current, correctly described, and suitable for analysis.

The Smart RDM CSV Integrator brings order to this area. It enables data from files to be loaded into the managed Smart RDM data model, mapped from source tags to the correct datapoints, checked for record quality, imported automatically on a recurring basis, and tracked throughout the entire process. As a result, CSV stops being a loose attachment and becomes a managed data ingestion channel.

The challenge: knowledge hidden in files

In practice, CSV and Excel files are often the only place where an organization holds the data required for analysis. This is particularly common with legacy systems, local applications, supplier-generated reports, laboratory data, periodic readings, and manually prepared production summaries.

As long as such a file is used only for a one-off check of a few figures, the risk may appear limited. The problem begins when the data is expected to support dashboards, KPIs, predictive models, quality analyses, business reports, or maintenance decisions.

At that point, questions arise that an ordinary file folder cannot answer effectively:

• Which version of the file is the correct one?

• Who prepared the data, and when?

• Were all records imported?

• Which records were rejected, and why?

• Does a tag in the file represent the same parameter that the system expects?

• Can the data be safely linked to a machine, production line, area, or process?

This is not only a technical issue. It is an operational knowledge management issue. The data exists within the organization, but without a shared model, validation, and history, it is difficult to use it in a repeatable and reliable way.

Why CSV still matters in industry

In an ideal world, every data source would provide a stable API, complete documentation, a clearly defined data model, and a ready-made integration. In real industrial environments, this is rarely the case. Some systems are legacy solutions, some operate locally, some are controlled by suppliers, and some data is created outside central systems.

CSV remains important because it is simple, widely supported, and easy to generate. For many data sources, it is the fastest and most cost-effective way to transfer data for further analysis. This does not mean, however, that the import process should be manual, undocumented, and uncontrolled.

The most common scenarios in which CSV delivers genuine business value include:

• Rapid pilot deployment. Before a full integration is developed, Smart RDM can be supplied with historical data to verify whether the business use case is viable.

• Batch data imports. Laboratory results, shift reports, audits, quality data, and production summaries are often sufficient when delivered daily, weekly, or monthly.

• Working with legacy systems. Not every source provides a convenient API. A file can serve as a practical bridge between a local system and an analytics platform.

• Organizing historical data. Before predictive analytics, dashboards, or KPIs can be launched, measurement history often needs to be imported from many distributed files.

• Supplying CRD and Smart RDM where no integration is available yet. In many organizations, a file is the first, and sometimes the only, practically accessible data source for the data model being built.

What the Smart RDM CSV Integrator changes

The main value of the integrator does not lie simply in its ability to “load a file.” Tools capable of doing that have existed for years. Its value lies in turning an import from a one-off operation performed outside the system into a structured process with mapping, validation, reporting, scheduling, and execution history.

A business user or administrator can configure an import without involving a developer for every batch of data. This does not mean that the process is unrestricted. The file must be connected to the Smart RDM data model, and the records must pass quality checks before they are saved.

In practice, this shortens the path from data to analysis. The organization can populate a dashboard more quickly, test a predictive model, analyze failure history, or verify the quality of source data. Only then does it need to decide whether a given use case requires a full integration through a ready-made connector, an API, OPC UA, PI System, or another target integration method.

From file to datapoint: a manageable import process

The import process can be described in simple terms. Smart RDM helps transform a file whose meaning may be local and informal into data assigned to specific datapoints in the platform model.

1. Uploading the file and checking its structure

The user selects a CSV file and specifies the data separator. The integrator supports common formats separated by commas, semicolons, or tabs. The system checks whether the file can be read, whether it contains any data, and whether column headers are present. The user can also use a sample template to reduce the risk of errors at the data preparation stage.

An example file structure may look as follows:

tag;value;timestamp;dataPoint

temperature_bearing_01;72.4;1777452965;MOTOR_01_BEARING_TEMP

energy_consumption;1240.8;1777452965;LINE_02_ENERGY

machine_status;true;1777452965;MACHINE_03_STATUS

2. Mapping source columns

Column names in files are rarely fully standardized. The same parameter may appear as temp, temperature, bearing_temp, T_01, or Bearing_Temperature. The integrator allows the user to specify which columns correspond to the logical fields tag, value, timestamp, and datapoint. The system can suggest a mapping automatically, and the user can adjust it manually.

3. Mapping tags to datapoints

This is the most important step from an information quality perspective. A tag is an identifier from the data source. A datapoint is an element of the Smart RDM model that can later be used by analyses, dashboards, quality rules, alarms, or workflows. The objective is therefore not simply to store the value 72.4. The system must understand that the value represents the temperature of a specific bearing, in a specific machine, on a specific production line.

4. Validation and the import report

Before saving the data, the system checks whether all required fields are present, whether the time format is correct, whether the value type is valid, and whether the record can be assigned to the correct datapoint. This makes the import part of the data quality process rather than a black box.

Import without a developer, but not without rules

In many organizations, every new data feed requires an IT request, a wait for team availability, and the manual preparation of a one-off script. This slows down business processes and places an unnecessary burden on IT teams with tasks that should be repeatable and manageable.

The CSV Integrator changes this model. It allows users to run imports independently, but within defined rules. Data does not enter the system through an uncontrolled process. It is mapped, validated, reported, and recorded in the execution history. As a result, the business gains speed, while IT retains control over how data is ingested into the platform.

This is particularly important during initial Smart RDM deployments. The team can test the product using real data without waiting for a full integration. If the pilot confirms value, the target connection can be implemented later through ready-made connectors or a dedicated integration with the source system.

Not all or nothing: record level error handling

One common problem with file imports is the all or nothing approach. A single invalid record may cause the entire batch to be rejected. In practice, this wastes time because most of the data may be correct and ready for use.

Smart RDM validates data at the record level. Valid records can be imported, while invalid records are rejected and described in the report. The user can see what was saved, what requires correction, and why.

The import report answers specific questions:

• How many records were imported successfully?

• How many records were rejected?

• Which tag or datapoint is affected by the issue?

• Is the issue caused by the time format, value type, a missing field, or a missing mapping?

• Did the import finish successfully, partially successfully, or with an error?

This approach reduces diagnostic time. The user does not need to search manually through thousands of rows. They can correct the source data or mapping configuration and repeat the import in a controlled manner.

One time or recurring

Not every use case requires a permanent integration. For this reason, the CSV Integrator supports two primary operating modes.

• A one-time integrator is suitable for historical data migration, pilots, one-off analyses, importing data from a specific event, or quickly populating a dashboard.

• A recurring integrator allows the user to specify a folder and a schedule for retrieving subsequent files. Imports can run every minute, hour, day, week, month, or year.

For example, a local production system may generate a CSV report every day after the end of a shift. Instead of emailing the file to an analyst, the organization can define a folder and a schedule. Smart RDM will regularly retrieve the files, validate the records, and supply the correct datapoints.

An administrator can pause the schedule, resume it, run an import immediately, modify the configuration, or delete the integrator definition.

Data types and quality

Operational data is not uniform. A single file may contain numeric values, text, dates, timestamps, or Boolean values. The CSV Integrator supports common data types, including floating point numbers, integers, text, date and time values, and true or false values.

This matters because many analytical problems begin much earlier than the dashboard or algorithm stage. If a numeric value is stored as text, a timestamp uses the wrong format, a tag does not match any datapoint, or columns are shifted because of an incorrect separator, the analytical result will not be reliable.

Data quality should therefore begin before data is saved. The CSV Integrator helps detect errors at the point of entry and turns importing into a repeatable data control process.

History, accountability, and change tracking

In a production environment, it is not enough to know that data is present in the system. It is also necessary to know when it was imported, by whom, from which file, with what result, and whether any errors occurred.

Smart RDM therefore separates integrator configuration from execution history. Users can review
one-time and recurring imports, filter them by integrator name, status, the user who initiated the import, or date range, and inspect the processing result.

This process is important not only for administrators. It also supports data owners, analysts, quality teams, maintenance teams, and people responsible for business processes. Whenever questions arise, the organization can return to the history and verify where the data used in an analysis came from.

CSV as a step toward full integration

The CSV Integrator should not be treated as an inferior substitute for an online integration. In many cases, it is the best first step. It enables the organization to verify the value of the data quickly, organize the model, identify quality issues, and launch initial analyses without a large integration project.

After this validation, it becomes easier to decide whether a given use case requires real-time integration or whether a recurring import is sufficient. If a target integration is needed, it can later be implemented using ready-made connectors or a dedicated connection to the data source.

The key point is that CSV and Excel files should not remain outside the data management system. If they are a genuine source of organizational knowledge, they should have a place in a controlled process that includes mapping, validation, scheduling, reporting, and history.

To sum up

There is no point pretending that CSV and Excel files do not exist. They are widely used in industry and will remain in use for a long time. The difference lies in whether an organization treats them as unmanaged attachments or as data sources that can be structured and used effectively.

The CSV Integrator makes it possible to move from distributed files to a reliable data feed for the Smart RDM data model. It enables the business to get started faster, gives IT greater control, and provides the organization with something even more important: the ability to recover knowledge previously lost in spreadsheets, folders, and emails.