Adding bulk data

The Pretectum CMDM allows you to add data through three modalities, via API, via a quick entry data form, and in bulk.

Adding bulk data is easily done via data staged in a CSV file, especially where the data schema is aligned.

Adding data when the columns in the dataset are different from the schema definition requires remapping.

Navigate to the schema you wish to add data to and establish a dataset.

Though datasets may all be described similarly, the datasets must all have unique names. The ability to have multiple datasets within a schema allows you to maintain multiple versions of the dataset independently of one another but still using the same schema. You may choose to do this for different business scenarios like test vs production systems.

The Schema overview screen will show the underlying datasets for each schema where your cursor is focused and the schema selected

Schema without datasets
Schema with three datasets

When adding a dataset you are only required to provide a name for the dataset and a description.

Provide a Name and Description for your unique dataset

You can add data from a file via the Load data from a file button or you can edit a dataset and add data that way by clicking on the pencil icon.

When adding data from a file you will be provided with a drop zone to add the file and a preview will be provided. Columns in the dataset that perfectly align with the schema will reflect a green checkmark. You can also remap columns at this stage ahead of the bulk load.

At load time you have the options to replace the contents of the dataset or append the dataset. Appending data might be appropriate if you have an upstream Change Data Capture data source that provides new records. Clicking on Verify and Upload will create a background job and start the process of loading and data verification.

The job overview screen will then automatically load or you can navigate to this screen from the ribbon.

Job in progress from the Jobs Monitor

Once the job has completed, it will reflect a record summary which contains additional data describing what has happened

Completed job

The summary for the dataset will have been updated

Summary of dataset

In addition, the dataset overview will describe whether the dataset load contains errors

A more detailed overview of the errors in the loaded data can be found in the home page or by navigation to the dataset directly.

Errors found in the dataset