Joe Thompson | Data Done Right

Overview

Data Engineering is concerned with the movement of data between systems. To use the master planned community metaphor again, Data Engineering is the streets, sewers, lighting and electrical that every house needs access to. Data Engineering implements the plans of Data Architecture.

I like to describe data engineering this way: It delivers the right data, at the right time, to the right destination in the right format.

Key Concepts

The most important attributes of data engineering.

Data Pipelines

Data pipelines capture data from a myriad of sources . It is a key feature of data integration. It involves Extract Transform Load or ETL to move data from one database to another, typically automated.

API

An API or Application Programming Interface is a way to receive data form a third party. This is used in both custom development to add features and in data engineering to retrieve data.

Data Cleaning

Data cleaning ensures that your data follows standards and is of high quality. It is an essentially part of data engineering and involves proper formatting, missing values and other issues.

Data Enrichment

Data enrichment adds additional data to your database. It helps your to have a more detailed picture to understand the customer or market better.

Data Migration

Data migration is typically used when moving a system to a new vendor or performing and upgrade. It moves the data from the old database structure to the new structure.

Data Processing

Data processing is performing automated transformations, cleaning or analysis of a data set. It is typically run overnight as new data becomes available.

Data Engineering