Joe Thompson | Data Done Right

Overview

What is quite common in many companies is data is everywhere and none of it is integrated. It is spread out on a spreadsheet on someone's computer, documents and a variety of apps. When this happens the company is full of data solos and they miss out on opportunities.

However, if you integrate your data you reap the rewards. Integrated data can give you a complete picture of your entire organization and how everything interacts together so you can seize upon opportunities as they present themselves.

Key Concepts

The most important attributes of data integration.

Integration Methods

Data Integration can be done via many methods. Point-to-point, Hub-and-spoke, Publish subscribe and more. The exact method depends upon project requirements.

Data Standards

Don't attempt to integrate data without standards. Standards give a level of consistency and reliability that data from multiple systems need.

Latency

Latency is the time it takes to integrate the data from when it was created. Does it need to be in real time or will an overnight batch job do? In most cases it would be the latter.

Mapping

Mapping matches the source data to a target destination. It would also consider ETL and any needed transformations.

Lineage

Lineage logs how data has changed over time from its creation to its final destination. Much of this would be stored as metadata.

Profile

Data profiling helps with integration as you better understand the structure and format of the data being integrated by examining what is assumed about data vs. what is actual.

Data Integration