What’s the difference and why it matters
The terms data conversion and data migration are still sometimes used interchangeably by many. However, they do mean different things. Let’s define what we mean.
Data integration: Combining data residing in different sources and providing users with a unified view or access to them.
Data migration: Permanently moving data between computer storage types, file formats or applications.
Data integration involves combining data from several disparate sources, which are stored using various technologies and provide a unified view of the data.
Data integration becomes increasingly important in cases of merging systems of two companies or consolidating applications within one company to provide a unified view of the company’s data assets or to gain access to information from other systems. Need for real-time operational access, analytics, oversight and reporting drive all data integration projects.
The challenge in these data integration projects is to map or link data elements, define rules and make these operational across platforms for constant use over many years.
Data migration is all about moving data and business logic from one system to another. It’s a one-off exercise even though some companies might have several systems to be merged into one in parallel or over time.
Typically, data migration occurs during an upgrade of or move to a completely new system. What usually drive such change is company or organisational mergers or need for new business opportunities to be released through upgrade to new system functionality.
A key question in many organisations is to which extend similar systems should be integrated or merged into one system through a data migration.
The effort doing data integrations and data migrations is often not well understood and the result often delayed, costly and overly time-consuming or an outright failure.
There are many pitfalls that all contribute to the failure of many data integrations or migration efforts.
• Lack of attention and proper management
• Difficult to estimate due to lack of experience
• Improper implementation – low quality full of “surprises”
• Data quality and business rules poorly understood
• Complex and difficult to fix
Migration and Integration are different
• Most often a “one-off” activity
• Often large volumes of data in short time
• Cost to fix is high
• Significant data cleansing needed
• Ongoing activity (synchronisation or replication)
• Managing incremental changes
• Different requirements for different designs (real-time, batch or messaging)
• Usually needed in a small-time window
Yes, it matters!
For the novice, most things are the same. You see a nail or screw and might the look the same, but one needs a hammer the other a screwdriver to work as intended.
Same goes for data migration and data integration. Yes, there are similarities, but you just don’t use the same tools when working with them.
Data integration calls for an ETL tool to build your code, rules and logic with to support a (near) real-time integration framework to last many years a data migration is a one-off where the focus is on speed and flexibility of deployment and less on long-life and performance. A data migration like migFx will do the job much better and cheaper.