A migration is much more than ETL

From a business perspective a migration isn’t just a technical extract – transform – load process there is a lot more to it than that.

The business is usually undertaking a migration for a particular reason; consolidate technology platforms due to acquisitions or such or replacing an existing platform with a new better platform. In either case how you move the data can have a large bearing on the success or failure of the migration process. Sitting down with the business and talking through how a thing in the old world is going to look in the new world and what business rules should be applied to transform the old world thing into the new world thing is hard. Partly it is hard because you can’t see it is generally an abstract discussion which isn’t generally the most successful of conversations to have.

So if you can get someone who knows the current systems at a good level of detail and someone who knows the new systems at a good level of detail you may be able to evolve the discussion to come up with the transformation rules for the migration. One by-product of this discussion will be something I have seen referred to as overs and unders. An over is a field that exists in the old systems that doesn’t exist in the new systems; is it redundant, did we miss something when we specified the new system. Then unders are fields that exist in the new systems but we can’t find a value in the old systems, this is generally some new functionality that will need a default value.

Then I need to extract al the data out of the old systems transform it based on the rules you have given me above and then load it all into the new systems. Sounds simple, except you have a 24×7 business, oops. Now comes the trade-off between adding scope by having no or minimal disruption to the business and the associated cost in time and money to do this.

This is where the fun starts and from an architecture perspective here are some considerations:

  • Do the old systems need to be aware that the migration is occurring to cache changes and the like?
  • How do I extract the data so the old systems can remain online?
  • If the old systems are running once I have taken the extract how to I apply the delta changes to the new systems?
  • How do I cut over from the old system to the new system with no/minimal outage?

One other complexity that you may have to deal with is where your implementation strategy isn’t big bang but a sequence of iterative migrations. The complexity then is that you have to load all of the data into the new systems while they are still running phew!

So a migration is just ETL but the context within which you run the ETL changes the nature of it. This is what I refer to as the configuration of the ETL. So be aware that moving the data is the easy bit working with the business to understand the context that you will run it in is a key part of a successful migration.

Advertisements

Tags:

2 Responses to “A migration is much more than ETL”

  1. Migrating a Large Volumes of Data « What is an Architect? Says:

    […] to deal with the problem of moving a significant volume of data as part of a migration. As outline migration is much more than ETL and the mapping rules that tell you what data to migrate and the systems to get it from won’t be […]

  2. Migration Summary « What is an Architect? Says:

    […] from the start is seemed that migration is just ETL. As the project evolved we soon realised that a migration is much more than ETL. We had to tackle several problems such as migrating a large volume of data. The importance but […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: