From a business perspective a migration isn’t just a technical extract – transform – load process there is a lot more to it than that.
The business is usually undertaking a migration for a particular reason; consolidate technology platforms due to acquisitions or such or replacing an existing platform with a new better platform. In either case how you move the data can have a large bearing on the success or failure of the migration process. Sitting down with the business and talking through how a thing in the old world is going to look in the new world and what business rules should be applied to transform the old world thing into the new world thing is hard. Partly it is hard because you can’t see it is generally an abstract discussion which isn’t generally the most successful of conversations to have.
So if you can get someone who knows the current systems at a good level of detail and someone who knows the new systems at a good level of detail you may be able to evolve the discussion to come up with the transformation rules for the migration. One by-product of this discussion will be something I have seen referred to as overs and unders. An over is a field that exists in the old systems that doesn’t exist in the new systems; is it redundant, did we miss something when we specified the new system. Then unders are fields that exist in the new systems but we can’t find a value in the old systems, this is generally some new functionality that will need a default value.
Then I need to extract al the data out of the old systems transform it based on the rules you have given me above and then load it all into the new systems. Sounds simple, except you have a 24×7 business, oops. Now comes the trade-off between adding scope by having no or minimal disruption to the business and the associated cost in time and money to do this.
This is where the fun starts and from an architecture perspective here are some considerations:
- Do the old systems need to be aware that the migration is occurring to cache changes and the like?
- How do I extract the data so the old systems can remain online?
- If the old systems are running once I have taken the extract how to I apply the delta changes to the new systems?
- How do I cut over from the old system to the new system with no/minimal outage?
One other complexity that you may have to deal with is where your implementation strategy isn’t big bang but a sequence of iterative migrations. The complexity then is that you have to load all of the data into the new systems while they are still running phew!
So a migration is just ETL but the context within which you run the ETL changes the nature of it. This is what I refer to as the configuration of the ETL. So be aware that moving the data is the easy bit working with the business to understand the context that you will run it in is a key part of a successful migration.