Defaulting rather than exception

As part of our fact-finding for the migration we were working on, we looked at several previous migration projects to learn from them. One idea that one particular migration had adopted was defaulting rather than exception out any data that fails. Obviously the type of data you are migrating and the capabilities of the target system have a big bearing on how successful this strategy will be.

This isn’t a substitute for not doing proper data profiling and associated data cleansing before the migration. When doing the trial runs you should also be reviewing the outcome of these and doing any data cleansing to fix any data errors. When the actual migration occurs, the number of exceptions should be minimal.

The goal of this strategy is to get as much data into the target system with each migration. If doing a single migration the chance of an exception occurring that requires backing out the entire migration is greatly reduced. If doing multiple migrations then this reduces the occurrence where something fails and has to be backed out and then scheduled into the next migration.

The first consideration is what type of data you are migrating and the types of errors that can occur during the migration. These errors can include: 

• Data type errors
• Referential integrity rules
• Business rules

For each of the data entities that you identify you need to look at the errors that can occur and develop an appropriate “default” entity that you can load these into. It could be as simple as creating a “draft” status for an order where no rules are enforced and then use this. It would be best to have this status unique for the migration as you will need to go over all the data placed in this status and manually repair these.

The range of exceptions that can and probably will occur will be varied, but if you have done adequate profiling and give enough thought you should come up with some appropriate scenarios. The key to doing this is to have an appropriate process in place to go through the entities that have been “defaulted” and make necessary fixes to move these entities into a valid state.

The concept is relatively simple but developing the business rules to apply will be the hard part. In effect it is just a mapping rule to say if it doesn’t fit one of these, then shove it in here.

Advertisement

Tags: Best Practice, Migration