Posts Tagged ‘Best Practice’

Migration Summary

7 January 2011

Over the last couple of weeks I have put together some of my key learning’s from the last nine month, leading the architecture team on a migration project. There were a lot of firsts in this project for me, both from the problem space of the project but also leading a relatively large architecture team. This has been a good change to reflect on the project and take some time to capture these learnings.

By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest.

Leading a team of architects presented several challenges one simple problem of having multiple architects but one architecture to document. A simple habit that the team got into which helped develop a culture of knowledge sharing was the Architecture Review Meeting. At the core of the whole project we developing a Reference Architecture to tie everything back to. Last but not least, we used a technique from the agile world called the retrospective to provide a bit of continuous improvement as we moved from one part of the project to the next.

Doing a migration project was challenging, from the start is seemed that migration is just ETL. As the project evolved we soon realised that a migration is much more than ETL. We had to tackle several problems such as migrating a large volume of data. The importance but also the frustration of having to define what would happen on the day. By looking at other migration projects we came across a technique of defaulting rather than exception.

What a journey!


Defaulting rather than exception

6 January 2011

As part of our fact-finding for the migration we were working on, we looked at several previous migration projects to learn from them. One idea that one particular migration had adopted was defaulting rather than exception out any data that fails. Obviously the type of data you are migrating and the capabilities of the target system have a big bearing on how successful this strategy will be.

This isn’t a substitute for not doing proper data profiling and associated data cleansing before the migration. When doing the trial runs you should also be reviewing the outcome of these and doing any data cleansing to fix any data errors. When the actual migration occurs, the number of exceptions should be minimal.

The goal of this strategy is to get as much data into the target system with each migration. If doing a single migration the chance of an exception occurring that requires backing out the entire migration is greatly reduced. If doing multiple migrations then this reduces the occurrence where something fails and has to be backed out and then scheduled into the next migration.

The first consideration is what type of data you are migrating and the types of errors that can occur during the migration. These errors can include: 

  • Data type errors
  • Referential integrity rules
  • Business rules

For each of the data entities that you identify you need to look at the errors that can occur and develop an appropriate “default” entity that you can load these into. It could be as simple as creating a “draft” status for an order where no rules are enforced and then use this. It would be best to have this status unique for the migration as you will need to go over all the data placed in this status and manually repair these.

The range of exceptions that can and probably will occur will be varied, but if you have done adequate profiling and give enough thought you should come up with some appropriate scenarios. The key to doing this is to have an appropriate process in place to go through the entities that have been “defaulted” and make necessary fixes to move these entities into a valid state.

The concept is relatively simple but developing the business rules to apply will be the hard part. In effect it is just a mapping rule to say if it doesn’t fit one of these, then shove it in here.

Multiple architects but one architecture

5 January 2011

One problem with large projects is managing multiple architects working on the one architecture. You need to be able to decompose the problem space into chunks that you can assign to individual architects to focus on. Then there is how to document the architecture, do you produce one architecture document or multiple architecture documents.

Unfortunately this seems to be one of those problems where there is no perfect solution. Every way you decide to tackle it you will get to the end and not want to use that approach again because of the issues it introduced. From previous projects there seem to be three logical ways to tackle a problem like this:

  1. Have multiple architects editing the one architecture document
  2. Break the project into domains and write an architecture document for each domain
  3. Have an architect document a domain and then consolidate into a single architecture document

The first problem is defining the domains that people will work on. For the recent migration project I was on we used a reference architecture to do this. Once you have the break up of the problem space you then have to deal with how you tackle documenting it.

1. Have multiple architects editing the one architecture document


  • The architecture of the project is in one document
  • Don’t need to repeat content across multiple documents
  • Easy to identify if there are any gaps in the architecture


  • Complexity of multiple people editing a single document
  • Domains don’t necessarily align to the document sections
  • Different people have different writing styles
  • Hard to track progress

2. Break the project into domains and write an architecture document for each domain


  • Architects are largely self-sufficient
  • Easy to track progress


  • Have to read multiple documents to understand the architecture of the project
  • Some content will need to be repeated across document
  • Hard to manage things like issues, risks, dependencies etc across documents
  • Gaps in the architecture aren’t always relevant

3. Have an architect document a domain and then consolidate into a single architecture document


  • The architecture of the project is in one document
  • Easy to track progress
  • Architects are largely self-sufficient
  • Don’t need to repeat content across multiple documents


  • Different people have different writing styles when consolidating

For the recent migration project that I was on we went for the “Have an architect document a domain and then consolidate into a single document”. We produced a lite version of the architecture document that we called a Solution Brief. This was just a cut down version of the architecture document with only some of the sections. But at the end of the process we were able to just cut and paste the content of the Solution Briefs into the single architecture document.

Look it wasn’t perfect but I think the process was the better of the three. The big learning was to define the domains that people are working on well as if you don’t get this right up front you are constantly addressing scope issues for the domains you have defined.

PowerPoint Tip

5 December 2008

I attended a presentation skills training course recently and was told about an interesting feature of PowerPoint. When you are displaying the slide show pressing B will make the screen Black and pressing W will make the screen White. I am amazed that after all these years I have not come across this.

Train…ing developers to write better code

8 August 2008

While sitting on the train the other morning coming into work, I was suprised when I looked up at the Message board and saw the following announcement.

Arriving Camberwell change for Unknown LineID [15] services

My mind started spinning as to what was the architecture that resulted in the message. Are the messages sent to the train or stored on the train? Either way you would have thought that the requirements for such a service would not have specified the displayed message but something a bit more user friendly.

Big Problems and Little Problems

13 June 2008

Listening to an architecture presentation on InfoQ there was an interesting comment made about how an architect should prioritise decisions. An interesting point raised was around the impact of resolving big architecture decisions early in that these decisions allow smaller problems can be easier. The reverse of this is even more significant, in that making the easy architecture decisions before the big ones constrains what decisions you can make to solve you big problems.

This is one of those common sense type statements that is good to make a conscious decisions when creating an architecture.

If the glove fits

20 May 2008

With the constant debate in the blogosphere about which technology is better than another it was refreshing to read in a post about the Facebook Chat Architecture the following quote:

Why Erlang? In short, because the problem domain fits Erlang like a glove.

People too often learn a tool and then decide any problem can be solved with this tool aka Give a man a hammer and everything looks like a nail. I prefer to add as many tools to my toolbox as I can while still knowing how to use them all and then pick the right tool for the right job. You also need to review your tool box and refresh it as new tools come on the market you don’t want to come out with a hammer when everyone else has a nail gun.

Ted on Pragmatic Architecture

11 February 2008

Ted Neward has had a series on Pragmatic Architecture on the MSDN site for a while, definitely some interesting content which as Ted points out is quite technology-neutral.

Inbox Zero

11 January 2008

Inbox ZeroBefore Christmas a colleague introduced me to a process called Inbox Zero which provides a way to manage your inbox as apposed to having your inbox manage you. Well being the new year I thought it would be a good time to implement it. There ia a good webcast of Merlin Mann the creator of Inbox Zero presenting it to what seems to be Google.

One thing I seem to be finding now that I have a pretty empty inbox and shouldn’t be scanning it every couple of minutes is that I am pretty sure that I was using my inbox as a procrastination tool. It reminds me of the homework days when I used to go from my room to the kitchen and scan the cupboard on the odd chance that a chocolate bar had majically appeared since I last looked five minute prior.

Well now that I have procrastinated by blogging about it I had better get back to my task at had.

Sorry Boss!

Domain-Driven Design: I think I should read this

20 December 2007

Over about a six month period I constantly came across the book Domain-Driven Design by Eric Evans on peoples desks. Because of this I also found myself picking this book up in the bookshop probably because it has an eye catching cover. Having had these experiences I decided it was a book that I had to read so I have started, I am through the introduction part of the book and am now into the meat.