Posts Tagged ‘Migration’

Migration Summary

7 January 2011

Over the last couple of weeks I have put together some of my key learning’s from the last nine month, leading the architecture team on a migration project. There were a lot of firsts in this project for me, both from the problem space of the project but also leading a relatively large architecture team. This has been a good change to reflect on the project and take some time to capture these learnings.

By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest.

Leading a team of architects presented several challenges one simple problem of having multiple architects but one architecture to document. A simple habit that the team got into which helped develop a culture of knowledge sharing was the Architecture Review Meeting. At the core of the whole project we developing a Reference Architecture to tie everything back to. Last but not least, we used a technique from the agile world called the retrospective to provide a bit of continuous improvement as we moved from one part of the project to the next.

Doing a migration project was challenging, from the start is seemed that migration is just ETL. As the project evolved we soon realised that a migration is much more than ETL. We had to tackle several problems such as migrating a large volume of data. The importance but also the frustration of having to define what would happen on the day. By looking at other migration projects we came across a technique of defaulting rather than exception.

What a journey!


Defaulting rather than exception

6 January 2011

As part of our fact-finding for the migration we were working on, we looked at several previous migration projects to learn from them. One idea that one particular migration had adopted was defaulting rather than exception out any data that fails. Obviously the type of data you are migrating and the capabilities of the target system have a big bearing on how successful this strategy will be.

This isn’t a substitute for not doing proper data profiling and associated data cleansing before the migration. When doing the trial runs you should also be reviewing the outcome of these and doing any data cleansing to fix any data errors. When the actual migration occurs, the number of exceptions should be minimal.

The goal of this strategy is to get as much data into the target system with each migration. If doing a single migration the chance of an exception occurring that requires backing out the entire migration is greatly reduced. If doing multiple migrations then this reduces the occurrence where something fails and has to be backed out and then scheduled into the next migration.

The first consideration is what type of data you are migrating and the types of errors that can occur during the migration. These errors can include: 

  • Data type errors
  • Referential integrity rules
  • Business rules

For each of the data entities that you identify you need to look at the errors that can occur and develop an appropriate “default” entity that you can load these into. It could be as simple as creating a “draft” status for an order where no rules are enforced and then use this. It would be best to have this status unique for the migration as you will need to go over all the data placed in this status and manually repair these.

The range of exceptions that can and probably will occur will be varied, but if you have done adequate profiling and give enough thought you should come up with some appropriate scenarios. The key to doing this is to have an appropriate process in place to go through the entities that have been “defaulted” and make necessary fixes to move these entities into a valid state.

The concept is relatively simple but developing the business rules to apply will be the hard part. In effect it is just a mapping rule to say if it doesn’t fit one of these, then shove it in here.

Multiple architects but one architecture

5 January 2011

One problem with large projects is managing multiple architects working on the one architecture. You need to be able to decompose the problem space into chunks that you can assign to individual architects to focus on. Then there is how to document the architecture, do you produce one architecture document or multiple architecture documents.

Unfortunately this seems to be one of those problems where there is no perfect solution. Every way you decide to tackle it you will get to the end and not want to use that approach again because of the issues it introduced. From previous projects there seem to be three logical ways to tackle a problem like this:

  1. Have multiple architects editing the one architecture document
  2. Break the project into domains and write an architecture document for each domain
  3. Have an architect document a domain and then consolidate into a single architecture document

The first problem is defining the domains that people will work on. For the recent migration project I was on we used a reference architecture to do this. Once you have the break up of the problem space you then have to deal with how you tackle documenting it.

1. Have multiple architects editing the one architecture document


  • The architecture of the project is in one document
  • Don’t need to repeat content across multiple documents
  • Easy to identify if there are any gaps in the architecture


  • Complexity of multiple people editing a single document
  • Domains don’t necessarily align to the document sections
  • Different people have different writing styles
  • Hard to track progress

2. Break the project into domains and write an architecture document for each domain


  • Architects are largely self-sufficient
  • Easy to track progress


  • Have to read multiple documents to understand the architecture of the project
  • Some content will need to be repeated across document
  • Hard to manage things like issues, risks, dependencies etc across documents
  • Gaps in the architecture aren’t always relevant

3. Have an architect document a domain and then consolidate into a single architecture document


  • The architecture of the project is in one document
  • Easy to track progress
  • Architects are largely self-sufficient
  • Don’t need to repeat content across multiple documents


  • Different people have different writing styles when consolidating

For the recent migration project that I was on we went for the “Have an architect document a domain and then consolidate into a single document”. We produced a lite version of the architecture document that we called a Solution Brief. This was just a cut down version of the architecture document with only some of the sections. But at the end of the process we were able to just cut and paste the content of the Solution Briefs into the single architecture document.

Look it wasn’t perfect but I think the process was the better of the three. The big learning was to define the domains that people are working on well as if you don’t get this right up front you are constantly addressing scope issues for the domains you have defined.

On the Day

4 January 2011

Unlike a normal project where you are delivering a technical solution to solve a problem that has a life of years or even decades, a migration can be a one-off event. As the migration project forms there are a lot of interest in understanding what will happen “On the Day”. A migration is usually a critical event that will impact a large portion of the organisation including people involved with the old systems as well as the new system being migrated to.

One of the key questions that we needed to answer has to do with the expected duration of the migration. Although the actual amount of time that is needed for the migration is hard to define at this level, we can identify what dependencies drive when things can occur. The first key point is to understand when we could start the migration, when does the business want to take the current systems offline or in what state do the new systems need to be in to receive the data. Is there some type of End of Day activity that has to occur before we can start the migration. Once you have figured out when you can start the migration there is a standard set of technical steps involving Extracting Transforming and then Loading the data.

Then there is a step where we need to reconcile the data that has been loaded to confirm that everything occurred as it should have. This is a complex task to define early in the project because the business will find it hard to define what they need to reconcile. The first stab will be to reconcile basically every field or pretty close to. It really isn’t until you are able to perform your first trial migration that you can narrow down what needs to be reconciled and at what level. I think there is an aspect of trusting something that hasn’t been built. It isn’t until you do a migration and there is a report that says I migrated 100,000 orders and there are 100,000 orders in then new systems that you can delegate that responsibility to a piece of software.

Reconciliation will have a large bearing on how long the running of the migration on the day will take. The first trial migration you perform the reconciliation step will take a long time. There will be no pressure time wise as it will be a trial and there will be a natural tendency to be ultra cautious. Once you have completed this initial trial migration it could be a good  check-point to review the requirements for reconciliation. I know a waterfall methodology wouldn’t cope with this approach but I think this is critical to ensuring the reconciliation is successful. It will also allow constructive conversations around the amount of information that needs to be visible versus presenting the information in summary to make the reconciliation more efficient. At the end of this review you should be able to come up with a straw-man schedule for the reconciliation and planned duration.

Laying out the migration activity you will have at a high level the following activities:

At some stage through this process you will have to identify what we referred to as “The Point of No Return”. This was a point in the migration past which you could not abort the migration and just re-enable the old system. The architecture of the migration to a large degree will influence when this point is in the migration and the goal will be to make it as late as possible. In conjunction with the Point of No Return there will need to be a business Go or No Go decision on the migration. The system constraints that will influence this will be unique for each migration, so will vary greatly. The key is to be aware that this point will exist and to identify and communicate it early.

What will happen “On the Day” is important but also is something that many people will influence so will evolve. I have outlined a high level process above, use it or create your own and progressively define the details of each of these steps as you go. The duration of the migration will be a key question as discussed in Migration is just ETL you need to identify points in the architecture where you can tune for performance.


31 December 2010

On an agile project a few years back I got to experience first hand the retrospective. The project ran one of these at the end of each iteration and it was an easy going meeting over a beer last thing on the Friday. More as an observer I found it to be an effective meeting where concerns were able to be raised in a constructive manner. It was also good to be able to see the effect these meetings had on behaviours on the project with issues in one retrospective becoming a pro in a future one.

On the migration project that I have been working on we hit a junction going from Part A of the project to Part B. I wanted a way to check in with the team and get some feedback on how thing were going. A retrospective format session seemed like the right way to do this. The first challenge I had, was coming up with the right questions to ask, after much searching I settled on the following:

  • What one aspect of this project would you keep unchanged?
  • What one aspect of this project would you change if you could?

We ran the meeting going around the room with each person having to provide an answer to the question being asked. This was to ensure an equal contribution from everyone on the team. We went around the room twice for each question to get a minimum of 2 answers to each question from each team member. After the two times around the room we opened it to the floor for other suggestions.

One criterion I imposed as we weren’t to solve the problem just raise them in this forum. Unfortunately we have a tendency to want to solve every problem. Keeping it to just raising an aspect we were able to keep the session to 60 minutes.

Coming out of the meeting it wasn’t necessary to have a follow-up meeting and develop a 10 point plan to address all of the issues blah blah blah. A retrospective is good in that it provides a check point to look back (retrospectively) at part of the project and get different opinions on how it went. Just taking the time to hear everyone’s thoughts causes a change in behaviour. Doing it as a one-off probably hasn’t caused a huge change compared to doing it weekly or fortnightly but feedback as been positive.

Most waterfall project I have been on do post-implementation reviews (PIRs) the problem with them is that they are usually after the event. It would be good to find a way to incorporate the retrospective into a project while it was running. This may be a challenge for the next project I am on.

Migrating a Large Volumes of Data

30 December 2010

One of the big problems when performing a migration is the movement of a large volume of data around the network. This may have to occur during normal business operation if you are dealing with an organisation that runs 24×7 or you may have exclusive use of the network out of hours. Either way you could be moving a large amount of data across a network. Transferring 1GB of data will take about 15 minutes over a 10 Mbps network.

Remember also that you will need to execute the migration multiple times throughout the project. In the early stages this will just be with test data but you will need to be running it several time with production data during mock runs.

The first scenario is that you just wear the time cost to move the data off your production systems onto the migration platform. This may or may not be acceptable depending on what outage will the business will accept for the entire migration. Some alternatives you have here are to increase the bandwidth of the network. If you increase the bandwidth from 10 Mbps to 100 Mbps that same 1GB of data would only take 1 min 30 sec to transfer.

Another way of tackling this problem is to reduce the amount of data you have to transfer on the day of the migration by pre-migrating some of the data. Some data can be categorised as read-only so can be migrated days or weeks before the actual migration is to occur. The key here is to identify this data; it could be reference data, orders that have been delivered etc. Once you have identified the data that can be migrated early, the migration on the day will only need to migrate that data that hasn’t already been migrated. This may be significant or not depending on the nature of the data that needs to be migrated.

Change data capture (CDC) provides a pattern where a copy of the data can be taken usually from a back-up or other means and then kept in sync by replicating the changes to the copy. Firstly getting the copy of the data from the source system can be done in a non evasive manner by restoring an offline backup. Then generally the tool will use the log for the source system to capture the changes to replicate to the copy. This generally puts a negligible performance impact on the source system so should not impact to production running of the source system. This will come down to the way CDC has been implemented by the vendor as to any impact to the source systems.

There are a few alternatives to how to deal with the problem of moving a significant volume of data as part of a migration. As outline migration is much more than ETL and the mapping rules that tell you what data to migrate and the systems to get it from won’t be know until relatively late in the project. So from an architecture perspective you may decide on an approach to move the data but may also need a strategy on an alternative if you find the timings are too long.

Architecture Review Meetings

29 December 2010

One technique I picked up from a colleague that I adopted on my most recent project, was having regular architecture review sessions. We ended up having these twice a week and it was a forum where we could review designs as a team. We were able to extend these sessions to include people like the security and infrastructure architects as well as the design lead for the project.

Being a fairly disorganised person we ran these sessions in the early stages as a free-flowing meeting more as a general discussion. As we progressed to align to project schedules and all that good stuff we were a bit more organised and had an agenda and had a minute taker to capture all those juicy bits of feedback that were raised in these sessions.

I found these sessions to be beneficial for the team firstly we developed a behaviour in the team of bringing designs in the early stages of forming to these sessions. With the varied experiences of everyone in the room there was some great alternatives presented that ultimately helped us come up with a better design. This behaviour was also good in that the team was happy to bounce ideas around in these forums, people have a tendency to take feedback personally but I think we got to the point where the feedback was constructive and in some cases was intentionally sought by the author.

Given the size of the team these sessions also provided a great forum for the team to get visibility of parts of the design being completed by others. If we had not had these sessions members of the team would have known their parts of the systems intimately, but not had a good end-to-end knowledge of the design. As we had a broad audience outside of just the architects, over time as we reviewed all of the components of the design there was a good history built up about how we go to the final design and the interconnectedness of all of the components.

The way the project unfolded people were assigned areas of the design more to meet the schedule than any natural strengths around domains. These sessions allowed the collective knowledge of the team to be harnessed to review the design. So although someone who had a history using ETL tools may not have been designing that component, within the Architecture Review meetings they were able to impart their knowledge on that part of the architecture.

Having picked this up and applied it to my first project it is definitely a habit to instil into a project of any size. Look for that balance between a free-flowing discussion and a regimented meeting that runs to an agenda. Get that balance right and you create a great forum where the behaviour of the team as a whole can be fostered to build a high performing collaborative team.

A migration is much more than ETL

24 December 2010

From a business perspective a migration isn’t just a technical extract – transform – load process there is a lot more to it than that.

The business is usually undertaking a migration for a particular reason; consolidate technology platforms due to acquisitions or such or replacing an existing platform with a new better platform. In either case how you move the data can have a large bearing on the success or failure of the migration process. Sitting down with the business and talking through how a thing in the old world is going to look in the new world and what business rules should be applied to transform the old world thing into the new world thing is hard. Partly it is hard because you can’t see it is generally an abstract discussion which isn’t generally the most successful of conversations to have.

So if you can get someone who knows the current systems at a good level of detail and someone who knows the new systems at a good level of detail you may be able to evolve the discussion to come up with the transformation rules for the migration. One by-product of this discussion will be something I have seen referred to as overs and unders. An over is a field that exists in the old systems that doesn’t exist in the new systems; is it redundant, did we miss something when we specified the new system. Then unders are fields that exist in the new systems but we can’t find a value in the old systems, this is generally some new functionality that will need a default value.

Then I need to extract al the data out of the old systems transform it based on the rules you have given me above and then load it all into the new systems. Sounds simple, except you have a 24×7 business, oops. Now comes the trade-off between adding scope by having no or minimal disruption to the business and the associated cost in time and money to do this.

This is where the fun starts and from an architecture perspective here are some considerations:

  • Do the old systems need to be aware that the migration is occurring to cache changes and the like?
  • How do I extract the data so the old systems can remain online?
  • If the old systems are running once I have taken the extract how to I apply the delta changes to the new systems?
  • How do I cut over from the old system to the new system with no/minimal outage?

One other complexity that you may have to deal with is where your implementation strategy isn’t big bang but a sequence of iterative migrations. The complexity then is that you have to load all of the data into the new systems while they are still running phew!

So a migration is just ETL but the context within which you run the ETL changes the nature of it. This is what I refer to as the configuration of the ETL. So be aware that moving the data is the easy bit working with the business to understand the context that you will run it in is a key part of a successful migration.

Reference Architecture

23 December 2010

I have had the pleasure of leading a team of architects recently on a migration project. What seemed like a small problem but quickly became a real leadership challenge was dividing up an architecture across multiple architects. I have been on projects with multiple architects in the past but this was my first time leading a team.

In the past on projects where we have had multiple architects, we have used one of the following ways of dividing up the work:

  • Assign a domain to each architect and write an architecture document for each domain
  • Have multiple people working on the one architecture document

For this project we ended up going down a slightly different path and came up with something that I will definitely put in my back pocket and use again. To start the process off we came up with what we referred to as the reference architecture. This was a high level component based view of the architecture. The example of a reference architecture below was evolved by an architect taking a component or service and doing the next level of design for the functions identified.

Despite the reference architecture giving us a great high level component based view of the architecture one of the challenges it presented was providing guidance on what each components did and didn’t do. We spent a fair bit of time putting together a pack that went with the reference architecture diagram to outline for each component what was in scope for it and what was out of scope for it. Despite out best effort we still ended up having quite a few discussions about areas of the architecture and what component was responsible for this. Just something to be aware of if you follow this approach.

Once we had the reference architecture in place we were able to assign services and components to individual architects. To go with this we produced a light architecture document that was completed by the architect and then formed the basis of a review process. The document captured:

  • Problem Statement (from the pack described earlier)
  • Context (Highlight the component on the reference architecture)
  • Issues and Risks
  • Logical View
  • Integration View
  • Dependencies
  • Alternative designs considered

Once all of these light architecture documents were completed the content of all of them was cut and pasted into the central architecture document for the project. This was a relatively painless process with things like standardised modelling and content making it much easier.

This process evolved but looking back on it, it definitely provides a different approach to firstly developing the architecture and secondly managing the workload of a team of architects.

Migration is just ETL

22 December 2010

Over the last 9 months I have been working on a migration project which has been a new domain for me. Coming into the project I spent a fair bit of time trying to understand the domain to translate that into an architecture. Over time I came back to the simple answer that it is Extract – Transform – Load and I have ended up with that as the architecture. Where the ambiguity comes into it is knowing what systems you are extracting data from and the systems that you are loading the data into.

The roadblock that I hit was as an architect was that I wanted to identify the target and source systems that would form the solutions. To answer this question it became fairly clear that we would not know this answer until we had done all of the mapping from source to target. From a SDLC perspective this is in effect the detailed requirements/design which is way after the architecting bit.

Unfortunately we were in a situation where the target systems were being designed while we were designing the migration so were hitting dependencies again and again that we were trying to map to something where the design hadn’t been locked down. As a result of this, the mantra came about that as a project we were “special” and I really do think that we were but from a project management perspective there was an expectation that the normal waterfall SDLC would be followed and done in parallel to the design of the target systems. If I had my time again this is one issue that we should have driven a little harder to get a more logical alignment to the target systems design. The other solution could have been to use an iterative SDLC but this had its own issues being an organisation to great at doing iterative.

Architecting the migration capability, the main Non-Functional question that we needed to address in the design was how long it would take to perform the migration. There were several approaches to this but once again the problem of not really knowing the answer to the problem until we got closer to the finish came into play. We did try and look at other examples that we could use as a baseline which sounds OK in theory but in practice turned out to be a little more difficult. The technology is one part of a migration i.e. moving X GB of data from point A to point B and transform it on the way using a ETL tool. Then there are the operational components, things like reconcilliation and excpetion management, that add a variable amount of time to the migration. Talking to others who had done migrations the operation part is huge the first time you test the process and progressively become more efficient.

The answer around performance really boiled down to having the project in a position to do some sort of a trial run as early as possible and to performance benchmarks at this point. From a design perspective we needed to know what parts of the design could be tuned to deliver increased performance this included scaling the infrastructure and pre-migrating some data.

So a migration is just an ETL the complexities come about in that the details that drive the tuning of this design to allow identification of the source systems and also to meet the performance requirements. Move past the ETL problem there are more than enough tools to solve this problem focus on the bigger ticket items. But be prepared to call out dependencies on requirements that are needed to solve these other problems and look at ways to iteratively design the solution as these requirements come about, as waterfall just won’t do it.