The most difficult part of upgrading from legacy software is data migration. It’s time-consuming, and the effort often uncovers previously unnoticed problems with the data. Careful planning and a realistic schedule are necessary for a successful migration.
Treat Data Migration as a Project
Migrating the data can’t be an afterthought to updating the software. It’s often the majority of the work by itself. It needs its own plan and schedule. Management should have a realistic sense at the start of how much it will cost and how long it will take.
Estimating any project carries the temptation of assuming the best case. Unexpected problems always turn up. The plan needs to allow for a realistic level of issues that will need addressing. A successful migration may take more than one attempt.
Assess the Data
The assessment for the migration project needs to take several factors into account.
- How much information is there to migrate?
- How clean is it?
- What tools can do the job?
- Is all of it still necessary?
- Are there confidentiality issues?
The biggest issue is likely to be defects in the data. This is especially true if the old data is in an unconstrained format, such as text records or a spreadsheet. It’s likely to be full of invalid data and blank fields.
The data may contain information which puts confidentiality at risk if not carefully handled. The migration process needs to make sure it will be adequately protected in the new system, even if it wasn’t before.
On the positive side, some information in the existing data may no longer be necessary. Perhaps it never really was. It could contain obsolete codes or irrelevant information. Some fields may correspond to data which is already in the new system, and migrating it again would create conflicts without contributing any value.
Determine the Requirements
The success of migration depends on whether people can use the new system and the data for their intended purposes without loss. Achieving this requires knowing those purposes. The team needs to talk with users and managers to understand how they use the data. They should look at sample reports with an eye for what goes into them.
There may be other applications that use the same data. If so, it’s necessary to figure out how they can work with the migrated data. Alternatively, they may be unnecessary because the new system does what previously took several independent applications.
Define the Process
Whenever possible, the migration should use existing, reliable tools. Writing custom code just for the occasion is extra work, and it’s likely to introduce errors. Some new code may be necessary, but it should be kept to a minimum.
The process should be automated to the greatest extent possible. Ideally, one command launches the entire process. At a minimum, there will be a test run before the live migration, and they should both do the same thing apart from moving the data to a production destination. If there are serious problems on the first try, the migration may have to run more than once. Automating the process will make sure that the same steps happen every time.
Verify the Results
After the migration, it’s essential to verify that the new system has information without errors or unintended losses. This isn’t just a matter of running the new system and getting plausible results; they need to be the correct results. The first attempt may not be error-free, and it’s better to discover that immediately rather than after people notice they’re getting incorrect information.
Data migration isn’t a process to take lightly, but with a well-managed plan, it can go smoothly.