What’s data migration for homelessness charities?

Data migration is the process of transforming data from one format to another without losing any information in the process. The transformation must be reliable and deterministic, and should handle edge cases where incoming data doesn’t meet the established structure.

An analogy for this could be two people with to-do lists. The first person likes to label their tasks by urgency, and the second by deadline. If the first person wanted to use the second’s format, they’d have to migrate their tasks.

Why is data migration important for homelessness services?

Similarly, for case management software, it’s extremely rare for two platforms to share a format. The structure of data is optimised for the functionality of the service, so it always needs to be migrated from the source format to the destination format. This differs from our to-do list example as these large-scale migrations involve millions of data points and typically include historical data too. This is why businesses exist whose sole purpose is to migrate data to or from well-known platforms.

Data must be migrated quickly and wholly so there is always a single source of truth. This is absolutely critical, especially for highly varied charity data, as managing two platforms at the same time will lead to major inconsistencies, mistakes, and much lower confidence in both platforms. This is especially true for frontline workers who do not have the time to check multiple places for the information they need - a single source of truth is essential. To ensure the integrity of this single source of truth all current and historical data must be migrated. This then allows Harker to create detailed reports about data that was inputted on different, older platforms.

Why is data migration so complicated?

It goes without saying that these migrations can get very complicated. The structure of the source data must be analysed and understood before any transformation can begin, and a mapping must be decided between all source types and all destination types.

For example, a frequent case is that one platform might store their dates and times as numbers, called Unix Timestamps, but the other platform stores them as strings. This difference has to be identified and some code must be written that takes every possible Unix Timestamp as input and outputs the corresponding string. This analysis-to-implementation flow has to be repeated for all types, and the complexity increases further as many of these platforms use custom formats and languages for storing data. These unique systems must be learned from the ground up.

The ultimate goal, though, isn’t just to transform a large set of data - but is instead to produce a reliable pipeline that can transform any data of that format. Then it can be reused and extended the next time that format arises.

How does Harker handle data migration for homelessness services?

At Harker, we handle data migrations in house and, during the process of migrating data for our first few customers, we’ve now built a dedicated tool for inputting and transforming data into our unique format. Designing this was an interesting challenge, and was lead by the question “How much of this can we automate?”

We were able to isolate the deterministic and automation-friendly parts of this process and put them in our core migration tool, which then changes its behaviour based on a configuration file provided by the developer. It’s inside version-controlled configuration that the important platform-specific decisions are made.

Ready to see Harker in action?

If you'd like to benefit from a single source of truth for data in your services, we'd love to show you how Harker can transform your delivery and reporting by booking a demo today.