Harvesting

Harvesting is an automatic process that makes sure your data is there where you need it when you need it.

At typical bank might have 1000 external files per week,  say in the form of email with an attachment.

Our Harvester uses php and sql to strip out the attachment and shake out the data. Then it’s ready for cleansing.

Growing

Once all the data is in a common structure the data really starts to work. In the case of a medical facility this structure can save lives, because it might be checking the potential drug conflict of prescription medicine,  before administering it.  This added value is what we call growth.

Cleansing

Someone’s sent you a big file, but out of the 30 columns on the spreadsheet attachment there are only 18 you need. Cleansing takes the spurious stuff, checks things like zeros haven’t become the letter O, date formats haven’t become American etc. Cleansing is what makes the data usable. After all the famous maxim rubbish in, rubbish out, and who wants rubbish?

Integration

Integrating is getting data into a common useable structure, despite very different origins. Imagine you have to take ice cream sales from your tills you might want to integrate that with hours of sunshine or weather data from the met office. Integrating is bringing these two together.

Transport

For all the processes that have to happen, data needs to be collected and moved to where it is used. Obviously it is important that this is done in a way that preserves the integrity of the data, using the most appropriate vehicle and protocols. It also needs to be done regularly, robustly and reliably, 24/7/365 without missing a beat; this we refer to as transport, a crucial yet often overlooked component.