Big Data and Hadoop

As a rule of thumb, I usually estimate that when it comes to take-up of new technologies, the UK is 12 to 18 months behind the US and that the rest of Europe is around 6 months behind the UK. Of course there are exceptions but these tend to be specific – for example; the take-up of complex event processing in capital markets wasn’t much different in London from New York – but in general I think the 12-18 months rule works well.

The reason I mention this is with respect to “Big Data” and Hadoop (not the same thing). While there are always early adopters, for some time we have very much been in the position where companies have been trying out big data technologies rather than deploying them in anger. However, I get the strong impression, at least in the States, that that position has now changed: that enterprises are now starting to get into full production with big data and, indeed, have been doing so for some months. We should therefore expect the same trend in the UK and by the end of this year I fully expect us to be in the same position.

Of course this will be variable, especially where there are knock-on effects. For example, smart metering is a classic big data opportunity but smart metering adoption is more advanced in the US than here (another 18 months?) but hopefully the utility companies will go straight for the big data deployments as they start installing the relevant meters. Much of the noise about big data is with respect to social media and this is where we are likely to see take-up first but I actually think that analysis of machine generated data (such as smart meters) will be more important in the long run. In any case look for a big data deployment coming to your neighbourhood sometime soon.