Dogbert to Dilbert: Information is gushing toward your brain like a firehose aimed at a teacup.

Every company, organization and individual is continuously gathering and creating all kinds of data. Most of this data collection is happening in separated silos, with very limited connections between the different data collections. This is true, even for data sources within the same organization. A shame really, because the value of the data rises in proportion with the ways it can be interlinked and connected – a network effect similar to the one that determines the value of social networks, telecommunication systems or even financial markets themselves.

Lack of common definitions, data schemes and meta-data, currently make these connections quite hard to make. This is the very problem that the semantic web promises to solve. However a lot of this data is already finding its way to the internet in one form or another, and those that make the effort to identify and collect the right bits can gain insights that give them competitive advantage in their markets. At – and around – the Money:Tech conference I was so fortunate to attend last week, several examples were given:

• Stock traders are monitoring Amazon’s lists of top sales in electronics and using them as indicators of the performance of the chip maker’s performance in the market. This is done by breaking down the supply chain for each of the top selling devices and thereby establishing who’s benefiting from – say – the stellar sales of iPods.
• Insurance companies, utilities (energy sector) and stock traders (again) are constantly analyzing weather data to predict things like insurance claims, electricity demand and retail sales patterns.
• By monitoring in “real time” publicly available sales data in the real estate market, companies like Altos have been able to accurately predict housing price indexes up to a month before the official government numbers are published. Similar insights might be possible to predict other major economical indicators, such as matching the number of job listings on online sites to changes in the unemployment rate, or online retail prices to predict inflation.

But it’s not only data gathered from the internet that’s interesting. Far more data is dug deep in companies’ databases and therefore (usually for a good reason) not publicly available.

Take the area of telecommunications. Mining their data in new ways could help telcos and ISPs to get into areas currently dominated by other players. Take the sizzling hot social networking area as an example: Call data records, cell phone contact lists and email sending patterns are the quintessential social network information. Whom do I call and how frequently? Who calls me? Who is the real hub of information flow within my company? These are all information that can be read pretty directly from the data that a telco is already gathering. Every customer’s social network can be accurately drawn, including the strength, the “direction” and – to some degree – the nature of the relationship. Obviously this would have to be transparent to the customer and used only on a opt-in basis, but in terms of data accuracy it is a “Facebook killer” from day one.

This is clearly something Google would do if they were a telco (and I’m pretty sure that Android plays to this to some extent).

Another interesting aspect of this whole data collection business is the value of data that one company gathers to other organizations. Again, the telco is my example. A telco has – by default – information about the rough whereabouts of every mobile phone customer. Plot these on a time axis and remove all person identifying information and you have a perfect view of the flow of traffic and people through a city – including seasonal and periodical changes in the traffic patterns. This is of limited value to the telco, but imagine the value to city planners, businesses deciding where to build a service station or the opening hours of their high-street store. Add a little target group analysis on top of this and the results are almost scary.

We are probably at the very beginning of realizing the potential in the business of data exchange and data markets, but I’ll go as far as predicting that in the coming years we’ll see the rise of a new industry focusing solely on enabling and commercializing this kind of trade.