The promise of going down the “big data road” for the typical enterprise is that the boundary between “our data” and the world’s data gets a little blurred, and the union of the two offers new insights.

It’s important to know what you’re looking for, though, or you can destroy your credibility in a flash.

Take transit systems. “Are we doing a good job?” is a reasonable senior management preoccupation.

Vehicle trip times, and inspectors checking on whether they’re on time at midpoints, gives data today. It can help with bunching up problems (followed by long gaps), up to a point.

So more transit systems are turning to integrating Twitter feeds and other social media into their analysis. If people are tweeting about lousy service, it’s an indicator.

Here’s the gotcha, though: they don’t tweet about good service.

They also stop tweeting about poor service when they’ve given up the hope that it’ll make a difference. So silence could mean satisfaction — or sullenness.

The data feed won’t tell you which.

Likewise, with the internal data, everything could be making the numbers, but if the routes don’t serve the public well, or if the way the numbers are measured lets too much customer aggravation through, you can miss the picture.

The process of using the combination effectively is iterative — moreover, it should lead to revisions to what you set as standards to measure by.

Otherwise the big data effort simply leads to a false sense of security.

On the product and service side, too, there are pitfalls.

Research at Simon Fraser University done by Dr. Jian Pei and his students into mining big data has shown how, for instance, a travel service (airline or agency) could provide a much more tailored response to prioritize flight choices.

People care about cost (to a point), total trip time (to a point), frequent flyer bonuses (to a point) — the “best” flight has different characteristics depending on circumstances such as when you’re going and when you’ll arrive. It’s not as simple as “sort by price” or “sort by number of connections”.

“Learning” patrons’ patterns would allow the service to provide the best tailored option — and thus make this place the preferred place to purchase the flights.

Here, too, though, there’d be a need to move beyond the mining of patterns and get feedback mechanisms installed to “tune” the service.

“Good” is situational. A Western Canadian travelling East sees the trip quite differently from an Eastern Canadian travelling West (a function of time zones and connection points if they’re required — it’s not uncommon to have to fly over your destination and double back to save time in the West). Again, a learning and adjustment process would be required to maximize the value and make the service more than a novelty.

Big data’s promise, in other words, is also a source of its potential pitfalls, leading to a rejection of the whole idea later.

Much as happened, come to think of it, with data warehouses and business intelligence/performance management toolsets.

You’re going to have to consult with the business to make this work.