Big data is being talked about today, but few IT organizations are grappling with it, yet.
Part of this is a chicken-and-egg problem: they don’t have a specific request to respond to, yet, that needs a big data infrastructure to drive it, and they don’t have the resources within IT to build that infrastructure out to produce some big data experiments they can use to “what if” with the business.
The other part is what the IT folk hear about big data: it requires massively parallel capabilities, blows the doors off their relational database managers, requires them to think differently about data.
So a lot of the championing of big data as a source of business insight is coming from the vendor community, and IT’s feeling a little pushed around — again.
Set all that to one side, just for a moment. What’s really happening here?
What big data means to the IT organization is the divorce between the systems that some data flows through and the overall information resources of the enterprise.
It’s a key signal of a core transition (along with social business and a few other trends) from IT as process-centric to IT as information-centric.
Or, to put it another way, a shift from IT as emphasizing the “T” of technology over the “i” of information, to emphasizing the “I” of information over the “t” of technology.
Both still matter, but their weights change.
In the “I” world, IT thinks of the information assets of the enterprise as a unit. Systems dip in and out of the pool. But we stop talking about the individual parts.
There’s also a lot of other information in that pool that isn’t associated in any way with any of the existing application systems being run. Some comes from RFID sensors, some from an Internet of Things, some from the social world, some from external sources (purchased inputs), some from plant-floor devices … it’s a long list.
Also embedded in this set of assets are intermediate data objects, or partially-processed entities. Maybe you’re able to collect radar data from around the country. You might assemble these traces into records of flights moving through the sky. For some purposes I may need the raw data still, for many others I may need the consolidations. These intermediate data objects are processed solely to facilitate analysis. (Another example is in billing details: I might pre-assemble days of activity, or types of activity, to avoid chewing through all the details every time.)
Yes, enough data can take us out of our existing infrastructure and the database technologies we’ve established skills in. Smart thinking about masses and consolidations — and getting an information studies-trained information architect and information manager working with database people and EA-trained architects — can get you going.
Going enough, with what you have, to do a few demos, potentially add a little value, get started with the business.
CEOs want value from information. Big data is how IT will deliver.