In the corporate sector, Enterprise Resource Planning (ERP) and Business Intelligence (BI) were buzz words in the late 1990’s to early 2000. Data Warehousing was a concept that gained ground in that time. By the time social media gained ground after 2005, a massive amount of data was being stored in an unstructured way. With Facebook going public tomorrow, big data is clearly a serious deal in the consumer space. The consumer space is of significance for the corporations wanting to market in social media networks.
After attending a Gartner local briefing earlier this week, the limitations of how corporate data is structured is clear. This is not to say that Big Data will ever replace BI, ERP, or data warehouses. It is to say that Big Data will need to work alongside existing systems.
Gartner weighed in on Big Data, recognizing that unstructured data decentralizes the business analyst function in corporations. The Business Intelligence function is pushed outside of the IT unit to the business unit.
Corporations should anticipate that Big Data projects start as a proof of concept. The cost of Big Data software (Hadoop) is very low. The rising acceptance for open source software contributes to the low cost for Big Data implementation. Large volume, complex, loosely structured data are characteristics that would fit a Big Data project.
The growth in Big Data is illustrated with these recent developments:
- Six government agencies in the US will be spending over $200 million to help find ways to store, analyze, and organize large data
- The significantly lower costs of Big Data prototypes as compared to BI/ERP/Data Warehouses is creating jobs in this specialty
- A Toronto hospital is using big data for the clinical detection of infections in premature infants
Big Data is getting a dose of legitimacy, through the participation of large IT companies. Microsoft acquired Fast Search and Transfer in 2008, making the tool available in its SharePoint. HP acquired Anatomy (although paying a significant premium), a company based in the U.K. Oracle acquired Endeca. Endeca provides tools for managing unstructured data. The company also supports web commerce and BI. EMC has a tool, Lucene Solr, to support text analytics.
Despite acquiring Cognos many years ago, IBM is building its own BI tool for Big Data. The company chose to organize its tool in the format of a tabbed spreadsheet for linked analysis.
Further discussion: where is your company at with Big Data?