Big data analytics opportunities appear to be in the eye of the beholder. While the hype surrounding big data has been generally high, the ability of most organizations to implement an analytics strategy that incorporates most of their data has been fairly limited.
Data fragmentation poses a major challenge. In a new survey of 402 business and IT professionals conducted by the industry IT association CompTIA, 45 percent said that a high degree of their data is fragmented, with another 42 percent conceding their data fragmentation issues are at least moderate.
Overall, the CompTIA study finds that just over half the organizations surveyed have at least one big data project in play, with another 36 percent reporting they are in the planning stages.
Given that adoption rate, big data analytics represents a major challenge and opportunity for solution providers. Although most of the big data focus to date has been on platforms such as Hadoop, in practice, big data analytics opportunities span a panoply of products and services that solution providers need to master.
Grasping Big Data Analytics Opportunities and Challenges
It’s important to understand the nuances of big data analytics opportunities and challenges.
“Big data encompasses a range of products across the spectrum,” said Seth Robinson, senior director for technology analysis at CompTIA. “There are a whole range of issues involving, for example, storage and security.”
The CompTIA survey makes it clear that businesses are already deriving value from their big data investments. A full 72 percent of the organizations that have started a big data project report that their projects have exceeded expectations.
A similar survey conducted by Teradata in partnership with McKinsey Consulting showed that roughly a quarter of decision-makers report that they are starting to see significant returns on their big data investments in the form of increased revenue and reduced costs.
However, much work remains to be done. Approximately three-quarters of organizations CompTIA surveyed said their businesses would be stronger if they could harness all their data, while 73 percent said they need better real-time analysis.
That latter requirement is driving many organizations to wrap a variety of complementary technologies around Hadoop, ranging from the Apache Spark in-memory clustering software to databases from vendors such as SAP and MarkLogic. In fact, one of the things driving so much interest in Apache Spark as an alternative to MapReduce as a programming tool is that it can be applied directly against multiple data sources
MarkLogic CEO Gary Bloom said what most companies still don’t fully appreciate about big data is what it takes to operationalize that data inside their businesses. Most organizations, Bloom said, can make use of Hadoop today to analyze what’s already occurred in their business in a batch mode process. However, it requires a database that can process transactions and analytics at the same time to operationalize data in a way that allows business to act on it in real time, for example, to curtail fraud, Bloom said.
“The MarkLogic database is designed to ingest and index data in real time,” Bloom said. “Hadoop is really just about consolidating data.”