Big Data challenges
Too Much Big Data Variety
The variety of big data may not be the spice of life after all. In fact, 71% of the data scientists surveyed said new types of data make developing analytics apps more difficult.
Biggest Big Data Challenge
Big data analytics is more complex than most appreciate: 40% said they struggle with new types of data while 36% said it takes too long to get the answer they are seeking.
Types of Data Being Used in Next Year
Data comes in many structured and unstructured forms. Time series and business transaction data are tied, at 66%, followed by geospatial data, at 55%.
Plans for Big Data Analytics
The use of big data analytics applications is already pervasive: 59% have already deployed big data analytics and 31% plan to do so in the next two years.
Data Movement Issues
Big data tends to stay where it gets generated. One of the bigger issues for 36% of data scientists is that the amount of data that needs to be analyzed is too big for their organization to move into an application.
Limitations of Hadoop/Spark
Hadoop is only one element of the big data equation: 39% said Hadoop is too difficult to program while 37% said it’s too slow for ad hoc queries and 30% said it’s too slow for real-time analytics.
Hadoop/Spark Abandonment
Failure rates involving Hadoop are still pretty high: 37% of the data scientists who have tried Hadoop/Spark have abandoned it.
The Role of the Relational DatabaseMany organizations are outgrowing traditional relational databases. Just under half, or 49%, said they are having trouble fitting their data in a relational database.
Big Data Stress Factors
Much is asked when much is given. The growth of big data has made the job of being a data scientist more stressful for 39% of data scientists.
The Big Data Question
A knowledge of science does not always translate into business-savvy insight. Almost a quarter, 24%, said they don’t know what questions to ask when they gain access to all that data.