Big Data challenges
The variety of big data may not be the spice of life after all. In fact, 71% of the data scientists surveyed said new types of data make developing analytics apps more difficult.
Big data analytics is more complex than most appreciate: 40% said they struggle with new types of data while 36% said it takes too long to get the answer they are seeking.
Data comes in many structured and unstructured forms. Time series and business transaction data are tied, at 66%, followed by geospatial data, at 55%.
The use of big data analytics applications is already pervasive: 59% have already deployed big data analytics and 31% plan to do so in the next two years.
Big data tends to stay where it gets generated. One of the bigger issues for 36% of data scientists is that the amount of data that needs to be analyzed is too big for their organization to move into an application.
Hadoop is only one element of the big data equation: 39% said Hadoop is too difficult to program while 37% said it’s too slow for ad hoc queries and 30% said it’s too slow for real-time analytics.
Failure rates involving Hadoop are still pretty high: 37% of the data scientists who have tried Hadoop/Spark have abandoned it.
Many organizations are outgrowing traditional relational databases. Just under half, or 49%, said they are having trouble fitting their data in a relational database.
Much is asked when much is given. The growth of big data has made the job of being a data scientist more stressful for 39% of data scientists.
A knowledge of science does not always translate into business-savvy insight. Almost a quarter, 24%, said they don’t know what questions to ask when they gain access to all that data.