Few would disagree that big data is maturing. Looking back over the past 11 years of Hadoop, we’ve seen many participants in the ecosystem – those with proprietary, others with ‘mostly’ open-source and, of course, 100% open-source distributions. In 2015, the Open Data Platform Initiative (ODPi) was created by a group industry leaders working collaboratively to define and promote a set of standard, open source technologies to increase compatibility of among big data platforms.
One by one, the major software vendors are shifting focus to “above the stack” of the Big Data ecosystem. The latest version came with IBM’s announcement at DataWorks Summit 2017 that its Hadoop customer base would shift to Hortonworks. Just as a Microsoft made Hortonworks its Hadoop partner (and provided a premier Cloud platform for Hortonworks) and Pivotal…pivoted, this relationship brings a premier data science partnership while continuing to build out the broader platform customers need.
Turning to the insurance industry, Strategy Meets Action (SMA) discussed its recent research (“2017 Data and Analytics Trends: Is the Insurance Industry in for a Wild Ride?”) in a webinar outlining how 92% of insurers are investing in data and analytics initiatives. Furthermore, at the core of those initiatives is a Big Data platform based upon Hadoop (see figure 1).
As the insurance industry progresses, it’s interesting to ponder upon what will we see in the areas of machine learning, artificial intelligence and cognitive computing? Each relies on big data’s foundational capabilities. In a recent article by McKinsey “Ask the AI experts: What are the applications of AI?” members from Google, Baidu Research Silicon Valley AI Lab, Arraiy and Bosch Research and Technology Center highlighted how it is going be in enterprise systems for everything from self-driving cars, forecasting weather to the kinds of crops to grow. Each of these examples have implications for the insurance industry, especially impacting how we assess risks and investigate claims.
Adam Coates, from Baidu Research Silicon Valley AI Lab summarizes it best, “AI is substantially driven right now by three critical pieces. One is data, another is computing power, and the third is talent. As much as the field is hot, there still are not enough engineers who know how to apply these machine-learning algorithms with a really high level of skill. It’s getting better, but it’s still a scarce talent.
“If you’re interested in solving AI problems for your business, then I think it’s important to think hard about whether you want to try to construct a machine-learning team within your company to solve a specific problem or whether you can now use enterprise platforms.”
While technologies continue to mature, an absolute constant is the continued growth of data along with its variety and velocity. We see investments and innovation in open source capabilities such as Apache Spark (machine learning, natural language processing) and Apache Zeppelin (data science platform) which are quickly becoming a foundational element, just like Hadoop.
Where are we headed? Many would refer to Hadoop as “plumbing”… certainly to keep my house in order, we need plumbing and we need it to interconnect and inter-operate! The insurance industry depends on “data” to operate, grow and protect customers. To support “preventive analytics” the data needs to move and be analyzed “freely”. What has allowed Hortonworks to be front and center of thefoundational layer is the company’s core value to be 100% open-source, aligning to the source code “trunk” with no proprietary extensions and absolutely no vendor lock-in. Sometimes it is “ok” to be the ‘plumber’!
About the Author: As GM for Insurance,Cindy manages the overall business & technology vision, go-to market strategy and executive engagement worldwide for Hortonworks market leading Hadoop offerings. This includes the many opportunities of working with customers & partners leveraging analytics for current day business growth and exploring usage of new data driving innovation in the evolving world of insurance.