3 Insurance Industry Best Practices for Succeeding with Big Data
The concept of paying someone else to accept all or partial risk for a property or monetary loss is not new. It is believed that the first such arrangement was established with merchant sea vessel owners and their lending companies in ancient Babylon, some 4,200 years ago. Vessel owners paid their lenders an extra sum for the privilege of canceling their outstanding loan balance if the ship were to be lost at sea. There was little data available to either the vessel owner or the lender, so it was largely a matter of guesswork to establish premiums for such arrangements. Either the sea vessel owner or the lender likely took a big hit.
Today, a lack of data is definitely not the problem for insurance companies. On the contrary; most insurers have difficulty managing their vast reservoirs of data, and more sources and data sets become available continuously. If you’re involved in a big data initiative, or plan to be in the near future, follow these best practices to set yourself up for success!.
- Start Small for an Early Win and Faster Time to Value
Big data and Hadoop are not an all-or-nothing proposition. Start with a small project, and as you reap success, you can add on to the initiative. Soon you will have a data-driven organization that is structured wisely and well-founded in smart principles.
Most of the mistakes that are made with big data can be worked out easily and relatively quickly if the initiative starts small and manageable and is grown as the business learns, adapts, and masters the big data tools.
As Hadoop adoption matures, advanced, predictive analytics is emerging as the top use case for Hadoop; however, it can be a complicated and lengthy deployment to start with, as it requires
additional infrastructure and wider enterprise adoption to derive value. It also requires a foundation of readily-accessible data blended from various sources.
The insurance companies that have the most success with Hadoop — and fastest time to value – begin with manageable, operational use cases focused on getting data from legacy platforms, like mainframes and the enterprise data warehouse, into Hadoop. Starting with operational use cases also frees up database capacity and budget and creates the foundation required to build a data hub, blending valuable mainframe, telemetry and security data, to power next-generation big data analytics.
2. Ensure You Have the Right Resources
Whether it’s to save money or to keep your proprietary secrets, it’s tempting to try to take on big data on your own. Don’t. Use trustworthy partners with experience, knowledge, resources, and skills sets to get your big data and Hadoop initiative up and running.
Before you can reap any success from big data and Hadoop, you have to go through the process of selecting the right tools. You need to consider all of the potential sources of data, including internal systems as well as outside sources like social media firehose data. Hadoop is capable of handling a virtually unlimited amount of data (depending on the number of clusters your infrastructure can support), including unstructured and semi-structured data — so don’t limit your information sources. Once you identify the tools to use, you need to make sure you have the right skillsets on your team to use them.
You may think that you can keep costs down, keep your Hadoop initiatives private, and sidestep other issues by keeping your data initiatives in-house — training your own teams on the newest languages and tools, or hiring those skillsets. Unfortunately, this can be a losing proposition. Those skills are in short supply so hiring them into your company can be extremely expensive – if you can find them at all. Alternatively, training your own staff usually comes with lost productivity and a steep learning curve – and, no sooner do they learn one technology, but there’s a newer, hotter one to conquer.
It is almost always more cost effective, efficient, faster, easier, and more fruitful to partner with vendors with experience in big data and Hadoop. There is also third-party software specifically designed to simplify the Big Data pipeline and insulate your organization from the underlying complexities of the technology, like Syncsort DMX-h.
3. Always Keep Security & Governance Top of Mind
Above all other best practices, security and governance reign supreme. Fortunately, Hadoop has made enormous strides in the past couple of years in terms of security and governance, and if it is set up and used properly, Hadoop can be made as secure as any other aspect of your IT infrastructure. Remember, proper big data security protocol isn’t just to protect your company from liability — it is crucial to remain compliant with government and industry regulations.
Syncsort is a leader in the industry of big data, offering a variety of solutions to get your Hadoop operations underway successfully – and securely, with support for Kerberos, Apache Ranger, Sentry, and more. Take a look at the Syncsort Big Data solutions that can help you get your big data initiative underway.