What are the Hadoop distribution vendors really selling?

IT teams can download Hadoop from the Apache website and deploy it on a hardware cluster themselves, without any vendor involvement. But Hadoop vendors are aware that the self-starter approach isn’t for everyone, so they provide prebuilt Hadoop distributions that can be downloaded from their websites — typically in both a free community edition and an enterprise edition that adds more features and requires the purchase of a license. But if these vendors are providing users with a product, what are they really selling? In other words, what do you actually get when you engage and pay a Hadoop software vendor?

Vendors offering commercial versions of open source technologies, such as those providing big data management systems based on Hadoop, follow an alternative system and services model in which customers effectively subscribe to the enterprise edition of the product. Benefits of subscribing to an enterprise edition include:

Access to enterprise features. The subscription relationship enables customers to access versions of Hadoop that have features and optimizations that haven’t been openly released to the open source community.

Release from restrictions. In some situations, the freely downloadable Hadoop distributions have been built with restrictions, such as a limit to the number of nodes on which the system can be run or the amount of data that can be managed. Buying an enterprise subscription lifts these restrictions.

Responsive technical support. Enterprise subscriptions provide availability of resources for support with 24/7 telephone access and response times that can be guaranteed under service-level agreements, depending on the level of support purchased.

Advanced training. While all website visitors may have access to some training materials and videos, enterprise subscribers typically are entitled to more advanced and extensive training sessions.

Access to deployment experts. Hadoop vendors have professional services teams that are experienced in big data management deployments and can help jump-start a customer’s implementation.

Key considerations for comparing Hadoop distribution vendors

The enterprise editions of vendor Hadoop distributions all provide the core components of the Hadoop ecosystem stack, which include the Hadoop Distributed File System (HDFS), the MapReduce programming and execution environment for batch processing, and the YARN job scheduler and cluster resource manager. They also commonly incorporate various other open source technologies, such as the Spark data processing engine and HBase database. But different vendors may support different releases of all those technologies, and newer or more specialized tools may not be universally supported. If your organization is looking to use a particular technology as part of a Hadoop deployment, you should ensure that the distributions you’re considering support it and, if so, which release they’re currently on.

Beyond these typical components, you should also compare and contrast how each vendor provides the following:

Access to enterprise-class features. Some Hadoop vendors offer additional tools that aren’t part of the open source distribution for system configuration, system performance, ongoing monitoring and administration. While these may add value to the enterprise distribution, recognize that integration with proprietary components may lock the customer into that vendor’s product.

Infrastructure deployment alternatives. Your organization may choose to adopt different underlying infrastructure options, such as running on-premises, in the cloud or in virtualized environments. Consider how the Hadoop distribution alternatives are adaptable to these infrastructure choices.

Interoperability with other data management systems. In most cases, an organization will have existing data warehousing, business intelligence and analytics systems in place. Hadoop typically doesn’t fully replace these systems, but rather augments and complements them. So it’s critical that the adopted Hadoop environment enable access and data exchange with existing data management platforms such as DB2, Oracle, SQL Server, Teradata and others.

Integration with end-user tools. End users will want to continue using their favorite tools for business intelligence, reporting, visualization and analytics. Assess how well the Hadoop big data management vendor’s distribution supports integration with the tools used in your organization.

Security and data protection. The Apache Hadoop ecosystem is still maturing, which means that not all of its components may meet enterprise expectations for data security and protection. Many Hadoop vendors provide security features as add-ons.

Support options. Consider what your support requirements are in terms of availability and response times. Vendors offer different plans for support availability as well as response windows.

Indemnification from litigation from use of open source technology. This increasingly important concept ensures that vendors of open source technologies protect their users from potential liabilities related to the use of the product.

Optimized performance. Enterprise distributions may be augmented with performance optimizations that enhance scalability and extensibility.

One additional consideration when comparing Hadoop distribution vendor offerings relates to the approach that vendors are taking toward compatibility within the open source community and interoperability between product offerings from different companies. Ideally, this means ensuring that Hadoop distributions will remain compatible with the open source versions of Hadoop and other Apache technologies, even as vendors make code changes and develop proprietary add-ons. That could help prevent vendor and version lock-in, in which an organization becomes bound to a particular distribution of Hadoop.

However, there’s a lack of unanimity among Hadoop vendors on how best to enable interoperability. Several have formed a group called the Open Data Platform Initiative, set up within the Linux Foundation open source consortium, to develop a common set of interoperability standards for Hadoop. But other vendors have declined to join the group, saying that compatibility and interoperability issues are already being sufficiently addressed within Apache. Assuring alignment with the open source distribution as a standard is certainly desirable in that it allows Hadoop users to maintain some flexibility in their choice of vendors.

Prior to engaging vendors, it’s also important to assess what types of applications your company plans to develop and run using the Hadoop ecosystem, and the required capabilities. Then determine which of these are provided by the community open source versions of Hadoop and other technologies and which require additional functions only provided by a specific Hadoop software vendor.

Weighing all of these factors will help prepare your organization to move forward and evaluate the available options. In our next article, we will assess the similarities and differences between the leading Hadoop distributions.