Big Data in the Real World: How Hadoop Helped One Company Store 600,000 New Transactions Per Week

Are you faced with the need for truly scalable, high-performance storage and processing power? If so, what are your options? You could, of course, do all the trial and error for yourself. Or, you could take a look at what others have tried and what worked for them. The second option is less expensive, faster, and likely to yield better results. Here is the story of how one cloud provider — responsible for handling 600,000 transactions every week — chose to solve their big data dilemma.

The Challenge

ThinkstockPhotos 482249437 Big Data in the Real World: How Hadoop Helped One Company Store 600,000 New Transactions Per Week
Hadoop clusters allowed this cloud service provider to improve processing power, deepen analytical functions, and speed service, and their customers never suspected there was an issue.

Responsible for handling some 600,000 transactions per week, and growing at a rate of 7 terabytes per month, a cloud service provider was doubling their storage needs every 16 months. The data was largely unstructured, consisting of lots of unstructured logs and system diagnostic information. About 40 percent of this data had to be transmitted during an 18-hour period of time on the weekends. This presented issues with I/O bottlenecks, which could adversely affect their ability to meet the demands of their SLA agreements with customers. They needed to upgrade, but were unsure of what solution would meet their needs for massive scalability, high performance, superior availability, and high demand for bandwidth applications.

The Solution

ThinkstockPhotos 186119919 Big Data in the Real World: How Hadoop Helped One Company Store 600,000 New Transactions Per Week
Your data needs are growing, not shrinking. Plan for scalability well into the future.

This cloud service provider tested several solutions, but the only one capable of managing their intense workload was Hadoop. Using Hadoop clusters, they were able to manage huge amounts of customer data that is essential to their business. Hadoop also offered rich insight, which was an important part of their demands. Other options simply did not offer the scalability, performance, and deep analytical capabilities that Hadoop could provide.

The Results

According to the cloud provider, Hadoop has allowed them to improve their response time to customer issues. Scanning their some 24 billion records used to take about four weeks using their old system. With the new Hadoop clusters, this time has been cut to less than 11 hours. That is an improvement of 64 times. The new system provides full redundancy and automated path failover. According to the cloud provider, they are now easily able to meet their SLA requirements, assuring that their customers are satisfied.

Most organizations don’t have the intense need for availability and performance as this cloud service provider does, but that is the beauty of Hadoop clusters. So long as your data is large enough to warrant setting up and using Hadoop (usually petabytes of information), then Hadoop easily scales to the needs of the organization. If you have high demands for performance and availability, have a data-intensive environment, and use high-bandwidth applications, Hadoop can offer analytical power that surpasses the other options.  Syncsort DMX-h was designed from the ground up for Hadoop, and provides everything you need to collect, prepare, blend, transform and distribute data with Hadoop.

Why go through all the trial and error yourself?
 Big Data in the Real World: How Hadoop Helped One Company Store 600,000 New Transactions Per Week

This entry passed through the Full-Text RSS service – if this is your content and you’re reading it on someone else’s site, please read the FAQ at fivefilters.org/content-only/faq.php#publishers.

Syncsort blog