• Home
  • About Us
  • Contact Us
  • Privacy Policy
  • Special Offers
Business Intelligence Info
  • Business Intelligence
    • BI News and Info
    • Big Data
    • Mobile and Cloud
    • Self-Service BI
  • CRM
    • CRM News and Info
    • InfusionSoft
    • Microsoft Dynamics CRM
    • NetSuite
    • OnContact
    • Salesforce
    • Workbooks
  • Data Mining
    • Pentaho
    • Sisense
    • Tableau
    • TIBCO Spotfire
  • Data Warehousing
    • DWH News and Info
    • IBM DB2
    • Microsoft SQL Server
    • Oracle
    • Teradata
  • Predictive Analytics
    • FICO
    • KNIME
    • Mathematica
    • Matlab
    • Minitab
    • RapidMiner
    • Revolution
    • SAP
    • SAS/SPSS
  • Humor

DataTorrent data ingestion tool aims to speed Hadoop feeds

August 15, 2015   BI News and Info

Big data analytics platform vendor DataTorrent has released its first standalone application, a fault-tolerant data ingestion and extraction tool for users of the Hadoop Distributed File System (HDFS). The software, called dtIngest, can move data between HDFS, Kafka, the Java Message Service and other data formats.

The new tool includes a point-and-click user interface and runs on Hadoop 2 clusters as a native YARN application. The software is designed to support both large and small files, allowing smaller ones to be aggregated into larger files to reduce the overall number of ingestion jobs that Hadoop ssytems have to process. Besides running in standalone mode, dtIngest can also work with DataTorrent’s flagship RTS 3 in-memory analytics engine. As such, it demonstrates the Santa Clara, Calif., company’s goal to go beyond streaming data and also cover batch processing. And coupled with other recent moves, the release of dtIngest shows DataTorrent’s intention to become more perceptibly a card-carrying member of the Apache Hadoop ecosystem.

For example, DataTorrent said it would allow unlimited free use of the ingestion software by organizations. The company also said that an open source implementation of RTS called Project Apex is now available on Github, following earlier word that the core engine would be released as an open source technology under the Apache 2.0 license.

Data ingestion as foot in door

Data ingestion could be an entry point into user organizations for DataTorrent, which was formed by expatriates from Yahoo in 2012 as the Hadoop software that originated at the Internet services company took early flight. Now, several years into the Hadoop experience, the challenge of loading data into HDFS remains one of several factors cited when people ponder slow Hadoop uptake in mainstream organizations.

“Getting data into Hadoop can be hard, and getting it out can be just as difficult,” said John Fanelli, vice president of marketing at DataTorrent. Fanelli said dtIngest enables point-and-click configuration of Hadoop ingestion and extraction jobs, easing the development burden.

According to a 2014 report by Jason Stamper, an analyst at 451 Research LLC, Hadoop data analysis becomes much more troublesome without the right ingestion and data management tools. He noted that DataTorrent’s founders have a strong real-time engineering pedigree.

Spark, Storm lurk in waiting

Stamper’s report notes as well that DataTorrent RTS faces serious competition from the Apache Storm and Apache Spark processing engines, both of which have gained attention since the advent of Hadoop 2.0 in late 2013.

DataTorrent’s claimed customers include PubMatic, which uses RTS as part of a real-time ad analytics platform; and Silver Spring Networks, which has deployed it to help power a sensor networking application. The competitive environment marked by Spark and Storm can be seen as a likely driver of the company’s recent moves to open up more access to its offerings.

Jack Vaughan is SearchDataManagement’s news and site editor. Email him at jvaughan@techtarget.com, and follow us on Twitter: @sDataManagement.

Next Steps

Take a look at Hadoop performance bottlenecks

Learn more about big data in motion

Find out how online ad companies use Spark for data streaming

This entry passed through the Full-Text RSS service – if this is your content and you’re reading it on someone else’s site, please read the FAQ at fivefilters.org/content-only/faq.php#publishers.


SearchBusinessAnalytics: BI, CPM and analytics news, tips and resources

Aims, data, DataTorrent, feeds, Hadoop, ingestion, Speed, tool
  • Recent Posts

    • Someone’s having surgery
    • C’mon hooman
    • Build and Release Pipelines for Azure Resources (Logic Apps and Azure Functions)
    • Database version control: Getting started with Flyway
    • Support CRM with New Dynamics 365 Field Service Mobile App
  • Categories

  • Archives

    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • March 2020
    • February 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    • December 2016
    • November 2016
    • October 2016
    • September 2016
    • August 2016
    • July 2016
    • June 2016
    • May 2016
    • April 2016
    • March 2016
    • February 2016
    • January 2016
    • December 2015
    • November 2015
    • October 2015
    • September 2015
    • August 2015
    • July 2015
    • June 2015
    • May 2015
    • April 2015
    • March 2015
    • February 2015
    • January 2015
    • December 2014
    • November 2014
© 2021 Business Intelligence Info
Power BI Training | G Com Solutions Limited