• Home
  • About Us
  • Contact Us
  • Privacy Policy
  • Special Offers
Business Intelligence Info
  • Business Intelligence
    • BI News and Info
    • Big Data
    • Mobile and Cloud
    • Self-Service BI
  • CRM
    • CRM News and Info
    • InfusionSoft
    • Microsoft Dynamics CRM
    • NetSuite
    • OnContact
    • Salesforce
    • Workbooks
  • Data Mining
    • Pentaho
    • Sisense
    • Tableau
    • TIBCO Spotfire
  • Data Warehousing
    • DWH News and Info
    • IBM DB2
    • Microsoft SQL Server
    • Oracle
    • Teradata
  • Predictive Analytics
    • FICO
    • KNIME
    • Mathematica
    • Matlab
    • Minitab
    • RapidMiner
    • Revolution
    • SAP
    • SAS/SPSS
  • Humor

LinkedIn open-sources WhereHows, a metadata management tool

March 6, 2016   Big Data

LinkedIn today announced that it’s open-sourcing a piece of its software called WhereHows, which allows anyone in a company to learn about and share information on data that company has under management. The software is now available on GitHub under an open-source Apache license.

LinkedIn has many systems for storing and processing data, including Teradata’s data warehousing technology, the open source Hadoop distributed file system, the open source Hive data warehousing software, and its own open source Pinot real-time analytics software. It’s not trivial to know exactly where a kind of data lives. WhereHows can help with that, because it lets people run wide-ranging searches across everything, and people can post about the data for which they have knowledge.

Rather than viewing data, WhereHows lets people track the specific types of data that are available. In other words, it’s a tool for discovering and managing metadata. WhereHows is available to people at LinkedIn in the form of a user interface and an application programming interface (API) for developers. It serves up information on more than 25,000 publicly shared data sets from HDFS alone. It also takes into consideration flows of data through multiple tools; so, for example, it surfaces 150,000 flows from its open source job scheduler. But instead of LinkedIn keeping the software to itself, the company is opening up and sharing it for other companies with complex systems to use and even build on.

“We are open sourcing WhereHows on GitHub, as well as our discussion group, to share our work with the broader data community,” LinkedIn staff data engineer Eric Sun wrote in a blog post. “We highly encourage contributors from different companies to create new features and commit important bug fixes. Though metadata management tends to be tightly coupled to other components in the company, we will continue to try to refactor LinkedIn-internal integrations into WhereHows into generic templates or plugins in open source.”

This is hardly LinkedIn’s first open source contribution. Pinot became available last year, and before that, there were Azkaban, Kafka, Samza, and Voldemort.

But data discovery, or the data catalog, is a whole other type of software. Many proprietary tools are available. For instance, startup Tamr came out with something last year. So the WhereHows release could be a big deal for companies with complex data infrastructures. In return, LinkedIn could easily find people willing to improve the technology and maybe even join the company’s ranks.

LinkedIn wants to enhance the software by giving it integration with tools like Kafka, Samza, Gobblin, and Nuage, and it could also add in information on joins between different types of data, wrote Sun.

Documentation for all parts of WhereHows is here.

LinkedIn is the world’s largest professional network on the internet, with more than 259 million members worldwide, including executives from Fortune 500 companies. Founded on May 5, 2003, by Reid Hoffman and founding team members fr… read more »

VB Profile Logo LinkedIn open sources WhereHows, a metadata management toolNew! Track LinkedIn’s Landscape to stay on top of the industry in 3 minutes a day. Understand the entire ecosystem, monitor innovation, and track deal flows. Learn more.

This entry passed through the Full-Text RSS service – if this is your content and you’re reading it on someone else’s site, please read the FAQ at fivefilters.org/content-only/faq.php#publishers.

VentureBeat » Big Data News | VentureBeat

LinkedIn, Management, Metadata, opensources, tool, WhereHows
  • Recent Posts

    • Kevin Hart Joins John Hamburg For New Netflix Comedy Film Titled ‘Me Time’
    • Who is Monitoring your Microsoft Dynamics 365 Apps?
    • how to draw a circle using disks, the radii of the disks are 1, while the radius of the circle is √2 + √6
    • Tips on using Advanced Find in Microsoft Dynamics 365
    • You don’t tell me where to sit.
  • Categories

  • Archives

    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • March 2020
    • February 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    • December 2016
    • November 2016
    • October 2016
    • September 2016
    • August 2016
    • July 2016
    • June 2016
    • May 2016
    • April 2016
    • March 2016
    • February 2016
    • January 2016
    • December 2015
    • November 2015
    • October 2015
    • September 2015
    • August 2015
    • July 2015
    • June 2015
    • May 2015
    • April 2015
    • March 2015
    • February 2015
    • January 2015
    • December 2014
    • November 2014
© 2021 Business Intelligence Info
Power BI Training | G Com Solutions Limited