Source Archives - Business Intelligence Info

Tag Archives: Source

ETL company Airbyte raises $5.2M to integrate open source data

March 3, 2021 Big Data

Join Transform 2021 for the most important themes in enterprise AI & Data. Learn more.

Airbyte today announced it has raised $ 5.2 million in seed funding as part of an effort to make open source tools for managing and integrating data more accessible.

The company, which offers an open source extract transform and load (ETL) tool used to create data pipelines, is now seeking to further democratize that process. This includes, for example, building complementary open source tools to govern and secure data, Airbyte cofounder and CEO Michel Tricot told VentureBeat.

Internal IT teams have historically employed ETL tools to move data between repositories. In recent years, however, data analysts have been using these tools to load data into warehouses without requiring any intervention on the part of an IT team.

All of those tools are licensed by individuals. Airbyte plans to eventually provide versions of its tools licensed by organizations, along with an option to access those tools via a service hosted by Airbyte. Tricot said the company is also planning a managed integration service. “We won’t be focusing on monetization until 2022,” he said.

Accel led the current round of funding, with participation from Y Combinator; 8VC; Segment cofounder Calvin French-Owen; former Cloudera GM Charles Zedlewski; Datavant cofounder and CEO Travis May; Machinify president Alain Rossmann; and Auren Hoffman, cofounder and CEO of LiveRamp and CEO of Safegraph.

As of the end of January, more than 600 organizations are using Airbyte ETL tools, including Safegraph, Dribbble, Mercato, GraniteRock, Agridigital, and Cart.com. Many of those organizations are attracted to Airbyte because they don’t have to wait for a provider of commercial ETL tools to create connectors for various data sources. Instead, the community collaboratively builds and supports the connectors it deems most critical, Tricot said. The community has thus far certified 50 connectors. Those connectors are encapsulated Docker containers, which enables them to be deployed on any platform.

ETL processes, along with other classes of data preparation tools, are being reevaluated as organizations increasingly realize that the quality of any AI model they build is dependent on how reliable the data used to train machine learning algorithms is. Data scientists also want to be able to easily update data needed to retrain models as business conditions evolve, which usually entails having more direct control over what data sources are employed to train those models.

As critical as control over any dataset may be, data scientists are discovering that much of the data stored within enterprise systems is not all that consistent or reliable. Data science teams can easily find themselves spending more time addressing data plumbing issues than they do constructing AI models. Successfully building an AI model, as a consequence, can often require months of time and effort.

ETL tools are not going to resolve that issue on their own. But the easier data becomes to manipulate, the less time it will take to build an AI model and then continuously maintain it as new data sources become available.

It’s not clear what impact the availability of open source ETL tools is having on providers of the rival commercial offerings some organizations have been employing for decades. But at a time when many organizations are under pressure to reduce the total cost of IT, the allure of open source software has proven undeniable.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

up-to-date information on the subjects of interest to you
our newsletters
gated thought-leader content and discounted access to our prized events, such as Transform
networking features, and more

Become a member

Let’s block ads! (Why?)

Big Data – VentureBeat

AI Weekly: The challenges of creating open source AI training datasets

February 20, 2021 Big Data

In January, AI research lab OpenAI released Dall-E, a machine learning system capable of creating images to fit any text caption. Given a prompt, Dall-E generates photos for a range of concepts, including cats, logos, and glasses.

The results are impressive, but training Dall-E required building a large-scale dataset that OpenAI has so far opted not to make public. Work is ongoing on an open source implementation, but according to Connor Leahy, one of the data scientists behind the effort, development has stalled because of the challenges in compiling a corpus that respects both moral and legal norms.

“There’s plenty of not-legal-to-scrape data floating around that isn’t [fair use] on platforms like social media, Instagram first and foremost,” Leahy, who’s a member of the volunteer AI research effort EleutherAI, told VentureBeat. “You could scrape that easily at large scale, but that would be against the terms of service, violate people’s consent, and probably scoop up illegal data both due to copyright and other reasons.”

Indeed, creating AI training datasets in a privacy-preserving, ethical way remains a major blocker for researchers in the AI community, particularly those who specialize in computer vision. In January 2019, IBM released a corpus designed to mitigate bias in facial recognition algorithms that contained nearly a million photos of people from Flickr. But neither the photographers nor the subjects of the photos were notified by IBM that their work would be included. Separately, an earlier version of ImageNet, a dataset used to train AI systems around the world, was found to contain photos of naked children, porn actresses, college parties, and more — all scraped from the web without those individuals’ consent.

“There are real harms that have emerged from casual repurposing, open-sourcing, collecting, and scraping of biometric data,” said Liz O’Sullivan, cofounder and technology director at the Surveillance Technology Oversight Project, a nonprofit organization litigating and advocating for privacy. “[They] put people of color and those with disabilities at risk of mistaken identity and police violence.”

Techniques that rely on synthetic data to train models might lessen the need to create potentially problematic datasets in the first place. According to Leahy, while there’s usually a minimum dataset size needed to achieve good performance on a task, it’s possible to a degree to “trade compute for data” in machine learning. In other words, simulation and synthetic data, like AI-generated photos of people, could take the place of real-world photos from the web.

“You can’t trade infinite compute for infinite data, but compute is more fungible than data,” Leahy said. “I do expect for niche tasks where data collection is really hard, or where compute is super plentiful, simulation to play an important role.”

O’Sullivan is more skeptical that synthetic data will generalize well from lab conditions to the real world, pointing to existing research on the topic. In a study last January, researchers at Arizona State University showed that when an AI system trained on a dataset of images of engineering professors was tasked with creating faces, 93% were male and 99% white. The system appeared to have amplified the dataset’s existing biases — 80% of the professors were male and 76% were white.

On the other hand, startups like Hazy and Mostly AI say that they’ve developed methods for controlling the biases of data in ways that actually reduce harm. A recent study published by a group of Ph.D. candidates at Stanford claims the same — the coauthors say their technique allows them to weight certain features as more important in order to generate a diverse set of images for computer vision training.

Ultimately, even where synthetic data might come into play, O’Sullivan cautions that any open source dataset could put people in that set at greater risk. Piecing together and publishing a training dataset is a process that must be undertaken thoughtfully, she says — or not at all, where doing so might result in harm.

“There are significant worries about how this technology impacts democracy and our society at large,” O’Sullivan said.

For AI coverage, send news tips to Khari Johnson and Kyle Wiggers and AI editor Seth Colaner — and be sure to subscribe to the AI Weekly newsletter and bookmark our AI channel, The Machine.

Thanks for reading,

Kyle Wiggers

AI Staff Writer

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

up-to-date information on the subjects of interest to you
our newsletters
gated thought-leader content and discounted access to our prized events, such as Transform
networking features, and more

Become a member

Let’s block ads! (Why?)

Big Data – VentureBeat

How the pandemic is accelerating enterprise open source adoption

January 27, 2021 Big Data

Enterprises increasingly shifted to open source software solutions in 2020 to meet their remote organizational needs and address new market demands for quality and speed. COVID-19 challenged not only the economy, but also enterprises’ existing frameworks for how, when, and at what volume people use information technology.

Last year, major players like LinkedIn and Spotify open-sourced tools they developed — from Java machine learning libraries to audio file processing ecosystems — for non-proprietary IT team members like data scientists and software engineers to use. Seed-level startups like Eradani and RudderStack that built their products on open source thrived despite launching only a few months before the pandemic began to escalate.

Most companies don’t publicize their internal tooling and infrastructure strategy transformations, but GitHub’s data suggests these have much to do with open source solutions. GitHub says 72% of Fortune 50 companies used GitHub Enterprise, which runs GitHub services on their local networks, between Q4 2019 and Q3 2020. GitHub also found over 40% YOY growth in open source project creation per any active user between late April 2019 and late April 2020. This first spiked with the pandemic in early March, when countries began to close schools, ban visitors, and initiate lockdowns en masse.

In an interview with VentureBeat, GitHub VP Mario Rodriguez said, “Open source project creation just kind of shoots up” after March. He added, “2020 is interesting because everything from a technology perspective got accelerated, and you were trying to do more and more.”

According to Rodriguez, “2020 also opened up a new set of talent from a software development perspective.”

Companies were previously limited by their geography and could only hire within certain cities, like San Francisco or New York. But the transition to remote work has begun to change that.

“Now, you do not have those restrictions … and the majority of software developers out there use open source today. And so you bring it into your enterprise now at an accelerated rate, which allows you to learn and continue to evolve your practices on how you develop software and how you use open source,” he said.

The pandemic disrupted existing tech trends and likely helped amplify the growing movement toward open source software solutions as enterprises’ distributed, remote workforces needed to use more custom applications internally, to innovate their IT quickly with the help of reusable code, and to retain developers by using the tools they prefer.

Speed: Accelerating technology with a remote, distributed workforce

Internet traffic skyrocketed by over 30% in March, especially to platforms for online learning and telecommuting. Microsoft Teams users set a new record for 2.7 billion meeting minutes in one day, and Microsoft, along with Netflix and YouTube, temporarily reduced video streaming quality and download speeds to cut back on bandwidth consumption.

These changes highlighted consumer demands for new digital communication tools and challenged enterprise IT teams to create and manage them — quickly. And enterprise IT teams, now distributed and in some cases fully remote, had to organize their own work around new kinds of applications and develop them in days or weeks, as opposed to months or years.

“From an IT perspective, you have to accelerate the [number] of apps that you are creating for your internal use as an enterprise,” Rodriguez said. “So that has actually allowed the enterprises to say, ‘You know what, if we are going to have these constraints, maybe we should start to research and figure out a way to empower more use of open source internally, as well.’ … You’re trying to go faster.”

While some enterprise frameworks and custom logic for business applications remain fully proprietary, integrating open source code can be a faster way to develop most software. Developers can import the existing work of thousands of people with a few inputs, making it easier to pull new applications together. For example, companies expedite their training and development of machine learning models with Google’s TensorFlow. Now that information is more democratized with open source software, some enterprise leaders suggest it’s hard to compete without it.

“I’m a member of the CTO forum, which is a group of 150 CTOs around the globe that talks a lot about tech,” Pinterest head of engineering Jeremy King said in a recent interview with VentureBeat. “And for sure, lifecycles have sped up.”

King described how companies used to try out maybe three or four vendors with an open source technology stack at a time for six months. Afterward, the company might figure out which vendor performed best and negotiate a rate.

“All that has gone away,” King said. He said if a company knows that a given open technology works, they’ll make a prototype and they’ll have changed the cycle by the next week.

“People are more rapidly adopting [open source], and I think it’s just because the tolerance for failure and making mistakes and moving quick has gone up,” he said. He explained that this is also true when it comes to “dealing with the consequences of moving fast in production, which, a year ago was unheard of.”

Elephant Ventures CEO Arthur Shectman also commented on the pandemic’s disruption of enterprise IT in an interview with VentureBeat. “The market kind of collapsed around people. In that moment of high volatility or prices and market stress, you reach for data,” Shectman said. “You’re like, ‘These decisions are going to have profound impacts … the net impact a week from now is huge.’”

Elephant Ventures is a digital transformation consultancy that helps corporations like Pfizer and American Express build their engineering capabilities. Shectman said he applies a technology readiness framework to his approach, finding ways for clients to create increments of business value with ETL technologies, API tooling patterns, and other strategies for deploying applications and workflows.

According to Shectman, “People were clamoring for data, and they were clamoring to transform their kind of data ecosystems, very rapidly.” He said that in the past few months, the conversation went from planning out three years of digital transformation retool to looking for immediate answers with a return on investment in 90 days. Shectman noted that waiting to purchase proprietary software could cost his enterprise clients $ 3 million in some instances.

Shectman said proprietary software’s cost and speed of deployment became greater roadblocks for his clients during 2020. “I felt like there was a lot more willingness to instantly adopt open source technologies and start applying them without any additional kind of software purchase cost to get a rapid ROI cycle project,” Shectman added. “Over the last year, if you could demonstrate that you knew how to implement it, you had a pattern that generated success in critical dimensions.”

Volume: Innovating with code reuse and existing tools

Developers can build applications at greater volumes by reusing open source code instead of starting from scratch. Enterprise IT teams can also integrate open source tools into their existing workflows to manage their data with improved precision. This control is increasingly important, given the pandemic’s overall shift toward digitalization, which increases the amount of data itself.

“The number of apps that get created right now are at an all-time high. And then the number of those apps that [use] open source are also at an all-time high. It’s probably because of code reuse and ability to just go from zero to 60% or 70% of what you need to create very, very quickly,” Rodriguez said.

These current trends in open source software are a continuation of many enterprises’ strategies. “I think we use a lot of open source technologies, largely due to the fact that our scale often prevents us from using a commercial product,” King said.

“Pinterest is a 10-year-old company, and we have billions of pins … and that’s not always something you can just buy off the shelf. And so whether it’s logging data, understanding images, searching — all these technologies didn’t exist when we started; they’re getting better and better over time. Even the off-the-shelf cloud providers are getting better. But we’ve made Pinterest very unique to what we’re looking for as a result of us using a lot of open source technology,” he added.

Other enterprises have tailored their tech stacks to maximize productivity and output with open source solutions more recently. Even maintaining the same products requires additional speed of deployment when competing in a volatile COVID-19 era market.

McKinsey reports that North American businesses accelerated their share of digitized product offerings by 20% between March and October 2020. Open source software might play a role in expediting this process if the bulk of digitized products’ code can be built from an existing repository’s code.

“For a long time, the digital economy and the computer revolution were just driven by Moore’s law, with your chips getting better and faster at this crazy growth rate,” Shectman said as he traced his client’s technology solutions from the early 2000s to late 2020. “Now, after you go through the early iterations of your product, you are able to do a certain amount of things with commercially available software, [but] you’re gonna have typically an easier time unwrapping and fixing or customizing the open source stuff.”

According to Shectman, understanding and acting upon data has become key to enterprises. “I think it’s the scale of data production. I don’t think anybody really had a great sense of how rapidly data would proliferate in the world.”

The market’s space for tools, particularly open source technologies that sieve and curate data, has widened as a result. In the past 30 years — and mostly in the last 10 — over 200 companies with open source technology at their core have raised over $ 10 billion in capital.

Startups such as data warehousing platform RudderStack, which provides an open source-based alternative to Segment, have recently capitalized on this change. RudderStack was launched in late 2019, and its 2020 growth mirrors the growth patterns of other early-stage startups, like Prisma and Streamlit, which contribute to GitHub community code and base their business strategies in open source software.

RudderStack marketing lead Gavin Johnson said in an interview with VentureBeat, “Being open source makes it easy to deploy RudderStack in your own environment. … Anyone can look through the code and figure out what is being done with their customer data in-product, something you can’t do with closed-source products.” According to Johnson, RudderStack’s open source software reduces the number of tools enterprise engineering teams would have to build for sending and collecting data and allows the teams to modify the platform with custom integrations. He also suggested end-user enterprise IT teams tend to save more money with these alternatives to proprietary platforms.

Novelty: Energizing IT teams and long-term growth

The Linux Foundation reports that in 2020, 93% of hiring managers found it difficult to recruit employees with open source software skills and 37% of them wanted to hire more skilled IT professionals, despite the pandemic’s economic impact.

Rodriguez told VentureBeat that in 2020, “We didn’t see companies shrink in developers. We actually saw them expand in developers.” He believes this growth has to do with enterprises’ goals that now, more than ever, revolve around IT capabilities.

“You’re trying to create all this software, which means that you need more developers in order to do it right. And if you’re trying to actually have more, [you need] to not only have the best tools for your developers but have the best practices as well.” He said that is another reason he thinks open source accelerated significantly in 2020.

The open source community organizes conferences, groups, and companies around making code more accessible. Developers contribute to repositories like Open Source for Good and develop personal projects with open source code.

Johnson said that data engineers and data scientists work a lot in open source. He said, “RudderStack’s mission is to help make engineering and data teams be heroes of their organizations. So we needed to build our product in a way they like, which means open source.”

Some enterprises are turning to open source tools to bring in skilled technical employees. King said open source attracts and retains engineers. “And the people who are the best in the world want to publish, and they want to work on things that are well known, and they want to contribute back. And so it is definitely a good return on investment as well,” he said.

In addition to using open source for internal transformations, enterprises have been contributing open source tools to the community. Open source contributions can improve enterprises’ visibility and relevance, and this business strategy is important now that most, if not all, of their public operations are digital. RedHat brings open source technologies to enterprises that need them and has since it was founded in 1993. But in 2020, end-user companies like Facebook, LinkedIn, Spotify, and Uber have also begun to open-source their own tools for the public.

“The open source movement is pretty well established and reasonably mature at this point, so it’s not a surprise for any large corporation,” Shectman said. “If there’s a bit of awareness, people can afford to shy away from some of the convenience of a particular vendor-specific thing in order to allow themselves fluidity.” Shectman suggested that businesses face similar constraints with their staffing, technology, and ability to create effective products. He added that he’s seen open source software help businesses remove constraints, including proprietary vendor lock-in.

Rodriguez suggested that the advancements open source can provide in regard to speed, volume, and overall business strategy aren’t easily matched. “What everyone now is realizing is you cannot out-innovate, or you cannot out-execute open source,” he said.

Will open source continue to grow in 2021 like it did in 2020? Rodriguez thinks it could. “For any company out there, from the most powerful company in the world to a startup, you’re not going to be able to hire [enough] people to build the software of that quality — open source gives you that immediately.”

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

up-to-date information on the subjects of interest to you
our newsletters
gated thought-leader content and discounted access to our prized events, such as Transform
networking features, and more

Become a member

Let’s block ads! (Why?)

Big Data – VentureBeat

#PowerBI – Change the data source in your composite model with direct query to AS/ Power BI Dataset

December 29, 2020 Self-Service BI

I have been playing around with the new awesome (preview) feature in the December Power BI Desktop release where we can use DirectQuery for Power BI datasets and Azure Analysis services (link to blogpost)

In my case I combined data from a Power BI dataset, Azure Analysis Services, and a local Excel sheet. The DirectQuery sources was in a test environment.

I then wanted to try this on the actual production datasets and wanted to change the datasources – and was a bit lost on how to do that but luckily found a way that I want to share with you.