Over the past decade, “big data” has become Silicon Valley’s biggest buzzword. When they’re trained on mind-numbingly large data sets, machine learning (ML) models can develop a deep understanding of a given domain, leading to breakthroughs for top tech companies. Google, for instance, fine-tunes its ranking algorithms by tracking and analyzing more than one trillion search queries each year. It turns out that the Solomonic power to answer all questions from all comers can be brute-forced with sufficient data.
But there’s a catch: Most companies are limited to “small” data; in many cases, they possess only a few dozen examples of the processes they want to automate using ML. If you’re trying to build a robust ML system for enterprise customers, you have to develop new techniques to overcome that dearth of data.
Two techniques in particular — transfer learning and collective learning — have proven critical in transforming small data into big data, allowing average-sized companies to benefit from ML use cases that were once reserved only for Big Tech. And because just 15% of companies have deployed AI or ML already, there is a massive opportunity for these techniques to transform the business world.
Above: Using the data from just one company, even modern machine learning models are only about 30% accurate. But thanks to collective learning and transfer learning, Moveworks can determine the intent of employees’ IT support requests with over 90% precision.
From DIY to open source
Of course, data isn’t the only prerequisite for a world-class machine learning model — there’s also the small matter of building that model in the first place. Given the short supply of machine learning engineers, hiring a team of experts to architect an ML system from scratch is simply not an option for most organizations. This disparity helps explain why a well-resourced tech company like Google benefits disproportionately from ML.
But over the past several years, a number of open source ML models — including the famous BERT model for understanding language, which Google released in 2018 — have started to change the game. The complexity of creating a model the caliber of BERT, whose aptly named “large” version has about 340 million parameters, means that few organizations can even consider quarterbacking such an initiative. However, because it’s open source, companies can now tweak that publicly available playbook to tackle their specific use cases.
To understand what these use cases might look like, consider a company like Medallia, a Moveworks customer. On its own, Medallia doesn’t possess enough data to build and train an effective ML system for an internal use case, like IT support. Yet its small data does contain a treasure trove of insights waiting for ML to unlock them. And by leveraging new techniques to glean these insights, Medallia has become more efficient, from recognizing which internal workflows need attention to understanding the company-specific language its employees use when asking for tech support.
Massive progress with small data
So here’s the trillion-dollar question: How do you take an open source ML model designed to solve a particular problem and apply that model to a disparate problem in the enterprise? The answer starts with transfer learning, which, unsurprisingly, entails transferring knowledge gained from one domain to a different domain that has less data.
For example, by taking an open source ML model like BERT — designed to understand generic language — and refining it at the margins, it is now possible for ML to understand the unique language employees use to describe IT issues. And language is just the beginning, since we’ve only begun to realize the enormous potential of small data.
Above: Transfer learning leverages knowledge from a related domain — typically one with a greater supply of training data — to augment the small data of a given ML use case.
More generally, this practice of feeding an ML model a very small and very specific selection of training data is called “few-shot learning,” a term that’s quickly become one of the new big buzzwords in the ML community. Some of the most powerful ML models ever created — such as the landmark GPT-3 model and its 175 billion parameters, which is orders of magnitude more than BERT — have demonstrated an unprecedented knack for learning novel tasks with just a handful of examples as training.
Taking essentially the entire internet as its “tangential domain,” GPT-3 quickly becomes proficient at these novel tasks by building on a powerful foundation of knowledge, in the same way Albert Einstein wouldn’t need much practice to become a master at checkers. And although GPT-3 is not open source, applying similar few-shot learning techniques will enable new ML use cases in the enterprise — ones for which training data is almost nonexistent.
The power of the collective
With transfer learning and few-shot learning on top of powerful open source models, ordinary businesses can finally buy tickets to the arena of machine learning. But while training ML with transfer learning takes several orders of magnitude less data, achieving robust performance requires going a step further.
That step is collective learning, which comes into play when many individual companies want to automate the same use case. Whereas each company is limited to small data, third-party AI solutions can use collective learning to consolidate those small data sets, creating a large enough corpus for sophisticated ML. In the case of language understanding, this means abstracting sentences that are specific to one company to uncover underlying structures:
Above: Collective learning involves abstracting data — in this case, sentences — with ML to uncover universal patterns and structures.
The combination of transfer learning and collective learning, among other techniques, is quickly redrawing the limits of enterprise ML. For example, pooling together multiple customers’ data can significantly improve the accuracy of models designed to understand the way their employees communicate. Well beyond understanding language, of course, we’re witnessing the emergence of a new kind of workplace — one powered by machine learning on small data.