researchers Archives - Business Intelligence Info

Tag Archives: researchers

Google’s ML-fairness-gym lets researchers study the long-term effects of AI’s decisions

February 6, 2020 Big Data

Determining whether an AI system is maintaining fairness in its predictions requires an understanding of models’ short- and long-term effects, which might be informed by disparities in error metrics on a number of static data sets. In some cases, it’s necessary to consider the context in which the AI system operates in addition to the aforementioned error metrics, which is why Google researchers developed ML-fairness-gym, a set of components for evaluating algorithmic fairness in simulated social environments.

ML-fairness-gym — which was published in open source on Github this week –is designed to be used to research the long-term effects of automated systems by simulating decision-making using OpenAI’s Gym framework. AI-controlled agents interact with digital environments in a loop, and at each step, an agent chooses an action that affects the environment’s state. The environment then reveals an observation that the agent uses to inform its next actions, so that the environment models the system and dynamics of a problem and the observations serve as data.

For instance, given the classic lending problem, where the probability that groups of applicants pay back a bank loan is a function of their credit score, the bank acts as the agent and receives applicants, their scores, and their membership in the form of environmental observations. It makes a decision — accepting or rejecting a loan — and the environment models whether the applicant successfully repays or defaults and adjusts their credit score accordingly. Throughout, ML-fairness-gym simulates the outcomes so that the effects of the bank’s policies on fairness to the applicants can be assessed.

ML-fairness-gym in this way cleverly avoids the pitfalls of static data set analysis. If the test sets (i.e., corpora used to evaluate model performance) in classical fairness evaluations are generated from existing systems, they may be incomplete or reflect the biases inherent to those systems. Furthermore, the actions informed by the output of AI systems can have effects that might influence their future input.

Google’s ML fairness gym lets researchers study the long term effects of AI’s decisions

Above: In the lending problem scenario, this graph illustrates changing credit score distributions for two groups over 100 steps of simulation.

Image Credit: Google

“We created the ML-fairness-gym framework to help ML practitioners bring simulation-based analysis to their ML systems, an approach that has proven effective in many fields for analyzing dynamic systems where closed form analysis is difficult,” wrote Google Research software engineer Hansa Srinivasan in a blog post.

Several environments that simulate the repercussions of different automated decisions are available, including one for college admissions, lending, attention allocation, and infectious disease. (The ML-fairness-gym team cautions that the environments aren’t meant to be hyper-realistic and that best-performing policies won’t necessarily translate to the real world.) Each have a set of experiments corresponding to published papers, which are meant to provide examples of ways ML-fairness-gym can be used to investigate outcomes.

The researchers recommend using ML-fairness-gym to explore phenomena like censoring in the observation mechanism, errors from the learning algorithm, and interactions between the decision policy and the environment. The simulations allow for the auditing of agents to assess the fairness of decision policies based on observed data, which can motivate data collection policies. And they can be used in concert with reinforcement learning algorithms — algorithms that spur on agents with rewards — to derive new policies with potentially novel fairness properties.

In recent months, a number of corporations, government agencies, and independent researchers have made attempts at tackling the so-called “black box” problem in AI — the opaqueness of some AI systems — with varying degrees of success.

“Machine learning systems have been increasingly deployed to aid in high-impact decision-making, such as determining criminal sentencing, child welfare assessments, who receives medical attention and many other settings,” continued Srinivasan. “We’re excited about the potential of the ML-fairness-gym to help other researchers and machine learning developers better understand the effects that machine learning algorithms have on our society, and to inform the development of more responsible and fair machine learning systems.”

In 2017, the U.S. Defense Advanced Research Projects Agency launched DARPA XAI, a program that aims to produce “glass box” models that can be easily understood without sacrificing performance. In August, scientists from IBM proposed a “factsheet” for AI that would provide information about a model’s vulnerabilities, bias, susceptibility to adversarial attacks, and other characteristics. A recent Boston University study proposed a framework to improve AI fairness. And Microsoft, IBM, Accenture, and Facebook have developed automated tools to detect and mitigate bias in AI algorithms.

Let’s block ads! (Why?)

Big Data – VentureBeat

SenseTime researchers create a benchmark to test face forgery detectors

January 16, 2020 Big Data

Face swapping is a category of deepfakes that extracts the faces of people in existing media and replaces them with other peoples’ features, typically with AI and machine learning. It’s been popularized by apps like MixBooth and SnapChat, and while the underlying techniques have enabled sophisticated image editing for legitimate purposes, they’ve also given rise to concerns about potential misuse or abuse.

Various groups have compiled manipulated media to support the development of face swapping detection methods, but the samples that have released so far are relatively few in number or overly artificial. That’s why researchers from SenseTime Research, the R&D division of Hong Kong-based tech startup SenseTime, partnered with Nanyang Technological University in Singapore to design a new large-scale benchmark for face forgery detection. They call it DeeperForensics-1.0, and they say it’s the largest corpora of its kind with over 60,000 videos containing roughly 17.6 million frames.

According to the researchers, all source videos in DeeperForensics-1.0 were carefully selected for their quality and diversity. They’re ostensibly more realistic than those in other data sets in that they’re closer in kind to real-world detection scenarios, and in that they contain compression, blurriness, and transmission artifacts matching those found in the wild.

To build DeeperForensics-1.0, the researchers collected face data from 100 paid male and female actors of 26 different nationalities and ranging in age from 20 to 45, all of whom were instructed to turn their heads in nine lighting conditions and speak naturally with over 53 expressions. They ran these through an AI framework — DeepFake Variational AutoEncoder, or DF-VAE — using 1,000 YouTube videos as target videos, where each of the 100 actors’ faces was swapped onto 10 targets. And they deliberately distorted each video in 35 different ways to simulate real-world scenarios, such that the final data set contained 50,000 unmanipulated videos and 10,000 manipulated videos.

“We find that the source faces play a much more critical role than the target faces in building a high-quality data set,” wrote the researchers in a preprint paper detailing their work. “Specifically, the expressions, poses, and lighting conditions of source faces should be much richer in order to perform robust face swapping.”

The researchers also created what they call a “hidden” test set within DeeperForensics-1.0 — a set of 400 videos carefully selected to better imitate fake videos in real scenes. Curating the set involved collecting fake videos generated by unknown face-swapping methods and obscuring them with distortions commonly seen in real scenes, and subsequently choosing only videos that fooled at least 50 out of 100 human observers in a user study.

To evaluate the quality of DeeperForensics-1.0 compared with other publicly available data sets, the researchers tasked 100 experts in computer vision with ranking the the quality of a subset of videos contained within it. They report that DeeperForensics-1.0 came out ahead on average in terms of realism for its scale compared with FaceForensics++, Celeb-DF, and other popular deepfake detection corpora.

In future work, the research team intends to expand DeeperForensics gradually and work with the research community toward identifying evaluation metrics for face forgery detection methods.

The fight against deepfakes appears to be ramping up. Last summer, members of DARPA’s Media Forensics program tested a prototypical system that could automatically detect AI-generated videos in part by looking for cues like unnatural blinking. Startups like Truepic, which raised an $ 8 million funding round in July, are experimenting with deepfakes “detection-as-a-service” business models. And in December 2019, Facebook together with the Partnership on AI, Microsoft, and academics launched the Deepfake Detection Challenge, which will offer millions of dollars in grants and awards to spur the development of deepfake-detecting systems.

Let’s block ads! (Why?)

Big Data – VentureBeat

Carnegie Mellon researchers propose AI that surfaces positive online comments

January 13, 2020 Big Data

Researchers at Carnegie Mellon’s Language Technologies Institute say they’ve developed a system that taps machine learning to analyze online comments and pick out those that defend or sympathize with disenfranchised peoples. Although it hasn’t been commercialized, they’ve used it in experiments to search for nearly a million YouTube comments, focusing on the Rohingya refugee crisis and the February 2019 Pulwama terrorist attack in Kashmir. And they hope it’ll form the foundation of a future system that’ll reduce the manual effort necessary to curate comments on publisher websites, social media, and elsewhere.

Tamping down on abusive online behavior is no easy feat, particularly considering the level of toxicity in some social circles. More than one in five respondents to a survey by the Anti-Defamation League, a nonprofit that tracks and fights anti-Semitism, reported having been subjected to threats of violence. And nearly one in five said they’d experienced sexual harassment or stalking and sustained harassment, while upwards of 20% said the harassment was the result of their gender identity, race, ethnicity, sexual orientation, religion, occupation, or disability.

As the researchers explain, improvements in AI language models — which learn from many examples to predict what words are likely to occur in a given sentence — made it possible to analyze such large quantities of text. The study’s contribution was a technique enabling those models to digest short texts originating from South Asia, which can be difficult to interpret because they often contain spelling and grammar mistakes and combine different languages and systems of writing.

Specifically, the researchers obtained embeddings — numerical representations of words — that revealed novel language groupings or clusters. Language models create these so that words with similar meanings are represented in the same way, making it possible to compute the proximity of a word to others in a comment or post.

The team reports that in experiments, their approach worked as well or better than commercially available solutions. Random samplings of the YouTube comments showed about 10% were positive, compared with the 88% found with the AI algorithm.

“Even if there’s lots of hateful content, we can still find positive comments,” said post-doctoral research Ashiqur R. KhudaBukhsh, a contributing author on aforthcoming paper describing the work. He’ll present his findings with coauthors Shriphani Palakodety and Jaime Carbonell at the Association for the Advancement of Artificial Intelligence annual conference next month in New York City.

The study follows the release of a data set by Jigsaw — the organization working under Google parent company Alphabet to tackle cyber bullying, censorship, disinformation, and other digital issues of the day — containing hundreds of thousands of comments and annotations with toxicity and identity labels. It’s intended to help measure bias in AI comment classification systems, which Jigsaw and others have historically measured using synthetic data from template sentences.

In a related development, researchers at the Georgia Institute of Technology and email marketing startup Mailchimp recently proposed RECAST, an interactive tool for examining toxicity detection models by visualizing explanations for predictions and providing alternative wordings for detected toxic speech. The plan to release an open source browser extension in the near future.

Let’s block ads! (Why?)

Big Data – VentureBeat

Intel researchers propose AI that recognizes faces from thermal images

January 10, 2020 Big Data

Is thermal imagery detailed enough to enable an AI model to recognize peoples’ facial features? That’s the question Intel and Gdansk University of Technology researchers sought to answer in study recently presented at the Institute of Electrical and Electronics Engineers’ 12th International Conference on Human System Interaction. In it, they investigated the performance of a model trained on visible light data that was subsequently retrained on thermal images.

As the researchers point in a paper describing their work, thermal imagery is often used in lieu of RGB camera data within environments where privacy is preferred or otherwise mandated, like medical facilities. That’s because it’s able to obscure personally identifying details like eye color and jaw line.

The team tapped two data sets of facial thermal images., the first of which — SC3000-DB — was created using a Flir ThermaCam SC3000 infrared camera. It contained 766 images of 40 volunteers from a cohort of 19 men and 21 women, who were asked to sit and look at the camera for a span of two minutes. As for the second data set — the IRIS data set, from the Visual Computing and Image Processing Lab at Oklahoma State University — it contained 4,190 images collected with the assistance of 30 people, many of whom moved their heads and used different facial expressions.

The researchers first cropped the images to regions containing only peoples’ faces, using a machine learning model. Then, they employed another model to extract facial features from the images and represent them numerically, as vectors (sets of values corresponding to specific features). Lastly, they employed a third model trained on visible light images to validate whether such a model could be applied to thermal images.

In the course of experiments, the researchers compared two types of facial feature vectors: one that identified the person depicted in a given image and a second based on the similarity of the person’s profile with an input image. They found that the model trained on visible image data generalized well to thermal images, successfully extracting facial features and distinguishing among volunteers with accuracies of 99.5% for SC3000-DB and 82.14% for IRIS.

The team hopes to extend their research to data collected in various other scenarios, such as when subjects’ heads are turned horizontally or vertically. “Many promising visual-processing applications, such as non-contact vital signs estimation and smart home monitoring, can involve private and or sensitive data, such as biometric information about a person’s health,” wrote the researchers. “Thermal imaging, which can provide useful data while also concealing individual identities, is therefore used for many applications.”

Let’s block ads! (Why?)

Big Data – VentureBeat

Researchers detail AI that de-hazes and colorizes underwater photos

December 31, 2019 Big Data

Ever notice that underwater images tend to be be blurry and somewhat distorted? That’s because phenomena like light attenuation and back-scattering adversely affect visibility. To remedy this, researchers at Harbin Engineering University in China devised a machine learning algorithm that generates realistic water images, along with a second algorithm that trains on those images to both restore natural color and reduce haze. They say that their approach qualitatively and quantitatively matches the state of the art, and that it’s able to process upwards of 125 frames per second running on a single graphics card.

The team notes that most underwater image enhancement algorithms (such as those that adjust white balance) aren’t based on physical imaging models, making them poorly suited to the task. By contrast, this approach taps a generative adversarial network (GAN) — an AI model consisting of a generator that attempts to fool a discriminator into classifying synthetic samples as real-world samples — to produce a set of images of specific survey sites that are fed into a second algorithm, called U-Net.

The team trained the GAN on a corpus of labeled scenes containing 3,733 images and corresponding depth maps, chiefly of scallops, sea cucumbers, sea urchins, and other such organisms living within indoor marine farms. They also sourced open data sets including NY Depth, which comprises thousands of underwater photographs in total.

Post-training, the researchers compared the results of their twin-model approach to that of baselines. They point out that their technique has advantages in that it’s uniform in its color restoration, and that it recovers green-toned images well without destroying the underlying structure of the original input image. It also generally manages to recover color while maintaining “proper” brightness and contrast, a task at which competing solutions aren’t particularly adept.

It’s worth noting that the researchers’ method isn’t the first to reconstruct frames from damaged footage. Cambridge Consultants’ DeepRay leverages a GAN trained on a data set of 100,000 still images to remove distortion introduced by an opaque pane of glass, and the open source DeOldify project employs a family of AI models including GANs to colorize and restore old images and film footage. Elsewhere, scientists at Microsoft Research Asia in September detailed an end-to-end system for autonomous video colorization; researchers at Nvidia last year described a framework that infers colors from just one colorized and annotated video frame; and Google AI in June introduced an algorithm that colorizes grayscale videos without manual human supervision.

Let’s block ads! (Why?)

Big Data – VentureBeat

Researchers want to use Mega Man 2 to evaluate AI

December 26, 2019 Big Data

Games have long served as a training ground for AI algorithms, and not without good reason. Games — particularly video games — provide challenging environments against which to benchmark autonomous systems. In 2013, a team of researchers introduced the Arcade Learning Environment, a collection of over 55 Atari 2600 games designed to test a broad range of AI techniques. More recently, San Francisco research firm OpenAI detailed Procgen Benchmark, a set of 16 virtual worlds that measure how quickly models learn generalizable skills.

The next frontier might be Mega Man, if an international team of researchers have their way. In a newly published paper on the preprint server Arxiv.org, they propose EvoMan, a game-playing competition based on the eight boss fights in Capcom’s cult classic Mega Man 2. As they describe it, competitors’ goal is to train an AI agent to defeat every enemy and evaluate their performances by common metrics.

Why Mega Man 2? The paper’s coauthors assert that few other environments test an AI’s ability to win against a single enemy, or how well an AI generalizes to win matches against waves of enemies or coevolves to create increasingly difficult enemies. To this end, in EvoMan, an AI-controlled Mega Man — a robot equipped with a powerful arm cannon — must beat eight so-called Robot Masters equipped with different weapons. Every time a Robot Master is defeated, the agent acquires its weapon, making it easier to defeat the bosses who remain.

Researchers want to use Mega Man 2 to evaluate AI

Above: A table of scores achieved by an agent in the EvoMan benchmark.

Image Credit: EvoMan

As proposed, EvoMan challenge entrants would train their agents on any four enemies and measure how well their learned strategy scales up to a whole set of enemies. The agents would be expected to learn how to identify and react to general patterns like avoiding being shot or shooting at the direction of the enemy, and to deplete an enemy’s health from 100 energy points to 0 by the end of each match.

“The main goal of this competition is that a given agent perform equally good for every boss,” wrote the coauthors, who suggest that competitors be ranked by the mean of performances across bosses. “The winner … will be the one agent that performs equally well on each one of the eight bosses, hopefully defeating them all.”

Atari titles and Mega Man 2 are far from the only games against which AI has been trained and evaluated. In July, a paper published by Facebook researchers posited that Minecraft was well-suited to refining natural language understanding algorithms, and both OpenAI and Google parent company Alphabet’s DeepMind have fine-tuned systems on Valve’s Dota 2 and Activision Blizzard’s Starcraft 2. Separately, scientists at Facebook AI Research, the Lorraine Research Laboratory in Computer Science and its Applications, and the University College London are developing LIGHT, an open source research environment in the form of a large-scale, crowdsourced text adventure within which AI and humans interact as player characters.

Let’s block ads! (Why?)

Big Data – VentureBeat

Researchers teach robots to use inference to complete complex tasks

November 16, 2019 Big Data

There’s much robots can achieve by observing human demonstrations, like the actions necessary to move a box of crackers from a counter to storage. But imitation learning is by no means a perfect science — demonstrators often complete subgoals that distract systems from overarching tasks.

To solve this, researchers at the University of Washington, Stanford University, the University of Illinois Urbana-Champaign, the University of Toronto, and Nvidia propose an “inverse planning” system that taps motions or low-level trajectories to capture the intention of actions. After evaluating their technique by collecting and testing against a corpus of video demonstrations conditioned on a set of kitchen goals, the team reports that their motion reasoning approach improves task success by over 20%.

The researchers lay out the full extent of the problem in a preprint paper detailing their work. In an environment like, say, a cluttered kitchen, they note that objects are configured in such a way that the goal is obfuscated. Recognizing an action sequence isn’t enough, because a task could have myriad motivations. For example, a demonstrator might move a tablecloth both to remove it from view and reach a knife underneath it.

The researchers’ AI system, then, outputs the symbolic goal of a task given a real-world video demonstration, which can then be used as input for robotics systems to reproduce said task. To test it, they had it learn a 24-task cooking objective where a human cook poured and prepped ingredients — tomato soup and spam — which were initially blocked by three objects, including a cracker box, a mustard bottle, and a sugar box. They recorded a total of four demonstrations for each task, resulting in a total of 96 demonstrations (excluding videos with substantial missing poses), and then they divided the tasks in two — 12 for system training and 12 for testing.

The researchers say that their full model explicitly performed motion reasoning about the objects in the demonstration, and thus wouldn’t blindly take all the object movements as intentional. Additionally, they note that it enabled imitation learning across different environments. In one experiment, the system managed to successfully extract the correct goal despite the manipulation of an object (the aforementioned sugar box). Although the sugar box appeared in the kitchen, the robot recognized it didn’t need to move it because it was already out of the way.

“Our results show that this allows us to significantly outperform previous approaches that aim to infer the goal based on either just motion planning or task planning,” wrote the coauthors. “In addition, we show that our goal-based formulation enables the robot to reproduce the same goal in a real kitchen by just watching the video demonstration from a mockup kitchen.”

Let’s block ads! (Why?)

Big Data – VentureBeat

Researchers tout AI that can predict 25 video frames into the future

November 8, 2019 Big Data

AI and machine learning algorithms are becoming increasingly good at predicting next actions in videos. The very best can anticipate fairly accurately where a baseball might travel after it’s been pitched, or the appearance of a road miles from a starting position. To this end, a novel approach proposed by researchers at Google, the University of Michigan, and Adobe advances the state of the art with large-scale models that generate high-quality videos from only a few frames. All the more impressive, it does so without relying on techniques like optical flows (the pattern of apparent motion of objects, surfaces or edges in a scene) or landmarks, unlike previous methods.

“In this work, we investigate whether we can achieve high quality video predictions … by just maximizing the capacity of a standard neural network,” wrote the researchers in a preprint paper describing their work. “To the best of our knowledge, this work is the first to perform a thorough investigation on the effect of capacity increases for video prediction.”

The team’s baseline model builds on an existing stochastic video generation (SVG) architecture, with a component that models the inherent uncertainty in future predictions. They separately trained and tested several versions of the model against data sets tailored to three prediction categories: object interactions, structured motion, and partial observability. For the first task — object interactions — the researchers selected 256 videos from a corpus of videos of robot arm interacting with towels, and for the second — structured motion — they sourced clips from Human 3.6M, a corpus containing clips of humans performing actions like sitting on a chair. As for the partial observability task, they used the open source KITTI driving data set of front car dashboard camera footage.

Researchers tout AI that can predict 25 video frames into the future

Above: The AI model predicts frames given videos from car dash cameras.

Team conditioned every model on between two input to five video frames and had the models predict between five to ten frames into the future during training, at a low resolution (64 by 64 pixels) for all tasks and at both a low and high resolution (128 by 128 pixels) for the objects interactions task During testing, the models generated up to 25 frames.

The researchers report that one of the largest models was preferred 90.2, 98.7%, and 99.3% of the time by evaluators recruited through Amazon Mechanical Turk with respect to the object interactions, structured motion, and partial observability tasks, respectively. Qualitatively, the team notes that it crisply depicted human arms and legs and made “very sharp predictions that looked realistic in comparison to the ground truth.

Above: The AI model predicts frames given videos of human activity.

“Our experiments confirm the importance of recurrent connections and modeling stochasticity [or randomness] in the presence of uncertainty (e.g., videos with unknown action or control),” wrote the paper’s coauthors. “We also find that maximizing the capacity of such models improves the quality of video prediction. We hope our work encourages the field to push along similar directions in the future – i.e., to see how far we can get … for achieving high quality video prediction.”

Let’s block ads! (Why?)

Big Data – VentureBeat

Amazon researchers reduce data required for AI transfer learning

October 28, 2019 Big Data

Cross-lingual learning is an AI technique involving training a natural language processing model in one language and retraining it in another. It’s been demonstrated that retrained models can outperform those trained from scratch in the second language, which is likely why researchers at Amazon’s Alexa division are investing outsize time investigating them.

In a paper scheduled to be presented at this year’s Conference on Empirical Methods in Natural Language Processing, two scientists at the Alexa AI natural understanding group — Quynh Do and Judith Gaspers — and colleagues propose a data selection technique that halves the amount of required training data. They claim that it surprisingly improves rather than compromises the model’s overall performance in the target language.

“Sometimes the data in the source language is so abundant that using all of it to train a transfer model would be impractically time consuming,” wrote Do and Gaspers in a blog post. “Moreover, linguistic differences between source and target languages mean that pruning the training data in the source language, so that its statistical patterns better match those of the target language, can actually improve the performance of the transferred model.”

In the course of experiments, Do, Gaspers, and team employed two methods to cut the source-language data set in half: the aforementioned data selection technique and random sampling. They pretrained separate models on the two halved data sets and on the full data set, after which they fine-tuned the models on a small data set in the target language

Do and Gaspers note that all of the models were trained simultaneously to recognize intents (requested actions) and fill slots (variables on which the intent acts), and that they took as inputs multilingual embeddings (a word or sequences of words from different languages mapped to a single point in a multidimensional space) to bolster model accuracy. The team combined the multilingual embedding of each input word with a character-level embedding that encoded information about words’ prefixes, suffixes, and roots, and they tapped language models trained on large text corpora to select the source-language data that’d be fed to the transfer model.

Within the system the researchers engineered, a bilingual dictionary translated each utterance in the source data set into a string of words in the target language. Four language models were applied to the resulting strings while a trigram model handled character embeddings. And for each utterance in the the sum of probabilities computed by the four language models, only those which yielded the highest normalized score were selected. .

To evaluate their approach, the team first transferred a model from English to German with different amounts of training data in the target language (10,000 and 20,000 utterances, respectively, versus millions of utterances in the full source-language data set). Then, they trained the transfer model on three different languages — English, German, and Spanish — before transferring it to French (with 10,000 and 20,000 utterances in the target language). They claim that the transfer models outperformed a baseline model trained only on data in the target language, and that relative to the model trained on the target language alone, the model trained using the novel data selection technique showed improvements of 3% to 5% on the slot-filling task and about 1% to 2% on intent classification.

Let’s block ads! (Why?)

Big Data – VentureBeat

Apple researchers develop AI framework that leverages users’ intents to automatically label data

September 23, 2019 Big Data

Personal assistants like Apple’s Siri accomplish tasks through natural language commands. However, their underlying components often rely on supervised machine learning algorithms requiring large amounts of hand-annotated training data. In an attempt to reduce the time and effort taken to collect this data, researchers at Apple developed a framework that leverages user engagement signals to automatically create data-augmenting labels. They report that when incorporated using strategies like multi-task learning and validation with an external knowledge base, the annotated data significantly improve accuracy in a production deep learning system.

“We believe this is the first use of user engagement signals to help generate training data for a sequence labeling task on a large scale, and can be applied in practical settings to speed up new feature deployment when little human-annotated data is available,” wrote the researchers in a preprint paper. “Moreover … user engagement signals can help us to identify where the digital assistant needs improvement by learning from its own mistakes.”

The researchers used a range of heuristics to identify behaviors indicating either positive or negative engagement. A few included tapping on content to engage with it further (a positive response), listening to a song for a long duration (another positive response), or interrupting content provided by an intelligent assistant and manually selecting different content (a negative response). Those signals were selectively harvested in a “privacy-preserving manner” to automatically produce ground truth annotations, and they were subsequently combined with coarse-grained labels provided by human annotators.

In order to incorporate the coarse-grained labels and the inferred fine-grained labels into an AI model, the paper’s coauthors devised a multi-task learning framework that treats coarse-grained and fine-grained entity labeling as two tasks. Additionally, they incorporated an external knowledge base validator consisting of entities and their relations. Given the prediction “something” as a music title and “the Beatles” as a music artist for the query “Play something by the Beatles,” the validator would perform a lookup for the top label alternatives and send them to a component that’d re-rank the predictions and return the best alternative.

The researchers conducted two separate test sets to evaluate the tasks performed by the multi-task model, which they compiled by randomly sampling from the production system and hand-annotating with ground truth labels. They say that across 21 model runs, adding 260,000 training examples “consistently” reduced the coarse-grained entity error rate on a prediction task compared with the baseline for all amounts of human-annotated data. Moreover, they report that adding weakly supervised fine-grained data had a larger impact when there was a relatively small amount of human-annotated data (5,000 examples). Lastly, they report that for examples where any of the top model hypotheses passed the knowledge base validator, the fine-grained entity error rate dropped by around 50%.

In another experiment, the team sought to determine whether more granular representations of the user’s intent would increase the likelihood of the system selecting the correct action. They sampled roughly 5,000 “play music” commands containing references to multiple bands, artists, and songs and sent them through a system incorporating their framework, after which they asked annotators to grade the response returned by the system as “satisfactory” or “unsatisfactory.” The results produced by the enhanced system achieved a relative task error rate reduction of 24.64%, the researchers report.

They leave to future work exploring using individual users’ engagement behaviors to improve personalization.

“We observe that our model improves user-facing results especially for requests that contain difficult or unusual language patterns,” wrote the coauthors. “For example, the enhanced system correctly handles queries such as ‘Can you play Malibu from Miley Cyrus new album’ and ‘Play Humble through my music Kendrick Lamar.’ Also, the enhanced model identifies entities that users are more likely to refer to in cases of genuine linguistic ambiguity. For example, in ‘Play one by Metallica,’ ‘one’ could either be a non-entity token (meaning play any song by Metallica), or it refer specifically to the song called ‘One’ by ‘Metallica.’ Since most users listen to the song ‘One’ by the ‘Metallica’ whenever they say ‘Play one by Metallica,’ our model trained on engagement-annotated data will learn to predict ‘one’ as [the music title], thus better capturing trends and preferences in our user population.”

The work comes on the heels of a paper describing Apple’s Overton, an AI development tool whose models have processed ‘billions’ of queries. Separately, the Cupertino company recently studied whether users preferred conversing with “chattier” AI assistants.

Let’s block ads! (Why?)

Big Data – VentureBeat

Tag Archives: researchers

Google’s ML-fairness-gym lets researchers study the long-term effects of AI’s decisions

SenseTime researchers create a benchmark to test face forgery detectors

Carnegie Mellon researchers propose AI that surfaces positive online comments

Intel researchers propose AI that recognizes faces from thermal images

Researchers detail AI that de-hazes and colorizes underwater photos

Researchers want to use Mega Man 2 to evaluate AI

Researchers teach robots to use inference to complete complex tasks

Researchers tout AI that can predict 25 video frames into the future

Amazon researchers reduce data required for AI transfer learning

Apple researchers develop AI framework that leverages users’ intents to automatically label data

Recent Posts

Categories

Archives