• Home
  • About Us
  • Contact Us
  • Privacy Policy
  • Special Offers
Business Intelligence Info
  • Business Intelligence
    • BI News and Info
    • Big Data
    • Mobile and Cloud
    • Self-Service BI
  • CRM
    • CRM News and Info
    • InfusionSoft
    • Microsoft Dynamics CRM
    • NetSuite
    • OnContact
    • Salesforce
    • Workbooks
  • Data Mining
    • Pentaho
    • Sisense
    • Tableau
    • TIBCO Spotfire
  • Data Warehousing
    • DWH News and Info
    • IBM DB2
    • Microsoft SQL Server
    • Oracle
    • Teradata
  • Predictive Analytics
    • FICO
    • KNIME
    • Mathematica
    • Matlab
    • Minitab
    • RapidMiner
    • Revolution
    • SAP
    • SAS/SPSS
  • Humor

Researchers propose data set to measure few-shot learning performance

February 20, 2020   Big Data
 Researchers propose data set to measure few shot learning performance

A plethora of AI models have been tailored to tackle few-shot classification, which refers to learning a classifier for new classes given only a few examples (an ability humans naturally possess). Improving on it could lead to more efficient algorithms capable of expanding their knowledge without requiring large labeled data sets, but to date, many of the procedures and corpora used to assess progress here are lacking.

That’s why researchers at Google AI, the University of California, Berkeley, and the University of Toronto propose in a preprint paper a benchmark for training and evaluating large-scale, diverse, and more “realistic” few-shot classification models. They say it improves upon previous approaches by incorporating multiple data sets of “diverse” distributions and by introducing realistic class imbalance, which they say allows the testing of robustness across a spectrum from low-shot learning onward.

The work was published in May 2019, but it was recently accepted to the International Conference on Learning Representations (ICLR) that’ll be held in Addis Ababa, Ethiopia in April.

As the team explains, as opposed to synthetic environments, real-life learning experiences are heterogeneous in that they vary by the number of classes and the examples per class. They also measure only within-corpus generalization, and they ignore the relationships between classes when forming episodes — i.e., the coarse-grained classification of dogs and chairs may present different difficulties than the fine-grained classification of dog breeds. (An “episode” encompasses states that come between an initial-state and a terminal-state, such as a game of Chess.)

By contrast, the researchers’ data set — the Meta-Dataset — leverages data from 10 different corpora, which span a variety of visual concepts natural and human-made and vary in the specificity of the class definition. Two are reserved for evaluation, meaning that no two classes from them participate in the training set, while the remaining ones contribute some classes to each of the training, validation, and test splits of classes.

Meta-Dataset separately employs an algorithm for sampling episodes, which aims to yield imbalanced episodes of variable shots (class precision) and ways (accuracy). A prescribed number of examples of each chosen class are chosen uniformly at random to populate the support and query sets.

In experiments, the team trained meta-learning models via training episodes sampled using the same algorithm as used for Meta-Dataset’s evaluation episodes. They say that, tested against Meta-Dataset, the models generally didn’t improve when provided multiple data sources and that they didn’t benefit from meta-learning across the data sets. Moreover, they report that the models weren’t robust to the amount of data in test episodes; rather, each excelled in a different part of the spectrum.

“We believe that our exploration of various models on Meta-Dataset has uncovered interesting directions for future work pertaining to meta-learning across heterogeneous data,” wrote the coauthors, who added that addressing the uncovered shortcomings constitutes an important research goal. “[I]t remains unclear what is the best strategy for creating training episodes, the most appropriate validation creation and the most appropriate initialization.”

Let’s block ads! (Why?)

Big Data – VentureBeat

data, fewshot, Learning, Measure, Performance, Propose, researchers
  • Recent Posts

    • Building AI for the Global South
    • Dapper Duo
    • AI Weekly: These researchers are improving AI’s ability to understand different accents
    • Why Choose RapidMiner for Your Data Science & Machine Learning Software?
    • How to Use CRM Integration to Your Advantage – Real World Examples
  • Categories

  • Archives

    • March 2021
    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • March 2020
    • February 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    • December 2016
    • November 2016
    • October 2016
    • September 2016
    • August 2016
    • July 2016
    • June 2016
    • May 2016
    • April 2016
    • March 2016
    • February 2016
    • January 2016
    • December 2015
    • November 2015
    • October 2015
    • September 2015
    • August 2015
    • July 2015
    • June 2015
    • May 2015
    • April 2015
    • March 2015
    • February 2015
    • January 2015
    • December 2014
    • November 2014
© 2021 Business Intelligence Info
Power BI Training | G Com Solutions Limited