BeyondLabeler - Human is More Than a Labeler

Can we better classify objects in images with labeled examples plus a textual description of the object per image? Or will it be also better with labeled examples plus a collection of rules that can be used to prove whether or not an object is a member of a particular class? Or how about improving the performance of learning systems via user interactions? The main purpose of our proposed ``First Workshop on Human is More Than a Labeler (BeyondLabeler)'' is to bring together researchers and practitioners who are working to instill more than label annotations into their machine learning algorithms.

Learning in those scenarios has been conceptualized as far back as 1990s and as recent as 2009. Some examples include knowledge-based learning 1,2,3,4, learning using privileged information (LUPI)5,6,7, learning with hints 8, interactive learning 9, multi-view learning 10, model compression/distillation 16,17,18reinforcement learning with guidance 11, and active and co-active learning for problem solving 12. This learning framework is commonly seen in a variety of application fields including natural language processing 13, computer vision 14, and medical informatics 15, to name a few.

CFP & Dates (We are in touch with prominent ML journals to explore the idea of a special issue on the topic)

The main goal of this workshop is to have discussion on the following non-exhaustive topics:
  • What are the similarities and differences among all existing learning concepts that try to incorporate information beyond label annotations?
  • Is there an opportunity for a hybrid system ensembling state-of-the-art BeyondLabeller learning methods?
  • What are other novel application areas for this learning with labels plus additional information?
  • What if the additional information is wrong or inconsistent with the label annotations?
  • Will random additional information be helpful to improve the performance of the learning system?

    Important Dates & Submission Instructions

  • Submission deadline for papers: April 30, 2016 [Extended]
  • Notification of acceptance: May 25, 2016 [Extended]
  • Workshop day: July 10, 2016
  • We encourage contributions either a technical paper (IJCAI style, 6 pages without references), a position statement (IJCAI style, 2 pages maximum) or a published work related to the workshop, clearly indicating in the paper that this is published work.
  • IJCAI Style files available here.
  • Papers submission is via the Easychair system. Please leave authors information visible.
  • At least one author of each accepted paper is required to attend the workshop to present the work. Authors will be required to agree to this requirement at the time of submission.

    • Invited Speakers

    Vladimir Vapnik

    is a Research Consultant at Facebook AI Research and a Professor at Royal Holloway University and Columbia University. Prior to Facebook, he held positions at NEC Labs (2002-2014), AT&T (1996-2002), Bell Laboratories (1989-1996), and Institute for Control Sciences (1961-1989). He is a member of the National Academy of Engineering since 2006. He has made contributions to the theoretical foundations of machine learning, particularly the notions of capacity and consistency, and he helped develop the Support Vector Machine method. His current research interests include Learning with Intelligent Teacher.

    Rogerio Feris

    is a Research Scientist at IBM T.J. Watson Research Center, New York, and an Affiliate Associate Professor at University of Washington. He received several honors and awards, including a recent IBM Master inventor honor and a prestigious IBM Outstanding Innovation Achievement Award in 2011. In addition to working on core research, he had a one-year assignment at IBM Global Technology Services as a senior software engineer to help the productization of the IBM Smart Surveillance System.

    Michael Littman

    is a Professor of Computer Science at Brown University. His research in machine learning examines algorithms for decision making under uncertainty. He has earned multiple awards for teaching and his research has been recognized with three best-paper awards on the topics of meta-learning for computer crossword solving, complexity analysis of planning under uncertainty, and algorithms for efficient reinforcement learning. Littman has served on the editorial boards for the Journal of Machine Learning Research and the Journal of Artificial Intelligence Research. He was general chair of International Conference on Machine Learning 2013 and program chair of the Association for the Advancement of Artificial Intelligence Conference 2013.

    Rich Caruana

    is a Senior Researcher at Microsoft Research. Before joining Microsoft, Rich was on the faculty in the Computer Science Department at Cornell University, at UCLA's Medical School, and at CMU's Center for Learning and Discovery (CALD). Rich's Ph.D. is from Carnegie Mellon University, where he worked with Tom Mitchell and Herb Simon. His thesis on Multi-Task Learning helped generate interest in a new subfield of machine learning called Transfer Learning. Rich received an NSF CAREER Award in 2004 (for Meta Clustering), best paper awards in 2005 (with Alex Niculescu-Mizil), 2007 (with Daria Sorokina), and 2014 (with Todd Kulesza, Saleema Amershi, Danyel Fisher, and Denis Charles), co-chaired KDD in 2007 (with Xindong Wu), and serves as area chair for NIPS, ICML, and KDD. His current research focus is on learning for medical decision making, deep learning, adaptive clustering, and computational ecology.

    Accepted Papers

    Workshop Schedule

    Room Gibson at New York Hilton Midtown Hotel, 1335 Avenue of the Americas.
    • Morning session: 8:50 - 11:40
    • 8:50 - 9:00 Introduction
    • 9:00 - 9:45 Invited Talk 1: Rich Caruana
      What Has Model Compression Taught us About Teaching?
      Abstract

      In model compression, the trick is to train a small student to mimic a large teacher model. If we can train a small student to accurately mimic the teacher we can then deploy a small, accurate model. The surprise is that this works --- we may not know how to train accurate small models on the original training data, but we are able to train small students to mimic large teachers because the teachers give the students richer information than the original training signal. This tells us about the value of different kinds of labels (e.g., 0/1 hard targets vs. soft probabilistic targets) and other ways cooperative teachers can help students learn better.

    • 9:45 - 10:30 Invited Talk 2: Michael Littman
      Teaching Rewards by Example
      Abstract

      Reinforcement-learning agents can acquire sophisticated behavior autonomously given a reward function and an opportunity to explore. Nevertheless, human expertise, delivered in the form of demonstrations of good behavior, can obviate the need for explicit reward functions and also make the exploration process itself significantly more efficient. In this talk, I will summarize several projects I have contributed to that use human demonstrations to improve machine-learning performance.

    • ===10:30-11:00 Coffee break===
    • 11:00 - 11:40 Spotlight Session (10 x 4mins)
    • ===11:40 - 13:00 Lunch break===
    • Afternoon session: 13:00 - 17:15
    • 13:00 - 13:45 Invited Talk 3: Rogerio Feris
      Representation Learning Beyond Human Labels: Practical Applications in Visual Analysis of People
    • 13:45 - 14:00 Contributed Talk:
      Xinxing Xu, Joey Tianyi Zhou, Ivor W. Tsang, Zheng Qin, Rick Siow Mong Goh and Yong Liu
      Simple and Efficient Learning using Privileged Information
      Abstract

      The Support Vector Machine using Privileged Information (SVM+) has been proposed to train a classifier to utilize the additional privileged information that is only available in the training phase but not available in the test phase. In this work, we propose an efficient solution for SVM+ by simply utilizing the squared hinge loss instead of the hinge loss as in the existing SVM+ formulation, which interestingly leads to a dual form with less variables and in the same form with the dual of the standard SVM. The proposed algorithm is utilized to leverage the additional web knowledge that is only available during training for the image categorization tasks. The extensive experimental results on both Caltech101 and WebQueries datasets for image categorization tasks show that our proposed method can achieve a factor of up to hundred times speedup with the comparable accuracy when compared with the existing SVM+ method.

    • 14:00 - 15:30 Poster Session
    • ===15:30 - 16:00 Coffee break===
    • 16:00 - 16:45 Invited Talk 4: Vladimir Vapnik
      Learning with Intelligent Teacher: Similarity Control and Knowledge Transfer
      Abstract

      In the talk I will introduce model of learning with Intelligent Teacher. In this model Intelligent Teacher supplies (some) training examples with additional (privileged) information forming training triplets. Privileged information is available only for training examples and not available for test examples. Using privileged information it is required to find a better training processes (that use less examples or more accurate with the same number of examples) than the classical ones. In the lecture I will present two additional mechanisms that exist in learning with Intelligent Teacher:
      - The mechanism to control Student’s concept of examples similarity and
      - The mechanism to transfer knowledge that can be obtained in space of privi- leged information to the desired space of decision rules.
      Privileged information exists for any inference problem and Student-Teacher interaction can be considered as the basic element of intelligent behavior.

    • 16:45 - 17:15 Award Session and Closing Remarks

    Organizers

    Program Committee

    • Jude W. Shavlik, University of Wisconsin-Madison
    • Prasad Tadepalli, Oregon State University
    • Bernt Schiele, Max Planck Institute for Informatics
    • Gautam Kunapuli, UtopiaCompression Corporation
    • Christoph Lampert, IST Austria
    • Andrea Passerini, University of Trento
    • Pietro Galliani, University of Sussex
    • Daniel Hernández-Lobato, Universidad Autónoma de Madrid
    • Emilie Morvant, Université Jean Monnet
    • Vijay Badrinarayanan, Magic Leap
    • David Lopez-Paz, University of Cambridge
    • Kshitij Judah, Oregon State University
    • Tushar Khot, Allen Institute for Artificial Intelligence
    • Karthik Raman, Google
    • Phillip Odom, Indiana University
    • Martin Mladenov, TU Dortmund
    • Joseph Taylor, University of Sussex

    Sponsors


    Selected References

    1 M. O. Noordewier, G. G. Towell, and J. W. Shavlik. Training knowledge-based neural networks to recognize genes. NIPS, 1990.
    2 D. W. Opitz and J. W. Shavlik. Heuristically expanding knowledge-based neural networks. IJCAI, 1993.
    3 Q. V. Le, A. J. Smola, and T. Gaertner. Simpler knowledge-based support vector machines. ICML, 2006.
    4 P. Odom, T. Khot, R. Porter, and S. Natarajan. Knowledge-based probabilistic logic learning. AAAI, 2015.
    5 V. Vapnik and A. Vashist. A new learning paradigm: Learning using privileged information. Neural Networks, pp. 544--557, 2009.
    6 D. Hernández-Lobato, V. Sharmanska, K. Kersting, C. Lampert, and N. Quadrianto. Mind the nuisance: Gaussian process classification using privileged noise. NIPS, 2014.
    7 V. Vapnik and R. Izmailov. Learning using privileged information: Similarity control and knowledge transfer. JMLR , pp. 2023--2049, 2015.
    8 Y. S. Abu-Mostafa. A method for learning from hints. NIPS, 1992.
    9 T. Joachims. Learning from user interactions. WSDM, 2015.
    10 C, Xu, D. Tao, and C. Xu. A survey on multi-view learning. CoRR, abs/1304.5634, 2013.
    11 K. Judah, S. Roy, A. Fern, and T. G. Dietterich. Reinforcement learning via practice and critique advice. AAAI , 2010.
    12 K. Judah, A. Fern, and T. G. Dietterich. Active imitation learning via reduction to i.i.d. active learning. UAI, 2012.
    13 E. Gabrilovich and S. Markovitch. Harnessing the expertise of 70, 000 human editors: Knowledge-based feature generation for text categorization. JMLR, pp. 2297--2345, 2007.
    14 V. Sharmanska, N. Quadrianto, and C. H. Lampert. Learning to rank using privileged information. ICCV, 2013.
    15 L. Duan, Y. Xu, W. Li, L. Chen, D. W. K. Wong, T.Y. Wong, and J. Liu. Incorporating privileged genetic information for fundus image based glaucoma detection. MICCAI, 2014.
    16 D. Lopez-Paz, L. Bottou, B. Schölkopf and V. Vapnik. Unifying distillation and privileged information. ICLR, 2016.
    17 L. J. Ba and R. Caruana. Do Deep Nets Really Need to be Deep? NIPS, 2014.
    18 G. Hinton, O. Vinyals, and J. Dean. Distilling the Knowledge in a Neural Network. arXiv, 2014.