New Frontiers for Learning with Limited Labels or Data

ECCV 2020 Tutorial, Glasgow, Scotland, UK

Sunday, August 23rd, 2020
Location: Room M1

Photo: Pixabay


Learning from limited data or supervision has garnered much research interest and many ideas to solve it are now well-established. Among them, learning from 1) partial or imperfect labels and 2) data from other related domains (synthetic data), modalities (text/audio/3D) or tasks have been extensively addressed in past tutorials/workshops. To improve network generalization when learning with limited data, the ideas of zero-, few-shot, meta- and task-transfer learning are ubiquitous. So, what's next? Are there new sub-problems that we have not yet investigated? Are there recent breakthroughs in other areas of vision and learning that can help us to better tackle this problem? Can we enable new applications?

To foster discussion and inspire new ideas along these lines, in this tutorial we will focus on new frontiers of learning with limited labels or data. We will explore emerging sub-problems, such as 1) omni-supervision, which is creating unified frameworks for learning from the entire spectrum of supervision -- with completely unsupervised, semi/weakly-supervised, to full-supervised data and 2) deep dreaming, which involves effectively inverting networks to generate training data. We will present some recent breakthroughs in other vision problems including, but not limited to, neural rendering; biologically-inspired self-learning and self-supervised image representation learning by exploiting video correspondences and 3D mesh reconstruction that can help to solve this problem. Finally, we will present some new applications including self-supervisedly learned discovery of visual object attributes.


# Time Item
8:00am - 8:15am Welcome and Opening Remarks, Shalini De Mello (NVIDIA)
1 8:15am - 8:50am Biologically Inspired Self-Learning, Animashree Anandkumar (NVIDIA, Caltech)
2 8:50am - 9:25am Inverting Neural Networks for Data-free Knowledge Transfer, Pavlo Molchanov (NVIDIA)
2 9:25am - 10:00am Limitless Labels in a Labelless World: Weak Supervision with Noisy Labels, Arash Vahdat (NVIDIA)
10:00am - 10:10am Coffee Break
3 10:10am - 10:45am Object Attribute Discovery from Image Collections, Varun Jampani (Google Research)
4 10:45am - 11:20am Self-supervised Learning for Video Correspondences and 3D Mesh Reconstruction, Sifei Liu (NVIDIA)
5 11:20am - 11:55pm Omni-supervised Learning: Unifying Levels of Supervision, Zhiding Yu (NVIDIA)
11:55pm - 12:00pm Closing Remarks, Shalini De Mello (NVIDIA)



Shalini De Mello is a Principal Research Scientist at NVIDIA. Her research interests are in computer vision and machine learning for human-computer interaction and smart interfaces. At NVIDIA, she has invented technologies for gaze estimation, and 2D and 3D head pose estimation, hand gesture recognition, face detection, video stabilization and GPU-optimized libraries for mobile computer vision. Her research over that past several years has pushed the envelope of HCI in cars and has led to the development of NVIDIA’s innovative DriveIX product for smart AI-based automotive interfaces for future generations of cars. She received doctoral and master’s degrees in Electrical and Computer Engineering from the University of Texas at Austin in 2008 and 2004, respectively.

Sifei Liu

Sifei Liu is a senior research scientist at Learning and Perception Research (LPR) at NVIDIA. She obtained her Ph.D. at the University of California Merced, department of EECS, advised by Professor Ming-Hsuan Yang. Her research interests are in computer vision (low-level vision, semantic segmentation and 3D scene understanding), deep learning (graph-structured modules, self-supervised learning), and the combination of both. She worked as an intern student in Baidu IDL from 2013 to 2014, multimedia lab in CUHK in 2015, and NVIDIA research in 2017. She is also the recipient of the Baidu Graduate Fellowship in 2013 and the Rising Star EECS in 2019. She has co-organized the Learning Representations via Graph-structured Networks tutorial in conjunction with CVPR 2019.

Zhiding Yu

Zhiding Yu is a Research Scientist with ML Research at NVIDIA. He obtained Ph.D. in ECE from Carnegie Mellon University in 2017, and M.Phil. in ECE from The Hong Kong University of Science and Technology in 2012. His research interests mainly focus on deep representation learning, weakly/semi-supervised learning, transfer learning and deep structured prediction, with their applications to vision and robotics problems. His research goal is to leverage the abundant domain knowledge, priors and structured information to eliminate uncertainties with minimum human supervision and uncover the secret towards true visual intelligence. He has conducted research internships at Adobe, Microsoft Research and Mitsubishi Electric Research Laboratories. He is the winner of the Domain Adaptation for Semantic Segmentation Track, WAD Challenge at CVPR 2018.


Pavlo Molchanov obtained his PhD from the Tampere University of Technology, Finland in the area of signal processing in 2014. His dissertation was focused on designing automatic target recognition systems for radars. Since 2015 he is with the Learning and Perception Research team at NVIDIA, currently holding a senior research scientist position. His research is focused on methods for neural network acceleration, and designing novel systems for human-computer interaction and human understanding. For network acceleration, he is interested in neural network pruning methods and conditional inference. For human understanding, he is working on landmark estimation, gesture recognition, hand pose estimation. He received the EuRAD best paper award in 2011 and EuRAD young engineer award in 2013.

Varun Jampani
Google Research

Varun Jampani is a Research Scientist at Google Research in Cambridge, US. Prior to that, he was a Research Scientist at NVIDIA Research. He works in the areas of machine learning and computer vision and his main research interests include content-adaptive neural networks, self-supervised visual discovery, motion understanding and novel view synthesis. He obtained his Ph.D. with highest honors at Max Planck Institute for Intelligent Systems (MPI) in 2017. He obtained his BTech and MS from International Institute of Information Technology, Hyderabad (IIIT-H), India, where he was a gold medalist. His work on “SplatNet” received the Best Paper Honorable Mention award at CVPR 2018.


Arash Vahdat is a senior research scientist at NVIDIA research specializing in machine learning and computer vision. Before joining NVIDIA, he was a research scientist at D-Wave Systems where he worked on weakly supervised learning from noisy labeled data and deep generative learning with discrete latent variable models. Prior to D-Wave, Arash was a PhD student under Greg Mori’s supervision working on latent variable frameworks for visual analysis, then a research faculty member at Simon Fraser University, where he led several research projects at the intersection of deep learning and video analysis. Arash completed his undergrad in computer engineering at the Sharif University of Technology. His area of research includes weakly-supervised learning, neural architecture search, probabilistic deep learning, and generative learning. More information regarding Arash's background can be found on his personal website.

Animashree Anandkumar
NVIDIA and California Institute of Technology

Anima Anandkumar is the Director of ML Research at NVIDIA and Bren Professor at Caltech. She was previously a Principal Scientist at Amazon Web Services. She has received several honors such as Alfred. P. Sloan Fellowship, NSF Career Award, Young investigator awards from DoD, and Faculty Fellowships from Microsoft, Google and Adobe. She is part of the World Economic Forum's Expert Network. She is passionate about designing principled AI algorithms and applying them in interdisciplinary applications. Her research interests include tensor methods, optimization, and large-scale learning.

Jan Kautz

Jan Kautz is VP of Learning and Perception Research at NVIDIA. Jan and his team pursue fundamental research in the areas of computer vision and deep learning, including visual perception, geometric vision, generative models, and efficient deep learning. Before joining NVIDIA in 2013, Jan was a tenured faculty member at University College London. He holds a BSc in Computer Science from the University of Erlangen-Nürnberg (1999), an MMath from the University of Waterloo (1999), received his PhD from the Max-Planck-Institut für Informatik (2003), and worked as a post-doctoral researcher at the Massachusetts Institute of Technology (2003-2006).