DEEP LEARNING IN THEORY AND PRACTICE LAB

mit.jpeg
Department of Computer Science

COM2-03-32
13 Computing Drive
Singapore 117417

JOIN OUR LAB

We are always looking for PhD students, postdocs and visiting students to join our lab! For more information please visit our recruitment page here.

ABOUT

theory-practice.png
What is intelligence? Can we understand it? The deep learning approach for this classical question is to build an intelligent machine with the natural yet very flexible prior with powerful computational tools. An idea is that the ability to build an intelligent machine is correlated with our understanding of intelligence. Therefore, if we work towards building an intelligent machine, we are closer to understanding intelligence. However, we hypothesize that the ability to build an intelligent machine is not sufficient for understanding intelligence. We could build a black-box of an intelligent machine without really understanding it by tremendous efforts of trial and error. Therefore, we hypothesize that we need mathematical theories for what we are building in the quest for intelligence.
​​​​​​​
Our lab aims to establish the positive feedback loop between theory and practice, to accelerate the development of the practical deep learning methods and to contribute to the understanding of intelligence.

Selected Publications

​​​​​​​(* indicates equal contribution)

Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang and Kenji Kawaguchi. Self-Distillation for Further Pre-training of Transformers. In International Conference on Learning Representations (ICLR), 2023.
[pdf] [BibTeX]

Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi and Aaron Courville. Simplicial Embeddings in Self-Supervised Learning and Downstream Classification. In International Conference on Learning Representations (ICLR), 2023.
[pdf] [BibTeX] [notable-top-25%]

Tianbo Li, Min Lin, Zheyuan Hu, Kunhao Zheng, Giovanni Vignale, Kenji Kawaguchi, A.H. Castro Neto, Kostya S. Novoselov and Shuicheng YAN. D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory. In International Conference on Learning Representations (ICLR), 2023.
[pdf] [BibTeX] [notable-top-25%]

Dong Bok Lee, Seanie Lee, Kenji Kawaguchi, Yunji Kim, Jihwan Bang, Jung-Woo Ha and Sung Ju Hwang. Self-Supervised Set Representation Learning for Unsupervised Meta-Learning. In International Conference on Learning Representations (ICLR), 2023.
[pdf] [BibTeX]

Kenji Kawaguchi, Zhun Deng, Kyle Luh, Jiaoyang Huang. Robustness Implies Generalization via Data-Dependent Generalization Bounds. International Conference on Machine Learning (ICML), 2022.
[pdf] [BibTeX] Selected for ICML long presentation (2% accept rate)

Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, Ethan Fetaya. Multi-Task Learning as a Bargaining Game. International Conference on Machine Learning (ICML), 2022.
[pdf] [BibTeX]

Linjun Zhang*, Zhun Deng*, Kenji Kawaguchi, James Zou. When and How Mixup Improves Calibration. International Conference on Machine Learning (ICML), 2022.
[pdf] [BibTeX]

Juncheng Liu, Bryan Hooi, Kenji Kawaguchi and Xiaokui Xiao. MGNNI: Multiscale Graph Neural Networks with Implicit Layers. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[pdf] [BibTeX]

Riashat Islam, Hongyu Zang, Anirudh Goyal, Alex Lamb, Kenji Kawaguchi, Xin Li, Romain Laroche, Yoshua Bengio and Remi Tachet des Combes. Discrete Compositional Representations as an Abstraction for Goal Conditioned Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[pdf] [BibTeX]

Seanie Lee*, Bruno Andreis*, Kenji Kawaguchi, Juho Lee and Sung Ju Hwang. Set-based Meta-Interpolation for Few-Task Meta-Learning. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[pdf] [BibTeX]

Zheyuan Hu, Ameya Jagtap, George Em Karniadakis and Kenji Kawaguchi. When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?. SIAM Journal on Scientific Computing, 44 (5), pp. A3158-A3182, 2022.
[pdf] [BibTeX]

Kenji Kawaguchi. On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers. In International Conference on Learning Representations (ICLR), 2021.
[pdf] [BibTeX] Selected for ICLR Spotlight (5% accept rate)

Linjun Zhang*, Zhun Deng*, Kenji Kawaguchi*, Amirata Ghorbani and James Zou. How Does Mixup Help With Robustness and Generalization? In International Conference on Learning Representations (ICLR), 2021.
[pdf] [BibTeX] Selected for ICLR Spotlight (5% accept rate)

Keyulu Xu*, Mozhi Zhang, Stefanie Jegelka and Kenji Kawaguchi*. Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth. International Conference on Machine Learning (ICML), 2021.
[pdf] [BibTeX]

Vikas Verma, Minh-Thang Luong, Kenji Kawaguchi, Hieu Pham and Quoc V Le. Towards Domain-Agnostic Contrastive Learning. International Conference on Machine Learning (ICML), 2021.
[pdf] [BibTeX]

Dianbo Liu*, Alex Lamb*, Kenji Kawaguchi, Anirudh Goyal, Chen Sun, Michael Curtis Mozer and Yoshua Bengio. Discrete-Valued Neural Communication. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[pdf] [BibTeX]

Ferran Alet*, Dylan Doblar*, Allan Zhou, Joshua B. Tenenbaum, Kenji Kawaguchi and Chelsea Finn. Noether Networks: meta-learning useful conserved quantities. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[pdf] [BibTeX]

Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi and James Zou. Adversarial Training Helps Transfer Learning via Better Representations. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[pdf] [BibTeX]

Clement Gehring, Kenji Kawaguchi, Jiaoyang Huang and Leslie Pack Kaelbling. Understanding End-to-End Model-Based Reinforcement Learning Methods as Implicit Parameterization. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[pdf] [BibTeX]

Ferran Alet, Maria Bauza Villalonga, Kenji Kawaguchi, Nurullah Giray Kuru, Tomás Lozano-Pérez and Leslie Pack Kaelbling. Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[pdf] [BibTeX]

Juncheng Liu, Kenji Kawaguchi, Bryan Hooi, Yiwei Wang and Xiaokui Xiao. EIGNN: Efficient Infinite-Depth Graph Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[pdf] [BibTeX]

Ameya D. Jagtap, Kenji Kawaguchi and George E. Karniadakis. Adaptive Activation Functions Accelerate Convergence in Deep and Physics-informed Neural Networks. Journal of Computational Physics, 404, 109136, 2020.
[pdf] [BibTeX]

Ameya D. Jagtap*, Kenji Kawaguchi* and George E. Karniadakis. Locally adaptive activation functions with slope recovery for deep and physics-informed neural networks. Proceedings of the Royal Society A, 476, 20200334, 2020.
[pdf] [BibTeX]

Kenji Kawaguchi. Deep Learning without Poor Local Minima. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[pdf] [BibTeX] [Spotlight Video] [TalkSelected for NeurIPS oral presentation (2% accept rate)

​​​​​​​Kenji Kawaguchi, Leslie Pack Kaelbling and Tomás Lozano-Pérez. Bayesian Optimization with Exponential Convergence. Advances in Neural Information Processing Systems (NeurIPS), 2015.
[pdf] [BibTeX] [Code]

MEMBERS

Principal Investigator (PI)

Kenji Kawaguchi is a Presidential Young Professor in the Department of Computer Science.
​​​​​​​
​​​​​​​ Kenji Kawaguchi received his Ph.D. in Computer Science from Massachusetts Institute of Technology (MIT). He then joined Harvard University as a postdoctoral fellow. He was also an invited participant at the University of Cambridge, Isaac Newton Institute for Mathematical Sciences program on "Mathematics of Deep Learning". He was one of 77 invited participants from around the world.

His research interests include deep learning as well as artificial intelligence (AI) in general. His research group aims to have a positive feedback loop between theory and practice in deep learning research through collaborations with researchers from both practice and theory sides.

PhD Students

Kin Whye Chew
Zekun Shi
Brian Chen
(co-advised)
Zhiyuan Liu
(co-advised)
Martin Frederik Schmitz
(co-advised)
Haonan Wang
(co-advised)
Depeng Li
(visiting student)

Interns

Yuzhen Mao
Zhiwei Zhang
Savya Khosla

Administrative Assistant

Phang Kenneth