Tongtian Zhu

Hi there 😃. I am Tongtian Zhu (prounced as “Tong-tee-an Joo”), a PhD student at the Computer Science Department of Zhejiang University (ZJU), supervised by Professors Can Wang and Chun Chen. I was a research intern at JD Explore Academy (JDE), working on the generalization theory of decentralized learning, mentored by Doctor Fengxiang He and Professor Dacheng Tao. My current research mainly focuses on science of deep learning. I am also interested in utilizing elegant theoretical insights to develop fast and generalizable deep learning algorithms.

Google Scholar Semantic Scholar Email WeChat GitHub

🦄 Research Interest

“Seek truth and beauty.”

In the science aspect, my primary research interest lies in understanding the mechanisms underpining the practical success of deep learning. In the technology aspect, I am interested in developing machine learning methodologies inspired by principles.

My current research focus includes:

Science of deep learning (e.g., symmetries in neural networks’ weight space, training dynamics of gradient methods);
Theory of decentralized learning (e.g., generalization, privacy);
Efficient & trustworthy algorithms for decentralized learning.

Please always feel free to contact (via email or WeChat) if you have any questions, are interested in collaborating, or just want to chat.

🔥 News

Jul 20, 2024	Very excited to attend ICML this time 😀! Feel free to reach out if you want to chat ☕️, whether it’s about deep learning theory (generalization, symmetries, training dynamics), fully decentralized learning (decentralized economies, privacy), or anything else.✨
Jul 12, 2023	I give a long talk (video, slides) at Beijing Academy of Artificial Intelligence (智源社区, BAAI) about our DSGD-SAM paper!
Jul 11, 2023	I give a long talk at the Deep Learning and Optimization Seminar in Westlake University on the generalization benefits of decentralization!
Jul 8, 2023	I give a talk at the Sixth World Artificial Intelligence Conference (第六届世界人工智能大会, WAIC) on the generalization benefits of decentralization!
Apr 25, 2023	Our paper “Decentralized SGD and Average-direction SAM are Asymptotically Equivalent” is accpeted at ICML 2023!
Feb 11, 2023	Our paper “Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition” is accpeted at AAAI 2023 (Oral)!
May 17, 2022	Our paper “Topology-aware Generalization of Decentralized SGD” is accepted at ICML 2022 (Spotlight)!

📝 Selected Publications

ICML 2022 (Spotlight)
Topology-aware Generalization of Decentralized SGD

Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao

In Proceedings of the 39th International Conference on Machine Learning, 17–23 jul 2022

My Comment Abs Bib PDF Slides Poster Code

This paper studies the algorithmic stability and generalizability of decentralized stochastic gradient descent (D-SGD). We prove that the consensus model learned by D-SGD is \mathcalO(N^-1+ m^-1 +λ^2)\mathcalO(N^-1+ m^-1 +λ^2)-stable in expectation in the non-convex non-smooth setting, where NN is the total sample size, mm is the worker number, and 1+\lambda1+λis the spectral gap that measures the connectivity of the communication topology. These results then deliver an \mathcalO(N^-(1+α)/2+ m^-(1+α)/2+λ^1+α + \phi_S)\mathcalO(N^-(1+α)/2+ m^-(1+α)/2+λ^1+α + \phi_S) in-average generalization bound, which is non-vacuous even when λλis closed to 11, in contrast to vacuous as suggested by existing literature on the projected version of D-SGD. Our theory indicates that the generalizability of D-SGD is positively correlated with the spectral gap, and can explain why consensus control in initial training phase can ensure better generalization. Experiments of VGG-11 and ResNet-18 on CIFAR-10, CIFAR-100 and Tiny-ImageNet justify our theory. To our best knowledge, this is the first work on the topology-aware generalization of vanilla D-SGD.

One-sentence summary: The first work on the topology-aware generalization analysis of vanilla Decentralized SGD algorithm.
@inproceedings{pmlr-v162-zhu22d, title = {Topology-aware Generalization of Decentralized {SGD}}, author = {Zhu, Tongtian and He, Fengxiang and Zhang, Lan and Niu, Zhengyang and Song, Mingli and Tao, Dacheng}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {27479--27503}, year = {2022}, month = {17--23 Jul}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, }
ICML 2023 (My Favorite)
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent

Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao

In Proceedings of the 40th International Conference on Machine Learning, 23–29 jul 2023

My Comment Abs Bib PDF Slides Poster Code

Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive device simultaneously without the control of a central server. Sharpness-aware minimization, or SAM, is a popular optimization technique that effectively improves model generalization by explicitly minimizing a sharpness-based measure alongside the training loss. In this paper, we prove that D-SGD asymptotically minimizes the loss function of an average-direction SAM. This asymptotic equivalence further demonstrates three advantages of D-SGD: (1) D-SGD exhibits a gradient smoothing effect; (2) there exists an uncertainty self-estimating mechanism in D-SGD to improve posterior estimation; and (3) the sharpness regularization effect of D-SGD does not decrease as total batch size increases, which justifies the superiority of D-SGD over centralized SGD (C-SGD) in large-batch settings. We conduct extensive experiments which are in full agreement with our theory. Our code will be made publicly available.

TLDR: The first work on the surprising sharpness-aware minimization nature (i.e., a kind of unique implicit bias) of decentralized learning. We provide a completely new perspective to understand model decentralization, which helps to bridge the gap between practice and exisiting theory in decentralized learning.
@inproceedings{pmlr-zhu23, title = {Decentralized SGD and Average-direction SAM are Asymptotically Equivalent}, author = {Zhu, Tongtian and He, Fengxiang and Chen, Kaixuan and Song, Mingli and Tao, Dacheng}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, year = {2023}, month = {23--29 Jul}, pages = {43005--43036}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, }

📚 Academic Services

Conference Reviewer: ICLR 2024; ICML 2022, 2023, 2024; NeurIPS 2022, 2023; AISTATS 2023, 2024; UAI 2023, 2024; ECML 2023; ACML 2022.
I am also a contributor of International Seminar on Foundational Artificial Intelligence (FAI-Seminar).

🤗 Miscellaneous

I am an (old-school) ACG fan. My favorites range from Detective Conan and Pokemon to Attack on Titan.
I was an amateur artist (or more accurately, an enthusiast), spending more than five years practicing my sketching skills. My dream was to become a manga artist ("漫画家" in Chinese). I firmly believe that he aesthetic sensibility nurtured through my fine arts studies considerably enhances my research.
I'm also a huge basketball fanatic! I have a profound love for basketball, particularly for the intrinsic beauty of the sport itself. I'm planning (possibly in 1 or 2 years) to start a blog series about the basics of basketball (e.g., rules, skills and tactics) through the lens of scientific research. Any comments are welcome!