Tongtian Zhu

Hi there 😃. I am Tongtian Zhu (prounced as “Tong-tee-an Joo”), a second-year PhD student at the Computer Science Department of Zhejiang University (ZJU), under the auspices of Professors Mingli Song and Chun Chen. I earned my B.S. degree in Mathematical Science from CUMTB, as well as my B.Econ. degree in Economics (double degree) from Peking University (PKU) in 2021. In the same year, I joined JD Explore Academy (JDE) as a a research intern, where I focused on the generalization theory of decentralized learning, fortunately supervised by Professors Fengxiang He and Dacheng Tao.

Google Scholar Twitter Email WeChat Github Zhihu

🦄 Research Interest

In the science aspect, my primary research interest lies in understanding the mechanisms underpining the practical success of deep learning. My ultimate goal is to understand the emergence of intelligence (both machine intelligence and biological intelligence). In the technology aspect, I am passionate about developing machine learning methodologies inspired by principles.

My current research focuses on examining the theoretical foundations of decentralized learning. I am also dedicated to utilizing elegant theoretical insights to construct fast and generalizable decentralized learning algorithms.

Please always feel free to contact (via email or WeChat) if you have any questions, are interested in collaborating, or simply want to chat. For regular updates and insights, feel free to follow my Twitter as well.

🔥 News

Apr 25, 2023	Our paper “Decentralized SGD and Average-direction SAM are Asymptotically Equivalent” is accpeted at ICML 2023!
Feb 11, 2023	Our paper “Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition” is accpeted at AAAI 2023 (Oral)!
Dec 6, 2022	I am awarded as a Merit PhD student/Triple A PhD student of Zhejiang University on 2021-2022 (top 10%)!
May 17, 2022	Our paper “Topology-aware Generalization of Decentralized SGD” is accepted at ICML 2022!

📝 Selected Publications

ICML 2022
Topology-aware Generalization of Decentralized SGD

Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao

In Proceedings of the 39th International Conference on Machine Learning, 17–23 jul 2022

My Comment Abs Bib PDF Slides

This paper studies the algorithmic stability and generalizability of decentralized stochastic gradient descent (D-SGD). We prove that the consensus model learned by D-SGD is \mathcalO(N^-1+ m^-1 +λ^2)-stable in expectation in the non-convex non-smooth setting, where N is the total sample size, m is the worker number, and 1+λis the spectral gap that measures the connectivity of the communication topology. These results then deliver an \mathcalO(N^-(1+α)/2+ m^-(1+α)/2+λ^1+α + \phi_S) in-average generalization bound, which is non-vacuous even when λis closed to 1, in contrast to vacuous as suggested by existing literature on the projected version of D-SGD. Our theory indicates that the generalizability of D-SGD is positively correlated with the spectral gap, and can explain why consensus control in initial training phase can ensure better generalization. Experiments of VGG-11 and ResNet-18 on CIFAR-10, CIFAR-100 and Tiny-ImageNet justify our theory. To our best knowledge, this is the first work on the topology-aware generalization of vanilla D-SGD.

One-sentence summary: The first work on the topology-aware generalization analysis of vanilla D-SGD.
@inproceedings{pmlr-v162-zhu22d, title = {Topology-aware Generalization of Decentralized {SGD}}, author = {Zhu, Tongtian and He, Fengxiang and Zhang, Lan and Niu, Zhengyang and Song, Mingli and Tao, Dacheng}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {27479--27503}, year = {2022}, month = {17--23 Jul}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, }
ICML 2023
Decentralized SGD and Average-direction SAM are Asymptotically Equivalent

Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao

In 40th International Conference on Machine Learning, 23–29 jul 2023

My Comment Abs Bib

Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on massive device simultaneously without the control of a central server. Sharpness-aware minimization, or SAM, is a popular optimization technique that effectively improves model generalization by explicitly minimizing a sharpness-based measure alongside the training loss. In this paper, we prove that D-SGD asymptotically minimizes the loss function of an average-direction SAM. This asymptotic equivalence further demonstrates three advantages of D-SGD: (1) D-SGD exhibits a gradient smoothing effect; (2) there exists an uncertainty self-estimating mechanism in D-SGD to improve posterior estimation; and (3) the sharpness regularization effect of D-SGD does not decrease as total batch size increases, which justifies the superiority of D-SGD over centralized SGD (C-SGD) in large-batch settings. We conduct extensive experiments which are in full agreement with our theory. Our code will be made publicly available.

One-sentence summary: The first work on the surprising sharpness-aware minimization nature of decentralized learning. We provide a completely new perspective to understand decentralization, which helps to bridge the gap between theory and practice in decentralized learning.
@inproceedings{pmlr-zhu23, title = {Decentralized SGD and Average-direction SAM are Asymptotically Equivalent}, author = {Zhu, Tongtian and He, Fengxiang and Chen, Kaixuan and Song, Mingli and Tao, Dacheng}, booktitle = {40th International Conference on Machine Learning}, year = {2023}, month = {23--29 Jul}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, }

📚 Academic Services

Conference Reviewer: ICML 2022, 2023; NeurIPS 2022, 2023; AISTATS 2023; UAI 2023; ECML 2023; ACML 2022.
Journal Reviewer: Information Science.

🤗 Miscellaneous

I am an (old-school) ACG fan. My favorites range from Detective Conan and Pokemon to Attack on Titan.
I am also a basketball fanatic.