报告人：曾锦山（江西师范大学）

邀请人：刘海霞

报告时间：2021年6月6日（星期日）10：00-12：00

报告地点： 科技楼（南楼）813室

报告题目：On ADMM in Deep Learning: Convergence and Saturation-Avoidance

报告摘要：In this talk, we introduce an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called sigmoid-ADMM pair), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of deep neural networks with sigmoid-type activations (called deep sigmoid nets) over their rectified linear unit (ReLU) counterparts (called deep ReLU nets) in terms of approximation. In particular, we prove that the approximation capability of deep sigmoid nets is not worse than deep ReLU nets by showing that ReLU activation fucntion can be well approximated by deep sigmoid nets with two hidden layers and finitely many free parameters but not vice-verse. We also establish the global convergence of the proposed ADMM for the nonlinearly constrained formulation of the deep sigmoid nets training to a Karush-Kuhn-Tucker (KKT) point at a rate of order O(1/k). Compared with the widely used stochastic gradient descent (SGD) algorithm for the deep ReLU nets training (called ReLU-SGD pair), the proposed sigmoid-ADMM pair is practically stable with respect to the algorithmic hyperparameters including the learning rate, initial schemes and the pro-processing of the input data. Moreover, we find that to approximate and learn simple but important functions the proposed sigmoid-ADMM pair numerically outperforms the ReLU-SGD pair.

报告人简介：曾锦山，江西师范大学计算机信息工程学院特聘教授，数据科学与大数据系主任。2015年博士毕业于西安交通大学。先后在美国加州大学洛杉矶分校、香港科技大学和香港城市大学从事博士后或访问合作研究。2020年入选江西省人才计划，曾两度获得“世界华人数学家大会最佳论文奖”（2018、2020年）。现已发表高水平论文近四十篇，其中在IEEE TPAMI、JMLR、ICML和IEEE TSP等期刊和会议上发表论文十余篇，ESI热点论文1篇，单篇论文最高引用逾600次。承担国家自然科学基金2项。主要研究方向包括非凸优化和机器学习。