Dan Alistarh (IST Austria): Distributed and Concurrent Optimization for Machine Learning
October 15th (Tuesday) 09:00 – 10:00
Slides (pdf)
 |
Dan Alistarh is currently an Assistant Professor at IST Austria. Previously, he was affiliated with ETH Zurich, Microsoft Research, and MIT. He received his PhD from the EPFL, under the guidance of Prof. Rachid Guerraoui. His research focuses on distributed algorithms and concurrent data structures, and spans from algorithms and lower bounds to practical implementations. He is the recipient of an ERC Starting Grant with a focus on distributed machine learning. |
Abstract: Machine learning has made considerable progress over the past decade, matching and even surpassing human performance on a varied set of narrow computational tasks. This progress has been enabled by the widespread availability of large datasets, as well as by improved algorithms and models. Distribution, implemented either through single-node concurrency or through multi-node parallelism, has been third third key ingredient to these advances.
The goal of this talk is to provide an overview of the role of distributed computing in machine learning, with an eye towards the intriguing trade-offs between synchronization and communication costs of distributed machine learning algorithms, on the one hand, and their convergence, on the other. The focus will be on parallelization strategies for the fundamental stochastic gradient descent (SGD) algorithm, which is a key tool when training machine learning models, from venerable linear regression, to state-of-the-art neural network architectures. Along the way, we will provide an overview of the ongoing research and open problems in distributed machine learning. The lecture will assume no prior knowledge of machine learning or optimization, beyond familiarity with basic concepts in algebra and analysis.
János Kertész (Central European University): Network science – a bridge between social and “hard” sciences
October 16th (Wednesday) 09:00 – 10:00
 |
János Kertész is a professor at the Department of Network and Data Science of the Central European University. He was visiting scientist in Germany, US, France, Italy and Finland. He is interested in statistical physics and its applications, including percolation theory, phase transitions, fractal growth, granular materials and simulation methods. His work has been awarded by several recognitions, including the Hungarian Academy Award, the Szent-Györgyi Award of the Ministry of Education and Culture, the Széchenyi Prize and the title of Finland Distinguished Professor. |
Seth Gilbert (National University of Singapore): When is an algorithm robust?
October 17th (Thursday) 09:00 – 10:00
Slides (pdf)
 |
Seth Gilbert is an Associate Professor at the National University of Singapore. He received his PhD from MIT, and spent several years as a postdoc at EPFL. His work includes research on backoff protocols, dynamic graph algorithms, wireless networks, robust scheduling, and the occasional blockchain. In fact, Seth’s research focuses on algorithmic issues of robustness and scalability, wherever they may arise. |
Abstract: We live in a world of increasingly fragile networks, where outages are common and failures can cascade through networks that are ever increasing in scale and connectivity. Worse, increasing levels of optimization lead to decreasing margin of error, causing catastrophic failures.
In the distributed algorithms community, we have a long history of focusing on robustness. (For example, more than half of the papers awarded the Dijkstra Prize have fault-tolerance as a critical motivation.) And yet, over the last several years, the classical approaches to fault-tolerance seem to be less and less effective at handling contemporary problems.
In this talk, I will posit that we need to develop new approaches to designing robust algorithms. Specifically, I would like to discuss two different approaches we have been exploring for managing uncertainty, a key cause of fragility.
The first approach focuses on uncertainty caused by noise: wireless networks, dynamic distributed networks, biological systems, and many others, suffer random noise. Some behaviors are driven by larger scale trends (i.e., adversarial), while others are caused by local noise (i.e., stochastic). By designing algorithms that tolerate these types of noise, and by analyzing performance in a “smoothed” manner, we can develop more robust algorithms.
The second approach focuses on uncertainty caused by future changes. In general, there is a trade-off between how well a solution is optimized and its robustness to change. We explore this trade-off in the context of resource allocation, showing how leaving slack in the system today can prevent catastrophic failures tomorrow.
Looking forward, the distributed algorithms community has a key role to play in designing more robust and less fragile networks, and I am optimistic that we will find new ways to think about and quantify robustness.
