Last week I had a pleasure to participate in the International Conference on Learning Representations (ICLR), an event dedicated to the research on all aspects of deep learning. Initially, the conference was supposed to take place in Addis Ababa, Ethiopia, however, due to the novel coronavirus pandemic, it went virtual. Iâm sure it was a challenge for organisers to move the event online, but I think the effect was more than satisfactory, as you can read here!
Over 1300 speakers and 5600 attendees proved that the virtual format was more accessible for the public, but at the same time, the conference remained interactive and engaging. From many interesting presentations, I decided to choose 16, which are influential and thought-provoking. Here are the best deep learning papers from the ICLR.
Best Deep Learning papers
1. On Robustness of Neural Ordinary Differential Equations
In-depth study of the robustness of the Neural Ordinary Differential Equations or NeuralODE in short. Use it as a building block for more robust networks.
4. Understanding and Robustifying Differentiable Architecture Search
We study the failure modes of DARTS (Differentiable Architecture Search) by looking at the eigenvalues of the Hessian of validation loss w.r.t. the architecture and propose robustifications based on our analysis.
5. Comparing Rewinding and Fine-tuning in Neural Network Pruning
Instead of fine-tuning after pruning, rewind weights or learning rate schedule to their values earlier in training and retrain from there to achieve higher accuracy when pruning neural networks.
Neural nets, while capable of approximating complex functions, are rather poor in exact arithmetic operations. This task was a longstanding challenge to deep learning researchers. Here, the novel, Neural Addition Unit (NAU) and Neural Multiplication Unit (NMU) are presented, capable of performing exact addition/subtraction (NAU) and multiplying subsets of a vector (MNU). Notable first author is an independent researcher đ
7.The Break-Even Point on Optimization Trajectories of Deep Neural Networks
In the early phase of training of deep neural networks there exists a “break-even point” which determines properties of the entire optimization trajectory.
9. Selection via Proxy: Efficient Data Selection for Deep Learning
We can significantly improve the computational efficiency of data selection in deep learning by using a much smaller proxy model to perform data selection.
11. A Signal Propagation Perspective for Pruning Neural Networks at Initialization
We formally characterize the initialization conditions for effective pruning at initialization and analyze the signal propagation properties of the resulting pruned networks which leads to a method to enhance their trainability and pruning results.
Depth and breadth of the ICLR publications is quite inspiring. Here, I just presented the tip of an iceberg focusing on the âdeep learningâ topic. However, this analysis, suggests that there were few popular areas, specifically:
In order to create a more complete overview of the top papers at ICLR, we are building a series of posts, each focused on one topic mentioned above. You may want to check them out for a more complete overview.
Happy reading!
Was the article useful?
Thank you for your feedback!
More about
The Best Deep Learning Papers from the ICLR 2020 Conference
Check out our
product resources
and
related articles below: