Ferenc Huszar - inFERENCe (Page 2)

inFERENCe

Sign in Subscribe

Ferenc Huszar

Notes on the Origin of Implicit Regularization in SGD

Notes on the Origin of Implicit Regularization in SGD

I wanted to highlight an intriguing paper I presented at a journal club recently: * Samuel L Smith, Benoit Dherin, David Barrett, Soham De (2021) On the Origin of Implicit Regularization in Stochastic Gradient Descent [https://openreview.net/forum?id=rq_Qr0c1Hyo] There's actually a related paper that came

$An information maximization view on the $\beta$-VAE objective$

An information maximization view on the $\beta$-VAE objective

guest post with Dóra Jámbor [http://dorajambor.com/] This is a half-guest-post written jointly with Dóra, a fellow participant in a reading group where we recently discussed the original paper on $\beta$-VAEs: * Irina Higgins et al (ICLR 2017): $\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

Some Intuition on the Neural Tangent Kernel

Some Intuition on the Neural Tangent Kernel

Neural tangent kernels are a useful tool for understanding neural network training and implicit regularization in gradient descent. But it's not the easiest concept to wrap your head around. The paper that I found to have been most useful for me to develop an understanding is this one:

Notes on Causally Correct Partial Models

Notes on Causally Correct Partial Models

I recently encountered this cool paper in a reading group presentation: * Rezende et al (2020) Rezende Causally Correct Partial Models for Reinforcement Learning [https://arxiv.org/abs/2002.02836v1] It's frankly taken me a long time to understand what was going on, and it took me weeks to

Meta-Learning Millions of Hyper-parameters using the Implicit Function Theorem

Meta-Learning Millions of Hyper-parameters using the Implicit Function Theorem

Last night on the train I read this nice paper by David Duvenaud and colleagues. Around midnight I got a calendar notification "it's David Duvenaud's birthday". So I thought it's time for a David Duvenaud birthday special (don't get too

The secular Bayesian: Using belief distributions without really believing

The secular Bayesian: Using belief distributions without really believing

The religious Bayesian My parents didn't raise me in a religious tradition. It all started to change when a great scientist took me under his wing and taught me the teachings of Bayes. I travelled the world and spent 4 years in a Bayesian monastery in Cambridge, UK.

Exponentially Growing Learning Rate? Implications of Scale Invariance induced by Batch Normalization

Exponentially Growing Learning Rate? Implications of Scale Invariance induced by Batch Normalization

Yesterday I read this intriguing paper about the midboggling fact that it is possible to use exponentially growing learning rate schedule when training neural networks with batch normalization: * Zhiyuan Li and Sanjeev Arora (2019) An Exponential Learning Rate Schedule for Deep Learning [https://arxiv.org/abs/1910.07454] The paper

$On Marginal Likelihood and Cross-Validation$

On Marginal Likelihood and Cross-Validation

Here's a paper someone has pointed me to, along the lines of "everything that works, works because it's Bayesian [https://www.inference.vc/everything-that-works-works-because-its-bayesian-2/]": * Edwin Fong, Chris Holmes (2019) On the marginal likelihood and cross-validation [https://arxiv.org/abs/1905.08737] I found this

Notes on iMAML: Meta-Learning with Implicit Gradients

Notes on iMAML: Meta-Learning with Implicit Gradients

This week I read this cool new paper on meta-learning: it a slightly different approach compared to its predecessors based on some observations about differentiating the optima of regularized optimization. * Aravind Rajeswaran, Chelsea Finn, Sham Kakade, Sergey Levine (2019) Meta-Learning with Implicit Gradients [https://arxiv.org/abs/1909.04630] Another

Invariant Risk Minimization: An Information Theoretic View

Invariant Risk Minimization: An Information Theoretic View

I finally got around to reading this new paper by Arjovsky et al. It debuted on Twitter with a big splash, being decribed as 'beautiful' and 'long awaited' 'gem of a paper'. It almost felt like a new superhero movie or Disney remake just