Learn how a decision tree splits nodes only to minimize its loss function — When talking about decision tree node splitting, I often hear phrases such as “variance reduction” and “information gain maximization”. And I’m always terrified by them. …

Learn how distributed data parallel and distributed model parallel works inside stochastic gradient descent to let you train gigantic models over huge datasets — You must have heard that recent successful models, such as ChatGPT, have trillions of parameters and trained with terabytes of data. …

A deep dive into the motivation behind the denoising diffusion model and detailed derivations for the loss function — The Denoising Diffusion Probabilistic Models by Jonathan Ho et. al. is a great paper. But I had difficulty understanding it. So I decided to dive into the model and worked out all the derivations. In this article, I will focus on the two main obstacles to understand the paper: why…

The attention mechanism is an influential idea in deep learning. Even though we often think about attention as the one implemented in transformers, the original idea came from the paper “Neural Machine Translation by Jointly Learning to Align and Translate” by Dzmitry Bahdanau et. al. Understanding the origin of the…

Learn why it is valid to use SGD on a linear regression model for parameter learning, see however, SGD can be inefficient, and appreciate what the Adam optimizer does to improve the efficiency of SGD. — We are told to use Stochastic Gradient Descent (SGD) because it speeds up optimization of loss functions in machine learning models. But have you thought about why it is OK to use stochastic gradient descant on your model?

Understand the std err of the learnt parameter weights in linear regression, and other metrics such as the the t metric, and the P>|t| metic. — I always assumed that I know linear regression well, until recently my brain went blank on the question from a colleague — where do the parameter confidence intervals come from when using Python’s ordinary least squares class OLS? I decided to derive all the relevant formulas for linear regression and…

Big data is the cure for many machine learning problems. But one person’s cure can be another’s poison. Big data causes many Bayesian methods to be unpractically expensive. We need to do something or Bayesian methods are left behind the big data revolution. In this article, I will explain why…

People may have the impression that Bayesian methods only work effectively when everything is Gaussian. That’s not true. Modern Bayesian machine learning can comfortably handle non-Gaussian distributions. I will describe the Gaussian Process binary classification model. It uses a Gaussian prior but a Bernoulli likelihood. I will explain why a…

Introduce the VeriTensor code method that you can apply to your Tensorflow code to make debugging more effective. — If you are stuck in front of the paywall, use the friend link. Tensorflow code is difficult to debug. I used to spend weeks debugging my code. Worse, most of the time, I didn’t know how to proceed — I can see my code does not learn, but I don’t…