Notes

Fastai Dropout: In the image below, the white ones are activations and the arrows are parameters. In one layer of a neural net, a vector of activations are multiplied by a matrix of parameters. And since parameters are part of a matrix, there are a lot of them below. We randomly throw away activations, not parameters, in each mini-batch with a probability p. Batch Norm: Works for continuous variables. Keeps loss landscape less bumpy, therefore allowing you to increase learning rate and arrive at good results quickly. If loss landscape is bumpy, you want to keep learning rate low lest you mistakenly jump off into some awful part of the weight space. Both a regularization technique and training helper, the latter probably because of the previous point. In neural networks, there are only two kinds of numbers: activations and parameters. The latter are learnable.

Fastai: Validation vs test data: Validation set: used actively during training to calculate and minimize loss. Test set: (at least in Fastai) it is used unlabeled, the assumption being that we need to predict on test data using our trained model. Force GC (i.e. garbage collection) in Python: learn = None import gc gc.collect()

I should try harder to create space for doing deep work. I had thought about it in the past but have been slipping on this lately. For instance, I have (re)started acting more as a tech lead and less as an IC in office and I don’t prioritize the Fastai course as much as I’d like to.

Fastai: While optimizing (using, say, SGD), you calculate derivative of loss with respect to each parameter separately thereby updating that particular parameter. Loss is a function of prediction and observed. Prediction is a function of model and inputs. Model is composed of parameters. Therefore, loss is a function of model parameters. When you call loss.backward(), Pytorch calculates the gradients - but where would it store those gradients? Since gradients are derivatives of the loss function with respect to the parameters and calculated per parameter, it makes sense to expect them inside parameter tensor. Gradients are found in parameter.grad. One should zero out the gradients after each execution on a mini-batch - using parameter.grad.zero_() - so that gradients from one run don’t affect the next run. Python: ...

I put a lot of effort in polishing my documents, as if I try to perfect each of them. (There has been at least one recent instance where this has dragged on literally for weeks.) That leads to two problems: The folks reading my documents aren’t English experts or generally smart enough to differentiate between a polished vs a somewhat half-baked document. At the end of the day, how many ideas I produced matters more than how good my English was in them. Moreover, if I have a lot of documents at end of the day, who will go through all of them to know the difference. It takes time to polish these documents and that leaves me less time to write more of them. I should start focusing on creating a lot of documents and only put minimal time in polishing them. Maybe do two sittings: one, to jot down all the ideas quickly, and two, maybe a day later, polish them a little.

Fastai: While updating weights, we multiply learning rate with derivative of the loss function with respect to the weights. Loss function is a function of independent variables, X, and weights. Cross-entropy loss function is useful for classification problems where you don’t care about how close your prediction was. Softmax is an activation function that allows your output to be between 0 and 1. Somewhat like the Sigmoid that conforms output to a range. Some discussion on when to use or the other of these: Softmax vs Sigmoid function in Logistic classifier? You generally want Cross-entropy loss and Softmax for single-label multi-class classification problems. They go well together. Regularization techniques allow you to avoid over-fitting. Weight decay. Dropout. Batch Norm. Data augmentation. One way to avoid over-fitting is to use less parameters. However, Jeremy proposes that we instead use a lot of parameters but penalize complexity. Weight decay is one way to do the latter.

I haven’t been working out as much as I’d like to. One reason is back pain that has become somewhat persistent lately. I need to fix that immediately lest it becomes a major problem. Stop working on the laptop in the living room - only use the desk and monitor. Other reason could be laziness. If so, I might just need to regain some motivation.

Pandas Series can have string indices (similar to how array indices in general are numbers). numpy.array is just a convenience function to create an ndarray. You can also create an array using numpy.ndarray but that is not recommended. Can I try out the following? Scatter plot of movie ratings vs their bias.

I am a fitness noob, but was a bigger one around the same time last year. I used to run once in a while in a small park near my place, but my form wasn’t correct. I would land on the front half of my feet, that would cause so much pain in between my knees and calves that I’d always have to rest for 2-3 days before going out. I didn’t know that my form was off - instead I thought my legs weren’t strong enough to run - until later. It was once I watched a bunch of Youtube videos did I start running correctly and enjoying it along the way. I have since run decent distances on Alki and Elliot Bay trails, and a half marathon in just 2h35m, a number I am personally proud of. ...

I used to blog, but then I quit because it was time-consuming. It used to take me a lot of time to coherently put down my ideas, often requiring multiple iterations to get the grammar right. And the latter was necessary because I am a grammar nazi and didn’t have tools like Grammarly back then. I, however, feel that it’s important to write for many reasons. One, my future self will be interested to know what I was thinking today. And even though pictures and tweets can serve a similar purpose, they can’t beat a well-thought blog post. Two, it gives clarity to thoughts. As Jeff Besoz says, “There is no way to write a six-page, narratively structured memo and not have clear thinking.” Of course, six isn’t a strict number, and the same should apply to a three hundred word blog post! Third, it improves my production vs consumption balance. I consume way more content than what I produce, a side-effect of the always online mindset of the contemporary world. This isn’t healthy, and writing can help fix it to a minor extent. ...