Notes

On the AI/Fastai front: If x is a torch.Tensor and x.shape is torch.Size([4, 3, 5]), it means x contains 4 2D matrices with 3 rows and 5 columns. Also, x is 3D matrix, the technical term for which would be a rank-3 tensor. When images are represented using PyTorch tensors, the first dimension corresponds to the number of channels in it. (For e.g., an RGB image will contain 3 channels and, therefore, the first element in the tensor shape will be 3.) Matplotlib’s imshow function expects number of channels at the end and you can modify a tensor to that by applying the permute function.

The following Youtube video reminded me to focus on the more important things in life. Lately, I’ve been worrying too much about non-essential things. I should instead constantly remind myself of what’s essential and reorient my priorities towards that. The video also mentioned how 20% effort leads to 80% result. And often times, I can offload the remaining 20% results to someone else (for some work-related projects) or not do them at all (in personal projects). So, it’d be a good idea to try and identify where to put the first 20% effort in any project. In other words, be effective before efficient.

Revert an environment: Find out the revision number to revert to by running: conda list --revisions. Then revert using: conda install --revision 22.

Pandas: If df2 is a DataFrame, df2['A'] returns a Series whereas both df2[['A']] and df2[['A', 'B']] return a DataFrame. Fastai: Embeddings are indeed created for categorical variables. For example, in the Excel sheet on collaborative filtering, Jeremy had created embeddings for user-ids and movie-ids both of which were categorical variables. Let’s say we do a stride 2 in a CNN. While the output height and width will halve, we still double the number of output channels probably because the output channels detect various features of the images and we care more about capturing those features.

Update one package in an environment: conda update -n pytorch_p36 numpy Update all (which is generally advised): conda update --all

Fastai Dropout: In the image below, the white ones are activations and the arrows are parameters. In one layer of a neural net, a vector of activations are multiplied by a matrix of parameters. And since parameters are part of a matrix, there are a lot of them below. We randomly throw away activations, not parameters, in each mini-batch with a probability p. Batch Norm: Works for continuous variables. Keeps loss landscape less bumpy, therefore allowing you to increase learning rate and arrive at good results quickly. If loss landscape is bumpy, you want to keep learning rate low lest you mistakenly jump off into some awful part of the weight space. Both a regularization technique and training helper, the latter probably because of the previous point. In neural networks, there are only two kinds of numbers: activations and parameters. The latter are learnable.

Fastai: Validation vs test data: Validation set: used actively during training to calculate and minimize loss. Test set: (at least in Fastai) it is used unlabeled, the assumption being that we need to predict on test data using our trained model. Force GC (i.e. garbage collection) in Python: learn = None import gc gc.collect()

I should try harder to create space for doing deep work. I had thought about it in the past but have been slipping on this lately. For instance, I have (re)started acting more as a tech lead and less as an IC in office and I don’t prioritize the Fastai course as much as I’d like to.

Fastai: While optimizing (using, say, SGD), you calculate derivative of loss with respect to each parameter separately thereby updating that particular parameter. Loss is a function of prediction and observed. Prediction is a function of model and inputs. Model is composed of parameters. Therefore, loss is a function of model parameters. When you call loss.backward(), Pytorch calculates the gradients - but where would it store those gradients? Since gradients are derivatives of the loss function with respect to the parameters and calculated per parameter, it makes sense to expect them inside parameter tensor. Gradients are found in parameter.grad. One should zero out the gradients after each execution on a mini-batch - using parameter.grad.zero_() - so that gradients from one run don’t affect the next run. Python: ...

I put a lot of effort in polishing my documents, as if I try to perfect each of them. (There has been at least one recent instance where this has dragged on literally for weeks.) That leads to two problems: The folks reading my documents aren’t English experts or generally smart enough to differentiate between a polished vs a somewhat half-baked document. At the end of the day, how many ideas I produced matters more than how good my English was in them. Moreover, if I have a lot of documents at end of the day, who will go through all of them to know the difference. It takes time to polish these documents and that leaves me less time to write more of them. I should start focusing on creating a lot of documents and only put minimal time in polishing them. Maybe do two sittings: one, to jot down all the ideas quickly, and two, maybe a day later, polish them a little.