Notes

Revert an environment: Find out the revision number to revert to by running: conda list --revisions. Then revert using: conda install --revision 22.

Pandas: If df2 is a DataFrame, df2['A'] returns a Series whereas both df2[['A']] and df2[['A', 'B']] return a DataFrame. Fastai: Embeddings are indeed created for categorical variables. For example, in the Excel sheet on collaborative filtering, Jeremy had created embeddings for user-ids and movie-ids both of which were categorical variables. Let’s say we do a stride 2 in a CNN. While the output height and width will halve, we still double the number of output channels probably because the output channels detect various features of the images and we care more about capturing those features.

Update one package in an environment: conda update -n pytorch_p36 numpy Update all (which is generally advised): conda update --all

Fastai Dropout: In the image below, the white ones are activations and the arrows are parameters. In one layer of a neural net, a vector of activations are multiplied by a matrix of parameters. And since parameters are part of a matrix, there are a lot of them below. We randomly throw away activations, not parameters, in each mini-batch with a probability p. Batch Norm: Works for continuous variables. Keeps loss landscape less bumpy, therefore allowing you to increase learning rate and arrive at good results quickly. If loss landscape is bumpy, you want to keep learning rate low lest you mistakenly jump off into some awful part of the weight space. Both a regularization technique and training helper, the latter probably because of the previous point. In neural networks, there are only two kinds of numbers: activations and parameters. The latter are learnable.

Fastai: Validation vs test data: Validation set: used actively during training to calculate and minimize loss. Test set: (at least in Fastai) it is used unlabeled, the assumption being that we need to predict on test data using our trained model. Force GC (i.e. garbage collection) in Python: learn = None import gc gc.collect()

I should try harder to create space for doing deep work. I had thought about it in the past but have been slipping on this lately. For instance, I have (re)started acting more as a tech lead and less as an IC in office and I don’t prioritize the Fastai course as much as I’d like to.

Fastai: While optimizing (using, say, SGD), you calculate derivative of loss with respect to each parameter separately thereby updating that particular parameter. Loss is a function of prediction and observed. Prediction is a function of model and inputs. Model is composed of parameters. Therefore, loss is a function of model parameters. When you call loss.backward(), Pytorch calculates the gradients - but where would it store those gradients? Since gradients are derivatives of the loss function with respect to the parameters and calculated per parameter, it makes sense to expect them inside parameter tensor. Gradients are found in parameter.grad. One should zero out the gradients after each execution on a mini-batch - using parameter.grad.zero_() - so that gradients from one run don’t affect the next run. Python: ...

I put a lot of effort in polishing my documents, as if I try to perfect each of them. (There has been at least one recent instance where this has dragged on literally for weeks.) That leads to two problems: The folks reading my documents aren’t English experts or generally smart enough to differentiate between a polished vs a somewhat half-baked document. At the end of the day, how many ideas I produced matters more than how good my English was in them. Moreover, if I have a lot of documents at end of the day, who will go through all of them to know the difference. It takes time to polish these documents and that leaves me less time to write more of them. I should start focusing on creating a lot of documents and only put minimal time in polishing them. Maybe do two sittings: one, to jot down all the ideas quickly, and two, maybe a day later, polish them a little.

Fastai: While updating weights, we multiply learning rate with derivative of the loss function with respect to the weights. Loss function is a function of independent variables, X, and weights. Cross-entropy loss function is useful for classification problems where you don’t care about how close your prediction was. Softmax is an activation function that allows your output to be between 0 and 1. Somewhat like the Sigmoid that conforms output to a range. Some discussion on when to use or the other of these: Softmax vs Sigmoid function in Logistic classifier? You generally want Cross-entropy loss and Softmax for single-label multi-class classification problems. They go well together. Regularization techniques allow you to avoid over-fitting. Weight decay. Dropout. Batch Norm. Data augmentation. One way to avoid over-fitting is to use less parameters. However, Jeremy proposes that we instead use a lot of parameters but penalize complexity. Weight decay is one way to do the latter.

I haven’t been working out as much as I’d like to. One reason is back pain that has become somewhat persistent lately. I need to fix that immediately lest it becomes a major problem. Stop working on the laptop in the living room - only use the desk and monitor. Other reason could be laziness. If so, I might just need to regain some motivation.