Notes

I am one of those people who function better by writing things down. One day, I realized that most of my notes don’t have to be private, so here they are - my second brain. Be warned that, if you stumble upon something here that doesn’t make sense to you, it isn’t meant to!

Today I learned

Fastai: Unets are useful when the size of your outputs is similar to that of the input. So, any kind of generative modeling such as segmentation or decrappification. They don’t make sense for classification problems. Unets have a downsampling and upsampling path. These parts are also called encoder and decoder respectively. A layer in a neural network is always an affine function, such as a matrix multiplication, followed by a non-linearity, such as a relu.

Change user information for Git commits

Run the following to change Git user information globally: git config --global user.name "Ketan Vijayvargiya" git config --global user.email "[email protected]" To change on a repository level, simply omit the --global flag.

Today I learned

On the AI/Fastai front: If x is a torch.Tensor and x.shape is torch.Size([4, 3, 5]), it means x contains 4 2D matrices with 3 rows and 5 columns. Also, x is 3D matrix, the technical term for which would be a rank-3 tensor. When images are represented using PyTorch tensors, the first dimension corresponds to the number of channels in it. (For e.g., an RGB image will contain 3 channels and, therefore, the first element in the tensor shape will be 3.) Matplotlib’s imshow function expects number of channels at the end and you can modify a tensor to that by applying the permute function.

Effectiveness vs efficiency

The following Youtube video reminded me to focus on the more important things in life. Lately, I’ve been worrying too much about non-essential things. I should instead constantly remind myself of what’s essential and reorient my priorities towards that. The video also mentioned how 20% effort leads to 80% result. And often times, I can offload the remaining 20% results to someone else (for some work-related projects) or not do them at all (in personal projects). So, it’d be a good idea to try and identify where to put the first 20% effort in any project. In other words, be effective before efficient.

Revert environments in Conda

Revert an environment: Find out the revision number to revert to by running: conda list --revisions. Then revert using: conda install --revision 22.

Today I learned

Pandas: If df2 is a DataFrame, df2['A'] returns a Series whereas both df2[['A']] and df2[['A', 'B']] return a DataFrame. Fastai: Embeddings are indeed created for categorical variables. For example, in the Excel sheet on collaborative filtering, Jeremy had created embeddings for user-ids and movie-ids both of which were categorical variables. Let’s say we do a stride 2 in a CNN. While the output height and width will halve, we still double the number of output channels probably because the output channels detect various features of the images and we care more about capturing those features.

Update packages in Conda

Update one package in an environment: conda update -n pytorch_p36 numpy Update all (which is generally advised): conda update --all

Today I learned

Fastai Dropout: In the image below, the white ones are activations and the arrows are parameters. In one layer of a neural net, a vector of activations are multiplied by a matrix of parameters. And since parameters are part of a matrix, there are a lot of them below. We randomly throw away activations, not parameters, in each mini-batch with a probability p. Batch Norm: Works for continuous variables. Keeps loss landscape less bumpy, therefore allowing you to increase learning rate and arrive at good results quickly. If loss landscape is bumpy, you want to keep learning rate low lest you mistakenly jump off into some awful part of the weight space. Both a regularization technique and training helper, the latter probably because of the previous point. In neural networks, there are only two kinds of numbers: activations and parameters. The latter are learnable.

Today I learned

Fastai: Validation vs test data: Validation set: used actively during training to calculate and minimize loss. Test set: (at least in Fastai) it is used unlabeled, the assumption being that we need to predict on test data using our trained model. Force GC (i.e. garbage collection) in Python: learn = None import gc gc.collect()

Deep work

I should try harder to create space for doing deep work. I had thought about it in the past but have been slipping on this lately. For instance, I have (re)started acting more as a tech lead and less as an IC in office and I don’t prioritize the Fastai course as much as I’d like to.