Notes

Deep Learning: https://www.fast.ai http://deeplearning.net/ http://neuralnetworksanddeeplearning.com/index.html https://d2l.ai Also, see this for resources on practicing these skills.

Project ideas: https://github.com/NirantK/awesome-project-ideas Datasets: http://academictorrents.com/ https://www.dataquest.io/blog/free-datasets-for-projects/

Bag of Tricks for Image Classification with Convolutional Neural Networks: This paper reviews a few tricks that have been proposed in the past for image classification using CNNs and proposes a few of its own. It then documents how those tricks improve training time or model accuracy. I found the following interesting: Increasing the batch size during training will decrease the variance (and, therefore, noise) of stochastic gradient, so we can increase learning rate and, therefore, reduce training time. Learning rates should be kept low initially, otherwise they can lead to numerical instability. Apply weight decay only to weights, not to biases and $\gamma$ and $\beta$ in Batch Norm layers. You can use 16-bit floating point precision to decrease training time. Cosine decay for learning rate. I didn’t know about the teacher student model, where we make the student imitate the teacher. This is different from transfer learning where we use one model as the starting point to train another. Mixup training: An augmentation technique that randomly chooses two samples and blends them together with a weighted linear interpolation. I couldn’t imagine something like that would work. I didn’t fully understand: Label smoothing.

DeepAnT: A Deep Learning Approach forUnsupervised Anomaly Detectionin Time Series: It’s a long paper but, at it’s core, it presents a straightforward idea. Use CNN as a forecasting model to predict data for a given timestamp. Calculate an anomaly score by comparing the prediction against the observed value and call the time-series anomalous (for that timestamp) if the score is higher than some threshold. I found two problems with their proposal of calculating anomaly scores: It doesn’t take variance into account and only uses absolute difference. If the latter is fixed, shouldn’t anomaly score be inversely proportional to variance? How do I interpret that score in business terminology? They used Mean Absolute Error as the loss function for the model. Why not Mean Squared Error? The idea of choosing time-series specific thresholds would be tricky but the paper doesn’t clarify how to automate that at scale.

Quick notes after reading Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning: When I read about their idea of using U-Nets for anomaly detection, I thought well, that’s cool but where will you find all that labeled data. It was (pleasantly) surprising when they later mentioned that they generated synthetic data themselves, for all the types of time-series and anomalies they wanted to train the model on. Instead of plain change point (i.e. whether there was a short spike or dip) or change of trend detection, their U-Net based approach allows them to go deeper. They are able to do multi-class (i.e. multiple types of anomalies) and both single and multi label for sub-sequences of time-series. Impressive! The paper lacked details around input normalization (based on time-series scale) and data augmentation. They also talked about up-sampling and down-sampling the data to fit into 1024 length sequences their model expects but I didn’t understand how they do that. I couldn’t find their code on the internet. That’s a pity because that could have filled in the holes left in their text.

Resources: https://stackoverflow.com/a/30352360/1517410 https://gist.github.com/0xjac/85097472043b697ab57ba1b1c7530274 https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/duplicating-a-repository

I got the following CDs from the library yesterday: To the Bone, by Steven Wilson: I had already heard this a few times but I was curious to see the art work on the CD. It didn’t disappoint. Kind of Blue, by Miles Davis.

Try something by Anekdoten. Blackfield. Marillion. Fates Warning. Soen. Review websites https://progreport.com/category/cd-reviews/ https://progressivemusicplanet.com/

PyTorch: You can initialize parameters through the nn.init module. Weight decay parameter is supplied to the optimizer (i.e. torch.optim module), not the loss function.

Couple of days ago, I learnt about GPG for general-purpose, and more specifically email, encryption. Unsurprisingly, I got excited to try it out for myself. However, after I spent an afternoon digging more into it, I learnt that, while a powerful tool, there are valid criticisms against using it nowadays. Here is what I understood: GPG is a monolith that tries to do a lot. That makes it cumbersome to use. In today’s world, it’s probably better to use more specific tools for the use case at hand. Given it has so many options, getting them right is error prone. GPG relies on long-term keys which are a bad idea from a security perspective nowadays. (This is also something I’ve learnt from my experience in Amazon.) Burdensome to keep them safe. How you securely transfer them across machines is another problem. Ephemeral keys that are, by design, rotated often are obviously more secure and easier to use. No forward secrecy. Again, bad. Very few people use GPG in the real world. (I don’t even use email all that much today. It only functions as a sink for my online orders etc. Who’s going to send me GPG-encrypted email?!) Interesting resources: ...