Notes

Setup Installation guide: https://docs.docker.com/engine/install/ubuntu/ No sudo: Check this to allow Docker containers to run as non-root. See what’s running docker ps Container-specific commands # SSH into a running container. docker exec -it 0061ecc7ef37 /bin/bash # Restart docker restart 52835f0907e8 # Stop docker container stop 23642fbe8515 # Stop all docker stop $(docker ps -a -q) # Prune all stopped containers docker container prune Parameters for docker run -d: Run as daemon. -p 8439:8080: Map port A externally to B internally. -v ~/Documents/TiddlyWikiDockerVolume:/var/lib/tiddlywiki: Map volume A externally to B internally. -e USERNAME="abc": Environment variable. --restart unless-stopped: Automatically bring up the container on Docker daemon restart. Compose # Assuming current directory contains a file named *docker-compose.yml* docker-compose up # Or docker-compose down Uprade an existing service The following assumes that the containers are stateless. ...

Deep Learning: https://www.fast.ai http://deeplearning.net/ http://neuralnetworksanddeeplearning.com/index.html https://d2l.ai Also, see this for resources on practicing these skills.

Project ideas: https://github.com/NirantK/awesome-project-ideas Datasets: http://academictorrents.com/ https://www.dataquest.io/blog/free-datasets-for-projects/

Bag of Tricks for Image Classification with Convolutional Neural Networks: This paper reviews a few tricks that have been proposed in the past for image classification using CNNs and proposes a few of its own. It then documents how those tricks improve training time or model accuracy. I found the following interesting: Increasing the batch size during training will decrease the variance (and, therefore, noise) of stochastic gradient, so we can increase learning rate and, therefore, reduce training time. Learning rates should be kept low initially, otherwise they can lead to numerical instability. Apply weight decay only to weights, not to biases and $\gamma$ and $\beta$ in Batch Norm layers. You can use 16-bit floating point precision to decrease training time. Cosine decay for learning rate. I didn’t know about the teacher student model, where we make the student imitate the teacher. This is different from transfer learning where we use one model as the starting point to train another. Mixup training: An augmentation technique that randomly chooses two samples and blends them together with a weighted linear interpolation. I couldn’t imagine something like that would work. I didn’t fully understand: Label smoothing.

DeepAnT: A Deep Learning Approach forUnsupervised Anomaly Detectionin Time Series: It’s a long paper but, at it’s core, it presents a straightforward idea. Use CNN as a forecasting model to predict data for a given timestamp. Calculate an anomaly score by comparing the prediction against the observed value and call the time-series anomalous (for that timestamp) if the score is higher than some threshold. I found two problems with their proposal of calculating anomaly scores: It doesn’t take variance into account and only uses absolute difference. If the latter is fixed, shouldn’t anomaly score be inversely proportional to variance? How do I interpret that score in business terminology? They used Mean Absolute Error as the loss function for the model. Why not Mean Squared Error? The idea of choosing time-series specific thresholds would be tricky but the paper doesn’t clarify how to automate that at scale.

Quick notes after reading Time Series Anomaly Detection Using Convolutional Neural Networks and Transfer Learning: When I read about their idea of using U-Nets for anomaly detection, I thought well, that’s cool but where will you find all that labeled data. It was (pleasantly) surprising when they later mentioned that they generated synthetic data themselves, for all the types of time-series and anomalies they wanted to train the model on. Instead of plain change point (i.e. whether there was a short spike or dip) or change of trend detection, their U-Net based approach allows them to go deeper. They are able to do multi-class (i.e. multiple types of anomalies) and both single and multi label for sub-sequences of time-series. Impressive! The paper lacked details around input normalization (based on time-series scale) and data augmentation. They also talked about up-sampling and down-sampling the data to fit into 1024 length sequences their model expects but I didn’t understand how they do that. I couldn’t find their code on the internet. That’s a pity because that could have filled in the holes left in their text.

Resources: https://stackoverflow.com/a/30352360/1517410 https://gist.github.com/0xjac/85097472043b697ab57ba1b1c7530274 https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/duplicating-a-repository

I got the following CDs from the library yesterday: To the Bone, by Steven Wilson: I had already heard this a few times but I was curious to see the art work on the CD. It didn’t disappoint. Kind of Blue, by Miles Davis.

Try something by Anekdoten. Blackfield. Marillion. Fates Warning. Soen. Review websites https://progreport.com/category/cd-reviews/ https://progressivemusicplanet.com/

PyTorch: You can initialize parameters through the nn.init module. Weight decay parameter is supplied to the optimizer (i.e. torch.optim module), not the loss function.