Today I learned

PyTorch:

  • You can initialize parameters through the nn.init module.
  • Weight decay parameter is supplied to the optimizer (i.e. torch.optim module), not the loss function.