Covid closure

We have a facebook group for comunication: https://www.facebook.com/groups/544273583166584/

Things you should get familiar with:

Machine translation

• See http://web.stanford.edu/class/cs224n/ mainly: Lecture8 about machine translation. Get familiar with sequence to sequence models, beam search decoding and attention.

Speech recognition (mainly CTC and Transducers)

• Get familiar with Connectionist temporal classification and Transducers. https://www.youtube.com/watch?v=3MjIkWxXigM
• I also recommend checking https://arxiv.org/pdf/1211.3711.pdf and http://www.cs.toronto.edu/~hinton/absps/DRNN_speech.pdf.

Self-attention and transformers

• Again http://web.stanford.edu/class/cs224n/ lecture 10

Bayessian modeling

• Check out PyMC3 tutorial https://docs.pymc.io/notebooks/getting_started.html (there will be more content about this comming soon).

Genarative adversarial networks (GANs)

• It would be great if you check basic of GANs, typical problems (saturation, mode colapse) and Wasserstein GANs.
• Checkout what can Earthmover distance capture compared to KL divergence.
• If you are brave, go for tutorials on https://fleuret.org/ee559/ (11.1, 11.2), but they are bit technical IMHO. There is also a video on older version of the course https://fleuret.org/ee559-2018/dlc/.
• These links might be little bit more accessible http://slazebni.cs.illinois.edu/spring17/lec11_gan.pdf

https://www.alexirpan.com/2017/02/22/wasserstein-gan.html, https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html

Reinforcement learning and policy gradients

• slides 1, slides 2, slides 3

Adversarial training

• Skim material from here https://adversarial-ml-tutorial.org/

Schedule

• Tuesday 11:30-13:00 F1-247
• Thursday 9:50-11:20 M-X

Contact

<surname>@fmph.uniba.sk

Grading

Three projects:

• Autograd - 20% - deadline 9th March
• Relativelly simple implementation of something exotic. Lottery tickets - 30% - deadline end of semester (22th May for 5th year students, 29th May for rest)
• Final project aka Replication of recent paper results - 50%
  • General deadline 25th June
  • Deadline for 5th year students - 27th May

Getting less than 50% from autograd project -> FX. Otherwise typical grading scheme (more than 90% - A, 89% - 80% - B, …, 59 % - 50% - E, less than 50% - FX).

Recommended prerequisites

• Either Machine learning or Neural networks course (ideally both)
• No fear of math (gradients)
• Reasonable coding skills in Python

Good other courses

• http://ttic.uchicago.edu/~shubhendu/Pages/CMSC35246.html
• https://fleuret.org/ee559/ and https://fleuret.org/ee559-2018/dlc/
• http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/
• http://introtodeeplearning.com/

Schedule

Similar to old. We will skip things over graphs (except semisupervised learning). We will probably add more about model compression and deployment (quantization).

Calculus for deep learning: https://explained.ai/matrix-calculus/

18. 2

• Introduction to DL. https://documents.epfl.ch/users/f/fl/fleuret/www/dlc/dlc-slides-1a-introduction.pdf, http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture1_pauses.pdf
• Typical modelling problem, which consist of:
  • parametrized function, which takes input and output prediction
  • loss function, which takes prediction and required output and outputs loss
  • optimizer, which find parameters with smallest loss as possible
• Pytorch demos

20. 2

• Optimization algorithms and initialization. Slides from: http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture6_flat.pdf, https://documents.epfl.ch/users/f/fl/fleuret/www/dlc/dlc-slides-5-init-optim.pdf
• About network initialization: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

25. 2

• Building blocks of neural networks
• Fully-connected (Pytorch Linear, Keras Dense) layers
• Convolutional layers
• Maxpooling

27. 2

• Recurrent neural networks, GRU, LSTMs
• Difficulty of training them

3. 3

• Relu
• BatchNormalization, LayerNormalization
• ResNets

5. 3

• Object detection, more material

Old schedule

20.2.

• Introduction to DL. https://documents.epfl.ch/users/f/fl/fleuret/www/dlc/dlc-slides-1a-introduction.pdf, http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture1_pauses.pdf
• Typical modelling problem, which consist of:
  • parametrized function, which takes input and output prediction
  • loss function, which takes prediction and required output and outputs loss
  • optimizer, which find parameters with smallest loss as possible

21.2.

• Demonstration of theano and chainer (TODO notebooks)
• How backprop works

27.2

• Optimization algorithms and initialization. Slides from: http://ttic.uchicago.edu/~shubhendu/Pages/Files/Lecture6_flat.pdf, https://documents.epfl.ch/users/f/fl/fleuret/www/dlc/dlc-slides-5-init-optim.pdf
• About network initialization: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
• Neural nets can fit anything: https://arxiv.org/pdf/1611.03530.pdf
• Small batch generalizes better than large batch: https://openreview.net/pdf?id=HkmaTz-0W

28.2

• Details from Understanding the difficulty of training deep feedforward neural networks
• Tricks for deep architectures:
  • Good initialization (see paper above)
  • Batch norm (see last lesson)
  • Relu units
  • Residual networks (just google it)
• Talking about vanishing and exploding gradients On the difficulty of training recurrent neural networks
• LSTM, GRU, IRNN
• Highway networks

6.3

• Generaliation in neural networks and dropout: slides, slides 19-25.

7.3

• Language generation: char-RNN
• Machine translation. Best course for that is CS224n. Relevant slides are 1, 2, 3

13.3

• ResNets
• Making convolutions smaller Inception, more inception
• Object detection, more material

14.3

• making nnets faster

20.3

• Generative adversarial networks
• Wasserstein GAN, Wasserstein GAN 2

21.3

• Introduction to Bayesian modelling
• Mostly stuff from here

27.3

• Sampling methods for probabilistic models
• Rejection sampling
• Importance sampling
• MCMC sampling

28.3

• MCMC sampling in more detail
• Metropolis and Metropolis hastings
• Gibbs sampling

4.4

• PyMC demos

10.4

• Reinforcement learning
• slides 1, slides 2

11.4

• Reinforcement learning demos

17.4

• Graphs in ML
• Smart reply for google inbox

18.4 and 24.4

• Graphs in ML course
• Selected topics from lecture 1-4

2.5

• no lesson

9.5

• Neareast neighbours
• nmslib

15.5, 16.5

• lecture canceled. If you have any questions about project send me an email and we can meet.