My Learnings from attending PyCon SG19 Tutorial on Deep Learning
Contents
My learnings from attending a 3 hour session on Deep Learning :
Code Links
GitHub repo: https://github.com/NVAITC/pycon-sg19-tensorflow-tutorial Workshop slides: https://drive.google.com/open?id=1RR0UhnvJ3PHL4sGRe2du4_w66Kg9KNVr
Contents
- Everyone is doing ResNet
- Transformers - Cutting edge models in NLP
- Rapids.ai - Lead by NVIDIA - Uses GPU
- ArXiv - 25000 papers every year
- AI’s compute > Moore’s law
- Changes in TF2.0
- No more graphs and sessions - Use Keras API + tf.function decorator
- tf.contrib – Tensor flow add-ons
- Eager execution
- Functional API solves 99% of the usecases
- Model Subclassing is possible - Autoencoder
- Encodes in a dense vector
- Decodes in to a transformed vector
- Choose encoder or decoder separately
- Standalone
keras
will be stopped development after2.3
- Send the data faster to GPU so that there is less time spent on sending data
- tf.data is quick - Data flows efficiently to GPU
- Idle time on GPU can be reduced
tf.data
can help you do a parallel processingshuffle, map, batch, prefetch
- Datasets that cannot fit in RAM- Loading code is same -
- Keras automatically runs on GPU
- XLA - Little performance boost if you are using standard architectures
- What parts of the model can be in 16 bit ? 32 bit ?
- TPUs - Mixed Precision
- What is the policy that keras should use for precision ?
- Tensorflow lite works for mobile devices etc
- XLA might not give a perf boost
- Model that cannot be fit on one GPU can be split on multiple GPUs - Quite rare
- Horovod - Train on one GPU script - Magically scale it to multiple GPUs
- Takes control of optimizer and makes scaling possible
- Automatically used relevant libraries to do it
tf.distribute
allows you to distribute to multiple GPUs- You can set distribution strategy to place model on different GPUs
Takeaways
- I don’t have to parallelize my code at all. I use the GPU and do it one time series after another. Instead of trying to parallelize stuff. Keras automatically uses all cores on the machine
- Access to a bunch of notebooks that I can use to learn ML on TF2
- Broad understanding of
tf.data
- Broad understanding of XLA and Automatic Mixed Precision
- Understanding of basic structure of colab
- Got inspired by the undergrad who had really put in a lot of effort in DL