My learnings from attending a 3 hour session on Deep Learning :

Code Links

GitHub repo: https://github.com/NVAITC/pycon-sg19-tensorflow-tutorial Workshop slides: https://drive.google.com/open?id=1RR0UhnvJ3PHL4sGRe2du4_w66Kg9KNVr

Contents

  • Everyone is doing ResNet
  • Transformers - Cutting edge models in NLP
  • Rapids.ai - Lead by NVIDIA - Uses GPU
  • ArXiv - 25000 papers every year
  • AI’s compute > Moore’s law
  • Changes in TF2.0
  • No more graphs and sessions - Use Keras API + tf.function decorator
  • tf.contrib – Tensor flow add-ons
  • Eager execution
  • Functional API solves 99% of the usecases
  • Model Subclassing is possible - Autoencoder
    • Encodes in a dense vector
    • Decodes in to a transformed vector
    • Choose encoder or decoder separately
  • Standalone keras will be stopped development after 2.3
  • Send the data faster to GPU so that there is less time spent on sending data
  • tf.data is quick - Data flows efficiently to GPU
  • Idle time on GPU can be reduced
  • tf.data can help you do a parallel processing
  • shuffle, map, batch, prefetch - Datasets that cannot fit in RAM
  • Loading code is same -
  • Keras automatically runs on GPU
  • XLA - Little performance boost if you are using standard architectures
  • What parts of the model can be in 16 bit ? 32 bit ?
    • TPUs - Mixed Precision
  • What is the policy that keras should use for precision ?
  • Tensorflow lite works for mobile devices etc
  • XLA might not give a perf boost
  • Model that cannot be fit on one GPU can be split on multiple GPUs - Quite rare
  • Horovod - Train on one GPU script - Magically scale it to multiple GPUs
    • Takes control of optimizer and makes scaling possible
    • Automatically used relevant libraries to do it
  • tf.distribute allows you to distribute to multiple GPUs
    • You can set distribution strategy to place model on different GPUs

Takeaways

  1. I don’t have to parallelize my code at all. I use the GPU and do it one time series after another. Instead of trying to parallelize stuff. Keras automatically uses all cores on the machine
  2. Access to a bunch of notebooks that I can use to learn ML on TF2
  3. Broad understanding of tf.data
  4. Broad understanding of XLA and Automatic Mixed Precision
  5. Understanding of basic structure of colab
  6. Got inspired by the undergrad who had really put in a lot of effort in DL