Astle's stream

ML/DL/AI subfields: present and future

Long gone are the days were ML students used to code up RNNs and CNNs in order to utilise their very own models on tasks. Transformers put an end to that, why ? Scalability. Transformers only outperform the other architectures when scaled, and hence the average individual person could only stand in awe as millions of dollars, thousands of GPUs and >terabytes of storage was used in order to train foundation models. This paradigm shift is reminiscing of technology which is beyond the individual. However, as any field matures, one can find niches to lodge our efforts into, and hence I have gathered a few possible paths here. These maybe obsolete or solved in the upcoming years, but I shall not remove, but only extend this list, to track the growth of this field:

• Mechanistic Interpretability

  • Scaling SAE
  • Sparse Models
  • Decomposition with Dictionary Learning
  • Practical Application
  • Gemma Scope

• Parallel/Distributed Training

  • Megatron LM
  • Parallelise Computational Graphs
  • Deep Distributed Learning
  • CUDA
    • CUDA Blog
    • Nvidia Guide
    • Course/Roadmap for CUDA

• Distillation of LLMs

  • Curse of Capacity Gap
  • Knowledge Distillation
  • On-Policy Distillation

• Reinforcement Learning

  • OpenAI's Site
  • Neural MMO
  • Reinforcement Learning Theory
  • Introductory Papers [1, 2, 3, 4]

• Automated Design

  • First Paper