AI in 2024

Table of Contents

Intro

As of this writing (April 5 2024) if you look at arxiv.org submissions for Machine Learning since April 1 there is 625 papers. In the past 12 months there was 17,468 papers just on on Neural Networks (Deep Learning). The field changes so much there is no way to keep up unless you can teach yourself how to read these papers and implement (optimize) them. We will learn here how exactly to do that because there is one constant that doesn't change and that's the original theory how all this works. There is also a resource called Depth First Learning where curriculums are built around papers so we can understand them written by the researchers of the paper themselves and try and implement them.

Tour of university AI courses

I've audited every public 'Intro to ML' or 'Intro to Deep Learning' course that can be found with a bot and honestly none of them teach you more than what you would learn doing the classic Andrew Ng course. If you insist on taking a typical undergrad course here are some of the better one's:

A totally different perspective:

  • Berkeley's CS188 which covers the old school Peter Norvig AI book
    • If you can't find all the 2024 videos look at 2023

I omitted a ton of other courses like ETH Zurich, Harvard, Berkeley, Waterloo, and countless more because it's just the same content at different levels of confusion.

Tour of Applied AI courses

The scikit-learn core developer Andreas Mueller taught this course at Columbia in NY and it's definitely one of the best introductions to software frameworks after you learn all the theory. Another is this Neural Networks/LLMs crash course by Andrej Karpathy. My favorite course is Scientific ML which is an applied course on optimization for physics-informed AI aka 'expert-guided AI'.

Biggest open research problem is reducing hardware overhead of neural networks that means redefining floating point, fine control of garbage collectors for AutoDiff and optimizing numerical linear algebra libraries. You probably never imagined the cutting edge AI research in the future would be in numerical methods for specialized integer hardware and ASICs but alas here we are.

Limits of current AI

Current popular AI is all foundation models like GPT-n, DALL-E etc. Have you wondered if we had infinite resources, infinite data, perfect training algorithms with no errors, can we use this model for everything? Someone from the Beijing Academy of AI used category theory to model this scenario to see what is possible.

Strategy

  • This playlist of theoretical foundations and book
    • Same book is still used in recent PhD courses

If you flip through that pdf it looks absolutely incomprehensible but the magic of the course we are doing is he walks through all the math and explains it. Then we can try a deep learning paper from Depth First Learning and implement it with (non)convex optimization.

While we finish the book/lectures I will go through some geometry-centric linear algebra videos because honestly they are best resource anywhere to see in your mind what is going on such as hyperplanes. Anything not covered will be picked up in the Depth First Learning content.

TODO setting up every workshop so I can just execute 1-3 hrs of work a day and finish everything here.

Home