AI in 2025

Current popular AI is all foundation models like Grok-n, GPT-n, DALL-E etc. Have you wondered if we had infinite resources, infinite data, perfect training algorithms with no errors, can we use this type of model for everything aka 'General AI'? Someone with help from the Beijing Academy of AI used category theory to model this scenario to see what is possible.

There is no shortage of excellent linear algebra courses. These are the one's I will do here:

• Coding the Matrix (Brown)
  • All the recorded lectures are open to anyone
  • Programmed w/Python and has a (nonfree) book

Created by Philip Klein a name you will recognize if you take any algorithms course. Probability and the SVD is covered which we'll need. The book is at least cheap ($35) and worth buying or you can use Anna's Archive or whatever the latest Library Genesis domain is to get a pdf but it will likely be missing a lot of graphical content. This is a very good course for anyone interested in game graphics or taking an algorithms design course and wants to manipulate graphs using linear algebra. If you hate everything else here then do this you'll be fine.

• Linear Algebra Done Right - Sheldon Axler
  • New completely free 4th version
  • He upgraded his cat too that was always in the about author pic
  • Abstract treatment which we need but is considered a second course
  • Contains some calculus
  • Seems designed to prepare students for functional analysis (and thus ML)

Neel Nanda says we should we do this and it can be completed in parallel with the Klein book.

We can audit some of the slides from MIT OpenCourseWare then the Matrix Calculus course will teach us the behavior of derivatives.

Reading lecture 1. If you want worked through examples then watch this from Clemson's MATH 1060 but this will all be properly explained in higher dimensions when we do calculus in the realm of linear algebra.

The answer to 'what is a tangent line exactly' is at that point P on the curve, super zoom until it's neighboring points appear to lay on a straight line. The slope of a tiny straight line between P and P + 0.001 is the tangent. Then you imagine doing this for the entire function creating many tiny little straight lines for every point in the function which is a linear approximation of that curve/function.

The geometric definition pretend 'delta-x' \(\Delta x\) is a tiny displacement dx = 0.001 then imagine P + dx distance shrinking as dx approaches zero. Here is a graph (desmos.com) showing what is going on. Notice on that graph in order to go from (2, 4) to (2.001, 4.004) move to the right along the x-axis 0.001 then up 0.004 to meet the graph again. This is what the derivative tells us that if a function is preturbed by some tiny extra input f(x + dx) how sensitive is this function when we analyze the change in output.

Example 1 of the MIT notes for f(x) = 1/x

• Plug into derivative equation
• Extract out 1/delta-x
• Perform a/b - c/d is (ad - bc)/bd
• Cancel the extracted delta-x leaving -1
• Take limit to zero of delta-x
• - 1/x²

Why is it negative? Look at the graph if 0.001 is added to the x input then the y output has to drop down -0.5 to meet the graph again of f(x) = 1/x

Finding the tangent line the equation for a line is y = mx + b and they have merely substituted in values but we will never use this feel free to skip it and the area computation after.

Try inputs to f(x) = x² to understand Big-O

• (2 + 0.001) = 4.004001
• (3 + 0.001) = 9.006001
• (x + dx) = y + dy + 0.000001
• (x + dx) = y + dy + O(dx)²

If dx is approaching zero then (dx)² will reach zero before dx and can discarded.

Those above inputs to f(x) = x² notice that dy = 2dx + (dx)² and if we ignore the (dx)² then dy/dx = 2 which is the derivative at that point. The derivative of the entire function is dy = (2x)dx or dy/dx = 2x.

'dx' means 'a little bit of x' and dy means 'a little bit of y' it is infintesimal notation where an infintesimal is some extremely small positive quantity which is not zero.

Reading lecture 2. Watch left and right limits and these techniques for computing limits such as multiplying by a conjugate which is how you eliminate square roots from limit calculations.

A limit was already described perfectly by Isaac Newton as the ability to make the difference between a quantity and a fixed value less than any given positive number. He called these the ultimate ratio of 'vanishing quantities' where the ultimate ratio is not before they vanish or after but the ratio with which they vanish.

These MIT notes tell us if f is differentiable at a point then f is continuous at that point which of course makes sense if we go back to the definition of a derivative being a linear approximation where around any super zoomed point is a tiny displacement point on a straight line (tangent).