Impressively concise implementation, really interesting paper! Benchmark looks quite questionable though -- e.g. they use fp64 (while any sane person would use at least f32, if not f16), batch 1 (while normally one would try to get the max batch size which fits into memory, and it would reach 1 only for much bigger models or inputs), and also measure the time including transfer to/from GPU (while it would normally be interleaved with GPU operations). Not sure what results would look like in a more realistic setup, but still getting within 2x of PyTorch even in such a setting looks impressive!
Interesting Futhark mention as well in the Related work section:
> Another approach to GPU-based array programming with
an APL focus is the TAIL/Futhark system [8], which is a
compiler chain taking APL to the TAIL (Typed Array Inter-
mediate Language) and then compiling TAIL code using the
Futhark GPU compiler backend.
Why? As a learning experience implementing backpropagation is extremely helpful, implementing an entire FNN/CNN from scratch is, to be honest.
Also Implementing some basic automatic differentiation is something you should probably have done once in your life you are interested in Machine learning or numerical mathematics.
Agreed. Implementing backprop myself—even if it was a crappy, slow version in MatLab—is what finally got me to understand it. I've worked as an ML researcher for 2 years since then and I'm still routinely happy that I have that deeper understanding of what's going on under the hood of the models I'm training.
I agree that implementing autodiff is nice exercise to grasp the whole stack. But its a ladder that has to be thrown away after you climbed it. In other words, write that autodiff for yourself, and then use production grade ML solution where you won't ever write backward pass by hand.
The APL paper doesn't have implementation of autodiff, rather 90s style hand derivations.
Yes, every ML grad student should write autodiff lib (ideally using dual numbers). But nobody should ever write backward pass for whole net. Researchers did that in 80s and 90s. Once we had autodiff, it became obvious that deriving backward pass by hand is no-go.
