I think PCA is better understood in theorem-proof style of mathematics rather than algorithmically. For example, PCA can be understood as an optimization problem. Whereas CART and k-NN and the likes are best understood algorithmically. I know k-NN was introduced as an algorithm and only later figured out in terms of probability, etc..