It really doesn't matter until one tries to enforce an AGPL provision. If you do, that's where as you say will be arguments whether one added something substantial enough; basically the AGPL terms can really only be enforced on those additions.