I frequently use TODOs and I don't get crazy if I don't go back and do all the TODOs. Some seem important at the time and become less so with age. Others become more important.
I feel like this leads down a dangerous path of no signal-to-noise ratio and destroys your hope of finding relevant TODOs. Better to, as part of revisiting them, rewrite the comment to explain the de facto behaviour for yourself (and others) in future. If you're not going to do them, anyhow.
You make good points. I don't use them so much as to bother me. I code for a team size of one (me) so that gives me liberties that I might not take elsewhere.