The Lean Startup advocates proportional investment in solutions. WHEN the problem comes up (again, after deciding to do this) determine how much your time percentage wise this took out of your week or month. Invest that amount to fix it, right now. My interpretation would be, spend that time trying to solve part of it. Every time that problem comes up keep investing in that thing, that way if you've made the wrong call you only waste a small portion of your time. But you also are taking steps to mitigate it if becomes more of an issue in the future.

Having gone down several myself, I can say it's hard. You lose time. You have to accept you've lost time and learn how not to do it in the future.

My advice is to collaborate with people who are much, much smarter than you and have the expectation that things actually get done because they know they could do it. You learn what productivity looks like first, at the most difficult and complex level you're capable of.

That sets the bar.

Everything has to be equal to or beneath that unless your experience tells you you'll be able to do something even greater (possibly) with the right help or inspiration

You gain experience by going down similar rat holes, until you feel that you can adequately compare the situation you are in now to an experience in the past.

You'll still be wrong, but perhaps less often.

For many particular examples, there have already been enough rathole spelunkers to provide useful data. Maybe start looking in the places where there isn't already useful data?

Any area in which enough such spelunkers are found is unlikely to be significantly improved by adding your own effort.

Agreed. It's often much, much harder to articulate why an idea is bad or a rat hole. You just move on.

I've come up with explanation by analogy. You can demonstrate quite easily in mathematics how you can create a system of notation or a function that quickly becomes impossible to compute. A number that is too large, or an algorithm that would take infinity amount of time and resources to solve...

It seems to be in nature that bad ideas are easy. Good ideas are harder, because they tend to be refinements of what already exists and what is already good.

So pursue good ideas. Pursue the thing that you have thought about and decided has the best balance between values and highest chance to succeed. Sometimes it's just a strong gut feeling. Go for it, but set limits, because you don't want to fall prey to a gut feeling originating from strong intuition but an equally strong lack of fundamental understanding.