Well these technologies are there to help bring Google type infrastructure for businesses that need it. If you're running a CRUD app just fine on Heroku you don't need to do any of this and you shouldn't.
When your availability starts having problems or your data is getting too much for one machine, you start having problems of scale. Where you are in terms of scaling issues should lead you to the next iteration of technology required to keep your service up.
Starting out of the gates with that one idea you're not sure anyone will be interested in? Just stick to deploying something fast and not worry about Google style scaling solutions.
Seems to be the problem with a lot of technologies. If you're a HN reader, you can start to get the feeling that you're falling behind if you're not doing these things. Even though they are totally inappropriate for the majority of us who are working on smaller systems.
Often there's some tension between individual career development and what makes sense for the project and company.
Ideally, managers create opportunities for people to exercise the former in a sandbox (e.g. some variation of "20% time"), without YAGNI-ing up the project.
I like the fact that I know about a lot of these things, so that if a need arises I'm aware of what's out there and current. I figure when/if the time comes then I'll actually learn how to use them. I think it's just more of a perception that everybody else is moving forward and I'm stuck here in pragmatic land for my work.
As an engineer I feel an obligation to know about these things and how they work. It might be appropriate for my work. For my own projects I am just going to start out with a Django or Meteor app and if I'm lucky enough to have to worry about scale I'll have some idea of what to do next.
so might be AS400, but few are trying to learn that I guess.
Let's be fair, it's quite likely that if anyone is looking into this stuff is more because it's the technology du jour, rather then the off chance that they end up with 100 million users.
It could be useful, and learning stuff is fun, but that's not the real drive.
As a sysadmin, I feel an obligation to know about these things so I have sufficient warning of the next trendy maintenance burden that's coming down the track. This is how I credit my HN reading as real work ;-)
When your availability starts having problems or your data is getting too much for one machine, you start having problems of scale. Where you are in terms of scaling issues should lead you to the next iteration of technology required to keep your service up.
Starting out of the gates with that one idea you're not sure anyone will be interested in? Just stick to deploying something fast and not worry about Google style scaling solutions.