Not to express an endorsement in either direction, but you may be interested in the finitist school of thought: http://en.wikipedia.org/wiki/Finitism
or its even stricter cousin, which not only denies the meaningful existence of infinity, but also the meaningful existence of technically finite but infeasibly large numbers: http://en.wikipedia.org/wiki/Ultrafinitism
That said, I will point out that math is essentially a game played by certain rules known as "axioms", and there is no one true set of axioms. If you choose one set, you get infinity; if you choose another, you don't (but you will pay for that). You can't say which is "more right" or "more wrong" without bringing other subjective standards into it, each of which may have their own utility. While "corresponds to the physical universe" is definitely one useful such standard, it is still not the only one.
Do read up on the limits of computable numbers though; they are interesting, but it's not a free lunch.
or its even stricter cousin, which not only denies the meaningful existence of infinity, but also the meaningful existence of technically finite but infeasibly large numbers: http://en.wikipedia.org/wiki/Ultrafinitism