One of the things I've tried to do differently in Arc is to design the language based on what actually happens in programs, rather than a priori theories. For example, let is for the one-variable case because I looked at a lot of CL code and found 76% of the lets bound one variable. Cond doesn't have implicit progn because I found most cond clauses didn't use it. Etc x 100.
So in reply to the a priori argument that shadowing global function names will cause problems (How big? I don't know. Big!) I reply that extensive experience has shown that it's just not a big deal.
I'll add modules eventually, of course. But it's not a high priority.
I didn't say it was a big problem, I said it was a real problem. And the larger your code base, and the more people start sharing code, the more real it will be. It is only a matter of time before someone writes something like:
(let foo snoz (jims-macro frotz))
and gets an error of the form "42 is not a valid function" and has to spend an hour puzzling over where that error came from before realizing that Jim's macro expands into Bob's macro which expands into Ann's macro which uses a helper function called foo.
Is it a "big" problem? Maybe not. But it's a real problem, and given that one of your stated goals was to design "the 100 year language" I think it's a shame that you chose to solve it by placing the burden on the programmers. (And I use the plural advisedly here, because the potential for encountering this problem grows as the community grows and people start re-using each other's code.)
But I would reiterate that the lack of abstract data types is IMO a much bigger problem than the lack of hygienic macros. (And just so you know I'm not just sniping from the sidelines, I have proposed solutions implemented in CL which I'd be happy to help port to Arc if you agree that this is in fact a problem.)
I'm not a lisp programmer neither I know the peculiarities of each lisp implementation, but I think I understand it a little bit.
My question is, what hinders Arc (or other Lisp if it has the same problem) to create an operator that finds any global symbol from the top, like when an absolute path starts with "/" to find a file?
For instance, let's stay that there is an "unshadowable" operator called "." and let's take your example:
(let foo snoz (jims-macro frotz))
Well, that shadowed "foo" is causing the problem. As you said ("Jim's macro expands into Bob's macro which expands into Ann's macro which uses a helper function called foo."), this symbol is shadowing the actual function "foo" used by Ann's macro.
However, if Ann's macro were written carefully (and all macro writes should know that macros can be used in any context, any scope), Ann would care to call "foo" from the top using the "." operator, like:
...
(. foo args) ; excerpt of Ann's macro.
...
In this case, in every "(let)" or another scope definition method, you could always escape to the global scope using this operator, behaving like a hole to the outside world.
Of course, I'm just a Lisp newbie, maybe I'm just creating a huge mess with the concepts :-).
That's kinda the right idea. There are two problems with your specific proposal: 1) it places a considerable burden on the programmer to annotate every invocation of a top-level function. For complex macros that can become quite annoying. 2) Baking the concept of a single top-level into the macro system makes it hard to add modules later.
If you pursue the idea of having the system walk the code and automatically insert those annotations you will (almost certainly) end up re-inventing hygienic macros.
(For the record, I much prefer the Lisp2 solution, because it's the hacker's solution. It's not mathematically elegant, but it's simple (compared to hygiene) and it gets the job done.)
How did you arrive at these conclusions? Did you use some programmatic analysis of CL source code to report these stats? Which codebases were your reference?
Yes. I used all the CL and Scheme source of my own that I could find, and I also asked Ken Anderson to analyze some big collection of CL (I think it was) code at BBN.
I'm sure I'm not a good representative Lisping example, but I routinely use let to bind more than one variable, especially when extracting program state from a URL in multiple variables, and probably use the implicit progn in cond 70% of the time. I can see the value of not allowing the implicit progn which would force creating more numerous and smaller functions, but in the case where I have only 2 expressions in a cond, a function just for that would seem a bit verbose. But again, this is really no big deal as with ARC, one can write a macro utility to put the multiple option back in.
So in reply to the a priori argument that shadowing global function names will cause problems (How big? I don't know. Big!) I reply that extensive experience has shown that it's just not a big deal.
I'll add modules eventually, of course. But it's not a high priority.