Macros achieve a limited version of a fexpr with much more complicated semantics. Hygienic macros take that a step further - you get implicit renaming of variables which is generally convenient - in exchange for even more complicated semantics.
The abstraction leaks as well. `(reduce and (list #t #t #f))` is probably an error. Calling a macro and calling a function looks the same and sometimes is the same and sometimes falls over.
Macros are sacrosanct to lisp. Hygienic macros are a crown jewel of scheme. They're still the wrong thing though.
In the beginning there was dynamic scope, possibly by accident. There were also fexpr - pass the unevaluated data and the environment, instead of passing evaluated data, which were also dynamically scoped. Sometimes called nlambda. That was difficult to program with and very difficult to compile.
Macros are easier to use than dynamically scoped fexpr. They're much easier to compile and lisp was getting a bit of a kicking for being too slow. There were some papers written, macros were the better choice, and here we are.
Lexical scoping turned up a little while after macros won the fight. Lexically scoped fexpr with first class environments are the right thing. Simpler and more capable than macros. First class environments being another thing sacrificed on the alter of performance decades ago.
Shutt noticed this and wrote about it at length. It's slightly subtle that a non-hygienic macro is a subset of fexpr. Force inline it at the caller, refuse to run various calls (e.g. pass it to functions), wrap the return value in an eval in the caller environment. A hygienic macro has to do reflective symbol renaming stuff which is also expressible as a fexpr by implementing the symbol renaming rules of scheme, which thus far I don't have the patience for.
Qualified as minor, in that the wrong thing we have is still very useful.
Disaster in the sense that we could have fexpr if history had happened in a slightly different sequence.
Opportunity in that lisp can be better. First class environments and first class macros can be done. The performance constraints of the past have been removed by hardware progress and to a lesser extent by progress in compiler design.
> Macros achieve a limited version of a fexpr with much more complicated semantics
Neither of these ideas rings true.
While there are things that fexprs can do that macros cannot, the reverse is also true.
Macros have the opportunity to work with the source code before it is executed. Macros can issue diagnostics. Also, macros have access to the compilation environment, which can be an entirely separate machine from the run-time environment. Macros can bring in materials from auxiliary files, which would not be found on the target.
A system build on fexprs rather than macros should incorporate the concept of static checking functions. For each fexpr, two additional functions should be required:
- a function which knows how to statically code-walk the form which is the target of the fexpr. Perhaps of this form:
(lambda (form recurse-fn)
;; function identifies, inside form, all expressions that are
;; forms and calls (recurse-fn subform) for each such subform
)
- a function which knows how to check the form for errors, so the programmer can be informed about misuses of the fexpr before it is actually called on the target system.
Unhygienic macros do not have complicated semantics. However, it should be acknowledged that hygiene is more naturally achieved in fexprs without effort: the fexpr's own local variables and functions are very clearly separated from evaluations of argument material, that being done by an explicit eval call, using the correct environment passed into the fexpr. A macro's local variables are likewise clearly separated, but those are expansion time. A macro has to inject new variables into the generated code which a fexpr doesn't have to do: a fexpr can use its own local variables to hold run-time temporaries related to the calculation, whereas a macro cannot do that: for run-time temporaries, it has to generate code, combining that with the argument material. Those temporaries are then in the same scope, and have to be gensyms.
Thank you for disagreeing. The relative complexity of macros vs fexpr is interesting and your reply uncovered unspoken assumptions in my thinking.
Common lisp's read macros are indeed a different beast. Mutating the bytes pulled from file before parsing them is more powerful than operating on the lists of symbols.
Macros imply a phase separation which fexpr do not. The compile time diagnostics and similar are an artifact of that phase separation, not of evaluation sequencing.
If one wants to build in that phase separation, I think it follows that unhygienic macros are the simple option. Hygienic macros still carry much implementation complexity.
If one wants hygienic behaviour, fexpr do that exactly as easily as lexically scoped lambdas, because they literally are lambda which pass the argument forms unevaluated. I implemented that as a bool tag on the function affecting calling convention - do you map eval across the arguments or not during the function call - and that's sufficient.
If the desired behaviour is variable capture from the calling environment, it takes more effort from a fexpr than from a macro. You write (eval env 'foo) instead of ,foo or similar.
I don't see a particular need for a code walker - in the simple case the argument gets passed around a little then evaluated, in the complicated case you need to walk the arguments the same way a macro would want to.
The phase separation thing I had forgotten about. As in warning messages during compilation so that you have invariants at runtime. Macros fit that model better. Fexpr and compilation are an uneasy combination.
I also don't see a nice way to implement something like with-slots as a fexpr. This relies on symbol-macrolet.
(with-slots (a b c) obj
;; a b c refer to slots (assignable!)
)
This has to evaluate obj to some hidden variable #:obj, and then replaces all occurrences of a, b, c with (slot-value #:obj 'a), etc.
I don't see how the fexpr can do that without performing the same replacement.
a, b and c are not lexical variables you can just bind and then evaluate the body. They are aliases: assigning to a sets slot a of the object.
One way is for the fexpr to contain its own eval implementation which takes an extra macro environment, indicating that when a, b and c are seen, alternative forms are to be evaluated in their place.
That just adds up to an inefficient implementation of symbol macros.
Also, how are places implemented. In particular, user-defined places. How can we have a set of fexprs like setf, push, incf, ... which can work with with arbitrary syntax denoting a place, including application-defined places.
Overall I agree: yes, with fexprs you lose some code introspection ability compared to macros. I haven't found it to be a big deal in my fexpr-based hobby Lisp so far.
Re your two points:
You could have "symbol fexprs", analogous to symbol macros, I guess.
For places I think the first-class solution as employed by T and others is better, and would work fine with fexprs:
(set (name-of person-1) "sam") simply stands for ((setter name-of) person-1 "sam").
IOW, name-of is expected to be a reader function. Every reader function has a writer function attached to it, that we extract with (setter name-of). Then we call that writer function with the rest of the original arguments.
That's a fun one. symbol-macros don't have an equivalent in lisp functions as far as I'm aware - `(symbol-lambda-let (x (lambda () (list foo x)) (list z z))` ~= `(list (foo z) (foo z))` isn't a thing.
Speculate that such a thing did exist. Call it symbol-lambda for less confusion with the symbol-function that does exist in common lisp. It's a function that takes no arguments, being a distinct premise from a function that takes the empty list. `(apply func some-list)` would be an error. But one wishes to call it implicitly when it shows up in lists. Related to and distinct from thunks.
Thus it's a function executed by eval, not by apply. Specifically if eval calls into eval-symbol when called on symbols, one way to go is perform the environment lookup etc as usual, and then afterwards check whether the thing one found in the environment is a symbol-lambda. Something like:
It doesn't take arguments, the same way symbol-macros don't take arguments.
The fexpr equivalent could be identical (since it's mostly distinguished from lambda by not evaluating arguments and there aren't any), or maybe the fexpr form gets passed the environment and the lambda form gets passed the empty environment instead on efficiency grounds.
I'm not totally convinced. Its straightforward to add to an fexpr-based evaluator. I think the proper model is that it's a function with a different calling convention to lambda and to fexpr. In the same way that eval-on-list tends to end up in a tail call to apply, eval-on-symbol can end up in a tail call to symbol-apply which usually returns the argument unchanged.
However passing one of these things directly to eval, as opposed to passing a symbol bound to one to eval, is also possible. So eval picks up a third case for eval-on-function, where fexpr and lambda get returned unchanged, but symbol-lambda/fexpr/name-tbd get evaluated.
Also the evaluation of the symbol-lambda might need to know what name it was invoked through. That usually isn't the case and interacts poorly with renaming variables. That also distinguishes passing it to eval as a function from calling it by writing down a symbol that resolves to one.
Overall this leaves me with a sense that I don't quite have the proper decomposition worked through yet.
Your thinking almost exactly parallels mine about this topic. I formed the same idea about lambdas that take only an environment being bound to symbols and whatnot.
I got to something resembling this part:
> With-slots then turns into something like:
That's when I hit the problem that "turns into" is macro code generation!
If the fexpr has to do code generation on the fly and eval it, it's doing a macro's job, badly, since the macro can do it upfront.
