Cédric Champeau
2014-10-07 17:20:32 UTC
Hi everyone,
You may know that we have long been complaining about the complexity of
AST transformations in Groovy. There are a very powerful tool, but it is
still a bit complex to handle. In Groovy 2.4, we planned to integrate
work from Sergei Egorov (@bsideup) that he called "Groovy macros".
Basically the idea is to have a simple way to replace classic method
class into more complex expressions. In this email, I'll try to
summarize the project and give some ideas about the orientations we want
for the Groovy language. In the end, your opinion matters so please feel
free to comment.
First of all, Sergei actually worked on two "macro" projects. The first
one consists of a "macro" block that can be illustrated by this:
defsomeVariable=newVariableExpression("someVariable");
ReturnStatementresult= macro {
return newNonExistingClass($v{someVariable});
}
As you can see, it is some kind of "super AstBuilder" which is capable
of handling variables/expressions from an external context thanks to
proper escaping with $v. This macro code is primarily aimed at being
used inside AST transformation themselves, and dramatically reduce the
amount of code required to generate an AST tree. An advantage of those
macro blocks is that if you use them in AST transformations, you're
actually not limited to expressions. You can generate anything, as long
as what is inside the macro { ... } block is supported by the Groovy
syntax. A drawback of the current implementation is that it uses a
global AST transformation. So as soon as you have the macro project on
classpath, every single method call corresponding to "macro" will be
interpreted as a macro block. There are multiple disadvantages of global
AST transformations. First of all, they are, as the name says, global,
meaning that they apply independently on *every* class being compiled by
Groovy. This also means that the transformation is run even if you know
the code you write is not using it. In particular, we need to inspect
the full AST, in-depth, to find potential method calls named "macro"
even if there's not a single one in the code (because you will only know
once you have visited the full AST). In short, a global AST
transformation has a clear performance impact, because it is executed
independently of the context. For the macro stuff, an easy workaround
would be to transform the global AST transformation into a local one.
For example, an AST transformation using the macro system could declare
it by annotating with @EnableMacros. I think it is reasonable, since it
is very unlikely that you would use the macro { ... } stuff in regular
code. The macro block is indeed useful, but in the end, it is limited to
writing other AST transformations that would be either global or local.
In short, the current macro { ... } block is useful for AST
transformation designers, but cannot be used by "regular" Groovy users
to define new language constructs.
In answer to that problem, Sergei worked on a second implementation of
macros named @Macro. The idea is both simple and elegant. Just like you
can define extension methods in Groovy (see
http://docs.groovy-lang.org/2.3.7/html/documentation/#_extension_modules),
you can write a macro like this:
public classTestMacroMethods {
@Macro
public staticExpression safe(MacroContext macroContext, MethodCallExpression callExpression) {
returnternaryX(
notNullX(callExpression.getObjectExpression()),
callExpression,
constX(null)
);
}
}
and for this to work, TestMacroMethods needs to be declared as a regular
Groovy extension module. In that case, *any* code using "safe" would be
transformed, so:
safe(x.foo()).bar()
will be expanded *at compile time* into x.foo?x.foo().bar():null
A more interesting example with pattern matching can be found here:
https://github.com/bsideup/groovy-pattern-match/blob/master/src/main/java/ru/trylogic/groovy/pattern/PatternMatchingMacroMethods.java
An important thing to understand is that the @Macro annotation is *not*
an AST transformation. Instead, it's a marker annotation which is looked
up at compile time, for each method call. So for each method call of the
AST, we try to find if an extension method node of the same name exists,
is annotated with @Macro and takes MacroContext as the first argument
and in that case, the original method call is transformed thanks to the
extension method at compile time. The code of the macro itself is
regular AST transformation code. It does *not* use the macro stuff from
the first implementation, hence doesn't simplify writing xforms. On the
other hand, it greatly simplifies the availability of transforms by
making them writable as simple extension methods, without the need of
ceremony (annotations). It is actually very cool, but it comes at a
price. Performance wise, it is still a global AST transformation, but
worse, for each and every call, it implies finding an extension method
node. It can (and will) cost a lot, but we can improve the situation by
introducing specific caches. Moreover, it also implies that the sole
fact of adding a macro "jar" on classpath could potentially affect the
semantics of your program: if a method call in your code matches the
name of a macro method call, then it would be applied at compile time.
Of course, one could argue that it is already the case for extension
modules, but the risk is lower: for extension modules, the only cases of
conflicts is when the receiver of the message matches the class of the
extension module. For @Macro, any method call on "implicit this" would
be matched. This draws the point of whether the @Macro stuff should also
be combined with a @UseMacro local AST transformation, in order to avoid
the problem of the global one.
If we do, then definitely, the performance issue is not one anymore,
because only marked code would be transformed. On the other hand, we
introduce ceremony again, which would mean that we loose the benefit of
extension methods being transparently visible. In the case of the
pattern matching stuff, this would mean that you would have to
explicitly annotate your code to enable the feature. Is it a problem?
I'm not sure actually. Also it's worth noting that the global AST
transformation issue is less of a problem if you have lots of macros on
classpath, because a single transformation would apply them all in a
single pass.
Last but not least, my feeling is that what we need is something in
between the first macro implementation and the second one. In
particular, I would like to be able to write the @Macro method body
using those macro { ... } blocks. It makes a lot of sense to me. Next, I
wouldn't like to be limited to expressions. I think a macro should be
able to produce any AST node. This implies expressions, but also
statements or even full methods.
I gave an example a little contrived, I admit, to Sergei, which is,
imagine that I want to generate two methods with the same body, but
accepting two different argument types. Then I could write a macro that
generates the method:
@Macro MethodNode createMultiply(MacroContext ctx, ClassNode argType) {
macro {
_argType_ multByTwo(_argType_ x) { 2 }
}
}
Then in a class I could write:
class Calculator {
createMultiply int
createMultiply double
}
Of course, this wouldn't compile because the grammar wouldn't allow
defining a method in a closure (in the macro block), and it would not
recognize the createMultiply calls directly in the class body, but I
kind of like the idea of a macro system that just allows expanding and
reasoning at the AST level anywhere, because it lets us create new
language constructs.
In any case, let us know what you think, what you expect from a macro
system in Groovy. We have very good starting points, let's make it rock
solid :-)
You may know that we have long been complaining about the complexity of
AST transformations in Groovy. There are a very powerful tool, but it is
still a bit complex to handle. In Groovy 2.4, we planned to integrate
work from Sergei Egorov (@bsideup) that he called "Groovy macros".
Basically the idea is to have a simple way to replace classic method
class into more complex expressions. In this email, I'll try to
summarize the project and give some ideas about the orientations we want
for the Groovy language. In the end, your opinion matters so please feel
free to comment.
First of all, Sergei actually worked on two "macro" projects. The first
one consists of a "macro" block that can be illustrated by this:
defsomeVariable=newVariableExpression("someVariable");
ReturnStatementresult= macro {
return newNonExistingClass($v{someVariable});
}
As you can see, it is some kind of "super AstBuilder" which is capable
of handling variables/expressions from an external context thanks to
proper escaping with $v. This macro code is primarily aimed at being
used inside AST transformation themselves, and dramatically reduce the
amount of code required to generate an AST tree. An advantage of those
macro blocks is that if you use them in AST transformations, you're
actually not limited to expressions. You can generate anything, as long
as what is inside the macro { ... } block is supported by the Groovy
syntax. A drawback of the current implementation is that it uses a
global AST transformation. So as soon as you have the macro project on
classpath, every single method call corresponding to "macro" will be
interpreted as a macro block. There are multiple disadvantages of global
AST transformations. First of all, they are, as the name says, global,
meaning that they apply independently on *every* class being compiled by
Groovy. This also means that the transformation is run even if you know
the code you write is not using it. In particular, we need to inspect
the full AST, in-depth, to find potential method calls named "macro"
even if there's not a single one in the code (because you will only know
once you have visited the full AST). In short, a global AST
transformation has a clear performance impact, because it is executed
independently of the context. For the macro stuff, an easy workaround
would be to transform the global AST transformation into a local one.
For example, an AST transformation using the macro system could declare
it by annotating with @EnableMacros. I think it is reasonable, since it
is very unlikely that you would use the macro { ... } stuff in regular
code. The macro block is indeed useful, but in the end, it is limited to
writing other AST transformations that would be either global or local.
In short, the current macro { ... } block is useful for AST
transformation designers, but cannot be used by "regular" Groovy users
to define new language constructs.
In answer to that problem, Sergei worked on a second implementation of
macros named @Macro. The idea is both simple and elegant. Just like you
can define extension methods in Groovy (see
http://docs.groovy-lang.org/2.3.7/html/documentation/#_extension_modules),
you can write a macro like this:
public classTestMacroMethods {
@Macro
public staticExpression safe(MacroContext macroContext, MethodCallExpression callExpression) {
returnternaryX(
notNullX(callExpression.getObjectExpression()),
callExpression,
constX(null)
);
}
}
and for this to work, TestMacroMethods needs to be declared as a regular
Groovy extension module. In that case, *any* code using "safe" would be
transformed, so:
safe(x.foo()).bar()
will be expanded *at compile time* into x.foo?x.foo().bar():null
A more interesting example with pattern matching can be found here:
https://github.com/bsideup/groovy-pattern-match/blob/master/src/main/java/ru/trylogic/groovy/pattern/PatternMatchingMacroMethods.java
An important thing to understand is that the @Macro annotation is *not*
an AST transformation. Instead, it's a marker annotation which is looked
up at compile time, for each method call. So for each method call of the
AST, we try to find if an extension method node of the same name exists,
is annotated with @Macro and takes MacroContext as the first argument
and in that case, the original method call is transformed thanks to the
extension method at compile time. The code of the macro itself is
regular AST transformation code. It does *not* use the macro stuff from
the first implementation, hence doesn't simplify writing xforms. On the
other hand, it greatly simplifies the availability of transforms by
making them writable as simple extension methods, without the need of
ceremony (annotations). It is actually very cool, but it comes at a
price. Performance wise, it is still a global AST transformation, but
worse, for each and every call, it implies finding an extension method
node. It can (and will) cost a lot, but we can improve the situation by
introducing specific caches. Moreover, it also implies that the sole
fact of adding a macro "jar" on classpath could potentially affect the
semantics of your program: if a method call in your code matches the
name of a macro method call, then it would be applied at compile time.
Of course, one could argue that it is already the case for extension
modules, but the risk is lower: for extension modules, the only cases of
conflicts is when the receiver of the message matches the class of the
extension module. For @Macro, any method call on "implicit this" would
be matched. This draws the point of whether the @Macro stuff should also
be combined with a @UseMacro local AST transformation, in order to avoid
the problem of the global one.
If we do, then definitely, the performance issue is not one anymore,
because only marked code would be transformed. On the other hand, we
introduce ceremony again, which would mean that we loose the benefit of
extension methods being transparently visible. In the case of the
pattern matching stuff, this would mean that you would have to
explicitly annotate your code to enable the feature. Is it a problem?
I'm not sure actually. Also it's worth noting that the global AST
transformation issue is less of a problem if you have lots of macros on
classpath, because a single transformation would apply them all in a
single pass.
Last but not least, my feeling is that what we need is something in
between the first macro implementation and the second one. In
particular, I would like to be able to write the @Macro method body
using those macro { ... } blocks. It makes a lot of sense to me. Next, I
wouldn't like to be limited to expressions. I think a macro should be
able to produce any AST node. This implies expressions, but also
statements or even full methods.
I gave an example a little contrived, I admit, to Sergei, which is,
imagine that I want to generate two methods with the same body, but
accepting two different argument types. Then I could write a macro that
generates the method:
@Macro MethodNode createMultiply(MacroContext ctx, ClassNode argType) {
macro {
_argType_ multByTwo(_argType_ x) { 2 }
}
}
Then in a class I could write:
class Calculator {
createMultiply int
createMultiply double
}
Of course, this wouldn't compile because the grammar wouldn't allow
defining a method in a closure (in the macro block), and it would not
recognize the createMultiply calls directly in the class body, but I
kind of like the idea of a macro system that just allows expanding and
reasoning at the AST level anywhere, because it lets us create new
language constructs.
In any case, let us know what you think, what you expect from a macro
system in Groovy. We have very good starting points, let's make it rock
solid :-)
--
Cédric Champeau
SpringSource - Pivotal
http://twitter.com/CedricChampeau
http://melix.github.io/blog
http://spring.io/ http://www.gopivotal.com/
Cédric Champeau
SpringSource - Pivotal
http://twitter.com/CedricChampeau
http://melix.github.io/blog
http://spring.io/ http://www.gopivotal.com/