[groovy-dev] JVM application installer via maven/gradle repo

Discussion:

Thibault Kruse

2015-01-06 19:22:01 UTC

Hi,

I was just thinking, packaging and distributing applications with Java
/ Groovy / etc. is painful.
Consider e.g. the installation instructions at
http://groovy.codehaus.org/Installing+Griffon

As one consquence, there is gvm, which provides somewhat easier
installation of hand-picked groovy libraries.

Wouldn't it be great if that could be extended such that any jar-based
application could easily be distributed and installed from
bintray/maven central?
I mean some generic variant of gvm, a wrapper to gradle or maven.

The jar could live in the local maven or gradle repository, some
additional tooling would be required to get start scripts.
Similar infrastructures exist for other languages as well.

What do you think? Does something like that already exist and I missed it?

regards,
Thibault

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Mario Garcia

2015-01-06 20:01:37 UTC

Permalink

Hi Thibault:

Can you put an example of features you see in other languages that you are
missing in Groovy ? About what you said, came to mind some ideas:

- Any jar file can contain its own dependencies and be executed if you
have the JDK available and accesible in your command line and you have
specified your entry point (java -jar myjar.jar) . Most of web development
is evolving to a fat-jar, one executable jar with an embedded web server.
Of course you can publish in a maven repository/bintray your fat jar and
download it to be executed.
- Gradle fat-jar plugin:
https://github.com/musketyr/gradle-fatjar-plugin

- Apart from that you can store more than jar files in a maven
repository, you can publish a zip file for example. That zip file could
contain anything.
- Maven assembly:
http://maven.apache.org/plugins/maven-assembly-plugin/
- Gradle doc:
http://www.gradle.org/docs/current/userguide/artifact_management.html

- "...wrapper to gradle..." In fact there's the gradle wrapper. You
don't even have to have installed gradle, the gradle wrapper downloads the
specific gradle distribution if it's not already installed, and downloads
the needed dependencies for you (Still you need to have a JDK already
installed of course)
- http://www.gradle.org/docs/current/userguide/gradle_wrapper.html

I hope this helped
Mario

Post by Thibault Kruse
Hi,
I was just thinking, packaging and distributing applications with Java
/ Groovy / etc. is painful.
Consider e.g. the installation instructions at
http://groovy.codehaus.org/Installing+Griffon
As one consquence, there is gvm, which provides somewhat easier
installation of hand-picked groovy libraries.
Wouldn't it be great if that could be extended such that any jar-based
application could easily be distributed and installed from
bintray/maven central?
I mean some generic variant of gvm, a wrapper to gradle or maven.
The jar could live in the local maven or gradle repository, some
additional tooling would be required to get start scripts.
Similar infrastructures exist for other languages as well.
What do you think? Does something like that already exist and I missed it?
regards,
Thibault
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

Paolo Di Tommaso

2015-01-06 20:20:55 UTC

Permalink

You may want to give a try to Capsule. https://github.com/puniverse/capsule

It allows you to deploy your dependencies as a "fat" jar, to download them
from maven when required

Cheers,
Paolo

Mario Garcia

2015-01-06 21:50:33 UTC

Permalink

Nice! I didn't know about it. Thanks :)

BTW: Reading capsule documentation I remembered about
https://github.com/johnrengelman/shadow which is the fat-jar-like gradle
plugin I normally use.

Cheers
Mario

Post by Paolo Di Tommaso
You may want to give a try to Capsule.
https://github.com/puniverse/capsule
It allows you to deploy your dependencies as a "fat" jar, to download them
from maven when required
Cheers,
Paolo

Mark Fortner

2015-01-06 22:59:34 UTC

Permalink

It sounds interesting. I wonder if the simplest approach to implementing
this would be to create a launcher script (with the appropriate Grab
annotations for dependencies) and register that script with a gvm-like
repo? A gvm-like client would simply be another such script with plugins
for handling different commands. Perhaps using gradlew.

Regards,

Mark

Thibault Kruse

2015-01-07 01:28:48 UTC

Permalink

To describe this some more:

To install a python package, in the shell I run:

$ pip search inx
$ sudo pip install sphinx
$ sphinx-build

Using ruby, the same goes for gems:

$ gem search juggler
$ sudo gem install taskjuggler
$ tj3

Using javascript, I can run

$ npm search coff
$ npm install -g coffee-script
$ coffee

For all those I just need to know the rough name, and I am good to go,
don't even need to open a browser, don't need to read install
instructions. Of course in the Windows world, the command line is not
too popular, but even there a packaging solution can be great. It is
literally just one command to set everything up. And to update to a
latest version or to remove from the system again is also just one
command.

With a JVM app, I have to download a jar/zip, put it somewhere,
configure a shortcut, configure paths, etc.
The very simple lookup and install procedures of the other languages
make it easy for programs to spread quickly, and for patches and
updates to those programs to spread quickly.

The JVM world already has centralized repositories, and excellent
build tools, but somehow no opinionated package manager to install
such small applications quickly and painlessly.

As to a solution draft: A launcher script is not good enough. After
running some install command, the application should be launchable
with a command reflecting it's name. So the installer needs to create
some launch script and put it on the PATH. It is acceptable for such
solutions that the user has to initially add some foldert to his PATH
where all scripts end up.
The installer needs to check for name conflicts when installing. Also
the installer needs to keep track of what was installed, so that those
things (at least the scripts), can also quickly be removed.

To be extra nice, it would be great if extensions could also be
loaded, such as tab completion for bash, zsh, etc, if a package
provides those. My approach would currently probably be to wrap ivy or
something similar, to download jars from bintry as usual, but then to
look into the jar for maybe a special folder/file with metadata on how
to setup scripts to make applications available.

The other languages should provide a great draft of what can and needs
to be done.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Mark Fortner

2015-01-07 01:54:46 UTC

Permalink

Why do you need to wrap ivy? Why not just use grab to get your
dependencies? You can also install libraries using the grape command.

Mark

Post by Thibault Kruse
$ pip search inx
$ sudo pip install sphinx
$ sphinx-build
$ gem search juggler
$ sudo gem install taskjuggler
$ tj3
Using javascript, I can run
$ npm search coff
$ npm install -g coffee-script
$ coffee
For all those I just need to know the rough name, and I am good to go,
don't even need to open a browser, don't need to read install
instructions. Of course in the Windows world, the command line is not
too popular, but even there a packaging solution can be great. It is
literally just one command to set everything up. And to update to a
latest version or to remove from the system again is also just one
command.
With a JVM app, I have to download a jar/zip, put it somewhere,
configure a shortcut, configure paths, etc.
The very simple lookup and install procedures of the other languages
make it easy for programs to spread quickly, and for patches and
updates to those programs to spread quickly.
The JVM world already has centralized repositories, and excellent
build tools, but somehow no opinionated package manager to install
such small applications quickly and painlessly.
As to a solution draft: A launcher script is not good enough. After
running some install command, the application should be launchable
with a command reflecting it's name. So the installer needs to create
some launch script and put it on the PATH. It is acceptable for such
solutions that the user has to initially add some foldert to his PATH
where all scripts end up.
The installer needs to check for name conflicts when installing. Also
the installer needs to keep track of what was installed, so that those
things (at least the scripts), can also quickly be removed.
To be extra nice, it would be great if extensions could also be
loaded, such as tab completion for bash, zsh, etc, if a package
provides those. My approach would currently probably be to wrap ivy or
something similar, to download jars from bintry as usual, but then to
look into the jar for maybe a special folder/file with metadata on how
to setup scripts to make applications available.
The other languages should provide a great draft of what can and needs
to be done.
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

Cédric Champeau

2015-01-07 08:27:10 UTC

Permalink

I must give a -0 to this "feature". I have always been annoyed by those
"cpan/npn/pip" tools. I have some ideas why, but it very often failed for
me. One obvious reason is those who build from source and of course,
dependencies are not well defined and that's often because of a poorly
designed dependency management tool behind. With that regards, I don't
think that relying on ivy would be a good idea compared to using what
Gradle or Aether offer. But even without that, I don't like the idea much.
I mean, if it was *applications* that were released this way, with all
dependencies embedded, that would probably be ok, but by experience on the
other languages, this is not the case.

When I see tutorials starting with:

npm -i install bower

I feel uncomfortable because it is a wrong approach to me. It goes the
opposite way of what we are doing for years, that is continuous
integration. It is a tool belonging to a toolchain, so the primary channel
should not be a standalone program but a plugin of a build tool. Promoting
those installers as first class citizen IMHO promotes a wrong design
paradigm which is executable first, where we should think about integration
first. Of course, it comes at the price of not being "simple" to install,
but in the end, you gain a lot because you don't fall into the traps of
incompatible versions, fighting with your build system to integrate it, not
having to rely on containers or virtual machine images to ensure
reproducible builds, ...

Last but not least, we live on the JVM. And any tools that does "gvm
install foo" might fail because of the installed JDK. Each of the programs
may require a different version (7, 8, ...). This might not be an issue for
some, but think about it.

Of course, you might say "hey, but you promote use of gvm to install Groovy
and Gradle!". Yes I do, because:

1. Groovy is build with integration in mind first. So there are the Maven
artifacts, *then* the binary distribution, *then* GVM
2. Gradle is a build tool, so at the top of the toolchain

Once we generalize this approach, I fear that people will go the opposite
way, that is runnable first, and embeddable maybe. That is, you don't find
artifacts in a Maven repo for example, or no APIs anymore. And from what we
see in other language communities, this is a very likely issue.

So, I can understand why people want this, but I wouldn't give it a
priority in development.

Post by Mark Fortner
Why do you need to wrap ivy? Why not just use grab to get your
dependencies? You can also install libraries using the grape command.
Mark

Thibault Kruse

2015-01-07 09:11:02 UTC

Permalink

Regarding Ivy, It is just a matter of choosing the best library to
depend on. Ivy was just an example.

Regarding Cedrics mixed feelings... I too made experienced failures
with pip/npm/gem, but somehow they never annoyed me much. Under
Ubuntu, I understand that the ideal way to get software is using the
distribution package manager (apt-get) but have to live with the
software being 1-3 years older than the most recent release. The
second-best clean approach is to build from source, where I have to
organize the software myself after it has been build. The least
reliable way is to use a language-specific installer. But since the
latter is just so quick to try out, it does not matter if it fails. As
a user I invested next to nothing to try it, and when it works I am
happy, when not I accept that and look for one of the other painful
ways.

I understand as an experienced user that this way of installation is
reliable only for small projects with no non-standard dependency, or
with just very simple dependencies.

(And by the way, gvm by now installs also gaiden, glide, grails, griffon, ...)

Regarding continous integration, the concept of such installers also
has benefits. Where it works, I can use the installers on Jenkins or
Travis or other CI servers in a script to quickly set up
compiler/tester/linter toolchains. Again, where it does not work, I
have to backtrack to use a more painful means, but god am I grateful
whenever I find out that I can just 'pip install' the latest version
of a tool for my CI setup in travis.

Also I believe the tradition of providing toolchain libraries over
maven repositories is strong enough in the JVM world so that I don't
expect it to suffer much if an alternative means of distribution
existed.

This is not necessarily a topic for the Groovy mailing list, because I
think it concerns all JVM languages.I think I should have asked this
on ***@... anyway, sorry.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Jochen Theodorou

2015-01-07 10:03:04 UTC

Permalink

Post by Thibault Kruse
Regarding Ivy, It is just a matter of choosing the best library to
depend on. Ivy was just an example.
Regarding Cedrics mixed feelings... I too made experienced failures
with pip/npm/gem, but somehow they never annoyed me much. Under
Ubuntu, I understand that the ideal way to get software is using the
distribution package manager (apt-get) but have to live with the
software being 1-3 years older than the most recent release.

If we would provide a deb package, that would not be the case for
Groovy. It is really quite simple... no package maintainers always
handling the most recent version, means outdated packages. Its not for
nothing that PPAs are so often used for Ubuntu

Post by Thibault Kruse
The
second-best clean approach is to build from source, where I have to
organize the software myself after it has been build.

ideally you won't have to organize anything here... That is, if there is
a source package with dependencies, building a package you then install.
Also quite common.... but not in the Java world.

Post by Thibault Kruse
The least
reliable way is to use a language-specific installer. But since the
latter is just so quick to try out, it does not matter if it fails. As
a user I invested next to nothing to try it, and when it works I am
happy, when not I accept that and look for one of the other painful
ways.

For me the main purpose of those language installers is to have a
cross-platform and cross-distribution way of installing. I mean how else
do you want to ensure a "gem install rails" working on for example
Redhat, Ubuntu, FreeBSD, MacOS and Windows? If not done like this you
would have to maintain 4 packages instead of one.

Post by Thibault Kruse
I understand as an experienced user that this way of installation is
reliable only for small projects with no non-standard dependency, or
with just very simple dependencies.
(And by the way, gvm by now installs also gaiden, glide, grails, griffon, ...)

Basically as soon as you depend on a native library things can get
difficult... but in theory you can make this work too, if your language
based installer can communicate with the package manager of the
system... as long as there is one.

That being said, I think that most of the language installers don't
really communicate with the possibly existing package manager. gems with
a native extension are sure to able to tell a story here.

bye blackdrag
--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Dylan Cali

2015-01-07 09:59:00 UTC

Permalink

Post by Thibault Kruse
As to a solution draft: A launcher script is not good enough. After
running some install command, the application should be launchable
with a command reflecting it's name. So the installer needs to create
some launch script and put it on the PATH. It is acceptable for such
solutions that the user has to initially add some foldert to his PATH
where all scripts end up.
The installer needs to check for name conflicts when installing. Also
the installer needs to keep track of what was installed, so that those
things (at least the scripts), can also quickly be removed.

An approach we use internally is to use the gradle wrappers and the
application plugin as an 'installer.' We ship a tarball or provide a
repo that has a simple build.gradle which only declares our app as a
dependency, but doesn't have any actual code. It sets the app main
class, and provides a little extra sugar such as allowing the user to
specify an install dir and symlinking the bin scripts into a common
bin. Something like:
https://gist.github.com/calid/be042b2025c8cd2937b5

So for example the user downloads our app tarball and runs

./gradlew installapp -Pinstalldir=/usr/local/java-apps

This would then create /usr/local/java-apps/ with the structure:
https://gist.github.com/calid/0e082cffb026e5957e9f

The user can then just have /usr/local/java-apps/bin on their PATH,
and seamlessly pick up any new apps installed there. Since each app
really gets its own directory, there's no concern about dependency
jars conflicting.

Some other nice properties of this model is its really easy to have
the app tests run as a sanity check before actually installing (i.e.
similar to how cpan does it). Also if the user wants a different
version of our app they just have to change the version in the
build.gradle.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Thibault Kruse

2015-01-07 12:43:39 UTC

Permalink

@Jochen, indeed the pain for anyone wanting to distribute software is
to provide packages for many linux distributions / package managers +
installers for Windows. There are tools that can help with that pain a
bit, but in some sense I believe this whole process stems from a
worldview where there is only one operating system for mankind.
Also building a package from source and then installing is also not as
straightforward even outside the java world. The main problems here
are to not avoid installing into system space because that conflicts
with the distributions package manager, to install dependencies in a
shared way, and to be able to cleanly uninstall again (removing
dependencies once nobody depends on them). Apart from that the pain is
the huge diversity of build systems, and conventions for what to
provide for source installs. Some communities seem to have a strong
culture of standards, I guess the BSD crowd is strong in that respect,
but many open-source projects are run by people who are not as
passionate or experienced in the area of providing a package that can
be cleanly build and installed from source.

So I do some digging on what can be done with grape today. First thing
I tried was to find an example application that we could use for
experimenting. I saw that applications like jedit, jscribble have an
ubuntu package, but found no up-to-date library on jcenter or maven
central (and even searching was a pain). So this somewhat contradicts
Cendrics point that thanks to the current JVM ecosystem, people
develop embeddable first, runnable later.

Anyway, I found jftp is maybe an okay example.

It can be graped like this:
$ grape install org.objectweb.joram jftp
Both group and id are required sadly. Uninstalling using grape also
requires the version as third argument, sadly.

Same for grape resolve:

$ grape resolve org.objectweb.joram jftp 1.52
/home/kruset/.groovy/grapes/org.objectweb.joram/jftp/jars/jftp-1.52.jar

This takes 10 solid seconds on my machine, I guess some we lookup is
done that does not seem necessary.
In any case, I can use this output to start the app:

$ java -jar /home/kruset/.groovy/grapes/org.objectweb.joram/jftp/jars/jftp-1.52.jar

So what is missing here from grape or a grape wrapper:

- Searching in all available packages (search in group, id, and description)
- Installing/removing/resolving just giving the module id (if unique)
- Providing a launch scripts name like 'jftp' in a location that can
be put on the path, such as ~/.groovy/grapes/local/bin
- (Selecting the current active / default version, similar to gvm)

This would just be a start, obviously, as a package should be able to
provide more than one script, but already that does not seem all that
bad.
It would seem realistic to provide a script which can do
$ grape2 install jftp
$ jftp

and it just runs. Such an installer does not seem too far-fetched to me.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Jochen Theodorou

2015-01-07 13:29:35 UTC

Permalink

Post by Thibault Kruse
@Jochen, indeed the pain for anyone wanting to distribute software is
to provide packages for many linux distributions / package managers +
installers for Windows. There are tools that can help with that pain a
bit, but in some sense I believe this whole process stems from a
worldview where there is only one operating system for mankind.
Also building a package from source and then installing is also not as
straightforward even outside the java world.

I did know the maintainer of the windows version of gimp... I know how
much of an adventure such things can be.

Post by Thibault Kruse
The main problems here
are to not avoid installing into system space because that conflicts
with the distributions package manager, to install dependencies in a
shared way, and to be able to cleanly uninstall again (removing
dependencies once nobody depends on them). Apart from that the pain is
the huge diversity of build systems, and conventions for what to
provide for source installs. Some communities seem to have a strong
culture of standards, I guess the BSD crowd is strong in that respect,
but many open-source projects are run by people who are not as
passionate or experienced in the area of providing a package that can
be cleanly build and installed from source.

believe me, even with things like autoconf and leverage all of those gnu
tools it can be quite the adventure. I think we can conclude together in
that it is simply too complicated.

Post by Thibault Kruse
So I do some digging on what can be done with grape today. First thing
I tried was to find an example application that we could use for
experimenting. I saw that applications like jedit, jscribble have an
ubuntu package, but found no up-to-date library on jcenter or maven
central (and even searching was a pain). So this somewhat contradicts
Cendrics point that thanks to the current JVM ecosystem, people
develop embeddable first, runnable later.

oh I think he thought it should be embeddable first and runnable
later... for something like JEdit I cannot see that...

Post by Thibault Kruse
Anyway, I found jftp is maybe an okay example.
$ grape install org.objectweb.joram jftp
Both group and id are required sadly. Uninstalling using grape also
requires the version as third argument, sadly.

well... this could maybe be improved

Post by Thibault Kruse
$ grape resolve org.objectweb.joram jftp 1.52
/home/kruset/.groovy/grapes/org.objectweb.joram/jftp/jars/jftp-1.52.jar
This takes 10 solid seconds on my machine, I guess some we lookup is
done that does not seem necessary.
$ java -jar /home/kruset/.groovy/grapes/org.objectweb.joram/jftp/jars/jftp-1.52.jar
- Searching in all available packages (search in group, id, and description)
- Installing/removing/resolving just giving the module id (if unique)
- Providing a launch scripts name like 'jftp' in a location that can
be put on the path, such as ~/.groovy/grapes/local/bin
- (Selecting the current active / default version, similar to gvm)
This would just be a start, obviously, as a package should be able to
provide more than one script, but already that does not seem all that
bad.
It would seem realistic to provide a script which can do
$ grape2 install jftp
$ jftp
and it just runs. Such an installer does not seem too far-fetched to me.

agreed... I am wondering though if this would not be better as an
extension to gvm

bye blackdrag
--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Thibault Kruse

2015-01-07 13:37:52 UTC

Permalink

Duh, actually something already exists: jpm4j:
http://www.jpm4j.org/#!/
Obviously, it is not yet very well-known. And not sure yet how it
works. But I guess it has a head-start.

I did know the maintainer of the windows version of gimp... I know how much
of an adventure such things can be.

believe me, even with things like autoconf and leverage all of those gnu
tools it can be quite the adventure. I think we can conclude together in
that it is simply too complicated.

oh I think he thought it should be embeddable first and runnable later...
for something like JEdit I cannot see that...

well... this could maybe be improved

Post by Thibault Kruse
$ grape resolve org.objectweb.joram jftp 1.52
/home/kruset/.groovy/grapes/org.objectweb.joram/jftp/jars/jftp-1.52.jar
This takes 10 solid seconds on my machine, I guess some we lookup is
done that does not seem necessary.
$ java -jar
/home/kruset/.groovy/grapes/org.objectweb.joram/jftp/jars/jftp-1.52.jar
- Searching in all available packages (search in group, id, and description)
- Installing/removing/resolving just giving the module id (if unique)
- Providing a launch scripts name like 'jftp' in a location that can
be put on the path, such as ~/.groovy/grapes/local/bin
- (Selecting the current active / default version, similar to gvm)
This would just be a start, obviously, as a package should be able to
provide more than one script, but already that does not seem all that
bad.
It would seem realistic to provide a script which can do
$ grape2 install jftp
$ jftp
and it just runs. Such an installer does not seem too far-fetched to me.

agreed... I am wondering though if this would not be better as an extension
to gvm
bye blackdrag
--
Jochen "blackdrag" Theodorou - Groovy Project Tech Lead
blog: http://blackdragsview.blogspot.com/
german groovy discussion newsgroup: de.comp.lang.misc
For Groovy programming sources visit http://groovy-lang.org
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Dylan Cali

2015-01-07 19:07:02 UTC

Permalink

Post by Thibault Kruse
- Searching in all available packages (search in group, id, and description)
- Installing/removing/resolving just giving the module id (if unique)
- Providing a launch scripts name like 'jftp' in a location that can
be put on the path, such as ~/.groovy/grapes/local/bin
- (Selecting the current active / default version, similar to gvm)
This would just be a start, obviously, as a package should be able to
provide more than one script, but already that does not seem all that
bad.
It would seem realistic to provide a script which can do
$ grape2 install jftp
$ jftp
and it just runs. Such an installer does not seem too far-fetched to me.

To put a finer point on it, if you clone
https://github.com/calid/jftp-gradlew.git you can run

./gradlew installapp
build/install/jftp-gradlew/bin/jftp-gradlew

and jftp runs. The cool thing is this works on both linux and
windows. This seems pretty powerful to me.

One can imagine with a little but of infrastructure and a nice
frontend you could provide the same user experience (or better) as
cpan/npm/gem/pip, including search and uninstallation. i.e. have a
repository of 'gradle recipes' for installing applications.

But to Cedric's point you would still be working with the maven/ivy
artifact ecosystem and the gradle/plugin paradigm. So you have the
best of both worlds in my opinion.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Thibault Kruse

2015-01-07 20:49:07 UTC

Permalink

Hi Dylan,

this is unsatisfactory in many cases. First, you need to find the
github repo. You need to verify that the repo is indeed containing the
software you were looking for. Then, you need to decide where on your
file system to place your local clone. Then you need to find out
whether jftp can be build from source that way, and what buildsystem
command is required (ant, maven, gradle, gant, buildr, leiningen,
buck, ...). Then you need to figure out which branch or tag to use.
Then you have to wait until gradle (or any other tool) has downloaded
half the internet for all the build dependencies, that you don't
generally care about (your intention is to run the thing, not
compile/test/document it). Then you need to figure out where the
executable was placed, and whether and how to get the classpath right.
Then you need to devise a clever way to get that executable onto your
PATH.

Then six month later, you have done this for 50-100 applications, now
comes the time you want to fetch updates. So what will you do now? Go
through all your directories, check for updates, select a new
tag/branch, invoke the scm update command, and then the build command?
When you buy a new computer (or set up some virtualbox or so), and you
want these 50-100 apps there as well, what will you do?

It is all possible, for sure. I like having mature build instructures
in the JVM ecosystem to be able to build projects from source reliably
when I want to do that. But it is not good enough for many situations
for just installing a little tool to try out and possibly dump 10
minutes later. You want to get it as fast as snipping your fingers,
and be rid of it again just as fast. I want just the jar, not the
whole source tree (which often is also more than just the tool I want,
e.g. jpm4j is part of the huge bndtools repo (>100MB).

Anyway...
Investigating jpm4j, I find that the project does not look like it's
going to be the next big thing. I believe it aims to high, and
requires too much special infrastructure (like an own website with
index). The connection to OSGI is also not encouraging for
'lightweight' tool.

So I started coding what I think would be good, at https://github.com/tkruse/gpi
Once a few use-cases work, I'll see whether I can find more contributors.

Post by Dylan Cali

To put a finer point on it, if you clone
https://github.com/calid/jftp-gradlew.git you can run
./gradlew installapp
build/install/jftp-gradlew/bin/jftp-gradlew
and jftp runs. The cool thing is this works on both linux and
windows. This seems pretty powerful to me.
One can imagine with a little but of infrastructure and a nice
frontend you could provide the same user experience (or better) as
cpan/npm/gem/pip, including search and uninstallation. i.e. have a
repository of 'gradle recipes' for installing applications.
But to Cedric's point you would still be working with the maven/ivy
artifact ecosystem and the gradle/plugin paradigm. So you have the
best of both worlds in my opinion.
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Dylan Cali

2015-01-07 21:07:48 UTC

Permalink

Hi Thibault,

I'm saying you could leverage the Gradle functionality as the
_backend_ of a new tool. They've already implemented most of the hard
stuff so why not reuse it? And then your still using the common
artifact ecosystem and build tools prevalent in the jvm community.

To continue with your 'grape2' usage, I'm saying you could implement

grape2 search jftp
grape2 install jftp
grape2 upgrade jftp
grape2 uninstall jftp

and many other commands by leveraging the Gradle API under the hood.
You could host 'gradle app specs' like the one in my repo on a common
web server or github repo, so it's easy to search and reuse... i.e.
jftp.gradle

Does that make more sense?

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Thibault Kruse

2015-01-07 23:16:51 UTC

Permalink

Hi Dylan,

I see. I had not understood before. Actually I did not notice you
posted twice, so I did not read your two messages in context. Sorry.

What your gradle spec approach solves then is the lookup of a
dependency on a remote repository, the installation to a local folder,
and the creation of a launchable script given a jar and a main class
name. Organizing these build results would remain to be done.

I would like to avoid hosting anything, but your approach can also
work with jcenter() to some degree, one can get the mainClassName from
a manifest, and run gradle locally.
That would not be quite as powerful, though one can imagine shipping
such a gradle spec file on maven central as well.

I have to think some more about it. I do not like to depend on gradle
(I was rather looking for ways to eventually only depend on Java + Ivy
or Java + Aether), but I can see that the application plugin of gradle
does a useful job of bundling distribution files in a folder and
creating a laauncher script.

Possibly other JVM build tools have similar / better capabilities of
creating launcher scripts, that would have to be investigated. If ant
had something similar, that would propbably be nicer.

It's worth trying that out in my gpi project as a branch. My search
function based on Maven central search already works, and that part is
something that gradle could not solve anyway.

Thanks,
Thibault

Post by Dylan Cali
Hi Thibault,
I'm saying you could leverage the Gradle functionality as the
_backend_ of a new tool. They've already implemented most of the hard
stuff so why not reuse it? And then your still using the common
artifact ecosystem and build tools prevalent in the jvm community.
To continue with your 'grape2' usage, I'm saying you could implement
grape2 search jftp
grape2 install jftp
grape2 upgrade jftp
grape2 uninstall jftp
and many other commands by leveraging the Gradle API under the hood.
You could host 'gradle app specs' like the one in my repo on a common
web server or github repo, so it's easy to search and reuse... i.e.
jftp.gradle
Does that make more sense?
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Dylan Cali

2015-01-08 05:43:25 UTC

Permalink

Some replies inline

Post by Thibault Kruse
Hi Dylan,
I see. I had not understood before. Actually I did not notice you
posted twice, so I did not read your two messages in context. Sorry.

No problem

Post by Thibault Kruse
What your gradle spec approach solves then is the lookup of a
dependency on a remote repository, the installation to a local folder,
and the creation of a launchable script given a jar and a main class
name. Organizing these build results would remain to be done.

Well like I mentioned, I think that's easy. Just default to
$HOME/.java-apps or something as the application plugin install dir,
and symlink into a common bin ($HOME/.java-apps/bin) the user can put
on their path (Windows would be a little trickier, but hey everything
on windows is trickier...)

Post by Thibault Kruse
I would like to avoid hosting anything, but your approach can also
work with jcenter() to some degree, one can get the mainClassName from
a manifest, and run gradle locally.
That would not be quite as powerful, though one can imagine shipping
such a gradle spec file on maven central as well.

I think the advantage of a spec based approach is that you don't have
to do complicated gymnastics to figure out stuff like the main class
name (by querying jcenter or whatever). Just save it in the spec for
each app and then there's no guessing.

Post by Thibault Kruse
I have to think some more about it. I do not like to depend on gradle
(I was rather looking for ways to eventually only depend on Java + Ivy
or Java + Aether), but I can see that the application plugin of gradle
does a useful job of bundling distribution files in a folder and
creating a laauncher script.

I don't think depending on gradle should be considered a bad thing...
reusing gradle's functionality and plugin ecosystem I think is a very
good thing.

Post by Thibault Kruse
Possibly other JVM build tools have similar / better capabilities of
creating launcher scripts, that would have to be investigated. If ant
had something similar, that would propbably be nicer.

I'd have a hard time being convinced that an ant based approach would
be nicer :) Although if ant did have some killer task there was no
equivalent gradle plugin for you can after all call ant tasks from
gradle.

Post by Thibault Kruse
It's worth trying that out in my gpi project as a branch. My search
function based on Maven central search already works, and that part is
something that gradle could not solve anyway.

Another advantage of a spec based approach I think is it makes search
easy. If every spec name corresponds to the app name (jftp.gradle),
then search could be a simple "is there a spec with this name?" And
that includes using regexes to find matching spec names.

A downside would be if there's no spec for an app, then it's like the
app doesn't exist. But maybe this is also an upside since it would
promote following a common app distribution pattern, which the jvm
doesn't really have right now. Also for projects that don't currently
host their projects on maven central or jcenter, it would force them
to join that ecosystem (make them "embeddable first" to use Cedric's
term).

Once a gradle build spec is loaded it's just groovy after all, so it
should make other things easy like adding hooks for descriptions.

Full disclosure: I've been having to do a ton of Gradle programming
of late (writing plugins and whatnot), so that's probably why my brain
is hot on Gradle right now... but frankly it's been an awesome
experience and has been a huge force multiplier for my team. And at
least based on my recent experience the functionality we're discussing
here would be very naturally done as a Gradle plugin (or plugins).

Cheers,
Dylan

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Thibault Kruse

2015-01-08 10:28:30 UTC

Permalink

Don't get me wrong, I would chose gradle over ant any time for
building a conventional project. But a tool like what I have in mind
"sells" best when it is small and has minimal dependencies.

The biggest problem I see right now with your approach are duplicate
and heavy dependencies. So when somebody writes a tiny shell tool in
Groovy and creates a jar for that depending on Groovy, I would not
want a package installer to download an individual copy of Groovy for
each such tiny application. A hello world Groovy 1KB script becomes a
6MB tar by including groovy-all. That's no way to make JVM languages
popular with the *nix crowd. Not even talking about Spring-based
applications and similar. Downloading groovy-all to .gradle and using
it from there is fine, even having multiple versions of groovy there,
I think that is quite acceptable.
installing duplicate jars in general is still a viable approach given
disk space is cheap, but if it is reasonably possible to avoid the
duplicate download of the same jar, that should be the preferred way,
I think.

That being said I still like your idea for hacking together a
prototype quickly. And maybe the gradle people have an idea for how to
reuse their launcher script generator with shared jars downloaded to a
local gradle/ivy/aether repository. (And again, the other high-level
building frameworks maven, sbt, ... have similar plugins, gradle is
not that special in that respect). I also think about just depending
on the respective classes from within gradle, and bundling those
classes with my app.

Regarding the mainClassName, many jars are already executable by
having a mainClass defined in their META-INF/MANIFEST.MF. By using
that, without any action by maintainers, I'd get thousands of working
applications already (like jftp). And maintainers would not be tied to
my application, they could just follow age-old Java standards.
For more ambitious projects delivering more than one executable in a
single jar, that sadly does not work, similar as other ideas like
manpages and bash completion, so for such things another metadata
format would be required, as far as I can see. Even then I would not
want specs written for gradle as target framework, but rather a
declarative format that can easily be mapped to any tool, and in
particlar one that allows easy generation of deb files and such as
well (two birds, one stone).

I agree that search over all jars on maven central is less useful than
searching just within the jars that are executable, but that is
something that can be solved later on one way or the other. Also while
it is not my main concern right now, it is possible to extend my idea
to 'install' library jars (without executables) to have them in a
classpath, possibly similar to virtualenv. Just like grape install
does (without the classpath bit). Not sure where that would lead, but
it is something lurking at the back of my mind.

However all my reservations do not mean that for your internal
solution, I would have done differently. For that, your approach seems
fine to me.

Post by Dylan Cali
Some replies inline

Post by Thibault Kruse
Hi Dylan,
I see. I had not understood before. Actually I did not notice you
posted twice, so I did not read your two messages in context. Sorry.

No problem

I don't think depending on gradle should be considered a bad thing...
reusing gradle's functionality and plugin ecosystem I think is a very
good thing.

Another advantage of a spec based approach I think is it makes search
easy. If every spec name corresponds to the app name (jftp.gradle),
then search could be a simple "is there a spec with this name?" And
that includes using regexes to find matching spec names.
A downside would be if there's no spec for an app, then it's like the
app doesn't exist. But maybe this is also an upside since it would
promote following a common app distribution pattern, which the jvm
doesn't really have right now. Also for projects that don't currently
host their projects on maven central or jcenter, it would force them
to join that ecosystem (make them "embeddable first" to use Cedric's
term).
Once a gradle build spec is loaded it's just groovy after all, so it
should make other things easy like adding hooks for descriptions.
Full disclosure: I've been having to do a ton of Gradle programming
of late (writing plugins and whatnot), so that's probably why my brain
is hot on Gradle right now... but frankly it's been an awesome
experience and has been a huge force multiplier for my team. And at
least based on my recent experience the functionality we're discussing
here would be very naturally done as a Gradle plugin (or plugins).
Cheers,
Dylan
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Dylan Cali

2015-01-08 10:46:56 UTC

Permalink

Post by Thibault Kruse
The biggest problem I see right now with your approach are duplicate
and heavy dependencies. So when somebody writes a tiny shell tool in
Groovy and creates a jar for that depending on Groovy, I would not
want a package installer to download an individual copy of Groovy for
each such tiny application. A hello world Groovy 1KB script becomes a
6MB tar by including groovy-all.

That's a good point, that didn't occur to me.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Cédric Champeau

2015-01-08 10:50:48 UTC

Permalink

Maybe a good point, but IMHO building from source is the worse of all
solutions. You will have a different version of the compiler than when
the script was written and no guarantee that it will compile. I much
prefer pre-compiled applications even if they bundle jars and have
redundant dependencies. We have TBs of storage nowadays, and we download
millions of cute cat videos on Youtube everyday that weight hundreds of
megabytes, why would 4MB of Groovy be a problem?

Post by Dylan Cali

That's a good point, that didn't occur to me.
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

--
Cédric Champeau
Groovy language developer
http://twitter.com/CedricChampeau
http://melix.github.io/blog

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Thibault Kruse

2015-01-08 11:08:41 UTC

Permalink

Not sure whether I misphrased. I was talking about writing a Groovy
Class with a main method to be distributed in a jar file. So no local
recompilation involved, but still there is the dependency to
groovy-all (unless I got something wrong). The some MB are maybe not a
'real' problem, but a 'perceived' problem. I do not want to evangelize
people about the cheapness of disk space, I want to persuade them to
use my tool regardless of their biases.

Just like when I install a python program, I do not want it to have
install its own python interpreter, or when I install a C++
application depending on boost, I do not want it to ship with it's own
copy of boost. Or, put another way, I also would not want an
application installer to install a dedicated JRE with every small
application (even though 30MB is also not that much compared to
streaming video data). For really tiny interpreters and libraries, it
may not matter as much (e.g. lua).

I am totally not complaining about the size of groovy-all.jar here. I
am just saying a pip/gem/npm/cpan-like tool for the JVM will have more
success if it manages to avoid duplicate jar downloads. And since my
main use-case for the tool are scripts in the 1KB range, any duplicate
dependencies in the MB range will be perceived as bloat. And I would
like this tool to boost the usage of groovy for the kind of tiny
applications currently commonly written in python or perl or bash. If
the tool seems bloated, it could have the effect of making people
biased against groovy.

On Thu, Jan 8, 2015 at 11:50 AM, Cédric Champeau

Post by CÃ©dric Champeau
Maybe a good point, but IMHO building from source is the worse of all
solutions. You will have a different version of the compiler than when the
script was written and no guarantee that it will compile. I much prefer
pre-compiled applications even if they bundle jars and have redundant
dependencies. We have TBs of storage nowadays, and we download millions of
cute cat videos on Youtube everyday that weight hundreds of megabytes, why
would 4MB of Groovy be a problem?

Post by Dylan Cali

That's a good point, that didn't occur to me.
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

--
Cédric Champeau
Groovy language developer
http://twitter.com/CedricChampeau
http://melix.github.io/blog
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Dylan Cali

2015-01-08 11:10:36 UTC

Permalink

On Thu, Jan 8, 2015 at 4:50 AM, Cédric Champeau

I don't think 4MB of Groovy is a problem per se, it's more the
ickiness of having redundant dependencies all over the place. So
instead of 100 apps all having their own copy of the same dependency
it would be nice if they could instead simply reference it from the
same location. Which is totally doable, it would just require more
work than a simple wrapper around the Gradle application plugin.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

Steve Amerige

2015-01-07 12:41:27 UTC

Permalink

Groovy Grape with the @Grab annotations might be disallowed
<http://groovy.329449.n5.nabble.com/Disabling-Grape-td5720694.html> in
production environments. If one is looking at installation issues, I'd
just like to point out that for stability and security reasons,
automatic dependency resolution might not be a viable choice.

Enjoy,
Steve Amerige
Principal Software Developer, Fraud and Compliance Solutions Development
SAS Institute, 100 SAS Campus Dr, Room U3050, Cary, NC 27513-8617

Post by Mark Fortner
It sounds interesting. I wonder if the simplest approach to
implementing this would be to create a launcher script (with the
appropriate Grab annotations for dependencies) and register that
script with a gvm-like repo? A gvm-like client would simply be
another such script with plugins for handling different commands.
Perhaps using gradlew.
Regards,
Mark
On Tue, Jan 6, 2015 at 11:22 AM, Thibault Kruse
Hi,
I was just thinking, packaging and distributing applications with Java
/ Groovy / etc. is painful.
Consider e.g. the installation instructions at
http://groovy.codehaus.org/Installing+Griffon
As one consquence, there is gvm, which provides somewhat easier
installation of hand-picked groovy libraries.
Wouldn't it be great if that could be extended such that any jar-based
application could easily be distributed and installed from
bintray/maven central?
I mean some generic variant of gvm, a wrapper to gradle or maven.
The jar could live in the local maven or gradle repository, some
additional tooling would be required to get start scripts.
Similar infrastructures exist for other languages as well.
What do you think? Does something like that already exist and I missed it?
regards,
Thibault
---------------------------------------------------------------------
http://xircles.codehaus.org/manage_email