No big changes this time, mainly compatibility fixes for JRuby 1.0. It is now also possible to build the project using Maven, for those too afraid to use rake. Documentation and code for springy are available here.

I’m also happy to announce that Craig Walls, the author of “Spring in Action”, is going to talk about Springy as part of his “Spring Cleaning: Tips for Managing XML Clutter” talk at this year’s No Fluff Just Stuff series of events as well as the Spring Experience 2007 in Florida.

While trying to create a unified testing framework (shared between Rails and our Java backend code) I came across ActiveRecordJDBC which is an adapter to use JDBC drivers with JRuby on Rails. It works fine, although it can be a bit complicated to get a DRY database.yml configuration. The goal is to get rid of our dbunit/manually crafted database tests on the Java side by using ActiveRecord fixtures. After some research I found out about the Rails integration project (now called GoldSpike), which tries to make it easy to deploy a Rails app on a Java servlet container such as jetty. As far as I know Thoughtworks uses this approach to deploy their new product, Mingle. GoldSpike is under constant development but it is already usable, although a few patches were required. After everything was set up, a simple

$ rake war:standalone:create
Reading user configuration
Assembling web application
Adding Java library commons-pool-1.3
Adding Java library rails-integration-1.1.1
Adding Java library activation-1.1
Adding Java library mysql-connector-java-5.0.5
Adding Java library bcprov-jdk14-124
Adding Java library jruby-complete-1.0
Adding web application
Adding Ruby gem ActiveRecord-JDBC version 0.4
Creating web archive

creates a .war file which can be directly deployed to a container. I’ve never been a big fan of war files, but in the context of JRuby+Rails it makes perfect sense, because Ruby and all the required gems can be packaged up in a single file. No need to worry about missing gems, C bindings or wrong versions of the installed software, all you need to deploy is Java and jetty (which is pretty lightweight).

Not quite sure what the implications of this are, but it seems like a good alternative to mongrel/CRuby deployments. Performance-wise it looks good, too, though we haven’t done any benchmarking. GoldSpike doesn’t have a project homepage yet, but you can find it on the jruby-extras project page. Let’s wait and see what this will mean for the adoption of Rails in corporate environments (Oracle is currently looking for Rails developers to join the Enterprise 2.0 team, for example).

We’ve just got ourselves a couple of new servers for running SONAR on, for internal use and demonstration. They are quite beefy, with 8GB RAM, 15k disks, and 8 x 3.2GHz threads. We decided to go with centos 5 as the operating system, since we have had pain installing Oracle on our debian platforms. Here is the install procedure I uncovered, which gets the server ready for SONAR on either MySQL or oracle. It covers installing Ruby, RubyGems, lots of useful Gems, Java, and Oracle 10g

yum install -y ruby
yum install -y ruby-devel ruby-docs ruby-ri ruby-irb ruby-devel ruby-rdoc
yum install -y w3m # for text based browsing and downloading, where curl won't work

# download rubygems from rubygems.org with w3m then...
gunzip rubygems-0.9.4.tgz
tar xf rubygems-0.9.4.tar
cd rubygems-0.9.4
ruby setup.rb

# get jdk rpm from java.sun.com and install

# add an /etc/profile.d/java.sh file, containing
##
JAVA_HOME=/usr/java/default
export JAVA_HOME
##
chmod +x /etc/profile.d/java.sh
source /etc/profile.d/java.sh

yum install -y ImageMagick-devel
yum install -y perl-DBI
yum remove -y sendmail
yum install -y postfix
chkconfig postfix on
yum install -y ntp
chkconfig ntpd on
yum install -y mysql-server mysql mysql-devel

chkconfig mysqld on
/usr/bin/mysqladmin -u root password '<password>'
/usr/bin/mysqladmin -u root -h <hostname> password '<password>'

yum install -y gcc
gem install builder
gem install camping
gem install capistrano
gem install cgi_multipart_eof_fix
gem install cheat
gem install chronic
gem install ci_reporter
gem install daemons
gem install fastthread
gem install gem_plugin
gem install hpricot
gem install image_science
gem install jerbil --source http://code.trampolinesystems.com
gem install json
gem install markaby
gem install metaid
gem install mongrel
gem install needle
gem install net-sftp
gem install net-ssh
gem install rake
gem install rmagick  # [ ignore errors about fonts... they are just for the docs ]
yum install -y libxml2-devel
yum install -y libxslt-devel
gem install ruby-xslt
gem install rubyforge
gem install RubyInline
gem install mysql -- --build-flags --with-mysql-config
yum install -y httpd
yum install -y subversion

# install rpmforge for centos 5 as detailed here :
# http://wiki.centos.org/Repositories/RPMForge#head-d766e7aacdee6150c6705d9369aa36c9bcabc139

yum install -y firefox
yum install -y xorg-x11-xauth
yum install -y vnc-server
yum install -y emacs
yum install -y libXtst
yum install -y xorg-x11-utils
yum install -y xorg-x11-resutils
yum install -y xorg-x11-server-utils

yum install -y compat-libgcc-296 compat-libstdc++-296 compat-libstdc++-33 openmotif22
yum groupinstall -y "Development Libraries"
yum groupinstall -y "Development Tools"
yum groupinstall -y "Legacy Software Development"
yum groupinstall -y "Server Configuration Tools"
yum groupinstall -y "X Software Development"
yum groupinstall -y "Administration Tools"

# fire up a vncserver on the remote machine :
vncserver
# and enable connections to the X server from other accounts :
xhost +

# connect an ssh session from your local machine, forwarding the vnc port [ 5900 + screen # ]
# connect your local vnc client to the server
# run firefox on the server
# download oracle 10g from www.oracle.com, thus avoiding having to upload 750M at 512Kb/s
# install as shown here : http://www.dizwell.com/prod/node/681?page=0%2C3
# and add an init.d startup script as described here : http://www.dizwell.com/prod/node/50?page=0%2C4

# done ! reboot to check that your init.d scripts are functioning correctly

I’ve just released a new version of springy, the JRuby-based alternative to Spring’s XML configuration format. It now has a “serialise-to-xml”-feature which allows you to write your whole config to a nicely formatted Spring XML file. Why would you want to do that ? In our case we’ve created a Ruby-based configuration framework modeled after Rails environments which pulls config files from different locations, deals with overrides and so on. The only downside is that this is pretty slow (especially when unit testing), hence the idea of serialising the whole context to XML.

The serialised XML config file is about twice the size of the springy context (~1600 vs. ~800 lines), which shows how expressive Ruby can be. A small example from our configuration:

DAOS = [ :ZoneDAO, :EmailDomainDAO, :DayDAO, :PreferenceDAO,
 :WhatEverDao… ]
DAOS.each do |dao|
bean(dao, “daos.hibernate.#{dao}Hibernate”)
{|b| b.new(“sonarSession”)}
end

This snippet uses metaprogramming to create 20 odd DAOs in 8 lines of code.

Springy has matured since the first release when it was still in the proof-of-concept-stage. We’re using it successfully in Sonar, and there is no way we’ll ever go back to using XML for configuration. Anyway, XML is for girls, according to Hani.

My new favourite feature in JRuby is JavaUtilities.extend_proxy. Basically it allows you to extend an existing Java class with Ruby code. All instances of this class will then have additional Ruby methods on them. This can be handy to fix completely useless APIs like org.w3c.dom.Document. If you’ve ever used Ruby’s xml builder you will know what I mean.

JRuby allows us to do this:

JavaUtilities.
extend_proxy(‘com.sun…internal.dom.DocumentImpl’) {
def tag(sym, *args, &block)
#ruby code omitted
end
}

doc = DocumentImpl.new
doc.tag(“beans”) do
doc.tag(“bean”, “id”=>“mybean”) do
…
end
end
doc.validate!
doc.serialize(System.err)

DOM is finally usable! See serialization.rb for the full code.

SONAR brings statistical natural language processing and social network analysis to the problem of browsing and filtering large archives. We permit users to see who is talking to who, how much and about what. There are applications in areas including expertise discovery, compliance, forensics, search and alerting.

We are a small company, with large ambition. We number currently 10, and have just secured £3M of venture funding to help us grow our R&D efforts and establish a North American presence.

The SONAR platform consists of a very Ajaxy web front-end in Ruby on Rails, with a Java back-end doing the asynchronous natural language processing and social network analysis.

We are looking for software developers with experience in the following technologies :

Java, Hibernate, Spring framework, JMS, Ruby, Ruby on Rails, Transactions, Transactional messaging, SQL, MySQL, Oracle, Linux, Windows

Experience in all of the above is not a necessity. We are looking for smart people who can learn quickly, love solving difficult problems and can grow with the company. These attributes are far more important than any tick list.

If you are interested, please contact :

Craig McMillan, craig AT trampolinesystems DOT com

Please note : if you are an agency and are not already talking to us, then do not contact us. Really.

I’m a big fan of SVN, and I know it’s in very common use throughout the development community, so I thought I’d give a shout out to an interesting project that extracts some fascinating SVN stats and draws lots of pretty pictures - StatSVN

We’ve now added a StatSVN task to our build scripts, and the figures it kicks out are just great - once you start looking, you can lose yourself in there for ages….

To give some random samples:

• As of three days ago, there were 75130 lines of code in the Sonar trunk, of which Craig has contributed 33.6%, myself 32.1% and Jan 22.2%. As Jan has mostly been working on the Ruby On Rails interface, and Craig and I have mostly been writing the Java backend, what does this say about the relative expressive efficiency and productivity of each language?
• Craig took “Developer Of The Month” for February, with 5286 lines, taking over from me in January with 7546 lines.
• Over 80% of my changes are additions, with under 20% being updates. Craig is more or less the same, with a few percent more modifications - probably due to fixing my occasional “EVIL” quick hacks…
• By far and away the most commits overall get done between 4pm and 5pm, but personally, I do three times as many commits between 2pm and 3pm than any other hour. Probably because I keep saying to myself got to finish this before I go to lunch…
• Jan has not done a single commit before 12 noon However, he’s the only person to have committed between 1am and 2am!
• I do a huge number of commits on a Monday (400+), followed by successively fewer every day until Friday (~60). Does this mean I’m fresher and more enthusiastic after the weekend, or is it that when I start the week, I go for the easiest tasks first, and ramp up towards the more difficult tasks as I go on? On the other hand, overall the most commits are done on a Friday, followed by Tuesday

You can also measure some metrics around files, rather than developers:

• Overall, across all file types, we have an average of around 40 lines of code per file
• 58.7% of the files in the repository are .java, but they contribute 73.3% of the total lines of code
• On average, each .rhtml template is just 26.6 lines of code
• The most amended file in the whole repository is the Rakefile!

I could go on for hours….but it’s nearly 2pm and I need to commit some code, dammit! So install StatSVN on your repository and have a play yourself. It’s fun.

i came across a problem with our java unit testing package, TestNG : our IHookable implementation, which sets up transaction contexts for test methods, wasn’t being given details of Exceptions which occurred during the running of the test methods

downloading the source for TestNG, it turned out to be an easy fix, a one-liner. i submitted a patch to cedric, and a couple of hours later it was accepted and checked in

if only all open source experiences were so smooth

The term “XML sit-ups” is often used by Rails developers to make fun of the XML-loving Java crowd. When I first heard it, I was slightly annoyed - there’re many valid use cases for XML. However, using it as a scripting language is clearly not one of them. The build system Ant fell into this trap, and so did the Spring Framework: the configuration is just too verbose and awkward.

<bean id="bean3" class="springy.beans.Bean3">
<constructor-arg value="${vic}"/>
<constructor-arg ref="bean2"/>
<property name="aProperty">
<list>
<ref bean="bean1"/>
<value>A String</value>
</list>
</property>
</bean>

This slightly contrived example creates a new instance of the class springy.beans.Bean3 (passing two values to the constructor) and eventually sets a property (in this case of type java.util.List) on the instance. The first constructor argument, ${vic}, is a placeholder which will get resolved at runtime (requiring more plumbing, I’ll spare you the details).

Fortunately, the bean/object management bits of Spring are completely unaware of the XML horrors happening around them, in best Java tradition (“if we need to change the implementation some day”). I decided that the day had arrived, inspired by a blog post talking about a successful Groovy-based Spring configuration (Programmatic Configuration in Spring).

Instead of Groovy I opted for Ruby (in its Java-based incarnation JRuby), a language known for its DSL-friendly syntax. Trying to make the DSL similar to Rake’s the above XML turned into:

bean :bean3, “springy.beans.Bean3″ do |b|
b.new(vic, :bean2)
b.aProperty = [ :bean1, “A String” ]
end

A fair bit of Ruby’s magic is used to make this possible: the notorious method_missing to set properties and :symbols to reference existing beans. The nice thing about using a DSL based on real programming language instead of pure XML is that you can use all features the host language has to offer. For example, constructs like <import resource=”x”/> can be replaced with Ruby’s require() or load() mechanisms. This is not only restricted to Ruby code, as JRuby allows you to use normal Java objects as well.

Extending the syntax is just a matter of adding methods, as opposed to having to deal with XML schema and coding definition parsers. The Extensible XML authoring example from the Spring documentation translates into the following Ruby code:

def dateformat(id, format, lenient=false, args = {})
register_bean(BeanDef.new(id, “java.text.SimpleDateFormat”, args)
.new(format).lenient(lenient))
end

This is three lines of code compared to writing the XML schema (20 lines), parsing handler (30 lines), namespace handler (10 lines) plus registering the handler and schema (two configuration files to edit).

In case that XML really is required, you can put additional XML markup inline:

inline_xml do <<XML
<beans>
<bean id="bean1-2" class="springy.beans.Bean1"/>
</beans>
XML
end

The implementation of this DSL requires ca. 200 lines of JRuby and 20 lines of Java. However, the obvious advantage is not the DSL itself but the access to both Ruby and Java from within the ‘configuration’ files, giving way to far more interesting possibilities like Ruby-based AOP (currently not implemented), which could look like this:

around_advice “expression(* foo(..))” do |joinpoint|
joinpoint.this.do_something
joinpoint.proceed
end

In case you want to give it a spin you can find the code here. However, bear in mind that it is far from being complete as many features are currently not implemented. It serves more as a proof of concept and starting point for a full implementation. Also, using an interpreted language there’s an obvious performance overhead but it’s not that noticeable (ca. 1 sec slower on my machine) and the increased flexibility makes more than up for it. Things can only get better: the JRuby guys are putting a lot of effort into performance improvements at the moment.

Edit 08/12/06: Simplified example

Edit 18/12/06: Got some interesting feedback from the InfoQ community. The readability of the DSL was a main concern of most of the posts. I must admit that it is probably difficult to read without some basic understanding of Ruby concepts and data types. It is probably more interesting to developers already familiar with Ruby. It’s a typical problem with internal DSLs.

Applets aren’t always evil

Applets are hated by users and developers alike. The infamous grey box combined with slow startup times caused them to disappear almost completely from most web developers’ toolboxes. They are now widely regarded as a failed and overhyped attempt to deliver a desktop-like user experience in browsers, eclipsed by more lightweight alternatives such as Flash or Ajax. However the Java runtime and browser plugins are already installed on most computers (our logs indicate 98% of our visitors) and the Java platform continues to thrive.

For the implementation of the visualiser in the SONAR platform (also included in the Enron Explorer), an interactive visualisation of complex social networks graphs [1], our first instinct was to consider alternatives to Java. After a bit of investigation we found that both Flash and DHTML were far too slow and would require a significant amount of bespoke work… So we turned our attention back to Java. Our earlier successful development of Metascope, a Java Web Start/desktop application based on the excellent Prefuse visualisation toolkit, tempted us to reconsider the applet route. Code reuse is a Good Thing after all.

After implementing a first prototype we came to the conclusion that applets and Ajax/JS are actually a good match, allowing many of the applets’ shortcomings to be worked around quite easily.

Making applets optional

We decided to make the applet completely optional - it is inserted with Javascript only after the user has ‘opted in’. This avoids the browser lock-up during the first page load, one of the reasons for the unpopularity of applets, and bypasses the large download for those who cannot or do not want to use it.

Adding Ajax

Unfortunately applets, still stuck in web prehistory, don’t integrate as seamlessly as other plugins with web pages. As soon as the user navigates away from the page containing the applet, the browser unloads the virtual machine, causing another delay as the page gets loaded again. HTML frames are commonly used to work around this issue, a big no-no in modern web design.

This is where Ajax comes to the rescue. Ajax and other techniques, like Lightboxing, have made the construction of complex single-page web applications very simple - so the applet can remain loaded while portions of the page are updated without refreshing the whole page. This gets around the long delays that the user would have to sit through with repeated loading and unloading of the Java virtual machine.

LiveConnect: Interaction with the browser

Applets are not usually self contained and need to interact in some way with their host environment. This is accomplished with LiveConnect, a Java-Javascript bridge which was first introduced in 1996 with Netscape 3.0 but never got much attention. The API is surprisingly simple to use and methods on the applet can be directly invoked from Javascript:

// “applet” is the id of the applet
document.applet.itemClicked(id);

The fact that modern web applications use Javascript libraries such as Dojo or Prototype makes integration very simple: they almost always provide callbacks which can be used to notify the applet of changes made to the page:

// example with Prototype:
// invoke method after successful ajax request
new Ajax.Request(url, {
…,
onSuccess: function(req, obj) {
document.applet.success(req.status);
}
});

On the applet side, partial page reloads (as a result of user interaction) are triggered by evaluating Javascript directly:

// “this” is referring to the applet object
public void itemClicked(int id) {
JSObject.getWindow(this).eval(“focusItemId(” + id
+ “);”);
}

In this example focusItemId is a custom Javascript function, included using normal <script> tags. It will typically issue an asynchronous request to update parts of the interface. Although LiveConnect is not widely used the implementation seems to work quite well with recent browsers and versions of Java. The class files needed during the development of the applet are contained in plugin.jar which can be found in the jre/lib subdirectory of the JDK.

Conclusion

In situations where Java is a good tool for the job (CPU and graphicly intensive client applications), a combination of Ajax, LiveConnect and simple Javascript can greatly reduce the pain usually associated with applets.

Looking to the future, Sun has finally realised that the Java Plugin implementation needs reworking. There is now a proposal for a minimal Java runtime environment named “Java Kernel”, which could be used to make the Java plugin indistinguishable from other plugins in terms of loading times.

Although Sun has already lost a lot of its credibility in the web development community this could be their chance to redeem themselves.

Notes

[1] Flash, the obvious choice, seems to get used quite a lot recently to build web-based infoviz solutions, but most of them tend to be custom built with little code reuse. Performance is another known issue with Flash.

Pure Javascript solutions like JSViz are interesting but currently too slow for visualisations of big datasets (as discussed recently on Ajaxian).

The best solution would probably be a mix of JS and Java, combinbing Java’s performance and powerful apis with an easy to use Javascript programming model.

Update: 20/11/06

Chet Haase at Sun suggests the name Applet-JAX: his blog post mentions the possibilities of mixing applets and other media in order to get away from Java-only solutions. However, the demo he provides does not make use of asynchronous updates.

Update: 28/11/06

My colleague Mike came up with “AppleJax”: a lot easier to use in conversations. Names are important!

i’ve just spent an age tracking down a problem parsing an xml file with commons digester. it turned out to be because someone had missed an element out of the hierarchy, and digester was trying to call a method which didn’t exist because of that

tracking down the problem took ages because there is no information in java stack traces other than a list of method calls. if each method called had an ability to annotate the current stack frame with some application related information [ the name or path of the xml element currently being processed in this case ], then i wouldn’t have needed to fire up the debugger and manually correlate the stack trace with objects in the debugger

all that would be needed would be some method like :

Thread.annotateStackTrace(String)