Many Java developers complain about the length of time it takes to deploy their webapps for development, or to start the embedded WebTools server in Eclipse, or to use the mvn jetty:run process with Maven.

At work, we’ve created a solution using class which starts up Jetty and gets it to serve your webapp from within the IDE. This means you have start up times of seconds, and can debug without using remote debugging.

Happily, we’ve been able to open source this, too, so people can try it for themselves rather than just here me talk about it.

The source code is here, but you are probably better off using Maven, adding the following repository to your pom.xml:


<repository>
<id>EdAuUtil</id>
<url>http://maven.uat.educationau.edu.au/EdAuUtil</url>
</repository>


and adding the following dependencies:

<dependency>
<groupId>au.edu.educationau.opensource</groupId>
<artifactId>EdAuUtils</artifactId>
<version>0.9</version>
</dependency>

<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<version>6.1.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<version>6.1.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>jasper</artifactId>
<version>6.0.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-ant</artifactId>
<version>0.8</version>
<scope>test</scope>
</dependency>

Then you can run the class au.edu.educationau.opensource.devsupport.TestWebServer from your IDE. I need to generate the javadocs and put them up somewhere, but for the moment the code is reasonably well documented.

My previous post I’ve already tried the ‘waving a dead chicken over our servers’ trick attracted a bit of attention, and quite a number of suggestions – thanks to all who contributed. The suggestions seemed to fall into four main categories:

1. Database tuning.
   • This is a good suggestion, and is something we’ve done a fair bit of. In this case it doesn’t really help because the problem wasn’t performance but stability.
2. Introduce a caching layer
   • We’d already done this, twice. We initially used an ehcache caching filter to fix some pretty serious performance problems. We later added some OSCache JSP cache tags in some critical areas in some templates (it was the addition of OSCache which caused the performance boost seen in my post on monitoring performance using the Google Webmaster Tools). As it turned out this combination may have been what caused our problem.
3. Rewrite everything
   • Thanks. Let me know when you get a job in the real world.
4. Debug the problem
   • This is what I figured we’d have to do. It’s something I was attempted to avoid because the issue seemed to be threading related, and we couldn’t reproduce it anywhere except our production environment.

We did have one stroke of good luck. We were able to predict when the site would stop working by monitoring the number of threads Apache was using and we could use this information to preemptively restart the site. We were able to modify the restart script to generate stack traces for all the JMV’s threads (kill -SIGQUIT <jvm pid>).

Since it looked like I’d actually have to start debugging this problem I started looking through the stack traces and I noticed that lots of the threads were in the ehcache filter. Now this wasn’t necessarily a bad thing, since all http request would be passed though it. However, it did make debugging harder, was easy to remove (just comment it out in the web.xml) and did have some potential to be a source of problems – in particular the cache-invalidation part.

So we took a punt and removed the filter and… it fixed the problem. Yay! I’m a genius and all that.

Except…. now the CMS is crashing with a NullPointerException deep in the data persistence layer. There’s also the small problem that I don’t have a clue why that change fixed it. Using the ehcache filter on its own worked fine, and there is no programmatic interaction between the ehcache and oscache code.

There is an alleged fix for the NullPointerException – but we have to take a point release of the CMS, and then patch it with a service pack to get it. Our previous experience with upgrades have been less than confidence inspiring.

In the mean time we have a script watching the site and restarting it when it crashes. It’s kind of like failover, without the over bit.

Me to Unix Admin @ work: Hey – so I’m doing estimates for a proposal which needs between 30 and 250 TB of storage – what do you know about mass storage?

Unix Admin: Hmm.. you should just buy Google….

Okay then! But seriously – Amazon S3 seems the obvious solution, and I’m also looking at OmniDrive. Any other suggestions are welcome. No hardware suggestions, though – I don’t have enough confidence in our operations group to do it in house (mainly because we don’t have an operations group…)

The title of this post says it all, really. But the long story follows… (this made TheServerSide. Unfortunately I can’t login there – I think that’s our work firewall problem – so I’ll update here to give a few more details)..

Continue reading Updated:I’ve already tried the ‘waving a dead chicken over our servers’ trick →