Category Archives: java

Why tech predictions are stupid (and a small prediction)

Every year hundreds of tech pundits go and make their predictions for the year – a trend I’m not immune to either. Alan Kay explained the problem with this the best: “The best way to predict the future is to invent it”. In a field like computing it is so easy for a single person to build something new it makes trying to make predictions a pointless Lose Weight Exercise.

None the less, here’s something that is less of a prediction and more an Lose Weight Exercise in deduction and rumor mongering. Sun is planning to launch a direct competitor to Amazon’s EC2 in the near future (not sure when exactly, but 2008 for sure). Note that this is different to the existing Sun Grid product (which will presumably continue).

Shipping software part 2

In shipping software I spoke briefly about me.edu.au (which is still taking a good amount of my time). Recently, though I’ve been spending a lot of time preparing education.au’s Java based federated search product (the Distributed Search ManagerOpenDSM) for release as an open source product. That’s been an interesting experience – the code is pretty old, and was glued together using static references. I had to pull it part, replace the static references with factories (changing to dependency injection wasn’t realistic for this release at least) and put it back together. It’s kind of odd working on a project like that – the code almost causes me pain at times, but with a product that is stable and reliable I don’t want to make too many changes just because I don’t like the style.

Some readers of this blog may be interested in it, because it allows federating of results from multiple Solr servers (or OpenSearch services) together into a single result set. That’s useful in quite a large set of places.

It’s also the first time I’ve been paid to create open source code as an explicit goal – most of my open source work has been for pragmatic reasons, not as a goal in itself.

Run your web code from your IDE

Many Java developers complain about the length of time it takes to deploy their webapps for development, or to start the embedded WebTools server in Eclipse, or to use the mvn jetty:run process with Maven.

At work, we’ve created a solution using class which starts up Jetty and gets it to serve your webapp from within the IDE. This means you have start up times of seconds, and can debug without using remote debugging.

Happily, we’ve been able to open source this, too, so people can try it for themselves rather than just here me talk about it.

The source code is here, but you are probably better off using Maven, adding the following repository to your pom.xml:


<repository>
<id>EdAuUtil</id>
<url>http://maven.uat.educationau.edu.au/EdAuUtil</url>
</repository>

and adding the following dependencies:

<dependency>
<groupId>au.edu.educationau.opensource</groupId>
<artifactId>EdAuUtils</artifactId>
<version>0.9</version>
</dependency>

<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty</artifactId>
<version>6.1.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<version>6.1.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>jasper</artifactId>
<version>6.0.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-ant</artifactId>
<version>0.8</version>
<scope>test</scope>
</dependency>

Then you can run the class au.edu.educationau.opensource.devsupport.TestWebServer from your IDE. I need to generate the javadocs and put them up somewhere, but for the moment the code is reasonably well documented.

Shipping software

http://me.edu.au/p/nlothian

Custom vertical social network, with feed integration (kind of like the Facebook activity stream, but driven by RSS), communities and single-sign on across a number of other applications.

Open Social support coming soon. I should do a post on how the open source implementation of that works. For the time being, though, this is all the code you need to show a Google Gadget into your app:

<c:import url=”/ifr”>
<c:param name=”url”>http://www.labpixies.com/campaigns/todo/todo.xml</c:param>
</c:import>

Guessing is much quicker than debugging

My previous post I’ve already tried the ‘waving a dead chicken over our servers’ trick attracted a bit of attention, and quite a number of suggestions – thanks to all who contributed. The suggestions seemed to fall into four main categories:

  1. Database tuning.
    • This is a good suggestion, and is something we’ve done a fair bit of. In this case it doesn’t really help because the problem wasn’t performance but stability.
  2. Introduce a caching layer
    • We’d already done this, twice. We initially used an ehcache caching filter to fix some pretty serious performance problems. We later added some OSCache JSP cache tags in some critical areas in some templates (it was the addition of OSCache which caused the performance boost seen in my post on monitoring performance using the Google Webmaster Tools). As it turned out this combination may have been what caused our problem.
  3. Rewrite everything
    • Thanks. Let me know when you get a job in the real world.
  4. Debug the problem
    • This is what I figured we’d have to do. It’s something I was attempted to avoid because the issue seemed to be threading related, and we couldn’t reproduce it anywhere except our production environment.

We did have one stroke of good luck. We were able to predict when the site would stop working by monitoring the number of threads Apache was using and we could use this information to preemptively restart the site. We were able to modify the restart script to generate stack traces for all the JMV’s threads (kill -SIGQUIT <jvm pid>).

Since it looked like I’d actually have to start debugging this problem I started looking through the stack traces and I noticed that lots of the threads were in the ehcache filter. Now this wasn’t necessarily a bad thing, since all http request would be passed though it. However, it did make debugging harder, was easy to remove (just comment it out in the web.xml) and did have some potential to be a source of problems – in particular the cache-invalidation part.

So we took a punt and removed the filter and… it fixed the problem. Yay! I’m a genius and all that.

Except…. now the CMS is crashing with a NullPointerException deep in the data persistence layer. There’s also the small problem that I don’t have a clue why that change fixed it. Using the ehcache filter on its own worked fine, and there is no programmatic interaction between the ehcache and oscache code.

There is an alleged fix for the NullPointerException – but we have to take a point release of the CMS, and then patch it with a service pack to get it. Our previous experience with upgrades have been less than confidence inspiring.

In the mean time we have a script watching the site and restarting it when it crashes. It’s kind of like failover, without the over bit.

“You should just buy Google”

Me to Unix Admin @ work: Hey – so I’m doing estimates for a proposal which needs between 30 and 250 TB of storage – what do you know about mass storage?

Unix Admin: Hmm.. you should just buy Google….

Okay then! But seriously – Amazon S3 seems the obvious solution, and I’m also looking at OmniDrive. Any other suggestions are welcome. No hardware suggestions, though – I don’t have enough confidence in our operations group to do it in house (mainly because we don’t have an operations group…)