@Geoff: Redhat AS3. It runs out of threads and stops responding (or actually Apache runs out of threads). Increasing the threadcount in Apache just delays hang.

CMS site with some custom code for dynamic content. The dynamic stuff was a bit slow so i put in some caching. It was still a bit slow so i guessed i could make things faster by eliminating synchronization of the cache access - hence the double-checking-locking. All went swimmingly, including up to 2 days of continuous heavy load testing on a single CPU server.

In case you’re not familiar with the double-checked problem, its when you write code like this..

Object getCachedThing() {
if( cached == null ) {
synchronized {
if( cached == null ) {
cached = lookupFromDatabase()
}
}
return cached
}
}

There’s lots of literature on it but the bottom line is it can’t ever work reliably in a multi CPU environment. Note that in my case i was looking up via a key and storing in a hashmap.

Once in production (on clustered multi CPU machines) the site would gradually grind to a halt. For some reason, the double checked thing resulted in threads deadlocking - not at all an expected result even once i knew that double-checking was a no no. I thought i’d get null pointers. Looks like its something in HashMap which can make it dead lock if accessed by multiple threads.

Might not be your problem, but sound like you have the same symptoms i had

Cheers

@Tim: We’ve tried the index thing. It is actually possible that the default database setup has too many indexes - but as I said, database load doesn’t seem to be an issue.

I suspect a threading issue, too. I have some experience debugging that kind of problem, and it’s not something I want to do - especially when it isn’t our code. Hence the waving of dead chickens…

__Crazy solutions__
You could write your own JDBC driver and catch the inefficient/unnecessary queries and handle them yourself.
If the queries are moderately well written, then it should be feasible enough to work.

__Stability__
In my experience, lock ups tend to be concurrency/synchronization issues. The last one I ran into was a log4j issue (in an older version) where all the threads were locked up on the async buffer.
The thread dumps don’t sound like fun, but they sound like the best path to enlightenment.

Just another couple of questions:
you say that “after a some hours running the site just stops responding”.

There is a “time pattern” in this behaviour?
A - it stops responding after X hours from the restart
B - it stops every day at about X o’clock
C - it stops every morning when the load begins to increase

Last question: have you the possibility to increase the number of the DB connections available to the CMS. If you’ve already tried it, what effect it had on performance and stability?

Very often, the dynamic parts are minimal: most of the page comes from authored content that is pretty static, time-wise.

There are different ways to “inject” bits of dynamic data in a static cached page, some ugly, some less.

You can have the CMS generate client-side scripting that will call back the server for the dynamic bits and display them in the right places on the page. This requires caching, either at client side with a cookie, or between the client and the server.

Another option is to use the CMS as a template provider and have your application using these templates to decorate their dynamic outputs with a framework like Sitemesh.