Fixing Firefox performance and lock-ups on Linux

I’ve been using Ubuntu (8.04 then 8.10) reasonably heavily over the last 12 months or so as my main operating system on two of my home computers (a Dell Mini 9″ and a quad core desktop).

I’ve been pretty happy with it except for the infuriating habit Firefox has for “locking up” periodically. The symptoms of this include non-reponsiveness, screen freezing and even the computer being unusable for 30 seconds at a time. The only clue I had was there seemed to be large amounts of disk usage.

After a while bitching and moaning about it I got so annoyed I started looking for a fix.

The first thing I tried was the Chromium Linux nightly builds. Google says “blogging about this isn’t helpful”, so I won’t except to say that I’ve got pretty high expectations of Chrome on Linux and so far I haven’t had to re-apprise that.

All the same, I wanted to fix Firefox. The next thing I tried was moving Firefox cache to a RAM drive. That’s pretty easy - just set it to use a directory under /dev/shm/ for the cache location.

I think that improved the situation marginally, but not enough to call it a fix.

The next thing I tried was to raise a Firefox bug. Somewhat to my surprise that got linked to another bug which was marked as fixed.

The comments on that bug are quite long, but the story is this:

  • Firefox uses SQLite as a database for its history and bookmarks.
  • SQLite, being a database is very concerned about data integrity, and to implement this it relies on the fsync system call.
  • fsync has performance issues on ext3 filesystems. See for example, http://lwn.net/Articles/328363/:

    The problem, in short, is this: the ext3 filesystem, when running in the default data=ordered mode, can exhibit lengthy stalls when some process calls fsync() to flush data to disk. This issue most famously manifested itself as the much-lamented Firefox system-freeze problem, but it goes beyond just Firefox.

  • SQLite has a no-sync mode, which trades reliability for performance.
  • Firefox can use this mode via a config setting.

So the outcome of all that is this:

Create a new config key “toolkit.storage.synchronous” and set it to the integer 0 to stop Firefox lock-ups on Linux (but be aware that there is some chance a power failure could cause loss of your history and/or bookmarks).

Comments (1)

The AppEngine is forking Java “controversy”

So there has been some noise from Sun about how Google AppEngine is evil because it’s not supporting the complete set of classes in the JRE. I’m sorry Sun - I’m a Java programmer, and I think that argument is shit. I’d much prefer a partial Java implementation with well defined limitations than PHP, or Python or Ruby.

AFAIK, no one has posted a list of classes missing. I can’t be bothered doing that either, but I did manually take a look at package level. Here’s it looks like GAE/J is missing:

java.applet
java.awt.*
javax.activation
javax.imageio.*
javax.jws.*
javax.management.*
javax.naming.*
javax.net.*
javax.print.*
javax.rmi.*
javax.sound.*
javax.swing.*
javax.tools
javax.xml.bind.*
javax.xml.crypto.*
javax.xml.soap
javax.xml.stream.*
javax.xml.ws
org.ietf.jgss
org.omg.*

From that list, I’d like to see javax.activation, javax.management and the remaining javax.xml.* and maybe javax.tools packages supported. The rest really don’t seem at all relevant to the AppEngine environment.

Comments (4)

Random MP3 metadata code

I’ve been doing random MP3 metadata work lately. Here’s some code which others might find useful.

Extracting MP3 tags from mp3 file hosted on server using HTTP Range queries.

So I was using Apache Tika for various metadata stuff. I wanted to get the song title for a file hosted on a server, but Tika only supports MP3 ID3v1 metadata, which exists at the end of a file. Downloading an entire MP3 just for the title is wasteful, but fortunatly HTTP Range queries can help us out.

HttpClient httpClient = new HttpClient();
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(10000);
httpClient.getHttpConnectionManager().getParams().setSoTimeout(10000);

String address = "http://address of mp3 file here";

HttpMethod method = new HeadMethod();
method.setURI(new URI(address,true));

Header contentLengthHeader = null;
Header acceptHeader = null;

httpClient.executeMethod(method);
try {
	//System.out.println(Arrays.toString(method.getResponseHeaders()));
	contentLengthHeader = method.getResponseHeader("Content-Length");
	acceptHeader = method.getResponseHeader("Accept-Ranges");
} finally {
	method.releaseConnection();
}

if ((contentLengthHeader != null) && (acceptHeader != null) && "bytes".equals(acceptHeader.getValue())) {
	long contentLength = Long.parseLong(contentLengthHeader.getValue());
	long metaDataStartRange = contentLength - 128;
	if (metaDataStartRange > 0) {
		method = new GetMethod();
		method.setURI(new URI(address,true));
		method.addRequestHeader("Range", "bytes=" + metaDataStartRange + "-" + contentLength);
		System.out.println(Arrays.toString(method.getRequestHeaders()));
		httpClient.executeMethod(method);
		try {
			Parser parser = new AutoDetectParser();

			Metadata metadata = new Metadata();
			metadata.set(Metadata.RESOURCE_NAME_KEY, address);
			InputStream stream = method.getResponseBodyAsStream();
			try {
				parser.parse(stream, new DefaultHandler(), metadata);
			} catch (Exception e) {
				e.printStackTrace();
			} finally {
				stream.close();
			}
			System.out.println(Arrays.toString(metadata.names()));
			System.out.println("Title: " + metadata.get("title"));
			System.out.println("Author: " + metadata.get("Author"));
		} finally {
			method.releaseConnection();
		}
	}
} else {
	System.err.println("Range not supported. Headers were: ");
	System.err.println(Arrays.toString(method.getResponseHeaders()));
}

The next thing I needed to do was extract song titles from a shoutcast stream. Shoutcast streams are kinda-but-not-quite http. Metadata is embedded in the stream (not as part of the MP3). That makes the code pretty ugly, but whatever… This code will open a connection, read the metadata and close, so you don’t need to keep downloading gigs of data.

URL url = new URL("http://scfire-ntc-aa01.stream.aol.com:80/stream/1074");
URLConnection con = url.openConnection();
con.setRequestProperty("Icy-MetaData", "1");

InputStream stream = con.getInputStream();
try {

	BufferedReader in = new BufferedReader(new InputStreamReader(stream));

	String metaIntervalString = null;
	// get the headers
	StringBuilder headers = new StringBuilder();
	char c;
	while ((c = (char)in.read()) != -1) {
		headers.append(c);
		if (headers.length() > 5 && (headers.substring((headers.length() - 4), headers.length()).equals("\r\n\r\n"))) {
			// end of headers
			break;
		}
	}

	//System.out.println(headers);
	// headers look like this:
	//		ICY 200 OK
	//		icy-notice1: 
This stream requires Winamp
// icy-notice2: Firehose Ultravox/SHOUTcast Relay Server/Linux v2.6.0
// icy-name: .977 The 80s Channel // icy-genre: 80s Pop Rock // icy-url: http://www.977music.com // content-type: audio/mpeg // icy-pub: 1 // icy-metaint: 16384 // icy-br: 128 Pattern p = Pattern.compile("\\r\\n(icy-metaint):\\s*(.*)\\r\\n"); Matcher m = p.matcher(headers.toString()); if (m.find()) { metaIntervalString = m.group(2); } if (metaIntervalString != null) { int metaInterval = Integer.parseInt(metaIntervalString.trim()); if (metaInterval > 0) { int b; int count = 0; int metaDataLength = 4080; // 4080 is the max length boolean inData = false; StringBuilder metaData = new StringBuilder(); while ((b = stream.read()) != -1) { count++; if (count == metaInterval + 1) { metaDataLength = b * 16; } if (count > metaInterval + 1 && count < (metaInterval + metaDataLength)) { inData = true; } else { inData = false; } if (inData) { if (b != 0) { metaData.append((char)b); } } if (count > (metaInterval + metaDataLength)) { break; } } String metaDataString = metaData.toString(); System.out.println(metaDataString); } } } finally { stream.close(); }

Comments

ROME 1.0 Released

I’ve just pushed out ROME 1.0 and ROME Fetcher 1.0.

As they say with open source projects - “it’s done when it’s done”. But nearly 5 years to get to version 1.0 is kind of long.

Comments (4)

Software of the year, 2008

Yeah, I know it’s kinda late for a post like this.

The award for oh-god-this-is-what-sharepoint-should-be: Dropbox. Also, drop.io, but dropbox has the backup problem solved, too.

The does-anyone-remember-just-how-buggy-browsers-used-to-be award: Chrome (note that Firefox is bad - just that Chrome is so amazingly good for a first release)

The year-of-linux-on-the-desktop-yes-really-this-time-well-maybe-anyway-award : Ubuntu 8.10 (it’s not perfect, but installs well, doesn’t crash, and doesn’t annoy me as much as Vista)

The this-search-shit-is-easy-anyway: Apache Solr (Solr is the first development tool where I’ve ever felt any reluctance in sharing around. It’s so good it sometimes seems to be a magic bullet)

Comments

ROME 1.0RC2 Release

I’ve just pushed out a release of ROME core, ROME Fetcher and ROME modules.

For those who don’t know, ROME is a (the?) Java library for handling RSS and Atom. Unlike some other libraries it is pretty stable (18 months since the last release) and has a low number of dependencies (one - JDom - if all you need is parsing)

The annoucement, including links is at https://rome.dev.java.net/servlets/ReadMsg?list=dev&msgNo=2656

The thing I’m most pleased about (and the number one source of complaints about ROME) is that I’ve pushed it to the java.net Maven repository, so now it will be easier to use from Maven. Further details are at http://wiki.java.net/bin/view/Javawsxml/RomeAndMaven2

Comments (2)

Installing Java on RedHat Linux by building your own RPM

It’s pretty easy to install Java on Linux - download the RPM from sun and install it. Then if you run “java -version” you’ll suddenly discover that it doesn’t really work:

java version "1.4.2"
gij (GNU libgcj) version 4.1.2 20070626 (Red Hat 4.1.2-14)

You can get around that by setting your path and JAVA_HOME, or by only using Java version that have a matching JPackage RPM and using the alternatives command

If you want to be able to build your own RPM, here’s how to do it.

 

# Be sure to enable the distro specific repository for your distro below:
# - jpackage-fc for Fedora Core
# - jpackage-rhel for Red Hat Enterprise Linux and derivatives

[jpackage-generic]
name=JPackage (free), generic
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=1.7
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=1

[jpackage-fc]
name=JPackage (free) for Fedora Core $releasever
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=fedora-$releasever&type=free&release=1.7
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=0

[jpackage-rhel]
name=JPackage (free) for Red Hat Enterprise Linux $releasever
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=rhel-$releasever&type=free&release=1.7
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=0

[jpackage-generic-nonfree]
name=JPackage (non-free), generic
mirrorlist=http://www.jpackage.org/jpackage_generic_nonfree_1.7.txt
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=1
  • Become root
  • Copy this file to /etc/yum.repos.d. Edit it, and make sure that enabled=1 is set for the [jpackage-generic-nonfree] section.
  • Make directories required by the RPM process (I suspect you can do this outside the /usr/src directory, though):  
mkdir -p /usr/src/redhat/SOURCES  
mkdir -p /usr/src/redhat/RPMS/i586/
  • Copy the Java installation file you previously downloaded to /usr/src/redhat/SOURCES and make it executable (chmod +x <name of file>)
  • Install the tools you need to build an rpm: yum install yum-utils jpackage-utils rpm-build  (At the moment this seems to fail on 64bit machines because of missing dependencies)
  • cd usr/src/redhat/SOURCES
  • yumdownloader –source java-1.6.0-sun
  • At the moment, that will download a file called java-1.6.0-sun-1.6.0.10-1jpp.nosrc.rpm
  • Run setarch i586 rpmbuild –rebuild java-1.6.0-sun*nosrc.rpm. At the moment that gives an error message, which seems to be able to be ignored:
sh: /usr/src/redhat/SOURCES/jdk-6u10-linux-i586.bin: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.6041 (%prep)
RPM build errors:
    user jasonc does not exist - using root
    group jasonc does not exist - using root
    user jasonc does not exist - using root
    group jasonc does not exist - using root
    user jasonc does not exist - using root
    group jasonc does not exist - using root
    Bad exit status from /var/tmp/rpm-tmp.6041 (%prep)
  • That previous command extracted a RPM SPEC file in the /usr/src/redhat/SPECS/ directory.
  • Edit /usr/src/redhat/SPECS/java-1.6.0-sun.spec. Find the part that says %define buildver and change the value to the build for the new version of Java
  • Run rpmbuild -ba /usr/src/redhat/SPECS/java-1.6.0-sun.spec. This extracts the JDK installer you previously downloaded and builds a set of RPMs from it.
  • cd /usr/src/redhat/RPMS/i586; ls;

java-1.6.0-sun-1.6.0.11-1jpp.i586.rpm        java-1.6.0-sun-fonts-1.6.0.11-1jpp.i586.rpm
java-1.6.0-sun-alsa-1.6.0.11-1jpp.i586.rpm   java-1.6.0-sun-jdbc-1.6.0.11-1jpp.i586.rpm
java-1.6.0-sun-demo-1.6.0.11-1jpp.i586.rpm   java-1.6.0-sun-plugin-1.6.0.11-1jpp.i586.rpm
java-1.6.0-sun-devel-1.6.0.11-1jpp.i586.rpm  java-1.6.0-sun-src-1.6.0.11-1jpp.i586.rpm
  • You can now install the RPM: rpm -i java-1.6.0-sun-1.6.0.11-1jpp.i586.rpm
  • For me that failed with a missing X dependency: libXtst.so.6 is needed by java-1.6.0-sun-1.6.0.11-1jpp.i586
  • I fixed that with yum -y install libX11-devel libXtst.
  • Use the alternatives command to set the correct version of Java: alternatives –config java
  • Finally: java -version:

java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)
That’s it - you finally have Java working on Linux! You also have a RPM which can be installed on other machines.

Comments (6)

My Google Interview

Well it’s just under 5 months since I promised a post about my Google interview. Being the highly active blogger that I am, I’d say its about time..

Back in early April I got an email from a recruiter from the Google European Recruiting Centre. I was a little puzzled how she got my name, but a bit of detective work found a guy I went to school with worked for Google in the London office and he’d recommended me.

After a phone converstation with the recruiter, we decided that a position in London would suit me best (I live in Australia, and have a wife and a then-2 year old child, so this was kinda a big deal), and I was handed over to a recruiter from London.

Google’s London office mostly does mobile applications, and I wanted to do web apps, but I was assured this wouldn’t be a problem because they had positions like that available too.

So the wheels were set in motion to setup the dreaded Google phone screen, and I started studying. It’s hard to explain how difficult I found that. Not only did I have to remember a whole lot of half-remembered computer science from ten years ago, but I also had to remember the Math that went with it, which was much harder to dig up. Things like big-O notion are easy enough conceptually, but when you are actually analyzing an algorithm you need to remember the maths for dealing with logorithms (for example), which isn’t something I’ve thought about much since doing Computer Science 4 back in 1997…

Interestingly, every recruiter I spoke to referred me to Steve Yegge’s “Get that Job at Google” post which I found ironic considering the “I don’t speak for Google” disclaimers he uses. 

Anyway, in early May I did the phone interview with a programmer from London. While I’m not going to go into specific questions, it did involve writing some code (in Google Docs), and some highish level problem solving. For me, I found that I’d read some of Steve Yegge’s other writing pretty valuable.

I finished the phone interview feeling pretty reasonable, but I was still pretty pleased when I got called to arrange some on-site interviews in Sydney. 

We arranged to do two in-person interviews with engineers in Sydney, and then two video-conference interviews with London, in late May.

So I went back to studying. Working through the stuff on Steve Yegge’s post was actually getting me more and more worried about all the stuff I didn’t know, but what else could I do?

In my first interview, the first question was pretty much my nightmare scenario. It was a (computer) math question, a (to quote the interviewer) “easy question to start you off” and it was something I didn’t know, and even worse - it was something I’d known I didn’t know but had left in favour of other things. So I muddled though the best I could and got the answer in the end, and the subsequent questions from that interviewer were better, but I was pretty worried that I’d blown it badly.

The next interview was much better. It was pretty clear that the first interviewer had told the second one that I was pretty nervous, because he kept telling me to slow down and not to worry too much. His first question was something I was much more confident about, and I got the naive solution out pretty quickly. Even better, I was able to identify that the class of problem was the same as something I’d been asked before, so I was able to skip the obvious improvement and got straight from the naive solution to the optimal solution in one step. The interviewer was happy about that, and let me choose if I wanted a low level or high level question next. 

I chose a high level question, and he gave me a “design and sketch-code an appropriate interface” problem. I was very happy with that question because it’s the kind of thing I deal with most days in my work. I have what I though was an adequate answer, although I could see a lot of problems with my implementation. the interviewer was very happy with it, though, and said it was the best answer he’d seen. That surprised me, because I could see areas to improve, but I’d run out of time, so suspect him telling me that was a technique to try and get me over my “nerves”.

The third interview was the first video-conference one. There was nothing the stood out at me in this one, except it was the only interview where there was a question that involved talking about design trade offs etc instead of coding. There was also an interesting question where I forgot a pretty basic computer science concept, but once I got a hint I solved it reasonably.

The fourth interview was the interesting one. The first question involved writing a solver for a puzzle-type game. Unfortunately, it wasn’t a game I’d played before and that really cost me because I didn’t know how I’d go about solving it. To be honest, I stuggled pretty badly with this one. I did write a checker, to determine if a given solution was valid, but it was the one question I had to give up on.

The second question from that interview was probably the best question I was given. I wish I could post it here, because the approach to solving it and the optimizations used were just so typical of all the other questions, and the optimal solution is glorious, and yet is easy to understand.

So that was it - the famous Google interview. I can’t say that there were any surprises, and I came out of it with mixed feelings.

I felt that I’d done reasonably well. I’d missed one question, and struggled in another, but I thought some of my other answers were pretty good, and I hoped my second interview might have been enough to get me over the line.

I was hoping to find out quickly how I’d gone, but that wasn’t to be. My next contact with Google was at a Developer Day here in Adelaide. I was fortunate in that the recruitement consultant from Sydney was at that event, and I’d met him at the interview. One of my interviewers was also there (the second interview - the one I’d done well at).

I spoke to them both, and both were pretty positive. The recruiter actually said he’d looked at my feedback and that I shouldn’t plan to be in my current job much longer, and both asked if I was fixed on a job in London or if I’d be interested in Sydney. They both mentioned again that I’d been very nervous, which I gladly agreed with (anything to excuse my bad answers!)

I came away from that event feeling pretty optimistic.

A couple of weeks later I finally got a response from London that the position I’d been going for had been taken by an internal applicant, but that they’d like to do more interviews with me for another postion on the mobile team. I also felt that the feedback on my interview was very mixed - some very good, and some not so good, which made them feel they wanted to do another interview. That pretty much brings me up to my last post on the topic.

Some common questions:

  • Do you get asked puzzle/brainteaser questions?
    • No - they were all algorithm and coding.
  • Was it as hard as everyone says?
    • Yes. By far the hardest 5 interviews I’ve ever done.
So would I do it again? Yes I would, but I’d probably go for a position nearer to where I live. I’d also do a few things differently WRT to studying. Instead of working my way though Steve Yegge’s study list, I think I’d probably concentrate a lot more on the TopCoder algorithm questions.

Comments

Sticky Sessions which aren’t

Exhibt A (the Apache 2.2 mod_proxy documentation): 

stickysession - Balancer sticky session name. The value is usually set to something like JSESSIONID or PHPSESSIONID, and it depends on the backend application server that support sessions. If the backend application server uses different name for cookies and url encoded id (like servlet containers) use | to to separate them. The first part is for the cookie the second for the path.

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

 

Exhibit B (the Apache 2.2 mod_proxy code):

    /*
     * If we found a value for sticksession, find the first '.' within.
     * Everything after '.' (if present) is our route.
     */
    if ((*route) && ((*route = strchr(*route, '.')) != NULL ))
        (*route)++;
    if ((*route) && (**route)) {
        ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
                                  "proxy: BALANCER: Found route %s", *route);
        /* We have a route in path or in cookie
         * Find the worker that has this route defined.
         */
        worker = find_route_worker(balancer, *route, r);
        if (worker && strcmp(*route, worker->s->route)) {
            /*
             * Notice that the route of the worker chosen is different from
             * the route supplied by the client.
             */
            apr_table_setn(r->subprocess_env, "BALANCER_ROUTE_CHANGED", "1");
            ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
                         "proxy: BALANCER: Route changed from %s to %s",
                         *route, worker->s->route);
        }
        return worker;
    }

http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_balancer.c?revision=687754&view=markup

Translation: In Apache mod_proxy_balancer, sitcky sessions need to be in the format <cookie value>.<route> where route refers to the backend server. 

 

There are a couple of useful docs about this: Mark Round and the Jetty Docs. Normally I’d submit a patch to the Apache docs to clear this up, but unfortunately I haven’t been able to get failover between servers working correctly, so I’m not convinved I completely understand all the issues yet.

However, once you have got this working it turns out to be a case of RTFM.

Exhibit C:

route - Route of the worker when used inside load balancer. The route is a value appended to session id.

 

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

I’d argue that only makes any sense after you’ve got it working…

Comments

Modify java.library.path at runtime

Linking to native code in Java is always a hassle. JNI isn’t exactly nice, and there are some oddities around classloaders and native libraries which are annoying if you run into them.

One thing I wasn’t aware of was exactly how hard it is to load a library it isn’t already in the directories specified by the java.library.path system property. 

Initially, I thought I’d just be able to alter that property and the JVM would pick up the new locations. That turns out not to be the case, as is shown by this (closed) bug report.

However, there is a solution, outlined in this post on the Sun forums, which revolves around altering the usr_paths field stored in java classes.

	public static void addDir(String s) throws IOException {
		try {
			// This enables the java.library.path to be modified at runtime
			// From a Sun engineer at http://forums.sun.com/thread.jspa?threadID=707176
			//
			Field field = ClassLoader.class.getDeclaredField("usr_paths");
			field.setAccessible(true);
			String[] paths = (String[])field.get(null);
			for (int i = 0; i < paths.length; i++) {
				if (s.equals(paths[i])) {
					return;
				}
			}
			String[] tmp = new String[paths.length+1];
			System.arraycopy(paths,0,tmp,0,paths.length);
			tmp[paths.length] = s;
			field.set(null,tmp);
			System.setProperty("java.library.path", System.getProperty("java.library.path") + File.pathSeparator + s);
		} catch (IllegalAccessException e) {
			throw new IOException("Failed to get permissions to set library path");
		} catch (NoSuchFieldException e) {
			throw new IOException("Failed to get field handle to set library path");
		}
	}

Obviously, I don’t think that’s portable across JVMs, though.

Comments (2)