Archive for tech

Cross domain, cross window Javascript communication

The scenario: A typical meeting, they want to pop up our search screen, select an item and have it post back via Javascript the selected item. “Sure, no problem” I say. “Can you do cross domain Javascript like that” their dev asks, and I think for a second about all the IFrame proxy hacks I’ve seen and say “yeah, no problems”.

Doh.

So I get back to work, and talk to Ben-out-local-Javascript-god, who points me at the location-fragment for communication hack (see, for example Cross Domain Frame Communication with Fragment Identifiers (for Comet?)) and away I go.

I quickly discover that all the documentation on that is for use with frames, not windows, and windows work slightly different. For example, the parent window can’t poll the child for it’s location.hash, nor can the child set the parent’s location.hash.

Doh Doh.

Anyway to cut a long story short and to document it for anyone else, you can make it work. The child window needs to contain a iframe from the parent domain, then proxy communication though that. Modern non IE browsers can use Window.postMessage to do the same thing (Actually, IE8 supports Window.postMessage, too, but I couldn’t get it to work properly).

The core of the IE specific code looks like this. The child window uses this object:

/*
  XDCommsChild - Cross Domain Communication, Child Object.

   Create this in the child window with the address of the xdomaincomms.html file on the same domain as the parent window
*/
var XDCommsChild = function(xdomaincommsAddress) {
    this.xdomaincommsAddress = xdomaincommsAddress;

    this.isIe = isInternetExplorer();

    if (isIe) {
        // need hidden frame for communication. IE8 is supposed to support postMessage, but I can't get it to work properly
        document.writeln("")
    }

    this.postBack = function(data) {
        if (isIe) {
            // this method tested against IE6,7 & 8
            window.open(this.xdomaincommsAddress + '#data=' + data, 'xdcomms'); // MUST use window.open. frame.src or frame.location both fail
        } else {
            // for everything else - tested against Firefox 3, Chrome, Safari
            window.opener.postMessage(data, '*'); // should really restrict the domain data can come from here
        }
    }

    this.postBackAndCloseWindow = function(data){
        this.postBack(data);
        setTimeout("window.close()", 200); // need to use a timeout to make sure the javascript in the frame has executed if we are using that communication model
    }
}

Then the included proxy frame (deployed on the parent domain) looks like this:

           var hash = null;

            function sendData() {
                hash = location.hash;
                if (hash != null && hash.length > 1) {
                    window.parent.opener.location.hash=hash;
                    location.hash = '#';
                }
            }

            setInterval("sendData()", 50);

The the parent window uses this object:

/*
  XDCommsParent - Cross Domain Communication, Parent Object.

   Create this in the parent window with a callback function that will recieve data.
*/
var XDCommsParent = function(dataRecievedCallback) {
    this.dataRecievedCallback = dataRecievedCallback;

    this.isIe = isInternetExplorer();

    this.receiveMessage = function(event) {
        dataRecievedCallback(event.data);
    }

    if (this.isIe) {
        window.setInterval("pollForData()", 200);
    } else {
        window.addEventListener("message", this.receiveMessage, false);
    }

}

The final piece of the solution is calling it from the child window. I do this:

<button onclick="javascript:xDCommsChild.postBackAndCloseWindow(document.getElementById('thevalue').value);">Send to Parent Window</button>

Comments (3)

Fixing Firefox performance and lock-ups on Linux

I’ve been using Ubuntu (8.04 then 8.10) reasonably heavily over the last 12 months or so as my main operating system on two of my home computers (a Dell Mini 9″ and a quad core desktop).

I’ve been pretty happy with it except for the infuriating habit Firefox has for “locking up” periodically. The symptoms of this include non-reponsiveness, screen freezing and even the computer being unusable for 30 seconds at a time. The only clue I had was there seemed to be large amounts of disk usage.

After a while bitching and moaning about it I got so annoyed I started looking for a fix.

The first thing I tried was the Chromium Linux nightly builds. Google says “blogging about this isn’t helpful”, so I won’t except to say that I’ve got pretty high expectations of Chrome on Linux and so far I haven’t had to re-apprise that.

All the same, I wanted to fix Firefox. The next thing I tried was moving Firefox cache to a RAM drive. That’s pretty easy – just set it to use a directory under /dev/shm/ for the cache location.

I think that improved the situation marginally, but not enough to call it a fix.

The next thing I tried was to raise a Firefox bug. Somewhat to my surprise that got linked to another bug which was marked as fixed.

The comments on that bug are quite long, but the story is this:

  • Firefox uses SQLite as a database for its history and bookmarks.
  • SQLite, being a database is very concerned about data integrity, and to implement this it relies on the fsync system call.
  • fsync has performance issues on ext3 filesystems. See for example, http://lwn.net/Articles/328363/:

    The problem, in short, is this: the ext3 filesystem, when running in the default data=ordered mode, can exhibit lengthy stalls when some process calls fsync() to flush data to disk. This issue most famously manifested itself as the much-lamented Firefox system-freeze problem, but it goes beyond just Firefox.

  • SQLite has a no-sync mode, which trades reliability for performance.
  • Firefox can use this mode via a config setting.

So the outcome of all that is this:

Create a new config key “toolkit.storage.synchronous” and set it to the integer 0 to stop Firefox lock-ups on Linux (but be aware that there is some chance a power failure could cause loss of your history and/or bookmarks).

Comments (1)

Random MP3 metadata code

I’ve been doing random MP3 metadata work lately. Here’s some code which others might find useful.

Extracting MP3 tags from mp3 file hosted on server using HTTP Range queries.

So I was using Apache Tika for various metadata stuff. I wanted to get the song title for a file hosted on a server, but Tika only supports MP3 ID3v1 metadata, which exists at the end of a file. Downloading an entire MP3 just for the title is wasteful, but fortunatly HTTP Range queries can help us out.

HttpClient httpClient = new HttpClient();
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(10000);
httpClient.getHttpConnectionManager().getParams().setSoTimeout(10000);

String address = "http://address of mp3 file here";

HttpMethod method = new HeadMethod();
method.setURI(new URI(address,true));

Header contentLengthHeader = null;
Header acceptHeader = null;

httpClient.executeMethod(method);
try {
	//System.out.println(Arrays.toString(method.getResponseHeaders()));
	contentLengthHeader = method.getResponseHeader("Content-Length");
	acceptHeader = method.getResponseHeader("Accept-Ranges");
} finally {
	method.releaseConnection();
}

if ((contentLengthHeader != null) && (acceptHeader != null) && "bytes".equals(acceptHeader.getValue())) {
	long contentLength = Long.parseLong(contentLengthHeader.getValue());
	long metaDataStartRange = contentLength - 128;
	if (metaDataStartRange > 0) {
		method = new GetMethod();
		method.setURI(new URI(address,true));
		method.addRequestHeader("Range", "bytes=" + metaDataStartRange + "-" + contentLength);
		System.out.println(Arrays.toString(method.getRequestHeaders()));
		httpClient.executeMethod(method);
		try {
			Parser parser = new AutoDetectParser();

			Metadata metadata = new Metadata();
			metadata.set(Metadata.RESOURCE_NAME_KEY, address);
			InputStream stream = method.getResponseBodyAsStream();
			try {
				parser.parse(stream, new DefaultHandler(), metadata);
			} catch (Exception e) {
				e.printStackTrace();
			} finally {
				stream.close();
			}
			System.out.println(Arrays.toString(metadata.names()));
			System.out.println("Title: " + metadata.get("title"));
			System.out.println("Author: " + metadata.get("Author"));
		} finally {
			method.releaseConnection();
		}
	}
} else {
	System.err.println("Range not supported. Headers were: ");
	System.err.println(Arrays.toString(method.getResponseHeaders()));
}

The next thing I needed to do was extract song titles from a shoutcast stream. Shoutcast streams are kinda-but-not-quite http. Metadata is embedded in the stream (not as part of the MP3). That makes the code pretty ugly, but whatever… This code will open a connection, read the metadata and close, so you don’t need to keep downloading gigs of data.

URL url = new URL("http://scfire-ntc-aa01.stream.aol.com:80/stream/1074");
URLConnection con = url.openConnection();
con.setRequestProperty("Icy-MetaData", "1");

InputStream stream = con.getInputStream();
try {

	BufferedReader in = new BufferedReader(new InputStreamReader(stream));

	String metaIntervalString = null;
	// get the headers
	StringBuilder headers = new StringBuilder();
	char c;
	while ((c = (char)in.read()) != -1) {
		headers.append(c);
		if (headers.length() > 5 && (headers.substring((headers.length() - 4), headers.length()).equals("\r\n\r\n"))) {
			// end of headers
			break;
		}
	}

	//System.out.println(headers);
	// headers look like this:
	//		ICY 200 OK
	//		icy-notice1: 
This stream requires Winamp
// icy-notice2: Firehose Ultravox/SHOUTcast Relay Server/Linux v2.6.0
// icy-name: .977 The 80s Channel // icy-genre: 80s Pop Rock // icy-url: http://www.977music.com // content-type: audio/mpeg // icy-pub: 1 // icy-metaint: 16384 // icy-br: 128 Pattern p = Pattern.compile("\\r\\n(icy-metaint):\\s*(.*)\\r\\n"); Matcher m = p.matcher(headers.toString()); if (m.find()) { metaIntervalString = m.group(2); } if (metaIntervalString != null) { int metaInterval = Integer.parseInt(metaIntervalString.trim()); if (metaInterval > 0) { int b; int count = 0; int metaDataLength = 4080; // 4080 is the max length boolean inData = false; StringBuilder metaData = new StringBuilder(); while ((b = stream.read()) != -1) { count++; if (count == metaInterval + 1) { metaDataLength = b * 16; } if (count > metaInterval + 1 && count < (metaInterval + metaDataLength)) { inData = true; } else { inData = false; } if (inData) { if (b != 0) { metaData.append((char)b); } } if (count > (metaInterval + metaDataLength)) { break; } } String metaDataString = metaData.toString(); System.out.println(metaDataString); } } } finally { stream.close(); }

Comments (3)

ROME 1.0 Released

I’ve just pushed out ROME 1.0 and ROME Fetcher 1.0.

As they say with open source projects – “it’s done when it’s done”. But nearly 5 years to get to version 1.0 is kind of long.

Comments (4)

Software of the year, 2008

Yeah, I know it’s kinda late for a post like this.

The award for oh-god-this-is-what-sharepoint-should-be: Dropbox. Also, drop.io, but dropbox has the backup problem solved, too.

The does-anyone-remember-just-how-buggy-browsers-used-to-be award: Chrome (note that Firefox is bad – just that Chrome is so amazingly good for a first release)

The year-of-linux-on-the-desktop-yes-really-this-time-well-maybe-anyway-award : Ubuntu 8.10 (it’s not perfect, but installs well, doesn’t crash, and doesn’t annoy me as much as Vista)

The this-search-shit-is-easy-anyway: Apache Solr (Solr is the first development tool where I’ve ever felt any reluctance in sharing around. It’s so good it sometimes seems to be a magic bullet)

Comments

Installing Java on RedHat Linux by building your own RPM

It’s pretty easy to install Java on Linux – download the RPM from sun and install it. Then if you run “java -version” you’ll suddenly discover that it doesn’t really work:

java version "1.4.2"
gij (GNU libgcj) version 4.1.2 20070626 (Red Hat 4.1.2-14)

You can get around that by setting your path and JAVA_HOME, or by only using Java version that have a matching JPackage RPM and using the alternatives command

If you want to be able to build your own RPM, here’s how to do it.

 

# Be sure to enable the distro specific repository for your distro below:
# - jpackage-fc for Fedora Core
# - jpackage-rhel for Red Hat Enterprise Linux and derivatives

[jpackage-generic]
name=JPackage (free), generic
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=1.7
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=1

[jpackage-fc]
name=JPackage (free) for Fedora Core $releasever
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=fedora-$releasever&type=free&release=1.7
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=0

[jpackage-rhel]
name=JPackage (free) for Red Hat Enterprise Linux $releasever
mirrorlist=http://www.jpackage.org/mirrorlist.php?dist=rhel-$releasever&type=free&release=1.7
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=0

[jpackage-generic-nonfree]
name=JPackage (non-free), generic
mirrorlist=http://www.jpackage.org/jpackage_generic_nonfree_1.7.txt
failovermethod=priority
gpgcheck=1
gpgkey=http://www.jpackage.org/jpackage.asc
enabled=1
  • Become root
  • Copy this file to /etc/yum.repos.d. Edit it, and make sure that enabled=1 is set for the [jpackage-generic-nonfree] section.
  • Make directories required by the RPM process (I suspect you can do this outside the /usr/src directory, though):  
mkdir -p /usr/src/redhat/SOURCES  
mkdir -p /usr/src/redhat/RPMS/i586/
  • Copy the Java installation file you previously downloaded to /usr/src/redhat/SOURCES and make it executable (chmod +x <name of file>)
  • Install the tools you need to build an rpm: yum install yum-utils jpackage-utils rpm-build  (At the moment this seems to fail on 64bit machines because of missing dependencies)
  • cd usr/src/redhat/SOURCES
  • yumdownloader –source java-1.6.0-sun
  • At the moment, that will download a file called java-1.6.0-sun-1.6.0.10-1jpp.nosrc.rpm
  • Run setarch i586 rpmbuild –rebuild java-1.6.0-sun*nosrc.rpm. At the moment that gives an error message, which seems to be able to be ignored:
sh: /usr/src/redhat/SOURCES/jdk-6u10-linux-i586.bin: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.6041 (%prep)
RPM build errors:
    user jasonc does not exist - using root
    group jasonc does not exist - using root
    user jasonc does not exist - using root
    group jasonc does not exist - using root
    user jasonc does not exist - using root
    group jasonc does not exist - using root
    Bad exit status from /var/tmp/rpm-tmp.6041 (%prep)
  • That previous command extracted a RPM SPEC file in the /usr/src/redhat/SPECS/ directory.
  • Edit /usr/src/redhat/SPECS/java-1.6.0-sun.spec. Find the part that says %define buildver and change the value to the build for the new version of Java
  • Run rpmbuild -ba /usr/src/redhat/SPECS/java-1.6.0-sun.spec. This extracts the JDK installer you previously downloaded and builds a set of RPMs from it.
  • cd /usr/src/redhat/RPMS/i586; ls;

java-1.6.0-sun-1.6.0.11-1jpp.i586.rpm        java-1.6.0-sun-fonts-1.6.0.11-1jpp.i586.rpm
java-1.6.0-sun-alsa-1.6.0.11-1jpp.i586.rpm   java-1.6.0-sun-jdbc-1.6.0.11-1jpp.i586.rpm
java-1.6.0-sun-demo-1.6.0.11-1jpp.i586.rpm   java-1.6.0-sun-plugin-1.6.0.11-1jpp.i586.rpm
java-1.6.0-sun-devel-1.6.0.11-1jpp.i586.rpm  java-1.6.0-sun-src-1.6.0.11-1jpp.i586.rpm
  • You can now install the RPM: rpm -i java-1.6.0-sun-1.6.0.11-1jpp.i586.rpm
  • For me that failed with a missing X dependency: libXtst.so.6 is needed by java-1.6.0-sun-1.6.0.11-1jpp.i586
  • I fixed that with yum -y install libX11-devel libXtst.
  • Use the alternatives command to set the correct version of Java: alternatives –config java
  • Finally: java -version:

java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)
That’s it – you finally have Java working on Linux! You also have a RPM which can be installed on other machines.

Comments (6)

Sticky Sessions which aren’t

Exhibt A (the Apache 2.2 mod_proxy documentation): 

stickysession - Balancer sticky session name. The value is usually set to something like JSESSIONID or PHPSESSIONID, and it depends on the backend application server that support sessions. If the backend application server uses different name for cookies and url encoded id (like servlet containers) use | to to separate them. The first part is for the cookie the second for the path.

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

 

Exhibit B (the Apache 2.2 mod_proxy code):

    /*
     * If we found a value for sticksession, find the first '.' within.
     * Everything after '.' (if present) is our route.
     */
    if ((*route) && ((*route = strchr(*route, '.')) != NULL ))
        (*route)++;
    if ((*route) && (**route)) {
        ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
                                  "proxy: BALANCER: Found route %s", *route);
        /* We have a route in path or in cookie
         * Find the worker that has this route defined.
         */
        worker = find_route_worker(balancer, *route, r);
        if (worker && strcmp(*route, worker->s->route)) {
            /*
             * Notice that the route of the worker chosen is different from
             * the route supplied by the client.
             */
            apr_table_setn(r->subprocess_env, "BALANCER_ROUTE_CHANGED", "1");
            ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
                         "proxy: BALANCER: Route changed from %s to %s",
                         *route, worker->s->route);
        }
        return worker;
    }

http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_balancer.c?revision=687754&view=markup

Translation: In Apache mod_proxy_balancer, sitcky sessions need to be in the format <cookie value>.<route> where route refers to the backend server. 

 

There are a couple of useful docs about this: Mark Round and the Jetty Docs. Normally I’d submit a patch to the Apache docs to clear this up, but unfortunately I haven’t been able to get failover between servers working correctly, so I’m not convinved I completely understand all the issues yet.

However, once you have got this working it turns out to be a case of RTFM.

Exhibit C:

route - Route of the worker when used inside load balancer. The route is a value appended to session id.

 

http://httpd.apache.org/docs/2.2/mod/mod_proxy.html

I’d argue that only makes any sense after you’ve got it working…

Comments

Modify java.library.path at runtime

Linking to native code in Java is always a hassle. JNI isn’t exactly nice, and there are some oddities around classloaders and native libraries which are annoying if you run into them.

One thing I wasn’t aware of was exactly how hard it is to load a library it isn’t already in the directories specified by the java.library.path system property. 

Initially, I thought I’d just be able to alter that property and the JVM would pick up the new locations. That turns out not to be the case, as is shown by this (closed) bug report.

However, there is a solution, outlined in this post on the Sun forums, which revolves around altering the usr_paths field stored in java classes.

	public static void addDir(String s) throws IOException {
		try {
			// This enables the java.library.path to be modified at runtime
			// From a Sun engineer at http://forums.sun.com/thread.jspa?threadID=707176
			//
			Field field = ClassLoader.class.getDeclaredField("usr_paths");
			field.setAccessible(true);
			String[] paths = (String[])field.get(null);
			for (int i = 0; i < paths.length; i++) {
				if (s.equals(paths[i])) {
					return;
				}
			}
			String[] tmp = new String[paths.length+1];
			System.arraycopy(paths,0,tmp,0,paths.length);
			tmp[paths.length] = s;
			field.set(null,tmp);
			System.setProperty("java.library.path", System.getProperty("java.library.path") + File.pathSeparator + s);
		} catch (IllegalAccessException e) {
			throw new IOException("Failed to get permissions to set library path");
		} catch (NoSuchFieldException e) {
			throw new IOException("Failed to get field handle to set library path");
		}
	}

Obviously, I don’t think that’s portable across JVMs, though.

Comments (2)

Webdesign

For the last I’ve been doing webdesign (yeah, that actual visual UI stuff, not just AJAX or something) at work, and – remakably – for the first time since 1997 (yes – 1997!) I’ve enjoyed it.

Generally speaking my design tastes are different – or perhaps I could better say they reflect my unique sense of humour. For example, the orginal – and best – design for nicklothian.com featured a colour scheme generated from converting universal constants (the speed of light, e, etc etc to hex values). It was unique, and is still yet to be duplicated (!!).

But doing serious webdesign led me to dig out an old, old review for the first website I ever built and maintained. This, was when the web was young, CSS didn’t really work, Netscape 4 (!) was my browser of choice and I think I was running a pre-Slackware 1.0 Linux install, which I’d downloaded onto 12 floppies.

Website review, 1997 (actually, the article is from the May 1998 edition of Adelaide Review but I did the site in ’97)

 

Website review

Website review

Comments

Nick’s Two Laws of Software Engineering

So I’ve been working in software for over 10 years, and so I figured its about time I came out with some highly scientific principles of software engineering. Follow these, and I personally guarantee success…

  1. Don’t try and do too much at once.
  2. Hire smart people.
It’s important that you do both of these – leave either out and you’ll probably fail. 

Firstly, software is too hard to try and get too much of it done at once. Don’t bother trying – no matter how smart your team is you’ll fail – or have to redefine success to something less than 100% satisfactory. If you’ve got a big project to do, then incrememental delivery is the only way forward.

Secondly, no matter how small the project is you need smart people. Software is too hard, and it has an amplifying effect on stupidity. Since the smallest bug can cause big problems, and the number of bugs decreases expoentially with smartness you just can’t afford to have less than the smartest people available (I’ve got no evidence this bug vs smartness relationship is true, but it sounds kinda scientific & shit, so I’ll leave it in). However, if you have smart people, you have to stop them trying to do too much at once. See rule one.

That’s it. I now declare software development to be a solved field.

Comments