The 1.5 million “words”
I claimed JavaBlogs has needs to be clarified
slightly. They are “words” as defined by running String.split(“\\W”)
on all the posts archived. The [\W] regular expression is defined as a “A
non-word character” – any character that is not in “a-zA-Z_0-9”. For normal
english sentences from a book that is probably a reasonable definition –
however when used on blogs where there is a large number of urls it doesn't
quite work. For instance, we suddenly find that “http” is one of the most
popular “words” in the english language. That's because all urls are split
on their non-word characters – so http://www.javablogs.com is split into
“http”, “www”, “javablogs” & “com”. Also, dates like 2-May-2003 or
25/12/2002 are split on the “-” and “/” characters, so “2002” and “2003” are
very common words.

My current thoughts are to try splitting on “\s” – ie whitespace.