I’ve been pretty impressed with some of the stuff ARKit can do, as well as the Microsoft Surface demos.
I decided that we need to do that on the web, too, so I build a virtual keypad, using HTML5 video and a webcam.
In HTML5, a common (example 1, example 2) pattern is to take an image from a video, copy it to a canvas and manipulate it here using Javascript. That’s the general pattern I’ve used.
I use a colored marker (setup in a calibration phase) and track its location. In order to take account of brightness changes, etc, I do cosine similarity tests on average color vectors for each “key” on the keypad using the Sylvester Javascript vector library.
I think there’s a fair bit of potential for video based interfaces. I’d like to see a combination of this and the HTML5 drumkit, for example.
You can try it out here: http://demo.nicklothian.com/keypad/. You’ll need to be using Firefox 3.5+.
Here’s a demo video (ironically as an embedded flash video):
Does the camera need to be static for that to remain workable?
Nice work, though the embedded video seems too wide for the page and gets clipped behind the left nav (on FF3.5 anyway).
Would be even better if you’d chuck it up on Github so people can fork it :)
@Ed – yes, good idea.
@Anees – it shouldn’t – it relies on color matching rather than position. I can’t say I’ve tried it, though.
But, this is not in real time, is it? it’s a recorded video..
It’s a recording of it working realtime.There is a working, real time demo at http://demo.nicklothian.com/keypad/
That’s awesome! I started a project somewhat like that last year which I ported to JS/Canvas+. Instead of using color tracking, my system requires the camera to be mounted at an angle (which makes the surface appear “shinier”, it reflects more light, and I don’t know the math to prove it but it’s true). So I have the thing scan from right to left vertically until it sees something finger-like and then looks to the immediate right and calculates whether or not it looks like there’s a reflection, since the real z-axis position is half of the percieved distance between the real and virtual image.
It also implements a nice matrix perspective transform thing which you might find useful :)
http://antimatter15.com/wp/2009/08/shinytouchjs/
(My site may be down, I’m in the process of switching hosts so the DNS may not have fully propagated)