Webcam and Annotation With HTML5

Nowadays, Webcam is widely used for both personal and business. It is simple, cheap, and yet powerful for image acquisition.

Some developers prefer to create their own webcam libraries. In this case, you need to learn technologies like TWAIN,WIA and/or DirectShow.

Note: the WIA and DirectShow APIs are recommended to develop a webcam SDK. TWAIN, however, is a good option for scanners.

However, due to the complexity of learning the APIs, as well as the technologies such as COM interfaces, it’s not an easy task for developers to create and embed Webcam SDK into their web applications. There are two options to simplify the whole process:

• Use a third-party plug-in/add-on to access and control the webcams, such as Flash, Silverlight, etc.
• Adopt the new technology – HTML5.

I will talk more about the second option in this article.

Webcam devices will be supported by the new video element and the getUserMedia API in HTML5. This means we can capture video streams and images from Webcam devices with just a few lines of JavaScript and HTML code, and the end users do not need to install anything for the browsers.

Information you need to know:

• HTML5 is not released yet. So this article is for early experimentations and it is not recommended to use the code snippet in the article for actual implementations.
• Currently only the following browsers support getUserMedia and WebGL:
  • Google Chrome (Version 21 and above) – Recommended for testing.
  • Firefox (Nightly Build, version 17 and above)
  • Opera (Version 12 and above, need to enable WebGL in opera:config)

Access the Webcam

Thanks to Wes Bos. He posted a wonderful article on how to access the Webcam. If you are new to HTML5, I suggest you read this article first (you can skip the content for ccv and scripts.js).

Basically, there are two steps:

1. Getting the Webcam stream

We use the getUserMedia API to get the stream.

 Collapse | Copy Code

navigator.getUserMedia_ = navigator.getUserMedia || navigator.webkitGetUserMedia || 
  navigator.mozGetUserMedia || navigator.msGetUserMedia;
//Get the API according to the browser.
navigator.getUserMedia_({ video: true, audio: false }, success, error);

Unfortunately, different browsers have different names for the getUserMedia API. So we have to check and pick the correct one in the first step.

When we call the API, video:true indicates that we want video access and audio:false indicates that we don’t want audio. Error function will be called when the stream is not accessible.

Success is the function to receive the stream:

 Collapse | Copy Code

function success(stream) {
	var domURL = window.URL || window.webkitURL;
	//start streaming via the video element
	document.getElementById(‘myVideo’).src =
	domURL ? domURL.createObjectURL(stream) : stream;
}

When you access the page, the browser will ask for the permission to use your webcam. If you click Deny, the web application cannot access the Webcam and it will trigger the error function.

2. Output to canvas

We have the video stream for the Webcam and now we can output the images to canvas element.

 Collapse | Copy Code

var video = document.getElementById(‘myVideo’);
var canvas = document.getElementById(‘myCanvas’);
var ctx = canvas.getContext(’2d’);
ctx.drawImage(video, 0, 0, canvas.width, canvas.height);

In the above code, we only capture an image from the video and draw on the canvas. So the question will be: how can we output the video? Well, you can set an interval and draw the image every few milliseconds, or, you can use therequestAnimationFrame API to perform an animation from the browser level. (You will find the sample in the attached source code).

Annotation

Annotation is always important for image processing. We can use it to mark or highlight some important information, add some notes, hide certain area, etc… If you have checked the face detection demo from Wes Bos, you will find it also useful for entertainment.

Since we have the canvas object, we can actually use the context of the canvas object to draw lines, arc, text or image on the canvas. It is pretty much just a simple annotation design (we won’t discuss the advanced annotation functions in this article).

You can find all the information from canvas and canvas 2D context.

For example, let’s draw a text on the video:

 Collapse | Copy Code

var canvas = document.getElementById(‘myCanvas’);
var ctx = canvas.getContext(’2d’);
ctx.font = “10px sans-serif”;
ctx.strokeText(“Hello World”, 0, 0); //output the text to (0,0)

Note: this code should be added into the repaint loop. Otherwise the text will be lost in the next repaint operation.

You can find a complete annotation sample in the source code .

Capture and save image from Webcam

Canvas object offers a method to save the current display to an image element. This allows us to capture and save both the image and the annotations.

 Collapse | Copy Code

var img = canvas.toDataURL(“image/png”);
window.open(img, “_blank”);

You can open a new window to display and save the captured image, or you can use the img element to hold the image.

A complete sample for Webcam and annotation with HTML5

Please test the web page as a web application (place the page in IIS/Apache/Tomcat/etc… and access the page via HTTP address). If you just double click the HTML file and run it from local file system, the browser will not be able to access the webcam resources.

Google Chrome Version 21 and above is recommended for the testing.

Download sample (source code included)

Conclusion

HTML5 provides a simple way for developers to create a webcam SDK. As you can see from the sample, we can easily enable our applications to interact with the webcams. However, this method also brings up some questions:

1. Compatibility with the browsers.
2. Security (since the app will access device directly)

Please let me know if you have the same concerns. Any comments are welcome.