fredagen den 10:e februari 2012

WebRTC getUserMedia and WebSockets (XSockets)

 

Just recently I posted a video showing how you can access local devices that can generate a multimedia stream . In my example I access the Webcam just using the WebRTC JavaScript API – getUserMedia.  It’s an early experiment that uses a nightly build of Opera ( 12.00 lab) which is the only (correct me if I’m wrong ) browser giving us the chance to experiment with the API today?

Short video showing how we can use WebRTC & WebSockets and more.

I will very short show you how you can try it out your self.

1. Install Opera Nightly (Lab)

First of all you need to install Opera Nightly on your system, I found the installer here.

http://dev.opera.com/articles/view/64-bit-opera-and-out-of-process-plug-ins/

2. Enable WebSocket and Camera to Canvas access in Opera

As Opera today has WebSockets disable we need to enable this,  type opera:config , Type ‘WebSockets’ in the search box , then you need to Allow the browser to copy “Camara to Canvas”   by typing “Allow camera to Canvas Copy” I the search box and check.  You will need to restart the browser to apply the changes.

3. Install a XSockets.NET WebSocket Server

The most convenient way of setting up a XSockets.NET WebSocket Server is to create an empty ASP.NET MVC3 Project I Visual Studio, open the Package Manager Console (View – > Other Windows – Package Manager Console )

The you type Install-Package XSockets , for more information have a look at this Video XSockets.NET 1.0.X RC1 – Getting started - http://xsockets.net/videos 

 

4. Create the “broadcast” page

The code snippet below shows you how you can use the getUserMedia, WebSockets (XSockets.NET) to send video stream, unfortunately we are not possible to send the “stream” as binary message. (XSockets.NET Supports this) so we have to Base64 encode/decode the image/stream.

JavaScript code

       var videoStream = null;
      
var canvas = null;
      
var ctx = null;
      
var ws = null;
      
var timer = 0;
      
$(function () {

          
// You need to active the following stuff in Opera
           // 1. Allow Camera To Canvas Copy
           // 2. WebSockets
           ws = new WebSocket("ws://127.0.0.1:4502/GenericText");
          
ws.onopen = function () {
              
$("#state").text("Connected to XSockets.NET Server!");
          
};
          
canvas = $("canvas");

          
ctx = canvas[0].getContext('2d');
          
ws.onerror = function () {
              
// do op
           }
          
// Assign the <video> element to a variable
           videoStream = document.getElementById('videoSource');
          
// Replace the source of the video element with the stream from the camera
           if (navigator.getUserMedia) {
              
navigator.getUserMedia('video', successCallback, errorCallback);

              
function successCallback(srm) {
                  
videoStream.src = srm;
              
}

              
function errorCallback(error) {

              
}
          
}
          
$("#start").bind("click", function () {
              
timer = setInterval(

              
function () {
                  
send(videoStream, ctx, 320, 240);
              
}, 100);
          
});

          
$("#stop").bind("click", function () {
              
clearInterval(timer);
          
});
      
});

      
function send(stream, ctx, w, h) {
          
ctx.drawImage(stream, 0, 0, w, h);
          
var el = $("canvas").get(0);
          
var d = el.toDataURL();
          
var msg = {
              
event: "Onstream",
              
data: {
                  
url: d
              
}
          
};
          
ws.send(JSON.stringify(msg));
      
}

Markup

<button id="start">Start</button>
  
<button id="stop">Stop</button>
   
<div>
   
<h1>Video element</h1>
     
<video id="videoSource" autoplay controls style="height:240px;width:320px"></video>
   
</div>  
   
<h1>Canvas element</h1>
   
<canvas width="320" id="canvas" height="240" style="border:1px solid #ff0"></canvas>
   
<div id="state">    
   
</div>  

We also need to include some external JavaScript , put this references into the header

<script src="/Scripts/jquery-1.7.1.min.js" type="text/javascript"></script>
<script src="/Scripts/JXSockets-1.0.beta.js" type="text/javascript"></script>

The jXSockets library will be added by the Nuget Package (see 3 above)

5. Create the “listener” page

This page will subscribe to the “media stream” that the broadcast page will send.   It consists of a few lines of JavaScript and markup , you will find the complete “source” here.

<!DOCTYPE html>
<html>
<head>
   
<title></title>
   
<script src="Scripts/jquery-1.7.1.min.js" type="text/javascript"></script>
   
<script src="Scripts/JXSockets-1.0.beta.js" type="text/javascript"></script>
   
<script type="text/javascript">
       
$(function () {
           
var binaryWs = new jXSockets.xWebSocket("ws://127.0.0.1:4502/GenericText");
           
binaryWs.bind("Onstream", function (blob) {
               
$("img").attr("src", blob.url);
           
});
       
});
   
</script>
</head>
<body>
   
<div id="images">
       
<img width="320" height="240" />
    </
div>
</body>
</html>

 

That is pretty much the thing’s you need to try this out.

Links

Kind regards
Magnus Thor

onsdagen den 14:e december 2011

Send and receive BLOBS using FileAPi and WebSockets (XSockets.NET)


This is  short blog post showing you all some of the work we have done on the XSockets.NET WebSocketServer.  The latest two month in the XSockets team has been working on improving the API’s, plugin ,configuration and protocols.  We are pleased that we decided to focus on issues, development and things in general regarding the protocol support – As we did this we will manage to deliver protocol support based on the W3C Candidate Recommendation from the 8th of December.

Don’t miss our free WebSockets CodeCamp in end of January 2012 in Stockholm Sweden

In this post I will show you a short video and some early code-snippets of how we can user the binary support of WebSockets, send and receive files in out Web browser just using WebSockets.  I’m also taking advantage of the File API.  

The demo reminds of an similar thing o wrote in April this year, but back back then we had to Base64 encode/decode as the protocols and browser lacked support of sending binary web sockets messages (http://dathor.blogspot.com/2011/04/html5-websockets-sending-binary-data.html for the old post )

 

Video (http://www.youtube.com/watch?v=GCQglS8cGQM)

JavaScript code

// Create an WebSockets instance using XSockets jsAPI
binaryWs = new jXSockets.xWebSocket("ws://127.0.0.1:4502/XSockets.Core.Handlers.Binary", "XSockets.Core.Handlers.Binary");

// Bind an "listner" for the onopen event
binaryWs.bind("open", function () {
   
$("<p>").text("connected to XSockets XSockets.Core.Handlers.Binary").insertAfter("div");
   
$("input").change(function (e) {
       
$(this.files).each(function () {
           
$("<p>").text("Sending Blob '" + this.fileName + "' size is " + this.size).insertAfter("div");
           
binaryWs.send(this); // send the file (blob)
        });
   
});
   
// Bind a "listner" for the "blob event 
    binaryWs.bind("blob", function (blob) {
       
var reader = new FileReader(); // Create a FileReader using the FileAPI
                   
       
// When the "blob" i loaded, add the "image to DOM (page)
        reader.onload = function (e) {                        
           
var data = e.target.result;
           
uri = data.replace("data:base64,", "data:image/png;base64,")
           
$("<img>").addClass("image").attr("src", uri).prependTo("div");
       
};                    
       
reader.readAsDataURL(blob); // Read the "blob" recived from the "WebSocket"
    })
});
});

Markup (HTML)

    <form>
   
<h1>
        XSockets.NET - FileAPI + Send Blobs using WebSockets
</h1>
   
<p>
        Get XSockets here http://xsockets.net
</p>
   
<input type="file" multiple />
    </
form>
   
<div>
   
</div>

XSockets.NET WebSocket Hander

using System;
using System.ComponentModel.Composition;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;
using XSockets.Common.XSocket;
using XSockets.Common.XSocket.Event.Arguments;
using XSockets.Common.XSocket.Interface;

namespace XSockets.Core.Handlers
{
   
[Export(typeof (IXBaseSocket))]
   
public class Binary : XBaseSocket
   
{
       
public Binary()
       
{
           
OnIncomming += Binary_OnIncomming;
       
}

       
private void Binary_OnIncomming(object sender, OnIncommingArgs e)
       
{

           
byte[] binaryStream = e.Bytes;
           
var ms = new MemoryStream(binaryStream);
           
Image image = Image.FromStream(ms);
           
image.Save(string.Format(@"c:\tmp\{0}.png", Guid.NewGuid()), ImageFormat.Png);
            
           
// Send to all connected users.
            foreach (var node in XNodes)
           
{
               
node.Value.XSocket.Send(binaryStream);
           
}

           
       
}
   
}
}

For more information about XSockets see http://xsockets.net

Links

  1. FileAPI - http://www.w3.org/TR/FileAPI/
  2. WebSockets API - http://www.w3.org/TR/websockets/
  3. XSockets.NET – http://xsockets.net
  4. XSockets Free CodeCamp in Stockholm Jan 2012 – http://xsocketscodecamp2.eventbrite.com 
  5. Video above at Youtube - http://www.youtube.com/watch?v=GCQglS8cGQM

Kind regards

Magnus Thor

måndagen den 12:e december 2011

KnockoutAuction a KnockoutJS,jQueryMobile + WebSockets example

 

I must confess, it has been a long time since I wrote something here at the blog, I have been busy doing HTML5, jQueryMobile teaching. Of course I also  got the time to work on the next release of XSockets.NET – A new release is coming during xmas!

So, in this post I will give I you an idea of how we cam build real-time web application using two of my favorite frameworks jQueryMobile and KnockoutJS.  The real-time parts is built on the XSockets.NET WebSocketServer (framework)

The example is an very simple “Auction” containing a few different views , it’s a single-page-web-app . There is no backend at all except the WebSocket Server, that enables the real-time parts of the auction.

In a few days there will be an enhancement of this simple demo, a complete backend will be implemented, until that I hope you will enjoy this sample.

 

We must keep in mind that I just want to show an concept, not a complete system,  but for those of you that wants more – its coming :-)

Ok lets have a look at the markup

JavaScript and other “dependencies "

    <title>KnockoutAuction</title>
   
<link rel="stylesheet" href="http://code.jquery.com/mobile/1.0/jquery.mobile-1.0.min.css" />
    <
script src="http://code.jquery.com/jquery-1.6.4.min.js" type="text/javascript"></script>
   
<script src="http://code.jquery.com/mobile/1.0/jquery.mobile-1.0.min.js" type="text/javascript"></script>    
   
<script src="http://xsockets.net/js/knockout-1.3.0beta.js" type="text/javascript"></script>    
   
<script src="http://xsockets.net/js/JXSockets-0.9.4.js" type="text/javascript"></script>

The auction contains the following views ( JQueryMobile )

Home (page)

The pages contains a couple of menu elements, I will just cover latest, create and details (where you place bids)

<div data-role="page">
   
<div data-role="header" data-position="fixed">        
       
<h1>
            KnockoutAuction
</h1>
   
</div>
   
<div data-role="content">
       
<p style="text-align: center">
            Welcome to this XSockets.NET show case using jQueryMobile and KnockoutJS - two utterly
            good frameworks
       
</p>
       
<ul data-role="listview" data-inset="true">
           
<li data-role="list-divider">Menu</li>
           
<li><a href="#latest">
               
<h2>
                    Latest auctions
</h2>
               
<p>
                    Show the 15 latest crazy auctions
</p>
           
</a></li>
           
<li><a href="#search">
               
<h2>
                    Find
</h2>
               
<p>
                   Find things you never thought existed...
</p>
           
</a></li>
           
<li><a href="#create">
               
<h2>
                    Create a new auction
</h2>
               
<p>
                   Sell ??your surplus stuff...
</p>
           
</a></li>
       
</ul>
   
</div>
   
<div data-role="footer" data-position="fixed">
       
<h2>
           Team XSockets.NET 2011
       
</h2>
   
</div>
</div>

Latest

This view contains a list of auctions , by using the data-bind=”” attribute (KnockoutJS) we bind the “listview/template” to the" model using foreach: auctions" ,  then by adding other data-bind attributes the the child element we gain control of the markup and rendering.  I also add a custom attribute auctionid, had some problems adding a “proper” html5 data attribute using the knockout templates.   The auctionid attribute will be used to “track” the element later.

 

<div data-role="page" id="latest">
   
<div data-role="header" data-position="fixed">
      
<a href="//" data-icon="back">Home</a>

       
<h1>
            Latest auctions
</h1>
   
</div>
   
<div data-role="content">
       
<ul data-bind="foreach: auctions" id="listofauctions" data-role="listview" data-inset="true">
       
<li>
       
<a data-bind="click:function() { auctionModel.show($data) }">
       
<h2 data-bind="text: Caption"></h2>
       
<p data-bind="attr: { auctionid: Id }"><strong>Latest bid:<span data-bind="text: Price" class="bid"></span></strong></p>
       
<p data-bind="text: Description">        
       
</p>
       
</a>
       
   
</li>
       
</ul>
   
</div>
   
<div data-role="footer" data-position="fixed">
       
<h2>
           Team XSockets.NET 2011
       
</h2>
   
</div>
</div>

Create a new auction

 

<div data-role="page" id="create">
   
<div data-role="header" data-position="fixed">
       
<a href="/" data-icon="back">Home</a>
       
<h1>
            Create
</h1>
       
<a href="#" data-icon="check" data-bind="click: newAuction">Save</a>
   
</div>
   
<div data-role="content">
       
<div data-role="fieldcontain">
           
<label for="caption">
                Caption:
</label>
           
<input type="text" id="caption" name="caption" data-bind="value: auction.Caption, valueUpdate: 'afterkeydown'" />
        </
div>
       
<div data-role="fieldcontain">
           
<label for="description">
                Description:
</label>
           
<input type="text" id="description" name="description" data-bind="value: auction.Description, valueUpdate: 'afterkeydown'" />
        </
div>
       
<div data-role="fieldcontain">
           
<label for="price">
                Price:
</label>
           
<input type="number" id="price" name="price" data-bind="value: auction.Price, alueUpdate: 'afterkeydown'" />
        </
div>
   
</div>
   
<div data-role="footer" data-position="fixed">
       
<h2>
           Team XSockets.NET 2011
       
</h2>
   
</div>
</div>

 

Placebid ( Details)

Lets the user place a bid.  The view contains of three elements bound to the “model”, the <p> tag , (<span )> that is the actual price and the input element.

<div data-role="page" id="details">
   
<div data-role="header" data-position="fixed">
   
<a href="#latest" data-icon="back">Back</a>
       
<h1 data-bind="text: bid.Caption">
           
</h1>
       
<a href="#" data-icon="check" data-bind="click: placeBid">Place bid</a>
   
</div>
   
<div data-role="content">
       
<h2 data-bind="text: bid.Caption"></h2>
       
<p data-bind="text: bid.Description">
        
       
</p>

       
<p data-bind="attr: { auctionid: bid.Id }"><strong>Current bid:<span data-bind="text: bid.Price" class="bid"></span></strong></p>

       
<div data-role="fielcontain">
           
<label for="bid">
                Your bid:
</label>
           
<input type="number" data-bind="value: bid.Price" />
        </
div>
   
</div>
   
<div data-role="footer" data-position="fixed">
       
<h2>
           Team XSockets.NET 2011
       
</h2>
   
</div>
</div>

JavaScript

Below is the applications JavaScript code,  you will find a quick explanation just below the code snippets

 var auctionModel = {
           
bid: {
               
Id: ko.observable(),
               
Caption: ko.observable(),
               
Description: ko.observable(),
               
Price: ko.observable()
           
},
           
auction: {
               
Id: ko.observable(),
               
Caption: ko.observable(),
               
Description: ko.observable(),
               
Price: ko.observable()
           
},
           
auctions: ko.observableArray([]),

           
newAuction: function () {
               
this.auction.Id = guidGenerator();

               
// Broadcast the new auction to listners and server for storage..
                ws.trigger("AuctionCreated", ko.toJS(this.auction), function () {
                   
$.mobile.changePage("#confirmcreate");
               
});

           
},
           
add: function (auction) {
               
this.auctions.push(auction);
           
},
           
find: function (query), { // not yet implemented },
            
           
placeBid: function () {
               
ws.trigger("UpdatePrice", ko.toJS(this.bid), function () {
                   
$.mobile.changePage("#latest");
               
});
           
},
           
show: function (auction) {

               
this.bid.Id(auction.Id)
               
this.bid.Caption(auction.Caption)
               
this.bid.Description(auction.Description);
               
this.bid.Price(auction.Price)
               
$.mobile.changePage("#details");
           
}
       
};

The JavaScript Code above represents the model of the applications, it contains object, arrays and methods.  The model consists of an KnockoutJS observable array that detects and responds to changes on collection of objects, in this case auctions.  The model also contains some observable objects (bid, auction) , the model is activated by passing it to applyBindings method of KnockoutJS; applyBindings(auctionModel);

There are some methods that needs to be explained a bit,  newAuciton triggers using jXSockets API (sends) a message (JSON) event named “AuctionCreated” to the WebSocket ws .  worth mentioning is that we need to serialize the auction object to be sent as JSON using the ko.toJS metod.  In the callback of the WebSocket trigger we redirects the user to the “ConfirmCreate” view of out application.

When we are dealing with our bids, w pretty much does the same thing, have a look at he placeBid method below.

The show(auction) method “maps” the event argument of the actual auction to be shown , and changes the jQueryMobile page to details. Not that the auction parameter of method is mapped to the bid object of out viewModel, giving us the possibility to keep track of changes and send those to the view and elements linked to each property of the bid object as shown in this codesnippet (markup)

        <h2 data-bind="text: bid.Caption"></h2>
       
<p data-bind="text: bid.Description"></p>

The text element of the <h2> and <p> tag is bound to the auctionModels bid object , when we from our array of object generates the markup using KnockoutJS astute template engine giving us the opportunity to bind our observable observable array to a listview widget (control) of jQueryMobile as shown here;

<ul data-bind="foreach: auctions" id="listofauctions" data-role="listview" data-inset="true">
   
<li><a data-bind="click:function() { auctionModel.show($data) }">
       
<h2 data-bind="text: Caption">
       
</h2>
       
<p data-bind="attr: { auctionid: Id }">
           
<strong>Latest bid:<span data-bind="text: Price" class="bid"></span></strong></p>
       
<p data-bind="text: Description">
       
</p>
   
</a></li>
</ul>

If we just have a short look at the attributes of the <ul> I suppose that you get point,  we got an data-bind attribute saying “foreach:auctions”  , this tells Knockout that this is the actual object/array to use for rendering ,  as you can see in the child elements of the <ul> we got some other data-bind attributes,  setting text and attributes of the elements.  The <a> tag has an click event defind, telling that when a user click’s we pass the complete auction object to out auctionModel.show(auction)

Inside the detail (placebid) markup above you will find a <a> tag with a data-bind that looks like this;

 

         <a href="#" data-icon="check" data-bind="click: placeBid">Place bid</a>

When the user hits the “Placebid” button the placeBid method of our viewModel is invoked and the following code is executed

ws.trigger("UpdatePrice", ko.toJS(this.bid), function () {
   
$.mobile.changePage("#latest");
});

We just triggers an and WebSockets event named UpdatePrice and passes the bid object, and within the callback there is a simple redirect.

So what's my point then, this may not be so interesting ?   As you may noticed we doesn’t deal with any “incoming” messages yet, nothing is changed in the model at all.

Lets have a look at how we wire up the application to listen to those messages we trigger .

Establish connection to the WebSocket and WebSocket handler

   ws = new jXSockets.xWebSocket("ws://127.0.0.1:4502/XSockets.Core.Handlers.Generic", jXSockets.WEBSOCKET);

The code above creates an new connection to our WebSocket Server and Handler, in this case we just use a generic handler ( Receives and broadcasts all incoming messages to all connected clients)

Is the KnockoutAuction open or close?

By subscribing to the “native” events of the WebSocket (open/close) , inside the callback of open you can se that we triggers an message called “LoadAuctions” , and just passes an empty JSON object, what we do here I simulating a call to a server that will respond with the auctions stored in any kind of database

ws.bind("open", function () {
   
ws.trigger("LoadAuctions", {});
});
ws.bind("close", function () { });

as there is no backend in this example , I just create some when receiving the “LoadAuctions” message.   By binding an eventhandler/subscription for the “LoadAuctions” message as follows


           
ws.one("LoadAuctions", function (auctions) {
               
// Add some sample auctions
                auctionModel.add(new auction(1, "XBox", "This is a test", 1200));
               
auctionModel.add(new auction(2, "Playstation 2", "This is a test", 500));
               
auctionModel.add(new auction(3, "Volvo v40", "This is a test", 4000));
               
auctionModel.add(new auction(4, "Summer house", "This is a test", 10000));
           
});

When we receives a messages named “LoadAuctions”  I add a few static auctions to the auctionModel by calling the add method of the model.  Also note that we are using the .one() method od jXSockets, Why?  we just need to listen to “LoadAuctions” once, as we will receive the rest in real-time, as they are created, we don’t need to poll for new auctions all the time…

Listen to new auction

 ws.bind("AuctionCreated", function (auction) {
               
auctionModel.add(auction);
               
$("#listofauctions").listview("refresh");
           
});

By just creating a event handler/subscription to the AuctionCreated messages ( triggered from the models newAuctions) we are able to just add it the the models (auctions) , as we use jQueryMobile we also need to refresh the listview widget.

Listen to bids

When a user hits the “Placebid” button ( as shown above) as UpdatePrice message is triggered , the following event handler/subscription will the be invoked.

  ws.bind("UpdatePrice", function (auction) {
               
$("[auctionid='" + auction.Id + "'] span").text(auction.Price);
           
});

As you see there is an jQuery selecor tracing all the elements containing an attribute called auctionid , the inboud messages id (auction.Id) is appended to the selector then the text node of its child span I set the the “new Price”, this could of can be handled by KnockoutJS and observables as well.

This pretty much explain how the "KnockutAuction” works, and how we can take advantages of WebSockets using the XSockets.NET WebSocketServer.

The code can be studied, executed , edited and tested at jsbin - http://jsbin.com/ukulum/ 

Links

 

Magnus Thor