Chapter 3

WebSockets

This chapter gives a detailed description on how to connect users and servers in real-time. In this course we will use WebSockets and cover a popular webSocket implementation for Node.js called Socket.io.

Push technology

On the web, we generally rely on short-lived request/response connections between clients and servers. When a client (e.g. web browser) sends a request to a server, the server will inspect this request and respond by sending data back. The data that your server sends can be anything from an html file to images or videos. Traditionaly, once the server has responded it will close the connection and move on.

This communication model works fine if you only want to fetch some files over the Internet, but what if you want to create a website that automatically refreshes a portion of the page when there is new content to show? In the traditional model your would need to program the client to contact the server every so often to ask it if there is new content for you to load.

In Ajax there is a nice way around this.
Ajax applications like Gmail, also start of with a client sending a request to a server. However, if the server has no data to send back to the client it will keep the connection to the client open. If the server now detects that there is new content, it will send this content to the connected clients. The client will process the data and send a new request to the server. The server will once again hold the connection open until there is new data available.

This method of communication is called "long polling" and is an often-used trick to create applications that respond in real time. There is however a problem with long pooling, and that is the amount of overhead it generates. Clients need to constantly initiate a request to the server in order for it to be able to send back data when it becomes available. In short; this means that you constantly need to send a bunch of instructions back and forth to make sure the client and server stay connected. This is generally not a big problem until latency starts to matter, like in multiuser games for example.

What we need is a protocol that allows us to connect clients and servers in a way where both ends hold the connection open until one disconnects. This connection can then be used to send data back and forth, without any re-connection overhead - Enter WebSockets.

What are WebSockets?

WebSockets allow for real-time two-way communication between two endpoints on a network. The WebSocket API is part of the HTML5 specification and supported by most major web browsers.

Once your client and your server have established a socket connection (handshake), they can send data to each other in real-time.

WebSockets are perfect for many different types of applications:

For a more detailed introduction to WebSockets you should read the HTML5Rocks, introduction to webSockets, article.

How about older browsers?

WebSockets are still a fairly new addition to the HTML5 standard and not as broadly supported as other push technologies. You should therefore always provide a fallback mechanism like, long polling or flash sockets. This way your application still works in older web browsers.

There are a couple of different libraries available that allow you to write connected applications with WebSockets. Most of these libraries automatically support fallback protocols.

There are WebSocket libraries available for most back-end languages like; Php, Java, Python, Rubi, Perl… This chapter will show you how to create your own WebSocket server in “Node.js” and the “Socket.io” library.

Node.js

In this course we will do all the back-end (server) development in Node.js. With Node.js, you can write system level programs in JavaScript.

Just like Php, Java, Python you write programs that are interpreted by a runtime. The runtime used in Node.js is based on the Chrome V8 engine. V8 is known for being extremely fast. This means that certain Node.js programs are faster or as fast as Php, Java or Python programs.
The Node.js website supplies the following product description:

Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.

NPM

Although Node.js was originally developed for networked applications, it can be used for much more. The runtime is expandable through libraries (often called packages). Packages allow you easily add new modules to a project and expand the possibilities of the core system. There are libraries for everything from Server programming to serial port communication.

You can find an overview of the available packages at: https://npmjs.org/

One of the best parts of Node is its package manager called NPM (Node Package Manager). The NPM makes package installation and dependency management simple.

Since we are going to build our server in Node.js. Lets just go ahead and install it now.

Node installation

  1. Go to the Node.js website.
  2. Click on the big green “Install” button, on the top of the page to download the Node.js installer.
  3. Open the installation file called something like: “node-v0.10.36.pkg”.
  4. The installation wizard will guide you through all the steps needed to install both “Node.js” and “NPM”.
  5. If all goes well the installation should end with a message telling you that Node.js is now installed on your system.

Test the Node.js installation

  1. Open the Terminal.
    The terminal is part of your OSX installation and can be found under:
    Applications > Utilities > Terminal.app, or just open Spotlight and type "Terminal".
  2. In your Terminal, type “node -h” and press enter.
    if Node.js is installed successfully you should now see the node help file.

Socket programming

Before we write our first Node.js program, please watch the Node.js introduction video from codeSchool.

You should also read:
http://net.tutsplus.com/tutorials/javascript-ajax/node-js-for-beginners/

Note that Node.js only gives you the core elements needed to write a server. This means that writing a full-fledged web server in Node.js is not a simple task, and far outside of the scope of this course.
If you at a later time want to return to Node.js, I can advice you the following book:
www.nodebeginner.org

If you are really serious about building your own web server in Node, you should also take a look at the Express web framework:
http://expressjs.com/

For general reference use the Node.js documentation:
http://nodejs.org/community/

For the sake of timeliness, we are going to jump straight in and install and use a simple static web server that I have prepared for you.

Download the server code

Setting up the server

You may not know it, but this website is actually running on the server you just downloaded.
Lets start coding:

  1. Create a folder for this project. Name the folder “ledServer”.
  2. Copy the code you downloaded into the “ledServer” folder.
  3. Your ledServer folder should now contain the following two files:
    • www
    • webServer.js
    • main.js
  4. Open the file: “main.js” in a JavaScript editor.
  5. Add the following line to main.js
    
    console.log("hello world");
                                
  6. Lets run the code.
    Open the Terminal and navigate to the ledServer folder.
    If you are not so familiar with the Terminal you may write “cd ” (note the space behind cd) into the terminal window. Then drag and drop the “ledServer” folder from the Finder into the terminal window and press enter.
  7. write “ls” to verify that you are inside the right folder. The console should list the files contained in the ledServer folder.
  8. To run run your server you need to write “node main.js” and press enter.
  9. In the terminal window it should now say. “hello world”.
  10. Super, you now know how to run node programs.
  11. Go back to your JavaScript editor and remove the code in the "main.js" file.
  12. To run the web server. You need to do the following.
    1. Add the following code to the "main.js" file
      
      var ip   = '127.0.0.1';
      var port = 8080;
      var server = require('./webServer').start(ip, port);
                                          
    2. Save and run the code again “node main.js”.
    3. Your terminal should now say:
      “Server running at: http://127.0.0.1:8080/”
  13. Open your web browser and go to http://127.0.0.1:8080/.
  14. You should now see the default page for the server.

The index page can be found in the folder “www”. This is the root folder your new webpage. Hence, all files from HTML, CSS, JavaScript to media like Images should be located in the www folder.

To turn the server off again and return to the terminal, press "ctrl + c"

🎓 Exercise

Update the default index.html page, provided by the server. You should add some CSS and a picture or two.

Socket.io

In order for your Node.js server to handle WebSocket requests you need to add this capability. In this course we will use the popular "socket.io" library. Socket.io is well tested and documented. As with the server, i have prepared a simple script to get you started.

Download WebSocket code

Download the socketServer.js file and add it to the “ledServer” folder.

Installation

Since Socket.io is not part of the standard Node.js installation you need to add it to the project.
In the terminal:

  1. Make sure you are (still) in the “ledServer” folder. (ls)
  2. Write “npm install socket.io” to install the socket.io library with the Node Package Manager (NPM). NPM will automatically find all the details it needs to download and install socket.io.
  3. Verify that socket.io is installed.
    But where did NPM install this package? Good question!
    All node modules are installed in a folder called "node_modules". Since this folder was not available, NPM created it for you. Modules by default are installed into the project folder only. This way you don't clutter up a global library folder that other projects rely on as well. It is however possible to install modules at a global level as well. To do so, write "npm install –g socket.io".
  4. Add the following line of code to your "main.js" file.
    
    var io = require('./socketServer').start(server);
                                
  5. Run the server again. If everything is working fine. You should see a status message that says:
    "info - socket.io started".

Congratulations your server is ready to handle webSockets.

WebSockets, the Client and the Server

Lets dive into webSockets by creating a simple app where the client connects to the server and the server in response sends some data to the client.

The Client

On the client-side (the HTML file loaded by the browser) you need to do the following.

  1. Load the socket.io JavaScript library from the server.
    
    <script src="/socket.io/socket.io.js"></script>
                                
    The socket.io library provides you with a simple set of instructions you can call to communicate to the server. Socket.io sends these instruction to the server using the best available protocol.
  2. Connect to the server.
    
    <script src="/socket.io/socket.io.js"></script>
    
    <script>
        
    	function init() {
    	
    	    // connect to the socket server
    	    var socket = io('http://127.0.0.1:8080');
            
    	}
    	window.addEventListener('DOMContentLoaded', init);
    	
    </script>
     
    
  3. Add an "connect" and "msg" event listener.
    
    <script src="/socket.io/socket.io.js"></script>
    
    <script>
            
        function init() {
            var socket = io('http://localhost:8080');
            
            // 
            socket.on('connect', function () {
                        console.log('The server is ready!');
            });
            
            // on msg
            socket.on('msg', function (data) {
                
                // log the message
                console.log(data);
                
                 // send message
                socket.emit('msg', 'hello server');
            });
        }
        window.addEventListener('DOMContentLoaded', init);
    	
    </script>
     
    

The Server

  1. Accept the connection.
                                
    "use strict";
    
    // start the webServer
    var ip   = '127.0.0.1';
    var port = 8080;
    var server = require('./webServer').start(ip, port);
    
    
    // start the socketServer
    var io = require('./socketServer').start(server);
    
    // client connection handler
    io.on('connection', function (socket) {
        
        console.log('client connected: ' + socket.id);
        
    });
                                
  2. Respond with some message.
                                
    "use strict";
    
    // start the webServer
    var ip   = '127.0.0.1';
    var port = 8080;
    var server = require('./webServer').start(ip, port);
    
    
    // start the socketServer
    var io = require('./socketServer').start(server);
    
    // client connection handler
    io.on('connection', function (socket) {
        
        console.log('client connected: ' + socket.id);
        
        // send message to the current client
        socket.emit('msg', 'hello new user');
        
    });
                                
  3. Detect disconnections.
                                
    "use strict";
    
    // start the webServer
    var ip   = '127.0.0.1';
    var port = 8080;
    var server = require('./webServer').start(ip, port);
    
    
    // start the socketServer
    var io = require('./socketServer').start(server);
    
    // client connection handler
    io.on('connection', function (socket) {
        
        console.log('client connected: ' + socket.id);
        
        // send message to the current client
        socket.emit('msg', 'hello new user');
        
        // client disconnect handler
        socket.on('disconnect', function () {
    
            console.log('client disconnected: ' + socket.id);
        });
    });
                                

Socket.io recipes

For detailed information, go to the socket.io "How to use" section or look at the socket.io's GitHub page..

🎓 Exercise

Create a simple application that allows users to post messages to the site, using the socket.io API.

Conclusion

Okay, lets take a minute and look how far we have come. So far we have talked about microcontrollers and looked into the Arduino. Additionally we have created a simple client-server app that uses the power of webSockets to communicate in near real-time. We now have almost all the pieces in place to create the light bulb server from chapter 1.

There is only one last bit missing now. The WebSocket Server and the Arduino need to learn talk to each other.
We will be looking into this in the next chapter.