http://www.developer.com/lang/jscript/building-a-twitter-streaming-client-with-node.js.html

Back to article

Introducing Node.js: JavaScript for Massively Concurrent Apps


January 24, 2011

I fondly recall sitting around the dinner table as a young boy, listening to my father and grandfather reminisce about "the old days," a simpler time when men were men, cars were cars, and humanity apparently walked perpetually uphill. I wonder whether in another 30 years if I'll be berating my own grandchildren about the good old days of computing, a time when RAM meant something, darn it!

The early days of the World Wide Web were also a simpler time, an era when developer's were primarily concerned with building websites which served a mix of dynamically and statically generated pages. Install Apache, build a series of interfaces which interact with a database, and the job is done.

With the mainstream Web now a teenager, things have become a bit more complicated. These days the emphasis has moved away from creating websites and toward building Web applications. This goal of creating Web-based applications which closely resemble their desktop-based brethren and involving traffic on a massive, worldwide scale has left developers grappling with a relatively new problem domain involving how to appropriately deal with concurrency.

If you're not familiar with the technical challenges surrounding building massively concurrent applications, you might be surprised to know that JavaScript has emerged as one of the front-running solutions helping to solve these challenges. Yes, JavaScript. As it turns out, JavaScript's event-based programming model makes it an ideal solution for taming the concurrency tiger, offering a poll-based solution in lieu of a thread-based approach (such as that used by the Apache Web Server) to handling concurrent requests. If you're not familiar with the differences between these two approaches, check out this excellent blog post.

But isn't comparing JavaScript to the Apache Web server rather unfair, if not downright nonsensical? As it turns out, a server-side JavaScript solution exists. It's called the V8 JavaScript Engine. V8 is an implementation of JavaScript written in C++ which allows you to write standalone JavaScript applications. A V8 library called Node.js (although referred to as simply Node) is fast becoming the toast of the party among those tasked with creating highly scalable real-time applications built-atop the Web thanks to its event-based approach to handling concurrency. Companies such as GitHub and Palm are already incorporating Node.js into their products, and a whole suite of amazing new Node-based products and games (see WordSquared for a really impressive example) are emerging seemingly by the day. Check out the list of applications posted on Node's GitHub wiki for more ideas of what's possible.

 

Installing Node

The easiest way to install Node is by cloning the project's GitHub-hosted repository:

$ git clone git://github.com/ry/node.git

Once the clone process has completed, you'll need to configure and install Node using the following command sequence:

$ ./configure
...
$ make
...
$ make install

Once finished, confirm Node has been properly installed by executing the following command from your terminal:

$ node --version
v0.3.6-pre

Building a Node-based Web Server

Most developers get started using a new programming language by writing and executing a language-specific implementation of Hello World. Node's particular implementation happens to be a simple Web server. Save the following code to a file named web.js:

 http = require('http');

var server = http.createServer(function (request, response) {
 response.writeHead(200, {"Content-type": "text/html"});
 response.end("<p>Hello, world!</p>");
});

server.listen(8000);

Once saved, start the Web server using the following command:

$ node web.js

Now test out the Web server by opening your browser and navigating to the address http://127.0.0.1:8000. You should see Hello, world! output to the browser window.

Building a Twitter Streaming Client

Building a simple Web server is an useful exercise for familiarizing oneself with Node syntax, however it's not a particularly compelling example of Node's true capabilities. So let's build something much more interesting which involves querying Twitter using the Twitter Search API.

This example works by periodically connecting to the Twitter API and searching the latest tweets for a specific keyword (which in the case of this example is defined by the variable keyword).

Begin by loading the sys and http modules:

 var sys  = require('sys');
var http = require('http');

Next, define the server you'll be connecting to using the client, which in this case is Twitter's search API endpoint. You'll also define a variable which holds the keyword you'd like to search for, before creating the HTTP client connection using the http module's createClient() method:

var host = "search.twitter.com"

var keyword = "php";

var connection = http.createClient(80, host);

Next you'll define the function which will repeatedly execute in accordance with a specified event, which in the case of our example is the conclusion of a specified waiting period (timeout) of five seconds. This function will send the request to Twitter's API endpoint, subsequently receiving and parsing the search results before outputting select result fields to the caller:

 function tweetStream() {

  var request = connection.request('GET', "/search.json?q=" + keyword 
     + "&amp;since_id=" + counter
     , {"host": host, "User-Agent": "TweetStreamer"});
    
  request.addListener("response", function(response) {

    var responseBody = "";

    response.setEncoding("utf8");

    response.addListener("data", function(chunk) { responseBody += chunk });

    response.addListener("end", function() {
      tweets = JSON.parse(responseBody);

      var results = tweets["results"];

      for(var x = 0; x &lt; tweets["results"].length; x++) {
        sys.puts("From: " + results[x].from_user + "n" + results[x].text + "nn");
      }
    });
  });

  request.end();
  setTimeout(tweetStream, 5000);

};

Finally, you'll execute the function:

tweetStream();

Save this code to a script called something like tweet.js and then execute it from the terminal like this:

$ node tweet.js

After a few moments you'll begin seeing the latest tweets containing the keyword mysql scroll by, a few of which are presented here:

 From: adeshosho
@pelosbriseno la verdad es que solo me faltan dos reportes, as&iacute; que me las ingeniare con vb y mysql, ni hablar

From: EmpregaTI
RT @tramposTI: DBRINTERATIVA #sp seleciona PROGRAMADORES. ninjas de Web com dom&iacute;nio em php, ajax e mysql. cv para vagas@dbrinterativa.com

From: ratzpo
RT @skysql_ab: Former Oracle MySQL Customers Drive SkySQL Sales to Seven Figures in its First Twelve Weeks - http://bit.ly/i9rLr5

Learning Resources

Are you doing something cool with Node.js? Tell us about it in the comments!

About the Author

Jason Gilmore, Contributing Editor, PHP, is the founder of EasyPHPWebsites.com, and author of the popular book, "Easy PHP Websites with the Zend Framework". Jason is a cofounder and speaker chair of CodeMash, a nonprofit organization tasked with hosting an annual namesake developer's conference, and was a member of the 2008 MySQL Conference speaker selection board.

Sitemap | Contact Us