Modular web applications with Node.js and Express

This short screencast describes how you can use Express’ app mounting feature to create modular self-contained applications to construct your product.

Express 3.0

Express 3.0 is here (finally) and while it is mostly a refinement release, between it and Connect 2.x there are some helpful new features. Sorry for the massive delay! Been busy and wanted to get a reasonable amount of documentation up on expressjs.com before releasing, more docs will be coming soon.

Keep in mind that the goal of Express is to aid you with HTTP, not to become a framework super-power like Rails, so this update may be underwhelming to some since it’s merely a refinement.

Connect 2.x

Changes introduced by Connect 2.x:

  • added err.status support to Connect’s default end-point
  • added session() “proxy” setting to trust “X-Forwarded-Proto”
  • added cookieSession() middleware
  • added compress() middleware for gzipped responses
  • added multipart() middleware
  • added json() middleware
  • added urlencoded() middleware
  • added limit option to the three above middleware
  • added defer option to multipart() to listen on formidable’s events
  • added debug() instrumentation to aid in debugging
  • changed basicAuth()’s req.remoteUser to req.user
  • changed session() to only set-cookie on modification (hashed session json)
  • changed bodyParser() to be an aggregate of json(), multipart() and urlencoded()
  • moved many cookie-related utils into npm
  • moved static()’s logic into a separate npm module named “send”
  • increase perf ~%20 by memoizing url parsing
  • removed router() middleware
  • fixed default encoding for logger(), now “utf8” instead of “ascii”

  • fixed mount-path case-sensitivity

    Connect docs are available at http://www.senchalabs.org/connect/, and will eventually be mirrored on expressjs.com as well for convenience, along with usage guides.

Express 3.x

Changes introduced by Express 3.x:

  • added several new examples in ./examples
  • added unit testing for the examples (most of them at least)
  • added res.jsonp() to explicitly opt-in to JSONP support
  • added ETags and conditional-GET handling to res.send() responses
  • added “jsonp callback name” setting
  • added support for status code as first or second arg to res.send() and res.redirect()
  • added req.range(size) to parse Range header fields
  • added req.auth for basic auth
  • added res.links(obj) to set response the Link header field for pagination
  • added res.format(obj) for content-negotiation
  • added req.fresh for conditional-GETs
  • added req.stale for conditional-GETs
  • added mount-point relative redirection support to res.redirect()
  • added req.ip for the remote address (supporting reverse proxies)
  • added req.ips for remote address(es) (supporting reverse proxies)
  • added [] support in jsonp callback
  • added app.get(name) to compliment app.set(name, val)
  • added app.engine() to register template engines (replaces app.register())
  • added req.subdomains to return an array of subdomains
  • added req.protocol to return the request protocol string (“http” or “https”)
  • added req.secure to assert that req.protocol is “https”
  • added req.path to return the parsed url’s pathname
  • added req.host to return hostname (Host void of port)
  • added debug() instrumentation to aid debugging
  • added req.accepts()
  • added req.acceptsLanguage()
  • added req.acceptsCharset()
  • added req.accepted
  • added req.acceptedLanguages
  • added req.acceptedCharsets
  • added “json replacer” setting to manipulate json responses (remove private keys etc)
  • added “json spaces” setting to compress or expand json as you like (defaults to 2 in dev)
  • added express.application prototype
  • added express.request prototype
  • added express.response prototype
  • added app.render() for app-level templates
  • added res.type() to replace old res.contentType()
  • added { signed: true } option to res.cookie()
  • added async signature to res.render(), engines in consolidate.js work OOTB
  • removed partial()
  • removed express-level layout support (engines provide similar)
  • renamed “case sensitive routes” to “case sensitive routing”
  • removed res.signedCookie()
  • removed “root” setting
  • removed res.redirect('home') support
  • removed req.notify()
  • removed app.register()
  • removed app.redirect()
  • removed app.is()
  • removed app.helpers()

  • removed app.dynamicHelpers()

    Head over to the New Features in 3.x wiki page for a more comprehensive list of additions, or to 3.x migration to help you upgrade if you wish to do so.

Express 3.x alpha reference docs

Since I don’t have the Express 3.x site up yet I thought I would whip up a quick set of markdown docs from the source comments using dox.

To view these markdown docs in your terminal install mad(1):

$ cd /tmp && git clone --depth 1 git@github.com:visionmedia/mad.git && cd mad && make install

Update your mad-pages:

$ mad --update

Profit!

$ mad express

HINT: Press “/”, type and press enter to search. The keys u and n move to the prev / next occurrences:

HINT HINT: There are also mad(1) pages for Jade, HTTP status codes, errnos and others.

Express 3.0.0 alpha1

I just released the first alpha, but beware! the documentation has not been updated. I wanted to defer a release until I had time to re-write the site, but meanwhile people have been using the master branch anyway, so we might as well make it official (and available via npm).

The alpha is by no means complete, some features still need adjusting, though mainly internal refactoring. The wiki contains both the 2.x to 3.x migration guide, and documentation for most of the new features added. It’s worth noting you’ll also gain new features from Connect 2.x as well, which Express 3.x depends on.

There are quite a few new http utilities, better reverse proxy support, and refinements to the view system. Express 3.x is even smaller than 2.x totalling 975 SLOC according to sloc(1), where Express 2.x was around 1300. Most of the effort goes into actually removing and refining code to increase quality, and lots of test coverage (4286 SLOC).

I’ve also started a Express 4.x Roadmap with some details on where things may go in the future.

Contributing

If you’d like to contribute, one of the best ways right now would be to help upgrade the ./examples ! The Express repo has many examples, roughly half of which still use 2.x code and may not work. This time around we have test coverage for these examples, but many still need to be upgraded. To contribute first install the development dependencies:

$ npm install

Then run the acceptance tests:

$ make test-acceptance

Add some tests for an example, write some tests, make sure they work, then open a pull-request :)

That’s all for now! When I have time I’ll be rewriting the site, updating the wiki, and of course finishing the release. If there’s anything you find yourself doing often and think has a home in core let me know!

cheers!

EDIT: If you’re looking for docs we now have some auto-generated docs that you can view in your terminal. View this post.

node recap #2

JSONSelect

This first library is JSONSelect. This library allows you to query JSON using css selectors! this seems like an obvious one, but I haven’t personally seen any other implementations.

It’s not node-specific, however it is JavaScript, so naturally it works with nodejs as well, and over all it’s just a neat concept. Suppose we have the following JSON:

JSONSelect

  {
      "name": {
          "first": "Lloyd",
          "last": "Hilaiel"
      },
      "favoriteColor": "yellow",
      "languagesSpoken": [
          {
              "language": "Bulgarian",
              "level": "advanced"
          },
          {
              "language": "English",
              "level": "native"
          },
          {
              "language": "Spanish",
              "level": "beginner"
          }
      ],
      "seatingPreference": [
          "window",
          "aisle"
      ],
      "drinkPreference": [
          "whiskey",
          "beer",
          "wine"
      ],
      "weight": 172
  }

To grab the languages “Bulgarian”, “English” and “Spanish” we could issue the following query:

.languagesSpoken .language

cool!

node-canvas Image#src=Buffer

A few days ago I added support to node-canvas for Image#src= to accept Buffer objects, not only path strings. This needs to be polished, and to increase format support but we can finally apply image data from arbitrary sources:

 var buf = fs.readFileSync(__dirname + '/textures/jpeg');
 var img = new Image;
 img.src = buf;
 ctx.drawImage(img, 0, 0, 400, 400);

When the Buffer is assigned we can sniff the bytes to determine the format by looking for the “magic number”. For example a PNG starts with the following bytes “89 50 4e 47 0d 0a 1a 0a”, while a JPEG starts with “ff d8”.

Express 2.3.8

A recent Express refactor moved Connect’s router middleware into Express so that I could modify it further. This also means that Connect 2.0 will no longer have router. At first it seemed like a good idea, but routing to specific and it gives people the wrong idea, Connect is meant to be an abstraction layer for higher level frameworks, not to be used directly for application logic. Though this is not true for all cases, such as simply serving static files from a directory or two.

This commit consists of a general refactor of the router, while retaining the public API. What does this mean for Express? flexibility! For example now we may query Express to see which (if any) routes match a specific route path:

 app.get('/user/:id');
 // => [Route]

or routes that match any HTTP method:

 app.all('/user/:id');
 // => [Route, Route, Route]

this is functionally equivalent to:

 app.lookup.all('/user/:id');

Another alternative is “matching” rather than “lookup”. With app.match.VERB() we can query Express to see which routes would match the url passed, for example:

app.match.get('/user/12/edit');
// => [Route]

app.match.del('/user/12');
// => [Route]

app.match.all('/user/12?foo=bar');
// => [Route, Route]

In this commit I’ve introduced the ability to register logic for defining parameters. Within Express core we only support middleware-style param pre-conditions, for example auto-loading a user for any route that includes the :uid param:

app.param('uid', function(req, res, next, uid){
  User.find(uid, function(err, user){
    if (err) return next(err);
    req.user = user;
    next();
  });
});

app.get('/user/:uid, function(){
  // req.user
});

while this is great, some cases can be less verbose, while still expanding to middleware-style functions, this is where express-params comes in. This plugin extends Express with additional app.param() logic, for example via RegExp:

  app.param('uid', /^[0-9]+$/);

  app.get('/user/:uid', function(req, res, next){
    var uid = req.params.uid;
    res.send('user ' + uid);
  });

  app.get('/user/:name', function(req, res, next){
    var name = req.params.name;
    res.send('user ' + name);
  });

or via return value, useful for validations and coercion:

  app.param('id', Number);

  app.get('/user/:id', function(req, res, next){
    var id = req.params.id;
    res.send('typeof ' + typeof id + ' ' + id);
  });

Fabric.js

This one is not node related, but I wanted to sneak it in :) Fabric is a slick client-side library providing an object model on top of canvas. It has a built-in drawing mode and path smoothing, SVG parsing and some other really cool features. If you do a lot of canvas work it’s worth checking out, the author did a good job at keeping the source clean and commented, which is very important when utilizing open source.

fabricjs

node-migrate

node-migrate is a very small abstract migration framework. All it cares about is that you supply an up() and down() method, and invoke the callback when your migration is complete:

 exports.up = function(next){
   next();
 };

 exports.down = function(next){
   next();
 };

node-migrate comes with the migrate(1) executable, allowing you to run and create migrations, for example here we add two migrations, populating ./migrations/0-add-pets.js etc.

 $ migrate create add-pets
 $ migrate create add-owners

The contents of our badass pet adding migration might look something like this:

 var db = require('./db');

  exports.up = function(next){
    db.rpush('pets', 'tobi');
    db.rpush('pets', 'loki');
    db.rpush('pets', 'jane', next);
  };

  exports.down = function(next){
    db.rpop('pets');
    db.rpop('pets', next);
  };

Now all we have to do is run the migrations:

 $ migrate
 up : migrations/0-add-pets.js
 up : migrations/1-add-jane.js
 up : migrations/2-add-owners.js
 up : migrations/3-coolest-pet.js
 migration : complete

The next time we run migrate, we’ll see that they have already been completed:

 $ migrate
 migration : complete

For more information check out the GitHub repo.

Move.js

The other night I noticed that Apple’s new iMac page showcased some really slick CSS3 animations, and wanted a really simple and intuitive way to create similar effects so I came up with Move.js.

Below is an example of the API:

move('#example-13 .box2')
  .set('background-color', 'red')
  .x(500)
  .scale(.5)
  .rotate(60)
    .then()
      .rotate(30)
      .scale(1.5)
      .set('border-radius', 5)
      .set('background-color', 'white')
      .then()
        .set('opacity', 0)
        .pop()
      .pop()
  .end();

It’s still in it’s early stages, and I don’t have much in the way of documentation yet however feel free to check out the site as well as the examples in the repository.