TJ Holowaychuk

Modular CSS preprocessing with rework

Thu, 28 Feb 2013 19:04:00 -0800

Several months ago I started a project named Rework, a very fast, simple, flexible, and modular CSS preprocessor. The biggest and most obvious question I get is how this tool compares to something like Stylus, LESS, or Sass, and why would you want to use it.

The simple answer is that Rework caters to a different audience, and provide you the flexibility to craft the preprocessor you want, not the choices the author(s) have force on you.

Our goal with Rework contrasts the others, as it does not provide a higher level language within the CSS documents, Rework’s goal is to make those constructs unnecessary, being the most transparent CSS pre-processor out there.

Speed

Rework was designed to be simple and fast out of the box, without implementing complex caching models allowing everyone to hack on Rework itself or associate plugins easily.

Powering rework is a css parser written in JavaScript that may be used with node.js or in the browser. On the other end there is the css compiler which accepts the AST from css-parse, outputting CSS.

So where does Rework come in? Rework is the plugin and transformation engine that sits in-between these two tools. The AST is manipulated by Rework and its plugins (or user-defined plugins), which is then passed to the compiler.

Because these transformations are performed in JavaScript and not in a specialized preprocessor language they are simple to implement and efficient.

Let’s check out what it can do!

Modularity & Flexibility

Before diving into the plugins bundled with Rework, let’s take a look at how they can be applied. Rework’s API is very simple, you can specify vendor prefixes with .vendors(), which all plugins have access to, or you may pass specific vendor prefixes to most plugins themselves. Plugins are passed to a .use() method, and then finally the CSS output is returned with .toString(). In practice this looks something like:

var rework = require('rework');

var css = rework('string of css here')
  .vendors(['-webkit-', '-moz-'])
  .use(rework.prefixValue('linear-gradient'))
  .use(rework.prefixValue('radial-gradient'))
  .use(rework.prefixValue('transform'))
  .use(rework.vars())
  .use(rework.colors())
  .toString()

Out of the box rework provides plugins for the following, which may be mixed and matched as you wish:

media macros — define your own @media queries
ease — several additional easing functions
at2x — serve high resolution images
prefix — add vendor prefixes to properties
prefixValue — add vendor prefixes to values
prefixSelectors — add prefixes to selectors
opacity — add IE opacity support
url — rewrite url()s with a callback function
vars — add css variable support
keyframes — add @keyframe vendor prefixing
colors — add colour helpers like rgba(#fc0, .5)
references — add property references support height: @width etc
mixin — add custom property logic with mixing
extend — add extend: selector support

Let’s look at some examples!

Vendor prefixes

The prefix, prefixValue, and keyframes plugins make vendor prefixing extremely simple. Here the prefix plugin is applied to only two properties for illustration:

var rework = require('..')
var read = require('fs').readFileSync

var css = rework(read('examples/prefix.css', 'utf8'))
  .vendors(['-webkit-', '-moz-'])
  .use(rework.prefix('border-radius'))
  .use(rework.prefix('box-shadow'))
  .toString()

console.log(css)

With the prefix.css file containing the following CSS:

.button {
  border-radius: 5px;
  box-shadow: inset 0 0 1px white;
}

Rework yields the following prefixed output:

.button {
  -webkit-border-radius: 5px;
  -moz-border-radius: 5px;
  border-radius: 5px;
  -webkit-box-shadow: inset 0 0 1px white;
  -moz-box-shadow: inset 0 0 1px white;
  box-shadow: inset 0 0 1px white
}

Extending

The extend plugin adds support for a special property named extend, which propagates the selector back up the AST so that the styles are applied without duplication. For example here’s a contrived green join button implementation:

button {
  padding: 5px 10px;
  border: 1px solid #eee;
  border-bottom-color: #ddd;
}

.green {
  background: green;
  padding: 10px 15px
}

a.join {
  extend: button;
  extend: .green;
  padding: 20px;
}

With this plugin enabled both of the extend properties are gone, and you’re left with the CSS you might expect:

button,
a.join {
  padding: 5px 10px;
  border: 1px solid #eee;
  border-bottom-color: #ddd
}

.green,
a.join {
  background: green;
  padding: 10px 15px
}

a.join {
  padding: 20px
}

Better rgba()

Ever had a color hex and then wanted to alter the alpha channel? Then you probably know how annoying this is with CSS. The colors plugin allows you to pass hex values to rgba(), for example rgba(#c00, .5) which is then compiled to rgba(204,0,0,0.5).

Writing your own plugins

Writing plugins for Rework is simple, suppose for example you want the ability to create your own properties, here expanding single-line position properties:

#logo {
  absolute: top left;
}

#logo {
  relative: top 5px left;
}

#logo {
  fixed: top 5px left 10px;
}

Into the valid CSS shown below:

#logo {
  position: absolute;
  top: 0;
  left: 0
}

#logo {
  position: relative;
  top: 5px;
  left: 0
}

#logo {
  position: fixed;
  top: 5px;
  left: 10px
}

The AST css-parse provides you is made up of simple objects and arrays, making it extremely easy to manipulate without extra knowledge of how the nodes are constructed or strange internal APIs. In the following plugin function we walk the declarations, check if the property is one of the positions, parse the value by splitting on whitespace and then inject the new css properties.

function positions() {
  var positions = ['absolute', 'relative', 'fixed'];

  return function(style){
    rework.visit.declarations(style, function(declarations){
      declarations.forEach(function(decl, i){
        if (!~positions.indexOf(decl.property)) return;
        var args = decl.value.split(/\s+/);
        var arg, n;

        // remove original
        declarations.splice(i, 1);

        // position prop
        declarations.push({
          property: 'position',
          value: decl.property
        });

        // position
        while (args.length) {
          arg = args.shift();
          n = parseFloat(args[0]) ? args.shift() : 0;
          declarations.push({
            property: arg,
            value: n
          });
        }
      });
    });
  }
}

To use this plugin you would simply pass it to .use():

var css = rework('#logo { absolute: top left; }')
  .use(positions())
  .toString()

Significant whitespace

Since CSS has a relatively simple grammar significant whitespace can help improve your productivity without turning the whole thing into an unreadable mess, this is especially true since Rework does not provide the higher level language constructs, which can get lost in the indentation. This is where css-whitespace comes in! It lets you write things like:

ul
  li
    background: #eee
    &:hover
      background: white
    &:active
      background: #ddd

Which yields:

ul li:hover {
  background: white;
}

ul li:active {
  background: #ddd;
}

ul li {
  background: #eee;
}

Future

In the near future we’ll be providing a more intuitive way to apply all or large chunks of the functionality Rework provides without manually adding all of the plugins, as well as improving the rework(1) executable.

Mocha JavaScript test framework matrix reporter:...

Sun, 24 Feb 2013 16:48:32 -0800

Mocha JavaScript test framework matrix reporter: https://github.com/visionmedia/mocha-matrix

Ark - a work in progress digital painting with Pixelmator

Thu, 07 Feb 2013 10:05:29 -0800

Ark - a work in progress digital painting with Pixelmator

First 30 minute digital painting with Corel Painter

Sun, 23 Dec 2012 10:48:13 -0800

First 30 minute digital painting with Corel Painter

Modular web applications with Node.js and Express

Sat, 22 Dec 2012 13:37:16 -0800

Modular web applications with Node.js and Express:

This short screencast describes how you can use Express’ app mounting feature to create modular self-contained applications to construct your product.

Creating components screencast

Wed, 19 Dec 2012 08:53:59 -0800

Creating components screencast:

This screencast walks through creation of web components and some fundamentals driving the movement. Improve your client-side workflow and help us build a strong foundation for the web!

Web components introduction screencast

Wed, 19 Dec 2012 08:53:25 -0800

Web components introduction screencast:

An introduction into using the javascript implementation of components, walking through a quick example of how you might integrate them into your application. Note that this is the node implementation only, in the future hopefully we’ll have python, ruby, java, etcetera tools as well, all consuming the same things.

Building a date picker component

Thu, 13 Dec 2012 06:34:56 -0800

First install component(1) with npm:

$ npm install -g component

Or if you don’t have node installed at all, on OSX execute:

$ (cd /usr/local && \
   curl -L# http://nodejs.org/dist/v0.8.15/node-v0.8.15-darwin-x86.tar.gz \
   | tar -zx --strip 1) \
  && npm install -g component \
  && printf "installed component(1) %s\n" $(component --version)

Next you’ll need to bootstrap a new component, you can do this manually or use the quick component-create(1) command:

$ component create datepicker
repo (username/project): component/datepicker
description: Datepicker ui component built on component/calendar
does this component have js? y  
does this component have css? y
does this component have html? 

      create : datepicker
      create : datepicker/index.js
      create : datepicker/datepicker.css
      create : datepicker/Makefile
      create : datepicker/Readme.md
      create : datepicker/History.md
      create : datepicker/.gitignore
      create : datepicker/component.json

Adding dependencies

Next cd into the new directory and install the component/calendar component as a dependency:

$ component install component/calendar

    install : component/calendar@master
        dep : component/range@master
    install : component/range@master
        dep : component/jquery@master
    install : component/jquery@master
        dep : component/emitter@master
    install : component/emitter@master
        sep : component/in-groups-of@master
    install : component/in-groups-of@master
      fetch : component/calendar:index.js
      fetch : component/calendar:lib/utils.js
      fetch : component/calendar:lib/template.js
      fetch : component/calendar:lib/calendar.js
      fetch : component/calendar:lib/days.js
      fetch : component/calendar:lib/calendar.css
      fetch : component/range:index.js
      fetch : component/jquery:index.js
      fetch : component/in-groups-of:index.js
      fetch : component/emitter:index.js
   complete : component/range
   complete : component/in-groups-of
   complete : component/emitter
   complete : component/jquery
   complete : component/calendar

If you cat the component.json file you’ll see that component-install(1) has added the dependency for us:

  cat component.json 
  {
    "name": "datepicker",
    "repo": "component/datepicker",
    "description": "Datepicker ui component built on component/calendar",
    "version": "0.0.1",
    "keywords": [],
    "dependencies": {
      "component/calendar": "*"
    },
    "development": {},
    "license": "MIT",
    "scripts": [
      "index.js"
    ],
    "styles": [
      "datepicker.css"
    ]

Creating the component

Like any other software it’s good to have a suite of tests, or at very least a functioning demo for developers and end-users. In this case we’ll just create example.html to test drive the date picker, create the file and add the following HTML:

<!DOCTYPE html>
<html>
  <head>
    <title>Datepicker</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <link rel="stylesheet" href="build/build.css">
    <script src="build/build.js"></script>
  </head>
  <body>
    <input type="text" name="date" placeholder="Choose a date">

    <script>
       var picker = require('datepicker');
       var el = document.querySelector('[name=date]');
       picker(el);
    </script>
  </body>
</html>

Our date picker is going to be pretty simple, we’re going to handle click events for the element passed to picker(), and then display the Calendar in a Popover instance, populating the input when a selection is made.

Before getting started you’ll want to install the popover component, event utility, and the optional “aurora” theme:

$ component install component/popover component/aurora component/event

Or with brace expansion:

$ component install component/{popover,aurora,event}

The following JavaScript snippet is what will go into ./index.js, the main entry-point script of the component. As you can see here the dependencies are explicitly required at the top of the file via the commonjs module system, making it very obvious what this module is interacting with. Next we export a single object, the Datepicker constructor itself which sets up a bit of event handling boilerplate.

var Calendar = require('calendar')
  , Popover = require('popover')
  , event = require('event')

module.exports = Datepicker;

function Datepicker(el) {
  if (!(this instanceof Datepicker)) return new Datepicker(el);
  this.el = el;
  event.bind(el, 'click', this.onclick.bind(this));
}

Datepicker.prototype.onclick = function(e){

};

NOTE: The use of “event” may seem low-level here, and that’s because it is, the event module is nothing more than a cross-browser event binding utility. These APIs are not designed to be cute, they’re designed to drastically reduce the dependency bloat, by only using what your component needs. “Cute” APIs are provided by higher level components such as the jQuery-like dom component, and are typically recommended for app-level use only

To finish up the JavaScript portion of the Datepicker instantiate a new Calendar in the constructor, and add a classname for styling purposes. Next in the .onclick handler a new Popover is passed the cal element for its content, and a datepicker-popover classname is given here for styling purposes as well, then finally the call to popover.show(this.el) shows the popover relative to the <input> element.

var Calendar = require('calendar')
  , Popover = require('popover')
  , event = require('event')

module.exports = Datepicker;

function Datepicker(el) {
  if (!(this instanceof Datepicker)) return new Datepicker(el);
  this.el = el;
  this.cal = new Calendar;
  this.cal.el.addClass('datepicker-calendar');
  event.bind(el, 'click', this.onclick.bind(this));
}

Datepicker.prototype.onclick = function(e){
  this.cal.on('change', this.onchange.bind(this));
  this.popover = new Popover(this.cal.el);
  this.popover.classname = 'datepicker-popover popover';
  this.popover.show(this.el);
};

Datepicker.prototype.onchange = function(date){
  el.value = date.getFullYear()
    + '/'
    + date.getMonth()
    + '/'
    + date.getDate();

  this.popover.hide();
};

Building the component

To build the component invoke the following component-build(1) command:

$ component build

Or simply use make, as the targets are already set up in the Makefile for you:

$ make

A new directory named ./build will now have been created for you with the resulting build.css and build.js files.

To make this process transparent in development I recommend using the watch command that some environments already have, for those that do not have it there’s watch(1). Before working on a component you may now simply watch the make command and chuck that process in the background to get out of your way:

$ watch make &

When you’re done invoke fg to foreground the job again, or kill it with:

$ kill %1

Styling the component

Without custom styling the Aurora calendar / popover themes will look something like the following, not exactly great for a date picker since the calendar already looks self-contained.

Add the following CSS to ./datepicker.css to cancel out the popover border for the datepicker popover only:

.datepicker-calendar {
  font: 10px "Helvetica Neue", Helvetica, Arial, sans-serif;
}

.datepicker-popover .tip-arrow {
  top: auto;
}

.datepicker-popover .tip-inner {
  border: none;
}

Now re-build and you should have the following completed datepicker:

Publishing

To “publish” the component all you need to do is push it to GitHub and then install with component install myname/datepicker! No fighting over names, you can call them whatever you like. To increase discoverability of your component you can then add it to the Wiki component listing and the component-search(1) command will index it.

Get the code for this component at component/datepicker.

For more on components check out the component repo and the Wiki.

Monitoring processes with mon

Mon, 12 Nov 2012 22:14:00 -0800

mon is an extremely small, simple, and light-weight alternative to monit for monitoring processes. Monit’s approach of using a DSL is at times inflexible and often annoying. Mon’s approach is to monitor only a single process, monitoring of several require a mon instance per process, however reducing single points of failure.

This may sound like it’s a lot of work to manage, but I promise you it’s not! There’s another tool for this called mongroup(1) to tie them together.

Monitoring a single process

Mon instances weigh in at about 400kb per process, so you don’t have to worry about spawning a bunch of them. The simplest use of mon accepts a command to execute, and keep alive:

$ mon "echo hello; sleep 5"

Mon will execute the command with /bin/sh -c, so brace expansion and other shell goodies are completely fine. Once the sub process exits, mon will bring it back to life. By default mon logs to stdout:

mon : child 94136
mon : sh -c "echo hey; sleep 5"
hey
mon : last restart less than one second ago
mon : 10 attempts remaining
mon : child 94138
mon : sh -c "echo hey; sleep 5"
hey
mon : last restart 5 seconds ago
mon : 9 attempts remaining
mon : child 94140
mon : sh -c "echo hey; sleep 5"
hey

Mon will continue to execute the command until mon itself is signalled with SIGQUIT or SIGTERM at which point it will signal the child using the same signal, and gracefully exit.

When a process continues to fail upon restart, cyclic spawning will be detected by mon, it will then exit and log a warning. By default 10 restart attempts within 60 seconds are allowed, however you may tweak this value with --attempts <n>.

Failure alerts

Mon provides two facilities for notification. The first is the --on-restart <cmd> flag, which mon will invoke upon any restart. This is useful for emailing administrators the tail of a log file, notifying your team via IRC, etcetera.

The second, and more crucial facility is the --on-error <cmd> flag, which is invoked only when a cyclic restart is detected, and mon has bailed out. Administrators should be notified immediately as the process is completely down.

Daemonization

Mon’s --daemonize flag may be used to background the process and disassociate with the terminal, at this point stdio will be redirected to the log file specified by --log, otherwise defaulting as “./mon.log”, typically this looks something like:

$ mon -d -l /var/log/app.log "node app.js"

Monitoring multiple processes with mongroup

Using mon with several flags is obviously not something you’d want to be typing all the time, but part of the benefit of using mon is that shell scripts become your DSL. If you’re not interested in doing this yourself but have several processes to manage I recommend checking out a project by jgallen23 called mongroup(1). I’ve forked the project to add some goodies, until merged you can find them here.

Mongroup uses a simple configuration file, defaulting to the name ./mongroup.conf, which simply lists out the pids and files directories, as well as one or more processes to spawn:

 pids = /var/run
 logs = /var/log
 web1 = node server 3000
 web2 = node server 3001
 web3 = node server 3002
 web4 = node server 3003
 redis = redis-server

Mongroup ships with typical init-style commands start, stop, restart, status and so on, without a lot of the boilerplate associated with writing init scripts.

Launching processes

To check the status of your monitoring group, simply invoke mongroup status, or mongroup, as it’s the default sub-command. You’ll see a list of process names, pids, and the status itself:

Firing up the entire suite of processes takes only a single command, mongroup start:

Checking the status again will show the process uptime:

You may also start, stop, or restart single processes at a time via:

$ mongroup stop [name]
$ mongroup start [name]
$ mongroup restart [name]

Since mongroup is working-directory sensitive unless the --config <file> flag is specified, I recommend adding a small snippet to your shell profile similar to the following:

procs() {
  (cd ~ && mongroup $@)
}

If you’re super paranoid you can monitor mon with mon:

  $ mon "mon 'node app'"

That’s all for now!

EDIT: changes to mongroup(1) have been merged! https://github.com/jgallen23/mongroup

Express 3.0

Tue, 23 Oct 2012 15:28:00 -0700

Express 3.0 is here (finally) and while it is mostly a refinement release, between it and Connect 2.x there are some helpful new features. Sorry for the massive delay! Been busy and wanted to get a reasonable amount of documentation up on expressjs.com before releasing, more docs will be coming soon.

Keep in mind that the goal of Express is to aid you with HTTP, not to become a framework super-power like Rails, so this update may be underwhelming to some since it’s merely a refinement.

Connect 2.x

Changes introduced by Connect 2.x:

added err.status support to Connect’s default end-point
added session() “proxy” setting to trust “X-Forwarded-Proto”
added cookieSession() middleware
added compress() middleware for gzipped responses
added multipart() middleware
added json() middleware
added urlencoded() middleware
added limit option to the three above middleware
added defer option to multipart() to listen on formidable’s events
added debug() instrumentation to aid in debugging
changed basicAuth()’s req.remoteUser to req.user
changed session() to only set-cookie on modification (hashed session json)
changed bodyParser() to be an aggregate of json(), multipart() and urlencoded()
moved many cookie-related utils into npm
moved static()’s logic into a separate npm module named “send”
increase perf ~%20 by memoizing url parsing
removed router() middleware
fixed default encoding for logger(), now “utf8” instead of “ascii”
fixed mount-path case-sensitivity

Connect docs are available at http://www.senchalabs.org/connect/, and will eventually be mirrored on expressjs.com as well for convenience, along with usage guides.

Express 3.x

Changes introduced by Express 3.x:

added several new examples in ./examples
added unit testing for the examples (most of them at least)
added res.jsonp() to explicitly opt-in to JSONP support
added ETags and conditional-GET handling to res.send() responses
added “jsonp callback name” setting
added support for status code as first or second arg to res.send() and res.redirect()
added req.range(size) to parse Range header fields
added req.auth for basic auth
added res.links(obj) to set response the Link header field for pagination
added res.format(obj) for content-negotiation
added req.fresh for conditional-GETs
added req.stale for conditional-GETs
added mount-point relative redirection support to res.redirect()
added req.ip for the remote address (supporting reverse proxies)
added req.ips for remote address(es) (supporting reverse proxies)
added [] support in jsonp callback
added app.get(name) to compliment app.set(name, val)
added app.engine() to register template engines (replaces app.register())
added req.subdomains to return an array of subdomains
added req.protocol to return the request protocol string (“http” or “https”)
added req.secure to assert that req.protocol is “https”
added req.path to return the parsed url’s pathname
added req.host to return hostname (Host void of port)
added debug() instrumentation to aid debugging
added req.accepts()
added req.acceptsLanguage()
added req.acceptsCharset()
added req.accepted
added req.acceptedLanguages
added req.acceptedCharsets
added “json replacer” setting to manipulate json responses (remove private keys etc)
added “json spaces” setting to compress or expand json as you like (defaults to 2 in dev)
added express.application prototype
added express.request prototype
added express.response prototype
added app.render() for app-level templates
added res.type() to replace old res.contentType()
added { signed: true } option to res.cookie()
added async signature to res.render(), engines in consolidate.js work OOTB
removed partial()
removed express-level layout support (engines provide similar)
renamed “case sensitive routes” to “case sensitive routing”
removed res.signedCookie()
removed “root” setting
removed res.redirect('home') support
removed req.notify()
removed app.register()
removed app.redirect()
removed app.is()
removed app.helpers()
removed app.dynamicHelpers()

Head over to the New Features in 3.x wiki page for a more comprehensive list of additions, or to 3.x migration to help you upgrade if you wish to do so.

“path” - 45 minute digial painting with pixelmator

Sat, 20 Oct 2012 18:31:28 -0700

“path” - 45 minute digial painting with pixelmator

more game WIP

Wed, 19 Sep 2012 21:37:39 -0700

more game WIP

Mocha 1.4.0

Wed, 22 Aug 2012 16:39:47 -0700

This is another relatively small release but still has some goodies!

.only()

By appending .only() Mocha will generate an internal .grep() call for you. This is very useful if you have a regression and a test-case fails. Instead of moving the test-case and associated code out in order to debug you can simply invoke it.only(title, callback) (for the BDD UI) and Mocha will ignore all others. Here’s an example that will execute “bar” and its test-cases only:

describe('foo', function(){
  it('should foo', function(){

  })
})

describe.only('bar', function(){
  it('should bar', function(){

  })
})

.skip()

By appending .skip() you can tell Mocha to ignore these tests. This was previously called xdescribe(), xit() etc to match other frameworks (Jasmine I believe?) but we decided to go with .skip() which is more readable. The jasmine-style functions will remain for now but they will be removed in the future. This is effectively the same as commenting out cases, however they remain in a pending state so that you do not forget to re-enable them.

describe('foo', function(){
  it.skip('should foo', function(){

  })
})

describe('bar', function(){
  it('should bar', function(){

  })
})

Filtering HTML passes and failures

The “passes” and “failures” labels are now clickable links that will filter respectively.

Retina

The canvas progress reporter is now retina-enabled via the autoscale-canvas component. No more ugly pixelated progress!

Changelog

Here’s the full changelog between 1.3 and 1.4:

add mkdir -p to mocha init. Closes #539
add .only(). Closes #524
add .skip(). Closes #524
add passes/failures toggling to HTML reporter
add pending state to xit() and xdescribe() [Brian Moore]
add the @charset “UTF-8”; to fix #522 with FireFox. [Jonathan Creamer]
add border-bottom to #stats links
add check for runnable in Runner#uncaught(). Closes #494
add 0.4 and 0.6 back to travis.yml
add -E, --growl-errors to growl on failures only
add prefixes to debug() names. Closes #497
add Mocha#invert() to js api
change str.trim() to use utils.trim(). Closes #533
change dot reporter to use sexy unicode dots
fix HTML progress indicator retina display
fix url-encoding of click-to-grep HTML functionality
fix exports double-execution regression. Closes #531
fix error when clicking pending test in HTML reporter
fix make tm

A ~2.5 hour work in progress digital painting

Fri, 17 Aug 2012 00:12:00 -0700

A ~2.5 hour work in progress digital painting

“Edge” - a ~3 hour digital painting

Thu, 09 Aug 2012 21:28:07 -0700

“Edge” - a ~3 hour digital painting

Components

Wed, 25 Jul 2012 08:34:00 -0700

With the advent of numerous client-side JavaScript package managers, I wanted to write up some of my thoughts about the fragmentation that we have today, and some ways that I think we could really improve delivering packages a community. Keep in mind that these are only my opinions, everyone has done a great job and there’s a lot of cool work going on out there.

I think there are a few problems with the current environment, though some are subjective. The first of these and the most important in my opinion is the lack of unification around package consumption and definitions. The second issue is package delivery, what’s going to transport them to your machine? The final issue is the use of the components, how are they exposed to the browser for your application.

Once I’m done rambling I’ll show you how we’ve been using components to build our application at LearnBoost, and how they really fare. Before getting into all that let’s dig into what I mean by “component”.

Building components

There have been many attempts to come up with some sort of client-side package manager, and some of them do it very well, however I think they are missing the big picture, a “component” is much more than just JavaScript.

Create components, not JavaScript packages

A component can be not just JavaScript, but CSS, images, fonts or other resources, or a component may be any combination of these. This is the main idea that I want to sell, we need to broaden modularity beyond just JavaScript source. This is not a new concept at all, Drupal has been employing it for over 7 years but it seems like something that hasn’t really caught on in most communities, at least not beyond the private application level.

A great example of a component would be a popover from the Twitter Bootstrap library (not picking on you guys, it’s just a good example):

This thing looks so sexy that I just want to install it with my package manager, and have it automatically available to me in the client. For this to happen we need to admit that a component is much more than JavaScript, and should be packaged accordingly. Like the “dialog” in UIKit for example:

UIKit even at this state is not a good example of a solution for this problem, it uses its own custom build script to produce a single ui.js and ui.css file from the components in the repo, so it’s still not something the community can consume directly without the custom build step. I’ve started moving UIKit components over to github.com/component.

Another problem in the community is the size and scope of projects.

Size and scope

The classic battle between DOM manipulation libraries such as jQuery and MooTools serve as an obvious example of fragmentation. Even if one is more popular than the other this doesn’t mean we don’t have a problem. Have you ever seen a great jQuery plugin and thought to yourself “damn! I’m using MooTools!” or perhaps the other way around? That highlights the problem right there, we should have no “jQuery plugins”, no “Dojo modules”, just simply “components” that we can all consume.

Components could then utilize these smaller, more modular dependencies to perform tasks. Instead of requiring jQuery as a dependency to convert a string of HTML into elements, one could simply add domify as a dependency and invoke domify(html), similar components could facilitate event handling etcetera. My point here is that ubiquitous libraries like jQuery will eventually be a thing of the past and fragmentation will hopefully decrease.

Another thing I think we really need to avoid, is the use of pre-processed assets within components, this includes things like Stylus, LESS, Sass, Jade, CoffeeScript among others. These are all great and can increase productivity in your application, my opinion is that they do not belong in public components, they fragment the community and reduce contributions, chances are if you’re depending on these tools for UI packages your component’s scope is too large.

A package such as “UIKit” could then simply aggregate all its parts into one convenient component for those who wish to consume the entire thing, but there’s no reason these need to live together, we’re just lacking the tools to make this convenient.

This leads to the next issue, modernization!

Modernization

There are a lot great tools being produced to deal with modernization, within reason we should not have to deal with these kinds of issues at the component level. Stylesheets are particularly bad for this, libraries are either faced with using CSS preprocessors to provide vendor prefixed versions of their styles, or of course need to manually declare them. This is a huge pain, and forever changing. What do we do? Nothing!

I strongly suggest that we write our public component stylesheets using regular old CSS. Other tools can still be utilized at the application level if you really need them. With explicit vendor prefixing out of the way we can enjoy building components since we’re not focused on vendor details (when possible), they will remain light-weight, and customizable via consumer manipulation via a library such as CSSOM or node-css.

On to the next topic! Structural styling.

Structural styling

Structural styling is another thing I frequently have issues with. JavaScript libraries, jQuery plugins and others often ship with stylesheets, this is great but these libs should only define structural styles.

What do I mean by “structural styling”? Take for example the Dialog from UIKit, it contains only styles necessary to make the component presentable, but does not force a ton of extra styling on you, and serves as a good base to build on. I even took this a little far, box-shadows etc should be omitted by default.

The basic idea here is let stylesheets be stylesheets, application should define the look and feel of things, if you must provide “themed” versions simply have a component named “dialog-dark-theme” with only a stylesheet in it, and so on.

package.json

Components could have a “component.json” much like the commonjs package.json, or we can simply extend package.json. This would of course act as the package manifest, letting the world know if it has stylesheets, templates, scripts, images and so on. I believe we should avoid the minimal gains of magical auto-globbing of files, we can just simply list these and avoid unnecessary complexity and I/O.

A Dialog component would simply look like the following, nothing fancy here, just an explicit manifest.

{
   "name": "dialog",
   "version": "1.0.0",
   "scripts": ["index.js"],
   "styles": ["dialog.css"],
   "templates": ["dialog.html"]
}

The reason I would name these styles, and templates instead of css and html would be that when using private components within your application, the build tool could simply sniff out things like ["login.jade"], realize it’s a Jade template and compile it appropriately. Like I’ve mentioned though I dont think these tools belong in public code.

It may also be useful in the future to define optional dependencies for legacy browser support. Suppose I dont care anything below IE 9, I should be able to tell the build system that I’m fine with omitting legacy functionality, and the packages that handle this sort of things for events, the DOM etcetera would simply not load those in, the APIs that those modules provide would remain identical.

Require fragmentation

Another huge issue in the community right now is fragmentation regarding javascript loader definitions, the most common probably being AMD. Personally I’m not a fan of AMD for one reason, it’s ugly.

Granted that’s not a very good reason to dislike something. AMD provides the nice benefit of working without a build step, however when you need a library to test out a script anyway I feel like a quick build step to convert the more convenient commonjs require() style wins here, at least for me.

The real problem today is that if you want to share your public client-side code, you end up with something like the following:

  // AMD support
  if (typeof define === 'function' && define.amd) {
      define(function () { return Cookies; });
  // CommonJS and Node.js module support.
  } else if (typeof exports !== 'undefined') {
      // Support Node.js specific `module.exports` (which can be a function)
      if (typeof module != 'undefined' && module.exports) {
          exports = module.exports = Cookies;
      }
      // But always support CommonJS module 1.1.1 spec (`exports` cannot be a function)
      exports.Cookies = Cookies;
  } else {
      window.Cookies = Cookies;
  }

This sucks. I hope to never look at this, nor support this sort of concept ever again! This is really painful. I completely understand that perhaps AMD nor sync-style requires are perfect, but if we pick one we can easily translate scripts to an async style for production builds if necessary.

Delivering components

We’ll obviously need a way to store and deliver components, existing tools are close but I think we can still do a little better and think a little larger.

Package managers

Tools like node’s npm(1) package manager are again very great, but there are several reasons I think these packages should not live in the npm registry.

The first reason being that not everyone uses node, only us as node developers can be convenienced by using it, other communities should be able to easily access and consume these components with whichever tools they prefer, or if they really want to use one written in another language then sure why not!

Client-side packages in the npm registry are ambiguous, without tagging them as such it’s unclear which are intended for which environment. Many packages also end up disambiguating with a suffix, for example debug works both on the client and in node, however without augmenting package.json one would need to publish “debug” and “debug-component” or similar to the registry.

Another reason is that the package manager simply should not matter, and it usually does not matter. You just want some packages on your local machine, these should absolutely be as decoupled as possible.

Component registries

Arguably the best registry would simply be Github, or git repositories in general. There are a few downsides to this of course:

- even shallow clones can be slow
- coupled with git both at the consumer and producer levels
- large dependency urls (unless we default github to "username/project" etc...)

Some nice benefits as well:

- clear & explicit dependency origins
- github is awesome

Utilizing components

Most “real” applications require build steps these days, however even with this being the case, build scripts are extremely implementation specific and may vary greatly for your private application(s) depending on the scope of the project.

Here I would propose a free-for-all, use whatever you like from whichever community you’re a part of. As long as the “package manager” can transfer these to your machine, you do whatever you need to integrate them into your application, the details here are largely irrelevant.

Components in practice

Traditionally most applications (that I’ve experienced) are built in more of a “vertical” approach, spreading a single concept into multiple sections of an application. Making the application, and the components of a specific feature very difficult to reason with. This is akin to how unix libraries splatter themselves all over your system into various directories and files.

At LearnBoost we’ve been using components for a while now, but like I’ve mentioned not only for abstract UI components, but for everything in our application, even the build system and application bootstrap are implemented as components.

What does look like? Nothing more than a simple list of directories as shown in the following screenshot. Each component here is comprised of any combination of server-side logic, styles, scripts, images, fonts and so on.

Beyond obvious organizational benefits this helps our team focus on specific tasks without stepping all over each other. Testing is also easier, as each component providing server functionality simply exports a fully self-contained express application, which has its own set of tests, then our root make test iterates those.

Our build system simply produces the output scripts and styles that we need into our root ./public directory, wrapping scripts with commonjs require()s, images, fonts and other assets are served from the self-contained servers.

My point here is that this concept goes beyond public components. Obviously it would be next to impossible, and ill-advised to start sharing components with server portions, but for your own application sticking to whatever language / platform you’re using it works very well.

What next?!

Keep building! The JavaScript community has been moving in great directions lately, take what I’ve said with a grain of salt but I hope to see some attempts at “component” management, not just JavaScript ;)

Until then you may want to keep your eyes on https://github.com/component for future work from us at LearnBoost in regards to this topic. EDIT: I’ve started a component wiki where we can toss around ideas.

EDIT: Work on the component(1) executable have begun, as well as a JavaScript implementation of a builder for components, aptly named component/builder.js. For now component-build(1) is shipped with component(1) to ease use, otherwise the tools are not coupled. For example you could use `component(1) to perform installation and then use builder.js directly in code and customize your build, use the command, or use an entirely custom solution if you wish. I’ve started documenting some frequently asked questions, and we’re starting a Wiki page listing some of the components available so far, with well over 70 within the last month or so we’re off to a good start!

git-extras introduction screencast

Tue, 10 Jul 2012 07:39:58 -0700

This screencast is an introduction to the git-extras project and covers the following commands:

git setup
git ignore
git summary
git effort
git changelog
git release

Git aliases

My personal git aliases as promised:

alias gd="git diff | mate"
alias ga="git add"
alias gbd="git branch -D"
alias gst="git status"
alias gca="git commit -a -m"
alias gm="git merge --no-ff"
alias gpt="git push --tags"
alias gp="git push"
alias grh="git reset --hard"
alias gb="git branch"
alias gcob="git checkout -b"
alias gco="git checkout"
alias gba="git branch -a"
alias gcp="git cherry-pick"
alias gl="git log --pretty='format:%Cgreen%h%Creset %an - %s' --graph"
alias gpom="git pull origin master"
alias gcd='cd "`git rev-parse --show-toplevel`"'

Mostly I just wanted to play with my new snowball mic :D (no more loud keyboard noise!!). Also wtf vimeo compression?… might use youtube next time.

Page.js 1.2.0

Thu, 05 Jul 2012 16:17:00 -0700

Along with better API docs the page.js 1.2.0 is released, the client-side micro router.

The additions are subtle, the first is a ctx.querystring property exposing the query string, followed by the complimenting ctx.pathname to reference the ctx.path void of query string. @ovaillancourt also added support for passing query strings in the first place, which was arguably a bug, and added .defaultPrevented support to opt-out of page’s handling (as well as browser defaults of course).

Mocha 1.3.0

Thu, 05 Jul 2012 13:03:14 -0700

A relatively minor release this time but we’ve still got some cool new features!

Scrolling HTML reporter

The window will now scroll to follow along with larger test suites, and the statistics are fixed to the upper right-hand corner, a small but nice touch.

Custom reporters

Mocha has quite a few bundled reporters now, so future reporters that are not deemed important enough to provide the convenience of being in core should be published to npm. Some less important reporters may be removed from core and published to npm in the future.

This new feature also allows you to use localized private reporters. The --reporter NAME flag now behaves like require(), so for example you could do --reporter ./myreporter, or a third-party npm module such as --reporter lcov-reporter.

For details on creating and finding third-party reporters visit the wiki.

Grep inversion

The awesome --grep feature can now be inverted with the aptly named --invert flag. For example suppose you had tests tagged as @slow, and you wanted to run only slow tests you might use --grep @slow, however if you wanted to run only fast tests --grep @slow --invert would do the trick.

Along with this change --grep PATTERN is now escaped, meaning chars that used to behave as regexp special chars are now escaped.

Changelog

Here’s the full changelog:

1.3.0 / 2012-07-05

add window scrolling to HTML reporter
add v8 --trace-* option support
add support for custom reports via --reporter MODULE
add --invert switch to invert --grep matches
fix export of Nyan reporter. Closes #495
fix escaping of HTML suite titles. Closes #486
fix done() called multiple times with an error test
change --grep - regexp escape the input

commander.c - C option parser

Wed, 20 Jun 2012 09:13:37 -0700

Every time I write a C program I end up rolling my own option parsing with a loop and strcmp(), so I figured it was time to port over commander.c from the original ruby commander, and there’s also a node.js port.

Commander.c is a super simple script that makes defining options declarative and does the dirty work for you, its usage looks like this:

#include <stdio.h>
#include "commander.h"

static void
verbose(command_t *self) {
  printf("verbose: enabled\n");
}

static void
required(command_t *self) {
  printf("required: %s\n", self->arg);
}

static void
optional(command_t *self) {
  printf("optional: %s\n", self->arg);
}

int
main(int argc, const char **argv){
  command_t cmd;
  command_init(&cmd, argv[0], "0.0.1");
  command_option(&cmd, "-v", "--verbose", "enable verbose stuff", verbose);
  command_option(&cmd, "-r", "--required <arg>", "required arg", required);
  command_option(&cmd, "-o", "--optional [arg]", "optional arg", optional);
  command_parse(&cmd, argc, argv);
  printf("additional args:\n");
  for (int i = 0; i < cmd.argc; ++i) {
    printf("  - '%s'\n", cmd.argv[i]);
  }
  return 0;
}

As a bonus it generates the --help from what it already knows about your program:

Usage: example [options]

Options:

  -V, --version                 output program version
  -h, --help                    output help information
  -v, --verbose                 enable verbose stuff
  -r, --required <arg>          required arg
  -o, --optional [arg]          optional arg

Commander handles optional and required args:

$ mon --sleep
 --sleep requires an argument

Warns about unrecognized flags:

$ ./test --foo
unrecognized flag --foo

Unparsed arguments are then provided as cmd.argv and of course its companion cmd.argc, which of course includes -- support to flag subsequent args as literals:

$ ./test foo -- --foo
additional args:
  - 'foo'
  - '--bar'

If you have a struct that you want to access within each callback just assign to cmd.data, a void *.

That’s it for now, I’ll be adding a few more things in the near future so check it out on github.