thinktime

Nominations for the statewide commitee have been received and we now have a poll up to confirm the uncontested positions and collect votes on the remaining ones.



The following positions were uncontested:

* President: Craige McWhirter (ruagair)

* Treasurer: Scott Bragg (mrfaulteh)

* Media Muse: Mathew Oakes



Congratulations to Craige, Scott and Mathew.



Nominees for the two remaining "Community Champion" positions are:

* Scott Bragg (mrfaulteh)

* Scott Evans (VK7HSE)

* Chris Neugebauer (chrisjrn)

* Tim Serong (tserong)





The poll can be found here: http://www.surveymonkey.com/s/GC7TKRZ



To reduce the impact of spam/duplicate votes, we are asking people to provide an email address that is associated with a TasLUG website account, or a TasLUG mailing list subscription (both are worth having anyway, and signup links can be found on this site). These email addresses will not be used for anything other than removing invalid/duplicate votes. Addresses will also not be associated with votes and the data will be destroyed at the end of the poll.



Many thanks to "returning officer" bleeter for coordinating this!



Please discuss further on the mailing list or in the forums.

Libferris supports many indexing libraries and technologies through its plugin interface. Larger systems can use a PostgreSQL plugin which is tailored explicitly to get the most out of that RDBMs for larger file server indexes. For smaller end, there are memory mapped files, clucene, soprano, or SQLite. I've been doing some tinkering trying to milk extra performance out of the indexing plugins for ARM machines lately. Note that if you are using debian, the CLucene you'll want is the 2.x series, currently only packaged for experimental.



For testing purposes I built a fairly tiny index of only 130k files. An interesting test case is looking for specific files which have paths that match against a regular expression and returns a fairly small chunk of results. For this case, about 115 resulting files using a four character substring search as the regex. These are a common query for looking for files when you don't recall the exact ordering of the directory names or where a directory was. Small number of results, regex to pick them.



The memory mapped index implementation (boostmmap) uses boost IPC and multi indexed collections created in memory mapped files to maintain the index. The index has also a digram index for each URL allowing regular expressions to resolve through index rather than needing evaluation against full URLs.



The SQLite index is fairly vanilla and doesn't include many customizations for sqlite. Whereas the PostgreSQL index implementation does use many of the features specific to that database. Neither the SQLite or boostmmap indexes in the public libferris repo attempt to do any compression on URL strings or the like.



A fairly basic index on 130k files is about 80mb using either memory mapped files or SQLite. Caches are cleared by echo 3 > drop_caches. Using an odroid-u2 with emmc flash, on a cold cache the SQLite index comes out about 10% faster than the boostmmap for a query finding 115 files. Turning off the regex prefilter index in the boostmmap makes it 10% slower again. This is a trade off, a very fast CPU and a disk with great file location and single extents will show less or no difference with the prefilter as reading 80mb from disk will take less time and the CPU can run 130k regexes very quickly. The prefilter requited only 124 regex evaluations, without the prefilter all 130611 URLs needed a regex evaluation.



The interesting part is with a warm cache the boostmmap is about twice as fast overall as the SQLite index. This is a big difference as the timing is for overall complete run time from the command line, and there is some overhead in starting up the index query itself. As usual, things vary depending on if you are expecting frequent queries (warm cache), have a very fast CPU (regex eval is relatively less costly), or need multiple updaters (SQLite allows it, my memory mapped doesn't).



To then experiment a little further, I brought the ferris clucene plugin into the mix. I disabled the explicit prefilter index on regex code for initial testing, the index became about 70mb and could resolve the query on a cold cache in about 65% the time of the SQLite plugin. On warm cache the clucene was slowest, which is mainly due to the prefilter being disabled and the fallback code making the URL query a WildcardQuery with no pre or postfix to anchor the query on.



Next time around I'll see how speed effective the prefilter index is on clucene. I know it slows down adding documents (you are indexing more), and is larger (I haven't optimized for index size), but it will be interesting to see the performance on the eMMC device for the prefilter.

The typical industrial-era organization is like a battleship. Hundreds or thousands of people onboard, and most of them are essential--but most of them aren't actually directly responsible for the work that we hired the battleship to do. Without the fuel people, the navigation team, the folks in the med corps and on and on, it doesn't work.

The battleship can go far, with impact, and change the course of history. While it has exactly one captain, it's the synchronized work of more than a million people (when you think about all the machinists and support folks back home) and it works. It does what we ask it to do.

One more thing about the people on the battleship: just about everyone has a punchlist, an itemized inventory of what they need to get done. And many of them are rewarded for doing that set of tasks more efficiently, more elegantly and with better quality than expected. Great people means the system works even better, but it's designed to survive with people who are merely good at what they do.

The typical professional services company, on the other hand, is a lot like a blueberry pancake. While there's an essential support team, the firm is all about blueberries working in parallel. Each blueberry can work independently, and sometimes they even work on projects that might have conflicting outcomes or views of the world. I don't care how many people report to you. I care about how connected and how brave you are.

As the firm gets bigger, it doesn't get thicker. You don't make a better pancake by making a thicker one. You make a better pancake by hiring ever better blueberries.

And, as you've guessed, most of the blueberries don't know exactly what they'll be doing in six weeks, and most don't work from a manual about the industry's best practices on how to do what they do. It's hard to measure blueberries, but a talented and motivated one can also change the world.

Apple is now a battleship. Most of the tens of thousands of people who work there have a line job, selling, building, fixing or interacting. Only a few are dreaming up something that you can't even imagine.

Your favorite record label, though, ought to be a blueberry pancake. Each musical group is mostly alone, figuring out something that just might work. The goal isn't to lock and repeat and scale. It's to go wide and stay interesting. Great record labels have both better blueberries and the support staff to launch them into the world.

I remember the day we transformed Yoyodyne from a pancake to a battleship. We hired 17 salespeople in 24 hours (increasing the size of the company by 25%) and for the first time, I didn't know every employee well. People had their orders, and we were ready to scale.

If you want to make your battleship work better, be really clear about defining the mission, the tactics, the chain of command and most of all, precisely what you measure from each person on the team.

Your pancake, on the other hand, gives up swing weight and firepower and instead gets flexibility and the possiblity of non-fatal failure (and game-changing magic).

Both work. The problem kicks in when a successful pancake thinks its future is in the battleship business. Or when battleships are asked to dance.

Wow, time moves quickly!



For anybody who needs a reminder, we've got out second ever TasLUG statewide gathering happening at the Ross Town Hall this weekend.



This is a chance for people from all around the state to come together and get to know/share knowledge with other Free/Open Source Software enthusiasts. People are encouraged to bring along a machine or a laptop or a mobile device or whatever's applicable to share with the community the sort of things they do with Free/Open Source Software.



Saturday May 4th

Ross Town Hall



Our rough schedule looks something like this:

* 10:00pm - 12:00pm: Mill around, get to know each other, show off any gear/projects we may have brought with us

* 12:00pm - 1:00pm: Break for lunch

* 1:00pm - 1:30pm: Housekeeping/Statewide committee stuff

* 1:30pm - 3:00pm: Presentations/lightning talks for anybody who wants to give one

* 3:30pm - 4:00pm: Pack up

* 4:00pm onwards: Go home/BBQ/whatever anybody is up for



Google Maps link



Hope to see everybody there!

Arguing for “separation of content from presentation” implies a neat division between the two. The reality, of course, is that content and form, structure and style, can never be fully separated. Anyone who’s ever written a document and played around to see the impact of different fonts, heading weights, and whitespace on the way the writing flows knows this is true. Anyone who’s ever squinted at HTML code, trying to parse text from tags, knows it too.

On one hand, the division of labor between writing and presentation can be seen at every point in our history. Ancient scribes chiseling stone tablets, medieval monks copying illuminated manuscripts, printers placing movable type—we’ve never assumed that the person who produces the document and the person who comes up with the ideas must be one and the same.

And yet, we know that medium and message are intertwined so tightly, they can’t be easily split apart. Graphic designers rail against the notion that “look and feel” can be painted on at the end of the process, because design influences meaning. The more skilled we are as communicators, the more we realize that the separation of content from presentation is an industrial-age feint, an attempt to standardize and segment tasks that are deeply connected.

Today, we try to enforce the separation of content and form because it’s good for the web. It’s what makes web standards possible. It enables social sharing and flexible reuse of content. It supports accessibility. It’s what will keep us sane as we try to get content onto hundreds of new devices and form factors.

When talking about how best to separate content from presentation, designers and developers tend to focus on front-end code—which makes sense, because that’s what we have the most control over. But, as with so many challenges we have with content on the web, the real issue lies in the tools we give content creators to help them structure, manage, and publish their content. The form that content takes depends as much on CMS as it does on CSS.

How should content management tools guide content creators to focus on meaning and structure? What’s the right amount of control over presentation and styling in the CMS? And how should these tools evolve as we break out of the web page metaphor and publish content flexibly to multiple platforms? Let’s look at three tools that sit at the intersection of content and form.

Preview button

Even the most die-hard structured content editors still like seeing what their work is going to look like. Writers print out documents for editing to give them a different view from what they see on the screen. Bloggers instinctively hit the preview button to look at their work the way a user will see it.

Whoops. Decades of work refining the emulators between desktop publishing programs and laser printers means that writers can feel confident that their document will look virtually identical, regardless of where it’s printed. We’ve carried that assumption over to the web, where it’s categorically untrue. Different browsers render content in their own vexingly special way. Users can change the font size—even add their own custom style sheet. Today, the same document will render differently on desktops, tablets, and mobile devices. The preview button is a lie.

Yet we can’t just throw the baby out with the bathwater. In fact, seeing content in context becomes even more important as our content now lives across devices and platforms. Instead of throwing up our hands and saying “preview is broken,” it’s time to invent a better preview button.

One publishing company I know of has built its own custom preview rendering interface, which shows content producers an example of how each story will appear on the desktop web, the mobile web, and an app. Is it perfect? Far from it. Content will appear in many more contexts than just those three. Is it better than nothing? Absolutely.

WYSIWYG

The desktop publishing revolution ushered in by the Macintosh allowed the user to see a document on screen in a form that closely mirrored the printed version. The toolbar at the top of the screen enabled the user to add formatting—change the font, insert an image, add typographic effects like headings and bullets, and much more.

In an effort to carry over this ease of use to the web, we allow content creators to embed layout and styling information directly into their content. Unfortunately, the code added by content creators can be at odds with the style sheet, and it’s difficult for developers to parse what’s style and what’s substance. When it comes time to put that content on other platforms, we wind up with a muddled mess.

What is the right amount of formatting control to give content creators? That’s a difficult question to answer, because it pierces right to the heart of what’s stylistic and what’s semantic. Even something as simple as adding bold and italic text forces us to ask if we’re really just styling the text, or adding semantic meaning (say, a book title or a warning message.)

Better content modeling can solve some of these problems, encouraging content creators to appropriately “chunk” their text. By banishing blobs of text with formatting embedded and replacing them with chunks of clean, presentation-independent content, we’re building in the distinction between content and form right from the start.

But imagining that each “chunk” of content is a field in the database (with its own input field) rapidly devolves into the absurd. That way lies madness. The real solution isn’t necessarily to “banish blobs,” but to replace the WYSIWYG toolbar with semantic markup. Rather than entering all text into discrete fields, content authors wrap text that describes what it is. Our book title doesn’t need to be a separate field if we can wrap it in the proper tags.

Defining what goes in a field and what goes in a tag requires a tighter collaboration between content authors, CMS architects, and front-end developers. It’s time we started having these conversations.

Inline editing

We’re evolving. Not satisfied to rely just on tools that are vestiges of the desktop publishing era, we’re developing new and innovative ways to mix up content and formatting that are unique to the way the web works. There’s no better example of this than inline editing.

Inline editing allows content creators to directly manipulate content in the interface, with no separation between the editing screen and the display. Medium offers an editing interface that’s identical to the desktop display and in-place editing is being added to Drupal 8 core.

One of the questions I get asked most frequently is “how can I get my content creators to understand why it’s so important to add structure and metadata to their content?” This, I believe, is one of the fundamental challenges we’re facing on the web, particularly as we adapt to a multi-channel future. Inline editing encourages content creators to focus on the visual presentation of the desktop interface. Just at the moment when we need content creators to think about the underlying structure, we’re investing in tools that obscure the “connective tissue.”

Jeff Eaton sums up this problem nicely in a post called Inline Editing and the Cost of Leaky Abstractions:

The editing interfaces we offer to users send them important messages, whether we intend it or not. They are affordances, like knobs on doors and buttons on telephones. If the primary editing interface we present is also the visual design seen by site visitors, we are saying: “This page is what you manage! The things you see on it are the true form of your content.”

The best solution isn’t to build tools that hide that complexity from the user, that make them think that the styling they’re adding to the desktop site is the “real” version of the content. Instead, our goal should be to communicate the appropriate complexity of the interface, and help guide users to add the right structure and styling.

The era of “desktop publishing” is over. Same goes for the era where we privilege the desktop web interface above all others. The tools we create to manage our content are vestiges of the desktop publishing revolution, where we tried to enable as much direct manipulation of content as possible. In a world where we have infinite possible outputs for our content, it’s time to move beyond tools that rely on visual styling to convey semantic meaning. If we want true separation of content from form, it has to start in the CMS.

To index data using a small ARM CPU without much RAM you might like to break the indexing run down into many parts, and get more explicit control over what is happening. The below will index all files on /DATA-PATH in batches of 5000 files at a time with libferris. This will use whatever index plugin you have setup for ~/.ferris/ea-index (the default metadata index). Be that PostgreSQL, SQLite, boost memory mapped files, clucene or whatever.



I'm currently racing the boost memory mapped index with the SQLite backed index on simple URL queries against the filesystem. This is being done on about 2ghz ARM machines with either 512 or 2048mb of RAM. The boostmmap plugin is of my own design and contains some smarts while executing regular expression matching against unanchored strings (.*foo.*). Unfortunately the boostmmap plugin is not as smart as it could be regarding scattered updates, transactions, and journaling, which slows it down a bit in the index creation phase relative to the SQLite plugin.



The below is a skeleton bash script to get started adding files. Another option is to ssh into the remote host and run find(1) there which can be much faster over network filesystems. The whitelist environment variable is to override which metadata libferris will index. If your index indicates it wants sha and md5 digests, the act of calculating those can dominate indexing time. An explicit whitelist keeps index adding times down with the obvious side effect of limiting what you can use in your queries. Such a limited list of metadata as in the below brings the index closer to what locate provides.



#!/bin/bash



rm -rf /tmp/fidxtmp

mkdir -p /tmp/fidxtmp

cd /tmp/fidxtmp



find /DATA-PATH | split -l 5000

 

export LIBFERRIS_EAINDEX_EXPLICIT_WHITELIST="name,size,mtime"



for if in x*

do

   echo "adding $if..."

   cat "$if" | feaindexadd -v -1 >>/tmp/ferris-index-progress.txt

done



Then you can find all your PDF files for example using the following:



feaindexquery -Z '(url=~pdf)'



The -Z tells libferris not to try to lstat() or resolve URLs to see if they exist currently. Much faster results but at the cost of not weeding out things which might have moved since they were last indexed.



And all the files which have been written this year



feaindexquery -Z '(mtime>=begin this year)'



Unfortunate about needing the quotes as bash wants to do things with naked parenthesis.



Save Ferris! Or just donate to an open source project or organization of your choice if you like the ferris posts.

Last week, I saw an extraordinary play on Broadway. It got the longest standing ovation I've ever seen in a theater, and Alan Cumming deserved every minute of it. The New York Times critic, though, didn't like the show.

What's the point of his review, then? Clearly the audience, discerning in their own right, disagreed. Do mainstream critics exist to tell us what to like, to warn us off from the not-so-good, or are they there to punish those that would dare to make a piece of work that doesn't match the critic's view of the world? Perhaps the critic is saying, "people like me will have an opinion like this," but of course, there just aren't that many people like him.

Have you noticed just how often the critics disagree with one another? And how often they're just wrong?

And yet we not only read them, but we believe them. Worse, we judge ourselves, contrasting our feelings with their words. Worse still, we sometimes think we hear the feared critic's voice before we even ship our work out the door...

For me, the opinion of any single critic is becoming less and less meaningful as I choose what to view or engage with. And the aggregate opinion of masses of anonymous critics merely tells me that the product or content is (or isn't) mass-friendly. I'm far more moved by the insistent recommendation of a credible, raving fan than I am the snide whispering of some people who just didn't get it.

The math is simple: no matter how big a critic's platform, what moves markets are conversations. And we are far more likely to have conversations about something we're raving about than something we didn't like (because when we don't like it, our friends never experience it and the conversation dies). The win, then, is creating raves, not avoiding pans.

Every single book I've written has gotten at least a few one star reviews on Amazon. Every one. The lowest possible rating, the rating of, "don't bother reading this, in fact it never should have been written." Not just me, of course. Far better writers, writers like Fitzgerald, Orwell and Kincaid have gotten even more one-star reviews on their books than I can ever hope to.

No one has ever built a statue to a critic, it's true. On the other hand, it's only the people with statues that get pooped on by birds flying by.

From their website. "Geant4 is a toolkit for the simulation of the passage of particles through matter. Its areas of application include high energy, nuclear and accelerator physics, as well as studies in medical and space science."

read more

From their website: "The CLHEP (Class Library for High Energy Physics) project was proposed by Leif Lönnblad at CHEP 92. It is intended to be a set of HEP-specific foundation and utility classes such as random generators, physics vectors, geometry and linear algebra. CLHEP is structured in a set of packages independent of any external package (interdependencies within CLHEP are allowed under certain conditions)."

read more

If your writing feels like nothing but easily defensible aphorisms, as if you're saying things that are obvious, it's entirely possible that no one is going to eagerly keep reading. Your real estate brochure or the ad copy you've written--if it's merely posturing or bragging, better to not say it at all. We already know you think you did something great.

Consider the alternative. Say the opposite. That your condo isn't right for everyone. That your software might be overpriced. That this new model car is in fact quite difficult to use.

And then tell us why. We'd love to know how you're going to wriggle out of that. And along the way, if your story is a good one, we might even give it a try.

Quite a few of the codebases I've worked on have some form of linked list implementation; generally a list node type:

struct list_node { struct list_node *prev; struct list_node *next; };

... and sometimes (for type-safety) a separate type for the lists themselves:

struct list { struct list_node head; };

Debugging these lists often means a lot of manual pointer-chasing, which can be error prone. If your list iterators are causing segfaults, it'd be nice to be able to track down invalid list entries.

python gdb extensions

It turns out we can write a little code to automate this, using the gdb module. Amongst other things, we can define our own commands to use in the standard gdb interpreter.

The skeleton code to implement a gdb command in python goes something like this:

import gdb class MyNewCommand(gdb.Command): def __init__(self): super(MyNewCommand, self).__init__("my-new-command", gdb.COMMAND_DATA, gdb.COMPLETE_SYMBOL) def invoke(self, argument, from_tty): args = gdb.string_to_argv(argument) expr = args[0] list = gdb.parse_and_eval(expr) # command implementation goes here ... MyNewCommand()

In a nutshell:

• We define a new subclass of gdb.Command
• Our new class' __init__() function will register the command name ("my-new-command" in this example) with the gdb interpreter, and pass a few hints about the command's usage.
• The invoke() function is called when the user invokes the command at the gdb interpreter interface.
• The invoke() implementation can access the debugger's state thorugh the gdb python module.
• Finally, we create an instance of the new class to get it registered with the interpreter.

The class' docstring is used by gdb's inline help command, which makes documenting your new command a total piece of cake.

Back to our list example, I've written a little python gdb extension to iterate and print a list:

#!/usr/bin/env python import gdb class ListPrintCommand(gdb.Command): """Iterate and print a list. list-print <EXPR> [MAX] Given a list EXPR, iterate though the list nodes' ->next pointers, printing each node iterated. We will iterate thorugh MAX list nodes, to prevent infinite loops with corrupt lists. If MAX is zero, we will iterate the entire list. List nodes types are expected to have a member named "next". List types may be the same as node types, or a separate type with an explicit head node, called "head".""" MAX_ITER = 20 def __init__(self): super(ListPrintCommand, self).__init__("list-print", gdb.COMMAND_DATA, gdb.COMPLETE_SYMBOL) def invoke(self, argument, from_tty): args = gdb.string_to_argv(argument) if len(args) == 0: print "Argument required (list to iterate)" return expr = args[0] if len(args) == 2: max_iter = int(args[1]) else: max_iter = self.MAX_ITER list = gdb.parse_and_eval(expr) fnames = [ f.name for f in list.type.fields() ] # handle lists with a separate list type.... if 'head' in fnames: head = list['head'] # ...and those with the head as a regular node elif 'next' in fnames: head = list else: print "Unknown list head type" return # if the type has a 'prev' member, we check for validity as we walk # the list check_prev = 'prev' in [ f.name for f in head.type.fields() ] print "list@%s: %s" % (head.address, head) node = head['next'] prev = head.address i = 1 while node != head.address: print "node@%s: %s #%d" % (node, node.dereference(), i) if check_prev and prev != node['prev']: print " - invalid prev pointer!" if i == max_iter: print " ... (max iterations reached)" break prev = node node = node['next'] i += 1 if check_prev and i != max_iter and head['prev'] != prev: print "list has invalid prev pointer!" ListPrintCommand()

This defines a new function, list-print, which takes an expression as an argument, and iterates through the list nodes:

(gdb) list-print handler->devices[0]->device->boot_options list@0x6128f0: {prev = 0x6129f8, next = 0x613918} node@0x613918: {prev = 0x6128f0, next = 0x613568} #1 node@0x613568: {prev = 0x613918, next = 0x6131b8} #2 node@0x6131b8: {prev = 0x613568, next = 0x612e08} #3 node@0x612e08: {prev = 0x6131b8, next = 0x6129f8} #4 node@0x6129f8: {prev = 0x612e08, next = 0x6128f0} #5 (gdb)

To use the gdb-list macro: download gdb-list.py (2.2 kB), and source it into your gdb session:

(gdb) source ~/devel/gdb-list/gdb-list.py

Then you should be able to invoke list-print <symbol> to debug your list structures. Typing help list-print will show the command's docstring:

(gdb) help list-print Iterate and print a list. list-print <EXPR> [MAX] Given a list EXPR, iterate though the list nodes' ->next pointers, printing each node iterated. We will iterate thorugh MAX list nodes, to prevent infinite loops with corrupt lists. If MAX is zero, we will iterate the entire list. List nodes types are expected to have a member named "next". List types may be the same as node types, or a separate type with an explicit head node, called "head". (gdb)

To compliment my existing packages of the libferris virtual filesystem and index/search suite for soft float debian, I now offer hot off the press, debian hard float! The distinction between how floating point is handled probably doesn't make a big difference to the operation of libferris, but if you are installing debian on an odroid-u2 then you are likely running hf, and as such having debs which are hf makes installing libferris a whole bunch simpler.



With the eMMC card on the u2, it is a really enjoyable little server machine to play around with. So far I've done the rudimentary test that XML is mountable as a filesystem and created one or two indexes with the Qt/SQLite index plugin for libferris. Note that in recent releases the sqlite backend is transaction backed which gives a huge performance increase, and on really IO constrained machines this is even more noticeable. This is a little tip for those using QtSQL, transactions are not just for making operations atomic, you may find that the whole show runs faster when it is transaction protected.



If you haven't played with libferris, things are auto mounted where possible, and there are many coreutils like tools to make interacting with ferris simple. The ferris-redirect is like the bash redirection but can write to any filesystem that libferris knows how to mount.



ben@odroidu2:/tmp/test$ cat example.xml

<top>

<node name="foo">bar

</node>

</top>

ben@odroidu2:/tmp/test$ fls example.xml

top

ben@odroidu2:/tmp/test$ fls example.xml/top

foo

ben@odroidu2:/tmp/test$ fcat example.xml/top/foo

bar

ben@odroidu2:/tmp/test$ date | ferris-redirect -T example.xml/top/foo

ben@odroidu2:/tmp/test$ cat example.xml

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<top>

  <node name="foo">Tue Apr 30 14:57:26 PDT 2013

</node>

</top>



The above interaction would also work for mounted Berkely DB and other filesystems of course.





I have noticed one of the binary scoped destructors doesn't like the hard float build for whatever reason. This can cause some of the command line tools to not exit gracefully, which is a shame. I can't get a good backtrace for the situation either, which makes tracking it down a nice day long adventure into trial and error.



So something productive has been generated during the last round of jet lag after all!



The goods http://fuuko.libferris.com/debian/debian-armhf/





Save Ferris!

Chris Coyier explains Magic Numbers:

Magic numbers in CSS refer to values which “work” under some circumstances but are frail and prone to break when those circumstances change. They are usually always related to fonts.

Many good examples in that post, and in the comments, but the one that stood out for me was Chris’ attempt to flank a heading with horizontal lines:

In a recent post Line-On-Sides Headers, I used a line-height value that was a magic number. Let’s say you used the technique around text with a fancy @font-face font. Let’s say that font doesn’t load or the user overrides it or the page is being viewed in a browser that don’t support @font-face. The fallback font will load, and that fallback font might have drastically different sizing than the custom font. The lines on the outside of the fallback font are now awkwardly placed, not centered like we wanted.

Of course, I don’t need to tell Chris (he was only trying to illustrate a technique and its shortcomings), but it bears mentioning whenever I get the chance: progressive enhancement is part of typography now. First, style text in a generic way (like, without flanking horizontal lines). Then, if the fonts you intend are active, use WebFont Loader (or Typekit font events) to follow up with rules that depend on the presence (and the dimensions) of those fonts.

It’s been too long since I did an economics blog post…

Way back when, I wrote fairly approvingly of the recommendations to simplify the income tax system. The idea being to more or less keep charging everyone the same tax rate, but to simplify the formulae from five different tax rates, a medicare levy, and a gradually disappearing low-income tax offset, to just a couple of different rates (one kicking in at $25k pa, one at $180k pa). The government’s adopted that in a half-hearted way — raising the tax free threshold to $18,200 instead of $25,000, and reducing but not eliminating the low-income tax offset. There’s still the medicare levy with its weird phase-in procedure, and there’s still four different tax rates. And there’s still all sorts of other deductions etc to keep people busy around tax time.

Anyway, yesterday I finally found out that the ATO publishes some statistics on how many people there are at various taxable income levels — table 9 of the 2009-2010 stats are the best I found, anyway. With that information, you can actually mess around with the tax rules and see what affect it actually has on government revenue. Or at least what it would have if there’d been no growth since 2010.

Anyway, by my calculations, the 2011-2012 tax rates would have resulted in about $120.7B of revenue for the government, which roughly matches what they actually reported receiving in that table ($120.3B). I think putting the $400M difference (or about $50 per taxpayer) down to deductions for dependants and similar that I’ve ignored seems reasonable enough. So going from there, if you followed the Henry review’s recommendations, dropping the Medicare levy (but not the Medicare surcharge) and low income tax offset, the government would end up with $117.41B instead, so about $3.3B less. The actual changes between 2011-2012 and 2012-2013 (reducing the LITO and upping the tax free threshold) result in $118.26B, which would have only been $2.4B less. Given there’s apparently a $12B fudge-factor between prediction and reality anyway, it seems a shame they didn’t follow the full recommendations and actually make things simpler.

Anyway, upping the Medicare levy by 0.5% seems to be the latest idea. By my count doing that and keeping the 2012-2013 rates otherwise the same would result in $120.90B, ie back to the same revenue as the 2011-2012 rates (though biased a little more to folks on taxable incomes of $30k plus, I think).

Personally, I think that’s a bit nuts — the medicare levy should just be incorporated into the overall tax rates and otherwise dropped, IMO, not tweaked around. And it’s not actually very hard to come up with a variation on the Henry review’s rates that both simplify tax levels, don’t increase tax on any individual by very much, and do increase revenue. My proposal would be: drop the medicare levy and low income tax offset entirely (though not the medicare levy surcharge or the deductions for dependants etc), and set the rates as: 35% for earnings above $25k, 40% for earnings above $80k, and 46.5% for earnings above $180k. That would provide government revenue of $120.92B, which is close enough to the same as the NDIS levy. It would cap the additional tax any individual pays to $2000 compared to 2012-13 rates, ie it wouldn’t increase the top marginal rate. It would decrease the tax burden on people with taxable incomes below $33,000 pa — the biggest winners would be people earning $25,000 who’d go from paying $1200 tax per year to nothing. The “middle class” earners between $35,000 and $80,000 would pay an extra $400-$500; higher income earners between $80,000 and $180,000 get hit between $500 and $2000, and anyone above $180,000 pays an extra $2,000. Everyone earning over about $34,000 but under about $400,000 would pay more tax than if the NDIS were just increased, everyone earning between $18,000 and $34,000 would be better off.

On a dollar basis, the comparison looks like:

Translating that to a percentage of income, it looks like:

Not pleasant, I’m sure, on the dual-$80k families in western Sydney who are just-scraping buy and all, but I don’t think it’s crazy unreasonable.

But the real win comes when you look at the marginal rates:

Rather than seven levels of marginal rate, there’s just three; and none of them are regressive — ie, you stop having cases like someone earning $30,000 paying 21% on their additional dollar of income versus someone earning $22,000 paying 29% on their extra dollar. At least philosophically and theoretically that’s a good thing. In practice, I’m not sure how much of a difference it makes:

There’s spikes at both the $80,000 and $35,000 points which involve 8% and 15% increases in the nominal tax rates respectively, which I think is mostly due to people transferring passive income to family members who either don’t work or have lower paid jobs — if you earn a $90,000 salary better to assign the $20,000 rental income from your units to your kid at uni and pay 15% or 30% tax on it, than assign it to yourself and pay 38%, especially if you then just deposit it back into your family trust fund either way. The more interesting spikes are those around the $20,000 and $30,000 points, but I don’t really understand why those are shaped the way they are, and couldn’t really be sure that they’d smooth out given a simpler set of marginal rates.

Anyway, I kind-of thought it was interesting that it’s not actually very hard to come up with a dramatically simpler set of tax rates, that’s both not overly punitive and gives about the same additional revenue as the mooted levy increase.

(As a postscript, I found it particularly interesting to see just how hard it is to get meaningful revenue increases by tweaking the top tax rate; there’s only about $33B being taxed at that rate, so you have to bump the rate by 3% to get a bump of $1B in overall revenue, which is either pretty punitive or pretty generous before you’re making any useful difference at all. I chose to leave it matching the current income level and rate; longer term, I think making the levels something like $25k, $100k, $200k, and getting the percentages to rounder figures (35%, 40%, 45%?) would probably be sensible. If I really had my druthers, I’d rather see the rate be 35% from $0-$100,000, and have the government distribute, say, $350 per fortnight to everyone — if you earn over $26k, that’s just the equivalent of a tax free threshold of $26k; if you don’t, it’s a helpful welfare payment, whether you’re a student, disabled, temporarily unemployed, retired, or something else like just wanting to take some time off to do charity work or build a business)

I send this report off to Linux Australia detailing our activities for the past few months. I’m posting it here for posterity, because we had a pretty good couple of months:

• Registrations have been open for a month now, we’re about to equal our record for Early Bird registrations, and should reach our limit of 80 Early Bird tickets this week (we actually sold our last 20 early bird tickets in one day. oops )
• Our CFP closed in early April, presentation submissions were up 25% — a record haul by quite some amount. International interest has increased a lot too, benefiting from on-the-ground promotion I did at PyCon US in March (many thanks to the Python Software Foundation for funding my trip).
• We’ve announced our first keynote speaker, Alex Gaynor (core Django, PyPy and CPython board member; PSF & DSF board member); our second keynote presenter is confirmed, and we’ll be announcing that in due course.
• Our programme committee met on Friday, we’ve selected our programme in its entirety. We have a great selection of local and international speakers lined up. Speaker acceptances will go out shortly.
• We’re finalising the details of our financial aid scheme. We hope that this will make PyCon Australia more accessible to people who could not otherwise afford to attend.

For those of you reading along at home, registrations are still open, and we really want you to come along. This is going to be the biggest PyCon Australia yet, and is going to feature one of the strongest programmes of any regional PyCon anywhere — all the details are up at http://2013.pycon-au.org/register/prices

In my first article, “Even Better In-Browser Mockups with Node.js,” I explained why Node.js makes designing applications easier and more efficient, and how to get started. Now it’s time to see your new design process in action.

Rather than figuring out all your requirements and API schemas just to design your comps with mockup content hard-coded and server interactions faked—only to throw it all away when you go back and implement things “for real”—you can use Node.js to skip the hard-coding and produce client-side code that’s ready for beta at the end of the design stage.

The process looks a lot like good ol’ designing in the browser, but with more JavaScript and an additional layer:

1. Design the layout and styling
2. Convert the markup to a JavaScript template
3. Create an initialization function
4. Create a simple Node.js server
5. Add a mockup data object to the server
6. Add server functions to serve static pages and JSON
7. Request and consume the JSON on the client

Sound daunting? Don’t worry. The first step takes approximately a zillion times longer than any of the others. So if you’ve already mastered the design, you’ll find the rest of these steps more than manageable.

In this walkthrough, we’ll build a feature for a mock art store. If you want to follow along at home, you can clone my GitHub repository. (If you need help installing, see the README, or just take a peek at the live demo—I’ll cover all the steps and code below.)

Creating templates

Once you have a solid design and the markup to accompany it, converting it to a template you can use for all examples is more efficient than creating duplicate markup for each one. The hard part’s over; you already thought about where data points would be used in the design when you created it. With those choices fresh in your mind, go back and mark up your HTML with data in whatever template language you prefer.

For my example, I’m using a store selling art prints. Here’s a snippet of my initial markup:

<h2>Two Acrobats with a Dog</h2> <h3>Pablo Picasso</h3> <img src="img/102.jpg" alt="Two Acrobats with a Dog" class="active" /> <ul class="info"> <li>8" x 11"</li> <li>acid-free paper</li> <li>suitable for matting</li> </ul> <span class="price">$49.99</span>

Think of your templates as places to define your requirements for both data and its formatting on the client side. If you can also reuse it for client-side rendering, that’s awesome—but that may not be relevant to your application. As long as you have good data, converting from one template language to another is trivial, so don’t agonize over which template engine to use.

You do need a template engine that will work in both the browser and Node.js, however. If you’re unsure, search for your template engine on GitHub and verify that there’s a guide to installing it via npm in the manual, as well as a minified script for use on the client. I prefer doT.js, so here’s that snippet again marked up to add data using doT:

<h2>{{=it.title}}</h2> <h3>{{=it.artist.name}}</h3> <img src="img/{{=it.id}}.jpg" alt="{{=it.title}}" class="active" /> <ul class="info"> {{~it.info :info_item}} <li>{{=info_item}}</li> {{~}} </ul> <span class="price">{{=it.price}}</span>

I like to save my templates in their own directory at the same level as my JavaScript directory, so now I store that as tmpl/detail.dot.

Initializing the client

Since we want to be able to use our templates in both Node and the browser, they need to be stored outside of the HTML and loaded and compiled when we open the page. To start, save the minified version of your template engine and add a script tag to your page to include it. Once that’s done, you can fetch the template, compile it, and then continue on with any other initialization work in your main JavaScript file. I’m using jQuery in my example, so my code looks like this:

var detailTmpl; $.when( $.get( "tmpl/detail.dot", function( tmpl ) { detailTmpl = doT.template( tmpl ); }, "text" ) ).then( init );

That mysterious init function? That’s where I’ll put any interactivity I want to add to my currently static mockup. For the moment I’m only creating one interaction, so my init function is pretty simple:

function init() { $( "div.content" ).on( "click", "div.result", showDetail ); }

This code can be made much more elegant using Require.js with its text plugin. That’s beyond the scope of this demo, but I highly encourage it for production.

We’ll handle template rendering in showDetail(), but we have to add a server and data store before writing that function, since right now we lack any data to render.

Creating a server

If I reload my page now and open the browser console, I get a JavaScript error. That’s because I’m trying to load my template via an XMLHttpRequest (XHR) on a page being served from the file system, in violation of the same origin policy. I can’t even check that my template works until the page is served properly (i.e., from a server).

To whip up a simple Node server that allows me to run my XHRs, I do a few things:

• Move all my existing assets into a new subdirectory called public
• Open my terminal or command line to my working directory and run npm install express
• Add a server.js file to the working directory

We could write everything from scratch, of course, but it’s more work than is necessary for a basic server. The Express framework provides a number of abstractions of server and application concepts. For the initial version of the server, the only one we’ll need is its ability to serve static resources. We can use it by adding four lines of code to server.js:

var express = require( "express" ), app = express(); app.use( express.static( __dirname + "/public" ) ); app.listen( 3000 );

Once you start your server by typing node server.js in your open terminal or command line, you can view your page at http://localhost:3000 (adding a filename if necessary), and the error related to loading the template ought to disappear.

Adding server-side data

While it’s certainly nice to be able to use XHRs, we’re creating the Node server to use it as a representation of the real server—and real servers store data. Though it’s not hard to create a data store that works with a Node server, it’s even less difficult to create one big object literal. For a mockup, that’s all we really need. One of the goals here is to define the data objects we need to support in our new design, so the format of this object can be determined by the template we just added. For my example, I need an object structured something like this:

var products = { "102": { id: 102, title: "Two Acrobats with a Dog", artist: { name: "Pablo Picasso" }, price: "$49.99", info: [ "8\" x 11\"", "acid-free paper", "suitable for matting" ] } };

Note that products could just as easily be an array, but I want to be able to quickly find my products—once I have more than one in my fake data store—by ID. Aside from that little twist, the data look exactly like the content hard-coded in my original HTML. If I want to add more data, including things that might break the layout in unpredictable ways, I can just copy this structure and make substitutions. Well, almost.

Returning data from the server

If you’ve dealt with other server-side frameworks, creating endpoints for XHRs might seem intimidating, but Express makes it really easy. We don’t need any special setup to define a server endpoint as a target for asynchronous requests. All we have to do is define the path on the server where we want to accept requests and a callback. The callback receives a request object (for doing things like getting passed-in data) and a response object (for defining what we return to the client). To return the data in my products object, I add a few lines of code at the bottom of server.js:

app.get( "/detail/:id", function( req, res ) { res.send( products[ req.params.id ] ); }); app.listen( 3000 );

See? Easy. If I restart my server and go to http://localhost:3000/detail/102, I should see my object data. To break down what’s going on with the ID in the path, we’ve named the data at that position in the path "id" with the :id bit, and it then becomes available as a property of req.params.

The names and positions of parameters are up to us, and if our path were super complex, we could also use regular expressions to split out multiple pieces of data. Express also gives us the option of accepting data from the query string or from a POST. Of all the pieces we’re creating, however, the paths are the most likely to change in production, so it’s to our advantage to keep them as readable as possible.

Besides sending pure data to the client, we also want to be able to send rendered HTML, in case a user is linked directly to a product detail or doesn’t have JavaScript available. We might also want HTML for our own consumption via XHR, if we find that client-side rendering is slowing us down. So we add a second endpoint below the one we just created to do that:

app.get( "/product/:id", function( req, res ) { res.render( "detail", products[ req.params.id ] ); });

For simplicity’s sake, and because the first path served JSON for an overlay while this provides a full page, I’ve used a different pathname, but kept the same pattern. This time, instead of the response’s send function, I use render(). Express provides some magic to make template rendering work out of the box, but since I’m using doT instead of Jade (the default template engine of Express), I have to do some additional setup.

First I have to go back to the terminal or command line, stop my Node server, and install my template engine using npm install doT and the consolidate module (which provides Express compatibility for a number of popular template engines) using npm install consolidate. Now I’ve got both of those in my node_modules directory and can use them in server.js.

Since doT (and probably your template engine of choice, as well) is accessed through consolidate, consolidate is the only additional module I need to require at the top of server.js:

var express = require( "express" ), app = express(), cons = require( "consolidate" );

I want to continue serving some of my other pages statically, so I add my template configuration stuff below the existing app.use line in my code:

app.use( express.static( _dirname + "/public" ) ); app.engine( "dot", cons.dot ); app.set( "view engine", "dot" ); app.set( "views", _dirname + "/public/tmpl" );

Those three new lines set doT (as exposed by consolidate) as the view engine, register files ending in .dot as templates, and tell Express to look in /public/tmpl for templates to use. So when Node sees res.render( "detail", { ... } ), it knows to expand "detail" to /public/tmpl/detail.dot and render it as a doT template. Now I can restart my server, go to http://localhost:3000/product/102, and see my template rendered statically, without creating a separate server-side file.

Fetching dynamic data

Our template now works as a static page, but there’s still one more step to get our mockup populated with the data from the server. Remember the showDetail function from our main client-side script? It’s time to flesh that out.

In my simple example, the overlay my template will populate already exists as a hidden div on the main page, and it appears when the user clicks a div containing a summary of the content. This div has a data attribute storing the ID of the product that corresponds to the key and id property in my server-side data object. Once that click event happens and showDetail() is called, I just need to do this:

function showDetail( e ) { var id = $( this ).data( "id" ); $.get( "detail/" + id, function( info ) { $( "div.detail" ).html( detailTmpl( info ) ); $( "div.detail" ).show(); } }

The path above is the same one I defined in server.js. If you chose a different name for yours, use that name here on the client. When I receive the data object from the server, I pass it to detailTmpl(), the compiled version of my template. The result of the detailTmpl function is the HTML to populate my overlay.

Onward

So there you have it! A mockup that mimics the interactions it will have with its production server precisely on the client, without the need for hard-coded data or temporary workarounds. Despite the simple exercise, the process I’ve outlined accomplishes a good deal of the setup necessary to create other workflows that require server interactions. For instance, I can fill my fake data store with more products and use that to render the initial page that triggers my overlay without having to revisit my mockup data, and my application will show the correct values in any view I add to it.

If you’d like to explore beyond just serving HTML and JSON, consider adding in Socket.io to allow real-time interaction for multiple clients or Require.js to manage your assets on the client. You could also move your CSS into templates and serve different builds of your site for different browsers or devices. Your mockup can be as sophisticated and reflect as many of its production requirements as you choose. At the end, the lion’s share of your client-side code is done and ready to use.

Designing in the browser has all sorts of benefits, like producing more accurate, comprehensive results and removing the extra step of converting from image file to markup and CSS. But even sites designed in a browser still require pasting in content, faking interactions with the server, and creating placeholder JavaScript that isn’t usable on the live site.

Wouldn’t it be nice if we could go from just designing layouts and interactions to designing the whole client side of the application during the same process?

This is where Node comes in.

Node.js is a server-side JavaScript platform. It isn’t a web server, but it allows you to easily create one. It also lets you create utilities that run on web servers, like setup and minification utilities and general-purpose command line tools.

Node started in 2009 and generated considerable interest, probably because it gave JavaScript developers an opportunity to write server-side code even if they lacked a server-side background. It didn’t hurt that Chrome had established a reputation for being solid and fast, and Node used its V8 engine.

Today, it’s possible to run production servers on Node, and many people are doing so successfully. Taking it that far, however, is an investment. The Node project, and all the community-created modules that make it so awesome, is still a moving target. But even if you’re not ready to write and launch entire sites with Node, it’s plenty stable enough to use as a development tool.

It’s JavaScript, so if you can wire up a jQuery event handler, you can write a web server. It’s lightweight, so you can run it on your laptop and keep streaming music in the background. It’s dead simple to download, set up, and build in, so you don’t need the esoteric knowledge of an IT support person to get going with it. And the payoff is that instead of mockups and hard-coded data, you can design a set of client-side assets, page templates, and data schemas that are ready to launch to production.

Getting started

Installing Node locally for the most common environments is a piece of cake. You can download installers that include Node as well as npm, its package manager, from the project’s site. Installing it on a remote server is not quite that easy, but good documentation is available to help you out. After running through the installation process, you should be able to go to your terminal or command line and test it out.

If you don’t tell Node to run a specific file, you get a Read-Eval-Print Loop, or REPL. If you type node in your terminal or command prompt, you can begin to execute arbitrary JavaScript. For example, after starting the REPL, type var a = 9;. The REPL will respond with undefined. Now type a * 3 (or any other number) and it will respond with the correct result. You can make things more interesting by defining a function and then calling it:

> function sayHello( name ) { return “Hello, “ + name; } undefined > sayHello( “A List Apart” ); ‘Hello, A List Apart’

To break out of the REPL, or end any other Node execution (like a running web server), press Ctrl+C. In the case of the REPL, you’ll need to press it twice to confirm.

While it’s nice to know Node can perform basic arithmetic and string concatenation, its value to us as developers is in running programs. You can see an example of one such program, a basic web server, on the project’s homepage. They suggest creating a file called example.js with this code:

var http = require(’http’); http.createServer(function (req, res) { res.writeHead(200, {’Content-Type’: ‘text/plain’}); res.end(’Hello World\n’); }).listen(1337, ‘127.0.0.1’); console.log(’Server running at http://127.0.0.1:1337/’);

This makes use of only one module, the core module http. As you can probably guess, the http module contains all the basic stuff you need to serve a site over HTTP. Node contains a tightly edited collection of core modules that provide things like event handlers, file system access, and abstractions for various network protocols. But just as you probably use a JavaScript library or framework to speed up and bulletproof development on the client side, for Node development beyond a simple "Hello World" you generally add other non-core modules using npm.

The http module does contain a createServer function, though, which is all you need to create a bare-bones web server. In the code above, once the server has been created, it listens to port 1337 on your local machine. When it receives a request, it sends back a text response.

One thing to note is that the work here is done in event handlers, as are most things in Node. The callback in createServer() handles a connection event, which occurs every time a new client contacts the server. To start this server, type node example.js in your terminal, and then open a browser to http://127.0.0.1:1337. This triggers the connection event, and you should see the message in the callback.

To obtain any serious value from a Node server—even one not intended to ever go to production—it’s best to get familiar with the modules in npm. There are thousands available, but those you’d need to create a basic web application are some of the oldest and most stable, so don’t feel obligated to research them all before getting started. One that definitely comes in handy for designing an application is Express, an uncomplicated web application framework.

If you’re accustomed to installing open source projects by cloning a GitHub repository or downloading a zip file, you’ll probably enjoy npm. To install Express with npm, for example, go to your terminal or command line and type npm install express. As long as you’re online and have permission to write to your machine, this fetches all the code and assets Express needs to run, as well as any modules it has as dependencies. The first time you run npm from your working directory, all these elements will end up in a new node_modules subdirectory. Now the module is ready to be used in Node programs the same way we used http, via the require function.

Designing applications

The ideal use case for designing your application with Node is a single-page application in which the server mostly provides data, but Node is still useful for a more traditional site. Of course, you want to begin development with requirements defined as precisely as possible, but implementation tends to expose requirements you hadn’t considered, and some of those can have a considerable impact on your timeline. Even in a server-driven application where it may not be possible to reuse data structures and templates as-is, creating client-only versions helps test your assumptions about the data you need and how you’ll use it.

If you’re developing a single-page app, the justification is much easier. You’ll need to think about optimizing your communication with the server to require as few requests as possible, which means knowing how data should be packaged up by each server endpoint and how you’ll cache the data on receipt (if at all).

An advantage of having JavaScript on the server is that templates can be rendered by the same template engine on either the client or server side. This allows you to experiment on both sides and optimize for your situation. It’s also a timesaver to render the templates with JavaScript objects and consider only the way data must eventually be grouped (not how it’s stored in a database). Designing these groupings of data is the bulk of the work we can do with Node toward the end of what we traditionally consider application design.

Piecing together each page from disparate parts from all over the server is messy for an application in any language. Instead, whatever renders a page should have clear dependencies, and the result of each page or data request should combine those dependencies into a cohesive and sensibly organized unit.

If you’ve worked in a server-side framework in which a page or view is tied to a single object or model, and where additional data is imported and exposed in a different way, you probably understand how the alternative gets to be a nuisance. You’re probably also aware that the solution is a good view-model whose data is defined by each view, not the models that feed it. With this exercise, we aim to map out what goes in those view-models.

Using templates

There’s a strong likelihood that your production server does not run JavaScript, so you may end up converting templates you produce in this design phase. You could attempt to mitigate this by choosing a template engine like Mustache with existing parsers for a huge list of languages. Or you might choose one with minimal logic available (I find that the only things I want for a truly flexible template are conditionals, loops, and partials) and the option of changing the delimiters around the template tags to agree with your server template language. I’d argue that the process of getting all the data correctly placed in your HTML is a lot more difficult than doing a find and replace on the end result, so creating a template in JavaScript that you can use easily is time well spent even if it can’t be parsed by your production server.

You could choose to design the UI of your pages using hard-coded mockup data first and add the template tags afterward, or you could start with a template and some mockup data ready to go in your Node server. Even though it’s an extra step, I find the former easier, because the latter tends to require extra shifting of the mockup data. Starting with hard-coded data lets me examine the finished mockup and see what kinds of "objects" are present (e.g., a user, an item for sale, or a status update). That helps me create a flexible object hierarchy in my data more easily. But you may be naturally amazing at creating object hierarchies, so, by all means, do what you feel.

Wherever you begin, hammering out your templates should give you an indication of which parts of each page are dynamic and which data each requires. If subsections of your pages are rendered separately because they’re reused on different parent pages or because they’re also rendered by the client, converting your markup to templates also allows you to find the right balance between never repeating code and having absurdly tiny templates.

Real fake server interactions

If you’ve created high-fidelity wireframes that run in a browser, you know the annoyance of having only parts of a page be interactive, since every click means having to create a new view (even if you have a series of items that share the same behavior when clicked). You also know about copying and pasting the same data into multiple places, and updating each of them separately if the manner of presenting data should change. Designing your app with a server behind it removes those frustrations.

With the support of a server, it’s not a problem if the same data shows up in different displays all over the workflow you’re designing. Since the data lives on your Node server, you can write it once and reuse it as many ways as you like. You still have to consider how you’ll move it from server to client, though. When a user clicks on one of many items in a list, will she be taken to a new page, or will more data appear inline? Is the former the non-JavaScript fallback for the latter? Working that out for your app will tell you which endpoints the server needs, and which parameters need to be sent to it in query strings, form posts, or URLs. It’ll also help define the API for those requests, telling anyone who might work on your production server which keys you expect to correspond to which pieces of data.

Having a server to work with is especially nice if you’re in the business of making asynchronous requests. Obviously, you can get your mockup data, which is excellent, but you can also lazy-load assets like templates or stylesheets to consume that data. Testing the process of getting data and assets to the client validates your assumptions about not only the way you’re packaging them, but how you’re storing and structuring them. And, of course, it means a lot less wasted client-side JavaScript.

A mockup you can use

The end result of all this should be that you’ve moved all the mockup pieces out of your client-side JavaScript and HTML. You have a Node server that might not match your production framework, but does have clear definitions of everything the client side expects to exist—possibly even all viewable in a single file. You have templates and client-side requests that may require substitutions, but also separate your data from everything else and are at minimum easily convertible to whatever format is needed for production.

Could you do the same with any other server under the sun? Absolutely. But if you already know JavaScript and aren’t aiming to become a server-side developer, it makes sense to use the skills you already have. Node makes that pretty easy, while also letting you dig as deeply as you want into more complex servers, should your ambitions change. It’s simple to get going and flexible to extend, making Node an awesome tool for designing applications.

Ready to take your new Node skills for a spin? In “Node at Work: A Walkthrough,” I’ll take you through a live demo, and get specific about how to refine your own mockups.

Some of the response to yesterday's post (and just about every time I talk about 'picking yourself') is predictable, sad and frustrated/frustrating. I'd have a lot easier time if I was in the business of telling people how to get picked, if I was working to uncover the proven, secret, time-saving tricks guaranteed to get you noticed...

"It's my turn."

I know you worked hard on paying your dues, on building your skills and in being next. We all know that. But that doesn't mean that the picking system is going to work when you need it to. It's not going to get you into the famous college of your dreams, or featured in a PR blitz or published by Knopf.

"The only way for me to do what I love (play the flute, trade stocks, volunteer with kids, spread the word about my cause...) is to get picked."

This is, to be really frank, nonsense. If, for example, you graduated from the Eastman School of Music, there are many ways to play the cello that don't involve auditioning for an orchestra. You can play house concerts, you can play on the street, you can build your own tribe, you can organize your own ad hoc orchestra. None of these things are official, none of these things are automatic, none of these things are guaranteed. So?

If you want to devote your work and your efforts to getting picked, that's your choice, and more power to you. But I think it's dangerous to start with the assumption that you have no choice.

I heard from a writer who invoked the Josh Bell story about the famous violinist who is treated shabbily by the mass commuter audience, because of course, to them, he's not famous at all. This is supposed to be proof that it matters if you're famous (picked) as opposed to good. In Josh's case, he's both. But if you can't be picked to be famous, at least you can become remarkable.

If you can't get invited to the main stage of TED, then do a TEDx talk, and make your talk so good it can't help but spread. And if you can't get invited to a TEDx, then start your own TED-like event. And if you can't figure out how to organize people, connect them and lead them, perhaps you could focus more energy and risk on that very skill.

If you've built an app that won't be profitable unless you're featured on the front page of iTunes, the problem isn't with the front page of iTunes, the problem is with the design of your app. Ideas built to spread are more likely to spread.

If your plan requires getting picked and you're not getting picked, you need a new plan. I'm betting it will turn out far better in the end, but yes, indeed, I understand that it's harder than being anointed. Your talent deserves the shift in strategy that will let you do your best work.

The problem isn't that it's impossible to pick yourself. The problem is that it's frightening to pick yourself. It's far easier to put your future into someone else's hands than it is to slog your way forward, owning the results as you go.

Grateful Dead vs. Bay City Rollers.

Today I downloaded Fedora 19 alpha to give it a spin. Some quick notes.

You can get MySQL by asking for the package community-mysql-server. This is 5.5.31. If you ask for stock “mysql” (i.e. yum install mysql-server), you automatically get MariaDB 5.5.30 (mariadb-server).

Fedora 19 runs systemd, so there is no longer /etc/init.d/mysql to start/stop/restart. So just do systemctl enable mysqld.service. Then perform: systemctl start mysqld.service. Replace start with: stop/status too. You can disable it too if you want.

MariaDB 10.0.2 compiles cleanly on Fedora 19 with gcc-4.8. Just perform: yum install bzr gzip tar gcc gcc-c++ make libtool bison ncurses-devel zlib-devel automake autoconf cmake. Get the source code (I just downloaded it). Do BUILD/compile-pentium64-max. Wait. Run make test. Enjoy. Refer to build environment setup, generic build instructions.

If running in a VM, set aside 15GB to ensure you always have sufficient space (I personally use 20GB as I like to test various upgrade scenarios too.

Related posts:

1. Using MariaDB on CentOS 6
2. workbench-5.1.1-alpha on Fedora 9
3. Eric Raymond says: “Goodbye, Fedora”

First of all, yes I'm a general Greens / Labour supporter so it's not surprising that I like the NBN. I work in a tech field (specifically datacomms) that also generally supports the NBN. I have plenty of reasons to support the NBN from those, but here's the "how it affects me" ones.



I live in inner-city Sydney, specifically Ultimo, literally in the afternoon shadow of (probably) the most wired building in the country, the GlobalSwitch datacenter, yet on a bad day I get 1Mb or so through my Internet connection.



I do have two basic options for a home Internet connection where I live, both through Telstra's infrastructure, either their HFC cable network, or classic copper. As Telstra (last time I tried) were unable to sell a real business-class service on HFC I use ADSL, and since I consider IPv6 support a requirement, I use Internode (there's other reasons I use them as well, but IPv6 sadly is still hard to get from other providers). Due to the location of my apartment 3/4g services aren't an option even if they were fast enough (or affordable).



Sadly the copper in Ultimo is in a sad state with high cross-talk and attenuation. I suspect this is likely due to passing underground through Wentworth Park[1] which is only a meter or two above sea level so water damage is highly likely.



Even on a good day I only barely sustain ~8Mb down, ~1Mb up, with no disconnects, on a bad day it can be as low as 1Mb down, with multiple disconnects per hour.



It's the last point I particularly care about, regular disconnects make the service unreliable and are the aspect that is most irritating.



Speed, although people often want it is of limited value to me, once I can do ~20Mb down and ~5Mb up that's enough for me, handles file uploads fast enough and offers a nice buffer for video conferencing (yes I really do join video conferences from home). I suspect I will subscribe to a 50/20 plan if/when NBN is available in my area. In theory NBNco should have commenced construction in the Pyrmont/Glebe/Ultimo/Haymarket area this month (per their map), but I'm not aware of anyone who has been contacted to confirm this.



Because I'm in an apartment complex it's likely that the coalition's plan would have a node in the basement (there's already powered HFC amps in the basement car parks so it wouldn't be unprecedented), this matches some versions of the NBN plan, although the current plan (last I saw) is fibre to each apartment. Once a node is within a building cable damage due to water is unlikely and I'd probably be able to sustain a decent service, but if I ever moved to a townhouse then I'd be back to potentially dodgy underground cable.



I don't believe forcing Telstra & Optus to convert their HFC networks into wholesale capable networks is a sensible idea for a variety of reasons, the two major being would they still be able to send TV, and the need to split the HFC segments into much smaller sections to be able to sustain throughput. Even with the current low take-up rate of cable Internet I know many people in Melbourne who find their usable throughput massively degrades at peak times, something that would almost certainly get worse, not better, if HFC became the sole method of high speed Internet in a region.



I also still maintain my server and Internet service at my old place in Melbourne, and will get at least 50/20 there, should it ever be launched, when I first got ADSL2+ service I managed to sustain 24/2 but now only 12/1, presumably due to degrading lines, which makes me wonder about how fast and reliable a VDSL based FTTN would actually be.



1: The only real alternate path would add a kilometre or more of cable to avoid that low lying area.