You are here


Russell Coker: Converting Mbox to Maildir

Planet Linux Australia - 4 hours 3 min ago

MBox is the original and ancient format for storing mail on Unix systems, it consists of a single file per user under /var/spool/mail that has messages concatenated. Obviously performance is very poor when deleting messages from a large mail store as the entire file has to be rewritten. Maildir was invented for Qmail by Dan Bernstein and has a single message per file giving fast deletes among other performance benefits. An ongoing issue over the last 20 years has been converting Mbox systems to Maildir. The various ways of getting IMAP to work with Mbox only made this more complex.

The Dovecot Wiki has a good page about converting Mbox to Maildir [1]. If you want to keep the same message UIDs and the same path separation characters then it will be a complex task. But if you just want to copy a small number of Mbox accounts to an existing server then it’s a bit simpler.

Dovecot has a script to convert folders [2].

cd /var/spool/mail mkdir -p /mailstore/ for U in * ; do ~/ -s $(pwd)/$U -d /mailstore/$U done

To convert the inboxes shell code like the above is needed. If the users don’t have IMAP folders (EG they are just POP users or use local Unix MUAs) then that’s all you need to do.

cd /home for DIR in */mail ; do U=$(echo $DIR| cut -f1 -d/) cd /home/$DIR for FOLDER in * ; do ~/ -s $(pwd)/$FOLDER -d /mailstore/$U/.$FOLDER done cp .subscriptions /mailstore/$U/ subscriptions done

Some shell code like the above will convert the IMAP folders to Maildir format. The end result is that the users will have to download all the mail again as their MUA will think that every message had been deleted and replaced. But as all servers with significant amounts of mail or important mail were probably converted to Maildir a decade ago this shouldn’t be a problem.

Related posts:

  1. Why Cyrus Sucks I’m in the middle of migrating a mail server away...
  2. Moving a Mail Server Nowadays it seems that most serious mail servers (IE mail...
  3. Mail Server Training Today I ran a hands-on training session on configuring a...
Categories: thinktime

Linux Users of Victoria (LUV) Announce: LUV Main October 2017 Meeting

Planet Linux Australia - 6 hours 3 min ago
Start: Oct 3 2017 18:30 End: Oct 3 2017 20:30 Start: Oct 3 2017 18:30 End: Oct 3 2017 20:30 Location:  Mail Exchange Hotel, 688 Bourke St, Melbourne VIC 3000 Link:


Tuesday, October 3, 2017
6:30 PM to 8:30 PM
Mail Exchange Hotel
688 Bourke St, Melbourne VIC 3000


  • TBA

Mail Exchange Hotel, 688 Bourke St, Melbourne VIC 3000

Food and drinks will be available on premises.

Linux Users of Victoria is a subcommittee of Linux Australia. October 3, 2017 - 18:30
Categories: thinktime

Linux Users of Victoria (LUV) Announce: LUV October 2017 Workshop

Planet Linux Australia - 6 hours 3 min ago
Start: Oct 21 2017 12:30 End: Oct 21 2017 16:30 Start: Oct 21 2017 12:30 End: Oct 21 2017 16:30 Location:  Infoxchange, 33 Elizabeth St. Richmond Link:

There will also be the usual casual hands-on workshop, Linux installation, configuration and assistance and advice. Bring your laptop if you need help with a particular issue. This will now occur BEFORE the talks from 12:30 to 14:00. The talks will commence at 14:00 (2pm) so there is time for people to have lunch nearby.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.) Late arrivals, please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

October 21, 2017 - 12:30
Categories: thinktime

sthbrx - a POWER technical blog: Stupid Solutions to Stupid Problems: Hardcoding Your SSH Key in the Kernel

Planet Linux Australia - 16 hours 6 min ago
The "problem"

I'm currently working on firmware and kernel support for OpenCAPI on POWER9.

I've recently been allocated a machine in the lab for development purposes. We use an internal IBM tool running on a secondary machine that triggers hardware initialisation procedures, then loads a specified skiboot firmware image, a kernel image, and a root file system directly into RAM. This allows us to get skiboot and Linux running without requiring the usual hostboot initialisation and gives us a lot of options for easier tinkering, so it's super-useful for our developers working on bringup.

When I got access to my machine, I figured out the necessary scripts, developed a workflow, and started fixing my code... so far, so good.

One day, I was trying to debug something and get logs off the machine using ssh and scp, when I got frustrated with having to repeatedly type in our ultra-secret, ultra-secure root password, abc123. So, I ran ssh-copy-id to copy over my public key, and all was good.

Until I rebooted the machine, when strangely, my key stopped working. It took me longer than it should have to realise that this is an obvious consequence of running entirely from an initrd that's reloaded every boot...

The "solution"

I mentioned something about this to Jono, my housemate/partner-in-stupid-ideas, one evening a few weeks ago. We decided that clearly, the best way to solve this problem was to hardcode my SSH public key in the kernel.

This would definitely be the easiest and most sensible way to solve the problem, as opposed to, say, just keeping my own copy of the root filesystem image. Or asking Mikey, whose desk is three metres away from mine, whether he could use his write access to add my key to the image. Or just writing a wrapper around sshpass...

One Tuesday afternoon, I was feeling bored...

The approach

The SSH daemon looks for authorised public keys in ~/.ssh/authorized_keys, so we need to have a read of /root/.ssh/authorized_keys return a specified hard-coded string.

I did a bit of investigation. My first thought was to put some kind of hook inside whatever filesystem driver was being used for the root. After some digging, I found out that the filesystem type rootfs, as seen in mount, is actually backed by the tmpfs filesystem. I took a look around the tmpfs code for a while, but didn't see any way to hook in a fake file without a lot of effort - the tmpfs code wasn't exactly designed with this in mind.

I thought about it some more - what would be the easiest way to create a file such that it just returns a string?

Then I remembered sysfs, the filesystem normally mounted at /sys, which is used by various kernel subsystems to expose configuration and debugging information to userspace in the form of files. The sysfs API allows you to define a file and specify callbacks to handle reads and writes to the file.

That got me thinking - could I create a file in /sys, and then use a bind mount to have that file appear where I need it in /root/.ssh/authorized_keys? This approach seemed fairly straightforward, so I decided to give it a try.

First up, creating a pseudo-file. It had been a while since the last time I'd used the sysfs API...


The sysfs pseudo file system was first introduced in Linux 2.6, and is generally used for exposing system and device information.

Per the sysfs documentation, sysfs is tied in very closely with the kobject infrastructure. sysfs exposes kobjects as directories, containing "attributes" represented as files. The kobject infrastructure provides a way to define kobjects representing entities (e.g. devices) and ksets which define collections of kobjects (e.g. devices of a particular type).

Using kobjects you can do lots of fancy things such as sending events to userspace when devices are hotplugged - but that's all out of the scope of this post. It turns out there's some fairly straightforward wrapper functions if all you want to do is create a kobject just to have a simple directory in sysfs.

#include <linux/kobject.h> static int __init ssh_key_init(void) { struct kobject *ssh_kobj; ssh_kobj = kobject_create_and_add("ssh", NULL); if (!ssh_kobj) { pr_err("SSH: kobject creation failed!\n"); return -ENOMEM; } } late_initcall(ssh_key_init);

This creates and adds a kobject called ssh. And just like that, we've got a directory in /sys/ssh/!

The next thing we have to do is define a sysfs attribute for our authorized_keys file. sysfs provides a framework for subsystems to define their own custom types of attributes with their own metadata - but for our purposes, we'll use the generic bin_attribute attribute type.

#include <linux/sysfs.h> const char key[] = "PUBLIC KEY HERE..."; static ssize_t show_key(struct file *file, struct kobject *kobj, struct bin_attribute *bin_attr, char *to, loff_t pos, size_t count) { return memory_read_from_buffer(to, count, &pos, key, bin_attr->size); } static const struct bin_attribute authorized_keys_attr = { .attr = { .name = "authorized_keys", .mode = 0444 }, .read = show_key, .size = sizeof(key) };

We provide a simple callback, show_key(), that copies the key string into the file's buffer, and we put it in a bin_attribute with the appropriate name, size and permissions.

To actually add the attribute, we put the following in ssh_key_init():

int rc; rc = sysfs_create_bin_file(ssh_kobj, &authorized_keys_attr); if (rc) { pr_err("SSH: sysfs creation failed, rc %d\n", rc); return rc; }

Woo, we've now got /sys/ssh/authorized_keys! Time to move on to the bind mount.


Now that we've got a directory with the key file in it, it's time to figure out the bind mount.

Because I had no idea how any of the file system code works, I started off by running strace on mount --bind ~/tmp1 ~/tmp2 just to see how the userspace mount tool uses the mount syscall to request the bind mount.

execve("/bin/mount", ["mount", "--bind", "/home/ajd/tmp1", "/home/ajd/tmp2"], [/* 18 vars */]) = 0 ... mount("/home/ajd/tmp1", "/home/ajd/tmp2", 0x18b78bf00, MS_MGC_VAL|MS_BIND, NULL) = 0

The first and second arguments are the source and target paths respectively. The third argument, looking at the signature of the mount syscall, is a pointer to a string with the file system type. Because this is a bind mount, the type is irrelevant (upon further digging, it turns out that this particular pointer is to the string "none").

The fourth argument is where we specify the flags bitfield. MS_MGC_VAL is a magic value that was required before Linux 2.4 and can now be safely ignored. MS_BIND, as you can probably guess, signals that we want a bind mount.

(The final argument is used to pass file system specific data - as you can see it's ignored here.)

Now, how is the syscall actually handled on the kernel side? The answer is found in fs/namespace.c.

SYSCALL_DEFINE5(mount, char __user *, dev_name, char __user *, dir_name, char __user *, type, unsigned long, flags, void __user *, data) { int ret; /* ... copy parameters from userspace memory ... */ ret = do_mount(kernel_dev, dir_name, kernel_type, flags, options); /* ... cleanup ... */ }

So in order to achieve the same thing from within the kernel, we just call do_mount() with exactly the same parameters as the syscall uses:

rc = do_mount("/sys/ssh", "/root/.ssh", "sysfs", MS_BIND, NULL); if (rc) { pr_err("SSH: bind mount failed, rc %d\n", rc); return rc; }

...and we're done, right? Not so fast:

SSH: bind mount failed, rc -2

-2 is ENOENT - no such file or directory. For some reason, we can't find /sys/ssh... of course, that would be because even though we've created the sysfs entry, we haven't actually mounted sysfs on /sys.

rc = do_mount("sysfs", "/sys", "sysfs", MS_NOSUID | MS_NOEXEC | MS_NODEV, NULL);

At this point, my key worked!

Note that this requires that your root file system has an empty directory created at /sys to be the mount point. Additionally, in a typical Linux distribution environment (as opposed to my hardware bringup environment), your initial root file system will contain an init script that mounts your real root file system somewhere and calls pivot_root() to switch to the new root file system. At that point, the bind mount won't be visible from children processes using the new root - I think this could be worked around but would require some effort.


The final piece of the puzzle is building our new code into the kernel image.

To allow us to switch this important functionality on and off, I added a config option to fs/Kconfig:

config SSH_KEY bool "Andrew's dumb SSH key hack" default y help Hardcode an SSH key for /root/.ssh/authorized_keys. This is a stupid idea. If unsure, say N.

This will show up in make menuconfig under the File systems menu.

And in fs/Makefile:

obj-$(CONFIG_SSH_KEY) += ssh_key.o

If CONFIG_SSH_KEY is set to y, obj-$(CONFIG_SSH_KEY) evaluates to obj-y and thus ssh-key.o gets compiled. Conversely, obj-n is completely ignored by the build system.

I thought I was all done... then Andrew suggested I make the contents of the key configurable, and I had to oblige. Conveniently, Kconfig options can also be strings:

config SSH_KEY_VALUE string "Value for SSH key" depends on SSH_KEY help Enter in the content for /root/.ssh/authorized_keys.

Including the string in the C file is as simple as:

const char key[] = CONFIG_SSH_KEY_VALUE;

And there we have it, a nicely configurable albeit highly limited kernel SSH backdoor!


I've put the full code up on GitHub for perusal. Please don't use it, I will be extremely disappointed in you if you do.

Thanks to Jono for giving me stupid ideas, and the rest of OzLabs for being very angry when they saw the disgusting things I was doing.

Comments and further stupid suggestions welcome!

Categories: thinktime

Your fast car

Seth Godin - Fri 22nd Sep 2017 18:09
Right there, in your driveway, is a really fast car. And here are the keys. Now, go drive it. Right there, in your hand, is a Chicago Pneumatics 0651 hammer. You can drive a nail through just about anything with...        Seth Godin
Categories: thinktime

sthbrx - a POWER technical blog: NCSI - Nice Network You've Got There

Planet Linux Australia - Fri 22nd Sep 2017 10:09

A neat piece of kernel code dropped into my lap recently, and as a way of processing having to inject an entire network stack into by brain in less-than-ideal time I thought we'd have a look at it here: NCSI!

NCSI - Not the TV Show

NCSI stands for Network Controller Sideband Interface, and put most simply it is a way for a management controller (eg. a BMC like those found on our OpenPOWER machines) to share a single physical network interface with a host machine. Instead of two distinct network interfaces you plug in a single cable and both the host and the BMC have network connectivity.

NCSI-capable network controllers achieve this by filtering network traffic as it arrives and determining if it is host- or BMC-bound. To know how to do this the BMC needs to tell the network controller what to look out for, and from a Linux driver perspective this the focus of the NCSI protocol.

Hi My Name Is 70:e2:84:14:24:a1

The major components of what NCSI helps facilitate are:

  • Network Controllers, known as 'Packages' in this context. There may be multiple separate packages which contain one or more Channels.
  • Channels, most easily thought of as the individual physical network interfaces. If a package is the network card, channels are the individual network jacks. (Somewhere a pedant's head is spinning in circles).
  • Management Controllers, or our BMC, with their own network interfaces. Hypothetically there can be multiple management controllers in a single NCSI system, but I've not come across such a setup yet.

NCSI is the medium and protocol via which these components communicate.

The interface between Management Controller and one or more Packages carries both general network traffic to/from the Management Controller as well as NCSI traffic between the Management Controller and the Packages & Channels. Management traffic is differentiated from regular traffic via the inclusion of a special NCSI tag inserted in the Ethernet frame header. These management commands are used to discover and configure the state of the NCSI packages and channels.

If a BMC's network interface is configured to use NCSI, as soon as the interface is brought up NCSI gets to work finding and configuring a usable channel. The NCSI driver at first glance is an intimidating combination of state machines and packet handlers, but with enough coffee it can be represented like this:

Without getting into the nitty gritty details the overall process for configuring a channel enough to get packets flowing is fairly straightforward:

  • Find available packages.
  • Find each package's available channels.
  • (At least in the Linux driver) select a channel with link.
  • Put this channel into the Initial Config State. The Initial Config State is where all the useful configuration occurs. Here we find out what the selected channel is capable of and its current configuration, and set it up to recognise the traffic we're interested in. The first and most basic way of doing this is configuring the channel to filter traffic based on our MAC address.
  • Enable the channel and let the packets flow.

At this point NCSI takes a back seat to normal network traffic, transmitting a "Get Link Status" packet at regular intervals to monitor the channel.

AEN Packets

Changes can occur from the package side too; the NCSI package communicates these back to the BMC with Asynchronous Event Notification (AEN) packets. As the name suggests these can occur at any time and the driver needs to catch and handle these. There are different types but they essentially boil down to changes in link state, telling the BMC the channel needs to be reconfigured, or to select a different channel. These are only transmitted once and no effort is made to recover lost AEN packets - another good reason for the NCSI driver to periodically monitor the channel.


Each channel can be configured to filter traffic based on MAC address, broadcast traffic, multicast traffic, and VLAN tagging. Associated with each of these filters is a filter table which can hold a finite number of entries. In the case of the VLAN filter each channel could match against 15 different VLAN IDs for example, but in practice the physical device will likely support less. Indeed the popular BCM5718 controller supports only two!

This is where I dived into NCSI. The driver had a lot of the pieces for configuring VLAN filters but none of it was actually hooked up in the configure state, and didn't have a way of actually knowing which VLAN IDs were meant to be configured on the interface. The bulk of that work appears in this commit where we take advantage of some useful network stack callbacks to get the VLAN configuration and set them during the configuration state. Getting to the configuration state at some arbitrary time and then managing to assign multiple IDs was the trickiest bit, and is something I'll be looking at simplifying in the future.

NCSI! A neat way to give physically separate users access to a single network controller, and if it works right you won't notice it at all. I'll surely be spending more time here (fleshing out the driver's features, better error handling, and making the state machine a touch more readable to start, and I haven't even mentioned HWA), so watch this space!

Categories: thinktime

In search of competition

Seth Godin - Thu 21st Sep 2017 18:09
The busiest Indian restaurants in New York City are all within a block or two of each other. Books sell best in bookstores, surrounded by other books, their ostensible competitors. And it's far easier to sell a technology solution if...        Seth Godin
Categories: thinktime

Can you live in a shepherd's hut?

Seth Godin - Wed 20th Sep 2017 18:09
The best way to plan a house on a vacant piece of land is to move into a tiny shepherd's hut on a corner of the property. It's not fancy, and it's not comfortable, but you can probably stay there...        Seth Godin
Categories: thinktime

Selling confusion

Seth Godin - Tue 19th Sep 2017 17:09
Over the last few decades, there's been a consistent campaign to sow confusion around evolution, vaccines and climate change. In all three areas, we all have access to far more data, far more certainty and endless amounts of proof that...        Seth Godin
Categories: thinktime

Beware of false averages

Seth Godin - Mon 18th Sep 2017 18:09
Some people like really spicy food. Some people like bland food. Building a restaurant around sorta spicy food doesn't make either group happy. It's tempting to look at pop music, network TV and the latest hot fashion and come to...        Seth Godin
Categories: thinktime

OpenSTEM: Those Dirty Peasants!

Planet Linux Australia - Mon 18th Sep 2017 09:09
It is fairly well known that many Europeans in the 17th, 18th and early 19th centuries did not follow the same routines of hygiene as we do today. There are anecdotal and historical accounts of people being dirty, smelly and generally unhealthy. This was particularly true of the poorer sections of society. The epithet “those […]
Categories: thinktime

Selfish marketing doesn't last

Seth Godin - Sun 17th Sep 2017 18:09
If it helps you, not the customer, why should she care? Sometimes there's an overlap between your selfish needs and hers, but you can save everyone a lot of time and hassle if you begin and end with a focus...        Seth Godin
Categories: thinktime

Dave Hall: Trying Drupal

Planet Linux Australia - Sat 16th Sep 2017 21:09

While preparing for my DrupalCamp Belgium keynote presentation I looked at how easy it is to get started with various CMS platforms. For my talk I used Contentful, a hosted content as a service CMS platform and contrasted that to the "Try Drupal" experience. Below is the walk through of both.

Let's start with Contentful. I start off by visiting their website.

In the top right corner is a blue button encouraging me to "try for free". I hit the link and I'm presented with a sign up form. I can even use Google or GitHub for authentication if I want.

While my example site is being installed I am presented with an overview of what I can do once it is finished. It takes around 30 seconds for the site to be installed.

My site is installed and I'm given some guidance about what to do next. There is even an onboarding tour in the bottom right corner that is waving at me.

Overall this took around a minute and required very little thought. I never once found myself thinking come on hurry up.

Now let's see what it is like to try Drupal. I land on d.o. I see a big prominent "Try Drupal" button, so I click that.

I am presented with 3 options. I am not sure why I'm being presented options to "Build on Drupal 8 for Free" or to "Get Started Risk-Free", I just want to try Drupal, so I go with Pantheon.

Like with Contentful I'm asked to create an account. Again I have the option of using Google for the sign up or completing a form. This form has more fields than contentful.

I've created my account and I am expecting to be dropped into a demo Drupal site. Instead I am presented with a dashboard. The most prominent call to action is importing a site. I decide to create a new site.

I have to now think of a name for my site. This is already feeling like a lot of work just to try Drupal. If I was a busy manager I would have probably given up by this point.

When I submit the form I must surely be going to see a Drupal site. No, sorry. I am given the choice of installing WordPress, yes WordPress, Drupal 8 or Drupal 7. Despite being very confused I go with Drupal 8.

Now my site is deploying. While this happens there is a bunch of items that update above the progress bar. They're all a bit nerdy, but at least I know something is happening. Why is my only option to visit my dashboard again? I want to try Drupal.

I land on the dashboard. Now I'm really confused. This all looks pretty geeky. I want to try Drupal not deal with code, connection modes and the like. If I stick around I might eventually click "Visit Development site", which doesn't really feel like trying Drupal.

Now I'm asked to select a language. OK so Drupal supports multiple languages, that nice. Let's select English so I can finally get to try Drupal.

Next I need to chose an installation profile. What is an installation profile? Which one is best for me?

Now I need to create an account. About 10 minutes I already created an account. Why do I need to create another one? I also named my site earlier in the process.

Finally I am dropped into a Drupal 8 site. There is nothing to guide me on what to do next.

I am left with a sense that setting up Contentful is super easy and Drupal is a lot of work. For most people wanting to try Drupal they would have abandonned someway through the process. I would love to see the conversion stats for the try Drupal service. It must miniscule.

It is worth noting that Pantheon has the best user experience of the 3 companies. The process with 1&1 just dumps me at a hosting sign up page. How does that let me try Drupal?

Acquia drops onto a page where you select your role, then you're presented with some marketing stuff and a form to request a demo. That is unless you're running an ad blocker, then when you select your role you get an Ajax error.

The Try Drupal program generates revenue for the Drupal Association. This money helps fund development of the project. I'm well aware that the DA needs money. At the same time I wonder if it is worth it. For many people this is the first experience they have using Drupal.

The previous attempt to have added to the try Drupal page ultimately failed due to the financial implications. While this is disappointing I don't think is necessarily the answer either.

There needs to be some minimum standards for the Try Drupal page. One of the key item is the number of clicks to get from d.o to a working demo site. Without this the "Try Drupal" page will drive people away from the project, which isn't the intention.

If you're at DrupalCon Vienna and want to discuss this and other ways to improve the marketing of Drupal, please attend the marketing sprints.

AttachmentSize try-contentful-1.png342.82 KB try-contentful-2.png214.5 KB try-contentful-3.png583.02 KB try-contentful-5.png826.13 KB try-drupal-1.png1.19 MB try-drupal-2.png455.11 KB try-drupal-3.png330.45 KB try-drupal-4.png239.5 KB try-drupal-5.png203.46 KB try-drupal-6.png332.93 KB try-drupal-7.png196.75 KB try-drupal-8.png333.46 KB try-drupal-9.png1.74 MB try-drupal-10.png1.77 MB try-drupal-11.png1.12 MB try-drupal-12.png1.1 MB try-drupal-13.png216.49 KB
Categories: thinktime

Fitting in all the way

Seth Godin - Sat 16th Sep 2017 18:09
It seems like a fine way to earn trust. Merely fit in. In every way. Don't do anything to draw attention to yourself, to be left out, to challenge the status quo. Go along with the crowd to get ahead....        Seth Godin
Categories: thinktime

Appropriate collusion (organizing the weaker side)

Seth Godin - Fri 15th Sep 2017 18:09
Businesses with power are prohibited from colluding with one another to set prices or other policies. For good reason. Public officials and economists realize that it’s quite tempting for an oligopoly to work to artificially create scarcity or cooperate--it creates...        Seth Godin
Categories: thinktime

Project Management for Humans

a list apart - Fri 15th Sep 2017 04:09

A note from the editors: We’re pleased to share an excerpt from Chapter 6 of Brett Harned's new book, Project Management for Humans, available now from Rosenfeld Media.

I loved the game Tetris as a kid. I played the Game Boy version for hours. It’s easy to get wrapped up in the concept of little shapes coming together in a logical way to clear a goal. The pieces complement one another, yet they all naturally work in different ways. The game has stuck with me since I was a kid (and, no, I’m not a gamer). I now have it on my phone and iPad and find myself playing it when I’m on a flight or bored, waiting for something to happen (which is never these days). Whether I’m playing the game a lot or not, the idea of making tiny boxes fit in neatly and clearing out rows of work is ingrained in my brain. It’s the project manager in me.

But here’s the thing: What project managers do on a daily basis when it comes to managing resources or staffing is similar to Tetris, and it’s a big project management challenge that we all face. The biggest difference between resourcing and Tetris? The team members we’re trying to assign tasks to aren’t blocks. They’re human beings, and they need to be treated as such.

Your Team Are People, Too!

Let’s move away from calling people “resources,” please. We’re really just staffing projects or assigning tasks. We’re not using people to just get things done. We’re asking them to solve challenges that are presented in our projects.

Set the Stage for Organized Resource Planning

The challenge of managing a team is making sure that they stay busy and working on tasks, yet are not completely overbooked. It’s a difficult balance to find, particularly when your projects require a variety of skills at different times, which seem to change all too often.

At the most basic level, you want to set up a system for tracking your projects and your team members’ time on those projects (see Figure 6.1). A simple goal is to ensure that you can confidently commit to deadlines on projects with the knowledge that your team is actually available to do the related work. It seems like a simple goal, but it’s often a difficult one to keep up with due to changes on projects, changes in personal schedules (hey, life happens), and an influx of new work and requests. But it’s not an insurmountable challenge. In fact, a simple spreadsheet could help you, particularly if you’re managing a smaller team. At the core, you want to track these items:

  • Projects (List them all, even the non-billable ones, or the other things that aren’t projects but end up taking a lot of time—like business development.)
  • People (List every person you work with.)
  • Estimated time (Track hours, days, weeks, etc. Make your best guess—based on your timeline or calendar—on how much each person will spend on a project or a task.)
Figure 6.1: Use a Google Spreadsheet, Numbers, or Excel to input your project and team data.

A couple of notes on how to use a spreadsheet to forecast team availability:

  • This should be set up on a week-by-week basis to minimize confusion (use tabs in your spreadsheet for each new week).
  • Always consider the “nonbillable” things that people must do (like stand-up meetings, internal tasks, sales, etc.).
  • The final cell contains a formula that tallies the hours for you; if the hours go over your typical limit (think of a 40-hour work week), it will turn red to notify you. You’ll want to have a good idea for just how “utilized” someone should be (32 hours/week is usually a good target).
  • You can input the actual hours logged in your time tracking system if you’d like. It could help with future estimating. (If you’re not tracking time, check in with your team on time percentages to get a gut check.)
  • Check your estimates with your team to make sure that the hours actually align with their assessment of the task (This might help with avoiding that red number!)
  • Communicate these hours to the entire team each week. Making sure that everyone “is in the know” will help on any project. Discussing it with individuals will help you understand effort, blockers, and possibly even different ways of working.

The landscape for project management tools is changing constantly. There are a number of tools in the marketplace for helping you manage and communicate this data. If you’ve working with a team of 10 or more, you might want to abandon the spreadsheet approach for something more official, organized and supported. Bonus: Many of these tools handle more than just resourcing!

Here’s the thing—it’s not just about numbers. The issue that makes estimating a team’s project hours difficult is that everyone works differently. There is no way to standardize the human factor here, and that’s what makes it tough. Forget the fact that no one on your team is a robot, and they all work at their own pace. Think about sick days, vacations, client delays, changes on projects, and so on. It’s a never-ending flow of shapes that must fit into the box that is a project. Be sure to have an ongoing dialogue about your staffing plans and challenges.

Match Resource Skills to Projects

Projects only slow down when decisions are not made. In that magical moment when things are actually going well, you want to make sure that your team can continue the pace. The only way to do that is by connecting with your team and understanding what motivates them. Here are some things to consider:

  • Interests: If you have a team member who loves beer, why not put that person on the beer design site? Maybe you have multiple people who want to be on the project, but they are all busy on other projects. These are the breaks. You’ve got to do what is right for the company and your budget. If you can put interests first, it’s awesome. It won’t always work out that way for everyone, but it’s a good first step to try.
  • Skill sets: It’s as simple as getting to know each and every team member’s work. Some people are meant to create specific types of designs or experiences. It not only has to do with interests, but it also has to do with strengths within those tasks. Sure, I may love beer, but that doesn’t mean that I am meant to design the site that caters to the audience the client is trying to reach.
  • Moving schedules: Projects will always change. One week you know you’re working against a firm deadline, and the next week that has changed due to the clients, the needs of the project, or some other reason someone conjured up. It’s tough to know when that change will happen, but when it does, how you’ll fill someone’s time with other work should be high on your mind.
  • Holidays: People always extend them. Plan for that!
  • Vacations: It’s great to know about these in advance. Be sure you know your company’s policies around vacations. You never ever want to be the PM who says “Well, you have a deadline on X date and that will conflict with your very expensive/exciting trip, so, um … no.” Ask people to request trips at least a month in advance so that you can plan ahead and make it work.
  • Illness: We’re all humans and that means we’re fine one day and bedridden the next. You’ve always got to be ready for a back-up plan. It shouldn’t fall on your client stakeholders to make up time, but sometimes it has to. Or sometimes you need to look for someone to pitch in on intermediate tasks to keep things of track while your “rock star” or “ninja” is getting better.
Align Plans with Staffing

When you’re working hard to keep up with staffing plans, you’ve got to have updated project plans. A small change in a plan could cause a change in staffing—even by a few hours—and throw everything else off.

Save Yourself and Your Team from Burnout

If you’re busy and not slowing down any time soon, you want to keep this spreadsheet (or tool) updated often. If you’re working at an agency, knowing what’s in your pipeline can also help you. Stay aligned with the person in charge of sales or assigning new projects so that you can anticipate upcoming needs and timelines. In some cases, you may even want to put some basic data in your spreadsheet or tool so that you can anticipate needs.

Good Resourcing Can Justify More Help

The value of tracking this data goes beyond your projects. It can help business owners make important decisions on growing a company.

No matter what you do, be sure to communicate about staffing as much as possible. If you’re in an organization that is constantly handling change, you’ll know that it’s a tough target to hit. In fact, your numbers will often be slightly off, but you’ll find comfort in knowing that you’re doing everything you can to stay ahead of the resource crunch. At the same time, your team will appreciate that you’re doing everything you can to protect their work-life balance.

Stakeholders Are Resources, Too

When you’re working on a team with a project, you have to consider the stakeholders as decision makers, too. Let’s face it—no one has ever been trained to be a good client, stakeholder, or project sponsor. In addition to that, they are likely to be working on several projects with several people at one time. Life as a client can be hectic! So do everything you can to help them plan their time appropriately. In general, you should let the stakeholders know they’ll have to plan for these things:

  • Meetings: You’ll conduct a kickoff meeting, weekly status updates, deliverable reviews, etc.
  • Scheduling: You’ll need stakeholders to wrangle calendars to get folks into said meetings.
  • Gathering feedback: This sounds easy, but it is not. You will need this person to spend time with all of the stakeholders to get their feedback and collate it for you to make sure there are no conflicting opinions.
  • Chasing down decisions: There are points on every project where one person will need to make sure there is agreement and decisions can be made to keep the project moving.
  • Daily ad hoc email, phone calls: Questions and requests will pop up, and you’ll need timely responses.
  • Operations: You might need invoices to be reviewed and approved or change requests to be reviewed and discussed. The stakeholders will need to make time to operate the project from their side of things.

This is a lot of work. And just like PM work, it is very hard to quantify or plan. If you’re in good hands, you’re working with someone who has good PM skills. If not, give them the list above along with a copy of this book. But seriously, if you can assist them with planning their time, it might be as simple as including action items or to-dos for them in a weekly email or in your status report. Just remember, they are busy and want the project to run smoothly as well. Help them make that happen.


Managing projects is hard enough, but being the person to manage who works on what and when can be even more difficult. However, if you don’t keep track of this basic information, you’ll likely find it hard to meet deadlines and wrap up projects without major issues. Here are some simple things you can do to make sure your that your team stays busy, yet not completely overbooked:

  • Set up a simple spreadsheet to forecast projects and hours per team member.
    • This data should be based on what’s included in your project scopes and timelines—be sure to double-check that.
    • You may want to check out one of the resourcing tools that are out there now.
  • Be sure to account for a number of factors that you can’t necessarily control in this process—for example, interests, skill sets, moving schedules, holidays, vacations, and so on.
  • Account for your sales process if you’re in an agency and stay ahead of new project requests.
  • Remember that you’re dealing with people here.
Want to read more?

This excerpt from Project Management for Humans will help you get started. Order the full copy today, as well as other excellent titles from Rosenfeld Media.

Categories: thinktime

Impossible, unlikely or difficult?

Seth Godin - Thu 14th Sep 2017 18:09
Difficult tasks have a road map. With effort, we can get from here to there. It might surprise you to realize that difficult is easy once you have the resources and commitment. Paving a road is difficult, so is customer...        Seth Godin
Categories: thinktime

OpenSTEM: New Dates for Human Relative + ‘Explorer Classroom’ Resources

Planet Linux Australia - Thu 14th Sep 2017 11:09
During September, National Geographic is featuring the excavations of Homo naledi at Rising Star Cave in South Africa in their Explorer Classroom, in tune with new discoveries and the publishing of dates for this enigmatic little hominid. A Teacher’s Guide and Resources are available and classes can log in to see live updates from the […]
Categories: thinktime

A List Apart volunteer update

a list apart - Wed 13th Sep 2017 23:09

A note from the editors: A few days ago, we announced a reimagined A List Apart, with you, our faithful readers of nearly 20 years, contributing your talents. The response from this community was humbling, thrilling, and, frankly, a bit overwhelming. If you volunteered to help A List Apart and haven’t heard back from us yet, here’s what’s up.

To the many wonderful souls who have so far volunteered to help A List Apart, thank you very, very much for your emails! And if you haven’t heard back from us yet,  please excuse the delay. We’ve been inundated with messages from hundreds of potential volunteers across a wide spectrum of disciplines and potential task groups, and we are going through your messages slowly and carefully, responding personally to each one.

Some of you have written asking if we might be interested in having you write for us. Gosh, A List Apart has always welcomed articles from our community. Guidelines (plus how to submit your first draft, proposal, or outline) are available at Please check them out—we’d love to look at any topically appropriate article you care to submit. 

But writing articles is far from the only way to support and make your mark at the new (19-year-old) ALA.

Meet the groups!

If you’ve expressed an interested in organizing or hosting an ALA-themed monthly meet-up, or have other ideas that can help grow community, we’ll invite you to join our newly forming COMMUNITY group. If EDUCATION AND OUTREACH is more your thing, we are starting a group for that, as well. There are other groups to come, as well—a list of our ideas appears in the original post on the topic, and there may be more groups to come.

How these groups will work, and what they will do, is largely going to be determined by the volunteers themselves. (That’s you folks.)

As we’re starting the work of supporting and organizing these groups on Basecamp, you can’t just add yourself to a group, as you could on, say, Slack. But that’s okay, because we want to approach this somewhat methodically, adding people a few at a time, and having little written conversations with you beforehand.

Our fear was that if we launched a bunch of Slack channels all at once, without speaking with each of you first, hundreds of people might add themselves the first day, but then nobody would have any direction as to what might be expected—and we might not have the resources ready to provide guidance and support.

By adding you to Basecamps a few at a time, and hopefully identifying leaders in each new group as it begins forming, we hope to provide a lightly structured environment where you can design your own adventures. It takes a little longer this way, but that’s by design. (A List Apart started in 1997 as a 16,000-member message board. Big open channels are great for letting everyone speak, but not necessarily the best way to organize fragile new projects.)

If you are interested in contributing to those projects, or curious about a particular area, and told us so in your initial email, we will eventually get to you and assign you to the right slot. If you haven’t yet volunteered, of course, you can still do so. (See the original post for details.)

Editors, developers, and designers

But wait, there’s more. Developers: if you have standards-oriented front-end development experience and would like to help out on day-to-day site maintenance, occasional minor upgrades, and an eventual redesign, just add yourself to A List Apart’s Github front-end repo:

Those with backend experience (particularly in ExpressionEngine and WordPress), you will hear from us as we work our way through your emails.

Editor-in-chief Aaron Gustafson and I have also been going slowly through your mails looking for additional editorial help. We’ve already found and added a few very promising people to our volunteer editorial staff, and will introduce them to you soon. If you’re an editor and we haven’t added you yet, not to worry! It likely means we haven’t gotten to your email yet. (So. Much. Email!)

As might be expected, a majority of those who volunteered offered their services as designers, developers, or both. The number of emails we’ve received from folks with these skills is humbling, touching, and a bit overwhelming. We have not yet begun to dig through this particular pile of mail. So if you haven’t heard from us, that’s why. (But, as I just mentioned, if you’re a developer, you can add yourself to our front-end repo. So do that, if you wish, and say hi!)

We love you

Hope this helps clarify what’s up. We are grateful for every single email we’ve gotten. We will eventually speak with you all. Thank you all again.



Categories: thinktime

Building on maximized systems

Seth Godin - Wed 13th Sep 2017 18:09
If you eat beef, you're probably using a maximized system. It's a commodity, and every part of the chain is under huge pressure to increase yield and cut corners. The animals are pushed to the brink, and so are the...        Seth Godin
Categories: thinktime


Subscribe to kattekrab aggregator