You are here

thinktime

On the danger of saying something for the first time

Seth Godin - Mon 25th Sep 2017 19:09
That's rare air, with no support, no foundation. Like the coyote, running of a cliff in pursuit of the roadrunner... What could be more important? When we synthesize and invent and leap, we create a rare sort of value.        Seth Godin
Categories: thinktime

OpenSTEM: What Makes Humans Different From Most Other Mammals?

Planet Linux Australia - Mon 25th Sep 2017 09:09
Well, there are several things that makes us different from other mammals – although perhaps fewer than one might think. We are not unique in using tools, in fact we discover more animals that use tools all the time – even fish! We pride ourselves on being a “moral animal”, however fairness, reciprocity, empathy and […]
Categories: thinktime

Dave Hall: Drupal Puppies

Planet Linux Australia - Sun 24th Sep 2017 21:09

Over the years Drupal distributions, or distros as they're more affectionately known, have evolved a lot. We started off passing around database dumps. Eventually we moved onto using installations profiles and features to share par-baked sites.

There are some signs that distros aren't working for people using them. Agencies often hack a distro to meet client requirements. This happens because it is often difficult to cleanly extend a distro. A content type might need extra fields or the logic in an alter hook may not be desired. This makes it difficult to maintain sites built on distros. Other times maintainers abandon their distributions. This leaves site owners with an unexpected maintenance burden.

We should recognise how people are using distros and try to cater to them better. My observations suggest there are 2 types of Drupal distributions; starter kits and targeted products.

Targeted products are easier to deal with. Increasingly monetising targeted distro products is done through a SaaS offering. The revenue can funds the ongoing development of the product. This can help ensure the project remains sustainable. There are signs that this is a viable way of building Drupal 8 based products. We should be encouraging companies to embrace a strategy built around open SaaS. Open Social is a great example of this approach. Releasing the distros demonstrates a commitment to the business model. Often the secret sauce isn't in the code, it is the team and services built around the product.

Many Drupal 7 based distros struggled to articulate their use case. It was difficult to know if they were a product, a demo or a community project that you extend. Open Atrium and Commerce Kickstart are examples of distros with an identity crisis. We need to reconceptualise most distros as "starter kits" or as I like to call them "puppies".

Why puppies? Once you take a puppy home it becomes your responsibility. Starter kits should be the same. You should never assume that a starter kit will offer an upgrade path from one release to the next. When you install a starter kit you are responsible for updating the modules yourself. You need to keep track of security releases. If your puppy leaves a mess on the carpet, no one else will clean it up.

Sites build on top of a starter kit should diverge from the original version. This shouldn't only be an expectation, it should be encouraged. Installing a starter kit is the starting point of building a unique fork.

Project pages should clearly state that users are buying a puppy. Prospective puppy owners should know if they're about to take home a little lap dog or one that will grow to the size of a pony that needs daily exercise. Puppy breeders (developers) should not feel compelled to do anything once releasing the puppy. That said, most users would like some documentation.

I know of several agencies and large organisations that are making use of starter kits. Let's support people who are adopting this approach. As a community we should acknowledge that distros aren't working. We should start working out how best to manage the transition to puppies.

Categories: thinktime

#perfect

Seth Godin - Sun 24th Sep 2017 17:09
Nothing ever is. Nothing is flawless, optimized and suitable for everyone. Instead, all we can hope for is, "the best we could hope for, under the circumstances." But, because there are circumstances, whatever happens is exactly what the circumstances created....        Seth Godin
Categories: thinktime

Tim Serong: On Equal Rights

Planet Linux Australia - Sun 24th Sep 2017 05:09

This is probably old news now, but I only saw it this morning, so here we go:

WOW. JILL POULSEN'S COLUMN TOMORROW. pic.twitter.com/5kmPa8LHiC

— The NT News (@TheNTNews) September 15, 2017

In case that embedded tweet doesn’t show up properly, that’s an editorial in the NT News which says:

Voting papers have started to drop through Territory mailboxes for the marriage equality postal vote and I wanted to share with you a list of why I’ll be voting yes.

1. I’m not an arsehole.

This resulted in predictable comments along the lines of “oh, so if I don’t share your views, I’m an arsehole?”

I suppose it’s unlikely that anyone who actually needs to read and understand what I’m about to say will do so, but just in case, I’ll lay this out as simply as I can:

  • A personal belief that marriage is a thing that can only happen between a man and a woman does not make you an arsehole (it might make you on the wrong side of history, or a lot of other things, but it does not necessarily make you an arsehole).
  • Voting “no” to marriage equality is what makes you an arsehole.

The survey says “Should the law be changed to allow same-sex couples to marry?” What this actually means is, “Should same-sex couples have the same rights under law as everyone else?”

If you believe everyone should have the same rights under law, you need to vote yes regardless of what you, personally, believe the word “marriage” actually means – this is to make sure things like “next of kin” work the way the people involved in a relationship want them to.

If you believe that there are minorities that should not have the same rights under law as everyone else, then I’m sorry, but you’re an arsehole.

(Personally I think the Marriage Act should be ditched entirely in favour of a Civil Unions Act – that way the word “marriage” could go back to simply meaning whatever it means to the individuals being married, and to their god(s) if they have any – but this should in no way detract from the above.)

Categories: thinktime

Constructive dissatisfaction

Seth Godin - Sat 23rd Sep 2017 18:09
It's never been easier to find ways to be disappointed in our performance. You can compare your output, your income, your success rate to a billion people around the globe... many of whom are happy to exaggerate to make you...        Seth Godin
Categories: thinktime

Russell Coker: Converting Mbox to Maildir

Planet Linux Australia - Sat 23rd Sep 2017 15:09

MBox is the original and ancient format for storing mail on Unix systems, it consists of a single file per user under /var/spool/mail that has messages concatenated. Obviously performance is very poor when deleting messages from a large mail store as the entire file has to be rewritten. Maildir was invented for Qmail by Dan Bernstein and has a single message per file giving fast deletes among other performance benefits. An ongoing issue over the last 20 years has been converting Mbox systems to Maildir. The various ways of getting IMAP to work with Mbox only made this more complex.

The Dovecot Wiki has a good page about converting Mbox to Maildir [1]. If you want to keep the same message UIDs and the same path separation characters then it will be a complex task. But if you just want to copy a small number of Mbox accounts to an existing server then it’s a bit simpler.

Dovecot has a mb2md.pl script to convert folders [2].

cd /var/spool/mail mkdir -p /mailstore/example.com for U in * ; do ~/mb2md.pl -s $(pwd)/$U -d /mailstore/example.com/$U done

To convert the inboxes shell code like the above is needed. If the users don’t have IMAP folders (EG they are just POP users or use local Unix MUAs) then that’s all you need to do.

cd /home for DIR in */mail ; do U=$(echo $DIR| cut -f1 -d/) cd /home/$DIR for FOLDER in * ; do ~/mb2md.pl -s $(pwd)/$FOLDER -d /mailstore/example.com/$U/.$FOLDER done cp .subscriptions /mailstore/example.com/$U/ subscriptions done

Some shell code like the above will convert the IMAP folders to Maildir format. The end result is that the users will have to download all the mail again as their MUA will think that every message had been deleted and replaced. But as all servers with significant amounts of mail or important mail were probably converted to Maildir a decade ago this shouldn’t be a problem.

Related posts:

  1. Why Cyrus Sucks I’m in the middle of migrating a mail server away...
  2. Moving a Mail Server Nowadays it seems that most serious mail servers (IE mail...
  3. Mail Server Training Today I ran a hands-on training session on configuring a...
Categories: thinktime

Linux Users of Victoria (LUV) Announce: LUV Main October 2017 Meeting

Planet Linux Australia - Sat 23rd Sep 2017 13:09
Start: Oct 3 2017 18:30 End: Oct 3 2017 20:30 Start: Oct 3 2017 18:30 End: Oct 3 2017 20:30 Location:  Mail Exchange Hotel, 688 Bourke St, Melbourne VIC 3000 Link:  http://mailexchangehotel.com.au/

PLEASE NOTE NEW LOCATION

Tuesday, October 3, 2017
6:30 PM to 8:30 PM
Mail Exchange Hotel
688 Bourke St, Melbourne VIC 3000

Speakers:

  • TBA

Mail Exchange Hotel, 688 Bourke St, Melbourne VIC 3000

Food and drinks will be available on premises.

Linux Users of Victoria is a subcommittee of Linux Australia. October 3, 2017 - 18:30
Categories: thinktime

Linux Users of Victoria (LUV) Announce: LUV October 2017 Workshop

Planet Linux Australia - Sat 23rd Sep 2017 13:09
Start: Oct 21 2017 12:30 End: Oct 21 2017 16:30 Start: Oct 21 2017 12:30 End: Oct 21 2017 16:30 Location:  Infoxchange, 33 Elizabeth St. Richmond Link:  http://luv.asn.au/meetings/map

There will also be the usual casual hands-on workshop, Linux installation, configuration and assistance and advice. Bring your laptop if you need help with a particular issue. This will now occur BEFORE the talks from 12:30 to 14:00. The talks will commence at 14:00 (2pm) so there is time for people to have lunch nearby.

The meeting will be held at Infoxchange, 33 Elizabeth St. Richmond 3121 (enter via the garage on Jonas St.) Late arrivals, please call (0421) 775 358 for access to the venue.

LUV would like to acknowledge Infoxchange for the venue.

Linux Users of Victoria is a subcommittee of Linux Australia.

October 21, 2017 - 12:30
Categories: thinktime

sthbrx - a POWER technical blog: Stupid Solutions to Stupid Problems: Hardcoding Your SSH Key in the Kernel

Planet Linux Australia - Sat 23rd Sep 2017 03:09
The "problem"

I'm currently working on firmware and kernel support for OpenCAPI on POWER9.

I've recently been allocated a machine in the lab for development purposes. We use an internal IBM tool running on a secondary machine that triggers hardware initialisation procedures, then loads a specified skiboot firmware image, a kernel image, and a root file system directly into RAM. This allows us to get skiboot and Linux running without requiring the usual hostboot initialisation and gives us a lot of options for easier tinkering, so it's super-useful for our developers working on bringup.

When I got access to my machine, I figured out the necessary scripts, developed a workflow, and started fixing my code... so far, so good.

One day, I was trying to debug something and get logs off the machine using ssh and scp, when I got frustrated with having to repeatedly type in our ultra-secret, ultra-secure root password, abc123. So, I ran ssh-copy-id to copy over my public key, and all was good.

Until I rebooted the machine, when strangely, my key stopped working. It took me longer than it should have to realise that this is an obvious consequence of running entirely from an initrd that's reloaded every boot...

The "solution"

I mentioned something about this to Jono, my housemate/partner-in-stupid-ideas, one evening a few weeks ago. We decided that clearly, the best way to solve this problem was to hardcode my SSH public key in the kernel.

This would definitely be the easiest and most sensible way to solve the problem, as opposed to, say, just keeping my own copy of the root filesystem image. Or asking Mikey, whose desk is three metres away from mine, whether he could use his write access to add my key to the image. Or just writing a wrapper around sshpass...

One Tuesday afternoon, I was feeling bored...

The approach

The SSH daemon looks for authorised public keys in ~/.ssh/authorized_keys, so we need to have a read of /root/.ssh/authorized_keys return a specified hard-coded string.

I did a bit of investigation. My first thought was to put some kind of hook inside whatever filesystem driver was being used for the root. After some digging, I found out that the filesystem type rootfs, as seen in mount, is actually backed by the tmpfs filesystem. I took a look around the tmpfs code for a while, but didn't see any way to hook in a fake file without a lot of effort - the tmpfs code wasn't exactly designed with this in mind.

I thought about it some more - what would be the easiest way to create a file such that it just returns a string?

Then I remembered sysfs, the filesystem normally mounted at /sys, which is used by various kernel subsystems to expose configuration and debugging information to userspace in the form of files. The sysfs API allows you to define a file and specify callbacks to handle reads and writes to the file.

That got me thinking - could I create a file in /sys, and then use a bind mount to have that file appear where I need it in /root/.ssh/authorized_keys? This approach seemed fairly straightforward, so I decided to give it a try.

First up, creating a pseudo-file. It had been a while since the last time I'd used the sysfs API...

sysfs

The sysfs pseudo file system was first introduced in Linux 2.6, and is generally used for exposing system and device information.

Per the sysfs documentation, sysfs is tied in very closely with the kobject infrastructure. sysfs exposes kobjects as directories, containing "attributes" represented as files. The kobject infrastructure provides a way to define kobjects representing entities (e.g. devices) and ksets which define collections of kobjects (e.g. devices of a particular type).

Using kobjects you can do lots of fancy things such as sending events to userspace when devices are hotplugged - but that's all out of the scope of this post. It turns out there's some fairly straightforward wrapper functions if all you want to do is create a kobject just to have a simple directory in sysfs.

#include <linux/kobject.h> static int __init ssh_key_init(void) { struct kobject *ssh_kobj; ssh_kobj = kobject_create_and_add("ssh", NULL); if (!ssh_kobj) { pr_err("SSH: kobject creation failed!\n"); return -ENOMEM; } } late_initcall(ssh_key_init);

This creates and adds a kobject called ssh. And just like that, we've got a directory in /sys/ssh/!

The next thing we have to do is define a sysfs attribute for our authorized_keys file. sysfs provides a framework for subsystems to define their own custom types of attributes with their own metadata - but for our purposes, we'll use the generic bin_attribute attribute type.

#include <linux/sysfs.h> const char key[] = "PUBLIC KEY HERE..."; static ssize_t show_key(struct file *file, struct kobject *kobj, struct bin_attribute *bin_attr, char *to, loff_t pos, size_t count) { return memory_read_from_buffer(to, count, &pos, key, bin_attr->size); } static const struct bin_attribute authorized_keys_attr = { .attr = { .name = "authorized_keys", .mode = 0444 }, .read = show_key, .size = sizeof(key) };

We provide a simple callback, show_key(), that copies the key string into the file's buffer, and we put it in a bin_attribute with the appropriate name, size and permissions.

To actually add the attribute, we put the following in ssh_key_init():

int rc; rc = sysfs_create_bin_file(ssh_kobj, &authorized_keys_attr); if (rc) { pr_err("SSH: sysfs creation failed, rc %d\n", rc); return rc; }

Woo, we've now got /sys/ssh/authorized_keys! Time to move on to the bind mount.

Mounting

Now that we've got a directory with the key file in it, it's time to figure out the bind mount.

Because I had no idea how any of the file system code works, I started off by running strace on mount --bind ~/tmp1 ~/tmp2 just to see how the userspace mount tool uses the mount syscall to request the bind mount.

execve("/bin/mount", ["mount", "--bind", "/home/ajd/tmp1", "/home/ajd/tmp2"], [/* 18 vars */]) = 0 ... mount("/home/ajd/tmp1", "/home/ajd/tmp2", 0x18b78bf00, MS_MGC_VAL|MS_BIND, NULL) = 0

The first and second arguments are the source and target paths respectively. The third argument, looking at the signature of the mount syscall, is a pointer to a string with the file system type. Because this is a bind mount, the type is irrelevant (upon further digging, it turns out that this particular pointer is to the string "none").

The fourth argument is where we specify the flags bitfield. MS_MGC_VAL is a magic value that was required before Linux 2.4 and can now be safely ignored. MS_BIND, as you can probably guess, signals that we want a bind mount.

(The final argument is used to pass file system specific data - as you can see it's ignored here.)

Now, how is the syscall actually handled on the kernel side? The answer is found in fs/namespace.c.

SYSCALL_DEFINE5(mount, char __user *, dev_name, char __user *, dir_name, char __user *, type, unsigned long, flags, void __user *, data) { int ret; /* ... copy parameters from userspace memory ... */ ret = do_mount(kernel_dev, dir_name, kernel_type, flags, options); /* ... cleanup ... */ }

So in order to achieve the same thing from within the kernel, we just call do_mount() with exactly the same parameters as the syscall uses:

rc = do_mount("/sys/ssh", "/root/.ssh", "sysfs", MS_BIND, NULL); if (rc) { pr_err("SSH: bind mount failed, rc %d\n", rc); return rc; }

...and we're done, right? Not so fast:

SSH: bind mount failed, rc -2

-2 is ENOENT - no such file or directory. For some reason, we can't find /sys/ssh... of course, that would be because even though we've created the sysfs entry, we haven't actually mounted sysfs on /sys.

rc = do_mount("sysfs", "/sys", "sysfs", MS_NOSUID | MS_NOEXEC | MS_NODEV, NULL);

At this point, my key worked!

Note that this requires that your root file system has an empty directory created at /sys to be the mount point. Additionally, in a typical Linux distribution environment (as opposed to my hardware bringup environment), your initial root file system will contain an init script that mounts your real root file system somewhere and calls pivot_root() to switch to the new root file system. At that point, the bind mount won't be visible from children processes using the new root - I think this could be worked around but would require some effort.

Kconfig

The final piece of the puzzle is building our new code into the kernel image.

To allow us to switch this important functionality on and off, I added a config option to fs/Kconfig:

config SSH_KEY bool "Andrew's dumb SSH key hack" default y help Hardcode an SSH key for /root/.ssh/authorized_keys. This is a stupid idea. If unsure, say N.

This will show up in make menuconfig under the File systems menu.

And in fs/Makefile:

obj-$(CONFIG_SSH_KEY) += ssh_key.o

If CONFIG_SSH_KEY is set to y, obj-$(CONFIG_SSH_KEY) evaluates to obj-y and thus ssh-key.o gets compiled. Conversely, obj-n is completely ignored by the build system.

I thought I was all done... then Andrew suggested I make the contents of the key configurable, and I had to oblige. Conveniently, Kconfig options can also be strings:

config SSH_KEY_VALUE string "Value for SSH key" depends on SSH_KEY help Enter in the content for /root/.ssh/authorized_keys.

Including the string in the C file is as simple as:

const char key[] = CONFIG_SSH_KEY_VALUE;

And there we have it, a nicely configurable albeit highly limited kernel SSH backdoor!

Conclusion

I've put the full code up on GitHub for perusal. Please don't use it, I will be extremely disappointed in you if you do.

Thanks to Jono for giving me stupid ideas, and the rest of OzLabs for being very angry when they saw the disgusting things I was doing.

Comments and further stupid suggestions welcome!

Categories: thinktime

Your fast car

Seth Godin - Fri 22nd Sep 2017 18:09
Right there, in your driveway, is a really fast car. And here are the keys. Now, go drive it. Right there, in your hand, is a Chicago Pneumatics 0651 hammer. You can drive a nail through just about anything with...        Seth Godin
Categories: thinktime

sthbrx - a POWER technical blog: NCSI - Nice Network You've Got There

Planet Linux Australia - Fri 22nd Sep 2017 10:09

A neat piece of kernel code dropped into my lap recently, and as a way of processing having to inject an entire network stack into by brain in less-than-ideal time I thought we'd have a look at it here: NCSI!

NCSI - Not the TV Show

NCSI stands for Network Controller Sideband Interface, and put most simply it is a way for a management controller (eg. a BMC like those found on our OpenPOWER machines) to share a single physical network interface with a host machine. Instead of two distinct network interfaces you plug in a single cable and both the host and the BMC have network connectivity.

NCSI-capable network controllers achieve this by filtering network traffic as it arrives and determining if it is host- or BMC-bound. To know how to do this the BMC needs to tell the network controller what to look out for, and from a Linux driver perspective this the focus of the NCSI protocol.

Hi My Name Is 70:e2:84:14:24:a1

The major components of what NCSI helps facilitate are:

  • Network Controllers, known as 'Packages' in this context. There may be multiple separate packages which contain one or more Channels.
  • Channels, most easily thought of as the individual physical network interfaces. If a package is the network card, channels are the individual network jacks. (Somewhere a pedant's head is spinning in circles).
  • Management Controllers, or our BMC, with their own network interfaces. Hypothetically there can be multiple management controllers in a single NCSI system, but I've not come across such a setup yet.

NCSI is the medium and protocol via which these components communicate.

The interface between Management Controller and one or more Packages carries both general network traffic to/from the Management Controller as well as NCSI traffic between the Management Controller and the Packages & Channels. Management traffic is differentiated from regular traffic via the inclusion of a special NCSI tag inserted in the Ethernet frame header. These management commands are used to discover and configure the state of the NCSI packages and channels.

If a BMC's network interface is configured to use NCSI, as soon as the interface is brought up NCSI gets to work finding and configuring a usable channel. The NCSI driver at first glance is an intimidating combination of state machines and packet handlers, but with enough coffee it can be represented like this:

Without getting into the nitty gritty details the overall process for configuring a channel enough to get packets flowing is fairly straightforward:

  • Find available packages.
  • Find each package's available channels.
  • (At least in the Linux driver) select a channel with link.
  • Put this channel into the Initial Config State. The Initial Config State is where all the useful configuration occurs. Here we find out what the selected channel is capable of and its current configuration, and set it up to recognise the traffic we're interested in. The first and most basic way of doing this is configuring the channel to filter traffic based on our MAC address.
  • Enable the channel and let the packets flow.

At this point NCSI takes a back seat to normal network traffic, transmitting a "Get Link Status" packet at regular intervals to monitor the channel.

AEN Packets

Changes can occur from the package side too; the NCSI package communicates these back to the BMC with Asynchronous Event Notification (AEN) packets. As the name suggests these can occur at any time and the driver needs to catch and handle these. There are different types but they essentially boil down to changes in link state, telling the BMC the channel needs to be reconfigured, or to select a different channel. These are only transmitted once and no effort is made to recover lost AEN packets - another good reason for the NCSI driver to periodically monitor the channel.

Filtering

Each channel can be configured to filter traffic based on MAC address, broadcast traffic, multicast traffic, and VLAN tagging. Associated with each of these filters is a filter table which can hold a finite number of entries. In the case of the VLAN filter each channel could match against 15 different VLAN IDs for example, but in practice the physical device will likely support less. Indeed the popular BCM5718 controller supports only two!

This is where I dived into NCSI. The driver had a lot of the pieces for configuring VLAN filters but none of it was actually hooked up in the configure state, and didn't have a way of actually knowing which VLAN IDs were meant to be configured on the interface. The bulk of that work appears in this commit where we take advantage of some useful network stack callbacks to get the VLAN configuration and set them during the configuration state. Getting to the configuration state at some arbitrary time and then managing to assign multiple IDs was the trickiest bit, and is something I'll be looking at simplifying in the future.

NCSI! A neat way to give physically separate users access to a single network controller, and if it works right you won't notice it at all. I'll surely be spending more time here (fleshing out the driver's features, better error handling, and making the state machine a touch more readable to start, and I haven't even mentioned HWA), so watch this space!

Categories: thinktime

How People Perceive Lossy Image Quality: A Study

a list apart - Thu 21st Sep 2017 22:09

The notion that lossy image quality is subjective is not an unreasonable hypothesis. There are many factors that play into how humans perceive quality: screen size, image scaling, and yes, even performance.

Many research projects have tackled this subject, but I’ve recently launched a survey that attempts to understand how people perceive image quality in a slightly different way: in the context of performance.

This image quality assessment serves up 25 different specimens, each of which is presented in a random lossy quality setting between 5 and 100, in both JPEG and WebP formats. As participants complete the survey, navigation, resource and paint timings are collected (when available) from the browser, as well as other client details such as a device’s resolution, pixel density, and many other pertinent details.

The real work of gathering data begins. This is where you can help out. If you have five to ten minutes to spare, please head over to https://imagesurvey.site and participate. When the survey is finished, I’ll post the raw data and write and article (or two) on the findings. If further experimentation is required, that will be pursued as well. I don’t know what we’ll find out, but we’ll find out together with your input. So please participate!

Thank you!

Note: If you have feedback for how to improve the survey, feel free to comment! Just be aware that your feedback can’t be implemented in this run of the survey, but it could be useful in constructing any follow-up surveys.

Categories: thinktime

In search of competition

Seth Godin - Thu 21st Sep 2017 18:09
The busiest Indian restaurants in New York City are all within a block or two of each other. Books sell best in bookstores, surrounded by other books, their ostensible competitors. And it's far easier to sell a technology solution if...        Seth Godin
Categories: thinktime

Can you live in a shepherd's hut?

Seth Godin - Wed 20th Sep 2017 18:09
The best way to plan a house on a vacant piece of land is to move into a tiny shepherd's hut on a corner of the property. It's not fancy, and it's not comfortable, but you can probably stay there...        Seth Godin
Categories: thinktime

The Ten Essentials for Good API Documentation

a list apart - Tue 19th Sep 2017 23:09

API documentation is the number one reference for anyone implementing your API, and it can profoundly influence the developer experience. Because it describes what services an application programming interface offers and how to use those services, your documentation will inevitably create an impression about your product—for better or for worse.

In this two-part series I share what I’ve learned about API documentation. This part discusses the basics to help you create good API docs, while in part two, Ten Extras for Great API Documentation, I’ll show you additional ways to improve and fine-tune your documentation. 

Know your audience

Knowing who you address with your writing and how you can best support them will help you make decisions about the design, structure, and language of your docs. You will have to know who visits your API documentation and what they want to use it for. 

Your API documentation will probably be visited and used by the following audiences. 

Developers

Based on their skills, experience, and role in projects, developers will generally be the largest and most diverse group. They’ll be using your docs in different ways.

At Pronovix, we started conducting developer portal workshops with our clients to help them learn more about what developers need and how to best support their work—and what they’re really looking for in API documentation. This is also supported by solid research, such as the findings published in Stephanie Steinhardt’s article following a two-year research program at Merseburg University of Applied Sciences.

Newcomers: Developers lacking previous experience with your API tend to need the most support. They will take advantage of quickstart guides that encourage them to start using your API—clear, concise, step-by-step tutorials for the most important topics, and sample code and examples to help them understand how to use it in real projects. If you can make onboarding pleasant for newcomers, they will be more likely to devote themselves to learning every nuance of your API.

External developers: Developers already working with your API will come back repeatedly to your docs and use them as reference material. They will need quick information on all the functionality your API offers, structured in an easy to understand way to help them quickly find what they need.

Debuggers: Developers using your API will encounter errors from time to time and use your documentation to analyze the responses and errors that crop up.

Internal developers: API providers tend to focus so much on their external audience that they forget about their own developers; internal teams working on the API will use the API documentation, as well.

These are just the most common use cases.

Decision makers

Decision makers like CTOs and product managers will also check out your API documentation and evaluate your API. They need to determine whether your API will be a good fit for their project or not, so it’s crucial to your business that this group can easily and quickly find what they’re looking for.

Other audiences

Although not as common, journalists, technical writers, support staff, developer evangelists, and even your competition might read your API documentation. 

Remember the purpose of documentation

The foundation of your API documentation is a clear explanation of every call and parameter.

As a bare minimum, you should describe in detail:

  • what each call in your API does
  • each parameter and all of their possible values, including their types, formatting, rules, and whether or not they are required.
Context-based structure

People won’t read your API documentation in order, and you can’t predict which part they will land on. This means, you have to provide all the information they need in context. So following the best practices of topic-based authoring, you should include all necessary and related information in the explanation of each call.

Context.IO, for example, did a great job documenting each of their API calls separately with detailed information on parameters and their possible values, along with useful tips and links to related topics.

Examples

In order to be able to implement your API, developers need to understand it along with the domain it refers to (e.g., ecommerce). Real world examples reduce the time they need to get familiar with your product, and provide domain knowledge at the same time.

Add the following to the description of each call:

  • an example of how the call is made
  • an explanation of the request
  • sample responses

Studies have shown, that some developers immediately like to delve into coding, when getting to know a new API; they start working from an example. Analysis of eye-tracking records showed that visual elements, like example code, caught the attention of developers who were scanning the page, rather than reading it line by line.  Many looked at code samples before they started reading the descriptions.

Using the right examples is a surefire way to improving your API docs. I’ll explore ways to turn good API docs into great ones using examples in my upcoming post “Ten Extras for Great API Documentation”.

Error messages

When something goes wrong during development, fixing the problem without detailed documentation can become a frustrating and time-consuming process. To make this process as smooth as possible, error messages should help developers understand:

  • what the problem is;
  • whether the error stems from their code or from the use of the API;
  • and how to fix the problem.

All possible errors—including edge cases—should be documented with error-codes or brief, human-readable information in error messages. Error messages should not only contain information related to that specific call, but also address universal topics like authentication or HTTP requests and other conditions not controlled by the API (like request timeout or unknown server error).

This post from Box discusses best practices for server-side error handling and communication, such as returning an HTTP status code that closely matches the error condition, human-readable error messages, and machine-readable error codes.

Quickstart guide

Newcomers starting to implement your API face many obstacles:

  • They are at the beginning of a steep learning curve
  • They might not be familiar with the structure, domain, and ideas behind your API
  • It’s difficult for them to figure out where to start.

If you don’t make the learning process easier for them, they can feel overwhelmed and refrain from delving into your API. 

Many developers learn best by doing, so a quickstart guide is a great option. The guide should be short and simple, aimed at newcomers, and list the minimum number of steps required to complete a meaningful task (e.g., downloading the SDK and saving one object to the platform). Quickstart guides usually have to include information about the domain and introduce domain-related expressions and methods in more detail. It’s safest to assume that the developer has never before heard of your service.

Stripe’s and Braintree’s quickstart guides are great examples; both provide an overview of the most likely tasks you’ll want to perform with the API, as well as link you to the relevant information. They also contain links to contact someone if you need help.

Tutorials

Tutorials are step-by-step walkthroughs covering specific functionality developers can implement with your API, like SMS notifications, account verification, etc.

Tutorials for APIs should follow the best practices for writing any kind of step-by-step help. Each step should contain all the information needed at that point—and nothing more. This way users can focus on the task at hand and won’t be overloaded with information they don’t need.

The description of steps should be easy to follow and concise. Clarity and brevity support the learning process, and are a best practice for all kinds of documentation. Avoid jargon, if possible; users will be learning domain-related language and new technology, and jargon can instill confusion. Help them by making all descriptions as easy to understand as possible. 

The walkthrough should be the smallest possible chunk that lets the user finish a task. If a process is too complex, think about breaking it down into smaller chunks. This makes sure that users can get the help they need without going through steps they’re not interested in.

Twilio’s tutorials explain the most-likely use cases with sample apps in a wide variety of programming languages and frameworks. Universal topics

To implement your API, there are some larger topics that developers will need to know about, for example:

  • Authentication. Handled differently by each type of API, authentication (e.g., OAuth) is often a complicated and error-prone process. Explain how to get credentials, how they are passed on to the server, and show how API keys work with sample code.
  • Error handling. For now, error handling hasn’t been standardized, so you should help developers understand how your API passes back error information, why an error occurs, and how to fix it.
  • HTTP requests. You may have to document HTTP-related information as well, like content types, status codes, and caching.

Dedicate a separate section to explaining these topics, and link to this section from each related API call. This way you can make sure that developers clearly see how your API handles these topics and how API calls change behavior based on them. 

Layout and navigation

Layout and navigation are essential to user experience, and although there is no universal solution for all API docs, there are some best practices that help users interact with the material.

Dynamic layout

Most good examples of API documentation use a dynamic layout as it makes navigation easier for users than static layouts when looking for specific topics in extensive documentation. Starting with a scalable dynamic layout will also make sure you can easily expand your docs, as needed.

Single page design

If your API documentation isn’t huge, go with a single page design that lets users see the overall structure at first sight. Introduce the details from there. Long, single page docs also make it possible for readers to use the browser’s search functionality.

Stripe managed to present extensive documentation in an easy to navigate single page. Persistent navigation

Keep navigation visible at all times. Users don’t want to scroll looking for a navigation bar that disappeared.

Multi-column layout

2- or 3-column layouts have the navigation on the left and information and examples on the right. They make comprehension easier by showing endpoints and examples in context.

Clearbit’s three-column layout displays persistent navigation (table of contents) on the left, references in the middle, and code examples on the right. Syntax highlighter

Improving the readability of samples with syntax highlighting makes the code easier to understand.

The syntax highlighter in action on Plaid’s API documentation site.

If you’d like to start experimenting with a layout for your docs, you might want to check out some free and open source API documentation generators.

To learn about the pros and cons of different approaches to organizing your API docs in the context of developer portals, this is an excellent article by Nordic APIs.

Editing

All writing that you publish should go through an editing process. This is common sense for articles and other publications, but it’s just as essential for technical documentation.

The writers of your API docs should aim for clarity and brevity, confirm that all the necessary information is there, and that the structure is logical and topics aren’t diluted with unnecessary content. 

Editors should proofread your documentation to catch grammar mistakes, errors, and any parts that might be hard to read or difficult to understand. They should also check the docs against your style guide for technical documentation and suggest changes, if needed.

Once a section of documentation is ready to be published, it’s a good idea to show it to people in your target audience, especially any developers who haven’t worked on the documentation themselves. They can catch inconsistencies and provide insight into what’s missing.

Although the editing process can feel like a burden when you have to focus on so many other aspects of your API, a couple of iterations can make a huge difference in the final copy and the impression you make.

Keep it up-to-date

If your API documentation is out of date, users will get frustrated by bumping into features that aren’t there anymore and new ones that lack documentation. This can quickly diminish the trust you established by putting so much work into your documentation in the first place.

When maintaining your API docs, you should keep an eye on the following aspects:

  • Deprecated features. Remove documentation for deprecated features and explain why they were deprecated.
  • New features. Document new features before launch, and make sure there’s enough time planned for the new content to go through the editorial process.
  • Feedback. Useful feedback you get from support, or analytics should be reflected in your docs. Chances are you can’t make your docs perfect at the first try, but based on what users are saying, you can improve them continuously.

For all this to work, you will have to build a workflow for maintaining your documentation. Think about checkpoints and processes for the above mentioned aspects, editing, and publication. It also helps if you can set up a routine for reviewing your docs regularly (e.g. quarterly).

Following these best practices, you can build a solid foundation for your API documentation that can be continuously improved upon as you gain more insight into how users interact with them. Stay tuned for part two, where I give you some tips on how to turn good API docs into amazing ones.

Categories: thinktime

Selling confusion

Seth Godin - Tue 19th Sep 2017 17:09
Over the last few decades, there's been a consistent campaign to sow confusion around evolution, vaccines and climate change. In all three areas, we all have access to far more data, far more certainty and endless amounts of proof that...        Seth Godin
Categories: thinktime

Beware of false averages

Seth Godin - Mon 18th Sep 2017 18:09
Some people like really spicy food. Some people like bland food. Building a restaurant around sorta spicy food doesn't make either group happy. It's tempting to look at pop music, network TV and the latest hot fashion and come to...        Seth Godin
Categories: thinktime

OpenSTEM: Those Dirty Peasants!

Planet Linux Australia - Mon 18th Sep 2017 09:09
It is fairly well known that many Europeans in the 17th, 18th and early 19th centuries did not follow the same routines of hygiene as we do today. There are anecdotal and historical accounts of people being dirty, smelly and generally unhealthy. This was particularly true of the poorer sections of society. The epithet “those […]
Categories: thinktime

Selfish marketing doesn't last

Seth Godin - Sun 17th Sep 2017 18:09
If it helps you, not the customer, why should she care? Sometimes there's an overlap between your selfish needs and hers, but you can save everyone a lot of time and hassle if you begin and end with a focus...        Seth Godin
Categories: thinktime

Pages

Subscribe to kattekrab aggregator - thinktime