You are here


Jonathan Adamczewski: A little bit of floating point in a memory allocator — Part 1: Background

Planet Linux Australia - Sat 06th Jan 2018 17:01

This post contains the same material as this thread of tweets, with a few minor edits.

Over my holiday break at the end of 2017, I took a look into the TLSF (Two Level Segregated Fit) memory allocator to better understand how it works. I’ve made use of this allocator and have been impressed by its real world performance, but never really done a deep dive to properly understand it.

The mapping_insert() function is a key part of the allocator implementation, and caught my eye. Here’s how that function is described in the paper A constant-time dynamic storage allocator for real-time systems:

I’ll be honest: from that description, I never developed a clear picture in my mind of what that function does.

(Reading it now, it seems reasonably clear – but I can say that only after I spent quite a bit of time using other methods to develop my understanding)

Something that helped me a lot was by looking at the implementation of that function from  There’s a bunch of long-named macro constants in there, and a few extra implementation details. If you collapse those it looks something like this:

void mapping_insert(size_t size, int* fli, int* sli) { int fl, sl; if (size < 256) { fl = 0; sl = (int)size / 8; } else { fl = fls(size); sl = (int)(size >> (fl - 5)) ^ 0x20; fl -= 7; } *fli = fl; *sli = sl; }

It’s a pretty simple function (it really is). But I still failed to *see* the pattern of results that would be produced in my mind’s eye.

I went so far as to make a giant spreadsheet of all the intermediate values for a range of inputs, to paint myself a picture of the effect of each step :) That helped immensely.

Breaking it down…

There are two cases handled in the function: one for when size is below a certain threshold, and on for when it is larger. The first is straightforward, and accounts for a small number of possible input values. The large size case is more interesting.

The function computes two values: fl and sl, the first and second level indices for a lookup table. For the large case, fl (where fl is “first level”) is computed via fls(size) (where fls is short for “find last set” – similar names, just to keep you on your toes).

fls() returns the index of the largest bit set, counting from the least significant slbit, which is the index of the largest power of two. In the words of the paper:

“the instruction fls can be used to compute the ⌊log2(x)⌋ function”

Which is, in C-like syntax: floor(log2(x))

And there’s that “fl -= 7” at the end. That will show up again later.

For the large case, the computation of sl has a few steps:

  sl = (size >> (fl – 5)) ^ 0x20;

Depending on shift down size by some amount (based on fl), and mask out the sixth bit?

(Aside: The CellBE programmer in me is flinching at that variable shift)

It took me a while (longer than I would have liked…) to realize that this
size >> (fl – 5) is shifting size to generate a number that has exactly six significant bits, at the least significant end of the register (bits 5 thru 0).

Because fl is the index of the most significant bit, after this shift, bit 5 will always be 1 – and that “^ 0x20” will unset it, leaving the result as a value between 0 and 31 (inclusive).

So here’s where floating point comes into it, and the cute thing I saw: another way to compute fl and sl is to convert size into an IEEE754 floating point number, and extract the exponent, and most significant bits of the mantissa. I’ll cover that in the next part.

Categories: thinktime

Jonathan Adamczewski: A little bit of floating point in a memory allocator — Part 2: The floating point

Planet Linux Australia - Sat 06th Jan 2018 17:01


This post contains the same material as this thread of tweets, with a few minor edits.

In IEEE754, floating point numbers are represented like this:


nnn is the exponent, which is floor(log2(size)) — which happens to be the fl value computed by TLSF.

sss… is the significand fraction: the part that follows the decimal point, which happens to be sl.

And so to calculate fl and sl, all we need to do is convert size to a floating point value (on recent x86 hardware, that’s a single instruction). Then we can extract the exponent, and the upper bits of the fractional part, and we’re all done :D

That can be implemented like this:

double sf = (int64_t)size; uint64_t sfi; memcpy(&sfi, &sf, 8); fl = (sfi >> 52) - (1023 + 7); sl = (sfi >> 47) & 31;

There’s some subtleties (there always is). I’ll break it down…

double sf = (int64_t)size;

Convert size to a double, with an explicit cast. size has type size_t, but using TLSF from, the largest supported allocation on 64bit architecture is 2^32 bytes – comfortably less than the precision provided by the double type. If you need your TLSF allocator to allocate chunks bigger than 2^53, this isn’t the technique for you :)

I first tried using float (not double), which can provide correct results — but only if the rounding mode happens to be set correctly. double is easier.

The cast to (int64_t) results in better codegen on x86: without it, the compiler will generate a full 64bit unsigned conversion, and there is no single instruction for that.

The cast tells the compiler to (in effect) consider the bits of size as if they were a two’s complement signed value — and there is an SSE instruction to handle that case (cvtsi2sdq or similar). Again, with the implementation we’re using size can’t be that big, so this will do the Right Thing.

uint64_t sfi; memcpy(&sfi, &sf, 8);

Copy the 8 bytes of the double into an unsigned integer variable. There are a lot of ways that C/C++ programmers copy bits from floating point to integer – some of them are well defined :) memcpy() does what we want, and any moderately respectable compiler knows how to select decent instructions to implement it.

Now we have floating point bits in an integer register, consisting of one sign bit (always zero for this, because size is always positive), eleven exponent bits (offset by 1023), and 52 bits of significant fraction. All we need to do is extract those, and we’re done :)

fl = (sfi >> 52) - (1023 + 7);

Extract the exponent: shift it down (ignoring the always-zero sign bit), subtract the offset (1023), and that 7 we saw earlier, at the same time.

sl = (sfi >> 47) & 31;

Extract the five most significant bits of the fraction – we do need to mask out the exponent.

And, just like that*, we have mapping_insert(), implemented in terms of integer -> floating point conversion.

* Actual code (rather than fragments) may be included in a later post…

Categories: thinktime

Ben Martin: That gantry just pops right off

Planet Linux Australia - Sat 06th Jan 2018 00:01
Hobby CNC machines sold as "3040" may have a gantry clearance of about 80mm and a z axis travel of around 55mm. A detached gantry is shown below. Notice that there are 3 bolts on the bottom side mounting the z-axis to the gantry. The stepper motor attaches on the side shown so there are 4 NEMA holes to hold the stepper. Note that the normal 3040 doesn't have the mounting plate shown on the z-axis, that crossover plate allows a different spindle to be mounted to this machine.

The plan is to create replacement sides with some 0.5inch offcut 6061 alloy. This will add 100mm to the gantry so it can more easily clear clamps and a 4th axis. Because that would move the cutter mount upward as well, replacing the z-axis with something that has more range, say 160mm becomes an interesting plan.

One advantage to upgrading a machine like this is that you can reassemble the machine after measuring and designing the upgrade and then cut replacement parts for the machine using the machine.

The 3040 can look a bit spartan with the gantry removed.

The preliminary research is done. Designs created. CAM done. I just have to cut 4 plates and then the real fun begins.

Categories: thinktime

Your theory

Seth Godin - Fri 05th Jan 2018 19:01
Of course, you have one. We all do. A theory about everything. You're waiting for 7:20 train into the city. Your theory is that every day, the train comes and brings you to work. Today, the train doesn't come. That's...        Seth Godin
Categories: thinktime

Discovery on a Budget: Part I

a list apart - Fri 05th Jan 2018 02:01

If you crack open any design textbook, you’ll see some depiction of the design cycle: discover, ideate, create, evaluate, and repeat. Whenever we bring on a new client or start working on a new feature, we start at the top of the wheel with discover (or discovery). It is the time in the project when we define what problem we are trying to solve and what our first approach at solving it should be.

Ye olde design cycle

We commonly talk about discovery at the start of a sprint cycle at an established business, where there are things like budgets, product teams, and existing customers. The discovery process may include interviewing stakeholders or pouring over existing user data. And we always exit the discovery phase with some sort of idea to move forward with.

However, discovery is inherently different when you work at a nonprofit, startup, or fledgling small business. It may be a design team of one (you), with zero dollars to spend, and only a handful of people aware the business even exists. There are no clients to interview and no existing data to examine. This may also be the case at large businesses when they want to test the waters on a new direction without overcommitting (or overspending). Whenever you are constrained on budget, data, and stakeholders, you need to be flexible and crafty in how you conduct discovery research. But you can’t skimp on rigor and thoroughness. If the idea you exit the discovery phase with isn’t any good, your big launch could turn out to be a business-ending flop.

In this article I’ll take you through a discovery research cycle, but apply it towards a (fictitious) startup idea. I’ll introduce strategies for conducting discovery research with no budget, existing user data, or resources to speak of. And I’ll show how the research shapes the business going forward.

Write up the problem hypothesis

An awful lot of ink (virtual or otherwise) has been spent on proclaiming we should all, “fall in love with the problem, not the solution.” And it has been ink spent well. When it comes to product building, a problem-focused philosophy is the cornerstone of any user-centric business.

But how, exactly, do you know when you have a problem worth solving? If you work at a large, established business you may have user feedback and data pointing you like flashing arrows on a well-marked road towards a problem worth solving. However, if you are launching a startup, or work at a larger business venturing into new territory, it can be more like hiking through the woods and searching for the next blaze mark on the trail. Your ideas are likely based on personal experiences and gut instincts.

When your ideas are based on personal experiences, assumptions, and instincts, it’s important to realize they need a higher-than-average level of tire-kicking. You need to evaluate the question “Do I have a problem worth solving?” with a higher level of rigor than you would at a company with budget to spare and a wealth of existing data. You need to take all of your ideas and assumptions and examine them thoroughly. And the best way to examine your ideas and categorize your assumptions is with a hypothesis.

As the dictionary describes, a hypothesis is “a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.” That also serves as a good description of why we do discovery research in the first place. We may have an idea that there is a problem worth solving, but we don’t yet know the scope or critical details. Articulating our instincts, ideas, and assumptions as a problem hypothesis lays a foundation for the research moving forward.

Here is a general formula you can use to write a problem hypothesis:

Because [assumptions and gut instincts about the problem], users are [in some undesirable state]. They need [solution idea].

For this article, I decided to “launch” a fictitious (and overly ambitious) startup as an example. Here is the problem hypothesis I wrote for my startup:

Because their business model relies on advertising, social media tools like Facebook are deliberately designed to “hook” users and make them addicted to the service. Users are unhappy with this and would rather have a healthier relationship with social media tools. They would be willing to pay for a social media service that was designed with mental health in mind.

You can see in this example that my assumptions are:

  • Users feel that social media sites like Facebook are addictive.
  • Users don’t like to be addicted to social media.
  • Users would be willing to pay for a non-addictive Facebook replacement.

These are the assumptions I’ll be researching and testing throughout the discovery process. If I find through my research that I cannot readily affirm these assumptions, it means I might not be ready to take on Mr. Zuckerberg just yet.

The benefit of articulating our assumptions in the form of a hypothesis is that it provides something concrete to talk about, refer to, and test. The whole product team can be involved in forming the initial problem hypothesis, and you can refer back to it throughout the discovery process. Once we’ve completed the research and analyzed the results, we can edit the hypothesis to reflect our new understanding of our users and the problems we want to solve.

Now that we’ve articulated a problem hypothesis, it is time to figure out our research plan. In the following two sections, I’ll cover the research method I recommend the most for new ventures, as well as strategies for recruiting participants on a budget.

A method that is useful in all phases of design: interviews

In my career as a user researcher, I have used all sorts of methods. I’ve done A/B testing, eye tracking, Wizard of Oz testing, think-alouds, contextual inquiries, and guerilla testing. But the one research method I utilize the most, and that I believe provides the most “bang for the buck,” is user interviews.

User interviews are relatively inexpensive to conduct. You don’t need to travel to a client site and you don’t need a fortune’s worth of equipment. If you have access to a phone, you can conduct an interview with participants all around the world. Yet interviews provide a wealth of information and can be used in every phase of research and design. Interviews are especially useful in discovery, because it is a method that is adaptable. As you learn more about the problem you are trying to solve, you can adapt your interview protocol to match.

To be clear, your interviewees will not tell you:

  • what to build;
  • or how to build it.

But they absolutely can tell you:

  • what problem they have;
  • how they feel about it;
  • and what the value of a solution would mean to them.

And if you know the problem, how users feels about it, and the value of a solution, you are well on your way to designing the right product.

The challenge of conducting a good user interview is making sure you ask the questions that elicit that information. Here are a couple tips:

Tip 1: always ask the following two questions:

  • “What do you like about [blank]?”
  • “What do you dislike about [blank]?”

… where you fill “[blank]” with whatever domain your future product will improve.

Your objective is to gain an understanding of all aspects of the problem your potential customers face—the bad and the good. One common mistake is to spend too much time investigating what’s wrong with the current state of affairs. Naturally, you want your product to fix all the problems your customers face. However, you also need to preserve what currently works well, what is satisfying, or what is otherwise good about how users accomplish their goals currently. So it is important to ask about both in user interviews.

For example, in my interviews I always asked, “What do you like about using Facebook?” And it wasn’t until my interview participant told me everything they enjoyed about Facebook that I would ask, “What do you dislike about using Facebook?”

Tip 2: after (nearly) every response, ask them to say more.

The goal of conducting interviews is to gain an exhaustive set of data to review and consider moving forward. That means you don’t want your participants to discuss one thing they like and dislike, you want them to tell you all the things they like and dislike.

Here is an example of how this played out in one of the interviews I conducted:

Interviewer (Me): What do you like about using Facebook?

Interviewee: I like seeing people on there that I wouldn’t otherwise get a chance to see and catch up with in real life. I have moved a couple times so I have a lot of friends that I don’t see regularly. I also like seeing the people I know do well, even though I haven’t seen them since, maybe, high school. But I like seeing how their life has gone. I like seeing their kids. I like seeing their accomplishments. It’s also a little creepy because it’s a window into their life and we haven’t actually talked in forever. But I like staying connected.

Interviewer (Me): What else do you like about it?

Interviewee: Um, well it’s also sort of a convenient way of keeping contacts. There have been a few times when I was able to message people and get in touch with people even when I don’t have their address or email in my phone. I could message them through Facebook.

Interviewer (Me): Great. Is there anything else you like about it?

Interviewee: Let me think … well I also find cool stuff to do on the weekends there sometimes. They have an events feature. And businesses, or local places, will post events and there have been a couple times where I’ve gone to something cool. Like I found a cool movie festival once that way.

Interviewer (Me): That seems cool. What else do you like about using Facebook?

Interviewee: Uh … that’s all I think I really use it for. I can’t really think of anything else. Mainly I use it just to keep in touch with people that I’ve met over the years.

From this example you can see the first feature that popped into the interviewee’s mind was their ability to keep up with friends that they otherwise wouldn’t have much opportunity to connect with anymore. That is a feature that any Facebook replacement would have to replicate. However, if I hadn’t pushed the interviewee to think of even more features they like, I might have never uncovered an important secondary feature: convenient in-app messaging. In fact, six out of the eleven people I interviewed for this project said they liked Facebook Messenger. But not a single one of them mentioned that feature first. It only came up in conversation after I probed for more.

As I continued to repeat my question, the interviewee thought of one more feature they liked: local event listings. (Five out of the eleven people I interviewed mentioned this feature.) But after that, the interviewee couldn’t think of any more features to discuss. You know you can move on to the next question in the interview when your participant starts to repeat themselves or bluntly tells you they have nothing else to say.

Recruit all around you, then document the bias

There are all sorts of ways to recruit participants for research. You can hire an agency or use a tool like But many of those paid-for options can be quite costly, and since we are working with a shoestring budget we have roughly zero dollars to spend on recruitment. We will have to be creative.

My post on Facebook to recruit volunteers. One volunteer decided to respond with a Hunger Games “I volunteer as tribute!” gif.

For my project, I decided to rely on the kindness of friends and strangers I could reach through Facebook. I posted one request for participants on my personal Facebook page, and another on the local FreeCodeCamp page. A day after I posted my request, twenty-five friends and five strangers volunteered. This type of participant recruitment method is called convenience sampling, because I was recruiting participants that were conveniently accessible to me.

Since my project involved talking to people about social media sites like Facebook, it was appropriate for my first attempt at recruiting to start on Facebook. I could be sure that everyone who saw my request uses Facebook in some form or fashion. However, like all convenience sampling, my recruitment method was biased. (I’ll explain how in just a bit.)

Bias is something that we should try—whenever possible—to avoid. If we have access to more sophisticated recruitment methods, we should use them. However, when you have a tight budget, avoiding recruitment bias is virtually impossible. In this scenario, our goals should be to:

  • mitigate bias as best we can;
  • and document all the biases we see.

For my project, I could mitigate some of the biases by using a few more recruitment methods. I could go to various neighborhoods and try to recruit participants off the street (i.e., guerilla testing). If I had a little bit of money to spend, I could hang out in various coffee shops and offer folks free coffee in exchange for ten-minute interviews. These recruitment methods also fall under the umbrella of convenience sampling, but by using a variety of methods I can mitigate some of the bias I would have from using just one of them.

Also, it is always important to reflect on and document how your sampling method is biased. For my project, I wrote the following in my notes:

All of the people I interviewed were connected to me in some way on Facebook. Many of them I know well enough to be “friends” with. All of them were around my age, many (but not all) worked in tech in some form or fashion, and all of them but one lived in the US.

Documenting bias ensures that we won’t forget about the bias when it comes time to analyze and discuss the results.

Let’s keep this going

As the title suggests, this is just the first installment of a series of articles on the discovery process. In part two, I will analyze the results of my interviews, revise my problem hypothesis, and continue to work on my experimental startup. I will launch into another round of discovery research, but this time utilizing some different research methods, like A/B testing and fake-door testing. You can help me out by checking out this mock landing page for Candor Network (what I’ve named my fictitious startup) and taking the survey you see there.

Categories: thinktime

Why we don't have nice things

Seth Godin - Thu 04th Jan 2018 20:01
The creation of worthwhile work is a duet. The creator has to do her part, but so does the customer. One of the best airport restaurants I've ever encountered breaks my first rule of airport eating. The sushi bar at...        Seth Godin
Categories: thinktime

Pia Waugh: Pivoting ‘the book’ from individuals to systems

Planet Linux Australia - Thu 04th Jan 2018 11:01

In 2016 I started writing a book, “Choose Your Own Adventure“, which I wanted to be a call to action for individuals to consider their role in the broader system and how they individually can make choices to make things better. As I progressed the writing of that book I realised the futility of changing individual behaviours and perspectives without an eye to the systems and structures within which we live. It is relatively easy to focus on oneself, but “no man is an island” and quite simply, I don’t want to facilitate people turning themselves into more beautiful cogs in a dysfunctional machine so I’m pivoting the focus of the book (and reusing the relevant material) and am now planning to finish the book by mid 2018.

I have recently realised four paradoxes which have instilled in me a sense of urgency to reimagine the world as we know it. I believe we are at a fork in the road where we will either reinforce legacy systems based on outdated paradigms with shiny new things, or choose to forge a new path using the new tools and opportunities at our disposal, hopefully one that is genuinely better for everyone. To do the latter, we need to critically assess the systems and structures we built and actively choose what we want to keep, what we should discard, what sort of society we want in the future and what we need to get there.

I think it is too easily forgotten that we invented all this and can therefore reinvent it if we so choose. But to not make a choice is to choose the status quo.

This is not to say I think everything needs to change. There is nothing is so simplistic or misleading as a zero sum argument. Rather, the intent of this book is to challenge you to think critically about the systems you work within, whether they enable or disable the things you think are important, and most importantly, to challenge you to imagine what sort of world you want to see. Not just for you, but for your family, community and the broader society. I challenge you all to make 2018 a year of formative creativity in reimagining the world we live in and how we get there.

The paradoxes in brief, are as follows:

  • That though power is more distributed than ever, most people are still struggling to survive.
    It has been apparent to me for some time that there is a growing substantial shift in power from traditional gatekeepers to ordinary people through the proliferation of rights based philosophies and widespread access to technology and information. But the systemic (and artificial) limitations on most people’s time and resources means most people simply cannot participate fully in improving their own lives let alone in contributing substantially to the community and world in which they live. If we consider the impact of business and organisational models built on scarcity, centricity and secrecy, we quickly see that normal people are locked out of a variety of resources, tools and knowledge with which they could better their lives. Why do we take publicly funded education, research and journalism and lock them behind paywalls and then blame people for not having the skills, knowledge or facts at their disposal? Why do we teach children to be compliant consumers rather than empowered makers? Why do we put the greatest cognitive load on our most vulnerable through social welfare systems that then beget reliance? Why do we not put value on personal confidence in the same way we value business confidence, when personal confidence indicates the capacity for individuals to contribute to their community? Why do we still assume value to equate quantity rather than quality, like the number of hours worked rather than what was done in those hours? If a substantial challenge of the 21st century is having enough time and cognitive load to spare, why don’t we have strategies to free up more time for more people, perhaps by working less hours for more return? Finally, what do we need to do systemically to empower more people to move beyond survival and into being able to thrive.
  • Substantial paradigm shifts have happened but are not being integrated into people’s thinking and processes.
    The realisation here is that even if people are motivated to understand something fundamentally new to their worldview, it doesn’t necessarily translate into how they behave. It is easier to improve something than change it. Easier to provide symptomatic relief than to cure the disease. Interestingly I often see people confuse iteration for transformation, or symptomatic relief with addressing causal factors, so perhaps there is also a need for critical and systems thinking as part of the general curriculum. This is important because symptomatic relief, whilst sometimes necessary to alleviate suffering, is an effort in chasing one’s tail and can often perpetrate the problem. For instance, where providing foreign aid without mitigating displacement of local farmer’s efforts can create national dependence on further aid. Efforts to address causal factors is necessary to truly address a problem. Even if addressing the causal problem is outside your influence, then you should at least ensure your symptomatic relief efforts are not built to propagate the problem. One of the other problems we face, particularly in government, is that the systems involved are largely products of centuries old thinking. If we consider some of the paradigm shifts of our times, we have moved from scarcity to surplus, centralised to distributed, from closed to openness, analog to digital and normative to formative. And yet, people still assume old paradigms in creating new policies, programs and business models. For example how many times have you heard someone talk about innovative public engagement (tapping into a distributed network of expertise) by consulting through a website (maintaining central decision making control using a centrally controlled tool)? Or “innovation” being measured (and rewarded) through patents or copyright, both scarcity based constructs developed centuries ago? “Open government” is often developed by small insular teams through habitually closed processes without any self awareness of the irony of the approach. And new policy and legislation is developed in analog formats without any substantial input from those tasked with implementation or consideration with how best to consume the operating rules of government in the systems of society. Consider also the number of times we see existing systems assumed to be correct by merit of existing, without any critical analysis. For instance, a compliance model that has no measurable impact. At what point and by what mechanisms can we weigh up the merits of the old and the new when we are continually building upon a precedent based system of decision making? If 3D printing helped provide a surplus economy by which we could help solve hunger and poverty, why wouldn’t that be weighed up against the benefits of traditional scarcity based business models?
  • That we are surrounded by new things every day and yet there is a serious lack of vision for the future
    One of the first things I try to do in any organisation is understand the vision, the strategy and what success should look like. In this way I can either figure out how to best contribute meaningfully to the overarching goal, and in some cases help grow or develop the vision and strategy to be a little more ambitious. I like to measure progress and understand the baseline from which I’m trying to improve but I also like to know what I’m aiming for. So, what could an optimistic future look like for society? For us? For you? How do you want to use the new means at our disposal to make life better for your community? Do we dare imagine a future where everyone has what they need to thrive, where we could unlock the creative and intellectual potential of our entire society, a 21st century Renaissance, rather than the vast proportion of our collective cognitive capacity going into just getting food on the table and the kids to school. Only once you can imagine where you want to be can we have a constructive discussion where we want to be collectively, and only then can we talk constructively the systems and structures we need to support such futures. Until then, we are all just tweaking the settings of a machine build by our ancestors. I have been surprised to find in government a lot of strategies without vision, a lot of KPIs without measures of success, and in many cases a disconnect between what a person is doing and the vision or goals of the organisation or program they are in. We talk “innovation” a lot, but often in the back of people’s minds they are often imagining a better website or app, which isn’t much of a transformation. We are surrounded by dystopic visions of the distant future, and yet most government vision statements only go so far as articulating something “better” that what we have now, with “strategies” often focused on shopping lists of disconnected tactics 3-5 years into the future. The New Zealand Department of Conservation provides an inspiring contrast with a 50 year vision they work towards, from which they develop their shorter term stretch goals and strategies on a rolling basis and have an ongoing measurable approach.
  • That government is an important part of a stable society and yet is being increasingly undermined, both intentionally and unintentionally.
    The realisation here has been in first realising how important government (and democracy) is in providing a safe, stable, accountable, predictable and prosperous society whilst simultaneously observing first hand the undermining and degradation of the role of government both intentionally and unintentionally, from the outside and inside. I have chosen to work in the private sector, non-profit community sector, political sector and now public sector, specifically because I wanted to understand the “system” in which I live and how it all fits together. I believe that “government” – both the political and public sectors – has a critical part to play in designing, leading and implementing a better future. The reason I believe this, is because government is one of the few mechanisms that is accountable to the people, in democratic countries at any rate. Perhaps not as much as we like and it has been slow to adapt to modern practices, tools and expectations, but governments are one of the most powerful and influential tools at our disposal and we can better use them as such. However, I posit that an internal, largely unintentional and ongoing degradation of the public sectors is underway in Australia, New Zealand, the United Kingdom and other “western democracies”, spurred initially by an ideological shift from ‘serving the public good’ to acting more like a business in the “New Public Service” policy shift of the 1980s. This was useful double speak for replacing public service values with business values and practices which ignores the fact that governments often do what is not naturally delivered by the marketplace and should not be only doing what is profitable. The political appointment of heads of departments has also resulted over time in replacing frank, fearless and evidence based leadership with politically palatable compromises throughout the senior executive layer of the public sector, which also drives necessarily secretive behaviour, else the contradictions be apparent to the ordinary person. I see the results of these internal forms of degradations almost every day. From workshops where people under budget constraints seriously consider outsourcing all government services to the private sector, to long suffering experts in the public sector unable to sway leadership with facts until expensive consultants are brought in to ask their opinion and sell the insights back to the department where it is finally taken seriously (because “industry” said it), through to serious issues where significant failures happen with blame outsourced along with the risk, design and implementation, with the details hidden behind “commercial in confidence” arrangements. The impact on the effectiveness of the public sector is obvious, but the human cost is also substantial, with public servants directly undermined, intimidated, ignored and a growing sense of hopelessness and disillusionment. There is also an intentional degradation of democracy by external (but occasionally internal) agents who benefit from the weakening and limiting of government. This is more overt in some countries than others. A tension between the regulator and those regulated is a perfectly natural thing however, as the public sector grows weaker the corporate interests gain the upper hand. I have seen many people in government take a vendor or lobbyist word as gold without critical analysis of the motivations or implications, largely again due to the word of a public servant being inherently assumed to be less important than that of anyone in the private sector (or indeed anyone in the Minister’s office). This imbalance needs to be addressed if the public sector is to play an effective role. Greater accountability and transparency can help but currently there is a lack of common agreement on the broader role of government in society, both the political and public sectors. So the entire institution and the stability it can provide is under threat of death by a billion papercuts. Efforts to evolve government and democracy have largely been limited to iterations on the status quo: better consultation, better voting, better access to information, better services. But a rethink is required and the internal systemic degradations need to be addressed.

If you think the world is perfectly fine as is, then you are probably quite lucky or privileged. Congratulations. It is easy to not see the cracks in the system when your life is going smoothly, but I invite you to consider the cracks that I have found herein, to test your assumptions daily and to leave your counter examples in the comments below.

For my part, I am optimistic about the future. I believe the proliferation of a human rights based ideology, participatory democracy and access to modern technologies all act to distribute power to the people, so we have the capacity more so than ever to collectively design and create a better future for us all.

Let’s build the machine we need to thrive both individually and collectively, and not just be beautiful cogs in a broken machine.

Categories: thinktime

Pia Waugh: Chapter 1.2: Many hands make light work, for a while

Planet Linux Australia - Thu 04th Jan 2018 11:01

This is part of a book I am working on, hopefully due for completion by mid 2018. The original purpose of the book is to explore where we are at, where we are going, and how we can get there, in the broadest possible sense. Your comments, feedback and constructive criticism are welcome! The final text of the book will be freely available under a Creative Commons By Attribution license. A book version will be sent to nominated world leaders, to hopefully encourage the necessary questioning of the status quo and smarter decisions into the future. Additional elements like references, graphs, images and other materials will be available in the final digital and book versions and draft content will be published weekly. Please subscribe to the blog posts by the RSS category and/or join the mailing list for updates.

Back to the book overview or table of contents for the full picture. Please note the pivot from focusing just on individuals to focusing on the systems we live in and the paradoxes therein.

“Differentiation of labour and interdependence of society is reliant on consistent and predictable authorities to thrive” — Durkheim

Many hands makes light work is an old adage both familiar and comforting. One feels that if things get our of hand we can just throw more resources at the problem and it will suffice. However we have made it harder on ourselves in three distinct ways:

  • by not always recognising the importance of interdependence and the need to ensure the stability and prosperity of our community as a necessary precondition to the success of the individuals therein;
  • by increasingly making it harder for people to gain knowledge, skills and adaptability to ensure those “many hands” are able to respond to the work required and not trapped into social servitude; and
  • by often failing to recognise whether we need a linear or exponential response in whatever we are doing, feeling secure in the busy-ness of many hands.

Specialisation is when a person delves deep on a particular topic or skill. Over many millennia we have got to the point where we have developed extreme specialisation, supported through interdependence and stability, which gave us the ability to rapidly and increasingly evolve what we do and how we live. This resulted in increasingly complex social systems and structures bringing us to a point today where the pace of change has arguably outpaced our imagination. We see many people around the world clinging to traditions and romantic notions of the past whilst we hurtle at an accelerating pace into the future. Many hands have certainly made light work, but new challenges have emerged as a result and it is more critical than ever that we reimagine our world and develop our resilience and adaptability to change, because change is the only constant moving forward.

One human can survive on their own for a while. A tribe can divide up the labour quite effectively and survive over generations, creating time for culture and play. But when we established cities and states around 6000 years ago, we started a level of unprecedented division of labour and specialisation beyond mere survival. When the majority of your time, energy and resources go into simply surviving, you are largely subject to forces outside your control and unable to justify spending time on other things. But when survival is taken care of (broadly speaking) it creates time for specialisation and perfecting your craft, as well as for leisure, sport, art, philosophy and other key areas of development in society.

The era of cities itself was born on the back of an agricultural technology revolution that made food production far more efficient, creating surplus (which drove a need for record keeping and greater proliferation of written language) and prosperity, with a dramatic growth in specialisation of jobs. With greater specialisation came greater interdependence as it becomes in everyone’s best interests to play their part predictably. A simple example is a farmer needing her farming equipment to be reliable to make food, and the mechanic needs food production to be reliable for sustenance. Both rely on each other not just as customers, but to be successful and sustainable over time. Greater specialisation led to greater surplus as specialists continued to fine tune their crafts for ever greater outcomes. Over time, an increasing number of people were not simply living day to day, but were able to plan ahead and learn how to deal with major disruptions to their existence. Hunters and gatherers are completely subject to the conditions they live in, with an impact on mortality, leisure activities largely fashioned around survival, small community size and the need to move around. With surplus came spare time and the ability to take greater control over one’s existence and build a type of systemic resilience to change.

So interdependence gave us greater stability, as a natural result of enlightened self interest writ large where ones own success is clearly aligned with the success of the community where one lives. However, where interdependence in smaller communities breeds a kind of mutual understanding and appreciation, we have arguably lost this reciprocity and connectedness in larger cities today, ironically where interdependence is strongest. When you can’t understand intuitively the role that others play in your wellbeing, then you don’t naturally appreciate them, and disconnected self interest creates a cost to the community. When community cohesion starts to decline, eventually individuals also decline, except the small percentage who can either move communities or who benefit, intentionally or not, on the back of others misfortune.

When you have no visibility of food production beyond the supermarket then it becomes easier to just buy the cheapest milk, eggs or bread, even if the cheapest product is unsustainable or undermining more sustainably produced goods. When you have such a specialised job that you can’t connect what you do to any greater meaning, purpose or value, then it also becomes hard to feel valuable to society, or valued by others. We see this increasingly in highly specialised organisations like large companies, public sector agencies and cities, where the individual feels the dual pressure of being anything and nothing all at once.

Modern society has made it somewhat less intuitive to value others who contribute to your survival because survival is taken for granted for many, and competing in ones own specialisation has been extended to competing in everything without appreciation of the interdependence required for one to prosper. Competition is seen to be the opposite of cooperation, whereas a healthy sustainable society is both cooperative and competitive. One can cooperate on common goals and compete on divergent goals, thus making best use of time and resources where interests align. Cooperative models seem to continually emerge in spite of economic models that assume simplistic punishment and incentive based behaviours. We see various forms of “commons” where people pool their resources in anything from community gardens and ’share economies’ to software development and science, because cooperation is part of who we are and what makes us such a successful species.

Increasing specialisation also created greater surplus and wealth, generating increasingly divergent and insular social classes with different levels of power and people becoming less connected to each other and with wealth overwhelmingly going to the few. This pressure between the benefits and issues of highly structured societies and which groups benefit has ebbed and flowed throughout our history but, generally speaking, when the benefits to the majority outweigh the issues for that majority, then you have stability. With stability a lot can be overlooked, including at times gross abuses for a minority or the disempowered. However, if the balances tips too far the other way, then you get revolutions, secessions, political movements and myriad counter movements. Unfortunately many counter movements limit themselves to replacing people rather than the structures that created the issues however, several of these counter movements established some critical ideas that underpin modern society.

Before we explore the rise of individualism through independence and suffrage movements (chapter 1.3), it is worth briefly touching upon the fact that specialisation and interdependence, which are critical for modern societies, both rely upon the ability for people to share, to learn, and to ensure that the increasingly diverse skills are able to evolve as the society evolves. Many hands only make light work when they know what they are doing. Historically the leaps in technology, techniques and specialisation have been shared for others to build upon and continue to improve as we see in writings, trade, oral traditions and rituals throughout history. Gatekeepers naturally emerged to control access to or interpretations of knowledge through priests, academics, the ruling class or business class. Where gatekeepers grew too oppressive, communities would subdivide to rebalance the power differential, such a various Protestant groups, union movements and the more recent Open Source movements. In any case, access wasn’t just about power of gatekeepers. The costs of publishing and distribution grew as societies grew, creating a call from the business class for “intellectual property” controls as financial mechanisms to offset these costs. The argument ran that because of the huge costs of production, business people needed to be incentivised to publish and distribute knowledge, though arguably we have always done so as a matter of survival and growth.

With the Internet suddenly came the possibility for massively distributed and free access to knowledge, where the cost of publishing, distribution and even the capability development required to understand and apply such knowledge was suddenly negligible. We created a universal, free and instant way to share knowledge, creating the opportunity for a compounding effect on our historic capacity for cumulative learning. This is worth taking a moment to consider. The technology simultaneously created an opportunity for compounding our cumulative learning whilst rendered the reasons for IP protections negligible (lowered costs of production and distribution) and yet we have seen a dramatic increase in knowledge protectionism. Isn’t it to our collective benefit to have a well educated community that can continue our trajectory of diversification and specialisation for the benefit of everyone? Anyone can get access to myriad forms of consumer entertainment but our most valuable knowledge assets are fiercely protected against general and free access, dampening our ability to learn and evolve. The increasing gap between the haves and have nots is surely symptomatic of the broader increasing gap between the empowered and disempowered, the makers and the consumers, those with knowledge and those without. Consumers are shaped by the tools and goods they have access to, and limited by their wealth and status. But makers can create the tools and goods they need, and can redefine wealth and status with a more active and able hand in shaping their own lives.

As a result of our specialisation, our interdependence and our cooperative/competitive systems, we have created greater complexity in society over time, usually accompanied with the ability to respond to greater complexity. The problem is that a lot of our solutions to change have been linear responses to an exponential problem space. the assumption that more hands will continue to make light work often ignores the need for sharing skills and knowledge, and certainly ignores where a genuinely transformative response is required. A small fire might be managed with buckets, but at some point of growth, adding more buckets becomes insufficient and new methods are required. Necessity breeds innovation and yet when did you last see real innovation that didn’t boil down to simply more or larger buckets? Iteration is rarely a form of transformation, so it is important to always clearly understand the type of problem you are dealing with and whether the planned response needs to be linear or exponential. If the former, more buckets is probably fine. If the latter, every bucket is just a distraction from developing the necessary response.

Next chapter I’ll examine how the independence movements created the philosophical pre-condition for democracy, the Internet and the dramatic paradigm shifts to follow.

Categories: thinktime

"We don't do rabbits"

Seth Godin - Wed 03rd Jan 2018 20:01
One thing that's often taught in amateur internet marketing school is the idea of keyword stuffing. List every possible thing that someone might want you to do on your website, so if they type that in, they'll find you. It's...        Seth Godin
Categories: thinktime

Today's the day

Seth Godin - Tue 02nd Jan 2018 23:01
The fourth session of The Marketing Seminar is open for enrollment today. No shortcuts, no magic spells, no secrets. Merely an effective, day by day approach to making a difference in the new year. A community of leaders, freelancers, managers...        Seth Godin
Categories: thinktime

Colin Charles: Premier Open Source Database Conference Call for Papers closing January 12 2018

Planet Linux Australia - Tue 02nd Jan 2018 23:01

The call for papers for Percona Live Santa Clara 2018 was extended till January 12 2018. This means you still have time to get a submission in.

Topics of interest: MySQL, MongoDB, PostgreSQL & other open source databases. Don’t forget all the upcoming databases too (there’s a long list at db-engines).

I think to be fair, in the catch all “other”, we should also be thinking a lot about things like containerisation (Docker), Kubernetes, Mesosphere, the cloud (Amazon AWS RDS, Microsoft Azure, Google Cloud SQL, etc.), analytics (ClickHouse, MariaDB ColumnStore), and a lot more. Basically anything that would benefit an audience of database geeks whom are looking at it from all aspects.

That’s not to say case studies shouldn’t be considered. People always love to hear about stories from the trenches. This is your chance to talk about just that.

Categories: thinktime

Craige McWhirter: Resolving a Partitioned RabbitMQ Cluster with JuJu

Planet Linux Australia - Tue 02nd Jan 2018 17:01

On occasion, a RabbitMQ cluster may partition itself. In a JuJu 1.25 environment this can often first present itself as nova-compute services stopping with errors such as these:

ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._sync_power_states: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c ... TRACE nova.openstack.common.periodic_task self._raise_timeout_exception(msg_id) TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/", line 218, in _raise_timeout_exception TRACE nova.openstack.common.periodic_task 'Timed out waiting for a reply to message ID %s' % msg_id) TRACE nova.openstack.common.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c

Merely restarting the stopped nova-compute services will not resolve this issue.

You may also find that querying the rabbitmq service may either not return or take an awful long time to return:

$ sudo rabbitmqctl -p openstack list_queues name messages consumers status

...and in an environment managed by juju, you could also see JuJu trying to correct the RabbitMQ but failing:

$ juju stat --format tabular | grep rabbit rabbitmq-server false local:trusty/rabbitmq-server-128 rabbitmq-server/0 idle 0/lxc/12 5672/tcp rabbitmq-server/1 error idle 1/lxc/8 5672/tcp hook failed: "config-changed" rabbitmq-server/2 error idle 2/lxc/10 5672/tcp hook failed: "config-changed"

You should now run rabbitmqctl cluster_status on each of your rabbit instances and review the output. If the cluster is partitioned, you will see something like the below:

ubuntu@my_juju_lxc:~$ sudo rabbitmqctl cluster_status Cluster status of node 'rabbit@192-168-7-148' ... [{nodes,[{disc,['rabbit@192-168-7-148','rabbit@192-168-7-163', 'rabbit@192-168-7-174']}]}, {running_nodes,['rabbit@192-168-7-174','rabbit@192-168-7-148']}, {partitions,[{'rabbit@192-168-7-174',['rabbit@192-168-7-163']}, {'rabbit@192-168-7-148',['rabbit@192-168-7-163']}]}] ...done.

You can clearly see from the above that there are two partitions for RabbitMQ. We need to now identify which of these is considered the leader:

maas-my_cloud:~$ juju run --service rabbitmq-server "is-leader" - MachineId: 0/lxc/12 Stderr: | Stdout: | True UnitId: rabbitmq-server/0 - MachineId: 1/lxc/8 Stderr: | Stdout: | False UnitId: rabbitmq-server/1 - MachineId: 2/lxc/10 Stderr: | Stdout: | False UnitId: rabbitmq-server/2

As you see above, in this example machine 0/lxc/12 is the leader, via it's status of "True". Now we need to hit the other two servers and shut down RabbitMQ:

# service rabbitmq-server stop

Once both services have completed shutting down, we can resolve the partitioning by running:

$ juju resolved -r rabbitmq-server/<whichever is leader>

Substituting <whichever is leader> for the machine ID identified earlier.

Once that has completed, you can start the previously stopped services with the below on each host:

# service rabbitmq-server start

and verify the result with:

$ sudo rabbitmqctl cluster_status Cluster status of node 'rabbit@192-168-7-148' ... [{nodes,[{disc,['rabbit@192-168-7-148','rabbit@192-168-7-163', 'rabbit@192-168-7-174']}]}, {running_nodes,['rabbit@192-168-7-163','rabbit@192-168-7-174', 'rabbit@192-168-7-148']}, {partitions,[]}] ...done.

No partitions \o/

The JuJu errors for RabbitMQ should clear within a few minutes:

$ juju stat --format tabular | grep rabbit rabbitmq-server false local:trusty/rabbitmq-server-128 rabbitmq-server/0 idle 0/lxc/12 5672/tcp 19 rabbitmq-server/1 unknown idle 1/lxc/8 5672/tcp 19 rabbitmq-server/2 unknown idle 2/lxc/10 5672/tcp

You should also find the nova-compute instances starting up fine.

Categories: thinktime

A sprint

Seth Godin - Tue 02nd Jan 2018 00:01
Most of us have two speeds. There's the grind, the day after day, a marathon, work work work. And there's the recovery, the sleep in, Netflix and chill zombie state that we compartmentalize into a day like today. But what...        Seth Godin
Categories: thinktime

Simon Lyall: Donations 2017

Planet Linux Australia - Mon 01st Jan 2018 11:01

Like in 2016 and 2015 I am blogging about my charity donations.

The majority of donations were done during December (I start around my birthday) although after my credit card got suspended last year I spread them across several days.

The inspiring others bit seems to have worked a little. Ed Costello has blogged his donations for 2017.

I’ll note that throughout the year I’ve also been giving money via Patreon to several people whose online content I like. I suspended these payments in early-December but they have backed down on the change so I’ll probably restart them in early 2018.

As usual my main donation was to Givewell. This year I gave to them directly and allowed them to allocate to projects as they wish.

  • $US 600 to Givewell (directly for their allocation)

In march I gave to two organization I follow online. Transport Blog re-branded themselves as “Greater Auckland” and is positioning themselves as a lobbying organization as well as news site.

Signum University produce various education material around science-fiction, fantasy and medieval literature. In my case I’m following their lectures on Youtube about the Lord of the Rings.

I gave some money to the Software Conservancy to allocate across their projects and again to the Electronic Frontier Foundation for their online advocacy.

and lastly I gave to various Open Source Projects that I regularly use.

Categories: thinktime


Seth Godin - Sun 31st Dec 2017 22:12
Even though it's usually at the end, the acknowledgments are often the most important part of a book. This year, thousands of people have helped. They've inspired those they engaged with, built things that mattered, gracefully handled pain and loss,...        Seth Godin
Categories: thinktime


Seth Godin - Sat 30th Dec 2017 20:12
You can't make an hourglass with a boulder. But break the boulder into sufficiently small bits of sand, and you can tell time. You wouldn't want to eat a baked loaf of ice cream, mustard, fish, bread, capers and cheese....        Seth Godin
Categories: thinktime

New habits

Seth Godin - Fri 29th Dec 2017 19:12
I bought a CD yesterday. That didn't used to be news. I used to buy a CD every week, week after week, year after year. It adds up. Hi-rez streaming changed that habit for me, but it took about a...        Seth Godin
Categories: thinktime

Are you day trading?

Seth Godin - Thu 28th Dec 2017 20:12
The volatility of bitcoin turns the people who own it into addicts. At any given moment, it's up $100 or down a thousand. When it's up, you think you're brilliant, that you somehow had something to do with it. And...        Seth Godin
Categories: thinktime

Craige McWhirter: First Look at Snaps

Planet Linux Australia - Wed 27th Dec 2017 22:12

I've belatedly come to have a close up look at both Ubuntu Core (Snappy), Snaps and the Snappy package manager.

The first pass was to rebuild my rack of Raspberry Pi's from Debian armhf to Ubuntu Core for the Raspberry Pi.

This proved to be the most graceful install I've ever had on any hardware, ever. No hyperbole: boot, authenticate, done. I repeated this for all six Pi's in such a short time frame that I was concerned I'd done something wrong. Your SSH keys are already installed, you can log in immediately and just get on with it.

Which is where snaps come into play.

Back on my laptop, I followed the tutorial Create Your First Snap which uses GNU Hello as an example snap build and finishes with a push to the snap store at

I then created a Launchpad Repo, related a snap package, told it to build for armhf and amd64 and before long, I could install this snap on both my laptop and the Pi's.

Overall this was a pretty impressive and graceful process.

Categories: thinktime

The power of the possible

Seth Godin - Wed 27th Dec 2017 21:12
Next year is almost here. And doing what you did this year probably isn’t going to be sufficient. That’s because you have more to contribute than you did this year. You have important work worth sharing. To reach your goals,...        Seth Godin
Categories: thinktime


Subscribe to kattekrab aggregator - thinktime