You are here

Planet Linux Australia

Subscribe to Planet Linux Australia feed
Planet Linux Australia - http://planet.linux.org.au
Updated: 50 min 39 sec ago

Joshua Hesketh: OpenStack infrastructure swift logs and performance

5 hours 28 min ago

Turns out I’m not very good at blogging very often. However I thought I would put what I’ve been working on for the last few days here out of interest.

For a while the OpenStack Infrastructure team have wanted to move away from storing logs on disk to something more cloudy – namely, swift. I’ve been working on this on and off for a while and we’re nearly there.

For the last few weeks the openstack-infra/project-config repository has been uploading its CI test logs to swift as well as storing them on disk. This has given us the opportunity to compare the last few weeks of data and see what kind of effects we can expect as we move assets into an object storage.

  • I should add a disclaimer/warning, before you read, that my methods here will likely make statisticians cringe horribly. For the moment though I’m just getting an indication for how things compare.
The set up

Fetching files from an object storage is nothing particularly new or special (CDN’s have been doing it for ages). However, for our usage we want to serve logs with os-loganalyze giving the opportunity to hyperlink to timestamp anchors or filter by log severity.

First though we need to get the logs into swift somehow. This is done by having the job upload its own logs. Rather than using (or writing) a Jenkins publisher we use a bash script to grab the jobs own console log (pulled from the Jenkins web ui) and then upload it to swift using credentials supplied to the job as environment variables (see my zuul-swift contributions).

This does, however, mean part of the logs are missing. For example the fetching and upload processes write to Jenkins’ console log but because it has already been fetched these entries are missing. Therefore this wants to be the very last thing you do in a job. I did see somebody do something similar where they keep the download process running in a fork so that they can fetch the full log but we’ll look at that another time.

When a request comes into logs.openstack.org, a request is handled like so:

  1. apache vhost matches the server
  2. if the request ends in .txt.gz, console.html or console.html.gz rewrite the url to prepend /htmlify/
  3. if the requested filename is a file or folder on disk, serve it up with apache as per normal
  4. otherwise rewrite the requested file to prepend /htmlify/ anyway

os-loganalyze is set up as an WSGIScriptAlias at /htmlify/. This means all files that aren’t on disk are sent to os-loganalyze (or if the file is on disk but matches a file we want to mark up it is also sent to os-loganalyze). os-loganalyze then does the following:

  1. Checks the requested file path is legitimate (or throws a 400 error)
  2. Checks if the file is on disk
  3. Checks if the file is stored in swift
  4. If the file is found markup (such as anchors) are optionally added and the request is served
    1. When serving from swift the file is fetched via the swiftclient by os-loganlayze in chunks and streamed to the user on the fly. Obviously fetching from swift will have larger network consequences.
  5. If no file is found, 404 is returned

If the file exists both on disk and in swift then step #2 can be skipped by passing ?source=swift as a parameter (thus only attempting to serve from swift). In our case the files exist both on disk and in swift since we want to compare the performance so this feature is necessary.

So now that we have the logs uploaded into swift and stored on disk we can get into some more interesting comparisons.

Testing performance process

My first attempt at this was simply to fetch the files from disk and then from swift and compare the results. A crude little python script did this for me: http://paste.openstack.org/show/122630/

The script fetches a copy of the log from disk and then from swift (both through os-loganalyze and therefore marked-up) and times the results. It does this in two scenarios:

  1. Repeatably fetching the same file over again (to get a good average)
  2. Fetching a list of recent logs from gerrit (using the gerrit api) and timing those

I then ran this in two environments.

  1. On my local network the other side of the world to the logserver
  2. On 5 parallel servers in the same DC as the logserver

Running on my home computer likely introduced a lot of errors due to my limited bandwidth, noisy network and large network latency. To help eliminate these errors I also tested it on 5 performance servers in the Rackspace cloud next to the log server itself. In this case I used ansible to orchestrate the test nodes thus running the benchmarks in parallel. I did this since in real world use there will often be many parallel requests at once affecting performance.

The following metrics are measured for both disk and swift:

  1. request sent – time taken to send the http request from my test computer
  2. response – time taken for a response from the server to arrive at the test computer
  3. transfer – time taken to transfer the file
  4. size – filesize of the requested file

The total time can be found by adding the first 3 metrics together.

 

Results Home computer, sequential requests of one file

 

The complementary colours are the same metric and the darker line represents swift’s performance (over the lighter disk performance line). The vertical lines over the plots are the error bars while the fetched filesize is the column graph down the bottom. Note that the transfer and file size metrics use the right axis for scale while the rest use the left.

As you would expect the requests for both disk and swift files are more or less comparable. We see a more noticable difference on the responses though with swift being slower. This is because disk is checked first, and if the file isn’t found on disk then a connection is sent to swift to check there. Clearly this is going to be slower.

The transfer times are erratic and varied. We can’t draw much from these, so lets keep analyzing deeper.

The total time from request to transfer can be seen by adding the times together. I didn’t do this as when requesting files of different sizes (in the next scenario) there is nothing worth comparing (as the file sizes are different). Arguably we could compare them anyway as the log sizes for identical jobs are similar but I didn’t think it was interesting.

The file sizes are there for interest sake but as expected they never change in this case.

You might notice that the end of the graph is much noisier. That is because I’ve applied some rudimentary data filtering.

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift Standard Deviation 54.89516183 43.71917948 56.74750291 194.7547117 849.8545127 838.9172066 7.121600095 7.311125275 Mean 283.9594368 282.5074598 373.7328851 531.8043908 5091.536092 5122.686897 1219.804598 1220.735632

 

I know it’s argued as poor practice to remove outliers using twice the standard deviation, but I did it anyway to see how it would look. I only did one pass at this even though I calculated new standard deviations.

 

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift Standard Deviation 13.88664039 14.84054789 44.0860569 115.5299781 541.3912899 515.4364601 7.038111654 6.98399691 Mean 274.9291111 276.2813889 364.6289583 503.9393472 5008.439028 5013.627083 1220.013889 1220.888889

 

I then moved the outliers to the end of the results list instead of removing them completely and used the newly calculated standard deviation (ie without the outliers) as the error margin.

Then to get a better indication of what are average times I plotted the histograms of each of these metrics.

Here we can see a similar request time.

 

Here it is quite clear that swift is slower at actually responding.

 

Interestingly both disk and swift sources have a similar total transfer time. This is perhaps an indication of my network limitation in downloading the files.

 

Home computer, sequential requests of recent logs

Next from my home computer I fetched a bunch of files in sequence from recent job runs.

 

 

Again I calculated the standard deviation and average to move the outliers to the end and get smaller error margins.

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift Standard Deviation 54.89516183 43.71917948 194.7547117 56.74750291 849.8545127 838.9172066 7.121600095 7.311125275 Mean 283.9594368 282.5074598 531.8043908 373.7328851 5091.536092 5122.686897 1219.804598 1220.735632 Second pass without outliers Standard Deviation 13.88664039 14.84054789 115.5299781 44.0860569 541.3912899 515.4364601 7.038111654 6.98399691 Mean 274.9291111 276.2813889 503.9393472 364.6289583 5008.439028 5013.627083 1220.013889 1220.888889

 

What we are probably seeing here with the large number of slower requests is network congestion in my house. Since the script requests disk, swift, disk, swift, disk.. and so on this evens it out causing a latency in both sources as seen.

 

Swift is very much slower here.

 

Although comparable in transfer times. Again this is likely due to my network limitation.

 

The size histograms don’t really add much here.

 

Rackspace Cloud, parallel requests of same log

Now to reduce latency and other network effects I tested fetching the same log over again in 5 parallel streams. Granted, it may have been interesting to see a machine close to the log server do a bunch of sequential requests for the one file (with little other noise) but I didn’t do it at the time unfortunately. Also we need to keep in mind that others may be access the log server and therefore any request in both my testing and normal use is going to have competing load.

 

I collected a much larger amount of data here making it harder to visualise through all the noise and error margins etc. (Sadly I couldn’t find a way of linking to a larger google spreadsheet graph). The histograms below give a much better picture of what is going on. However out of interest I created a rolling average graph. This graph won’t mean much in reality but hopefully will show which is faster on average (disk or swift).

 

You can see now that we’re closer to the server that swift is noticeably slower. This is confirmed by the averages:

 

  request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift Standard Deviation 32.42528982 9.749368282 245.3197219 781.8807534 1082.253253 2737.059103 0 0 Mean 4.87337544 4.05191168 39.51898688 245.0792916 1553.098063 4167.07851 1226 1232 Second pass without outliers Standard Deviation 1.375875503 0.8390193564 28.38377158 191.4744331 878.6703183 2132.654898 0 0 Mean 3.487575109 3.418433003 7.550682037 96.65978872 1389.405618 3660.501404 1226 1232

 

Even once outliers are removed we’re still seeing a large latency from swift’s response.

The standard deviation in the requests now have gotten very small. We’ve clearly made a difference moving closer to the logserver.

 

Very nice and close.

 

Here we can see that for roughly half the requests the response time was the same for swift as for the disk. It’s the other half of the requests bringing things down.

 

The transfer for swift is consistently slower.

 

Rackspace Cloud, parallel requests of recent logs

Finally I ran just over a thousand requests in 5 parallel streams from computers near the logserver for recent logs.

 

Again the graph is too crowded to see what is happening so I took a rolling average.

 

 

request sent (ms) – disk request sent (ms) – swift response (ms) – disk response (ms) – swift transfer (ms) – disk transfer (ms) – swift size (KB) – disk size (KB) – swift Standard Deviation 0.7227904332 0.8900549012 434.8600827 909.095546 1913.9587 2132.992773 6.341238774 7.659678352 Mean 3.515711867 3.56191383 145.5941102 189.947818 2427.776165 2875.289455 1219.940039 1221.384913 Second pass without outliers Standard Deviation 0.4798803247 0.4966553679 109.6540634 171.1102999 1348.939342 1440.2851 6.137625464 7.565931993 Mean 3.379718381 3.405770445 70.31323922 86.16522485 2016.900047 2426.312363 1220.318912 1221.881335

 

The averages here are much more reasonable than when we continually tried to request the same file. Perhaps we’re hitting limitations with swifts serving abilities.

 

I’m not sure why we have sinc function here. A network expert may be able to tell you more. As far as I know this isn’t important to our analysis other than the fact that both disk and swift match.

 

Here we can now see swift keeping a lot closer to disk results than when we only requested the one file in parallel. Swift is still, unsurprisingly, slower overall.

 

Swift still loses out on transfers but again does a much better job of keeping up.

 

Error sources

I haven’t accounted for any of the following swift intricacies (in terms of caches etc) for:

  • Fetching random objects
  • Fetching the same object over and over
  • Fetching in parallel multiple different objects
  • Fetching the same object in parallel

I also haven’t done anything to account for things like file system caching, network profiling, noisy neighbours etc etc.

os-loganalyze tries to keep authenticated with swift, however

  • This can timeout (causes delays while reconnecting, possibly accounting for some spikes?)
  • This isn’t thread safe (are we hitting those edge cases?)

We could possibly explore getting longer authentication tokens or having os-loganalyze pull from an unauthenticated CDN to add the markup and then serve. I haven’t explored those here though.

os-loganalyze also handles all of the requests not just from my testing but also from anybody looking at OpenStack CI logs. In addition to this it also needs to deflate the gzip stream if required. As such there is potentially a large unknown (to me) load on the log server.

In other words, there are plenty of sources of errors. However I just wanted to get a feel for the general responsiveness compared to fetching from disk. Both sources had noise in their results so it should be expected in the real world when downloading logs that it’ll never be consistent.

Conclusions

As you would expect the request times are pretty much the same for both disk and swift (as mentioned earlier) especially when sitting next to the log server.

The response times vary but looking at the averages and the histograms these are rarely large. Even in the case where requesting the same file over and over in parallel caused responses to go slow these were only in the magnitude of 100ms.

The response time is the important one as it indicates how soon a download will start for the user. The total time to stream the contents of the whole log is seemingly less important if the user is able to start reading the file.

One thing that wasn’t tested was streaming of different file sizes. All of the files were roughly the same size (being logs of the same job). For example, what if the asset was a few gigabytes in size, would swift have any significant differences there? In general swift was slower to stream the file but only by a few hundred milliseconds for a megabyte. It’s hard to say (without further testing) if this would be noticeable on large files where there are many other factors contributing to the variance.

Whether or not these latencies are an issue is relative to how the user is using/consuming the logs. For example, if they are just looking at the logs in their web browser on occasion they probably aren’t going to notice a large difference. However if the logs are being fetched and scraped by a bot then it may see a decrease in performance.

Overall I’ll leave deciding on whether or not these latencies are acceptable as an exercise for the reader.

Categories: thinktime

Andrew Pollock: [life] Day 265: Kindergarten and startup stuff

Tue 21st Oct 2014 21:10

Zoe yelled out for me at 5:15am for some reason, but went back to sleep after I resettled her, and we had a slow start to the day a bit after 7am. I've got a mild version of whatever cold she's currently got, so I'm not feeling quite as chipper as usual.

We biked to Kindergarten, which was a bit of a slog up Hawthorne Road, given the aforementioned cold, but we got there in the end.

I left the trailer at the Kindergarten and biked home again.

I finally managed to get some more work done on my real estate course, and after a little more obsessing over one unit, got it into the post. I've almost got another unit finished as well. I'll try to get it finished in the evenings or something, because I'm feeling very behind, and I'd like to get it into the mail too. I'm due to get the second half of my course material, and I still have one more unit to do after this one I've almost finished.

I biked back to Kindergarten to pick up Zoe. She wanted to watch Megan's tennis class, but I needed to grab some stuff for dinner, so it took a bit of coaxing to get her to leave. I think she may have been a bit tired from her cold as well.

We biked home, and jumped in the car. I'd heard from Matthew's Dad that FoodWorks in Morningside had a good meat selection, so I wanted to check it out.

They had some good roasting meat, but that was about it. I gave up trying to mince my own pork and bought some pork mince instead.

We had a really nice dinner together, and I tried to get her to bed a little bit early. Every time I try to start the bed time routine early, the spare time manages to disappear anyway.

Categories: thinktime

David Rowe: SM1000 Part 7 – Over the air in Germany

Tue 21st Oct 2014 09:10

Michael Wild DL2F2 in Germany recently attended a Hamfest where he demonstrated his SM1000. Michael sent me the following email (hint: I used Google translate on the web sites):

Here is the link to the review of our local hamfest.

At the bottom is a video of a short QSO on 40m using the SM-1000 over about 400km. The other station was Hermann (DF2DR). Hermann documented this QSO very well on his homepage also showing a snapshot of the waterfall during this QSO. Big selective fading as you can see, but we were doing well!

He also explains that, when switching to SSB at the same average power level, the voice was almost not understandable!

SM1000 Beta and FreeDV Update

Rick KA8BMA has been working hard on the Beta CAD work, and fighting a few Eagle DRC battles. Thanks to all his hard work we now have an up to date schematic and BOM for the Betas. He is now working on the Beta PCB layout, and we are refining the BOM with Edwin from Dragino in China. Ike, W3IKIE, has kindly been working with Rick to come up with a suitable enclosure. Thanks guys!

My current estimate is that the Beta SM1000s will be assembled in November. Once I’ve tested a few I’ll put them up on my store and start taking orders.

In the mean time I’ve thrown myself into modem simulations – playing with a 450 bit/s version of Codec 2, LPDC FEC codes, diversity schemes and coherent QPSK demodulation. I’m pushing towards a new FreeDV mode that works on fading channels at negative SNRs. More on that in later posts. The SM1000 and a new FreeDV mode are part of my goals for 2014. The SM1000 will make FreeDV easy to use, the new mode(s) will make it competitive with SSB on HF radio.

Everything is open source, both hardware and software. No vendor lock in, no software licenses and you are free to experiment and innovate.

Categories: thinktime

Chris Samuel: IBM Pays GlobalFoundries to take Microprocessor Business

Tue 21st Oct 2014 07:10

Interesting times for IBM, having already divested themselves of the x86 business by selling it on to Lenovo they’ve now announced that they’re paying GlobalFoundries $1.5bn to take pretty much that entire side of the business!

IBM (NYSE: IBM) and GLOBALFOUNDRIES today announced that they have signed a Definitive Agreement under which GLOBALFOUNDRIES plans to acquire IBM’s global commercial semiconductor technology business, including intellectual property, world-class technologists and technologies related to IBM Microelectronics, subject to completion of applicable regulatory reviews. GLOBALFOUNDRIES will also become IBM’s exclusive server processor semiconductor technology provider for 22 nanometer (nm), 14nm and 10nm semiconductors for the next 10 years.

It includes IBM’s IP and patents, though IBM will continue to do research for 5 years and GlobalFoundries will get access to that. Now what happens to those researchers (one of whom happens to be a friend of mine) after that isn’t clear.

When I heard the rumours yesterday I was wondering if IBM was aiming to do an ARM and become a fab-less CPU designer but this is much more like exiting the whole processor business altogether. The fact that they seem to be paying Global Foundries to take this off their hands also makes it sound pretty bad.

What this all means for their Power CPU is uncertain, and if I was nVidia and Mellanox in the OpenPOWER alliance I would be hoping I’d know about this before joining up!

This item originally posted here:



IBM Pays GlobalFoundries to take Microprocessor Business

Categories: thinktime

Andrew Pollock: [life] Day 264: Pupil Free Day means lots of park play

Mon 20th Oct 2014 21:10

Today was a Kindergarten (and it seemed most of the schools in Brisbane) Pupil Free Day.

Grace, the head honcho of Thermomix in Australia, was supposed to be in town for a meet and greet, and a picnic in New Farm Park had been organised, but at the last minute she wasn't able to make it due to needing to be in Perth for a meeting. The plan changed and we had a Branch-level picnic meeting at the Colmslie Beach Reserve.

So after Sarah dropped Zoe off, I whipped up some red velvet cheesecake brownie, which seems to be my go to baked good when required to bring a plate (it's certainly popular) and I had some leftover sundried tomatoes, so I whipped up some sundried tomato dip as well.

The meet up in the park was great. My group leader's daughters were there, as were plenty of other consultant's kids due to the Pupile Free Day, and Zoe was happy to hang out and have a play. There was lots of yummy food, and we were able to graze and socialise a bit. We called it lunch.

After we got home, we had a bit of a clean up of the balcony, which had quite a lot of detritus from various play dates and craft activities. Once that was done, we had some nice down time in the hammock.

We then biked over to a park to catch up with Zoe's friend Mackensie for a play date. The girls had a really nice time, and I discovered that the missing link in the riverside bike path has been completed, which is rather nice for both cycling and running. (It goes to show how long it's been since I've gone for a run, I really need to fix that).

After that, we biked home, and I made dinner. We got through dinner pretty quickly, and so Zoe and I made a batch of ginger beer after dinner, since there was a Thermomix recipe for it. It was cloudy though, and Zoe was more used to the Bunderberg ginger beer, which is probably a bit better filtered, so she wasn't so keen on it.

All in all, it was a really lovely way to spend a Pupil Free Day.

Categories: thinktime

Francois Marier: LXC setup on Debian jessie

Mon 20th Oct 2014 13:10

Here's how to setup LXC-based "chroots" on Debian jessie. While this is documented on the Debian wiki, I had to tweak a few things to get the networking to work on my machine.

Start by installing (as root) the necessary packages:

apt-get install lxc libvirt-bin debootstrap Network setup

I decided to use the default /etc/lxc/default.conf configuration (no change needed here):

lxc.network.type = veth lxc.network.flags = up lxc.network.link = virbr0 lxc.network.hwaddr = 00:FF:AA:xx:xx:xx lxc.network.ipv4 = 0.0.0.0/24

but I had to make sure that the "guests" could connect to the outside world through the "host":

  1. Enable IPv4 forwarding by putting this in /etc/sysctl.conf:

    net.ipv4.ip_forward=1
  2. and then applying it using:

    sysctl -p
  3. Ensure that the network bridge is automatically started on boot:

    virsh -c lxc:/// net-start default virsh -c lxc:/// net-autostart default
  4. and that it's not blocked by the host firewall, by putting this in /etc/network/iptables.up.rules:

    -A INPUT -d 224.0.0.251 -s 192.168.122.1 -j ACCEPT -A INPUT -d 192.168.122.255 -s 192.168.122.1 -j ACCEPT -A INPUT -d 192.168.122.1 -s 192.168.122.0/24 -j ACCEPT
  5. and applying the rules using:

    iptables-apply
Creating a container

Creating a new container (in /var/lib/lxc/) is simple:

sudo MIRROR=http://ftp.nz.debian.org/debian lxc-create -n sid64 -t debian -- -r sid -a amd64

You can start or stop it like this:

sudo lxc-start -n sid64 -d sudo lxc-stop -n sid64 Connecting to a guest using ssh

The ssh server is configured to require pubkey-based authentication for root logins, so you'll need to log into the console:

sudo lxc-stop -n sid64 sudo lxc-start -n sid64

then install a text editor inside the container because the root image doesn't have one by default:

apt-get install vim

then paste your public key in /root/.ssh/authorized_keys.

Then you can exit the console (using Ctrl+a q) and ssh into the container. You can find out what IP address the container received from DHCP by typing this command:

sudo lxc-ls --fancy Fixing Perl locale errors

If you see a bunch of errors like these when you start your container:

perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LANG = "fr_CA.utf8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C").

then log into the container as root and use:

dpkg-reconfigure locales

to enable the same locales as the ones you have configured in the host.

Categories: thinktime

Simon Lyall: Links: Docker, Dollar Vans, Bus Bunching and Languages

Mon 20th Oct 2014 10:10
Categories: thinktime

linux.conf.au News: Speaker Feature: Pavel Emelyanov, Alasdair Allan

Mon 20th Oct 2014 07:10
Pavel Emelyanov Libcontainer: One lib to rule them all

10:40 pm Friday 16th January 2015

Pavel Emelyanov is a principal engineer at Parallels working on server virtualization projects. He holds a PhD degree in Applied Mathematics from the Moscow Institute of Physics and Technology. His speaking experience includes the talk on network namespaces at LinuxCon 2009 and the presentation of the Virtuozzo resource management at the joint memory management, storage and filesystem summit in April 2011.

For more information on Pavel and his presentation, see here. You can follow him as @xemulp and don’t forget to mention #LCA2015.



Alasdair Allan Open Source Protocols and Architectures to Fix the Internet of Things…

3:40pm Friday 16th January 2015

Alasdair is a scientist, author, hacker, tinkerer and co-founder of a startup working on fixing the Internet of Things. He spends much of his time probing current trends in an attempt to determine which technologies are going to define our future.

He has also written articles for Make magazine. The latest entitled “Pick up your tools and get started!” posted 1 September 2014.

For more information on Alasdair and his presentation, see here. You can follow him as @aallan and don’t forget to mention #LCA2015.

Categories: thinktime

Sridhar Dhanapalan: Twitter posts: 2014-10-13 to 2014-10-19

Mon 20th Oct 2014 01:10
Categories: thinktime

Mark Terle: A preponderance of yak shaving….

Sun 19th Oct 2014 15:10

It is often observed that attempting to undertake one task begets another, with the corollary that two days later you’ve built a bikeshed painted in a multitude of colours.

So, dear readers, this tale of woe begins with the need to update my blog to something useful after 18 months of neglect and more. I had been writing a travel blog from when I took some leave off work to wander the globe. For this task, a new more generic DNS entry and an upgrade to the WordPress installation and syndication with my Advogato blog. Easily accomplished and a sense of progress.

This blog entry is going to be mostly a technical one. I’ll try incorporating more of real life in other entries.

Great, now I can tell the world about my little project toying with Vagrant and Puppet.

It is called “Browser In A Box”. It is up on Github https://github.com/mtearle/browser-in-a-box

It is very simple, a Vagrant file and a set of Puppet manifests/modules to launch Chromium in kiosk mode inside a VM to hit a certain URL. This is part of planned later work to look at creating a Vagrant development environment for Concerto.

At this point, I got distracted … aside from the liberal upgrades of bash on various machines to address Shellshock

Then I accidentally purchased a new Ultrabook. My previous netbook had been getting long in the tooth and it was time to upgrade. I ended up purchasing a Toshiba Satellite NB10, a reasonable processor Intel N2830, 4 Gig of RAM and 500 Gigs of spinning rust. Those are the nice bits.

On the negatives, Crappy Toshiba keyboard layout with the ~ key in a stupid spot and a UEFI bios. It is now blatantly apparent why Matthew Garrett drinks copious quantities of gin.

Special brickbats go to the Ubuntu installer for repartitioning and eating my Windows installation and recovery partition. (The option to install over my test Debian installation got over enthusiastic).  The wireless chipset (Atheros) has a known problem where it confuses the access point.

The next distraction ended up being a fit of procastination in terms of rearranging my tiny apartment. I’ve now modelled it in a program called Sweet Home 3D. Easy and straight forward to use. Needs a few more furniture models, but perfectly functional. I shall use it again next time I move.

Finally, we arrive at the the original task. I want to start syncing my calendars between various locations (written here for my benefit later).

They are:

  • Work stream – From my Work (Exchange) to my private host (Radicale) to Google Calendar (which will get to my Android phone)
  • Personal stream – From my private host (Radicale) to Google Calendar (and back again)
  • Party stream – From Facebook’s ical export to my private host and Google Calendar

In addition, various syncing of contacts but not my primary focus at the moment.

It appears that syncevolution will do most of what I want here. The challenge revolves around how to get it working. Ultimately, I want to have this live headless hosted on a virtual machine not running a desktop.

In a fit of enthusiasm, I decided upon attempting to build it from source as opposed to using the packages provided from the upstream (to avoid dragging in unnecessary dependencies.

I need to build from HEAD due to recent code added to syncevolution to support the change in Google’s CALDAV API to be behind OAuth V2.

This was not an overly successful exercise, I ended up getting something built but it didn’t ultimately work.

Problems encountered were:

  • libwbxml2 – The upstream at opensync.org is down. There appears to be forks, so playing the game of guessing the current head/release version.
  • activesyncd – Build system is currently broken in parts. There appears to be bit rot around the evolution bindings as the evolution API has changed

I gave up at that point. I’ve since spun up a different virtual machine with Debian Jessie and an install of Gnome. The packages from the syncevolution upstream installed cleanly, but have yet to work out the incarnations to make it work. However, that my friends is a story for a later blog entry…

Categories: thinktime

linux.conf.au News: Multimedia and Music Miniconf - Call for Papers

Sun 19th Oct 2014 13:10

The Multimedia and Music Miniconf at LCA2015 will be held in Auckland, New Zealand, on Monday 12 January 2015. We are pleased to formally open the miniconf's Call for Papers. Submissions are encouraged from anyone with a story to tell which is related to open software for multimedia or music.

Examples of possible presentations include:

  • demonstrations of multimedia content authored using Open Source programs
  • audio recording examples
  • Open Source games
  • video and image editing on Linux
  • new multimedia software being written
  • multimedia web APIs and applications
  • unusual uses of Open Source multimedia software
  • codec news

In addition, we are planning to hold an informal jam session at the end of the Miniconf, giving community members a change to showcase their compositions and multimedia creations. Expressions of interest for this are also invited. If musical instruments are required it is preferable if participants arranged this themselves, but with sufficient lead time it might be possible to arrange a loan from locals in Auckland.

The miniconf website at annodex.org/events/lca2015 has further details about the miniconf.

To submit a proposal or for further information, please email Jonathan Woithe (jwoithe@atrad.com.au) or Silvia Pfeiffer (silviapfeiffer1@gmail.com).

Jonathan Woithe and Silvia Pfeiffer

(Multimedia and Music miniconf organisers)
Categories: thinktime

Andrew Donnellan: KDE 4/Plasma system tray not displaying applications, notifications appearing twice

Sat 18th Oct 2014 23:10

So I was trying to get RSIBreak working on my machine today, and for some reason it simply wasn’t displaying an icon in the Plasma system tray as it was meant to.

It took some searching around, but eventually I came across a comment on a KDE bug report that had the answer.

I opened up ~/.kde/config/plasma-desktop-appletsrc, searched for “systemtray“, and lo and behold, there were two Containments, both using the systemtray plugin. It seems that at some point during the history of my KDE installation, I ended up with two system trays, just with one that wasn’t visible.

After running kquitapp plasma to kill the desktop, I removed the first systemtray entry (I made an educated guess and decided that the first one was probably the one I didn’t want any more), saved the file and restarted Plasma.

Suddenly, not only did RSIBreak appear in my system tray, but so did a couple of other applications which I forgot I had installed. This also fixed the problem I was having with all KDE notifications appearing on screen twice, which was really rather annoying and I’m not sure how I coped with it for so long…



Filed under: Linux, Uncategorized Tagged: KDE, linux, Linux Tips
Categories: thinktime

Andrew Donnellan: The r8169 driver and mysterious network problems

Sat 18th Oct 2014 23:10

A few months ago, a friend of mine was having a problem. When he hooked up his Toshiba laptop to the Ethernet port in his bedroom, it would work under Windows, but not under Linux. When he hooked it up to the port in the room next door, it would work under both.

I headed over with my Samsung Ultrabook, and sure enough – it worked fine under Windows, but not Linux, while the room next door worked under both.

As it turns out, both our laptops used Realtek RTL8168-series Ethernet controllers, which are normally handled just fine by the r8169 driver, which can be found in the kernel mainline. However, Realtek also releases a r8168 driver (available in Debian as r8168-dkms). Upon installing that, everything worked fine.

(At some point I should probably go back and figure out why it didn’t work under r8169 so I can file a bug…)



Filed under: Hardware, Linux Tagged: Computing, Drivers, linux, Linux Tips
Categories: thinktime

Lev Lafayette: PRINCE2 Checklist and Flowchart

Sat 18th Oct 2014 15:10

Recently a simple statement of PRINCE2 governance structures was provided. From this it is possible to derive a checklist for project managers to tick off, just to make sure that everything is done. Please note that this checklist is tailored and combines some functions. For example, there is no Business Review Plan as it is argued that any sensible project should incorporate these into the Business Case and the Project Plan.

A simple graphic is provided to assist with this process

read more

Categories: thinktime

Lev Lafayette: File Creation Time in Linux

Sat 18th Oct 2014 13:10

Linux offers most of the expected file attributes from the command line, including the owner of a file, the group, the size, the date modified and name. However often users want find out when a file was created. This requires a little bit of extra investigation.

read more

Categories: thinktime

Erik de Castro Lopo: Haskell : A neat trick for GHCi

Sat 18th Oct 2014 09:10

Just found a really nice little hack that makes working in the GHC interactive REPL a little easier and more convenient. First of all, I added the following line to my ~/.ghci file.

:set -DGHC_INTERACTIVE

All that line does is defines a GHC_INTERACTIVE pre-processor symbol.

Then in a file that I want to load into the REPL, I need to add this to the top of the file:

{-# LANGUAGE CPP #-}

and then in the file I can do things like:

#ifdef GHC_INTERACTIVE import Data.Aeson.Encode.Pretty prettyPrint :: Value -> IO () prettyPrint = LBS.putStrLn . encodePretty #endif

In this particular case, I'm working with some relatively large chunks of JSON and its useful to be able to pretty print them when I'm the REPL, but I have no need for that function when I compile that module into my project.

Categories: thinktime

Paul Wayper: That time that I registered an electric vehicle

Fri 17th Oct 2014 23:10
So, tell us a story, Uncle Paul.

Sure. One time when I was in Rovers, ...

No, tell us the story of how you got your electric motorbike registered!

Oh, okay then.

It was the 20th of February - a Friday. I'd taken the day off to get the bike registered. I'd tried to do this a couple of weeks before then, but I found out that, despite being told a month beforehand that the workload on new registrations was only a couple of days long, when I came to book it I found out that the earliest they could do was the 20th, two weeks away. So the 20th it was.

That morning I had to get the bike inspected by the engineer, get his sign-off, and take it down to the motor registry to get it inspected at 8:30AM. I also had to meet the plumber at our house, which meant I left a bit late, and by the time I was leaving the engineer it was already 8:15AM and I was in traffic. Say what you like about Canberra being a small town, but people like driving in and the traffic was a crawl. I rang the motor registry and begged for them to understand that I'd be there as soon as possible and that I might be a couple of minutes late. I squeaked into the entrance just as they were giving up hope, and they let me in because of the novelty of the bike and because I wasn't wasting their time.

The roadworthy inspection went fairly harmlessly - I didn't have a certificate from a weighbridge saying how heavy it was, but I knew it was only about eight kilos over the original bike's weight, so probably about 240 kilos? "OK, no worries," they said, scribbling that down on the form. The headlights weren't too high, the indicators worked, and there was no problem with my exhaust being too loud.

(Aside: at the inspection station there they have a wall full of pictures of particularly egregious attempts to get dodgy car builds past an inspection. Exhaust stuffed full of easily-removable steel wool? Exhausts with bit burnt patches where they've been oxy'd open and welded shut again? Panels attached with zip ties? Bolts missing? Plastic housings melted over ill-fitted turbos? These people have seen it all. Don't try to fool them.)

Then we came up to the really weird part of my dream. You know, the part where I know how to tap dance, but I can only do it while wearing golf shoes?

Er, sorry. That was something else. Then we came to the weird part of the process.

Modified vehicles have to get a compliance plate, to show that they comply with the National Code of Practice on vehicle conversions. The old process was that the engineer that inspected the vehicle to make sure it complied had blank compliance plates; when you brought the vehicle in and it passed their inspection, they then filled out all the fields on the plate, attached the plate to the vehicle, and then you transported it down to Main Roads. But that was a bit too open to people stealing compliance plates, so now they have a "better" system. What I had to do was:

  1. Get the bike inspected for road worthiness.
  2. They hand me a blank compliance plate.
  3. I then had to take it to the engineer, who told me the fields to fill in.
  4. He then told me to go to a trophy making place, where they have laser etchers that can write compliance plates beautifully.
  5. I arrive there at 11AM. They say it'll be done by about 2PM.
  6. Go and have lunch with friends. Nothing else to do.
  7. Pick etched compliance plate up.
  8. Take compliance plate back to engineer. Because he's busy, borrow a drill and a rivet gun and attach the plate to the bike myself.
  9. Take it back to Main Roads, who check that the plate is attached to the bike correctly and stamp the road worthiness form. Now I can get the bike registered.
Yeah, it's roundabout. Why not keep engrave the plates at Main Roads with the details the Engineer gives to them? But that's the system, so that's what I did.

And so I entered the waiting department. It only probably took about fifteen minutes to come up next in the queue, but it was fifteen minutes I was impatient to see go. We went through the usual hilarious dance with values:

  • Her: What are you registering?
  • Me: An electric motorbike.
  • Her: How many cylinders?
  • Me: Er... it's electric. None.
  • Her: None isn't a value I can put in.
  • Me: (rolls eyes) OK, one cylinder.
  • Her: OK. How many cubic centimetres?
Many months ago I had enquired about custom number plates, and it turns out that motorbikes can indeed have them. Indeed, I could by "3FAZE" if I wanted. For a mere $2,600 or so. It was very tempting, but when I weighed it up against getting new parts for the bike (which it turned out I would need sooner rather than later, but that's a story for another day) I thought I'd save up for another year.

So I finally picked up my new set of plates, thanked her for her time, and said "Excuse me, but I have to do this:" and then yelled:

"Yes!!!!"

Well, maybe I kept my voice down a little. But I had finally done it - after years of work, several problems, one accident, a few design changes, and lots of frustration and gradual improvement, I had an actual, registered electric motorbike I had built nearly all myself.

I still get that feeling now - I'll be riding along and I'll think, "wow, I'm actually being propelled along by a device I built myself. Look at it, all working, holding together, acting just like a real motorbike!" It feels almost like I've got away with something - a neat hack that turns out to work just as well as all those beautifully engineered mega-budget productions. I'm sure a lot of people don't notice it - it does look a bit bulky, but it's similar enough to a regular motorbike that it probably just gets overlooked as another two-wheeled terror on the roads.

Well, I'll just have to enjoy it myself then :-)

Categories: thinktime

Andrew Pollock: [life] Day 261: Lots of play dates with boys, TumbleTastics, and a fairy gathering

Fri 17th Oct 2014 19:10

Today was a typical jam packed day. Zoe had a brief wake up at at some point overnight because she couldn't find Cowie, right next to her head, but that was it.

First up, the PAG fundraising committee come over for a quick (well, more like 2 hour) meeting at my place to discuss planning for the sausage sizzle tomorrow. Because I don't have Zoe, I've volunteered to do a lot of the running around, so I'm going to have a busy day.

Mel had brought Matthew and Olivia with her, so Zoe and Matthew had a good time playing, and Olivia kept trying to join in.

That meeting ran right up until I realised we had to head off for TumbleTastics, so Zoe got ready in record time and we scootered over and made it there just as her class was starting. I was sure we were going to be late, so I was happy we made it in time.

Lachlan and his Mum, Laura, and little sister came over for lunch again afterwards, and stayed for a little while.

After they left, we started getting ready for the Fairy Nook's attempt to break the Guiness Book of Records record for the most fairies in one place. We needed to get a wand, so once Zoe was appropriately attired, we walked around the corner to Crackerjack Toys and picked up a wand.

After that, I popped up to Mel's place to collect a whole bunch of eskies that the local councillor had lent us for the sausage sizzle. Mel had also picked up a tutu for Zoe from the local two dollar store in her travels.

We got home, and then walked to the Hawthorne AFL oval where the record attempt was. Initially there were like two other fairies there, but by 4:30pm, there was a pretty good turnout. I don't know what the numbers were, but I'm pretty sure they were well under the 872 they needed. There was a jumping castle and a few of Zoe's friends from Kindergarten, so it was all good.

Sarah arrived to pick up Zoe from there, and I walked home.

Categories: thinktime

linux.conf.au News: Speaker Feature: Laura Bell, Michael Cordover

Fri 17th Oct 2014 07:10
Laura Bell Why can't we be friends? Integrating Security into an Existing Agile SDLC

3:40pm Friday 16th January 2015

Laura describes herself as an application security wrangler, repeat dreamer, some-time builder, python juggler, Mom and wife.

For more information on Laura and her presentation, see here. You can stalk her as @lady_nerd and don’t forget #LAC2015.



Michael Cordover Using FOI to get source code: the EasyCount experience

3:40pm Wednesday 14th January 2015

Michael is interested in the law, science, politics and everything in between. He worked in computing, event management and project management. He a policy wonk and systems-oriented and he loves variety but is interested in detail.

His life goal as a child was to know everything. He says that's impossible but is still trying to get as close as he can.

For more information on Michael and his presentation, see here. You can stalk him as @mjec and don’t forget #LAC2015.

Categories: thinktime

Andrew Pollock: [life] Day 260: Bedwetting, a morning tea play date, and swim class

Thu 16th Oct 2014 21:10

Zoe woke up at something like 3:30am because she'd wet the bed. She hasn't wet the bed since before she turned 4. In fact, I bought a Connie pad and she promptly never wet the bed again. I was actually thinking about stopping using it just last night, so I obviously jinxed things.

Anyway, she woke up, announced she'd had an accident, and I smugly thought I'd have it all handled, but alas, the pad was too low down, so she'd still managed to wet the mattress, which was annoying. Plan B was to just switch her to the bottom bunk, which still worked out pretty well. I've learned an important lesson about the placement of the Connie pad now.

Unfortunately for me, it seems that if I get woken up after about 4am, I have a low probability of getting back to sleep, and I'd gotten to bed a bit late the night before, so I only wound up with about 5 hours and felt like crap all day.

Vaeda and her mum, Francesca came over for a morning tea play date. I'd been wanting an excuse to try out a new scone recipe that I'd discovered, so I cranked out some scones for morning tea.

Vaeda and Francesca couldn't stay for too long, but it was a nice morning nonetheless. Then we popped out to Woolworths to pick up a $30 gift card that the store had donated towards the weekend sausage sizzle. Not quite 70 kg of free sausages, but better than nothing.

After we got back, we had some lunch, and I tried to convince Zoe to have a nap with me, without success, but we did have a couple of hours of quietish time, and I got to squeeze in some reading.

We biked over to swim class and then biked home, and I made dinner. Zoe was pretty tired, so I got her to bed nice and easily. It'll be an early night for me too.

Categories: thinktime

Pages