AUCKLAND, New Zealand – Thursday 18th December 2014 – linux.conf.au 2015 organisers are proud to announce an update to our funding programme!
Python Software Foundation Outreach Programme
LCA 2015 and the Python Software Foundation are proud to support our community. To supplement the existing InternetNZ Diversity fund the PSF have donated additional funds for candidates within the Python community.The Python Software Foundation appreciates LCA 2015's commitment to diversity, and is proud to add its own contribution in the form of the Python Software Foundation Outreach Fund. Much system software for Linux is written in Python (including both distro level tools and open source system management projects like OpenStack, Salt and Ansible), and Linux is often the default choice for deployment of Python web services and other networked applications. This contribution is intended to strengthen ties between the Python and Linux communities by assisting under-represented delegates who participate in the Python community in the region but, without financial assistance, would not be able to attend LCA 2015.
For more information please see our funding registration page.
linux.conf.au is one of the world's best conferences for free and open source software! The coming linux.conf.au; LCA 2015 will be held at the University of Auckland, New Zealand from Monday 12 January to Saturday 16 January 2015. LCA 2015 will be fun, informal and seriously technical, bringing together Free and Open Source developers, users and community champions from around the world. LCA 2015 is the third time linux.conf.au has been held in New Zealand. The first was in Dunedin in 2006 and the second was in Wellington in 2010.
For more information please visit our websiteAbout Linux Australia
Linux Australia is the peak body for Linux User Groups (LUGs) around Australia, and as such represents approximately 5000 Australian Linux users and developers. Linux Australia facilitates the organisation of this international Free Software conference in a different Australasian city each year.
For more information see: http://www.linux.org.au/Emperor Penguin Sponsors
LCA 2015 is proud to acknowledge the support of our Emperor Penguin Sponsors, Catalyst IT, HP and IBM, and our diversity sponsor Internet NZ.
For more information about our sponsors click below -
At bath time last night, Zoe had some spots on her torso. Interestingly, he first reaction upon seeing them in the mirror was "Chicken!". I was more sceptical, because she's been vaccinated for chicken pox, and wasn't showing other symptoms. I thought it may have been from crawling along the tree branch. So I put her to bed and said we would check them in the morning.
After a good night's sleep, but a ridiculously early start at 5am, she still had spots, but was otherwise fine, so I decided to make a doctor's appointment. I managed to get one for 12:15am, so we just hung out at home in the morning, and Zoe watched some TV. It was ridiculously hot, so it was a good day to be indoors with the air conditioning cranked up.
After an early lunch, we went to the doctor. She said that Zoe had a slight fever, but she was also doubtful if it looked like chicken pox. She said to give it 48 hours to see what happened. She said if it was chicken pox, it'd be a mild case, given she's vaccinated.
I guess the school holidays is as good a time as any to be out of commission. Hopefully we both won't go too stir crazy.
She also said that given how Zoe was presenting we didn't need to go too overboard on isolation, so we made a quick trip out to Westfield Carindale to pick up some birthday cards, before heading home again.
Zoe's temperature got a bit higher in the afternoon, and she ended up taking a long, late nap on the couch. I used the time to work on the next unit of my real estate licence course, and made some good progress.
I pretty much had to wake her up when it was time for Sarah to pick her up, and she still had a low grade fever, but was otherwise in good spirits.
1:20pm Wednesday 14th January 2015
Brenda Wallace is an Open Source contibutor from Wellington. She likes all the programming languages, but especially the ones beginning with P. Brenda works with the mighty wonderful people at Rabid Tech. Also, she's not a werewolf.
For more information on Brenda and her presentation, see here.
David Airlie Displayport MST: why do my laptop dockoutputs not work?
2:15pm Wednesday 14th January 2015
David Airlie is the upstream kernel graphics maintainer and work for Red Hat out of their Brisbane office. He is part of the maintainer team for Red Hat Enterprise Linux graphical components. He recently branched into virtualisation for graphics project and is trying to create a fully open source virtualised 3D graphics device capable of supporting modern operating-system requirements. He also gets distracted from this task my many random other graphics projects, of which support for Displayport MST is one.
For more information on David and his presentation, see here.
Dirk Hohndel Sustaining Momentum - or the Gap Between User Request and Developer Capacity
3:40pm Friday 16th January 2015
Dirk is Intel's Chief Linux and Open Source Technologist. He has been an active developer and contributor in the Linux space since its earliest days, among other roles, he worked as Chief Technology Officer of SuSE and as Unix Architect at Deutsche Bank. Dirk joined Intel in 2001 and since then has been working in the Software and Services Group with a focus on the technology direction of Intel's Open Source Technology Center and Intel's engagements in open source. His interests range from kernel to user interaction, from massively scalable cloud services to mobile operating systems. He is an active contributor in many open source projects and organizations, various program committees and advisory boards and currently maintains the Subsurface dive log project. Dirk holds a Diploma in Mathematics and Computer Science from the University of Würzburg, Germany. He lives in Portland, OR, USA.
For more information on Dirk and his presentation, see here.
It has been an extremely long time between beers (10 months!). I’ve gotten out of the habit of blogging and somehow I never blogged about the talk I co-presented at PyCon AU this year on Pallet and Forklift the standard and tool we’ve developed at Infoxchange to help make it easier to develop web-applications on Docker1.
Infoxchange is one of the few places I’m aware of that runs Docker in prod. If you’re looking at using Docker to do web development, it’s worth checking out what we’ve been doing over on the Infoxchange devops blog.
- There’s also Straddle Carrier, a set of Puppet manifests for loading Docker containers on real infrastructure, but they’ve not been released yet as they rely too much on our custom Puppet config.
Zoe slept all night and even slept in a little bit, which was nice, given her late night.
I thought that given it was a nice day and the tide times were well suited for it, that we could go out to Wellington Point again, and walk out to King Island. I suspect the school holidays are going to be a bit of a "best of" things that we've done throughout the year.
I whipped up a quick picnic lunch after breakfast, and we made it out there in good time for low tide. We didn't end up walking all the way out to King Island. Zoe had a great time looking at all the baby crabs running around and went fossicking for shells instead. After a while doing that and not making a lot of progress towards King Island, she'd had enough, so we turned around and had a bit of a play in the park, which included some climbing on the big climbing tree. Zoe wasn't particularly confident this time around, and was resorting to shimmying along the tree, which wasn't terribly compatible with her choice of clothing.
After that, we pulled out the picnic blanket and had a lovely picnic in the shade. The weather really was beautiful today. Not a cloud in the sky, not too hot, and a nice cool breeze.
After lunch, we went back to the playground, and Zoe had another go climbing the tree. This time, after I pointed out that it was just like the balance beam at Tumble Tastics, she veritably charged up the tree walking upright.
She was actually a little too confident, and once she reached the trunk headed up the higher branch running perpendicular to the long low one. I lost my nerve once she got about 10 metres above the ground and out over the concrete and picnic tables, and asked her to come back down. She was doing fine, but I was more worried about how she was going to turn around, and if she was going to lose her nerve and get stuck up there.
I was glad when she made it back down safe and sound. I'm proud to have such a confident and capable daughter, but sometimes it's hard being a free range parent.
We headed home after that, and did a spot of grocery shopping for dinner on the way home. Zoe wanted to go to the park, so after we got home and unpacked, we biked back to the park for a little while, before biking to our haircut appointment.
After that, it was dinner and bed time. I'm hoping we'll have another good night's sleep.
The screws had pulled out of the door frame on the bottom hinges of Zoe's door. I'd found a pretty straightforward looking Instructable on how to repair the situation. As I had a lot of dowel left over from when I built a couple of clothes lines for Zoe, I cut a few short pieces from the long length I had.
Unfortunately getting an exact length was impossible, so I had a bit of dowel sticking out that I needed to sand down, so after Sarah dropped Zoe off, we headed over to Bunnings to get a small drill-mounted sanding disc so I could sand them flush with the door frame.
After I successfully fixed the door, I thought we should go visit Bryce, since it's been quite a while since we've seen him. He wasn't feeling up for an outing, so we just visited him in the Masonic Centre at Sandgate and took him some mince pies.
Since we were relatively close to my parents, we dropped in on them for lunch afterwards, and we watched the photo slideshow DVD that Zoe's Kindergarten had given me on her second-last day, and flicked through her "yearbook" and portfolio.
Zoe napped in the car on the way home, and based on some behaviour in the morning, I figured she could do with it, so I let her nap a bit longer and we drove into the city to pick up her lunchbox from Biome. I probably blew the benefit of shaving on shipping by using their "click and collect" option by paying to park in the Myer Centre, but Zoe was certainly perkier after her nap.
After that, we went home, and I made a quick dinner. I wasn't going to attend my final Thermomix branch meeting because I had Zoe, but I decided in the morning, that given it wasn't a "school night", and the meeting was closer to home than usual. that I might try getting her all ready for bed and bringing her with me.
Fortunately I still had her Trunki all packed with amusements from our US trip in July, so I brought that with us, and that kept her sufficiently amused. She came up for a few cuddles at various points, but was otherwise happy to play quietly at the back of the room. She was really well behaved, and my Group Leader again complimented her on how well behaved she was.
That made for a bit of a late bedtime, but she did well. The nap in the car definitely helped.
11:35am Wednesday 14th January 2015
Jonathan Corbet is the lead editor of LWN.net, co-author of Linux Device Drivers, a member of the Linux Foundation's Technical Advisory Board, and a occasional kernel contributor.
For more information on Jonathan and his presentation, see here.
Josh Berkus PostgreSQL Replication Tutorial
1:20pm Wednesday 14th January 2015
Josh Berkus is best known as a core team member of the global PostgreSQL database project. He's also CEO of PostgreSQL Experts Inc., and sits on the board of several database startups. As well as PostgreSQL, Josh dabbles Python, Perl, Redis, and Docker these days, but ask him for an update when you see him. He's had a Linux desktop since 1998.
For more information on Josh and his presentation, see here.
Mark McClain Tunnels and bridges: A drive through OpenStack Networkings
1:20pm Thursday 15th January 2015
Mark McClain is a Senior Principal Architect at Yahoo!, member of the OpenStack Technical Committee, and is a core reviewer of the the OpenStack Networking Project. He served as the Technical Lead for Neutron during the Havana and Icehouse cycles. Mark has 14 years of software development experience and OpenStack Networking combines two of his favorite interests: networking and Python.
For more information on Mark and his presentation, see here.
Trigger warning, I suppose.
This like a Tom Clancy book, but with weirder sex, much of it non-consensual. Also, not as well thought through or as well researched or as believable. I couldn't bring myself to finish it.
Tags for this post: book john_ringo terrorism nuclear
Related posts: Citadel; Hell's Faire; Princess of Wands; East of the Sun, West of the Moon; Watch on the Rhine; Cally's War Comment Recommend a book
- Add more detailed network information to the metadata server: review 85673.
- Add separated policy rule for each v2.1 api: review 127863.
- Add user limits to the limits API (as well as project limits): review 127094.
- Allow all printable characters in resource names: review 126696.
- Consolidate all console access APIs into one: review 141065.
- Expose the lock status of an instance as a queryable item: review 127139 (abandoned); review 85928 (approved).
- Extend api to allow specifying vnic_type: review 138808.
- Implement instance tagging: review 127281 (fast tracked, approved).
- Implement the v2.1 API: review 126452 (fast tracked, approved).
- Improve the return codes for the instance lock APIs: review 135506.
- Microversion support: review 127127 (approved).
- Move policy validation to just the API layer: review 127160.
- Nova Server Count API Extension: review 134279 (fast tracked).
- Provide a policy statement on the goals of our API policies: review 128560 (abandoned).
- Sorting enhancements: review 131868 (fast tracked, approved).
- Support JSON-Home for API extension discovery: review 130715.
- Support X509 keypairs: review 105034 (approved).
- Expand support for volume filtering in the EC2 API: review 104450.
- Implement tags for volumes and snapshots with the EC2 API: review 126553 (fast tracked, approved).
- Actively hunt for orphan instances and remove them: review 137996 (abandoned); review 138627.
- Check that a service isn't running before deleting it: review 131633.
- Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705 (abandoned).
- Implement a daemon version of rootwrap: review 105404.
- Log request id mappings: review 132819 (fast tracked).
- Monitor the health of hypervisor hosts: review 137768.
- Remove the assumption that there is a single endpoint for services that nova talks to: review 132623.
- Allow direct access to LVM volumes if supported by Cinder: review 127318.
- Cache data from volumes on local disk: review 138292 (abandoned); review 138619.
- Enhance iSCSI volume multipath support: review 134299.
- Failover to alternative iSCSI portals on login failure: review 137468.
- Give additional info in BDM when source type is "blank": review 140133.
- Implement support for a DRBD driver for Cinder block device access: review 134153.
- Refactor ISCSIDriver to support other iSCSI transports besides TCP: review 130721 (approved).
- StorPool volume attachment support: review 115716.
- Support Cinder Volume Multi-attach: review 139580 (approved).
- Support iSCSI live migration for different iSCSI target: review 132323 (approved).
- Cells Scheduling: review 141486.
- Create an instance mapping database: review 135644.
- Flexible cell selection: review 140031.
- Implement instance mapping: review 135424 (approved).
- Populate the instance mapping database: review 136490.
- Initial specification: review 114044 (abandoned).
- Enforce instance uuid uniqueness in the SQL database: review 128097 (fast tracked, approved).
- Nova db purge utility: review 132656.
- Online schema change options: review 102545.
- Support DB2 as a SQL database: review 141097 (fast tracked, approved).
- Validate database migrations and model': review 134984 (approved).
- Migrate the Docker Driver into Nova: review 128753.
- Implement support for FreeBSD networking in nova-network: review 127827.
- Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190 (approved).
- Instance hot resize: review 141219.
- Add config drive support: review 98930 (approved).
- Pass through flavor capabilities to ironic: review 136104.
- Add ephemeral disk support to the VMware driver: review 126527 (fast tracked, approved).
- Add support for the HTML5 console: review 127283.
- Allow Nova to access a VMWare image store over NFS: review 126866.
- Enable administrators and tenants to take advantage of backend storage policies: review 126547 (fast tracked, approved).
- Enable the mapping of raw cinder devices to instances: review 128697.
- Implement vSAN support: review 128600 (fast tracked, approved).
- Support multiple disks inside a single OVA file: review 128691.
- Support the OVA image format: review 127054 (fast tracked, approved).
- Add Quobyte USP support: review 138372 (abandoned); review 138373 (approved).
- Add VIF_VHOSTUSER vif type: review 138736 (approved).
- Add a Quobyte Volume Driver: review 138375 (abandoned).
- Add finetunable configuration settings for virtio-scsi: review 103797 (abandoned).
- Add large page support: review 129608 (approved).
- Add support for SMBFS as a image storage backend: review 103203 (approved).
- Allow scheduling of instances such that PCI passthrough devices are co-located on the same NUMA node as other instance resources: review 128344 (fast tracked, approved).
- Allow specification of the device boot order for instances: review 133254.
- Allow the administrator to explicitly set the version of the qemu emulator to use: review 138731 (abandoned).
- Consider PCI offload capabilities when scheduling instances: review 135331.
- Convert to using built in libvirt disk copy mechanisms for cold migrations on non-shared storage: review 126979 (fast tracked).
- Derive hardware policy from libosinfo: review 133945.
- Implement COW volumes via VMThunder to allow fast boot of large numbers of instances: review 128810 (abandoned); review 128813 (abandoned); review 128830 (abandoned); review 128845 (abandoned); review 129093 (abandoned); review 129108 (abandoned); review 129110 (abandoned); review 129113 (abandoned); review 129116; review 137617.
- Implement configurable policy over where virtual CPUs should be placed on physical CPUs: review 129606 (approved).
- Implement support for Parallels Cloud Server: review 111335 (approved); review 128990 (abandoned).
- Implement support for zkvm as a libvirt hypervisor: review 130447 (approved).
- Improve total network throughput by supporting virtio-net multiqueue: review 128825.
- Improvements to the cinder integration for snapshots: review 134517.
- Quiesce instance disks during snapshot: review 128112; review 131587 (abandoned); review 131597.
- Real time instances: review 139688.
- Stop dm-crypt device when an encrypted instance is suspended or stopped: review 140847 (approved).
- Support SR-IOV interface attach and detach: review 139910.
- Support StorPool as a storage backend: review 137830.
- Support for live block device IO tuning: review 136704.
- Support libvirt storage pools: review 126978 (fast tracked, approved).
- Support live migration with macvtap SR-IOV: review 136077.
- Support quiesce filesystems during snapshot: review 126966 (fast tracked, approved).
- Support using qemu's built in iSCSI initiator: review 133048 (approved).
- Volume driver for Huawei SDSHypervisor: review 130919.
- Allow portions of an instance's uuid to be configurable: review 130451.
- Attempt to schedule cinder volumes "close" to instances: review 130851; review 131050 (abandoned); review 131051 (abandoned); review 131151 (abandoned).
- Dynamic server groups: review 130005 (abandoned).
- Improve the performance of unshelve for those using shared storage for instance disks: review 135387.
- A lock-free quota implementation: review 135296.
- Automate the documentation of the virtual machine state transition graph: review 94835.
- Fake Libvirt driver for simulating HW testing: review 139927 (abandoned).
- Flatten Aggregate Metadata in the DB: review 134573 (abandoned).
- Flatten Instance Metadata in the DB: review 134945 (abandoned).
- Implement a new code coverage API extension: review 130855.
- Move flavor data out of the system_metadata table in the SQL database: review 126620 (approved).
- Move to polling for cinder operations: review 135367.
- PCI test cases for third party CI: review 141270.
- Transition Nova to using the Glance v2 API: review 84887.
- Transition to using glanceclient instead of our own home grown wrapper: review 133485 (approved).
- Enable lazy translations of strings: review 126717 (fast tracked).
- Add a new linuxbridge VIF type, macvtap: review 117465 (abandoned).
- Add a plugin mechanism for VIF drivers: review 136827.
- Add support for InfiniBand SR-IOV VIF Driver: review 131729.
- Neutron DNS Using Nova Hostname: review 90150 (abandoned).
- New VIF type to allow routing VM data instead of bridging it: review 130732.
- Nova Plugin for OpenContrail: review 126446 (approved).
- Refactor of the Neutron network adapter to be more maintainable: review 131413.
- Use the Nova hostname in Neutron DNS: review 137669.
- Wrap the Python NeutronClient: review 141108.
- Dynamically alter the interval nova polls components at based on load and expected time for an operation to complete: review 122705.
- A nested quota driver API: review 129420.
- Add a filter to take into account hypervisor type and version when scheduling: review 137714.
- Add an IOPS weigher: review 127123 (approved, implemented); review 132614.
- Add instance count on the hypervisor as a weight: review 127871 (abandoned).
- Allow extra spec to match all values in a list by adding the ALL-IN operator: review 138698 (fast tracked, approved).
- Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530 (abandoned).
- Allow the remove of servers from server groups: review 136487.
- Convert get_available_resources to use an object instead of dict: review 133728 (abandoned).
- Convert the resource tracker to objects: review 128964 (fast tracked, approved).
- Create an object model to represent a request to boot an instance: review 127610 (approved).
- Decouple services and compute nodes in the SQL database: review 126895 (approved).
- Enable adding new scheduler hints to already booted instances: review 134746.
- Fix the race conditions when migration with server-group: review 135527 (abandoned).
- Implement resource objects in the resource tracker: review 127609.
- Improve the ComputeCapabilities filter: review 133534.
- Isolate Scheduler DB for Filters: review 138444.
- Isolate the scheduler's use of the Nova SQL database: review 89893.
- Let schedulers reuse filter and weigher objects: review 134506 (abandoned).
- Move select_destinations() to using a request object: review 127612 (approved).
- Persist scheduler hints: review 88983.
- Refactor allocate_for_instance: review 141129.
- Stop direct lookup for host aggregates in the Nova database: review 132065 (abandoned).
- Stop direct lookup for instance groups in the Nova database: review 131553 (abandoned).
- Support scheduling based on more image properties: review 138937.
- Trusted computing support: review 133106.
- Dynamic Management of Server Groups: review 139272.
- Make key manager interface interoperable with Barbican: review 140144 (fast tracked, approved).
- Provide a reference implementation for console proxies that uses TLS: review 126958 (fast tracked, approved).
- Strongly validate the tenant and user for quota consuming requests with keystone: review 92507.
- Pacemaker service group driver: review 139991.
- Transition service groups to using the new oslo Tooz library: review 138607.
- Add soft affinity support for server group: review 140017 (approved).
Tags for this post: openstack kilo blueprint spec nova
Related posts: Specs for Kilo; One week of Nova Kilo specifications; Compute Kilo specs are open; Specs for Kilo; Juno nova mid-cycle meetup summary: slots; Juno nova mid-cycle meetup summary: nova-network to Neutron migration
First off, there is a periodic task run the nova-compute process (or the computer manager as a developer would know it), which runs every reclaim_instance_interval seconds. It looks for instances in the SOFT_DELETED state which don't have any tasks running at the moment for the hypervisor node that nova-compute is running on.
For each instance it finds, it checks if the instance has been soft deleted for at least reclaim_instance_interval seconds. This has the side effect from my reading of the code that an instance needs to be deleted for at least reclaim_instance_Interval seconds before it will be removed from disk, but that the instance might be up to approximately twice that age (if it was deleted just as the periodic task ran, it would skip the next run and therefore not be deleted for two intervals).
Once these conditions are met, the instance is deleted from disk.
Tags for this post: openstack nova instance delete
Related posts: One week of Nova Kilo specifications; Specs for Kilo; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno Nova PTL Candidacy; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic
A well known set of security enhancements to the Linux kernel is the grsecurity patch. The grsecurity patch is a (large) patch that applies cleanly against selected supported stock Linux kernel versions. It brings with it PAX, which protects against various well known memory exploits, plus a number of other hardening features including logging time and mount changes. In particular it enables features such as Non-executable stack (NX) on platforms that do not provide NX in hardware, such as MIPS devices and older x86.OpenWRT hardening
OpenWRT is a widely adopted embedded / router Linux distribution. It would benefit greatly from including grsecurity, in particular given most MIPS platforms do not support NX protection in hardware. However for a long time the differences between the OpenWRT kernel and the kernel revisions that grsecurity is supported on have been significant and would likely have taken an extreme effort to get working, let alone get working securely.
This is a shame, because there is malware targeted at consumer embedded routers, and it must only be a matter of time before OpenWRT is targeted. OpenWRT is widely regarded as relatively secure compared to many consumer devices, at least if configured properly, but eventually some bug will allow a remote binary to be dropped. It would be helpful if the system can be hardened and stay one step ahead of things.
The OpenWRT development trunk (destined to become the next release, ‘Chaos Calmer’ in due course) has recently migrated most devices to the 3.14 kernel tree. Serendipidously this aligns with the long term supported grsecurity revision 3.14. When I noticed this I figured I’d take a look at whether it was feasible to deploy grsecurity with OpenWRT.Applying grsecurity – patch
In late November I pulled the latest OpenWRT sources and the kernel version was 3.14.25, which I noticed matched the current grsecurity stable branch 3.14.25
The grsecurity patch applies cleanly against a stock kernel, and OpenWRT starts with a stock kernel and then applies a series of patches designed to extend hardware support to many obscure embedded things not present in the mainline kernel, along with patches that reduce the memory footprint. Some of the general patches are pushed upstream but may not yet have been accepted, and some could be backports from later kernels. Examples of generic patches include a simplified crash report.
Anyway, I had two choices, and tried them both: apply grsecurity, then the OpenWRT patches; or start with the OpenWRT patched kernel. In both cases there were a number of rejects, but there seemed to be less when I applied grsecurity last. I also decided this would be easier for me to support for myself going forward, a decision later validated successfully.
OpenWRT kernel patches are stored in two locations; generic patches applying against any platform, then platform specific patches. My work is tested against the Carambola2, an embedded MIPS board supported by the ‘ar71xx’ platform in OpenWRT, so for my case, there were ar71xx patches.
To make life easy I wrote a script that would take a directory of OpenWRT kernel patches, apply to a git kernel repository and auto-commit. This allowed me to use gitg and git difftool to examine things efficiently. It also worked well with using an external kernel tree to OpenWRT so I didnt have to worry yet about integrating patches into OpenWRT. This script is on github, it should be easily adaptable for other experiments.
(Note: to use an external tree, managed by git, use config options like the following:CONFIG_KERNEL_GIT_CLONE_URI="path/to/linux-stable" CONFIG_KERNEL_GIT_LOCAL_REPOSITORY="path/to/linux-stable" CONFIG_KERNEL_GIT_BRANCH="owrt_grsec_v3.14.25"
There were four primary rejects that required fixing. This involved inspecting each case and working out what OpenWRT had changed in the way. Generally, this was caused because one or the other had modified the end of the same structure or macro, but luckily it turned out nothing significant and I was able to easily reconcile things. The hardest was because OpenWRT modifies vmstat.c for MIPS and the same code was modified by grsecurity to add extra memory protections. At this point I attempted to build the system, and discovered three other minor cases that broke the build. These mispatches essentially were due to movements in one or two lines, or new code using internal kernel API modified by grsecurity, and were also easily repaired. The most difficult mispatch to understand was where OpenWRT rewrites the kernel module loader code, apparently to make better use of MIPS memory structures and it took me a little while to understand how to try and fix things.
The end result is on github at https://github.com/pastcompute/openwrt-cc-linux-3.14.x-grsecurityApplying grsecurity – OpenWRT quirks
One strange bug that had to be worked around was some new dependency in the kernel build process, where extra tools that grsecurity adds were not being built in the correct order with other kernel prerequisites.
In the end I had to patch how OpenWRT builds the kernel to perform an extra ‘make olddefconfig‘ to sort things out.
I also had to run ‘make kernel_menuconfig‘ and turn on grsecurity.
As the system built, I eventually hit another problem area: building packages. This was a bit of an ‘OH-NO’ moment as I thought it had the potential to become a big rabbit hole. Luckily as it turned out, only one package was affected in the end: compat-wireless. This package builds some extra user space tools and wifi drivers, and used a macro, ACCESS_ONCE, that was changed by grsecurity to be more secure; and required use of a new macro to make everything work again, ACCESS_ONE_RW. There were rather a number of calls to this macro, but luckily it turned out to be fixable using sed!Booting OpenWRT with grsecurity – modules not loading
I was able to then complete an INITRAMFS image that I TFTP’d into my carambola2 via uboot.
Amazingly the system booted and provided me with a prompt.U-Boot 1.1.4-g33f82657-dirty (Sep 16 2013 - 16:09:28) ===================================== CARAMBOLA2 v1.0 (AR9331) U-boot Starting kernel ... [ 0.000000] Linux version 3.14.26-grsec (andrew@atlantis4) (gcc version 4.8.3 (OpenWrt/Linaro GCC 4.8-2014.04 r43591) ) #3 Sun Dec 14 18:08:52 ACDT 2014
I then discovered that no kernel modules were loading. A bit of digging and it turns out that a grsecurity option, CONFIG_GRKERNSEC_RANDSTRUCT will auto-enable CONFIG_MODVERSIONS. One thing I learned at this point is that OpenWRT does not support CONFIG_MODVERSIONS=y, due to the way it packages modules with its packaging system. So an iteration later with the setting disabled, and everything appeared to be “working”Testing OpenWRT with grsecurity
Of course, all this work is moot if we cant prove it works.
Easy to check is auditing. For example, we now had these messages:[ 4.020833] grsec: mount of proc to /proc by /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 [ 4.020833] grsec: mount of sysfs to /sys by /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0 [ 4.041666] grsec: mount of tmpfs to /dev by /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0, parent /[swapper:0] uid/euid:0/0 gid/egid:0/0
However, the acid test would be enforcement of the NX flag. Here I used the code from http://wiki.gentoo.org/wiki/Hardened/PaX_Quickstart to test incorrect memory protections. Result:[19111.666360] grsec: denied RWX mmap of <anonymous mapping> by /tmp/bad[bad:1497] uid/euid:0/0 gid/egid:0/0, parent /bin/busybox[ash:467] uid/euid:0/0 gid/egid:0/0 mmap failed: Operation not permitted
Success!Revisiting Checksec, and tweaking PAX
In an earlier blog I wrote about experimenting with checksec. Here I used it to double-check that the binaries were built with NX protection. MOst were, due to a patch I previously submitted to OpenWRT for MIPS. However, openssl was missing NX. It turns out that OpenSSL amongst everything else it has been discussed for this year, uses assembler in parts of the encryption code! I was able to fix this by adding the relevant linker ‘.note.GNU-stack‘ directive.
The PAX component can be tweaked using the paxctl command, so I had to build that with the OpenWRT toolchain to try it out. I discovered that it doesnt work for files on the JFFS2 partition, only in the ramdisk. Further to enable soft mode, you need to add a kernel boot command line argument. To do this for OpenWRT, edit a file called target/linux/$KERNEL_PLATFORM/generic/config-default where in my case, $KERNEL_PLATFORM is ar71xxMoving Targets
Right in the middle of all this, OpenWRT bumped the kernel to 3.14.26. So I had to exercise a workflow in keeping the patch current. As it happened the grsecuroty patch was also updated to 3.14.26 so I presume this made life easier.
After downloading the stock kernel and pulling the latest OpenWRT, I again re-created the patch series, then applied grsecurity 3.14.26. The same four rejects were present again, so fingers crossed I cherry-picked all my work from 3.14.25 onto 3.14.26. As luck would have it this was one smooth rebase!Recap of OpenWRT grsecurity caveats
- CONFIG_GRKERNSEC_RANDSTRUCT is not compatible with the OpenWRT build system; using it will prevent modules loading
- Some packages may need to be modified to support NX – generally, if these use assembly language and don’t use the proper linker directive.
- For some reason paxctl only seems to work on files in /tmp not in the JFFS overlay. This is probably only a problem when debugging
- Your experience with the debugger gdb will probably be sub-optimal unless you put the debug target on /tmp and use paxctl to mark it with exceptions
After concluding the above, I converted the change set from my local Linux working copy into a set of additional patches on OpenWRT and rebuilt everything to double check.
The branch ‘ar71xx-3.14.26-grsecurity’ in https://github.com/pastcompute/openwrt-cc-ar71xx-hardened has all the work, along with some extra minor fixes I made to some other packages related to checksec scan results.
THIS MAY EXPLODE YOUR COMPUTER AND GET YOU POWNED! This has been working for me on one device with minimal testing and is just a proof of concept.
I’m interested in developing a VHF mode for FreeDV. One intriguing possibility is to connect a modem to legacy analog FM radios, which would allow them to be re-purposed for digital voice. One candidate is FSK at about 1200 bit/s, which is often used over FM for APRS. This operates through FM radios using the mic/speaker ports on $50 HTs, no special data ports required.
So I want to know the performance of FSK over FM in terms of Bit Error Rate (BER) for a given SNR. That got me thinking. When you send FSK through a SSB radio, it faithfully mixes them up to RF and you get FSK over the channel. The SSB radio just adds a frequency translation step. So we can model FSK like this:
However sending a FSK modem signal through a FM radio is very different:
FSK over FM is not FSK when you look at the over the air waveform. The spectrum is no longer two tones bouncing back and forth. So what is it?
I wrote a simulation called fsk.m to find out. This involved building up a FSK modem, and an analog FM radio simulation. The modem took me only a few hours but I was struggling with the analog FM simulation for a week! In particular making my FM demodulator get the same results as the theory. FM is a bit old school for me, so I had to hit the ARRL handbook and do a bit of research.
It’s a BEL202 simulation (as used for the APRS physical layer); 1200/2200 Hz tones, 1200 bit/s. I’m using the integrate and dump demodulation method and it matches the theoretical curves for non-coherent BFSK. Here is the FSK modem in action. First the FSK time domain signal and spectrum. The spectrum is a bunch of energy between 1200 and 2200 Hz. Makes sense as the modulator keeps moving back and forth between those two frequencies.
The next figure shows the sames signals with a 10dB SNR. Although the time domain signal looks bad, it actually has a BER one error in every 1000 bits (1E-3). The reason it looks so bad is that in the time domain we are seeing the noise from the entire bandwidth (our sample rate is Fs=96kHz). The demod effectively filters most of that out.
This next plot shows the output from the 1200 and 2200Hz integrators in the FSK demodulator for the 10dB SNR case. The height measures the energy of the tone during that bit period. As we would expect, they are mirror images. When one detects a large amount of energy, the other detects a small amount of the other tone.
The next step was to build a simulation of the modulator and demodulator in an analog FM radio. I wrote some code to test the input Carrier to Noise Ratio (CNR) versus output SNR. The test signal was a 1000 Hz tone, and the modulator had a maximum deviation of 5kHz, and a maximum input audio frequency of 3 kHz. After the demodulator I notched out the 1000 Hz tone so I could measure the noise power, the input to the notch filter was signal plus noise.
Here is the spectrum at the FM demodulator input for a 1000 Hz test tone:
The top plot is the tx signal centred on a 24 kHz carrier, in the bottom plot it has been mixed down to baseband and filtered. The FM signal is 16 kHz wide, as per Carsons rule. Here is the output of the FM demodulator:
At the top is a nice sine wave, and the bottom also shows the sine wave. You can see the effect of the output 3kHz low pass filter used to limit the noise bandwidth of the demod output.
When tested over a range of CNR inputs, I achieved output SNRs (red) in line with the text books (green):
At about 9dB the demodulator falls away from theory as the FM demodulator falls over, this is pretty typical. The theoretical model I have used is only valid above this 9dB threshold. You often hear this threshold effect in FM. The blue line is SSB for comparison. Over a certain threshold FM does quite a bit better in terms of output SNR for the same input CNR.
FSK over FM
OK so lets combine the simulations and look at the BER performance:
Oh dear. If my simulations are accurate, it appears FSK over FM is a lemon. About 7dB worse than regular FSK for the same BER. So using a FSK modem over a SSB radio would allow you to use 7dB less power than running the same modem through a FM radio. Coherent PSK is 3dB better again that FSK so that would get you a 10dB improvement. Simple FSK or PSK transmitters are easy to build too, and needing 7-10dB less output power would simplify them again (e.g. 100mW versus 1W).
Here is the spectrum at the FM demodulator input when sending FSK:
Note the FM spectrum looks nothing like regular FSK “over the air”, which looks like this:
So What went Wrong?
Given the plot of analog FM performance (say compared to SSB) above I had expected better results from FSK over FM.
I think I know where the problem lies. The input CNR is a measure of carrier power to noise power in the input bandwidth of the demodulator. Another way of looking at the VHF channel noise is a “floor”, which can be modelled as the average noise power per 1 Hz of bandwidth, called No.
So the Universe has given us a fixed “noise floor”, which will be the same for any modem. The FM demod input bandwidth is much wider, so it’s sucking up much more noise from the channel, which the poor demodulator has to deal with.
Lets plot the analog FM demod performance again, this time against C/No rather than C/N:
This takes into account the noise bandwidth, everything is “normalised” to the noise floor. When the C/No is beneath 48dB SSB looks much better. We can see a 7dB improvement over FM at low C/No values. This also explains why the microwave guys prefer SSB for their long shots.
Here is the BER curve scaled for C/No:
It appears the key to good modem performance is the RF bandwidth of the signal. Given a constant noise floor No, the signal bandwidth sets the total noise power N=NoB the demodulator has to deal with.
This has put me off the idea of a FreeDV VHF mode based on BEL202 FSK through legacy FM radios. I’d really like to come up with a mode that has sparkling BER versus SNR performance. I haven’t spent years making Codec 2 operate at low bit rates just to throw all those gains away in the modem!
Couple of ways forward:
- Take a look at GMSK.
- Consider developing a version of the SM1000 into an (open source) VHF SDR radio that can do PSK. Not as crazy as it seems. We are already planning a HF SDR version. Radio hardware is getting simple now the signal processing is all moving to software. We can make the modem so efficient that the PA can be modest (100s of mW).
- Dream up waveforms that can pass through legacy FM radios and have a low over-the air bandwidth. For example FSK that shifts between 300 and 400 Hz. In the past I’ve dreamed up new Codec 2 modes (1300 and 450 bit/s) to suit the properties of HF channels. So why no design a modem waveform to suit us? Go open source!
- Cop the performance hit and use BEL202 FSK. It might still be useful to use legacy FM radios for DV even with a 7dB loss in modem performance. It seems to work fine for APRS. If your C/No is high (as is often the case) then FSK over FM will have zero errors.
Zoe slept solidly until 6:48am. It was overcast and cooler, so I dare say that helped. Uninterrupted sleep is always nice. We had a nice snuggle in bed before we started the day.
First up, we had another doctor's appointment so the doctor could have another go at freezing off the wart on her hand. Despite some initial uncertainty, Zoe was much braver this time, and the doctor got to really hit it this time. Zoe was very proud of herself.
After the obligatory Freddo Frog for bravery, we headed home via the Valley to clear my PO box.
After a little bit of TV, we scootered to Tumble Tastics for her final class.
Tumble Tastics has been really great for Zoe. Zoe's always enjoyed gymnastics, and has definitely enjoyed this. She was very fond of Mr Fletcher, her teacher (she seems to really like male teachers) and especially loved the rope swing they had in the classroom. I was personally impressed by the theme that they did each week, and their ability to keep the activities in the relatively small room fresh and varied each week. They use the limited space that they have quite effectively. The fact that it was an easy distance from home was a bonus.
On our way back home, we discovered a stray dog on the side of Hawthorne Road. I checked its collar, and it had a mobile phone number on it, so I gave it a call. It turned out the owner was down at the supermarket, and his wife was at home with a baby, so I offered to return the dog for him.
It was only about a 500 metre walk, but it was very back-breaking, as the dog was pretty dumb and wouldn't follow us, so I head to lead it by the collar all the way, which involved me having to walk bent over all the way. Zoe wanted to help, but he was a bit to big and heavy for her to lead.
He was an interesting cross-breed. He had the markings of a blue heeler, but the head and general body shape of a terrier of some sort.
Due to some ambiguous letterboxes, we ended up at the the wrong house (off by one) and this house had a black Siamese cat that emerged from a boat parked in the front yard when I knocked on the door. Of course the dog decided to chase off after the cat, and I thought all was lost at that point, but he came back after having chased the cat away.
We then proceeded to the right house, returned the dog and went home for a well earned lunch.
After lunch, we went for a walk in the rain to post a letter. Zoe had a great time puddle jumping in her rain boots. We also made an opportunistic Christmas present purchase, and then went home again.
We had an unplanned afternoon of silly play for a while, with lots of running around and tickles and laughter. It was nice. Our downstairs neighbour, Deana, popped up to hang out for a bit as well, which was nice.
Zoe watched a bit of TV after that, and then Sarah arrived to pick her up.