Planet Linux Australia
OK, so FreeDV 700 was released a few weeks ago and I’m working on some ideas to improve it. Especially those annoying R2D2 noises due to bit errors at low SNRs.
I’m trying some ideas to improve the speech quality without the use of Forward Error Correction (FEC).
Speech coding is the art of “what can I throw away”. Speech codecs remove a bunch of redundant information. As much as they can. Hopefully with whats left you can still understand the reconstructed speech.
However there is still a bit of left over redundancy. One sample of a model parameter can look a lot like the previous and next sample. If our codec quantisation was really clever, adjacent samples would look like noise. The previous and next samples would look nothing like the current one. They would be totally uncorrelated, and our codec bit rate would be minimised.
This leads to a couple of different approaches to the problem of sending coded speech over channel with bit errors:
The first, conventional approach is to compress the speech as much as we can. This lowers the bit rate but makes the coded speech very susceptible to bit errors. One bit error might make a lot of speech sound bad. So we insert Forward Error correction (FEC) bits, raising the overall bit rate (not so great), but protecting the delicate coded speech bits.
This is also a common approach for sending data over dodgy channels. For data, we cannot tolerate any bit errors, so we use FEC, which can correct every error (or die trying).
However speech is not like data. If we get a click or a pop in the decoded speech we don’t care much. As long as we can sorta make out what was said. Our “Brain FEC” will then work out what the message was.
Which leads us to another approach. If we leave a little redundancy in the codec speech, we can use that to help correct or at least smooth out the received speech. Remember that for speech, it doesn’t have to be perfect. Near enough is good enough. That can be exploited to get us gain over a system that uses FEC.
Turns out that in the Bit Error Rate (BER) ranges we are playing with (5-10%) it’s hard to get a good FEC code. Many of the short ones break – they introduce more errors than they correct. The really good ones are complex with large block sizes (1000s of bits) that introduce unacceptable delay. For example at 700 bit/s, a 7000 bit FEC codeword is 10 seconds of coded speech. Ooops. Not exactly push to talk. And don’t get me started on the memory, MIPs, implementation complexity, and modem synchronisation issues.
These ideas are not new, and I have been influenced by some guys I know who have worked in this area (Philip and Wade if you’re out there). But not influenced enough to actually look up and read their work yet, lol.
So the idea is to exploit the fact that each codec model parameter changes fairly slowly. Another way of looking at this is the probability of a big change is low. Take a look at the “trellis” diagram below, drawn for a parameter that is represented by a 2 bit “codeword”:
Lets say we know our current received codeword at time n is 00. We happen to know it’s fairly likely (50%) that the next received bits at time n+1 will be 00. A 11, however, is very unlikely (0%), so if we receive a 11 after a 00 there is very probably an error, which we can correct.
The model I am using works like this:
- We examine three received codewords: the previous, current, and next.
- Given a received codeword we can work out the probability of each possible transmitted codeword. For example we might BPSK modulate the two bit codeword 00 as -1 -1. However when we add noise the receiver will see -1.5 -0.25. So the receiver can then say, well … it’s most likely -1 -1 was sent, but it also could have been a -1 1, and maybe the noise messed up the last bit.
- So we work out the probability of each sequence of three codewords, given the probability of jumping from one codeword to the next. For example here is one possible “path”, 00-11-00:
total prob =
(prob a 00 was sent at time n-1) AND
(prob of a jump from 00 at time n-1 to 11 at time n) AND
(prob a 11 was sent at time n) AND
(prob of a jump from 11 at time n to 00 at time n+1) AND
(prob a 00 was sent at time n+1)
- All possible paths of the three received values are examined, and the most likely one chosen.
The transition probabilities are pre-computed using a training database of coded speech. Although it is possible to measure these on the fly, training up to each speaker.
I think this technique is called maximum likelihood decoding.
Demo and Walk through
To test this idea I wrote a GNU Octave simulation called trellis.m
Here is a test run for a single trellis decode. The internal states are dumped for your viewing pleasure. You can see the probability calculations for each received codeword, the transition probabilities for each state, and the exhaustive search of all possible paths through the 3 received codewords. At the end, it get’s the right answer, the middle codeword is decoded as a 00.
For convenience the probability calculations are done in the log domain, so rather than multiplies we can use adds.
Here is a plot of 10 seconds of a 4 bit LSP parameter:
You can see a segments where it is relatively stable, and some others where it’s bouncing around. This is a mesh plot of the transition probabilities, generated from a small training database:
It’s pretty close to a “eye” matrix. For example, if you are in state 10, it’s fairly likely the next state will be close by, and less likely you will jump to a remote state like 0 or 15.
Here is test run using data from several seconds of coded speech:
loading training database and generating tp .... done
loading test database .... done
Eb/No: 3.01 dB nerrors 28 29 BER: 0.03 0.03 std dev: 0.69 1.76
We are decoding using trellis based decoding, and simple hard decision decoding. Note how the number of errors and BER is the same? However the std dev (distance) between the transmitted and decoded codewords is much better for trellis based decoding. This plot shows 10 seconds of a 4 bit decoded parameter:
See how the trellis based decoding produces smaller errors?
Not all bit errors are created equal. The trellis based decoding favours small errors that have a smaller perceptual effect (we can’t hear them). Simple hard decision decoding has a random distribution of errors. Sometimes you get the most significant bit of the binary codeword flipped which is bad news. You can see this effect above, with a 4 bit codeword, a MSB means a jump of +/- 8. These large errors are far less likely with trellis decoding.
Hear are some samples that compare trellis based decoding to simple hard decision decoding, when applied to Codec2 at 700 bit/s on a AWGN channel using PSK. Only the 6 LSP parameters are tested (short term spectrum), no errors or correction are applied to the excitation parameters (voicing, pitch, energy).Eb/No (dB) BER Trellis Simple (hard dec) big 0.00 Listen Listen 3.0 0.02 Listen Listen 0.0 0.08 Listen Listen
At 3dB, the trellis based decoding removes most of the effects of bit errors, and it sounds similar to the no error reference. At 0dB Eb/No, the speech quality is improved, with some exceptions. Fast changes, like the “W” in double-you, and the “B” in Bruce become indistinct. This is because when the channel noise is high, the probability model favours slow changes in the parameters.
Still – getting any sort of speech at 8% bit error rates with no FEC is pretty cool.
These techniques could be applied to FreeDV 1600, improving the speech quality with no aditional overhead. Further work is required to extend these ideas to all the codec parameters, such as pitch, energy, and voicing.
I need to train the transition probabilities with a larger database, or make it train in real time using off air data.
We could include other information in the model, like the relationship of adjacent LSPs, or how energy and pitch change slowly in strongly voiced speech.
Now 10% BER is an interesting, rarely explored area. The data guys start to sweat above 1E-6, and assume everyone else does. At 10% BER FEC codes don’t work well, you need a really long block size or a low FEC rate. Modems struggle due to syncronisation issues. However at 10% the Eb/No versus BER curves start to get flat, so a few dB either way doesn’t change the BER much. This suggests small changes in intelligibility (not much of a threshold effect). Like analog.
However for speech, we don’t need to correct all errors, we just need to make it sound like they are corrected. By leaving some residual redundancy in the coded speech parameters we can use probability models to correct errors in the decoded speech with no FEC overhead.
This work is another example of experimental work we can do with an open source codec. It combines knowledge of the channel, the demodulator and the codec parameters to produce a remarkable result – improved performance with no FEC.
This work is in it’s early stages. But the gains all add up. A few more dB here and there.
Binh Nguyen: Electronics (TV) Repair, Working at Amazon, and Dealing With a Malfunctioning Apple iDevice
- take precautions. If you've ever watched some of those guys on YouTube, you'll realise that they are probably amateur electrcians and have probably never been shocked/electrocuted before. It's one thing to work with small electronic devices. It's an entirely different matter to be working with mains voltage. Be careful...
- a lot of the time electronic failure will take occur gradually over time (although the amount of time can vary drastically obviously)
- don't just focus on repairing it so that power can flow through the circuit once more. It's possible that it will just fail once more. Home in on the problem area, and make sure everything's working. That way you don't have to keep on dealing with other difficulties down the track
- it may only be possible to test components outside of circuit. While testing components with a multimeter will help you may need to purchase more advanced and expensive diagnostic equipment to really figure out what the true cause of the problem is
- setup a proper test environment. Ideally, one where you have a seperate circuit and where there are safety mechanisms in place to reduce the chances of a total blackout in your house and to increase your personal safety
- any information that you take from this is at your own risk. Please don't think that any of the information here will turn you into a qualified electronics technician or will allow you to solve most problems that you will face
- a lot of the time information on the Internet can be helpful but only applies to particular conditions. Try to understand and work the problem rather than just blindly following what other people do. It may save you a bit of money over the long term
Philips 32PFL5522D/05 - Completely dead (no power LED or signs of life) - Diagnosis and repair
https://www.youtube.com/all_comments?v=TrphsEw8slw - electronics repair is becoming increasingly un-economical. Parts may be impossible to find and replacing the TV rather than fixing it may actually be cheaper (especially when the screen is cracked. It's almost certain that a new replacement is going to cost more than the set itself). The only circumstances where it's likely to be worth it is if you have cheap spare parts on hand or the type of failure involves a relatively small, minor, component. The other thing you should know is that while the device may be physically structured in such a way to appear modularised it may not fail in such a fashion. I've been reading about boards which fail but actually have no mechanism to stop it from bleeding into other modules which means you end up in an infinite, failure loop. Replace one bad component with a good one and the leftover apparently good component fails and takes out the new, good board eventually. The cycle then continues on forever before the technician realises this or news of such design spreads. You may have to replace both boards at the same time which then makes the repair un-economical
- spare parts can be extremely difficult to source or are incredibly expensive. Moreover, the quality of the replacement parts can vary drastically in quality. If at all possible work with a source of known quality. Else, ask for demo parts particularly with Asian suppliers who may provide them for free and as a means of establishing a longer term business relationship
- be careful when replacing parts. Try to do your bet to replace like for like. Certain systems will operate in a degraded state if/when using sub-par replacements but will ultimately fail down the line
- use all your senses (and head) to track down a failure more quickly (sight and smell in particular for burnt out components). Sometimes, it may not be obvious where the actual failure is as opposed to where it may appear to be coming from. For instance, one set I looked at had a chirping power supply. It had actually suffered from failures of multiple components which made it appear/sound as though the transformer had failed though. Replacement of all relevant components (not the transformer) resulted in a functional power supply unit and stopping of the chirping sound
- as with musical instruments, teardowns may be the best that you can get with regards to details of how a device should work
- components may be shared across different manufacturers. It doesn't mean that they will work if swapped though. They could be using different version of the same base reference board (similar to the way in which graphics and network cards rely on reference designs in the computing world)
Magnavox has a very similar layout to a similar size Phillips LCD TV
Apparently, Amazon are interested in some local talent.
There are some bemusing tales of recruitment and the experience of working there though.
If your iPhone, iPad, or iPod touch doesn't respond or doesn't turn on
Identify your iPad model
I recently got an email from my local member, Andrew Leigh, that raised an issue I feel passionately about; here is my response.
On 09/07/15 14:55, Andrew Leigh wrote:[snip] > > ▪ Some people have asked me *why Labor supported the government’s bill to > continue regional processing*. This is a tough question, on which reasonable > people can disagree, but the best answer to this is to read Bill Shorten’s > speech to the House of Representatives > > on the day the legislation was introduced. Hi Andrew,
I'm sorry, but I cannot agree with the logic Bill Shorten and the Labor party has expressed in that speech.
Firstly, anyone watching the international problems with refugees will realise that Australia's intake is pitiful and stingy compared to some of its key allies and comparable nations and especially when compared to its population size and lifestyle. It is hypocritical to say "we don't want people to risk journeying across the sea from Indonesia, but we're happy for them to remain illegal immigrants there", especially when you look at the life that those people face as refugees there.
As an aside, though, I would say that it is still partly correct - it is more humane for them to remain in Indonesia than to be detained indefinitely in the inhuman, underresourced and tortuous conditions on Manus Island and Nauru. It is shameful to me that the Labor party can ignore this obvious contradiction.
But more importantly, the logic that we're somehow denying "people smugglers a product to sell" by pushing boats back into international waters shows no understanding of people smuggling as a business. Australia is still very much a destination, it's just that people now come with visas on planes and they pay even more for this than they used to. There is still a thriving trade in getting people into Australia, it's just been made more expensive - in the same way that making heroin illegal has not caused it to suddenly vanish from the face of the earth.
All we're doing by punishing people who come by boat to seek refuge in Australia is punishing the very desperate, the worst off, the people who have literally fled with their clothes and nothing else.
Other people with money still arrive, overstay their visas, get jobs as illegal immigrants or on tourism visas. The ABC has exposed some of these ridiculous, unethical companies trading on foreign tourists and grey market labourers. The Labor party, of all parties, should be standing up for these people's rights yet it seems remarkably silent on this issue.
The point that I think Labor needs to learn and the point I ask you to express to your colleagues there is that we don't want Labor to return to its policies in 2010. We thought those were inhuman and unjust then, and we still do now. Invoking them as a justification for supporting the Government now is bad.
Personally, I want Labor to do three things with regard to refugees:
- Move back to on-shore detention and processing. The current system is vastly more expensive than it needs to be, and makes it more difficult for UN officials and our own members of parliament and judiciary to be able to examine the conditions of detention. The Coalition keeps telling everyone about how expensive their budget is but seems remarkably silent on why we're paying so much to keep refugees offshore.
- Provide better ways of settling refugees, such that one can cut the
"people smuggler" middle men out of the deal.
For example, set up refugee processing in places such as Sri Lanka and Afghanistan where many refugees come from. Set a fixed price per person for transportation and processing in Australia, such that it undercuts the people smugglers - according to figures I read in 2010 this could be $10,000 and still be 50% less than black market figures.
- Ensure accountability and transparency of the companies such as Serco that are running these centres. If the government was running them and people were being abused, the government would be held accountable; when private companies do this the government wipes its hands and doesn't do a thing.
I do appreciate your updates and information and I look forward to more of your podcasts.
All the best,
Panel: “How we work” featuring Lance Wiggs, Dale Clareburt, Robyn Kamira, Amie Holman – Moderated by Nat Torkington
- Flipside of Startups given by Nat
- Amie – UX and Services Designer for the Govt, thinks her job is pretty cool. Puts services online.
- Lance – Works for NZTE better by capital programme. Also runs an early stage fund. Multiple starts and fails
- Dale – Founded of Weirdly. Worked her way up to top of recruitment company (small to big). Decided to found something for herself.
- Robyn – Started business 25 years ago. IT consultant, musician, writer.
- Nat – Look at what you are getting from the new job. Transition to new phase in life. Want ot be positive.
- Types of jobs: Working for someone else, work for yourself, hire other people, investor. Each has own perks, rewards and downsides.
- Self employed
- Big risk around income, peaks and troughs. Robyn always lived at the bottom of the trough level of income. Some people have big fear where next job is coming from.
- Robyn – Charged Govt as much as possible. Later on charged just below what the really big boys charged. Also has lower rates for community orgs. Sniffed around to find out the rates. Sometimes asked the client. Often RFPs don’t explicityly say so you have to ask.
- Pricing – You should be embarrassed about how much you charge for services.
- Robyn – Self promotion is really hard. Found that contracts came out of Wellington. Book meetings in cafes back to back. Chat to people, don’t sell directly.
- Working for others
- Amie – Working in a new area of government. But it an area that is growing. Fairly permissive area, lots of gaps that they can fill.
- Dale – Great experience as an employee. In environment with lot of autonomy in a fast growing company.
- Lance – Worked from Mobile – Lots of training courses, overseas 6 months after hired. 4 years 4 different cities, steep learning curve, subsidized housing etc. “Learning curve stopped after 4 years and then I left”.
- Big companies downside: Multiple stakeholders, Lots of rules
- Big company upside: Can do startup on the side, eg a Family . Secure income. Get to play with big money and big toys.
- Everything on steroids
- Really exciting
- Starting all parts of a company at once
- Responsibility for business and people in it
- Crazy ups and downs. Brutal emotional roller-coaster
- Lance lists 5 businesses off the top of his head that failed that he was at. 3 of which he was the founder
- Worst that can happen is that you can lose your house
- Is this life for everyone? – Dale “yes it can be, need to go in with your eyes open”. “Starting a business can be for everyone. I’m the poorest I’ve ever been now but I’m the happiest I’ve ever been”
- At a startup you are not working for yourself, you are working for everybody else. Dale says she trys to avoid that.
- Robyn – “If you life is gone when you are in a business then you are doing it wrong.”
- If you are working from home you can get isolated, get some peer support and have a drink, coffee with some others.
- Robyn – Recomends “How to make friends and influence People”
- Jobhunters – Look for companies 1st and specific job 2nd
- Startup – Meet everyone that you know and ask their opinion on your pitch
- Young People going to Uni – You have to get work experience, as a recruiter she looks at experience 1st and pure academic history second.
- Balance between creating income, creating wealth, learning
- Know what you are passionate about and good at
- It is part of our jobs to support everyone around us. Promote other people
- Find the thing that is your passion
- When you are deliverying your passion then you are delivering sometime relevant
Pick and Mix
- Random Flag generator – @polemic
- See Wikipedia page for parts of a flag
- 3 hex numbers are palet
- 4 numbers represent the pattern
- Next number will be the location
- next number which color will be assigned
- Last number will be a tweak number
- Up to 8 or 9 of the above
- Took python pyevolve and run evolution on them.
- Alex @4lexNZ , @overtime
- E-sports corporate gaming league
- untested in NZ
- Someone suggested cold calling CEOs or writing them letter
- Simon @slyall (yes me)
- Low volume site for announcements
- Mutate testing
- Tweak test values of code, to reverse fuzzing
- Landway learning – @kiwimrdee
- Looking for computers to borrow for class
- They teach lots of stuff
- Poetry for computers – @kiwimrdee
- Hire somebody english/arts background who understand language rather than somebody from a CS background who understand machines
- Lossless image compression for the web
- Tools vary across the platform
- Glen – Make computers learn to play Starcraft 1
- Takes replays of humans playing starcraft
- Getting computer to learn to play from that DB
- It is struggling
- Emergent political structures in tabletops games
Never check in a bag – How to pack
- 48 hour bag
- Laptop and power
- Always – Zip up pouch, tissues , hand sanitizer, universal phone charger, breath mints, the littlest power plug (check will work in multiple voltages), Food bar, chocolate.
- If more than 48 hours – notebook, miso soup, headphones, pen, laptop charger, apple plugs ( See “world travel kit” on apple site)
- Get smallest power plug that will charge your laptop
- Bag 3 – Every video adapter in the world, universal power adapter, airport express.
- TP-link battery powered wifi adapters
- If going away just moves laptop etc to this bag
- Packing Cell
- Enough clothes to get me through 48 hours
- 2 * rolled tshirts (ranger rolling)
- 2 pairs of underwear
- 2 pairs of socks
- Toileties. Ziplock back that complies with TSA rules for gels etc.
- Other toiletries in different bag
- Rip off stuff from hotels, also Kmart and local stores.
- Put toiletries ziplock near door to other bag so easy to get out for security.
- Leave packing cell in Hotel when you go out
- Learn to Ranger roll socks and shirts etc.
- 6 weeks worth of stuff
- In the US you can have huge carry-on
- Packs 2 weeks worth of clothes
- Minaal Bag (expensive but cool).
- Schnozzel bag – Vacuum pack clothing bag
- Airlines allow 1 carryon bag up to 7 kgs + 1 bag for other items (heavy stuff can go into that)
- Pick multi-color packing sell so you can color-code them.
- Elizabeth Holmes and Matilda Kahl and Steve Jobs all wear same stuff every day.
- Wear Ballet Heals on the plane
- Woman no more than 2 pairs of shoes every, One of which must be good for walking long distances
Today I’m at the Gather 2015 conference. This was originally “Barcamp Auckland” before they got their own brand and went off to do random stuff. This is about my 5th year or so here (I missed one or two).
Website is gathergather.co.nz . They do random stuff as well as the conference.
- Welcome and intro to conference history from Ludwig
- Rochelle thanks the sponsors
- Where to go for dinner, no smoking, watch out for non-lanyard people., fire alarms, etc
- Quiet room etc
Lessions learnt from growing a small team from 5-15
- Around 30 people. Run by Ben, works at sitehost, previously worked at Pitch
- Really hard work. Takes a lot of time and real effort to build a great team
- Need dedicate time and resources to growing team, Need someone who is focussed on growing the team and keeping the current team working
- Cringe when people say “HR” but you need some in the sort of role and early on.
- At around 16 people and doesn’t have full HR person yet. Before FT have someone with scheduled time to focus on team or company culture. In ideal world that person might not be in a manager role but be a bit senior (so they hear what the lower level employees say.
- Variety and inclusion are keep to happy team
- Once you are at 10+ members team will be diverse so “one size fits all” won’t work anymore. Need to vary team activities, need to vary rewards. Even have team lunches at different places.
- Hire for culture and fit
- From the first person
- Easier to teach someone skills than to be a good team member
- Anecdote: Hired somebody who didn’t fit culture, was abrasive, good worker but lost productivity from others.
- Give people a short term trial to see if they fit in.
- You will need to change the way communicate as a team as it grows
- A passing comment is not enough to keep everybody in the loop
- Nobody wants to feel alienated
- Maybe chat software, noticeboard, shared calendar.
- Balance the team work the members do
- Everybody needs to enjoy the work.
- Give people interesting rewarding work, new tech, customer interaction
- Share the no-fun stuff too. Even roster if you have to. Even if somebody volunteers don’t make them always do it.
- Appreciate you team members
- Praise them if they have put a lot of work into something
- Praise them before you point out the problems
- Listen to ideas no matter who they come from.
- 5 Questions/Minutes rule
- If someone is working not well, wonder if problem is elsewhere in their life. Maybe talk to them. Job of everyone in the team
- Appreciate your teams work, reward them for it
- Do what feels right for your team. What works for some teams might not work for all. No “one size fits all”
- Building great teams isn’t science it is an art. Experiment a bit.
- Taking the time to listen to 10 people instead of just 5 takes longer. Maybe this can be naturally taken on by others in the team, no just the “boss”.
- Have a buddy for each new hire. But make sure the buddys don’t get overloaded my constantly doing this with each new hire.
- Going from 10 to 100 ppl. They same thing doesn’t work at each company size.
- The point where you can get everybody in a room till when you can’t. At that point you have multiple teams and tribalism.
- If you have a project across multiple teams then try and put everybody in that project together in a room.
- Have people go to each others standups
- Hire people who can handle change
- Problem if you you buy a small company, they small company may want to keep their culture.
- Company that does welcome dinners not farewell dinners
- Make sure people can get working when they arrive, have an email address etc, find out if they have preferences like nice keyboard.
- Don’t hire when you are extremely busy that you can’t properly get them onboard (or you may pick the wrong person). Never hire impulsively. Hire ahead of time.
- Don’t expect them to be fully productive straight away. Give them something small to start on, no too complicated, no to crazy dependant on your internal crazy systems. But make sure it is within their skill level in case they struggle.
- Maybe summer student projects. Find good people without being stuck with someone. Give them a project that isn’t high enough priority for the FT people.
- Create training material
I often find myself describing the digital domain to people who don't live and breathe it like I do. It's an intangible thing, and many of the concepts are coded in jargon. It doesn't help that every technology tool set uses it's own specific language, sometimes using the same words for very different things, or different words for the same things. What's a page? A widget? A layout? A template? A module, plugin or extension? It varies. The answer "depends".
Analogies can be a helpful communication tool to get the message across, and get everyone thinking in parallel.
One of my favourites, is to compare a web development project, to a landscape design project.
One of the first things you need to know, is who is this landscape for and what sort of landscape is it? The design required for a public park is very different to one suitable for the back courtyard of an inner city terrace house.
You also need to know what the maintenance resources will be. Will this be watered and tended daily? What about budget? Can we afford established plants, or should we plan to watch the garden grow from seeds or seedlings?
The key point of comparison, is that a garden, whether big or small, is a living thing. It will change, it will grow. It may die from neglect. It may become an un-manageable jungle without regular pruning and maintenance.
What analogies do you use to talk about digital design and development?
Image: XIIIfromTOKYO - Plan of the gardens of Versailles - Wikipedia - CC-BY-SA 3.0
Safety and security in SMEs
- Biggest challenge for one SME IT person very bad password practises
- PABX issues, default passwords on voicemail resulting in calls getting forwarded overseas, racking up a big bill
- Disable countries you don’t need
- Credit Limits on your account
- Good firewall practice
- Good pin/password practice
- SMEs wanted problem to go away since they had a business to run.
- No standards for IT in small business, everywhere is setup different
- 9 times out of 10 IT stifles business and makes things worse.
- Small businesses recognise value, don’t want to spend on stuff that doesn’t return value
- So many attack directions very hard to secure.
- If you let other people using your business devices its a huge risk. Do you let your kids play with your work phone/laptop?
- Biometrics don’t seem to be there yet
- Maybe cloud-based software is a solution.
- Pictures of before/after of satellite downlink and comms centre in Vanuatu after Cyclone
- Cellular network survived, Datacentre survived, Fibre network survived
- One month after disaster 80% of comms were restored
- NZ team just sent over material via Govt CIO
- Various other groups on the ground
- Lots of other people doing stuff. Some were uncoordinated with main efforts
- NZ people (Dean, Andy) Spent 90% of time on logistics and 10% of time on IT stuff
- Vanuatu people very busy. eg offshore people had own mailing list to discuss things and then filter them through to people on the ground
- Lots of offers from people.
- Plan not in place in Vanuata, they now have one though
- What people wanted was Generators and Satellite phones. Both of them are hard to ship via air due to Petrol/Lithium.
- Very hard for non-regular (not the top 5 NGOs) to get access to shipping in military planes etc
- Echo from people who had similar problems in Christchurch working with the regular agencies
- Guy from vodafone said their company (globally) has a cellphone site that can be split between normal plane luggage
- Twitter accounts for Wellington suburbs had a meeting with council
- Some community outreach from the councils to coordinate with others. community resilience. Paying for street BBQs etc. “Neighbours day”
- Vital infrastructure needs to have capacity in disaster.
- Orgs need to have plans in place beforehand
- Good co-operation between telcos in Christchurch Earthquake
- Mobile app for 111 currently being looked at
- Some parts of the privacy act can be loosened when disasters are declared to enable information sharing with agencies
- Options for UPS on UFB “modems”
Panel: Adapt or die? News media, new media, transmedia
- Panelists: Megan Whelan (Radio New Zealand), Alex Lee (Documentary Edge Festival), Walid Al-Saqaf (Internet Society), Tim Watkin (Executive Producer of The Nation and blogger), Carrie Stoddart-Smith (blogger).
- Panel moderator: Paul Brislen.
- Intro Megan
- Been at Radio NZ for 10 years. Website back then just frequencies and fax number
- Good at Radio, not doing Internet very well
- New Job as community engagement editor
- Internet completely changed how the job is done.
- Sacrifice accuracy and context sometimes to get the story out fast
- Because people can now get their first and publish. They are no longer the gatekeepers of information. Getting used to others knowing more thna we do
- Sees himself as creative entrepreneur
- Content a few years ago seeing documentaries play in the cinema
- Storytelling being distributed. Communities already telling their own stories.
- 2 types of people in Audience. Skimmers and people wanting to do a deep dive
- Story tellers only know who to tell the story, sometimes not so much on the technology
- Developing collaboration between technologists and creatives together
- Blogging and social media provided new spaces for stories
- Maori TV. Maori people in the “Ngati blogosphere”
- Telling our own stories not just having others telling them
- Media still highlights negative rather than positive stories about Maori
- “Social media & blogging and facilitate stories and getting to know each other online”
- But Internet allows Maori to bypass media to get positive stories out to to National/International audience
- Internet should be empowering tool
- Problem with Internet are people on it not the Internet themselves
- Characteristics of traditionalist media is that there is a gatekeeper
- New media is that everybody is responsible for their own actions
- 60% of what is on social media is fake.
- Every newsroom in NZ is running digital-first
- No sustainable profit model for media orgs online
- Digital tools give media a lot more tools and ability to create to stories
- Speed comes a loss of quality, loss of subeditors
- Internet has sucks a lot of money out of journalism (especially loss of classifieds)
- Nostalgia has forgotten how bad journalism has used to be.
- So much pressure on resources but less money
- Example of real-time fact checking during interviews
- Question for Alex.
- You want people to Interaction with docus, but past has shown people don’t really?
- Alex says that people have in the past
- Refers to national Film board of Canada websites and interaction with their documentaries
- These days all need docs are required by broadcasters and funders to have interaction and social media strategy
- Mixing of Advertising and journalism undermine content?
- A bit but it is a source of money that helps keep the rest afloat.
- Is mainstream media actually verified compared to Social media
- Yes it is
- Use varified accounts on twitter to at least ensure the person is real
- Opinion on tools such as “data miner” which takes news across internet and aggregates it?
- Newsrooms have a lot of expertise
- But less now as newsrooms get hallowed out
- 8 feature editors at NZ Herald 10 years ago. Just 1.5 now
- People can some fact-check journalism instantly
- Good in one way
- But diversity of knowledge means fact checking harder
- What the Economic side of this? Where do you see economic support for high-quality contact coming from?
- Sugar Daddy. Eg Washington Post supported by Jeff Bezos
- Some kind of paywall seems be an main option
- Responsibility to highlight stories and come back to old/ongoing stories
- Yes they are revisited by media
- How far though a digital day could somebody go and only experience Maori?
- Some people only tweet in Maori
- Work at places where people primary work in Maori
- If money is tight and media companies consolidate does media have the room to push against the “powers that be”
- Pretty much has always been the case
- Getting harder but not astronomically harder than it used to be.
A new snapshot version of DataparkSearch Engine has been released. You can get it on Google Drive.
Here is the list of changes since previous snapshot:
- Crossword section is now includes value of TITLE attribute of IMG tag and values of ALT and TITLE attributes of A and LINK tags found on documents pointing to the indexing document
- Meta property is now indexing
- URL info data is now stored for all documents with HTTP status code < 400
- configure is now understands --without-libextractor switch to build dpsearch without libextractor support even it has been installed
- robots.txt support is enabled for sites crawling using HTTPS scheme
- AuthPing command has been added to send authorisation request before getting documents from a web-site. See details below.
- Cookie command has been added.
- Add support for SOCKS5 proxy without authorisation and with username authorisation. See details below.
- A number of minor fixes
Some web-sites may serve different content to a logged in user. In most cases logging in process consists of sending a POST or GET HTTP request to a specific URL before you start to receive targeted content. You may use AuthPing command to send such authentication request before requesting any document from the web-site.
E.g.:AuthPing "POST https://commercial-site.ext.au/user/login.php u=bot%40user.ext.au&p=super%40pass"
This command specify a POST request to be send to the URL address https://commercial-site.ext.au/user/login.php with the following CGI loading: u=bot%40user.ext.au&p=super%40pass
AuthPing command should be specified before each Server/Realm/Subnet command it affects. And specified request is sent each time an indexing thread access a web-server for the first time in a run session.Using SOCKS5 proxy
Proxy command is now accepting proxy type option with value either http either socks5. If you need to use username authentication with SOCKS5 proxy please use ProxyAuthBasic command to specify username and password.
E.g.:Proxy socks5 localhost:9050
In this example a SOCKS5 proxy connection to local Tor system is specified which uses no authentication method for connection.
If you don't know what this is please see the following...
Easy to miss Push features
Helpful Push information.
Jeremy Ellis meets Ableton Push
Mad Zach Push Performance Walkthrough
It's basically an advanced, modern musical instrument/MIDI controller.
There have been others who have attempted to de-compile and extend/modify the behaviour of the device but while information and the extensions that have been provided have been interesting and useful they have been somewhat limited.
ableton: just release the py midi remote scripts
Live 9 MIDI Remote Scripts revealed...
I'm beginning to understand why. The following link provides an update to automatically generated documentation (via epydoc) of decompiled Ableton Remote Script code (my scripts for decompilation and automated documentation have been included in the package). https://sites.google.com/site/dtbnguyen/Decompiled-Documentation-Ableton-MIDI-Remote-Scripts.zip
If you want to make any additional modifications of behaviour you'll need to be aware of the following:
- you'll need to catch up on your Python coding
- you'll need knowledge of how the device works, music and mathematical theory, Ableton, and core computing knowledge. It is not sufficient to know how they work seperately. You need to know how everything fits together.
- sounds obvious but start small and move up. This is critical particularly with reference to the awkward style of programming that they can sometimes resort to. More on this to below
- the code can vary in quality and style quite significantly at times. At times it seems incredibly clean, elegant, and well documented. At other times, it there is no documentation at all and doesn't seem to be well designed or engineered or have keep maintenance in mind. For instance, a commonly used design pattern is MVC. This doesn't seem to follow that. They use a heap of sentinels throughout there code. Moreover, the characters that are used can be a bit confusing. They don't use preprocesser directives/constants where they may be better suited. If you break certain aspects of the code you can end up breaking a whole lot of other parts. This may be deliberate (to reduce the chances of third party modification which is likely particularly as there seems to be some authentication/handshake mechasnisms in the code to stop it from working with 'uncertified devices') or not (they lack resources or just have difficult timelines to deal with)
- be prepared to read through a lot of code just to understand/make a change to something very small. As stated previously strictly speaking at times they don't adhere to good practice. That said, other aspects can be changed extremely easily without breaking other components
- due to the previous two poits it should seem obvious that it can be very difficult to debug things sometimes. Here's the other thing you should note,
- the reason why Ableton suffers from strange crashes and hangs from time to time becomes much more obvious when you look at the way they code. In the past, I've built programs (ones which rely on automated code generation in particular) that relied on a lot of consecutive steps that required proper completion/sequencing for things to work properly. When things work well, things are great. When things break, you feel and look incredibly silly
- you may need to figure out a structure for ensuring and maintaining a clean coding environment. I try to have two screens with one for clean code and another for modified code. Be prepared to restart from scratch by reverting to a clean pyc code and only one or a small number of modified py files.
- caching occurs in situations where you may not entirely expect. If you can not explain what is happening and suspect caching just restart the system. Better yet, maintain your development environment in a virtual machine to reduce hardware stress caused by continual restarts.
- you will need patience. As stated previously, due to the way code has been structured (sometimes) you'll need to understand it properly to allow you to make changes without breaking other parts. Be prepared to modify, delete, or add code just to help you understand it
- if you've ever dealt with firmware or embedded devices on a regular basis you would be entirely familiar with some of what I'm talking about. Like a lot of embedded devices you'll have limited feedback if something goes wrong and you'll be scratching your head with regards to how to work the problem
You may require a lot of Linux/UNIX based tools and other debugging utilities such as IDA Pro, Process Explorer and Process Monitor from the Sysinternals Suite. Once you examine Ableton using such utilities, it becomes much clearer how the program has been structured, engineered, and designed. One thing that can cause mayhem in particular is the Ableton Indexer which when it kicks in at the wrong time can make it feel as though the entire system has frozen.
Ablton indexing Crashes
(42474187) Disable "Ableton Index" possible?
The actual index file/s are located at
- the most relevant log file is located at,
The timestamps works on the basis on amount of time since program startup. Time of startup is clearly outlined.
Ableton takes 130 - 2 mins to start up ?
Delete it if you need to if you get confused about how it works.
- be aware that there are some things that you can't do anything about. The original Novation Launchpad was considered somewhat sluggish in terms of refresh rate and latency. The electronics were subsequently updated in the Novation Launchpad S to deal with it. You may encounter similar circumstances here.
Push browser - slow, freezing, sluggish :(
- they have a strong utility/systems engineering mentality. A lot of files are archives which include relatively unobfuscated content. For instance, look in a
C:\Users\[username]\AppData\Roaming\Ableton\Live 9.1.7\Live Reports
and you'll find a lot of 'alp' files. These are 'Crash Reports' which are sent to Ableton to help debug problems. Rename them to a gz file extension and run it through hexedit. Same with 'adg' audio device group files. Rename to gz and gunzip to see a flat XML file containing some encoded information but mostly free/human readable content. It will be interesting to see how much of this can be manually altered achieving flexibility in programming without having to understand the underlying file format.
- each version of Ableton seems to have a small version of Python included. To make certain, advanced extensions others have suggested installing a libraries seperately or a different version of Python...
- be prepared to learn multiple protocols and languages in order to make the changes that you want
How to control the Push LCD text with sysex messages
For a lot of people, the device seems incredibly expensive for what amounts to a MIDI controller. It was much the same with me. The difference is that it's becoming increasingly clear how flexible the device can be with adequate knowledge of the platform.
http://blog.dubspot.com/5-ableton-push-software-devices/Push feature requests
The Ableton Push is a good platform but it will never realise it's full potential if the software isn't upgraded.
If you are interested in signing up to test the latest Beta version of Ableton please see the following...
Domains: growth, change, transition
- Transition of .nz to second level domains
- Some stuff re moving root zone control away from the US
- Problem with non-ascii domains (IDNs). They work okay, but not 3rd party apps or apps in Organisations. Eg can’t register on Facebook or other websites.
- 60% of Government Depts don’t accept IDNs as email addresses, lots of other orgs
- 1/3 of all new .nz domains created at second level
- Around 95k or 600k .nz domains now at second level (about 2/3s of these from rights are 3LD holder)
- Some people when you give them your address.nz change it into address.co.nz
- 1st principles of .nz whois public policy.
- People are in danger if they address is published
- But what the ability to contact the real owner of a domain
- 4 people in room with signed domains
- 300 signed .nz domains. 150 with DS record
- Around 3 people in room with new TLDs. See ntldstats.com for current stats
Internet of Things
- Where does the data from your house appliances go?
- Forwarded to other companies
- Issues need to be understandable by ordinary citizens especially terms and conditions
- Choose the data that you choose to share with the company rather than company choosing what it shares with you (and others)
- In health care area people worried about sharing data if it will affect their insurance premiums or coverage
- Many people don’t understand what their data is, they don’t understand that if every time they do something (on a device) it is stored and can be used later. How to educate people without sounding paranoid?
- “IoT is connecting things whose primary purpose is not connecting to the Internet”
- “The cost of sharing is bearable, because the sharing is valuable.”
- More granularities of trust. No current standards or experience or feeling for this since such a new area and rapidly evolving
- NZ law should override overly aggressive agreements (by overseas companies)
- Some discussion about standards, lots of them, full stack, piecemeal, rapidly changing
- Will the IoT make everything useless after the zombie apocalypse?
- “Denial of Service attack on your IoT pill bottle would be bad!”
- Concern that something like a pill bottle failing can put life in danger. Very high level of reliability needed which is rare and hard in software
Panel: Parliamentary Internet Forum
- With Gareth Hughes (Green Party), Clare Curran (Labour Party), Brett Hudson (National Party), Ria Bond (NZ First), Karen Melhuish Spencer (Core Education), Nigel Robertson (University of Waikato)
- What roles does the Education system play in the Internet
- National guy mostly talked about UFB and RBI programmes, computers in homes
- Gareth Hughes adopts the “I went out to XYZ School” story. Pushes Teachers not trained and 1 in 4 homes don’t have Internet access.
- Claire – Got distracted about discussion re her pants. But she said 40% of jobs at risk over next 10-15 years due to impact of technology
- Karen – I got distracted about another clothing related discussion on twitter
- Nigel – 1. Use the Internet to do what we already do better. Help people to use the Internet better (digital literacy)
- Lots of discussion about retraining older people to handle jobs in the future as their present jobs go away
- How much should government be leading vs getting out of the way and just funding it?
- Nigel – Government should provide direction. Different in tertiary and other sectors
- Karen – Collaborative and connected but not mandating
- “We need to prepare people not just for the jobs of the future, but also to create the companies of the future” – Martin Danner
- Lots of other stuff but I got distracted.
The schedule is out for Percona Live Europe: Amsterdam (September 21-23 2015), and you can see it at: https://www.percona.com/live/europe-amsterdam-2015/program.
From MariaDB Corporation/Foundation, we have 1 tutorial: Best Practices for MySQL High Availability – Colin Charles (MariaDB)
And 5 talks:
- Using Docker for Fast and Easy Testing of MariaDB and MaxScale – Andrea Tosatto (Colt Engine s.r.l.) (I expect Maria Luisa is giving this talk together – she’s a wonderful colleague from Italy)
- Databases in the Hosted Cloud Colin Charles (MariaDB)
- Database Encryption on MariaDB 10.1 Jan Lindström (MariaDB Corporation), Sergei Golubchik (Monty Program Ab)
- Meet MariaDB 10.1 Colin Charles (MariaDB), Monty Widenius (MariaDB Foundation)
- Anatomy of a Proxy Server: MaxScale Internals Ivan Zoratti (ScaleDB Inc.)
OK, Ivan is from ScaleDB now, but he was the SkySQL Ab ex-CTO, and one of the primary architects behind MaxScale! We may have more talks as there are some TBD holes to be filled up, but the current schedule looks pretty amazing already.
What are you waiting for, register now!
Ministerial address: Hon. Amy Adams, Minister for Communications
- Mentions she was at community group meeting where people were “shocked” when it was suggested that minutes be sent via email
- Talk up of the UFB rollout. Various stats about how it is going
- Also mentioned that Mobile build is part of UFB, better cellular connectivity in rural regions
- Notes that this will never be 100% complete. The bar keeps moving
- Very different takeup in different regions. 2% in some 19% in others. Local organisations pushing
- Good Internet is especially important for remote countries like New Zealand
- Talk about getting better access in common areas (eg shared driveways) for network builds
- Notes how Broadcasting and Communications as well as other areas are converging. Previously they were separate silos. Similar for other areas.
- Harmful Digital Communications Act.
- Says new framework, adjustment may be needed and bedding down the courts.
- Says that majority of cases will go to mediation
- Similar Act in Australia very few things going to courts
- Gave similar silly literal readings of others acts ( RMA requires a permit to sneeze )
- 5 “Questions” to minister. 2 on TPP, 1 on Captions, 1 pushing some project and one actual question that she got to answer.
- Maybe they should look at this idea for the Questions
Keynote: Kathy Brown, ISOC CEO
- GDP of a National is highly correlated with the growth of the Internet
- 75% of the benefit of the Internet goes to existing businesses
- ISOC Global Internet Report 2015
- Huge growth in Mobile Internet
- “94% of the global population is covered by mobile networks. Mobile broadband covers 48% of global population”
- Huge gap between developed and developing counties
- Report is Online and “Interactional”
- Openness of the Internet means information is out there, exposed and gettable by the wrong people sometimes
- Generational divide in attitude to privacy
- Privacy is a matter of personal choice. The tools should be available should you wish to use them
Govt 2.0: Digital by default
- Rachel Prosser and David Farrar facilitating.
- Room full
- Result 10 programme background
- NZ Government Web toolkit
- 50,000 registered with NZ Realme site
- Shared rules between local governments, problems with same rules everywhere. Some limitations,. Perhaps at least similar technical standards
- People don’t care about governments structure, they just want a service, don’t care how depts are arranged.
Building an access network for demand and scale – new challenges – Kurt Rogers, Chorus
- Over 1 million broadband connections on access network
- 70-80% of BB connections
- Average connection sped now near 20Mb/s due to VDSL and Fibre
- Busiest 15 minute period (around 9pm Thursday) of week averaging 0.5Mb/s per user ( up from 100kb/s just 3 years ago )
- Jump in mid-2013 when Netflix and Lightbox launched
- Average bandwidth per user growing 50%/year. Grown that much in 1st half of 2015
- Quite a few people still on ADSL1 modems when ADSL2 would work
- Same a lot of people can get VDSL that don’t realize
- Lots of people on 30Meg fibre plan at the start, now most going for 100Mb/s
- Rural broadband (RBI)
- 85k lines upgraded to FTTN
- Average speed jumped 5.6Mb/s to 15Mb/s after a single rural cabinet upgraded cause everybody could now use ADSL2 and faster uplink. One fibre guy got 48Mb/s on VDSL, other 37Mb/s
- More speed out there than some people realize
- VDSL bandplan moving from 997 to 998. Trail average speed increases were from 32 to 46Mb/s for downstream. Minimal change on upstream speed.
- Aggregation link bandwidth. Alert threshold at 70%, Max threshold at 90%
- Technology down the road to speed up aggregation links with Next Generation PON technology
The new smart ISP – Colin Brown, GM of Networks at Spark
- Working on caching infrastructure, bigger and closer to their edge
- Big traffic growth this year
- Big growth in mobile traffic especially upload
- 60% of phones in stores are 4G capable
- Providers investing a lot of money , profits lower. Less like banks, more like airlines
- Technology refresh every 5 years rather than every 10
This content is password protected. To view it please enter your password below:
Way back in 2010, MySQL Bug 57241 was filed, pointing out that the “swap insanity” problem was getting serious on x86 systems – with NUMA being more and more common back then.
The swapping problem is due to running out of memory on a NUMA node and having to swap things to other nodes (see Jeremy Cole‘s blog entry also from 2010 on the topic of swap insanity). This was back when 64GB and dual quad core CPUs was big – in the past five years big systems have gotten bigger.
Back then there were two things you could do to have your system be usable: 1) numa=off as kernel boot parameter (this likely has other implications though) and 2) “numactl –interleave all” in mysqld_safe script (I think MariaDB currently has this built in if you set an option but I don’t think MySQL does, otherwise perhaps the bug would have been closed).
Anyway, it’s now about 5 years since this bug was opened and even when there’s been a patch in the Twitter MySQL branch for a while (years?) and my Oracle Contributor Agreement signed patch attached to bug 72811 since May 2014 (over a year) we still haven’t seen any action.
My patch takes the approach of you want things allocated at server startup to be interleaved across nodes (e.g. buffer pool) while runtime allocations are probably per connection and are thus fine (in fact, better) to do node local allocations.
Without a patch like this, or without running mysqld with the right numactl incantation, you end up either having all your memory on one NUMA node (potentially not utilising full memory bandwidth of the hardware), or you end up with swap insanity, or you end up with some other not exactly what you’d expect situation.
While we could have MySQL be more NUMA aware and perhaps do a buffer pool instance per NUMA node or some such thing, it’s kind of disappointing that for dedicated database servers bought in the past 7+ years (according to one comment on one of the bugs) this crippling issue hasn’t been addressed upstream.
Just to make it even more annoying, on certain workloads you end up with a lot of mutex contention, which can end up meaning that binding MySQL to fewer NUMA nodes (memory and CPU) ends up increasing performance (cachelines don’t have as far to travel) – this is a different problem than swap insanity though, and one that is being addressed.
Introduction – Dean Pemberton, InternetNZ
Dean was going to do an intro but got cock-blocked by some guy in a High-Vis vest.
The People Factor: what users want – Paul Brislen, ex-CEO of TUANZ
- Working from home since 1999, 30kb/s at first. Made it work
- Currently has 10Mb/s shared with busy family, often congested, not using much TV yet
- Television driving demand.
- Some infrastructure showing the strain
- Southern cross replacement will be via Sydney. A couple of thousand km in the wrong direction when going to the US
- Rural broadband still to deliver on the promise, no uptake stats, not great service level
- Internet access critical path for economic development. lack of political will
- Dean got to do his intro talk now.
- Will Internet be priced on peak usage? A: Already offpeak discounts, some ISPs manage home/biz customer ratio to keep traffic balanced
- Average usage per customer is 5Mb/s for ISP with streaming orientated ISP (acct sold with device).
- 60% of International traffic going to Aus (to CDNS)
- Consumers don’t accept buffering, high quality video (bitrate and production quality). Want TV to just-work.
- NZ doesn’t want to be a “rural” level of internet access, equiv to a farm in more connected countries
- Could multicast work for live events like sport?
- Hard to get overage to work to work when people leave TV on all day
- Plenty of people in Auckland not getting UFB till 2017 (or later)
The connected home and the Internet of Things – Amber Craig, ANZ
- At top of Hype cycle
- Has home Switches on Wemo (have to get upgraded)
- Lots of devices generating a lot of data
- Video Blogging – 10GB of raw data, 1GB of finished for just 5 minutes. Uploading to shared drives, sending back and forth through multiple edits
- Network capacity if probably not much for IoT compared to video, but home will be a source of a lot more uploads
- With IPv6 maybe less NAT, harder to manage (since people are not used to it).
- Whose responsibility is it to ensure that Internet works in every room
- Building standards, what are customers, government, ISP each prepared to pay for?
- What about medical dependency people who need Internet. A lot of this goes over GSM since that is more “reliable”
Lightbox – content delivery in New Zealand – Kym Nyblock, Chief Executive of Lightbox
- Lightbox is part of Spark ventures, morepork, skinny, bigpipe
- Lighbox – On line TV service, $12.99/month thousands of hours of online content
- 40% of US household have SVOD, but pay-TV only down 25%
- Many providers around the world, multiple providers in many countries. Youtube also bit player in the corner
- SVOD have some impact on piracy, especially those who only pirate cause they want content same day as programme airs in the US
- Lots of screens now in the house, TV not only viewed on TVs
- Lightbox challenges
- Rights issues, lots of competition with other providers, some with fuzzy launch dates
- NZ Internet not too bad
- Had to work within an existing company
- Existing providers
- Sky – 850k homes, announced own product, has most sports
- Netflix – approx 30k homes, coming to NZ soon
- From Biz plan to launch in 12 months
- Marketing job to be very simple – “Grandma Rule” ( can be explained to Grandma, used by her)
- Express service delivers content right after views in the US. Lots of views for the episodes that are brand new. One new episode can be 10% of days total views
- Very agile company, plans changed a lot.
- Customers will have several providers and change often
- Multiple providers in the market, more to come
- Premium and exclusive content will drive, simple interface will keep it
- Rights issues are a problem but locked into the studio system
- Try to “grow the category”, majority on consumers still using linear, scheduled TV
- Try to address local rights ownership. This is the bit where they dug at US based providers and people using them.
- Working on a Sports offering
- and then she showed a Lightbox ad
- Question costs of other ISPs of getting good lightbox due to charges from Spark-Wholesale for bandwidth exchanged. Not really answered
Quickflix – another view of content delivery in New Zealand – Paddy Buckley, MD of Quickflix NZ
- 1st service to launch in March 2012
- Subscription service for movies and TV shows and Standalone pay-per-view service for new-release movies and some TV shows
- Across lots of devices, Smart TVs, phones, computers, games consoles, tablets, tivo, chromecast. No Linux Client
- Just 15% of views via the website now
- Content: New release movies, subscriptions content movies, TV shows
- Uses Akamai for delivery. Hosting Centers in Sydney and Perth. AWS/Azure
- Unwritten 5 second rule. Content should play within 5 seconds of pressing play
- The future
- Multiple Models, Not just SVOD, eg TVOD, AVOD, EVOD, EST
- More fibre, fast home wifi and better hardware
- VOD content getting nearer to the viewer. HbbTV combines broadcast and on-demand being done by freeview
- Android TV
- Viewing levels to increase (volume and frequency), people will pick and mix between providers
- Aiming at 50% of households, 1 million is quite a lots for any scale.
- Coming soon
- 1080p/4K , 5.1 surround sound
- Fewer device limits. All services and all devices
- More streams
- Changing release windows
- Live streaming
- PPV options to compliment
- Download now, view later
- What we need from ISPs
- Significant bandwidth
- Mooorrreee bandwidth
- People will change ISPs if the ISP can’t provide the level of service
- Netflix is naming and shaming. Netflix best/worst list
- Prediction that NZ could hit 50% SVOD within a couple of years
- Asked if they will be going broke in next few months. Says he’s done deal with Presto in Aus and will ease funding problems but business as normal in the NZ
- SVOD has evolved from back-catalog TV shows a few years ago to first-run now. Will probably keep going forward with individual shows being provider-exclusive for now, especially since services are fairly low cost per month
- A few questions about subtitles. Usually available (although can cost extra) but not good support with end devices to turn on/off .
The refereed talks are:
- CC3: An Identity Attested Linux Security Supervisor Architecture – Greg Wettstein, IDfusion
- SELinux in Android Lollipop and Android M – Stephen Smalley, NSA
- Linux Incident Response – Mike Scutt and Tim Stiller, Rapid7
- Assembling Secure OS Images – Elena Reshetova, Intel
- Linux and Mobile Device Encryption – Paul Lawrence and Mike Halcrow, Google
- Security Framework for Constraining Application Privileges – Lukasz Wojciechowski, Samsung
- IMA/EVM: Real Applications for Embedded Networking Systems – Petko Manolov, Konsulko Group, and Mark Baushke, Juniper Networks
- Ioctl Command Whitelisting in SELinux – Jeffrey Vander Stoep, Google
- IMA/EVM on Android Device – Dmitry Kasatkin, Huawei Technologies
There will be several discussion sessions:
- Core Infrastructure Initiative – Emily Ratliff, Linux Foundation
- Linux Security Module Stacking Next Steps – Casey Schaufler, Intel
- Discussion: Rethinking Audit – Paul Moore, Red Hat
Also featured are brief updates on kernel security subsystems, including SELinux, Smack, AppArmor, Integrity, Capabilities, and Seccomp.
The keynote speaker will be Konstantin Ryabitsev, sysadmin for kernel.org. Check out his Reddit AMA!
See the schedule for full details, and any updates.
This year’s summit will take place on the 20th and 21st of August, in Seattle, USA, as a LinuxCon co-located event. As such, all Linux Security Summit attendees must be registered for LinuxCon. Attendees are welcome to attend the Weds 19th August reception.
Hope to see you there!
The Internet is going encrypted. Revelations of mass-surveillance of Internet traffic has given the Internet community the motivation to roll out encrypted services – the biggest of which is undoubtedly HTTP.
The weak point, though, is SSL Certification Authorities. These are “trusted third parties” who are supposed to validate that a person requesting a certificate for a domain is authorised to have a certificate for that domain. It is no secret that these companies have failed to do the job entrusted to them, again, and again, and again. Oh, and another one.
However, at this point, doing away with CAs and finding some other mechanism isn’t feasible. There is no clear alternative, and the inertia in the current system is overwhelming, to the point where it would take a decade or more to migrate away from the CA-backed SSL certificate ecosystem, even if there was something that was widely acknowledged to be superior in every possible way.
This is where Certificate Transparency comes in. This protocol, which works as part of the existing CA ecosystem, requires CAs to publish every certificate they issue, in order for the certificate to be considered “valid” by browsers and other user agents. While it doesn’t guarantee to prevent misissuance, it does mean that a CA can’t cover up or try to minimise the impact of a breach or other screwup – their actions are fully public, for everyone to see.
Much of Certificate Transparency’s power, however, is diminished if nobody is looking at the certificates which are being published. That is why I have launched sslaware.com, a site for searching the database of logged certificates. At present, it is rather minimalist, however I intend on adding more features, such as real-time notifications (if a new cert for your domain or organisation is logged, you’ll get an e-mail about it), and more advanced searching capabilities.
If you care about the security of your website, you should check out SSL Aware and see what certificates have been issued for your site. You may be unpleasantly surprised.
Since I was creating large blocks (41662 transactions), I added a little code to time how long they take once received (on my laptop, which is only an i3).
The obvious place to look is CheckBlock: a simple 1MB block takes a consistent 10 milliseconds to validate, and an 8MB block took 79 to 80 milliseconds, which is nice and linear. (A 17MB block took 171 milliseconds).
Weirdly, that’s not the slow part: promoting the block to the best block (ActivateBestChain) takes 1.9-2.0 seconds for a 1MB block, and 15.3-15.7 seconds for an 8MB block. At least it’s scaling linearly, but it’s just slow.So, 16 Seconds Per 8MB Block?
I did some digging. Just invalidating and revalidating the 8MB block only took 1 second, so something about receiving a fresh block makes it worse. I spent a day or so wrestling with benchmarking…
Indeed, ConnectTip does the actual script evaluation: CheckBlock() only does a cursory examination of each transaction. I’m guessing bitcoin core is not smart enough to parallelize a chain of transactions like mine, hence the 2 seconds per MB. On normal transaction patterns even my laptop should be about 4 times faster than that (but I haven’t actually tested it yet!).So, 4 Seconds Per 8MB Block?
But things are going to get better: I hacked in the currently-disabled libsecp256k1, and the time for the 8MB ConnectTip dropped from 18.6 seconds to 6.5 seconds.So, 1.6 Seconds Per 8MB Block?
I re-enabled optimization after my benchmarking, and the result was 4.4 seconds; that’s libsecp256k1, and an 8MB block.Let’s Say 1.1 Seconds for an 8MB Block
This is with some assumptions about parallelism; and remember this is on my laptop which has a fairly low-end CPU. While you may not be able to run a competitive mining operation on a Raspberry Pi, you can pretty much ignore normal verification times in the blocksize debate.
 I turned on -debug=bench, which produced impenetrable and seemingly useless results in the log.
So I added a print with a sleep, so I could run perf. Then I disabled optimization, so I’d get understandable backtraces with perf. Then I rebuilt perf because Ubuntu’s perf doesn’t demangle C++ symbols, which is part of the kernel source package. (Are we having fun yet?). I even hacked up a small program to help run perf on just that part of bitcoind. Finally, after perf failed me (it doesn’t show 100% CPU, no idea why; I’d expect to see main in there somewhere…) I added stderr prints and ran strace on the thing to get timings.