The DBCDE Pilot Report

In mid December last year, the Department of Broadband, Communications and the Digital Economy released the results of their live pilot of the proposed internet filter on their website.

 

Summary of Results

At face value, the report suggests that the pilot trial was a universal success.

I’m certainly not a systems expert – I don’t know the ins and outs of how to maintain or operate an ISP, or how to implement such a large-scale IT project.

That said, I am concerned at the language used in each ‘objectives’ sections of the report. Basic academic logic states that you need to clearly define your criteria for success before addressing your results and drawing a conclusion.

The objectives outlined in each section do not clearly state any criteria for success. The language is vague and does not provide any hard figures for comparison.

Indeed, the whole report - and Communications Minister Stephen Conroy’s comments – all speak in very vague terms about the success of the filter. The statistics they offer are suspiciously perfect and particularly general.

For example, in the press release published to announce the release of the pilot report, Senator Conroy states that “…blocking RC-rated material can be done with 100% accuracy and negligible impact on internet speed.”

These kind of sweeping statements are troubling for such a technically-complicated issue such as the filter.

A deeper investigation into the report reveals a couple of troubling issues.

‘Accuracy Testing’

This was detailed in three sections – blocking the ACMA blacklist, blocking additional content based on user request, and blocking of innocuous content (overblocking).

The first section reported the before mentioned 100% accuracy of all sites on the ACMA blacklist. This is not surprising, as blocking sites with known URLs and IP addresses is a task simple enough for any systems administrator with access to the appropriate DNS and network controls.

The second approach provides less favourable figures. Blocking of additional content – the key function of the government’s plan to ‘protect children’ – registered a success rate of between 78.80% and 84.65%. Given the sheer number of websites available to anyone with an internet connection, this would leave tens of millions of websites still available with ‘objectionable content’.

Similarly, the third section provides unfavourable figures, stating that the filters blocked ‘less than 3.37% of innocuous content’. Again, when scaled to the sheer volume of websites available on the internet, these seemingly small percentages become quite significant.

The first set of results might seem impressive, but it fails to take into account the dynamic nature of the internet. Content can be moved and created at such rapid rates that any ‘blacklist’ that exists would be outdated within days, if not hours, making the 100% claim a little redundant.

‘Network Performance Testing’

Again, this section of the report provides figures favouring the filter, with no serious reports of performance degradation across the trial ISPs. Taken on surface level, this seems to be a clear cut case against the naysayers.

What’s not mentioned in the report however is the full technical scope of the trials. The filter was only tested on DSL services up to 8Mbit – not on the now standard 24Mbit connections that most DSL subscribers have access to. Whilst not mentioned in the reports, the trial ISPs were, with the exception of Optus, small operators with similarly small customer bases.

The issue therefore is that the trials were not a true representation of the kind of network loads the filters would be subjected to if installed across all major ISPs. Additionally, the trial period – a minimum of 6 weeks – suggests that it may not have collected enough data to examine how the filter would perform over a sustained period of time, with a constantly changing blacklist and the emergence of new methods of circumvention. This brings me to my next point.

‘Circumvention Testing’

One of the fundamental flaws in the scheme is how easily the filter could potentially be bypassed by those with a small amount of technical knowledge. The report does little to combat this prediction. On the service filtering just the ACMA blacklist, the participating ISPs reported a success rate in blocking circumvention of 8% to 16%.

The second circumvention test was against both the blacklist and additional blocked content. This test appears to prove more successful, with results ranging from 37.8% of attempts blocked, through to 94.5%, but clearly far more unpredictable.

Unfortunately the report does not mention the circumvention techniques used for testing ‘for public interest reasons’, but without knowledge of what the techniques were, the percentages above become somewhat irrelevant. Without knowing this, the public is completely in the dark about how effective it will be at blocking circumvention attempts.

Summary & Final Thoughts

We’ve only begun scratching the surface of the report, and already there’s cause for concern. It seems like the report is designed to paint a favourable picture of the filter, providing detail where convenient and leaving it out where it’s not.

This appears to a common occurrence when it comes to the official government stance on the filter. Whether it’s through reports, press releases or conversations with the press, one gets the feeling that there’s plenty that’s not being said.

At the end of the day though, that’s politics. We might not like it, but we don’t have the power to change it.

Right… ?