The coming open data battle – government versus commercial interests

I’m a big fan of opening up as much public sector information as possible in easily discoverable and reusable ways (taking into account privacy, security and commercial-in-confidence considerations).

The data allows citizens and organisations to build a more informed view of their government’s activities, a good accountability measure.

It also allows the development of useful applications and services at low cost and even lower (frequently free) prices. Sure they may not be as polished as multi-million dollar services developed by governments or big business, however they allow citizens to choose the tools that work best for them. Government or big business can always use these learnings to build on.

Open data also allows government agencies to see what data other agencies have, and lets them use it to improve their models, understanding and policy. While often overlooked in the rush to provide data to citizens, often agencies have as much trouble discovering and accessing data from other agencies as citizens do.

However as more public sector data gets released, losers are also emerging, some with deep pockets and effective lobbyists.

Who loses when government data is released for free? Several groups spring to mind.

First are companies that make their living from licensing public information and selling it on (often with value-adds) at a mark-up. These companies allow agencies to extract a market price for their data without having to contend with the complexities of the open market. They often have a monopoly position, controlling access to a source of public data, and can be very resistant to losing their monopoly or seeing the data ‘devalued’ through free release.

Second are companies that rely on getting data first to build their edge. This includes stock market traders, where having information a few hours earlier than the market may be worth millions. It can also include the media, who thrive on ‘exclusives’. Where data is released to specific journalists under Freedom of Information or through other channels ahead of others they have an informational edge over their rivals.

Next are organisations who prefer to obscure the true cost of goods and services in favour of complexity. Where customers can’t compare prices effectively they can’t make the best price decision, therefore they may choose expensive services based on brand and never realise they are paying more than they should. Sound like any industry you know?

Finally there’s groups within government who prefer to keep citizens at arms length. Those who do not want too much scrutiny of their decisions or who believe the public won’t understand the broad context under which they were made. This group believes in only telling the public what they think the public needs to know.

We’re starting to see some of these groups flex their muscles in jurisdictions that are releasing a great deal of public sector information, or who are legislating for organisations to become more transparent.

One group currently resisting openness in the US are airlines. In the New York Times article, This Data Isn’t Dull. It Improves Lives, the journalist reports that,

…the Department of Transportation is considering a new rule requiring airlines to make all of their prices public and immediately available online. The postings would include both ticket prices and the fees for “extras” like baggage, movies, food and beverages. The data would then be accessible to travel Web sites, and thus to all shoppers.

The airlines would retain the right to decide how and where to sell their products and services. …

The approach would make markets more transparent and efficient – allowing consumers to make a decision on flights based on complete knowledge.

So do airlines support this approach? Well, not completely. They wish the right to choose when and how they display their fees – choosing to control the flow of information and force consumers to continue to make sub-optimal decisions on partial information.

This reflects the situation in Australia with the Rudd Government’s attempt to launch Fuelwatch and GroceryWatch websites. Petrol and grocery companies weren’t particularly supportive of having the true cost of their products visible to consumers before they were at the service station or in the store. Once consumers were there it was far less likely they’d leave and shop elsewhere because of price. Of course the reason given was the complexity of exposing the prices publicly, although they don’t seem to have this issue at the checkout.

Another example I have been watching is in Canada, where there’s been an active discussion of the decision of BC Ferries to release FOI requests online at the same time they are released to the requester (where the request doesn’t involve personal information).

Journalists have complained that the approach means they won’t get an exclusive, removing their financial incentive for requesting government information in the first place. One journalist in particular, Chad Skelton, has written a series of pieces detailing why it is so important that governments allow media to profit off FOI requests, as otherwise they are unlikely to ask for this information and it won’t be exposed for the public good. One of his articles worth reading is Why David Eaves is wrong about BC Ferries’ Freedom of Info policies.

It is an interesting point, however I tend to sympathise with David’s view – government information laws should not be designed to support the financial goals of media outlets, or any other organisations, over the goals of public openness and transparency. These laws should be designed to ensure that public information gains public scrutiny, not so that journalists can ‘make’ their careers with exclusives.

As we see more public sector information released by governments I expect we’ll see more battles over its release. Some forms of opposition will be passive, providing information in the least usable formats possible or hidden away in websites; other forms will be active, direct refusals to release information (because it is incomplete, the context wouldn’t be understood, or it isn’t useful), court cases from commercial interests asking for information to be suppressed, or even active information sabotage where data is destroyed rather than published.

Reputations and fortunes can be made and lost over access to information. It is unlikely that entrenched interests will support changes to the playing field without putting up an ongoing fight.