Building an open data infrastructure

Click the image for an interactive timeline of data releases in 2014

2014 was a great year for open data in Philadelphia – we saw departments release over 30 datasets, including big ones like Lobbyist Activity, Contracts for Professional Services, and Commercial Building Energy Usage, along with the publication of the Open Data Strategic Plan. But if you look at the release dates in our Open Data Census, you’ll see that we’ve gone over thirty days without any departments releasing a dataset! We thought an update was in order.

In the Strategic Plan, we laid out a vision to scale open data and make it “part of the way we do business as a government.” Rather than approaching data releases on a case-by-case basis as we would do as part of an initiative, we want departments to be able to look at the “full picture” of datasets and prioritize based on measured public demand, and we want a system in place to ensure data releases are consistent, responsible, and automated. This amounts to building an “open data infrastructure,” and it’s a bit different than what we’re used to working on. So far, we’ve:

Met with 43 department heads and Deputy Mayors to review the Open Data Strategic Plan

Step one of the process outlined in the Strategic Plan is “Meet with each department.” We’ve hit the ground running and learned a great deal about the people we’ll be working with and some of the data challenges they have in their operations. Most importantly, the openness and enthusiasm we received gave us a great sense of optimism.

Kicked off data inventories with 12 departments

Step two of the process involves putting together a list of all the datasets at each department. As you can imagine, this is a huge endeavor, but a critical one. It allows us, the department, and the public to see the full picture of what data exists, and determine up-front which datasets can be shared as-is, which ones need sensitive data removed, and which ones can never be shared. Most importantly, it lets us prioritize. Once these priorities are established, it’s just a matter of executing. This means more releases at a faster rate.

Of those 12, some are just getting started (like the Streets Department) and others are inches away from completion (like the Commerce Department). Follow along on our Open Data Census in the “Current Pipeline” section.

Built a Data Services team

There’s a lot of work ahead on open data. Fortunately we have friends in the Civic Technology, Application Services, and GIS teams to lend a hand, but we’ve also built a team specifically focused on implementing the Open Data Strategic Plan, including:

  • Stacey Mosley, Data Services Manager
  • Lauren Ancona, Data Scientist
  • Jessica Magness, Data Science Intern

We’re pretty serious about this stuff 🙂

Acquired an open data automation tool

In the “What we’ve learned” section of the Strategic Plan, we highlighted the importance of automation. Where avoidable, open data shouldn’t be a burden, and departments shouldn’t have to manually refresh their data every month. We’ve selected a tool (Safe FME) to automatically extract data, transform it to a shareable state, and publish it to various destinations. More importantly, we’ve been learning how to use it! In addition to the GIS Services Group, Andrey Mun, our Senior Software Engineer, has been training his team in preparation to scale our automation efforts.

Formed an Open Data Advisory Group

In June, we began to convene a small group of “open data stakeholders” that represent diverse communities of data users. These include academia, technology, business, non-profit, journalism, and more to help us gauge and understand public demand in a more holistic way. As we near completion of the first few inventories, we’ll discuss them with the group and put step three of the Strategic Plan to the test.

Kicked off a new redesign of phila.gov that will help us reach the general public

Step five of the Strategic Plan is about open data reaching people through a compelling “digital front door,” helping citizens connect to their government through more than just CSV files. Last month we kicked off alpha.phila.gov, a project to create a user-centered redesign of the City’s website. Check out the blog posts by Aaron Ogle and Kyle Odum, who are managing the project, to learn more.

There’s still a lot left to do to realize an open data infrastructure, but we’ve got our plan in front of us and we’re excited by the momentum we’ve been able to build in just a few months. Follow along at the Open Data Census, join the conversation on the Open Data Forum, and stay tuned for more progress updates!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s