Philadelphia Open Data Pipeline

Today we’re making public the list of open data sets that the Innovation Management Team at OIT is working to release to the public. This will give people a view into what we are working on, and when we hope that specific data sets will be published and listed in the Open Data Philly data portal.


This is a step we’ve been discussing internally for some time, and there is language in the Mayor’s Executive Order on open data that requires city departments to “develop a schedule for making information available to the public and updating it on a regular basis.” Our internal discussions on this issue have centered around the best way to publish such a schedule and the work processes we would use to support it.

Internally, we use Trello to manage our tasks around open data releases – it’s a great tool that is powerful, web-based (so we can access it from anywhere) and very easy to use. There are ways to export data from Trello, so one option we explored was the possibility of an export that would populate a web page or public document that could provide information on our open data pipeline. Ultimately, we decided that the easiest and most useful approach we could take would be to make the Trello board that contains our open data pipeline public.

View the public Trello board containing the City of Philadelphia’s open data pipeline here.

Note – you don’t need a Trello account to view this board, but if you have one you can take advantage of some of the additional features we may be rolling out in the future, like opening voting on specific cards and subscribing to receive notice of changes.

Several caveats when reviewing the information on this public Trello board.

First, our release schedule is organized into lists with a specific date targeted for the release of various data sets (represented by “cards” within that list). We’re using a monthly release cycle to guide our work to open up data sets. However, the fact that a data set is contained in a list for release on a specific date does not reflect a commitment from the underlying data producer to publish the data by that specific date. We’re using these dates to manage the work of our team, and to prioritize our interactions with the departments that maintain these data sets. Trello contains some nice features for reprioritizing tasks and moving things around, and we make ample use of them when needed. The specific release data for a data set may change, but we’re now providing visibility into this process.

Second, the items contained in the “Ideas / Suggestions” list (far left) are items that have been aggregated from a variety of sources – internal discussions with data producers, suggestions from external data users, nominations in the Open Data Philly site, entries in the Open Data Race, comments in the City’s open data forum, etc.

For a variety of reasons we can’t release every data set that is suggested as an open data release, so we need to use our best judgment about what our team will be able to accomplish – for example, some data sets are created and maintained by entities not covered under the Mayor’s Executive Order. We may include these suggestions in our pipeline to call attention to them (particularly if they are high value data sets), and to inform discussions with the data producers through forums like our Open Data Working Group. Ultimately, what we propagate from the “Ideas / Suggestions” list to one of our release lists reflects those data sets that we believe merit public release as an open data set, and those that we’ve had substantive discussions about with the agency or department that produces it.

Our hope is that by making this Open Data Pipeline public that we’ll give people better insight into what we’re working on, and how soon specific data sets could be released to the public. As with all of our work products that we release, public comments are welcomed.

Please feel free to post a comment below, or post a message to the City’s open data forum. You can also send comments or suggestions to

3 thoughts on “Philadelphia Open Data Pipeline

  1. Pingback: How Unique is the New U.S. Open Data Policy? - Sunlight Foundation Blog

  2. Pingback: Updates to Philly Open Data Pipeline | PhillyInnovates

  3. Pingback: Data Innovation Day 2014 | PhillyInnovates

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s