As we set out to build a more effective pipeline for publishing open data, we also had a very big question to answer: What data does the City collect in the first place? Sure, there are data sets that cities across the country have published, and others that are frequently the topic of journalistic inquiry or right to know requests. But there’s a ton of information collected that people inside and outside of the organization just flat out don’t realize exists.
So we began working with departments to get a better understanding of the data they collect. Instead of diving immediately into technical detail, we began asking existential questions like, “Why does your unit do what it does?” followed by, “What information do you collect in the process?” Individual data sets were identified and additional contextual information was gathered about each one.
The result is what we are calling the Data Inventory, a new resource for public and internal use. It is a work in progress – we’ve kicked things off with twelve departments so far, and today marks completion of the first four, including the:
- Commerce Department
- Board of Ethics
- City Commissioners Office
- Office of Emergency Management
Take a look and you’ll find a list of all 75 data sets housed by these four departments, with not only names and descriptions of their contents, but information about the quality of the data, current formats, and who you can contact at the agency of origin for more information. Some of the data sets are already available to the public while others, like Critical Infrastructure, are highly classified and cannot be released.
The other reason for creating the Data Inventory is to get feedback from you(!), the public, as to which data sets you are interested in and why they should be prioritized for release above the others. To do so, first go to the Data Inventory and check out the list of data sets. Find something of particular interest? Click “view” to the right of the data set, scroll to the feedback form on the bottom of the screen, and tell us why that data set is of interest to you. We’ll take your input back to each department for them to make informed prioritization decisions on which data sets to release first.
With eight more agencies currently taking an inventory, and many more to come, we are just beginning to scratch the surface of the data that is collected by the City. There is lots more on the way, so check back on a regular basis for new releases and to provide input along the way. We look forward to hearing from you!