As you may know, we’re building a platform for the delivery of city government services, available for public feedback right now at alpha.phila.gov. We’re starting small but thinking big, so we’re paying some attention now to devops and our overall deploy process.
In planning for the development and deployment of alpha.phila.gov I set a couple of requirements:
- Automatic deployment via push to GitHub.
- Infrastructure as code, also committed to GitHub.
These give us continuous delivery on a stack that can be easily modified and reproduced across production, staging, and testing environments.
The city’s Office of Innovation and Technology (OIT) has been steadily moving services to Amazon Web Services (AWS), so I explored our options there. It’s more flexible in development stages than running machines internally, and provides a clearer path for scaling up as we grow into production.
At this point in the evolution of AWS, the users of Amazon’s services suffer from a paradox of choice. Among the deployment management options, we evaluated Elastic Beanstalk, CodeDeploy, OpsWorks, and rolling our own with straight EC2. For the reasons stated below, we eventually decided on OpsWorks, though we used CodeDeploy for a month while I set up our cookbooks.
While I know my way around a Linux box, I had never written a Chef recipe before, so it was tempting to stick with CodeDeploy and simply script the server setup with Bash. With OpsWorks, however, we have a more conventional way to set up configuration as code that can be committed, reviewed, and rolled back when necessary (see those recipes at github.com/CityOfPhiladelphia/phila.gov-cookbooks).
The UI for OpsWorks is also a cut above the rest of AWS, with very pretty overview layouts to gain an understanding of the whole stack at a glance. It really is a great feeling to add an instance to a layer in the stack and know that within a few minutes a new machine will be configured behind the load balancer, connected to the database, accepting requests and returning responses.
The local development environment
To set up a local development environment, all one has to do is clone the repo at github.com/CityOfPhiladelphia/phila.gov and run “vagrant up” in that directory. That process relies on Vagrant, a great tool for development.
Our Vagrant setup uses a simple bash script to approximate our production setup. I would like to use the Chef cookbooks for Vagrant as well, but the specifics for AWS (relying on deploy variables and connecting to the database at RDS) have delayed that across-the-board consistency. In addition, enough of our team has issues with Vagrant and VirtualBox on their machines (Windows and old MacBooks) that I’ve been investigating another route: EC2 instances in an OpsWorks testing stack belonging to each developer. This involves a slightly more complicated development setup (maybe using sshfs to edit those remote files), but gives us parallel environments between production, staging, and testing that are easy to keep in sync.
WordPress and Composer
We have separate repos at GitHub for our custom WordPress theme and plugin for alpha.phila.gov. In development, those repos are checked out within the main phila.gov repo and commits are pushed from there. When a developer on our team (usually the awesome Karissa Demi) is ready to test new code on staging, the following steps are taken:
- Push a new release for the theme or plugin
- Switch to a “clean” checkout of the phila.gov repo
- Run “composer update” to pull in the latest versions
- Commit the new composer.lock file into the staging branch on the phila.gov repo
The third and fourth steps above are made possible by Composer, a version-locking dependency manager for PHP. With Composer we can define dependencies in the repo, enabling us to manage our WordPress stack with committed code.
Getting WordPress to work with Composer involves modifications to the wp_config, such as overriding the WP_CONTENT_DIR. A number of other configuration values are set via environment variables, which allows us to use this public repository in our deploy environments. The update command also pulls in the latest releases of all of our third-party plugins, and can even update WordPress itself.
Once that composer.lock file has been committed, a push to the staging branch on GitHub will make any changes live on our staging stack at OpsWorks. This is managed by Travis CI, which sees any push to GitHub and tells OpsWorks to run a deployment. See our .travis.yml for configuration details.
Once the team has reviewed staging and decided the changes there should land in production, we create a pull request at GitHub between the staging and master branches. After that merge is complete, the same process involving Travis CI and OpsWorks handles the production deployment.
I’ve been pleasantly surprised by OpsWorks. Little details make all the difference. For example, when I ssh into a testing box to land a hotfix or connect to the DB, the MOTD includes all of the machine information, so I know I’m in the right place before I start hacking away. Another nicety is that it’s easy to copy stacks in OpsWorks, which I’ve done to create our parallel environments. Also, while I took some time to climb up the Chef learning curve, our cookbooks ended up containing less than a hundred lines of code. I managed this on two assumptions:
- They rely on the built-in recipes at OpsWorks to do much of the basic machine setup, such as user accounts and database connections, and
- All of our machines are Ubuntu 14.04. Package names and configuration file locations are therefore specific to Ubuntu.
There are a few caveats, however. Those new instances that are so easy to spin up without any manual intervention take an absurdly long time to do so. Deploys are also slow due to all of the back and forth between GitHub, Travis CI, and OpsWorks. Finally, I fear that even though it relies on industry standards like Chef, OpsWorks encourages vendor lock-in because we get used to its way of doing things. That’s what a good hosted product should do, though.
We have yet to test the assumed scalability of this setup, and we’re just starting to build out our resource monitoring, but we’ve already reaped some benefits. One example is the recent launch of alpha.phila.gov/property. A small change to the nginx configuration in our cookbook set up the proxy for GitHub Pages. That change was deployed to staging and reviewed there. Once we were fairly confident it worked, we updated the cookbooks in production, and the service went live without a hitch.
You wouldn’t want Philly Tech Week to interfere with your weekly quizzo. But what if it didn’t have to?
The Office of Innovation and Technology has partnered with the Philadelphia School District to bring “Open Data Query Quizzo” to Philly Tech Week, an interactive dive into the school district’s open data. Quizzo participants will break up into teams to answer three rounds of questions, each round relating to a unique data set from the school district. In conjunction with the event, the Philadelphia School District has released all expenditure data, expanded to reflect all of the district’s expenditures, including those under $50,000.
Query Quizzo will be held at National Mechanics (22 3rd Street in Old City) this Thursday, April 23 from 4:00-6:00pm. The event is “bring your own device” and wifi will be available. There will be prizes awarded to the team with the most correct answers. We hope to see you there!
Well we know where we’re going
But we don’t know where we’ve been
And we know what we’re knowing
But we can’t say what we’ve seen
The opening lines of Road to Nowhere by the Talking Heads perfectly describe the process of making changes to web properties without determining how well the current site is working for its intended audience.
With the ongoing, iterative development of alpha.phila.gov, it’s become more crucial than ever that we evaluate the way the current phila.gov is being used — and perhaps more importantly — where we could improve. In addition to offering a feedback form on each page of the site and conducting live testing with citizens, we want to find out what users aren’t telling us.
To that end, today we launch analytics.phila.gov: a public, near-real-time dashboard representing traffic to the City of Philadelphia’s web properties. The application is based on an open source project developed by 18F in conjunction with the Digital Analytics Program, both housed inside of the General Services Administration, an independent federal agency.
In December, we began the process of implementing a unified digital analytics program for all city websites. This meant making sure each page of every site includes a snippet of code to send data about its web traffic to Google Analytics. This is the first attempt to capture a “big picture” website report card in Philadelphia, and it’s still a rough draft. There are currently almost 30 offices or agencies included in the reporting, with around 25 still to add, as well as several non-phila.gov domains.
We will continue to add more of the city’s web properties to this reporting in the coming months. As the alpha.phila.gov team continues to build out both content pages and redesign digital services, we’ll also add new metrics to these reports. Just as we develop in the open, so shall we measure and publicly report on the results of our efforts. This means we’ll need to look at benchmarking, create funnels and goals relevant to city services, and add custom events to fill in the gaps in measurement. But first, we’ll look at what users are already telling us – via their actions.
This data is publicly available for download, refreshed at the same intervals as the dashboard.
In the coming weeks, we’ll detail the specifics of our implementation and include instructions for deploying a similar reporting instance.
You hear it all the time, “It’s the little things” that make all the difference. Whether it’s in our personal or professional lives, our attention to detail is often the differentiator between mediocre and magnificent. I learned this lesson from a former group president of a great Fortune 500 company. His name is John Babiarz and, at the time, he was tasked with leading a billion dollar managed services company in the health care space. John’s company offered services in broad areas including: food and facilities management, maintenance, call centers, security and clinical technology equipment services in thousands of sites across the country. Despite that scope, as its leader, he always made time to travel and connect with his clients and employees. On one such trip, I learned a valuable lesson.
As a member of the senior leadership team, it was not uncommon for me to travel with John to meet with some of our largest clients. On this day, we traveled to Texas for a negotiation and site visit with one of the largest and most prestigious hospital systems in the country. After the formal meeting, we embarked on a tour of the impressive facility. As you can imagine, the staff worked overtime to make sure the hospital was spotless as we toured. John graciously met and acknowledged the hard work of the staff as we visited. And then it happened. He stopped talking, squinted his eyes and starred into the distance down the hall. I turned and looked, we all turned and looked, I didn’t see anything. Without a word, John slowly made his way all the way down the long hallway. He bent over and picked up what appeared to be a small cigarette butt and, without saying anything, quietly deposited it in a trash bin. You could see that what he did had an impact on the staff. The president of this big company was not above stopping and going out of his way to pick up the smallest piece of trash for his customer.
When I later asked John about this, he went into long detail about how critical cleanliness is in the hospital setting. How it often drives prestigious hospital rankings. How that is often the way infectious disease is spread and how in a hospital keeping the environment pristine is an important life and death issue that can significantly impact public health. I got it. I never forgot that lesson. It’s the little things.
Great leaders think BIG and act SMALL. Finding the right balance between “digging in” on details (and risk getting caught up in the minutia) and the big thinking strategic leadership your direct reports require from you is always a challenge. You can’t do just one or the other. The best leaders strike a careful balance of both. You can’t set a powerful and aspirational vision and not occasionally examine and confirm with specificity whether it’s being implemented and whether you are making true progress. You can’t set a bold road map and personally stray off the course. Your everyday actions must be consistent with the broader vision you have set for the organization. Great leaders find the delicate balance.
LEADERS THINK BIG
Leadership 101 requires a leader to set a clear vision, mission and core values for the organization. The leader must also work very hard to personally exemplify those values. Far too often leaders fall short of the standards they have set for the organization. We have all known that leader: “do as I say, not as I do.” That is not authentic and just doesn’t work. The effort may start off well enough but sooner or later (and usually when you’re faced with real challenges) the failure to truly embrace and reflect your vision shows up. If you are not meeting your goals as an organization, first look at whether your leaders (across the board) reflect the type of success you expect. If they do, ask yourself whether they are effectively driving the blueprint for success through the organization. That is something that requires everyday small actions and constant communication to achieve. (See my “Leadership,-Brick-by-Brick” post)
GREAT LEADERS ACT SMALL
Great leaders do the small things to live their values everyday. After they’ve set the broader vision, they ensure others in the organization are delivering with behaviors consistent with that vision on the front line. They personally live those values by demonstrating their personal commitment to the vision and the values of the organization. They dig in with probing questions, receive frequent briefings and are a real physical presence in the field to see outcomes and results for themselves. In doing so, they communicate a high level of commitment and engagement. I’m not talking about grand gestures but small behaviors and interactions that make all the difference.
Today, I frequently travel between my building and City Hall, almost every day, to meet with the Mayor and Chief of Staff. Because of that lesson, as I make the short walk across the street, I try to pick up just one small piece of litter every day. I do that not just for the symbolic gesture, but as a reminder that if we all do small acts, we can drive big change in our City. The big vision is we all need to be working to clean up our City. The small act is the one thing I try to do everyday that sends a broader message. Whether you are a City Manager, Deputy Mayor, a company president or one of our everyday heroes, if we all think big and act small, we can make our worlds a better place. How do you think big and act small?
Rich Negrin is the City of Philadelphia’s Managing Director and Deputy Mayor for Administration and Coordination. Service Centered Leadership is the Managing Director’s blog series appearing on PhillyInnovates. Follow Rich on Twitter @RichNegrin.
The goal in redesigning phila.gov is to build a municipal website that starts with the needs of people who use it. As we dug into our web analytics, it became clear that many Philadelphians come to phila.gov for information about real estate.
Bits and pieces of property information are currently spread about, so one has to visit several sites to get the full picture. Ideally, all of that information would be in one place, with the data organized and prioritized to best serve user needs. The search at property.phila.gov goes some distance toward that ideal, but we wanted to go further.
How do we go about determining data priority? We added a survey to the current search to find out what people are looking for, then scoured over 400 responses. One insight gained was a top five list:
With that new understanding we designed a property app that emphasizes the real estate information people most care about. You can see it in action at alpha.phila.gov/property.
The technology is fairly straightforward. The Property app is entirely client-side, fetching data via open APIs from the Office of Property Assessment and the Geographic Services Group. It will be easily extended to include other sources like Licenses and Inspections. The code (and issues we’re working on) can be found at github.com/CityOfPhiladelphia/property2.
In a trend we’re hoping will become the routine, Property is now public as alpha software. We’re testing both our ideas and the technology to see if this works for all of you. Please check it out and leave us some feedback.
Last week, StateScoop announced the nominees of the 2015 StateScoop 50 Awards, honoring the best and the brightest who make state government more efficient and effective. The City of Philadelphia was represented in two categories.
Chief Innovation Officer Adel Ebeid was nominated as a “State Executive of the Year.” (Last year, Adel received a “State Leadership Award”) Here’s what StateScoop had to say about this year’s nomination:
The new CIO, Adel Ebeid, came in 2011 with a bold vision to change the culture from “building IT” to “managing IT” while inspiring staff to embrace the role of a “tech broker” as opposed to “tech providers.” With a strong push for more cloud/mobile/social/data, the team was able to adjust its tech portfolio so that 30 to 50 percent of the tech footprint resides in the cloud, leaving behind things that the organization must be accountable for.
The City’s efforts in municipal IT (specifically increasing access to technology and democratizing data) were also nominated as an “Innovation of the Year.” Here’s what StateScoop had to say:
Philadelphia’s effort to make data publicly available and accessible, coupled with equally strong work providing the city’s underserved communities with access to technology and the Internet, has yielded outcomes of greater benefit than if either open data or digital inclusion were pursued individually. Integrating these two frequently compartmentalized initiatives so that they interact with and complement one another provides Philadelphia’s communities with a unique experience in civic technology. Our outcomes express themselves in civic applications that allow residents to engage in personally meaningful ways with their neighborhood and government.
The StateScoop50 Awards are represented by the most innovative minds and initiatives in city and state government. Last year, the City of Philadelphia was nominated, and received awards in, three categories. In addition to Adel’s State Leadership Award, the City’s Academy of Municipal Innovation and Innovation Management Program Manager, Ashley Del Bianco both won StateScoop Awards. Click here to vote for Adel and the City’s municipal IT efforts in this year’s awards. Voting is open until April, 17th.
The City has long discussed ways to improve its contracting and procurement processes and broaden its vendor pool. These efforts were part of a national conversation on how to deploy technology and other updated approaches to improve the way government buys goods and services. Recently, the Office of Innovation and Technology dedicated a resource towards this initiative by hiring a “procurement advocate.” The new position helps small and mid-sized firms navigate municipal processes and aims to make it easier to do business with the City. It also supports more experimental and pilot-style contracting. Todd Baylson, who has worked for both city government and a small technology start-up, was hired as the procurement advocate in late 2014.
Internally, Todd’s work is about engagement, reaching out to relevant stakeholders to help shape future processes. The City’s cross-departmental procurement working group has convened and the procurement advocate is its first dedicated resource to help move forward on shared objectives and aspirations. A number of departments and agencies within city government deal with contracting, procurement, or vendors: the Department of Finance, Procurement, Law, Commerce, the Chief Integrity Officer, the Mayor’s Office of New Urban Mechanics, and the Office of Innovation and Technology. The working group aims to leverage the expertise of these agencies, examine best practices around the country, and craft an agenda to improve the way the City buys goods and services.
Externally, Todd’s efforts are focused on raising awareness around contracting and procurement opportunities for vendors, specifically technology firms. Municipal processes can be mysterious—daunting even—to the average Philadelphian or business owner. Firms looking to do business with the City have a partner in the procurement advocate, someone to make potential opportunities easier to find, someone who is accessible for questions about the process, someone who is seeking more bids and responses to the City’s opportunities.
There’s a natural crossover between these external efforts and the work of the Department of Commerce, which offers similar services to businesses looking to operate within the city and works to improve public policy in support of small business and economic development. Most recently, the Department of Commerce’s Archna Sahay, Manager of Entrepreneurial Investments, and Todd collaborated in the creation of the City’s first “Entrepreneur Office Hours.” The monthly initiative will offer open drop-in hours to entrepreneurs looking to either open a business in the city or understand and apply for City contracts (or both). Archna and Todd will host Entrepreneur Office Hours in coworking spaces across Philadelphia. Here’s the schedule for the next four months if you’d like to drop-in:
April 20th: Pipeline
May 20th: Impact Hub
June 20th: Benjamin’s Desk
July 20th: Turnkey
(Check-out @PHLGovContracts for specific hours)
By engaging internal and external stakeholders and helping to increase awareness, contracting and procurement opportunities have a chance to become more efficient and attract a larger pool of vendors. A larger pool of vendors ultimately means better buying for the City.
What do you think of the procurement advocate position?
Today, the Office of the City Commissioners released six data sets. The new data sets could be used to improve the Election Day experience and promote civic engagement in Philadelphia.
The released election data includes:
- Elected Committee People (Elected Committee People by ward, division, and party)
- Election Board Officials who worked on Election Day
- Polling Places API (Polling places, organized by ward/division, accessibility rating, or type of building)
- Unofficial Candidates (Names, addresses, parties, pursued positions, and unofficial signature totals for individuals running for office)
- Qualified Voter Registry (Registered voters and election participation aggregated by voter attributes, election, and district)
- Voter Turnout (Number of individuals by party affiliation that turned out to vote in each precinct for the General Election in 2014.)
The data was released in conjunction with the Apps for Philly Democracy Hackathon, a Code for Philly event that will bring together technologists, community organizers, and civic-engagement enthusiasts to prototype apps relating to democracy and/or civic-engagement.
If you have an idea for how the democratic experience or civic-engagement could be improved through technology, consider participating in the Apps for Philly Democracy Hackathon. The hackathon is a great event for the engaged or curious Philadelphian, even for those without technical skills. The event’s opening reception will include a brainstorm session on what to build or how to utilize data. Hackathon participants with advanced skills will likely lead in technical development but team members of every skill level can assist in each project’s research, design, marketing, and eventual launch. The opening reception will take place on Friday, March 27th in the City Hall Caucus Room from 5pm-7:30pm. The hackathon will take place on Saturday and Sunday in the City’s Innovation Lab on the 16th floor of the Municipal Services Building. City Commissioner Al Schmidt will be one of the judges for the hackathon’s final presentation at 2:00pm on Sunday.
“My office is grateful for the opportunity to be a part of this important event. The Data Inventory Initiative is consistent with my commitment to improve the transparency and accountability of government by making election information available to the public. I look forward to seeing all of the great uses of these data sets by the participants in the Apps for Philly Democracy Hackathon. Together we can make election data accessible to everyone who wants to participate in the democratic process,” said Commissioner Schmidt.
Today’s data release has obvious external benefits but it is an important internal accomplishment as well. The election data is the first in a batch of upcoming releases that have gone through the City’s new open data process. The open data team worked with the Office of City Commissioner Al Schmidt to first create a data inventory (a comprehensive list of all datasets owned by a department, detailing its accuracy and sensitivity). Unpublished datasets then received feedback from both the public (via the City’s data inventory website) and the City’s data advisory group (representing diverse communities of public data users), to gauge which datasets were the most valuable. Next, datasets were prioritized by the department and entered into the open data team’s pipeline of work to scrub data, build automated processes for future updates, and release datasets into a central data store for public use. These releases are an indication that the City’s open data process is working and could be a repeatable way of doing business for Philadelphia city government.
Today marks a big win for transparency in Philadelphia City government. Through a collaborative effort between the Procurement Department, the Chief Integrity Officer, and the Office of Innovation and Technology, the City of Philadelphia released data on “Commodities Contracts” (or contracts for supplies, equipment, non-professional services, and public work services.)
This is important. By releasing commodities contract data, the City has successfully released data on all City contracts. (City contracts are broken down into two main categories: “Commodities” and “Professional Services.”) “Commodities” refers to equipment such as office supplies, vehicles, and “non-professional” services like as pest control or janitorial services. “Professional Services” refers to social services, legal services, and consulting, among others.
Professional Services data was released in 2014 and can be found here.
Today’s data release includes a breakdown of contract dollars by vendor, department, and service type. It also includes a list of expiring contracts, making the data not only valuable in the interest of government openness but in how it empowers a larger pool of vendors to do business with the City. Although not all expiring contracts will bid out again, the list could make vendors aware of potential, upcoming opportunities.
As with most of the City’s recent data releases, the commodities contract data is accompanied by visualizations, including a pie chart of the “Top 10 Contracts by Vendor” by the most recent fiscal quarter. This data dates back to July 1, 2013 (Fiscal Year 2014) and will be refreshed quarterly moving forward.
You can view, download, and read more about the commodities contract data here. Please let us know what you think!
Alpha.phila.gov hasn’t had many cosmetic changes since we made it public, but that doesn’t mean there hasn’t been a lot going on behind the scenes. We have been focusing our efforts on building a pattern portfolio to define and share all the common design elements currently on the site. Our goal is to provide a repository of common building blocks (styles, HTML markup, and colors) that developers, our team, and outside vendors can reference when building new sites and services. It is with great pleasure we present Phila.gov Patterns.
A pattern portfolio is helpful in a few ways. First, it allows developers to rapidly develop applications and websites with the same look and feel. This makes for a consistent experience, allowing users to know what to expect when they come to a phila.gov website. Second, it helps bring order to chaos. Anyone who wants to make an application using Phila.gov standards will be able to reference one source for that information. By making many important design decisions up front and documenting them as patterns, consistent, beautifully designed interfaces can be built in a fraction of the time. The ePay Gateway was recently launched and is the first website outside of alpha.phila.gov to use Phila.gov Patterns. We were able to cut in half the time it would normally take to build the interface because almost all of the design elements were already present in the pattern portfolio.
Phila.gov Patterns is built on Foundation 5. We started with a great front-end framework with a large community behind it. Working with Foundation makes creating grids, menus, and buttons extremely fast and easy and lets us focus on the harder development tasks. It is designed to be customized, so it is simple to repurpose common elements like headers, footers, and lists, but also to build more complex patterns as we continue to refine and add to alpha.phila.gov. Additionally, we also built test pages, so that it would be easy to see an example of the kind of things developers can create. These pages help define how the patterns work together to create a full-fledged website.
Phila.gov Patterns is running on Jekyll and hosted on GitHub Pages. We take advantage of Jekyll’s collections feature to allow us to make each pattern standalone, including markup and metadata, and then automatically compile all the pattern content into one master list on the home page.
It is worth noting that originally we were using Pattern Lab, another fantastic tool for creating pattern portfolios, bundled up within Daisy, from the brilliant folks over at Harvard Business Review. Pattern Lab gave us a vocabulary to classify our patterns and Daisy taught us about HBR’s pattern workflow, but ultimately we opted to use Jekyll for a lighter tech footprint.
We have come a long way from the early days and early ideas of this project and we hope that you will continue with us on this journey as we strive to improve our process and and products into something we all can be proud of.