Search

How a Snow Fiasco Uncovered Critical Issues with a $35 Million Dollar Software System

Updated: Jan 3, 2020

Just a week after 2 cyclist deaths and just before the city urges the biking community to memorialize victims of traffic deaths the city of Chicago failed to clear snow from bike lanes for more than 3 business days. This set off a deep dive investigation into how well the 311 system was working.




Many businesses and government buildings further exacerbated the issue by pushing snow into bike lanes. In the video below, you can see University of Illinois at Chicago (UIC) facilities crew knowingly pushing snow into the Halsted street bike lane, a regulator occurrence that is widely documented around campus.



Some cyclists turned to the city's 311 system to report snow packed bike lanes. The 311 system was down for an extended period of time. When the system was back up and running, users were given estimated completion times of 14 days to have the lanes cleared. These same cyclists later found their service requests had been been marked as completed due to "warm weather".


While cyclists were encountering treacherous snow and ice-packed bike lanes, the Chicago Department of Transportation continued to send out requests, urging groups like ours to promote their Vision Zero World Remembrance Day. The focus of the event was to pay tribute to traffic deaths. The timing of the request was a bit tone-deaf, considering, how dangerous the bike lanes were that week. The Department of Transportation are themselves responsible for clearing barrier protected bike lanes.


Below is a recording from the Mayor’s Bicycle Advisory Council (MBAC). In the recording, an attendee asks if this year's snow storm caught the city by surprise. Acting commissioner, Tom Carney, stated that it had not. He further stated the timing of the snow wasn't the most conducive to the city's snow clearing schedule. Carney explained the city prefers snow to hit right around 9 pm so crews can clear the roads by the next day. In reality, after as long as 3 business days post-snowfall, many bike lanes had not been touched.


Also during the MBAC meeting, the Chicago Department of Transportation (CDOT) explained snow removal of barrier protected bike lanes is the responsibility of CDOT to clear. They further explained all other bike lanes are the responsibility of Streets and Sanitation to clear. In response to November's snowfall fiasco, CDOT added an additional code to the city's 311 system. This new code requires citizens to correctly identify which type of bike lane they are reporting in order to move forward in the system. In the video recording below, we strongly urged the city to conduct user testing on this update. It appears user testing has not yet been conducted on this new user flow.


As we highlighted during the MBAC meeting, there are inherent usability risks with the execution of the new 311 codes. These codes require users to accurately identify the type of bike lane they are reporting. How many new cyclists or even veteran cyclists know what a barrier protected bike lane is? What a bike lane looks like in reality versus what it's labeled in the city's out of date bike network map are also two very different things. This means, bike lanes with bollards would require a citizen to first select the barrier protected bike lane option. This would then submit the service request to CDOT, not Streets and Sanitation.


Ask any cyclist you know, bollards are damaged and mowed down daily. In many areas, ALL bollards are missing from bike lanes. What happens when a submitter marks the lane incorrectly because bollards are missing entirely from a "barrier protected bike lane"? Even if a cyclists is able to accurately identify a bike lane, there is a high probability users will not understand the user experience of the new layout and submit the to the incorrect department.


Let's be honest, most of us are in a rush. Not everyone is a subject matter expert on bike lane design. Do you think you could pick the correct category from the two options listed below?


No matter how you answered the question above, there's no way for you to actually know if you selected the correct option. To be honest, we would likely be unable to submit accurately due to flaws in the system's usability and data management flow.


From the data collected in our Bike Lane Uprising database, we identified there are already critical usability and data management issues with the newly redesigned 311 system. We shared a few of these insights with PBS's WTTW in the recording below.


Later during the MBAC meeting, an attendee requested CDOT provide comment on the issues highlighted in the WTTW news segment above. From CDOT's response in the video below, it's clear they do not understand the issues with the 311 system. After post-MBAC follow-up with CDOT, its also clear the city of Chicago is still unaware of the depth of the usability and data flaws associated with bike related 311 service requests. CDOT further stated unless leadership deems it important, the issues with 311 will not be investigated further.



Our Deep Dive Analysis

We conducted a deep dive analysis on bike related 311 submissions. Our analysis was focused primarily on a specific genre of service requests that should have kicked off work to be conducted. We used publicly available data in the city of Chicago's data portal as well as data from our Bike Lane Uprising database to assist in this cross-analysis. We also followed up with submitters directly.


Questions We Wanted to Answer:

  1. Do users understand how to use the system accurately?

  2. Are 311 service requests effectively deploying city services so that work is conducted?



The Results: Key Problems with 311

From our analysis, here are a few key issues with the 311 system:

  1. Lack of Historical Records - It's important to point out the new 311 system no longer sends confirmation emails. This caused makes it impossible for even the most savvy 311 users to be able to look up data. The mobile app doesn't allow users to see their entire submission history which means, even with an account, users are unable to stay up to date on their service requests.

  2. Communication of Intent - While some citizens know 311 submissions documenting vehicles parked in a bike lane will not kick off immediate response from the city, there is a portion of people who do.

  3. Auto-Closed Service Requests - Many requests for service should have kicked off work within the city, however they did not. These requests included complaints of snow, debris, potholes, and even construction blocking bike lanes. Further twisting the knife, each auto-closed service request counts positively towards the 311 system's key performance indicators (KPI's).

  4. Falsely Closed Service Requests - There are multiple service requests closed without work being performed.  Citizens submitted as many as 6 service requests for the same issues, only to find they work requests marked. Notes in the service requests indicate work was completed yet, no work had actually been performed. Citizens who escalated these requests with Alderman Reilly, were able to find resolution, while submitters who contacted Alderman King were still waiting for a resolution after 6 months.

  5. Dormant Service Requests - There are also many service requests that have simply gone untouched. There is no indication what is causing these requests to remain dormant.

  6. Incompliant Construction Sites - For service requests involving dangerous construction sites, data shows it either took the city MONTHS to inspect these sites, or the sites were simply never inspected. As the transportation community knows all too well, construction sites increase the risk cyclists and pedestrians can be injured or killed. Leaving these sites unchecked is a critical issue.

  7. Erroneous Labeling - Multiple illegal construction sites were erroneously coded as "vehicle in bike lane" which then auto-closed the service request and stripped all data from public view. We suspect these sites were never inspected as it is our understanding "Vehicle in bike lane" submissions are only sent to the Department of Finance. Multiple service requests for potholes (potholes so bad they caused flat tires) were coded as vehicle in bike lane and auto closed. Again, if the system illegitimately labels a service request, such as construction, debris, or a pothole, as a "Vehicle in bike lane" instead, then no information on the status of the record is shared in 311.

  8. Data Triaging - The data submitted to 311 does not appear be triaged to relevant city departments in an impactful output. This means the city places the responsibility and burden on average citizens to stumble through learning exactly how the system works and report to each and every department individually.  



So how did this happen?

How was Bike Lane Uprising the first one to uncover these issues? Why did a software system (that cost taxpayers no less than $35 million dollars to redesign) operate for over a year without any of the city's countless employees and contracting companies noticing? Many of these issues should have been resolved during product development and/or shortly after launch.



Six opportunities these issues should have been uncovered during product development and post launch:

  1. Research & Development - These issues should have been uncovered during the research and development phase of the new 311 system. The contracted company, Salesforce, should have aided the city to develop the 311 system in a way that would be effective to users both front