Just a week after 2 cyclist deaths and just before the city urges the biking community to memorialize victims of traffic deaths the city of Chicago failed to clear snow from bike lanes for more than 3 business days. This set off a deep dive investigation into how well the 311 system was working.
Many businesses and government buildings further exacerbated the issue by pushing snow into bike lanes. In the video below, you can see University of Illinois at Chicago (UIC) facilities crew knowingly pushing snow into the Halsted street bike lane, a regulator occurrence that is widely documented around campus.
Some cyclists turned to the city's 311 system to report snow packed bike lanes. The 311 system was down for an extended period of time. When the system was back up and running, users were given estimated completion times of 14 days to have the lanes cleared. These same cyclists later found their service requests had been been marked as completed due to "warm weather".
While cyclists were encountering treacherous snow and ice-packed bike lanes, the Chicago Department of Transportation continued to send out requests, urging groups like ours to promote their Vision Zero World Remembrance Day. The focus of the event was to pay tribute to traffic deaths. The timing of the request was a bit tone-deaf, considering, how dangerous the bike lanes were that week. The Department of Transportation are themselves responsible for clearing barrier protected bike lanes.
Below is a recording from the Mayor’s Bicycle Advisory Council (MBAC). In the recording, an attendee asks if this year's snow storm caught the city by surprise. Acting commissioner, Tom Carney, stated that it had not. He further stated the timing of the snow wasn't the most conducive to the city's snow clearing schedule. Carney explained the city prefers snow to hit right around 9 pm so crews can clear the roads by the next day. In reality, after as long as 3 business days post-snowfall, many bike lanes had not been touched.
Also during the MBAC meeting, the Chicago Department of Transportation (CDOT) explained snow removal of barrier protected bike lanes is the responsibility of CDOT to clear. They further explained all other bike lanes are the responsibility of Streets and Sanitation to clear. In response to November's snowfall fiasco, CDOT added an additional code to the city's 311 system. This new code requires citizens to correctly identify which type of bike lane they are reporting in order to move forward in the system. In the video recording below, we strongly urged the city to conduct user testing on this update. It appears user testing has not yet been conducted on this new user flow.
As we highlighted during the MBAC meeting, there are inherent usability risks with the execution of the new 311 codes. These codes require users to accurately identify the type of bike lane they are reporting. How many new cyclists or even veteran cyclists know what a barrier protected bike lane is? What a bike lane looks like in reality versus what it's labeled in the city's out of date bike network map are also two very different things. This means, bike lanes with bollards would require a citizen to first select the barrier protected bike lane option. This would then submit the service request to CDOT, not Streets and Sanitation.
Ask any cyclist you know, bollards are damaged and mowed down daily. In many areas, ALL bollards are missing from bike lanes. What happens when a submitter marks the lane incorrectly because bollards are missing entirely from a "barrier protected bike lane"? Even if a cyclists is able to accurately identify a bike lane, there is a high probability users will not understand the user experience of the new layout and submit the to the incorrect department.
Let's be honest, most of us are in a rush. Not everyone is a subject matter expert on bike lane design. Do you think you could pick the correct category from the two options listed below?
No matter how you answered the question above, there's no way for you to actually know if you selected the correct option. To be honest, we would likely be unable to submit accurately due to flaws in the system's usability and data management flow.
From the data collected in our Bike Lane Uprising database, we identified there are already critical usability and data management issues with the newly redesigned 311 system. We shared a few of these insights with PBS's WTTW in the recording below.
Later during the MBAC meeting, an attendee requested CDOT provide comment on the issues highlighted in the WTTW news segment above. From CDOT's response in the video below, it's clear they do not understand the issues with the 311 system. After post-MBAC follow-up with CDOT, its also clear the city of Chicago is still unaware of the depth of the usability and data flaws associated with bike related 311 service requests. CDOT further stated unless leadership deems it important, the issues with 311 will not be investigated further.
Our Deep Dive Analysis
We conducted a deep dive analysis on bike related 311 submissions. Our analysis was focused primarily on a specific genre of service requests that should have kicked off work to be conducted. We used publicly available data in the city of Chicago's data portal as well as data from our Bike Lane Uprising database to assist in this cross-analysis. We also followed up with submitters directly.
Questions We Wanted to Answer:
Do users understand how to use the system accurately?
Are 311 service requests effectively deploying city services so that work is conducted?
The Results: Key Problems with 311
From our analysis, here are a few key issues with the 311 system:
Lack of Historical Records - It's important to point out the new 311 system no longer sends confirmation emails. This caused makes it impossible for even the most savvy 311 users to be able to look up data. The mobile app doesn't allow users to see their entire submission history which means, even with an account, users are unable to stay up to date on their service requests.
Communication of Intent - While some citizens know 311 submissions documenting vehicles parked in a bike lane will not kick off immediate response from the city, there is a portion of people who do.
Auto-Closed Service Requests - Many requests for service should have kicked off work within the city, however they did not. These requests included complaints of snow, debris, potholes, and even construction blocking bike lanes. Further twisting the knife, each auto-closed service request counts positively towards the 311 system's key performance indicators (KPI's).
Falsely Closed Service Requests - There are multiple service requests closed without work being performed. Citizens submitted as many as 6 service requests for the same issues, only to find they work requests marked. Notes in the service requests indicate work was completed yet, no work had actually been performed. Citizens who escalated these requests with Alderman Reilly, were able to find resolution, while submitters who contacted Alderman King were still waiting for a resolution after 6 months.
Dormant Service Requests - There are also many service requests that have simply gone untouched. There is no indication what is causing these requests to remain dormant.
Incompliant Construction Sites - For service requests involving dangerous construction sites, data shows it either took the city MONTHS to inspect these sites, or the sites were simply never inspected. As the transportation community knows all too well, construction sites increase the risk cyclists and pedestrians can be injured or killed. Leaving these sites unchecked is a critical issue.
Erroneous Labeling - Multiple illegal construction sites were erroneously coded as "vehicle in bike lane" which then auto-closed the service request and stripped all data from public view. We suspect these sites were never inspected as it is our understanding "Vehicle in bike lane" submissions are only sent to the Department of Finance. Multiple service requests for potholes (potholes so bad they caused flat tires) were coded as vehicle in bike lane and auto closed. Again, if the system illegitimately labels a service request, such as construction, debris, or a pothole, as a "Vehicle in bike lane" instead, then no information on the status of the record is shared in 311.
Data Triaging - The data submitted to 311 does not appear be triaged to relevant city departments in an impactful output. This means the city places the responsibility and burden on average citizens to stumble through learning exactly how the system works and report to each and every department individually.
So how did this happen?
How was Bike Lane Uprising the first one to uncover these issues? Why did a software system (that cost taxpayers no less than $35 million dollars to redesign) operate for over a year without any of the city's countless employees and contracting companies noticing? Many of these issues should have been resolved during product development and/or shortly after launch.
Six opportunities these issues should have been uncovered during product development and post launch:
Research & Development - These issues should have been uncovered during the research and development phase of the new 311 system. The contracted company, Salesforce, should have aided the city to develop the 311 system in a way that would be effective to users both front and back end users. Salesforce should have advocated for usability flows that allow average citizens to submit service requests to the system accurately and effectively. Salesforce and the city employees overseeing the project, should have planned for, tested, and iterated the systems data flow until the data accurately flowed through the system.
Usability Testing - It's no secret that new software goes through usability testing. It's a critical phase to developing any software. Usability testing allows you to witness how users will interact with your product. How you think a user with interact with your software is likely different than what users will do in real life. Testing offers the opportunity to uncover these incongruencies, pain points, heuristics that are generally confusing, as well as technical bugs and glitches. Usability testing should have been conducted on the 311 system, for both front end users (meaning citizens like you and I), as well as back end users (meaning anyone in the city effected by the data output). Less than a month before the 311 system launched, the city of Chicago conducted what they called usability testing. We personally attended one of these sessions and can assure you this was anything but user testing. As someone who has worked in product development and conducted countless usability tests, I can assure you these sessions were far from industry standard. These sessions were not set up in a way that would allow the development team to take away meaningful insights and modify the 311 system based on that feedback. Nothing in the study was scored, notes were not being gathered, and because the sessions were hosted a month before roll out, there wasn't time for any modifications to be made to the system. These sessions were nothing more than a cattle call that allowed the development team to check a box stating they had completed the task of "usability testing".
Post-Launch Data Management - On multiple occasions, the city of Chicago has stated the data submitted by citizens to 311 is critical to the success of their service deployment efforts. They've stated The Department of Finance uses the data to deploy parking aids for the day. They have also stated The Department of Transportation and The Department of Streets and Sanitation use the data in their maintenance efforts. Based on our analysis, it's clear the system has erroneously mislabeled service requests, triaging data, and closing service requests in a way that should have flagged a review. Why haven't these issues been uncovered by any of the employees and contracting companies working in those 3 departments? Has no one been reviewing the data for accuracy?
Aldermen - When citizens become frustrated with their 311 service requests closed or ignored, they've reached out to Aldermen. Depending on the ward the issue is located, citizens receive wildly differing responses. Alderman Reilly, who called the 311 system a "black hole" has demonstrated rather quick resolutions to bike lane issues escalated to his office. Meanwhile, Aldermen like King dismisses these requests for escalation, stating CDOT is too busy. Blasé responses from aldermen have even resulted in citizens reaching out to CDOT directly. While acting commissioner Tom Carney might think it's fine to casually dismiss our claims (as shown in the video above), our team has been included on multiple email chains to aldermen (and CDOT) where citizens have attempted to escalate issues with the 311 system. If aldermen and citizens have both reached out to CDOT about the issues with the 311 system, why hasn't the the system's issues been investigated?
Social Media - Out of desperation, cyclists have taken to social media on multiple occasions to escalate issues with the 311 system. Various city agencies and city leaders have been tagged in these posts. These departments include CDOT, Streets and Sans, aldermen, and even the mayor. These posts provide "notice" something is wrong with the 311 system, yet a review has not been conducted.
Process Transformation - Technology alone will not solve problems with bike lanes. City departments appear to be utilizing the 311 system to automate the deployment of city services. These automations require data submitted to the 311 system be perfect. If the data is not perfect, the system will spit out an inaccurate "answer". Due to the usability issues built into the 311 system, the average citizens would need to be a subject matter expert on the city's transportation system as well as the cumbersome usability flow of the 311 system in order to effectively submit to 311. In other words, the 311 system and the city services relying on its data are victim to a "garbage in, garbage out" data flow. There is no way for city services to operate meaningfully until the system's usability and data flow issues are resolved.
In conclusion:
The list above shows a system that is not working as intended. It also shows a great deal of city employees and companies with multi-million dollar city contracts who might not be providing effective outputs. We need to take a step back and perform a value analysis on the services and city contracts paid by tax payers against the value they are actually providing.