A field research and design project exploring how computer vision might help first responders in their duties. Our team created a research plan, interviewed experts, designed an intervention and tested its viability.
To begin the project, we conducted secondary research to formulate a research question: How do first responders assess emergency situations?
Our team then prepared a study guide consisting primarily of semi-structured interviews and guided storytelling. The interviews were semi-formal, loosely structured and conversational. Questions were formulated through secondary research and combined into a loosely followed script. The guided storytelling activity asked first responders to both “Walk us through a typical day” and to “walk us through a recent call.” This approach helped us identify both the critical moments in the job and the greater context of the work.
Our participant recruiting began with contacts in our own networks. We asked these first responders to refer co-workers and friends. Through targeted emails, we contacted 13 first responders and interviewed ten. These participants included EMTs, fire fighters, a street medic, and members of the Coast Guard.
As recorded interviews accumulated, the team organized and began to synthesize our data. Each significant piece of information was written on a sticky note. Together, we organized the notes into patterns and themes. Insights were derived from each theme and actionable design principles were formulated.
We found that first responder roles vary greatly but have many common threads. While time responding to emergencies is considerable, first responders experience extensive down time between calls. The cycle of work contains three main phases:
Throughout the work cycle, documentation plays a crucial role. There is always somebody in charge of keeping track of all decisions, actions, vitals, aid administered, any unusual activity, and this documentation goes through multiple hands and departments.
Three insights in particular went on to inform our design direction and intervention.
“The people who put the information into the database are the people who went out... they’ve gotten beat up in the elements for how many hours in the cold and wet and now they’re expected to come back and make a thorough report of what they’ve done”
“Garbage in, garbage out. If the reports are badly written, they get thrown out”
“So all of that is tracked so on a more macro scale we can see those trends... this is the average response time”
Responders are one part of a larger system. Every type of first responder relies on other teams to get the job done. Every participant mentioned the process of handing a case off to another team or working closely with another branch of government responders to respond or accurately assess an emergency situation.
There’s a clear need to update documentation systems. Documentation goes through multiple hands using different systems. Nothing about the documentation is universal, but each system overlaps others, leading to redundancy and duplicated work. The quality of these tools can be low, with one responder describing the Sun Pro system as “30 years out of date, 30 years ago.” Faulty tools can also lead to confusion, such as two devices reading conflicting vitals. These issues must be resolved by hand for each call.
These insights and others were synthesized into actionable design principles.
Support flexible roles in critical moments. There’s no typical call. Responders have to adapt on the spot.
Support order. There’s a strong hierarchy of command and procedure that needs to be supported.
Be aware of existing processes and procedures. There’s a unique, interdependent system. We can’t reinvent the system - there’s too much to change all at once. Any design solution needs to fit into this larger system.
Do not hinder in critical moments. We need to appreciate how valuable time and attention can be in emergency contexts.
Facilitate clear communication and collaboration. Everyone wants to collaborate, but the problems are often in communication.
With the opportunities uncovered through research, we defined a clear product goal: improve documentation through computer vision. We proposed a device that can track vitals and take notes for documentation purposes.
To arrive at this product and to validate our approach, we began with different product concepts. Given the large proportion of first responders’ time spent on documentation, as well as its universal presence among participants, we sought to improve this process. A requirement of the project was to apply computer vision technology to any product outcome.
With our design principles in mind, we created a behavioral prototype to explore how a computer vision device could assist in the documentation process. We wanted to know how first responders would use a “magic” device and what functions they would expect it to provide them. Our first prototype was nonfunctional but built with plausible hardware - a battery pack, a Raspberry Pi computer, and a webcam. This hardware was meant to look technical enough that people wouldn’t get caught up in how it worked, but would take it at face value so that we could see how they respond to it.
We tested the prototype by creating a hypothetical emergency situation where someone had been stabbed. The first responders used the device to scan a patient, which then provided notes on a tablet. A member of the research team added vitals and injury notes to an image on the tablet in real time, then handed it to our participant. We observed the participant and later asked how the device’s information would be used in their documentation. He expected the prototype to record what was done to the patient, including which drugs were given and what kind of gauze was applied. This information could be handed off to doctors and used for both providing care and later reports.
Over several tests of the initial behavioral prototype, first responders valued the potential to minimize documentation. We saw different expectations and ways of using the prototype. One responder only stabilized the victim and left basic notes for doctors, concluding his involvement. Another tester had to write up a report of the entire event based on multiple sets of notes. He was more interested in the logging and timekeeping aspects of the design.
Some features were particularly valued.
The form factor of the first prototype didn’t work - nobody wanted to hold the device, and in the field there’s no good place to set it down. Because medical emergencies can be messy, a small and easy to clean device is important. Based on this feedback, the next iteration was made smaller and affixed to the body.
When interviewing people after testing the behavioral prototypes, we found that saving time was a major benefit. Having more time in the moment is like having another team member. Most importantly, the product allows more time for an emergency responder to focus on the situation, rather than documentation.
The proposed product aided focus, which diminishes miscommunication. By partially automating documentation, a responder can focus better on their situation. Because the product documents many actions, this also eliminates the need for handwriting, making communication more clear. Since automated documentation can be made to match any type of form, responders do not have to worry if this technology matches other providers, such as hospitals. Rather than create a new technology ecosystem, this product integrates with existing systems.
With a validated product concept and an understanding of the problem’s context, next steps would include designing specific interactions and planning for technology. This project in its current form is a high level concept, made on the assumption that improving computer vision will make the proposed functions possible. In reality, technical limitations will inform what specific tasks can be automated. The actual level of technical maturity will inform the feature set and level of interaction between the product and first responders.