Achieving Symbiosis and the Conversational User Interface
An intelligent CUI concept that helps criminal investigators better collect, make sense of, and share information.
Find a market for a conversational user interface (CUI) that utilizes human-computer symbiosis and design a concept for a target audience with use cases.
- Client: Microsoft Design Expo 2016
- Target Audience: Criminal Investigators
- A Carnegie Mellon University Design Studio Project
- Collaborative project with Raghavendra Kandala and Min Kim
- Project Length: 17 weeks
- My Role: Research, synthesis, storyboard development, conversational prototyping, and video production
- Credit: Filming and GUI by Min Kim
More detailed process documentation can be found here.
My team developed a concept for a contextually aware intelligent conversational agent that utilizes CUI + GUI to help criminal investigators collect, make sense of, and share information more effectively— enhancing officer safety and collaboration efforts. The conversational agent and investigator work together and play to their strengths resulting in better investigative work.
Investigator Key Functions:
- Interact and engage more deeply with people rather than become distracted by note-taking
- Decide which information is relevant to the case
- Develop a narrative to share with other law enforcement officials
- Make decisions that require critical thinking
Intelligent Conversational Agent Key Functions:
- Listen to, transcribe, and summarize conversations
- Provide real-time contextual intel
- Review details from the investigator's work and help trigger reflection and reframing of problems
- Suggest and document action items and connect the investigator with relevant resources
The Design Process
Learning About The Stakeholders
Once my team decided to focus on criminal investigation, I spoke to a detective to learn about the stakeholders and how they relate to each other. I created a diagram to illustrate these relationships. Next I reached out to criminal investigation organizations, each of which agreed to participate in our design research. These included:
- Pittsburgh Bureau of Police
- Pittsburgh Office of Municipal Investigations
- Pittsburgh Federal Bureau of Investigation
My team and I wanted to learn what the main pain points are in investigative work and determine how a symbiotic CUI could help.
Participatory Design Workshop 1
- Objective: Learn investigators' work processes, challenges, and visions for the future
Methods: Interviews, journey maps, card sorting, brainstorming, affinity mapping, dot voting
- Collecting and sharing information is difficult
- Investigators have to sift through large amounts of data and determine which points are relevant
- Determining which data points are relevant to a case can be a guessing game
Participatory Design Workshop 2
- Objective: Learn what qualities investigators look for in a symbiotic relationship and what symbiotic functions could help investigators achieve better work.
- Methods: Symbiotic worksheet, build environments with blocks, fill environments with dolls, sculpt concepts with clay
I created a method to determine which tasks investigators believe are inherently human and which could be offloaded to a "magical helper". Participants used cards to brainstorm tasks that are enjoyable, unpleasant, or could benefit from a magic ability. Then they determined which of those tasks they would keep for themselves and which they would delegate to a "magical helper" in order to achieve the most effective work as a team.
After this exercise, we gave the investigators blocks and dolls and asked them to make a scene that shows how their magical helper would fit into their daily work. Lastly, we gave the investigators clay and asked them to build their magical helper and show us what it looked like.
- Investigators wish they had access to consolidated reliable data
- Investigators want to spend more time collaborating and less time managing data
- Investigators want to feel like they can focus on the problem and not be distracted by less important tasks
Testing Concept Storyboards
- Objective: Determine which CUI concepts resonate with investigators and which concepts they object to
- Method: Speed date 25 storyboards
Based on the insights we collected from our initial research sessions with investigators, we put together several storyboards illustrating CUI human-computer symbiosis scenarios. The feedback we got from officers helped guide our prototyping process.
- Officers are not interested in holding a verbal conversation with technology
- Officers appreciate accessing data analytics in a hands-off way
- Objective: Explore the interaction points, forms, and factors of a CUI in the investigative field
- Methods: Hardware prototyping, role playing, conversational prototyping
We started to prototype CUI technologies using simple electronics, cell phones, tablets, ear buds, and roleplaying. This started to introduce us to the types of interaction design considerations we needed to keep in mind. We realized, for example, that a CUI alone was not enough. Visual communication is also an important way to quickly communicate time-sensitive information, so we decided to test a combination of CUI and GUI. We also learned more about the CUI and the investigator relationship—what aspects were cumbersome and which were convenient.
Defining the Interactions
After concluding that we wanted to combine CUI with GUI, we started thinking about the interactions involved in the user experience: speaking (sometimes multiple speakers), tactile interactions, audio feedback, visual feedback, haptic feedback, etc. We wanted the interactions to be as simple and non-cumbersome as possible and started mapping interactions for different scenarios.
Narrowing Down the Concepts
We narrowed down our concepts to two main ideas: "Active Listener" and "Reflective Investigator". These concepts were most valuable because they touch on investigators' toughest pain points: collecting, making sense of, and sharing information.
- Active Listener: Helps investigators focus on conversation by offloading multitasking such as taking notes or checking databases
- Reflective Investigator: Helps investigators think through their day, document insights, make to-do lists, and coordinate action plans
Testing Prototypes with investigators
We built out our concept prototypes and took them to the police department. The investigators reacted positively to the Active Listener and Reflective Investigator prototypes. We also tested a prototype called "Sense Maker" which was designed to be a bit frustrating and riddled with errors so we could see how participants would react at points of failure.
- Investigators want to be sure that the CUI concept would be able to function in a noisy, chaotic environment
- Investigators want a collaborative component to the concept so they can better share information with their peers
- Investigators want to know how they can differentiate between personal notes and official documentation
Making the Deliverable
Defining the Form
When thinking about the form that the CUI + GUI should take, my team decided to build on the preexisting practices of the investigators. They still used paper and pen to take notes during an investigation. We developed a notebook-style digital device with an accompanying recorder/stylus. This allows investigators to make notes which are instantly digitized, and see visual updates produced by the intelligent conversational agent.