gui_0000_Layer Comp 3.jpg

Achieving Symbiosis and the Conversational User Interface

An intelligent CUI concept that helps criminal investigators better collect, make sense of, and share information. 

Objective

Find a market for a conversational user interface (CUI) that utilizes human-computer symbiosis and design a concept for a target audience with use cases.

Context

  • Client: Microsoft Design Expo 2016
  • Target Audience: Criminal Investigators
  • A Carnegie Mellon University Design Studio Project 
  • Collaborative project with Raghavendra Kandala and Min Kim
  • Project Length: 17 weeks
  • My Role: Research, synthesis, storyboard development, conversational prototyping, and video production
  • Credit: Filming and GUI by Min Kim

More detailed process documentation can be found here


The Solution

My team developed a concept for a contextually aware intelligent conversational agent that utilizes CUI + GUI to help criminal investigators collect, make sense of, and share information more effectively— enhancing officer safety and collaboration efforts. The conversational agent and investigator work together and play to their strengths resulting in better investigative work. 

Investigator Key Functions:

  • Interact and engage more deeply with people rather than become distracted by note-taking
  • Decide which information is relevant to the case
  • Develop a narrative to share with other law enforcement officials
  • Make decisions that require critical thinking

Intelligent Conversational Agent Key Functions:

  • Listen to, transcribe, and summarize conversations
  • Provide real-time contextual intel 
  • Review details from the investigator's work and help trigger reflection and reframing of problems
  • Suggest and document action items and connect the investigator with relevant resources
The investigators and intelligent conversational agent (which is a combination of CUI and GUI) work together as they move through the five phases of investigative work. The investigators take on humanesque tasks (written in blue) and the intelligent conversational agent takes on computing tasks (written in white).  

The investigators and intelligent conversational agent (which is a combination of CUI and GUI) work together as they move through the five phases of investigative work. The investigators take on humanesque tasks (written in blue) and the intelligent conversational agent takes on computing tasks (written in white).  

 

The Design Process

Exploratory Research

Learning About The Stakeholders

Once my team decided to focus on criminal investigation, I spoke to a detective to learn about the stakeholders and how they relate to each other. I created a diagram to illustrate these relationships. Next I reached out to criminal investigation organizations, each of which agreed to participate in our design research. These included:

  • Pittsburgh Bureau of Police
  • Pittsburgh Office of Municipal Investigations
  • Pittsburgh Federal Bureau of Investigation

My team and I wanted to learn what the main pain points are in investigative work and determine how a symbiotic CUI could help.

 

Generative Research

Participatory Design Workshop 1

  • Objective: Learn investigators' work processes, challenges, and visions for the future
  • Methods: Interviews, journey maps, card sorting, brainstorming, affinity mapping, dot voting

Key Insights

  • Collecting and sharing information is difficult
  • Investigators have to sift through large amounts of data and determine which points are relevant
  • Determining which data points are relevant to a case can be a guessing game
 

Participatory Design Workshop 2

  • Objective: Learn what qualities investigators look for in a symbiotic relationship and what symbiotic functions could help investigators achieve better work. 
  • Methods: Symbiotic worksheet, build environments with blocks, fill environments with dolls, sculpt concepts with clay

I created a method to determine which tasks investigators believe are inherently human and which could be offloaded to a "magical helper". Participants used cards to brainstorm tasks that are enjoyable, unpleasant, or could benefit from a magic ability. Then they determined which of those tasks they would keep for themselves and which they would delegate to a "magical helper" in order to achieve the most effective work as a team. 

After this exercise, we gave the investigators blocks and dolls and asked them to make a scene that shows how their magical helper would fit into their daily work. Lastly, we gave the investigators clay and asked them to build their magical helper and show us what it looked like. 

Key Insights

  • Investigators wish they had access to consolidated reliable data
  • Investigators want to spend more time collaborating and less time managing data
  • Investigators want to feel like they can focus on the problem and not be distracted by less important tasks
 

Evaluative Research

Testing Concept Storyboards

  • Objective: Determine which CUI concepts resonate with investigators and which concepts they object to
  • Method: Speed date 25 storyboards

Based on the insights we collected from our initial research sessions with investigators, we put together several storyboards illustrating CUI human-computer symbiosis scenarios. The feedback we got from officers helped guide our prototyping process. 

Key Insights 

  • Officers are not interested in holding a verbal conversation with technology
  • Officers appreciate accessing data analytics in a hands-off way
 

Scrappy Prototyping

  • Objective: Explore the interaction points, forms, and factors of a CUI in the investigative field
  • Methods: Hardware prototyping, role playing, conversational prototyping

We started to prototype CUI technologies using simple electronics, cell phones, tablets, ear buds, and roleplaying. This started to introduce us to the types of interaction design considerations we needed to keep in mind. We realized, for example, that a CUI alone was not enough. Visual communication is also an important way to quickly communicate time-sensitive information, so we decided to test a combination of CUI and GUI. We also learned more about the CUI and the investigator relationship—what aspects were cumbersome and which were convenient. 

Defining the Interactions

After concluding that we wanted to combine CUI with GUI, we started thinking about the interactions involved in the user experience: speaking (sometimes multiple speakers), tactile interactions, audio feedback, visual feedback, haptic feedback, etc. We wanted the interactions to be as simple and non-cumbersome as possible and started mapping interactions for different scenarios. 

Above is a map of the interactions between investigator and CUI in terms of verbal field notes, interviews, contextual intel, and micro reviews. Graphic Credit: Raghavendra Kandala.

Above is a map of the interactions between investigator and CUI in terms of verbal field notes, interviews, contextual intel, and micro reviews. Graphic Credit: Raghavendra Kandala.

Above is a map of the interactions between investigator and CUI in terms of narrating, editing a report, searching data, making a to-do list, and receiving suggestions. Graphic Credit: Raghavendra Kandala.

Above is a map of the interactions between investigator and CUI in terms of narrating, editing a report, searching data, making a to-do list, and receiving suggestions. Graphic Credit: Raghavendra Kandala.

Narrowing Down the Concepts

We narrowed down our concepts to two main ideas: "Active Listener" and "Reflective Investigator". These concepts were most valuable because they touch on investigators' toughest pain points: collecting, making sense of, and sharing information.

  • Active Listener: Helps investigators focus on conversation by offloading multitasking such as taking notes or checking databases
  • Reflective Investigator: Helps investigators think through their day, document insights, make to-do lists, and coordinate action plans
Above is a diagram of the forms, interactions, value and symbiosis associated with the Active Listener and Reflective Investigator concepts. Graphic Credit: Kaylee White.

Above is a diagram of the forms, interactions, value and symbiosis associated with the Active Listener and Reflective Investigator concepts. Graphic Credit: Kaylee White.

Testing Prototypes with investigators

We built out our concept prototypes and took them to the police department. The investigators reacted positively to the Active Listener and Reflective Investigator prototypes. We also tested a prototype called "Sense Maker" which was designed to be a bit frustrating and riddled with errors so we could see how participants would react at points of failure. 

Key feedback

  • Investigators want to be sure that the CUI concept would be able to function in a noisy, chaotic environment
  • Investigators want a collaborative component to the concept so they can better share information with their peers
  • Investigators want to know how they can differentiate between personal notes and official documentation
Following the concept testing, each investigator filled out a form explaining what they liked and disliked about the prototypes. 

Following the concept testing, each investigator filled out a form explaining what they liked and disliked about the prototypes. 

Making the Deliverable

Defining the Form

When thinking about the form that the CUI + GUI should take, my team decided to build on the preexisting practices of the investigators. They still used paper and pen to take notes during an investigation. We developed a notebook-style digital device with an accompanying recorder/stylus. This allows investigators to make notes which are instantly digitized, and see visual updates produced by the intelligent conversational agent. 

gui_0000_Layer Comp 4.jpg