How can prototyping facilitate design discovery?

Within the design process, prototyping is most often used once an initial concept has been identified, helping to iteratively define that concept.

Through my project Unstable Label, I show a different approach to prototyping by using it as a method in the early design discovery phases of the project to think through ideas before any individual concept has been identified. While the prototyping takes different forms, each prototype was developed without a particular outcome in mind. Instead, they were all built as a way to explore, communicate, and make sense of machine learning in the civic context.

Research & Sensemaking

The first phase of work involved amassing research within the space of civic machine learning through a variety of methods.

Process diagram with the research and sensemaking section highlighted

Secondary Research

Secondary research was an extremely important part of this early design work, serving as the starting point for much of the prototyping. My research engaged with the substantial amount of existing design projects on machine learning, as well as the critical scholarship happening in Science & Technology Studies, Anthropology, and New Media Studies.

Quotes printed on reciepts, with sections highlighted in yellow

Take a look at the curated are.na boards to get an overview of this research:

Into the Black Box - broadly focused on critical perspectives on algorithms, ML and AI
Data/Labeling/Information - research at the intersection of data labeling, datasets for ML, information, and contestability

Prototyping

Prototyping and making in this stage of work took the form of cardboard models (sometimes with simple electronics), quick web apps, or even just diagrams. These prototypes were often responding to other threads of research, delving into the materiality of the concepts I was encountering.

Cardboard box with reciepts coming out of one end (left) Code displayed in a Jupyter Notebook (right)

Cardboard metal detector type device (left) Interface labeled Become a Hotdog (right)

Someone being scanned (left) Box with a computer mounted inside, filled with reciepts (right)

Handheld cardboard device covered in buttons (left) Someone looking through a google cardboard device (right)

Some of the initial ideas that were conceived in this early stage of prototyping would consistently reappear in later stages of the design discovery process:

Cardboard device covered in numbered buttons and a screen (left) Interface listing categories mapped to numbers

The prototype above was where I first explored ideas of data collection devices that allows user to create their own set of categories, a concept that reappears throughout the rest of the design process.

Primary Research in Context

In addition to working in the design studio, I also spent time working to understand how machine learning was being operationalized in my local community, specifically with data-driven policing. There is an active community of abolitionist organizers in Los Angeles, focused on dismantling data-driven poicing, whose work I participated in and learned from.

Zines being held in the air (left and middle) Diagram labeled Algorithmic Ecology (right)

Much of this work took the form of collaboratively created popular education material. For more on the materials you see above, see the Algorithmic Ecology, and PredPol Is...

The learnings from this work greatly informed my design work, forming the foundational values upon which my machine learning speculations were built from. Some important takeaways included:

An analysis of machine learning has to explicitly understand how power dynamics are embedded within the algorithm.
Understanding an algorithm involves more than understanding the mechanics of its operation. We also need to understand how it was developed, whose interest it serves, and the historical context for its application.
The impact on the community where an algorithm is developed needs to be centered, especially when those communities aren't always considered "users"

Sensemaking through diagrams

Creating conceptual diagrams are an incredibly useful way for me to start synthesizing some of the research and prototyping. Creating these visual metaphors and flows helped to focus my view into particular aspects of the machine learning process.

Diagram that looks like an engineering drawing, labeled predpol model evaluation

Diagram that looks like an engineering drawing, labeled deep learning training process

Diagram that shows particles being sifted through two frames

Diagram illustrating a pyramid with blocks, labeled neural network architecture

Diagram that looks like an engineering drawing, labeled machine learning terminology

Mickey mouse as the sorcerer's apprentice, in front of a background annotated with if-else loops

These diagrams were not primarily designed as communication tools. Instead, they served as a form of prototyping with ideas.

Developing Scenarios

Process diagram with the section labeled Synthesis highlighted

The next phase of work was to do some synthesis of the broad range of research by creating specific scenarios that would put some of those initial explorations into particular contexts as a way to dig deeper into the subject matter.

From the initial research, I chose to focus on research threads on the process of data collection for machine learning as well as themes of surveillance, safety, and policing from my experiences working on dismantling predictive policing.

The scenarios I developed responded to the prompt:

What would data collection devices that collected data on fear look like? And how might you intervene in such data collection systems?

Collage of sketches and images of data collection devices

Collage of sketches and images of surveillance devices

To address this brief, the research effort became more focused on mechanisms for collecting data. In this case, the research involved prototypes as well as secondary research on the material qualities of various data collection devices.

Collage of different cardboard and electronic prototypes of data collection devices

The development of the scenarios emerged through the prototypes I created. As the prototypes gained fidelity, I also began "prototyping" stories of the near-futures where these devices would exist, filling in details of the entire system of data collection.

To see a more detailed look at these scenarios, take a look at the Machined Data project page.

Scenario 1: Municipal Data Collection

The first scenario imagined a municipal "Data Surveyors" program, where residents participated in a "fear mapping" of their neighborhoods, which would impact where resources were deployed.

Closeup annotated view of a orange, triangular electronic device covered in LEDs

Person wearing a high visability vest with an orange triangular shaped device attached to the middle

It posed questions about the subjective nature of data representing "fear" as well as questions about whose data would be collected in a program like this.

Scenario 2: Corporate Data Collection

The second scenario was focused on corporate data collection, imagining a system where a big tech company allowed users of home security technology to train their own algorithms to automate their cameras.

Homemade labels placed haphazardly ontop of a binoculars device

Someone looking through a binoculars device

This scenario imagines people intervening in such a consumer system by "re-labeling" the default categories provided.

A key insight that came out of this scenario work was:

Machine learning is a process of worldbuilding facilitated by the labeling of data. Each algorithm is fed data to generate a computational vision of the world, which is deployed into the world as if it reflects reality.

While these scenarios largely just communicate a critique of machine learning systems, elements and themes developed for these scenarios were key parts of imagining entirely new, more experimental approaches to machine learning.

Moving from Research to Concept

Process diagram with the section labeled Concept highlighted

Up until this part of the process, the design work was largely about exploring and critiquing existing machine learning systems. The next step was to pull together all the insights from those explorations into new concepts for machine learning.

Feeding an algorithm fictional data could be used to generate aspirational worlds.

Building from the insights I found while developing scenarios, I realized that feeding an algorithm fictional data could be used to generate aspirational worlds. I sketched out an initial prototype for users to create imaginary data ontop of google street view:

A prototype website showing a draggable box ontop of a google street view image

The design brief then became how do you prompt users of this tool for imaginary data? How do you spark their imagination? I developed the following variations to try to answer that question.

Grid of different UIs, with one highlighted captioned Relabeling

First Prototype

The most promising and intriguing approach was using the idea of relabeling, a theme which first appeared in the early phases of this design research.

The first prototypes of this data labeling application design around relabeling are shown below. For a user to label data, they had to create their own categories instead of using ones that were pre-defined for them.

Screenshot of an app that annotates google street view with labels

A long display table with an iPad sitting at the front (left) Dialog box asking user to contribute to a machine learning algorithm (right)

This prototype was used as a part of a physical installation that illustrated the re-labeling concept at the ArtCenter Media Design Practices Thesis Show.

Testing the Experience

The prototype, gallery show, and thesis presentation allowed me to test the concept.

The big insight from testing was that the "descriptions" written by the contributors, which represented the stories behind the relabeling, was the real "data" being collected, not the newly labeled images.

The key insight was that the written descriptions are the real "data" being collected, not the labeled images.

Top down view of two people using an ipad at a display table in a gallery

Detailed Design

Following the gallery show, and testing of the prototype, I began to develop the next iteration of the app focused on the visual aesthetics of the software. How would I represent the ideas at the core of the concept through the visual aesthetic? And how could I enhance the main features to highlight the stories behind the data?

How would you represent the concept and ideas in the visual aesthetics of the app?

Some of the design explorations I went through can be seen in the sketches as well as Figma mockups illustrating some of the flows I explored.

Final Prototype

The final prototype was developed into a working prototype with an associated user manual, which explains to users how the system works. Some of the key changes and additions made to the design are outlined below.

Interface design for a data labeling application, with multiple overlapping modal windows

Using lots of overlapping UI elements to represent the real messiness in developing categories for a machine learning algorithm.

A UI dropdown with the option Google Street View highlighted

Screenshot of a busy web application where someone is relabeling the word construction site

(Left) Pointing towards a future iteration of the app, where users could bring their own data instead of just using google street view
(Right) Using user-drawn bounding boxes to show visualize existing labels

Web app showing an image from google street view, covered in handrawn shapes

Web app showing a network diagram over the top of the entire interface

(Left) Allowing users to draw bounding boxes freehand instead of resizeable rectangles.
(Right) Representing the “model” as a network diagram showing the branching nature of the categories used

This resulting application represents an experimental approach to machine learning. It serves as a provocation and represents a set of critiques of contemporary machine learning’s approach to data collection.