2019 – 2020

Virtual Device System Architecture

A logic model for programming connected tools at scale.

Tulip Interfaces Enterprise Governance & Edge ComputingProduct Design Lead

Problem

The user experience was suffering from technical debt stemming from how physical devices were modeled in the software. Conversely, how physical devices were modeled in the software was limiting the user's ability to manage fleets of devices.

Goal

Streamline existing software objects to enable process engineers to program the required devices at a station. Create a mental model and workflow that makes the system architecture easy to learn and set-up.

An engineer in safety glasses and a lab coat drives a screw into a part with a tethered electric screwdriver at a workbench, under a monitor, a camera on an arm, a light strip and wall-mounted gateway boxes.

Architecture Audit

To scale the platform quickly, each piece of a "device" was historically built and owned by a different team. Over time, configuring devices becomes confusing and complex.

To kick-off the project, I ran a cross-functional workshop to understand everyone's perspective (problem definition and solution ideas) when it came to feature enablement. This would help me align stakeholder concerns to push the project through when it came to making engineering and design trade-offs downstream.

A map of the internal challenges of connected devices, from the word "Devices" through five primary factors, the technical situation and the user experience they lead to, down to internal effects such as longer deployments and sales cycles and their impact on the SWAT, Sales and C-Team teams, with proposed solutions marked along the way.

Problem Architecture Map

Cross-functional alignment requires explicit documentation to drive conversations forward effectively (and efficiently). Using the existing and proposed solutions identified during the workshop, I decided to construct a map of the problem space. The goal was for everyone to see where their solutions were located in relation to a problem architecture so that we could identify the most promising course of action. I started with the various internal definitions of a device and mapped down to their impact on internal teams, unfolding the technical and user challenges in-between. By making the endpoint the impact to internal teams, groundwork was also laid early on to be further decomposed into objectives and key results (OKRs) as the project concretized.

Learning 1

Each team had a different definition of a device.

Learning 2

Broadly construed, devices were defined in different ways within the platform, leading to a fragmented user experience.

Learning 3

The IT archetype needs to be strategically managed. There are some tensions here as devices are managed by OT but present a large threat surface which is IT's responsibility to manage. The working intersection between OT and IT is still evolving within manufacturing.

Card Sort

To learn more about how each internal team defined devices, I ran eight rounds of open and closed card sorts to generate more specific mental models. This forced the participants to concretize and work through their models instead of giving me unanchored facts (and opinions).

Head of Product's Card Sort (Left)

The results successfully reflected how each person interacted with the concept of devices in ways that were reflective of their job within the organization. For example, our Head of Product categorized devices according to how form related to function because he was thinking of how the user would assume an interaction with the platform. Meanwhile, our IoT engineer thought of devices in terms of how they "behaved," or how their data needed to be configured and managed in relation to the Tulip platform.

A card sort of device photos under device behavior, from sensors and scanners to a stack light, controllers and gateways, grouped along the bottom by streaming data, discrete data and data collectors, and by how much configuration they need.

IoT Engineer Card Sort (Above)

Learning 4

Network protocols are tied to devices and are the first touchpoint to set-up, but only 1 stakeholder mentioned them explicitly as part of a "device." They're also the part of devices that IT cares about most.

Decomposition 2x

Having gathered the technical requirements and challenges strewn across various teams, it was time to synthesize. The first pass at a device architecture took all our devices, internally and externally packaged and manufactured, and positioned them at various levels of abstractions from the network level down to the computational level.

Device trees across four levels of abstraction, cloud server, internet, station and environment, each broken down from general context and purpose through network protocol, device config and function types and information use context to data type.

Device Architecture v.1

The first pass at a device architecture took all our devices, internally and externally packaged and manufactured, and positioned them at various levels of abstractions from the network level down to the computational level.

Learning 5

As a part of this exercise, I identified the need for a middle layer doing conversions between different units. For example, a user would want the "weight" for a part, but "weight" could be in pounds or kilograms, and may be user-defined or manufacturer-defined by default. Handling this for an operator at an admin level would reduce errors, simplify set-up, and streamline reporting, especially as factories were often located around the world and data needed to be rolled up. We often think of internationalization as the translation of language, but units as well as timezones are primitives that can also be considered to fall under the same umbrella.

What is a "camera"?

A device architecture tree from displays and controllers through gateways to five enums of device types, each broken into device config types, devices, data collection and data types, colored by a key.

Device Architecture v.2

With all the devices structured and positioned, I started to simplify the model by creating an enumerated type system of different device types. These enums were defined according to how the user wanted to use our platform, which goes beyond the use of the device itself. This ended up mirroring the relationship the device had with data which is rather intuitive in hindsight given that our clients used Tulip to digitize their operations.

Learning 6

I was unconvinced that "machine vision cameras" were differently different from "cameras." While they were developed for a specific use case in Tulip (operator guidance), my intuition was that we needed to de-couple them from use cases. While "cameras" and "machine vision camera" are similar forms, image and video data are fundamentally different from tabular data from a processing standpoint. The space of what a Tulip user could potentially do with images and videos is also fundamentally different. For lack of time, I put a pin in it. The question of image and video data still haunts me.

System Abstraction

The device architecture and enums represent how our platform needed to interact with devices to enable our core value prop of "digitization" (data processing), but they were not representative of how our users saw them. Although our users wanted to manipulate data, the majority understood its computation as arithmetic; they were not data literate in a procedural programming sense. Devices were still conceptually rooted in function and purpose: what they did and why in relation to manufacturing lines. How a device processed data was only a means to an end for our users.

This presented some challenges as the function and purpose of devices are wildly open-ended. Defining all of them in advance would be equivalent to defining all the possible products anyone would ever make in advance: a foolish task. I needed to develop a conceptual model that balanced technical requirements with functional/formal intuition to present a system image that would guide our users' understanding of the platform. Currently, users all had different mental models of the platform, impending their ability to effectively accomplish tasks and troubleshoot errors.

A conceptual model of devices in three tiers, deployment context, control device and edge device, with tool, sensor and machine at the edge, mapped below a front of house and back of house line to five enums and machines.

Device Conceptual Model

The device architecture was collapsed into a simplified conceptual model. Edge cases identified during the card sorts were included in its development as breakpoints for modular design. While I had taken a first stab at the enums, pulling the edge cases back in demonstrated that some of them had to be composable into higher order objects that mapped back to our custom kits packaged for common manufacturing use cases. While the conceptual model was initially developed with explainability to the user in mind, it also ended up helping our internal teams map strategy back to technical product requirements.

This conceptual model also took a first pass at the product terminologies breaking down the amorphous mass of devices into smaller concepts that mapped to the platform architecture. All this would help users orient themselves to the connected world that is the Tulip platform.

Device Abstraction

After gathering and structuring all the technical requirements into a conceptual model, I could telescope in and start design!... of a Tulip device concept that would describe the device to a user and an application.

As a course of this research, it was evident how little control we had over a device. Most of the challenges were from the heterogenous ways that manufacturers had decided to set up their device configurations and defaults. This pointed to how definitions would need to be surfaced and aligned from the manufacturer, how we would have to introduce some of our own for usability and findability, and how the user should be able to edit the definitions of most parameters for the use cases they were trying to enable after some smart defaulting on our part. It also showed how we would have to continue to modularize devices into configurations that could be shared across the set.

A table defining a device's contextual, performance and information attributes, filled in for five sample devices from a barcode scanner to a tablet, with a check for whether each attribute is manufacturer-, Tulip- or user-defined.

Device Concept Definition

Along with distilling the structure of the device architecture further, I approached the concept definition as an integration problem and included all the attributes that would provide an endpoint linking a Tulip device into other parts of the platform. To start managing the complexity and provide some form of progressive disclosure downstream, I started considering how some of these "attributes" could be pulled out into their own modules. I tested the concept across multiple devices to assess their fingerprints and find any misfit scenarios.

Learning 7

Network protocols are killing me. After creating the device concept, There was still something crunchy about them, a sign that they probably needed to be pulled into their own object, especially since it was being managed by IT. My only concern was whether this would slow down OT deployment initially.

A hand sketch of an object-entity model linking network config, gateway, station, apps, device config and device config type, with notes on device types and data.

A hand sketch of the gateway object with connect, configure, use and disconnect actions, its attributes, what it sits in and holds, and notes on why each is needed.

A hand sketch of the network config object with create, edit, use and delete actions, its attributes, the gateways it holds, and questions about passwords and permissions.

An object model linking network protocol configuration, gateway, station, app, edge device instance, device configuration and device type configuration with one-to-many relationships, proposed objects dashed and existing objects solid.

Conceptual Data Model v.*

After working the network protocols a bit more and relating them to the other existing and proposed objects in our system to shake out the crunchiness, I developed an object model that covered the surface area of the device attributes identified.

An object model linking network protocol configuration, display device, Tulip player, active station, app, device, driver and data attribute with their cardinalities, annotated with notes on each relationship.

Conceptual Data Model, Solidified

After the objects and relationships are hammered out, it needs to be documented for team assessment. Certain design cases and product opinions surfaced during research that would be relevant to the engineering team are called out. For example, we can see in the relationship between displays and stations how the business bakes its strategic opinions directly into the data model. Being able to have a diagrammatic view of what is linearly represented in the code / text base opens up new collaborative avenues for discussion about structural product decisions. Proposed changes to any of these parts are understood in relation to the whole, allowing for more fluid and precise mental simulations of alternative systems for risk mitigation.

Four object definitions, network protocol configuration, device, driver and data attribute, each with its purpose, attributes and the objects in it and that it is in, linked in a row by their cardinalities.

Object Definitions, Solidified

In keeping with working at two levels of abstraction, all proposed objects are defined as well. Each object is given a purpose that is tied to a user's job-to-be-done. Designing objects explicitly around users at this stage ensure that flows and interactions are streamlined and handled in the backend for smoother frontend development and sane design evolution. With this structure in place, design thinking and agile development processes can proceed without incurring outsized engineering and design debt. While the data model is primarily a result of internal constraints and product vision, the user's conceptual model of the system is reflected in the objects' purpose, its data attributes, and relationships. These directly influence the information that can be displayed and how easily they can be updated.

Data ModelsData GovernanceIndustrial IoT