Architecture Agent
Project Type: Industry Sponsored Project (HCDE 517 - Usability Studies)
Client: Actual AI
Timeline: Jan 2026 - Mar 2026
Team: J.P Nguyen, Elnaz Mille, Sara Chong, Nimesh Mohanakrishnan
My Role: Client POC, Test Moderator, Design Recommendations
Overview
Actual AI is a platform that helps engineering teams manage architecture and code quality in an era where both humans and AI agents are writing code. We partnered with their team through HCDE 517 to run a moderated usability study on one feature: the Architecture Agent, which surfaces and applies Architecture Decision Records (ADRs) across a codebase.
Over five sessions with practicing software engineers and delivery managers, we found that people could generally follow the interface's structure, but consistently lost track of what the system had actually done at the moments that mattered most: right after accepting a change, and while that change was syncing in the background.
This case study walks through three of our 29 findings in depth. A link to the full report is at the end.
Banner: Architecture Agent
Explanation of Architecture Agent
An ADR is a recommendation; a Context File is what makes that recommendation stick.
Before I get into what we found, it's worth explaining what architecture does and what an ADR is.
The Architecture Agent is an AI agent that crawls a codebase, understands and spots technical architecture patterns, and proposes Architecture Decision Records, things like "adopt a consistent testing convention" or "standardize how environment configuration is handled." Architecture Decision Records (ADRs) are the architecture recommendations generated by the architecture agent.
In the product, a user looks at each ADR and accepts, rejects, or archives it. Once someone accepts an ADR, it still has to make its way into the codebase's Context Files. These are the reference docs that AI coding tools (Claude, Cursor, Kiro, you name it) actually read before they touch any code, so that whatever gets built next follows the same rules. Getting an ADR into a Context File means the system generates a pull request, and a human still has to go review and merge it on GitHub.
Architecture Agent Flow
Problem
Actual AI needed to know if the Architecture Agent's recommendations were actually usable, not just automated.
The team didn't know whether developers could actually understand what the AI was recommending or confidently apply those recommendations without guidance. Our usability study focused on answering two questions:
Discover, Interpret, Act:
Can users find, understand, and act on ADRs without guidance?Evaluate:
How do users decide whether to accept, reject, or revert a decision, and what information do they lean on most to make that call?
Initial Call with Actual AI CEO and Designer
Study Design
Five 60-minute sessions, one shared test repo, and a research team that rotated roles every time.
Participants
Format
Data Collected
5 engineers & engineering managers
60-min remote sessions
Time on task
2–20 yrs experience
Zoom, recorded w/ consent
Likert ratings (ease, trust)
Recruited via screener survey
Think-aloud protocol
Interaction counts
-
Shared test org + sample repo
Post-task debrief
Throughout each session, we collected qualitative observations alongside quantitative metrics including task completion time, interaction counts, ease-of-use ratings, and participant confidence.
As part of the research team, I rotated between moderator, note-taker, and timekeeper across sessions and independently moderated one complete usability session.
Team working on drafting study plan
Answering the Research Questions
Three moments where the interface said one thing and the system quietly did another.
Finding 1:
Developers could accept AI recommendations (ADRs), but couldn't confidently complete the workflow.
Every participant successfully located and accepted an Architecture Decision Record. However, accepting the recommendation created a second required step: synchronizing Context Files. Because this status appeared at the bottom of the page, participants interpreted the confirmation message as the end of the workflow and missed the remaining action entirely. Rather than failing to perform an interaction, participants failed to understand the system's current state.
This finding directly answered the first research question. Participants could discover recommendations and act on them, but they could not reliably complete the workflow because the interface failed to communicate that another action was still required. The usability issue wasn't interaction — it was workflow visibility.
Finding Evidence
Evidence
5 of 5 participants assumed accepting the ADR completed the task.
Task completion time ranged from 3:18 to 12:08, depending on how quickly participants discovered the unsynchronized Context Files.
Quotes:
"I feel like I've completed the test."
— Participant 3"I would expect some sort of either a notification or some sort of progress indicator... to indicate the state of it."
— Participant 2
Design Recommendation
Based on our usability findings, we recommended making the synchronization step visible within the primary workflow so users could clearly understand what remained after accepting an ADR. Specifically, we proposed:
Surface Context File synchronization immediately after accepting an ADR.
Clearly communicate synchronization status and required next steps through persistent workflow feedback.
Production Outcome
Following the study, Actual AI removed the manual Context File synchronization step from the workflow. After developers accept an ADR, Context Files are now generated automatically in the background, allowing users to review the resulting pull request directly in their codebase before merging. By eliminating an entire step, the updated workflow reduces the ambiguity observed during testing and makes architectural decisions feel complete once accepted.
Production Outcome
Finding 2:
Developers struggled to build a mental model of the Architecture Agent before completing tasks.
Before participants could evaluate Architecture Decision Records, they first had to understand how the Architecture Agent itself worked. Across sessions, participants spent considerable time exploring the interface, asking clarifying questions, and trying to infer relationships between Repositories, Categories, Decisions, Context Files, and pull requests. Rather than learning through the interface itself, participants relied on the moderator's explanations to understand the workflow.
This wasn't simply unfamiliarity with a new product. The Architecture Agent introduces concepts—such as ADRs and Context Files—that are unique to Actual AI and have no established mental model for most developers. Without guidance, participants entered the tasks with an incomplete understanding of how the system was organized. That increased cognitive load before meaningful work could begin.
Finding Evidence
Evidence
2 of 5 participants explicitly identified terminology as an overall usability issue during the post-test debrief.
Ease ratings ranged from 2–5/5. The participants giving the lowest rating (2–3/5) specifically attributed it to terminology.
Quotes:
"Took a bit to make the link between 'Decision' and recommendations."
— P4
"Like, tools that, as you first open up a new page, they can walk you through how to use the experience."
— P2
Design Recommendation
Rather than introducing every feature upfront, we recommended progressive onboarding that helps developers form a mental model while they explore. Specifically:
Introduce a first-run walkthrough explaining the Architecture Agent workflow.
Provide contextual explanations for concepts such as ADRs and Context Files.
Use progressive disclosure so information appears when it becomes relevant.
Production Outcome
Based on my observations, I provided 2 recommendations:
Move Context File status directly into the primary workflow, making synchronization the obvious next step after accepting an ADR.
Remove the context files entirely and make the ADR acceptance an automated experience.
Production Outcome