Jetpack Compose Semantics: A Guide for AI Agents

Teaching AI Agents to Understand Your UI: A Practical Guide to Jetpack Compose Semantics

Introduction

If you want AI agents and test automation tools to accurately understand "what this button does" in Jetpack Compose, proper Semantics design is the key.

Compose maintains an internal "Semantics Tree" — a semantic structure used by accessibility services, UI testing frameworks, and automation AI tools like Appium, UIAutomator, and MCP. Think of it as the Compose equivalent of ARIA and the DOM on the web.

Core Semantics Attributes

contentDescription

The most fundamental attribute for giving meaning to icon-only buttons.

IconButton(
    onClick = onDelete,
    modifier = Modifier.semantics {
        contentDescription = "Delete"
    }
) {
    Icon(Icons.Default.Delete, contentDescription = null)
}

AI and accessibility tools will recognize this as "Delete button, clickable, Button role."

role

Explicitly declares the UI type for custom components. A plain Box(clickable) registers as "something clickable," but adding a role makes it clearly a Button, Switch, Checkbox, Tab, and so on.

Modifier.semantics {
    role = Role.Button
}

stateDescription

Communicates the current UI state to AI.

Modifier.semantics {
    stateDescription =
        if (subscribed) "Subscribed"
        else "Not subscribed"
}

testTag

The most important identifier for MCP, Appium, and automated UI agents. It enables stable identification without depending on coordinates, OCR, or text changes.

Modifier.testTag("login_button")

Configuring UIAutomator Integration

Because Compose doesn't create a native View hierarchy, testTag values can be invisible to UIAutomator. Add this to your root to fix it:

Scaffold(
    modifier = Modifier.semantics {
        testTagsAsResourceId = true
    }
)

This makes testTag values visible as resource IDs, allowing UIAutomator to retrieve them directly.

The Best Pattern for AI Comprehension

Combining multiple attributes is the best practice:

Button(
    onClick = login,
    modifier = Modifier
        .testTag("login_button")
        .semantics {
            contentDescription = "Log in"
            role = Role.Button
        }
) {
    Text("Log in")
}

Collapsing Complex UI into a Single Meaning

When you have complex compositions of icons, text, and badges that should appear as a single button to AI, use clearAndSetSemantics:

Modifier.clearAndSetSemantics {
    contentDescription = "Add item to cart"
    role = Role.Button
}

This lets you pass just the meaning to AI, ignoring the internal nesting and icon structure.

What Semantics Design Enables

Stable Automation

With testTag, you can automate interactions reliably without coordinates or OCR. If the testTag stays the same after a UI redesign, your automation won't break. It also handles multi-language apps cleanly.

Token Reduction

Screenshot-only approaches require image analysis, which consumes a lot of tokens. A Semantics Tree is structured data — much lighter to process.

[
  {"type": "Button", "label": "Log in"},
  {"type": "TextField", "label": "Email"},
  {"type": "TextField", "label": "Password"}
]

This is especially effective for long-running automation sessions and large-scale screen analysis, and helps prevent context overflow.

AI State Awareness

stateDescription lets AI infer the current screen state, enabling decisions like "playback is paused, so tap the play button." It makes it possible to model your screen as a state machine.

Structured Analysis via MCP

By fetching the Semantics Tree through an MCP Server and passing it to an LLM agent, you can enable full-screen semantic understanding, prediction of next actions, and state transition analysis.

Compose UI → Semantics Tree → MCP Server → LLM Agent

Semantics Attribute Reference

Summary

Compose lets you separate "how it looks" from "what it means." The essence of Semantics design is the ability to design the human-facing UI and the AI-facing semantic structure independently.

If you're using AI agents, MCP, UIAutomator, or automated E2E tests, standardizing the following pattern across your components delivers strong ROI:

Modifier
    .testTag(...)
    .semantics {
        contentDescription = ...
        role = ...
    }

Color, position, and size matter less than "what is this button for, what happens when pressed, and what's its current state" — passing that through Semantics is the core of AI-friendly UI design.