Voice in App Panels

Avi has a voice mode: users talk to it, and it talks back — while the panels they're looking at respond to speech directly. Panels built with Avi's component library (@avihq/ui — see App React UI) are voice-operable automatically: the same components that render your lists, filters, buttons, and dialogs also declare what they answer to when spoken to. There is no separate voice API to integrate — compose the standard components and your panel speaks the fleet's language.

What users can say

With your panel open in voice mode, the components you composed answer for it:

The agent also knows what the panel is showing. With a record open, "what am I looking at?" or "reply to this" work without the user reading anything out — the panel describes its current view (what kind of thing is open, its title, who it's from, how it begins) so the agent can act on "this."

Two rules hold everywhere:

• A voice command is exactly a click. Speaking a target runs the same handler as clicking it — never a parallel code path, never something the pointer couldn't do.
• Only declared things are speakable. Voice never guesses at your UI. A control you didn't build from the library (or register yourself) simply doesn't answer.

Which components carry voice

Declaring spoken rows

Lists opt their rows into voice with a small config — a noun, a spoken label, and optionally the other names people use:

<RecordBrowser
  title="Invoices"
  voice={{
    noun: "invoice",
    recordLabel: (invoice) => invoice.vendor,
    recordPhrases: (invoice) => [invoice.number],
  }}
  /* ...records, columns, openId, onOpenChange... */
/>

• noun is what rows answer to as a type — it resolves "open the third invoice" when several lists are visible, and "close the invoice" on the open record.
• noun may also be a function of the record, so one list can carry several nouns — a file tree registers folders as "folder" and files as "file", and "the second folder" counts folders only. nounAliases (a list, or a function of the record) adds the other names people use for the noun ("pr" for "pull request" — a multi-word noun's initials are understood automatically).
• recordLabel is the name a user would say. Return null for records with no speakable name — they stay clickable, but voice will never guess at them.
• recordPhrases adds the other ways people refer to a record — an email's sender, a deal's company.

DataTable takes the same config as its voice prop (rowLabel / rowPhrases), so custom list bodies get spoken rows the same way.

Free-text parameters (template phrases)

An action that needs a value the user speaks — a new name, a folder title — declares a template phrase: a phrase ending in *. The words before the * are the literal prefix; everything the user says after it is captured verbatim (casing kept, "and"s and commas kept) and handed to your invoke as its parameter.

useVoiceTargets([
  {
    id: "action:rename-note",
    label: "rename note",
    phrases: ["rename note to *", "rename note *", "rename the note to *", "call it *", "change the title to *"],
    kind: "action",
    invoke: (value) => { if (value) renameNote(value) },
  },
  {
    id: "action:new-folder",
    label: "new folder",
    phrases: ["new folder *", "create a folder called *"],
    kind: "action",
    invoke: (value) => { if (value) createFolder(value) },
  },
])

Rules: a template is only ever a template (the bare prefix never presses it — give the target a plain label or phrase for that); an empty remainder never matches; the longest prefix wins; the prefix must not be a host verb ("open *" or "close *" is refused); and the prefix is matched article-tolerant, so "rename the note to Q3 plan" and "rename note to Q3 plan" are the same ask. A one-word template prefix ("move *", "set *") never captures a host command: "move down", "set the zoom to 120" and "change to dark mode" stay the host's. A multi-word prefix ("zoom to *") is a deliberate claim and keeps its precedence. Templates are matched before plain phrases, so a plain phrase that starts with your template's prefix never reaches its button.

Setting a value on a listed item

"Mark groceries as complete", "change the tax return to priority 3", "move the second task to in progress", "make it urgent": declare ONE action with updates — the row noun and the values it can set, each with the ways it's said:

useVoiceTargets([{
  id: "update-task",
  label: "update a task's status or priority",
  kind: "action",
  updates: {
    noun: "task", // rows whose voice noun is "task" are the subjects
    values: [
      { id: "status:complete", phrases: ["complete", "completed", "done"] },
      { id: "priority:4", phrases: ["urgent", "priority 4", "priority four"] },
    ],
  },
  invoke: (param) => {
    const [rowId, valueId] = String(param).split("|"); // rowId "" = "it"/"this" (the open record)
  },
}]);

The host matches only when a verb (mark, set, change, update, make, move, switch, put), a listed row of that noun, and a declared value are ALL said — the row resolved by the same forgiving matcher as "open the tax one" (ordinals, punctuation, compounds, word forms, one misheard word). Anything short of that — "update the notes of this task with the latest…" — goes to the agent untouched. Two rows that both fit is a spoken refusal, never a guess.

An action can bring its own verbs with updates.verbs, which replace the default set. With them, the value can also come first, linked to the row by "to", "on" or "from": "add the errands label to groceries", "remove the finance tag from the second task". Tasks declares its label actions this way:

{
  id: "add-label",
  label: "add a label to a task",
  kind: "action",
  updates: {
    noun: "task",
    verbs: ["add", "put", "apply", "attach"],
    values: labels.map((l) => ({ id: l.id, phrases: [`${l.name} label`, `${l.name} tag`] })),
  },
  invoke: (param) => { /* "<rowId>|<labelId>" */ },
}

Keep the value phrases specific when the verb is common. "Add" plus a bare label name would also catch "add a task to the list", so Tasks makes "add" and "remove" say "label" or "tag", and keeps bare names for its own "tag" and "untag" verbs. An action takes up to 64 values. Verbs the host owns ("open", "show", "go to" and the like) are dropped.

Collapsible blocks

A block that opens in place — a message in a thread, a folder in a tree — is an item like any row, with one extra flag: collapsible: true. The fold verbs then reach it ("expand the second message", "collapse the one from Jane", "fold up the latest"), and your invoke receives the fold as its parameter — "collapse" to close, "expand" to open; a plain open ("open the second message") passes nothing and should open it. Keep a spoken open idempotent: an already-open block stays open.

useVoiceTargets(
  messages.map((m) => ({
    id: `message:${m.id}`,
    label: `${m.sender} — ${m.date}`,
    phrases: [m.sender, `the one from ${m.sender}`],
    noun: "message",
    nounAliases: ["email"],
    collapsible: true,
    kind: "item",
    invoke: (fold) => (fold === "collapse" ? collapse(m.id) : expand(m.id)),
  })),
)

Only foldable blocks answer the fold verbs — a list row never does — so an alias shared with another list is safe: beside inbox rows (noun "email"), "open the second email" still means the rows (an item's own noun outranks another item's alias), while "expand the second email" counts the messages. Fold verbs are not reserved: a panel that folds its own things ("expand *" over a file tree, "collapse all" over a thread) keeps declaring them, and a declared action still answers by its object ("unfold everything" presses an "expand everything" action).

Custom targets and screen descriptions

For affordances no component carries, two hooks from @avihq/ui/panel complete the surface:

• useVoiceTargets([...]) registers your own spoken targets — an id, a label, extra phrases, and an invoke that must be the same code path as clicking. Declare as many phrases as an action deserves; the library handles the bookkeeping.
• A dialog that waits for one value (a name, a title) marks that field's parameterized action dictation: true. While the dialog is open, speech that matches nothing else becomes the action's param — so a user can just say the name instead of a phrase you had to predict. It only takes effect inside an open dialog, only the innermost dialog's field takes it, and the action must fill the field, not submit it.
• useVoiceScreen({...}) describes what the current view is, as structured fields — kind ("list", "record", or "draft"), noun, title, from, a short snippet of how the content begins — so the agent can answer "what am I looking at" and act on "this" without a tool call. RecordBrowser publishes a baseline for you; add your own only for richer inner views (an open reader, a draft editor) — the innermost view wins.

useVoiceScreen({
  kind: "draft",
  noun: "email",
  title: subject,
  from: recipients.join(", "),
  snippet: body.slice(0, 200),
  editing: true,
});

Making your app more voice-capable

• Cast a generous phrase net. Declare every natural way to say each action — "send it", "send the email", "send this". An unmatched phrasing is a failed command; extra aliases cost nothing.
• Give lists and rows good nouns. "Email", "event", "invoice" — nouns are how ordinals resolve and how "close the email" finds its target.
• Add the other names. People say senders, companies, and participants, not subjects — a row titled "Quarterly invoice" from Stripe should also answer to "Stripe" (recordPhrases).
• Keep labels speakable. A label is what the user must say — prefer the natural spoken name ("Q3 report" beats "q3-report.pdf"). Matching tolerates punctuation ("q3 report v2" finds "q3-report_v2.pdf"), but when a label has to stay file-like, also add the speakable stem ("q3-report_v2") as a phrase.
• Describe the screen, with a snippet. Fill useVoiceScreen's snippet and from — the agent then knows what the user is reading and can act on "this" immediately.
• Keep affordances the same in every mode. A voice user doesn't know whether the panel is in read or edit mode — mount the same spoken affordances (especially delete) in both, or a spoken command will behave differently depending on invisible state.
• Confirm destructive actions, always. Route deletes through DeleteArmedButton/ConfirmDialog so the destructive step only ever happens inside the dialog's exclusive scope — that discipline is exactly what makes a bare "yes" safe.

Telling the agent what is selected

It is automatic: whatever the person highlights inside a selection scope reaches the agent with their next message, typed or spoken, as part of the screen. The panel's voice bridge watches the selection and adds it to the open view's screen state (selection, the passage around it as selectionContext, or a selected image as selectionImage); the backend tells the agent that "this" means exactly that text.

• Mark where reading happens. Put data-selection-scope on the content area — never the toolbar, list rows or crumbs (a selection there is not what the person is talking about). The library Editor already marks its own content, so an app built on it needs nothing.
• Say where, when you know. A view that can name the place ("lines 41–43", "page 2") reports the selection itself and wins over the automatic one: spread useSelectionState() into useVoiceScreen with a selectionWhere. The Code app's readers do this for their line selections.
• A renderer that keeps its selection off the DOM (it draws its own highlight) reports it with reportSelection(state) — Drive's Word editor does.