Voice in App Panels

Avi has a voice mode: users talk to it, and it talks back — while the panels they're looking at respond to speech directly. Panels built with Avi's component library (@avihq/ui — see App React UI) are voice-operable automatically: the same components that render your lists, filters, buttons, and dialogs also declare what they answer to when spoken to. There is no separate voice API to integrate — compose the standard components and your panel speaks the fleet's language.

What users can say

With your panel open in voice mode, the components you composed answer for it:

The agent also knows what the panel is showing. With a record open, "what am I looking at?" or "reply to this" work without the user reading anything out — the panel describes its current view (what kind of thing is open, its title, who it's from, how it begins) so the agent can act on "this."

Two rules hold everywhere:

• A voice command is exactly a click. Speaking a target runs the same handler as clicking it — never a parallel code path, never something the pointer couldn't do.
• Only declared things are speakable. Voice never guesses at your UI. A control you didn't build from the library (or register yourself) simply doesn't answer.

Which components carry voice

Declaring spoken rows

Lists opt their rows into voice with a small config — a noun, a spoken label, and optionally the other names people use:

<RecordBrowser
  title="Invoices"
  voice={{
    noun: "invoice",
    recordLabel: (invoice) => invoice.vendor,
    recordPhrases: (invoice) => [invoice.number],
  }}
  /* ...records, columns, openId, onOpenChange... */
/>

• noun is what rows answer to as a type — it resolves "open the third invoice" when several lists are visible, and "close the invoice" on the open record.
• recordLabel is the name a user would say. Return null for records with no speakable name — they stay clickable, but voice will never guess at them.
• recordPhrases adds the other ways people refer to a record — an email's sender, a deal's company.

DataTable takes the same config as its voice prop (rowLabel / rowPhrases), so custom list bodies get spoken rows the same way.

Custom targets and screen descriptions

For affordances no component carries, two hooks from @avihq/ui/panel complete the surface:

• useVoiceTargets([...]) registers your own spoken targets — an id, a label, extra phrases, and an invoke that must be the same code path as clicking. Declare as many phrases as an action deserves; the library handles the bookkeeping.
• useVoiceScreen({...}) describes what the current view is, as structured fields — kind ("list", "record", or "draft"), noun, title, from, a short snippet of how the content begins — so the agent can answer "what am I looking at" and act on "this" without a tool call. RecordBrowser publishes a baseline for you; add your own only for richer inner views (an open reader, a draft editor) — the innermost view wins.

useVoiceScreen({
  kind: "draft",
  noun: "email",
  title: subject,
  from: recipients.join(", "),
  snippet: body.slice(0, 200),
  editing: true,
});

Making your app more voice-capable

• Cast a generous phrase net. Declare every natural way to say each action — "send it", "send the email", "send this". An unmatched phrasing is a failed command; extra aliases cost nothing.
• Give lists and rows good nouns. "Email", "event", "invoice" — nouns are how ordinals resolve and how "close the email" finds its target.
• Add the other names. People say senders, companies, and participants, not subjects — a row titled "Quarterly invoice" from Stripe should also answer to "Stripe" (recordPhrases).
• Keep labels speakable. A label is what the user must say — prefer the natural spoken name ("Q3 report" beats "q3-report.pdf"). Matching tolerates punctuation ("q3 report v2" finds "q3-report_v2.pdf"), but when a label has to stay file-like, also add the speakable stem ("q3-report_v2") as a phrase.
• Describe the screen, with a snippet. Fill useVoiceScreen's snippet and from — the agent then knows what the user is reading and can act on "this" immediately.
• Keep affordances the same in every mode. A voice user doesn't know whether the panel is in read or edit mode — mount the same spoken affordances (especially delete) in both, or a spoken command will behave differently depending on invisible state.
• Confirm destructive actions, always. Route deletes through DeleteArmedButton/ConfirmDialog so the destructive step only ever happens inside the dialog's exclusive scope — that discipline is exactly what makes a bare "yes" safe.