HTML explainer artifact

TipTour macOS

Source: https://github.com/milind-soni/tiptour-macos
Generated: 2026-05-22T06:48:30Z

What this is

TipTour is a native macOS menu-bar voice companion for guided computer use: ask how to do something on the Mac, then watch a cursor/overlay guide or execute the next action. It is not just a static product-tour app; the repo is a SwiftUI/AppKit macOS assistant with Gemini Live voice, optional screen streaming, local UI perception, and a localhost harness for outside orchestrators.

Repo/source map

TipTour/App        menu-bar lifecycle + CompanionManager state
TipTour/UI         floating panel, cursor cat/overlay, detection visuals
TipTour/Voice      Gemini Live WebSocket, PCM audio, transcripts, tool calls
TipTour/Workflow   single-action plans, grounding, runner, validation pauses
TipTour/Perception AX tree, browser coordinates, OCR/YOLO/local target cache
TipTour/Core       TipTourEngine facade for external callers
TipTour/Harnesses  localhost HTTP harness at 127.0.0.1:19474
harnesses/hermes-mcp  MCP bridge exposing TipTour to Hermes
worker/            optional Cloudflare Worker proxy for Gemini key handling

How it works

• User holds push-to-talk; Gemini Live gets voice plus optional JPEG screen frames.
• Gemini can call one tool: submit_workflow_plan(goal, app, steps).
• The app clamps the model to one action per turn, then grounds the action through AX/local perception rather than trusting raw model coordinates.
• WorkflowRunner executes cursor flight/click guidance, pauses if the user switches apps or a modal appears, and validates visible state change before advancing.
• External systems can call the local harness: observe, list targets, plan/execute one grounded action, inspect recent action history.

Maturity signals

• Recent active commits as of capture; ~347 stars / 61 forks; Swift-dominant codebase with small TypeScript/Python support pieces.
• No GitHub releases yet, no README surfaced through the GitHub README API, and license is NOASSERTION; treat as promising source/architecture reference, not polished install-and-run software.
• The project already contains agent-facing docs (AGENTS.md, docs/hermes-harness.md, docs/source-layout.md), which makes it unusually useful as a study target for Dab/Mac automation.

Install/use path to inspect later

1. Clone the repo on a Mac with Xcode.

2. Open tiptour-macos.xcodeproj and build the app.

3. Provide a Gemini API key in the panel; source builds store it in macOS Keychain.

4. Grant Accessibility/screen permissions, then test push-to-talk and one-step action guidance.

5. For Dab/Hermes integration experiments, run TipTour and connect to http://127.0.0.1:19474; optionally configure the included Hermes MCP bridge.

Why Ananth/Dab should care

• It is a concrete blueprint for “Dab can see/control my Mac” without embedding the whole assistant inside a desktop app.
• The best pattern to steal is the boundary: native macOS app owns perception/action safety; Hermes/Dab remains the orchestrator that reasons, remembers, and decides next steps.
• The one-action loop is the right reliability shape for assistant-controlled UI: observe → choose one grounded target → act → validate → repeat.
• The harness/MCP bridge is directly relevant to future Mac/phone/Pi control surfaces and to a safer computer-use capability for Mission Control.

Open questions before adopting

• Does the repo build cleanly on Ananth’s Mac/Xcode version?
• Is Gemini Live the desired backend, or should the harness be model-agnostic?
• What privacy mode should be default for screenshot streaming?
• Is the license permissive enough for reuse, or should this stay as architecture inspiration only?