01 · PHYSICAL TOPOLOGY — HARDWARE

graph LR classDef default fill:#1a1d24,stroke:#2b303b,stroke-width:1px,color:#e8eaed,rx:8px,ry:8px classDef pi fill:#1a1d24,stroke:#5c7cfa,stroke-width:2px,color:#e8eaed,rx:8px,ry:8px classDef ext fill:#1a1d24,stroke:#fcc419,stroke-width:1px,color:#fcc419,rx:8px,ry:8px classDef user fill:#1a1d24,stroke:#20c997,stroke-width:1px,color:#20c997,rx:8px,ry:8px cam["Camera Module
5 MP · CSI ribbon"] screen["Touchscreen
800x480 · face kiosk"] mic["Speaker & Mic
Bluetooth Audio
profile locked"] psu["Power Supply"] pi["CORE COMPUTE MODULE

Custom Logic Board · Fast Storage
Local Neural Network Inference
Wi-Fi 6 · BT 5.0

Custom case · fans + RGB mood LEDs
Sensor Array"]:::pi arm["Arm — 4 DOF
Custom firmware
4x Precision Servos"] user["User (person)
taps face · talks to mic
hears via speaker"]:::user cloud["CLOUD SERVICES
Speech Processing
Vision Intelligence
Secure Tunnel"]:::ext cam -- CSI-2 --> pi screen -- DSI --> pi mic -- BT · PipeWire --> pi psu --> pi pi -- USB · CH340 · /dev/ttyUSB0 --> arm pi -- Wi-Fi · WebRTC + HTTPS --> cloud cloud --> user user -.-> mic
LAYER 1 · REFLEX

Deterministic scripts

Keyword→gesture, greeting wave, fast local vision labels (~125 ms), mood-LED map. Instant, free, works with zero connectivity.

LAYER 2 · COGNITION

Cloud Intelligence

Advanced conversational models and high-fidelity vision processing. The personality lives here — locked, de-hardcoded (works for anyone), live date injected. Canonical config in the cloud.

LAYER 3 · RESILIENCE

Local fallback

Vision routes to fallback cloud → offline on-device models. Wi-Fi watchdog + tunnel re-registration heal the links.

03 · SIGNAL PATHS

SIGHT "What do you see?" — server tool look
graph LR classDef default fill:#1a1d24,stroke:#2b303b,stroke-width:1px,color:#e8eaed,rx:4px,ry:4px v["Cloud tool-call"] -->|tunnel| p["POST /look"] --> c["capture_clean()"] --> llm["Vision Model ~2-5 s"] --> r["result → LLM speaks it"] subgraph capture_clean logic led["RGB LEDs off"] -.-> bl["backlight 25"] -.-> wait["1.0 s"] -.-> crop["crop 10% vignette"] -.-> gray["30% gray-world"] -.-> rest["restore lights"] end c -.-> led
MOTION Gestures — client tool + deterministic reflex
graph LR classDef default fill:#1a1d24,stroke:#2b303b,stroke-width:1px,color:#e8eaed,rx:4px,ry:4px l["LLM picks gesture"] --> evt["client event"] --> face["face.html fetch localhost"] --> g["GET /gesture?type=..."]

04 · DATA CONTRACTS — SCHEMA

VAPI server-tool webhook → Pi
{ // POST /look /search /volume
  "message": {
    "toolCalls": [{
      "id": "toolu_01X...",
      "function": {
        "name": "look"
      }
    }]
  }
}
Pi → VAPI tool result
{ // blocking: LLM waits for this
  "results": [{
    "toolCallId": "toolu_01X...",
    "result": "A gray-and-white cat..."
  }]
}

06 · KNOWN LIMITS — BY DESIGN HONESTY

8 kHz Bluetooth mic
  • HFP narrowband garbles names; spell-it-out protocol + keyterm boost mitigate
  • Best next upgrade: USB speakerphone puck → hi-fi A2DP returns, buttons work, STT jumps in quality
Dim-room color fidelity
  • ov5647 lens shading (red vignette) + low light mute colors; crop + mild correction + light-source blanking recover object IDs
  • Screen + case LEDs are in-scene light sources — both blanked during captures
Conversation has no local fallback
  • Realtime voice needs cloud LLM (Gemma is ~159 s); quota exhaustion = failed turn — solved by paid Claude credits
Arm: 3 of 4 servos
  • Elbow servo dead (runaway casualty) — replacement pending; all gestures + pick-and-place IK stack ready in deploy/arm/ when it arrives