Saathi is a Hindi-first, voice-first personal agent for elderly and low-vision users. One button hides a system that combines speech, camera, messages, durable memory, and carefully bounded actions.
What did Priya say on WhatsApp?
I captured three messages from Priya today. The latest, at 4:20 PM, says she will call after dinner.
Read this medicine.
The right edge is cut off. Move the camera slightly left.
I can read Amlodipine, 5 milligrams. Would you like me to repeat that?
Log my BP as 128 over 78.
Saved: 128 over 78, measured now.
The design constraint
The user should not have to learn modes, prompts, app boundaries, or where a piece of information came from. He asks in ordinary speech; Saathi decides whether the answer lives in a message, a camera frame, or remembered history.
That simplicity moves complexity into reliability engineering: handling an interruption, admitting weak visual evidence, surviving a lost connection, and distinguishing the user's current truth from an old transcript.
These are the initial pilot contracts. Each crosses modalities or sessions; none is a one-shot model demo.
“When is my appointment?”
SMS and visible WhatsApp notifications become timestamped events. Saathi resolves relative dates, retrieves the relevant sender or group, and states the captured date range when it summarises.
“Read this medicine for me.”
A guided loop checks framing, blur, glare, and whether the label is cut off. It asks for one physical correction at a time, reads only high-confidence fields, and keeps the result available for follow-up questions.
“What denomination is this?”
The same camera contract handles currency: acquire enough visual evidence, answer the denomination aloud, and refuse to turn an uncertain frame into a confident financial fact.
“Log my BP as 128 over 78.”
A structured, user-scoped reading is written to the event log with its time. Later, “What was my last blood pressure?” returns the value and when it was measured—even after the service restarts.
“Stop.” · “Repeat that.” · “That was wrong.”
Interrupts and corrections are first-class events. Saathi can stop, replay, carry an incomplete request into the next turn, or ask a narrow clarification before it takes an action.
System architecture
Every observation becomes an immutable event. Memories record what happened; the world model projects what is true now. Plans and answers retain their evidence, so the caregiver can see why the agent acted.
Model providers sit behind narrow interfaces for reasoning, vision, speech, and embeddings. The durable product does not disappear when one model is replaced by another.
Microphone · CameraX · SMS receiver · Android notification listener
SQLite outbox · Signed device identity · Retry · Stable event IDs
Append-only event log · Memory · Current-truth projection · Provenance
Skill routing · Planning · Safety checks · Clarification · Explanation
Speech · Camera guidance · Recall · Permissioned device actions
Outcome · Trace · Memory · Vision · Safety · Reliability over trials
The receiver writes to the phone’s private SQLite outbox before returning.
A foreground service retries while stable IDs prevent duplicate memories.
A heartbeat reports permission, service, and queue health instead of claiming perfect coverage.
Saathi × Multivon
Component accuracy is not the same as a dependable agent. Multivon grades the completed journey: what the agent remembered, which tools it chose, whether its visual answer was grounded, and whether it stopped safely.
Explore MultivonWrong sender, wrong date, missing provenance, or a summary with no time boundary.
AgentMemoryEval · Faithfulness
Premature capture, invented visual detail, or no recovery from glare and cropping.
VQAFaithfulness · TaskCompletion
Assistant continues, repeats the wrong turn, or triggers an action after “stop.”
Conversation rules · ToolCallNecessity
Digits are transposed, a different user’s value leaks in, or restart loses the reading.
AgentMemoryEval · Exact-value checks
An event disappears, arrives twice, or stale sync health is presented as current.
Deterministic invariants · pass^k
The automated server and app contracts exist today. Visual evaluation in Multivon is experimental, and end-to-end field numbers will be published only after the real-device pilot—with sample size, conditions, failures, and confidence intervals attached.
Honest status
Automated coverage proves system logic. It does not prove a microphone, camera, notification service, or elderly user experience in the field.
The point
Saathi is where our world-model architecture meets a real human constraint: make the interface disappear, keep the evidence visible, and treat recovery as part of intelligence rather than an edge case.
