SCREEN EVIDENCE — HIGHLIGHTED
SCREEN STRUCTURED STATE
Screenshots show pixels.Agents need state.
Prots turns screen evidence, application data, and user actions into persistent structured state and explicit changes for AI agents.
Designed for lower observation latency, fewer tokens, and smaller context updates.
Local-first prototype • Source-grounded state • Explicit deltas
72-SECOND DEMO
See the screen become structured state.
The left side shows normal desktop activity. The right side visualizes the structured state Prots derives from it.
PROTS04 / DEMO
01:12
72-SECOND DEMO
Prots demonstration
A silent, captioned walkthrough of screen evidence becoming structured state.
Open the MP4 directlyPROTS STATE — VISUALIZED
The complete demo is silent and designed to remain understandable without audio.
The right pane is a visualization for humans. Agents receive the underlying structured events—not another screenshot.
- Spatial structure
- Source-grounded text
- Meaningful deltas
- Evidence provenance
WHAT THE AGENT RECEIVES
A visible change for humans.A structured event for agents.
The highlighted screenshot and illustrative JSON connect one visible change to the structured event an agent can receive.
What the agent receives
{}
"app": "Gmail","change": "fields_updated","region": "Bottom right 'New Email' region","fields": {"Email_address": "[email protected]","Subject": "Prots demo"}◆AI AGENT
A DIFFERENT OBSERVATION MODEL
From pixels to continuous structured state.
Most computer-use agents see pixels. From those pixels, they must reconstruct what exists and what happened.
Prots maintains a continuous state stream built from screen evidence, application data, spatial structure, user actions, and temporal changes.
- 01Observe — Gather screen and application evidence
- 02Structure — Resolve text, controls, and geometry
- 03Maintain state — Preserve context between observations
- 04Emit changes — Expose affected updates explicitly
Built for computer-use agents, browser agents, desktop automation, agent evaluation, and multimodal interaction systems.
WORKFLOW EVALUATION
Evaluate Prots on a real workflow.
Bring a browser or desktop workflow relevant to your agent. I will run it through the current Prots prototype and show how it can be represented as persistent state and explicit changes rather than repeated screenshots.
No installation is required for the initial evaluation.
Evaluate a workflow- Structured state and detected changes
- Observation, latency, token, and context-efficiency analysis
- Limitations revealed by the workflow
HOW PROTS FITS
One observation layer. Many evidence sources. Many agents.
Current prototype inputs and outputs are labeled separately from adapter and interface directions.
Evidence sources
Screen frames
CURRENTCaptured local pixels as spatial evidence.
Windows source elements
CURRENTAccessibility-derived controls and text.
Visual layout
CURRENTNative layout regions and geometry.
User-input events
CURRENTActions aligned with observed changes.
Browser DOM
PLANNEDFirst-class browser structure adapter.
OCR fallback
PLANNEDAdditional fallback evidence path.
Terminal, IDE + telemetry
PLANNEDSpecialized developer-work adapters.
Prots observation layer
PROTS
- Normalize
- Ground
- Validate coordinates
- Structure
- Apply privacy policy
- Compute changes
- Preserve provenance
- Expose uncertainty
Agent interfaces
Structured state
CURRENTInspectable state derived from evidence.
Delta events
CURRENTMeaningful affected-region changes.
Observation output
CURRENTAgent-readable timestream text and artifacts.
SDK + local API
PLANNEDProductized developer interface.
MCP
PLANNEDA future agent-tool interface direction.
Adapter ecosystem
PLANNEDDeveloper-defined evidence sources.
REPRESENTATION MATTERS
Screenshots vs. Prots state
| Comparison dimension | SCREENSHOT LOOP | PROTS STATE |
|---|---|---|
| Representation | ▧Pixels or image descriptions | ⌗Structured, source-grounded state |
| State over time | ▧Reconstructed repeatedly | ⌗Persistent between observations |
| Change handling | ▧Compare or reinterpret full frames | ⌗Explicit affected changes |
| Efficiency goal | ▧Repeated screen interpretation | ⌗Designed for lower latency, fewer tokens, and smaller context updates |
WHY OBSERVATION INFRASTRUCTURE
One observation layer between computer activity and AI.
If computer-use AI becomes continuous, each system will need a reliable way to understand changing interfaces, preserve context, and verify what happened. Prots is building that layer.
Continuous assistance
Recognize relevant changes without requiring the user to reconstruct the situation.
Cross-application context
Follow approved context across browser, email, editor, terminal, and desktop tools.
Action verification
Inspect explicit state changes after clicking, typing, or navigating.
Private local context
Keep processing local and expose only approved fields, changes, or events.
CURRENT STATUS
Working prototype. Larger platform direction.
A working local Windows prototype maintains structured state and explicit deltas. Broader application coverage and production hardening remain in progress.
- Working Windows prototype
- Local processing
- Persistent state + explicit deltas
- Coverage and hardening in progress
FAQ
The important distinctions.
Does Prots replace screenshots?
No. Screen pixels can remain evidence; Prots prevents downstream agents from having to reconstruct everything from full frames alone.
Does the agent receive the white reconstruction?
No. The reconstruction is a human-readable visualization of the underlying state. Agents receive structured events or state rather than a rendered image of that visualization.
What does the agent receive?
Structured state and explicit change events, with evidence and uncertainty where available.
Does it work in every application?
Not yet. Current coverage depends on available evidence. Prots is designed around multiple source adapters, explicit provenance and honest blocked or partial states.
How mature is the prototype?
It is a working local Windows prototype. Broader coverage, interfaces, and production hardening are still in progress.
BUILD WITH PROTS
Help evaluate the observation layer for computer-use AI.
I am Maciej Wajda, a mathematics student at the Technical University of Munich building Prots as an experimental infrastructure layer for AI agents.
Bring a workflow to evaluate, or get in touch to discuss a technical integration.