pizzaro

Mini Project · Computer-Use Harness

Pizzaro

My computer-use harness: it learns UI actions, verifies them with a human, and graduates them to programmatic replay

C#.NETWPFSQLiteAnthropic APIOllama

Overview

Pizzaro is my computer-use harness. It turns a task prompt into UI actions, and then earns the right to replay them without a model.

A planner breaks a task prompt into an ordered plan of clicks, typing, scrolls and keys, reusing actions it already knows. A human confirms or edits each step before it becomes part of a knowledge graph of actions.

Actions move from Exploring to Verifying to Trusted as they pass verified replays. Trusted actions replay programmatically using template matching and cheap pre/postchecks that need no model. When one fails during an autonomous run, it is demoted to a repair backlog with the screenshot at failure, so it can be fixed and re-baked.

Key Features

Knowledge Graph

  • Actions at three levels: atomic, procedure, job
  • Typed edges: composed of, alternative to, precedes
  • SQLite storage with embeddings for retrieval

Trust Lifecycle

  • Exploring → Verifying → Trusted
  • No-LLM pre- and postconditions
  • Pixel signatures to judge replay success

Automation

  • Screen capture and template matching
  • Input simulation
  • Repair backlog for failed actions
  • Settings tab with API usage and cost