OmniParser

OmniParser is a research-grade screen paRSIng tool developed by Microsoft Research to bridge the gap between raw visual interface input and structured, executable understanding. Unlike traditional automation tools that rely on accessibility APIs or DOM access, OmniParser operates purely from pixel-level screen captures — making it uniquely suited for cross-platform, zero-accessibility environments like legacy APPlications, kiOSks, or sandboxeD systems. It functions as a preprocessor for visual language models (VLMs), transfORMing unstructured screenshots into annotated, semantically rich representations that describe both layout and intent of UI elements.

What it does

OmniParser ingests a screenshot and ouTPUts a structured representation of interACTive regions — buttons, text fields, icons, slIDErs — along with natural-language descriptions of their likely function (e.g., 'CLIck to subMIT form', 'dRAG to adjust volume'). It does not execute actions itself but enables downstream VLMs like GPT-4V or GPT-4o to generate precise, region-grounded instructions such as 'click the blue 'Send' button located in the bottom-right corner'. This capability is foundational for building General-purpose GUI Agents that work across Windows, macOS, mobile emulators, or web apps without requiring source code, HTML, or OS-level hooks.

Key capabilities

• Two-stage multimodal parsing: A detection model localizes interactive UI elements; a description model generates functional semantics for each region using vision-language fine-tuning.
• benchmark-leading performance: Outperforms HTML-augmented GPT-4V on Mind2Web, suRPAsses Android-specific icon detectors on AITW, and achieves state-of-the-art results on ScreenSpot and WindowsAgentArena — all using vision-only input.
• Open training methodology: Includes two novel UI datasets — one for interactive icon detection, another for functional description — released alongside full training details to support reproducible research.
• Model-agnostic integration: designed as a modular plug-in: outputs JSON-structured bounding boxes + captions compatible with any VLM pipeline, whether hosted or self-deployed.

Who it is for

OmniParser targets AI researchers, multimodal systems engineers, and developers building autonomous desktop or cross-app agents — especially those working in constrained environments where DOM, accessibility tree, or platform SDKs are unavailable or unreliable. It’s also valuable for HCI researchers studying vision-based interaction, low-code automation designers, and educators exploring grounded VLM reasoning.

How to get started

The reference implementation is available on GitHub under an open license. Users can run inference locally using provided PyTorch weights and preprocessing scripts, or integrate the output format into existing VLM orchestration pipelines. No cloud service or API is offered — OmniParser is a self-hosted component, not a turnkey SaaS product. documentation includes Inference examples, dataset usage guides, and benchmark reproduction steps.

Pricing and access caveat

OmniParser is fully open-source and free to use, modify, and extend. There is no official hosted version, subscription, or Commercial License tier. As a research prototype, it may require engineering effort to adapt to production-scale latency, diverse DPI/resolution handling, or real-time streaming. Versioning, support, and long-term maintenance are community-driven and may vary.