ScreenAgent

ScreenAgent is a research-grade AI workflow automation tool designed to interACT with real computer interfaces—not simulations or APIs—by interpreting visual screen content and issuing precise input commands. Developed by Jilin university’s AI Institute and the Ministry of Education’s engineering Research Center for Knowledge-Driven AI, it bridges the gap between large multimodal models and actionable desktop control. Unlike traditional RPA tools, ScreenAgent relies on end-to-end perception-action loops grounded in visual language modeling (VLM) and custom reinforcement learning environments, enabling Generalization across diverse GUI-based tasks without hard-coded selectors or scripting.

What it does

ScreenAgent watches live screen frames—captured via VNC—and reasons over them alongsIDE natural language instructions to generate executable sequences of mouse movements, CLIcks, scrolls, and keyboard inputs. It operates in closed-loop cycles: first planning a high-level task into substeps; then executing low-level actions; and finally reflecting on outcomes using visual feedback to adjust or retry. This tripartite control flow supports robust, iterative task completion—from launching APPlications and navigating browsers to filling fORMs and manAGIng files—all within unmodified desktop environments.

Key capabilities

• Screen-aware reasoning: Uses VLMs to jointly interpret screenshots and text Prompts, grounding language in pixel-level interface elements.
• VNC-integrated RL environment: Trains and deploys within a simulated yet realistic desktop sandbox, supporting safe experimentation and reproducible evaluation.
• JSON-native action ouTPUt: Generates Standardized, machine-readable command sequences (e.g., {"action":"click","x":120,"y":340}) for inteRoPErability and auditing.
• CC-Score eValuation Framework: Introduces a novel metric measuring both correctness and completeness of GUI task execution, backed by a curated dataset of annotated desktop Workflows.

Who it is for

Researchers in AI agent design, Human-Computer Interaction, and embodied intelligence will find ScreenAgent especially valuable as a testbed for vision-language grounding and desktop automation. developers building no-code automation tools or accessibility assistants may adapt its architecture for real-world UI control. While not production-ready for Enterprise IT, it serves as a foundational open-source reference for next-generation Autonomous Agents that operate directly on visual interfaces.

How to get started

The project is hosted publicly on GitHub. Users need bASIc Python proficiency and access to a Linux or macOS system with VNC server support. Setup involves cloning the repository, installing dependencies (including PyTorch and VLM backbones), configuring the VNC connection, and running inference scripts with Sample tasks. documentation is currently research-oriented and assumes familiarity with Reinforcement Learning concepts and multimodal model fine-tuning.

Pricing and access caveat

ScreenAgent is fully Open Source and free to use under its MIT-licensed codebase. There are no subscription tiers, cloud services, or commercial licensing options at this time. However, deployment requires local compute resources—including GPU Acceleration for VLM Inference—and technical setup may vary depending on target OS and display environment. No official web app or Windows installer exists; users must self-host and configure the VNC bridge manually.