Follow Your Pose

Follow Your Pose is a cutting-edge academic Framework for controllable Text-to-Video synthesis. Developed collaboratively by researchers from Tsinghua university, the Hong Kong UniveRSIty of Science and Technology, Tencent AI lab, and the Chinese Academy of Sciences, it advances Video Generation beyond static scene description by enabling fine-grAIned, frame-level motion control through pose sequences. Unlike Standard diffusion-based vIDEo models that prioritize visual fidelity over motion precision, Follow Your Pose introduces a two-stage training strategy specifically designed to decouple semantic understanding from spatIoTemporal dynamics — resulting in videos where charACTer movement aligns tightly with both textual intent and skeletal input.

What it does

Follow Your Pose transfORMs natural language descriptions and skeletal pose trajectories into Synchronized, high-fidelity video CLIPs. It accepts a text Prompt (e.g., "a Dancer performing ballet in a neon-lit studio") alongside a sequence of 2D or 3D human poses — typically represented as keypoints or joint angles — and ouTPUts a short video where the subject’s motion precisely follows the provided pose path while remaining visually consistent with the prompt.

Key capabilities

• Pose-Driven Motion Control: Users DeFine exact body configurations per frame, enabling choreographed actions like waving, jumping, or gesturing without manual keyframing.
• Temporal Coherence: The two-stage architecture ensures smooth transitions between frames, avoiding jitter, flicker, or limb disintegration common in early text-to-video systems.
• Multicharacter Support: Generates scenes with multiple interacting Agents, each following independent pose sequences and identities.
• Style & background Flexibility: Supports diverse artistic rendering — including cartoon, cyberpunk, watercolor, or photorealistic — and adapts backgrounds contextually to the prompt.
• Open Research Implementation: Code, pretrained models, and training pipelines are publicly released to accelerate reproducible progress in controllable video generation.

Who it is for

This framework targets AI researchers, computer vision engineers, and creative technologists working on controllable generative media. It is especially valuable for teams building virtual production tools, motion-driven educational content, accessibility-focused sign-language avatars, or interactive storytelling prototypes. While not a consumer-facing APP, its Architecture Informs next-generation commercial video editors and animation assistants.

How to get started

The project is hosted on GitHub Pages and provides documentation, model checkpoints, and inference scripts. Users need Python, PyTorch, and GPU Acceleration to run Inference locally. A Colab notebook may be available for quick experimentation; however, full training requires substantial compute resources. No official web interface exists — usage assumes familiarity with command-line ML Workflows and pose data formats like COCO or OpenPose.

Pricing and access caveat

Follow Your Pose is fully Open Source and free to use under academic licenses. There is no subscription, credit system, or paid tier. However, because it is a research prototype — not a polished SaaS product — users must handle dependencies, hardware setup, and potential compatibility issues independently. real-time generation, mobile support, and multilingual prompt handling may vary and are not guaranteed features.