PortraitGen

PortraitGen is a research-driven AI video editing Framework developed by the 3D Vision Lab at the university of Science and Technology of China. It reimAGInes portrait vIDEo manipulation as a spatIoTemporal 4D problem—transfORMing 2D input videos into dynamic 4D Gaussian fields that preserve GEOmetry, APPearance, and motion coherence across frames. Unlike conventional frame-by-frame editing tools, PortrAItGen maintains structural integrity of faces and bodies while enabling expressive, controllable modifications grounded in 3D-aware representations.

What it does

PortraitGen edits portrait videos with unprecedented fidelity and consistency. It accepts raw RGB portrait videos and reconstructs them into editable 4D Gaussians—combining spatial 3D structure with temporal dynamics. This foundation allows users to modify lighting, identity, pose, clothing, and artistic style without introducing flicker, warping, or identity collapse. The system supports both text-guided instructions (e.g., 'soft studio lighting' or 'cyberpunk outfit') and image-based reference editing (e.g., applying a fashion look from a reference photo).

Key capabilities

• Multimodal editing: Seamlessly switch between text Prompts and reference images to drive changes in appearance, expression, or attire.
• Lighting-aware relighting: Adjust illumination direction, intensity, and color temperature using natural language, with physically plausible shadow casting and specular response.
• Face-perceptual preservation: LeveRAGes SMPL-X body modeling and facial landmark constraints to retain identity, micro-expressions, and anatomical plausibility during edits.
• virtual try-on: Projects gArments onto the reconstructed 4D human mesh with realistic drAPIng, occlusion handling, and material interACTion.
• style transfer & animation: Applies global stylistic transformations—including cartoon, oil-painting, or anime Aesthetics—while preserving motion continuity and lip-sync accuracy.
• High-speed rendering: Achieves up to 100 FPS ouTPUt via optimized Gaussian rasterization, suitable for iterative creative workflows.

Who it is for

PortraitGen targets researchers, VFX artists, indie filmmakers, digital fashion designers, and AI developers working on generative video. Its open architecture makes it especially valuable for teams exploring 3D-aware diffusion, neural rendering, or controllable video synthesis. While not yet a turnkey SaaS product, its modular design supports integration into larger pipelines for advertising, social media content creation, or game asset prototyping.

How to get started

The project is hosted publicly on GitHub and includes training scripts, inference Notebooks, and Sample datasets. Users need Python 3.9+, PyTorch, and CUDA-compatible GPUs. documentation assumes familiarity with 3D Reconstruction concepts; no GUI is provided—setup requires command-line execution and configuration tuning. Pretrained models are available for evaluation, but fine-tuning on custom subjects may require multi-hour training on high-end hardware.

Pricing and access caveat

PortraitGen is fully Open Source and free to use under academic and Commercial Licenses. However, it is a research prototype—not a production-ready application. real-time web deployment is not offered; all usage occurs locally or on private Cloud Infrastructure. Performance, compatibility, and feature completeness may vary depending on hardware, input quality, and scene complexity. No official support, cloud API, or subscription tier exists at this time.