Research
PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review
PrivacyPR is a new dataset of 4,131 merged GitHub pull requests from 1,469 repositories, created between June 2018 and January 2024, with LLM-generated labels for privacy relevance, fix type, and GDPR principles. It includes 1,169 high-confidence privacy fixes and 1,643 non-fix PRs, plus manually verified splits. Benchmark results show coding agents detect only 42-51% of privacy defects under generic prompts but 88-91% when explicitly asked to check privacy, with 11-15% false alarms on non-fix PRs.
Privacy-aware code reviewAI coding agentsGDPR compliance