MIDS Capstone Project Fall 2024
PolicyPulse
Team members
Problem & Motivation
The global RegTech market is valued at $15.80 billion in 2024, growing at a CAGR of 23.6%. This rapid growth reflects increasing regulatory pressures as jurisdictions worldwide enact stricter data privacy laws. Following the landmark GDPR in Europe, states like California have implemented their own privacy frameworks, and similar regulations are emerging globally. While regulatory compliance is essential, privacy policies are often lengthy and complex, making them challenging to assess for compliance. Fintechs risk hefty fines, lawsuits, and reputational damage if they fail to comply with these regulations. Law firms, on the other hand, face mounting pressure to deliver efficient and accurate compliance guidance to clients. High-profile non-compliance fines, such as €290 million against Uber and €1.2 billion against Meta, underscore the critical need for scalable, efficient compliance solutions. Recognizing this challenge, PolicyPulse delivers an AI-powered platform that simplifies privacy compliance workflows and reduces manual effort. PolicyPulse helps fintechs and law firms navigate evolving regulatory requirements with confidence.
Minimum Viable Product (MVP)
PolicyPulse was built using advanced AI technologies to automate compliance tasks, reducing costs and manual effort. Its core features include:
The global RegTech market is valued at $15.80 billion in 2024, growing at a CAGR of 23.6%. This rapid growth reflects increasing regulatory pressures as jurisdictions worldwide enact stricter data privacy laws. Following the landmark GDPR in Europe, states like California have implemented their own privacy frameworks, and similar regulations are emerging globally. While regulatory compliance is essential, privacy policies are often lengthy and complex, making them challenging to assess for compliance. Fintechs risk hefty fines, lawsuits, and reputational damage if they fail to comply with these regulations. Law firms, on the other hand, face mounting pressure to deliver efficient and accurate compliance guidance to clients. High-profile non-compliance fines, such as €290 million against Uber and €1.2 billion against Meta, underscore the critical need for scalable, efficient compliance solutions. Recognizing this challenge, PolicyPulse delivers an AI-powered platform that simplifies privacy compliance workflows and reduces manual effort. PolicyPulse helps fintechs and law firms navigate evolving regulatory requirements with confidence.
Minimum Viable Product (MVP)
PolicyPulse was built using advanced AI technologies to automate compliance tasks, reducing costs and manual effort. Its core features include:
- Q&A Chatbot: Built on Cohere’s Command R-Plus to provide detailed, GDPR article-specific answers to compliance queries. Ground truth datasets were generated using Claude and refined through SME verification for accuracy.
- Privacy Policy Summary: Utilizes GPT-4o to generate concise summaries of privacy policies, enabling organizations to quickly identify key compliance issues.
- GDPR Checklist: Powered by a policy classifier designed for future integration with a Retrieval-Augmented Generation (RAG) pipeline. Utilizes a knowledge graph to retrieve GDPR articles and evaluate privacy policies for compliance readiness.
- Policy Categorization: Fragmentation of privacy policies into meaningful categories for efficient analysis with one liner explanation for every policy category.
- Compliance Insights: Retrieval of GDPR/CCPA documentation to identify areas of non-compliance and provide actionable recommendations.
These tools automate critical privacy compliance workflows, helping fintechs avoid fines and enabling law firms to provide efficient, accurate guidance.
Key Insights
During the development of PolicyPulse, several critical learnings shaped the final product with SME collaboration to maintain context accuracy.
Key Insights
During the development of PolicyPulse, several critical learnings shaped the final product with SME collaboration to maintain context accuracy.
- Model Evaluation: Cohere’s Command R-Plus emerged as the top-performing model for Q&A tasks, balancing correctness and speed over alternatives like Llama.
- Model Improvement: Achieving incremental gains required extensive fine-tuning, highlighting the complexity of aligning AI outputs with regulatory standards.
- Inference & Integration: Real-time deployment posed significant challenges, requiring extensive debugging and infrastructure optimization to integrate RAG pipelines and APIs into a seamless workflow.
System Highlights
A key aspect of our project, ground truth dataset generation ensures accurate, reliable datasets for training and evaluation. Here are the overall components of the system:
A key aspect of our project, ground truth dataset generation ensures accurate, reliable datasets for training and evaluation. Here are the overall components of the system:
- SME Validation: Subject Matter Experts reviewed and verified generated outputs to ensure consistency, reliability, and accuracy.
- QA System: A robust question-answering system powered by Claude that provides precise, article-specific responses to compliance queries. Each response includes a main answer paragraph summarizing the query's resolution, key points for clarity, and direct references to relevant GDPR articles to ensure compliance accuracy.
- Summary System: Utilizes a MapReduce technique to process lengthy privacy policies, enabling scalable and effective summarization.
- Policy Classifier: A DeBERTa-based classifier designed to categorize policy sections into key categories such as data retention, user choice/control, and more.
- GDPR Checklist: Highlights existing and missing GDPR articles within the policy. Articles are displayed with their interconnections visualized through a knowledge graph, providing a comprehensive overview of compliance gaps.
- Evaluation Metrics: Outputs were complemented by manual inspections to validate relevance and coherence of answers and summaries.
Evaluation Insights
To ensure PolicyPulse delivers reliable outputs, we evaluated key models for Q&A and summary tasks.
To ensure PolicyPulse delivers reliable outputs, we evaluated key models for Q&A and summary tasks.
- Q&A Evaluation: Cohere Command R-Plus emerged as the top performer with a weighted accuracy of 72%, correctness of 52%, and semantic similarity of 92%. While it excelled in precision, its context recall (28%) indicates room for improvement in retrieving complete information for complex queries.
- Summary Evaluation: GPT-4o delivered the most concise and accurate summaries, achieving a ROUGE precision of 56.25%, recall of 32.79%, and F1 score of 41.43%. Manual inspections validated model outputs, ensuring relevance and accuracy while identifying improvement areas.
Mission Statement
“Unlock seamless data privacy compliance with our cutting-edge AI solutions, empowering fintechs to identify risks early, safeguard user data, and foster unwavering trust in the digital age.”
Acknowledgments
The development of PolicyPulse would not have been possible without the support of our instructors and classmates. Special thanks to Jared Maslin, Kevin Kuc, Mark Butler, and Timothy Cradle for their invaluable guidance and feedback throughout this journey.
“Unlock seamless data privacy compliance with our cutting-edge AI solutions, empowering fintechs to identify risks early, safeguard user data, and foster unwavering trust in the digital age.”
Acknowledgments
The development of PolicyPulse would not have been possible without the support of our instructors and classmates. Special thanks to Jared Maslin, Kevin Kuc, Mark Butler, and Timothy Cradle for their invaluable guidance and feedback throughout this journey.
Course
Data Science 210. Capstone , Fall 2024More Information
PolicyPulse is a tool designed for startups and legal professionals to assess privacy policies for GDPR compliance. It allows users to ask specific questions about a policy and GDPR guidelines, provides concise summaries of the policy, and offers a checklist of GDPR categories to evaluate compatibility and compliance. By simplifying policy analysis and ensuring alignment with GDPR standards, PolicyPulse helps users save time, reduce legal risks, and enhance data protection efforts.
Video
Last updated:
December 16, 2024