Data Governance in AI: The Pillar of Responsible AI Development

AI governance has emerged as the fundamental pillar for the responsible development of AI. Amid their accelerated business integration, implementing ethical and operational frameworks is essential to mitigate the risks inherent in deploying AI systems, explains AMITI.

The reliability of these systems depends on their training quality, explains Manuel O’Brien, Leader of the AI and Emerging Technologies Committee, AMITI. “Your AI will only be as good as the data with which you train it,” says O’Brien. For that reason, rigorous oversight and robust governance architecture should be established from the design phase.

The dominant AI paradigm now centers on Foundation Models. These Generative AI models are built on transformer architecture designed to generate data sequences. They are trained with vast multimodal datasets, including text, images, speech, and structured data, to execute a wide range of tasks.

The adoption of these models has been fast. “By 2025, nearly 70% of AI research involves models of this type,” says O’Brien. However, their adoption introduces significant risks, including accountability, inherent bias, and the difficulty of explainability, which often results in "black box" systems.

“The social and ethical challenges posed by AI are increasingly visible in public debates around privacy, bias, and accountability,” says O’Brien.

These issues have led to serious failures with severe consequences. For example, algorithmic bias became a public issue in 2020 in the United Kingdom, where an algorithm used for student grades sparked protests over an unfair system that perpetuated class biases. That same year, the Gender Shades project revealed structural biases in facial recognition. That study found these algorithms consistently showed the poorest accuracy for darker-skinned women, with error rates reaching 34.4% for IBM, 33.7% for Face++, and 20.8% for Microsoft. 

Recent models continue to struggle with veracity. A Newsguard Tech report issued in August 2025 on false information in LLM responses showed concerning rates for Inflection (56.67%), Perplexity (46.67%), Meta (40%), and Copilot (36.67%).

The formalization of AI governance could help to address these problems, with large institutions like JPMorgan Chase and global policy think tanks actively shaping standards. “AI governance has become a geopolitical discussion,” says O’Brien.

O’Brien outlines five core ethical principles organizations must embed in AI governance:

1. Explainability: Ensuring AI systems provide human-interpretable reasoning for predictions and outcomes.

2. Fairness: Guaranteeing equitable treatment of individuals and groups, with sensitivity to application context.

3. Robustness: Building resilient systems capable of handling anomalies or irregular inputs.

4. Transparency: Disclosing how models are designed, developed, and deployed.

5. Privacy: Safeguarding user data and prioritizing privacy rights throughout AI processing.

These principles are being codified into regulations, explains O’Brien. The European Union has been a pioneer with the General Data Protection Regulation (GDPR) and the AI Act. The latter applies a risk-based approach, classifying risks as:

• Unacceptable: Prohibited systems, such as real-time biometric surveillance.

• High: Systems in critical sectors (for example, recruitment or healthcare) requiring strict compliance.

• Limited: Systems with transparency obligations, such as customer service chatbots.

• Minimal: Systems like video game AI, subject to voluntary codes of conduct.

Practical implementation requires organizations to define accountability, transparency (who participates in testing), and explainability (how model answers are explained). 

“In corporate settings, AI deployment success increasingly depends on governance readiness. Transparent data handling, well-defined roles, and robust model oversight have become both ethical imperatives and competitive differentiators,” says O’Brien.