What is AI governance — and why does AI need more governance than traditional software?

For many organisations, the first question about AI governance is reasonable: we already govern software, information security, privacy, change and operational risk — why do we need something separate for AI?

The answer is not that every AI system needs a new committee, a 40-page questionnaire or a parallel compliance bureaucracy. Traditional technology governance remains essential. But AI introduces characteristics that can make familiar controls incomplete when used on their own.

The US National Institute of Standards and Technology (NIST) makes this distinction explicitly. Its AI Risk Management Framework notes that AI risks can differ from, or intensify, traditional software risks, and identifies features such as changing data, emergent behaviour, complex human-AI interaction and difficult-to-measure impacts. NIST: How AI risks differ from traditional software risks.

Start with a simple definition

AI governance is not one policy. It is the operating system around AI decision-making.

At a minimum, an organisation should be able to answer:

• What AI systems and AI-enabled products are we using?
• What is each system intended to do, and where is it actually being used?
• Who owns the business decision to use it?
• Who provided or developed the model, and what other providers sit underneath it?
• What data does it use, and what happens to that data?
• How much autonomy does the system have?
• What decisions or people can it affect?
• What testing and evidence support the claims we are making about it?
• Where is human judgement required?
• What happens when the model, supplier, data, purpose or regulation changes?

If those answers exist only in separate procurement files, privacy assessments, architecture diagrams, model cards and the memory of individual employees, the organisation has controls — but not yet a coherent AI governance record.

Why ordinary software governance is not enough on its own

1. AI behaviour can be probabilistic rather than fixed

Traditional software is often tested against defined rules: given a particular input and state, the expected output is known. Many AI systems behave probabilistically. The same prompt can produce different outputs, a model can perform differently across populations or contexts, and acceptable performance has to be understood statistically rather than as a simple pass/fail.

That changes the governance question from only “does the software work?” to “how well does it work, for whom, in which context, within what limits, and what happens when it is wrong?”

2. Data is part of the behaviour, not just an input

In AI, training, validation, retrieval and operational data can materially shape system behaviour. Data quality, representativeness, provenance, lawful use and drift therefore become governance concerns as well as engineering or privacy concerns.

NIST highlights the importance of data and context in AI risk. The EU AI Act likewise places specific data-governance requirements on certain high-risk systems and treats risk management as a continuous lifecycle process rather than a one-time software release activity. EU Artificial Intelligence Act.

3. A system can change without your code changing

An organisation may integrate an external model through an API and make no change to its own application, while the provider changes the model behind that API. A software-as-a-service product can add an AI feature after procurement. A retrieval source can change. A prompt, system instruction or guardrail can be updated. The risk position can therefore move even when the traditional application release process has not.

AI governance needs to define which changes are material and which should trigger review, retesting or reassessment.

4. The supply chain is often deeper than the vendor you bought from

An AI application may involve an application vendor, a foundation-model provider, a cloud platform, retrieval sources, safety tools and client configuration. Responsibility and evidence do not sit in one place.

Governance therefore needs to distinguish what the organisation controls itself, what it relies on a supplier to do, what evidence the supplier can provide, and what limitations must be accepted or mitigated.

5. Human oversight has to be designed, not merely declared

“Human in the loop” is not a control description. A meaningful oversight control identifies who the person is, what they can see, whether they understand the system's limits, what authority they have to disagree, when they must intervene and what evidence shows that this happens in practice.

The EU AI Act's requirements for high-risk AI systems make that concrete: human overseers should be able to understand relevant capabilities and limitations, detect anomalies, avoid automation bias, interpret outputs and, where appropriate, disregard, override or stop the system. That is substantially more specific than adding a manual approval box to a workflow.

6. AI risk is socio-technical

An AI system's effect depends not only on code but on how people use it, the incentives around it, the population it affects and the decision process into which it is inserted. NIST describes AI systems as socio-technical for precisely this reason.

A technically accurate system can still create poor outcomes if users over-rely on it, if it is used outside its intended context, if affected people cannot challenge decisions, or if accountability becomes blurred between the machine, the operator and the supplier.

So what does AI governance add?

Good AI governance does not replace software engineering, cybersecurity, privacy, procurement, legal review or enterprise risk management. It connects them around the AI system and its context of use.

That generally requires five things:

1. Inventory and context: know what the system is, what it does, who provides it, who owns it and who it can affect.
2. Risk-based routing: do not govern a low-impact drafting assistant exactly like a system influencing employment, credit, healthcare or safety.
3. Defined human decisions: specify who can approve, reject, condition or escalate use.
4. Evidence: keep the policies, test results, supplier material, logs, approvals and other records that support the governance conclusion.
5. Lifecycle review: revisit the decision when something material changes.

Different jurisdictions, the same operational problem

The legal and regulatory approaches are not identical. The EU has enacted the AI Act with risk-based obligations. The US NIST AI RMF is voluntary and organised around Govern, Map, Measure and Manage. The UK has used cross-sector principles including safety, transparency, fairness, accountability and contestability, applied through existing regulators. India has developed responsible-AI policy work alongside its broader digital and data-protection framework.

For a multinational organisation, the answer should not be four disconnected AI inventories and four versions of the truth. The stronger operating model is to establish a governed factual record for each AI system, then map the applicable obligations and evidence requirements across jurisdictions and frameworks.

Governance should be proportionate

There is a danger that AI governance becomes so heavy that employees route around it. That is not good governance either.

A low-impact internal summarisation tool should not automatically go through the same process as an AI system making or materially influencing a consequential decision. The governance process should become more demanding as the context, autonomy, sensitivity, regulatory exposure and potential impact increase.

The objective is not maximum documentation. It is a defensible decision with enough evidence for the risk involved.

The practical distinction

Traditional software governance asks important questions about requirements, security, testing, release and change. AI governance keeps those questions and adds another layer:

That is why AI governance is required. Not because AI deserves bureaucracy, but because the decision surface is wider than the software itself.

Primary sources

• NIST — Artificial Intelligence Risk Management Framework
• NIST — How AI Risks Differ from Traditional Software Risks
• European Union — Regulation (EU) 2024/1689 (Artificial Intelligence Act)
• UK Government — A pro-innovation approach to AI regulation
• NITI Aayog — Responsible AI for All

This article is general information about AI governance. It is not legal advice, certification guidance or a statement that every cited requirement applies to every AI system.