Security and Access Control

An ML system has every attack surface of a normal service plus a few of its own: model weights, training data, feature pipelines, and the prompts and outputs users send. Most of the work is ordinary access control and data hygiene applied to new asset types, plus a short list of ML-specific threats worth designing against on purpose.

The idea

Start with the assets: training data, model artifacts and weights, the feature store, serving endpoints, and prediction logs. Then apply the standard controls:

• Least privilege. A training job reads one bucket; it does not need write access to production data.
• Short-lived credentials. Workload identity and OIDC tokens replace long-lived keys copied into notebooks and CI variables.
• Encryption in transit and at rest, with production and development data separated so a test run cannot see real customer records.
• Supply chain. Pin dependency versions and model artifact hashes, and prefer safetensors over pickle-based weights, since loading a pickle can execute arbitrary code.

Then the ML-specific threats:

• Data poisoning — an attacker influences training data or its labels. Provenance, source allowlists, and sampled review raise the cost.
• Model extraction — many queries plus confidence scores let someone approximate your model. Rate limits, quotas per account, and rounded outputs all help.
• Prompt injection — untrusted text in the model's context changes its behavior. Treat model output as untrusted input to any tool call.
• Adversarial inputs — small perturbations flip predictions in security-critical classifiers.
• Training-data leakage — memorized samples can be returned verbatim, so treat logs and outputs as potentially containing sensitive data.

Worked example

A support assistant can call a refund tool. A customer ticket contains text instructing the model to ignore its instructions and refund a specific order. If the model's output is passed straight to the refund tool, the attacker has an authorization bypass. Fixing this is not a prompt-engineering problem: the tool must verify the authenticated user owns the order, cap the refund amount, and check the order's eligibility. The model proposes an action; only policy code authorizes it. That separation is the single most valuable security rule for tool-using systems, because it holds no matter how the model was convinced.

Check yourself

1. Why is "the model said to do it" not a valid authorization check?
2. How does returning raw confidence scores help a model-extraction attack?
3. Why is data provenance a security concern as well as a reproducibility concern?

Key takeaways

• Models propose actions; only policy code authorizes them.
• Least privilege and short-lived credentials cover most of the practical surface.
• Poisoning, extraction, prompt injection, and leakage are the ML-specific threats to design against.