Governance Is An Operational Asset, Not Paperwork
A CISO finds a dozen models running in production with no lineage and no procurement records. That is not an IT failure. It is a business risk. Governance artifacts should be measurable operational assets: tracked in your risk register, included in procurement evidence packs, and presented to auditors and the board.
Before you ask to "put them on the balance sheet," define what you will measure and where it lives. We recommend an auditable evidence pack (inventory changes, vendor attestations, and tabletop remediation lists) and a resilience score in your risk register as the accounting mechanism.
If You Cannot List Production Models Fast, You Do Not Have Inventory
Inventory is a live dependency map, not a checklist. Define production and an authoritative entry up front.
Definitions you must publish
- Production: models serving inference for customers or internal decisioning in an environment exposed to live data. Exclude training-only experiments until they are promoted.
- Authoritative inventory entry: a single, signed record that includes owner, environment, runtime endpoint, model hash, and linked data stores. An entry is authoritative when created by the deployment pipeline or procurement acceptance and timestamped in the central registry.
- Identity hygiene: MFA-enabled principals, role-based permissions limited to documented scopes, and automated rotation for short-lived credentials. Non-rotating perpetual tokens do not meet hygiene criteria.
Measure this, with phased targets
- Coverage: percent of production models with authoritative entries. Target: 95% for tier-1 production models within 90 days; broader coverage phased to 95% within 6 months. Full 100% is a long-term control objective where vendor capability allows.
- Freshness: median time since last inventory update for production models. Target: median <7 days for high-impact models; apply longer windows for low-impact models during ramp.
Do this now
- Minimal inventory schema (example fields): model_id, owner, environment, runtime_endpoint, model_hash, data_sources, vendor, procurement_ref, last_updated, incident_playbook_ref.
- Enforce: block deployments from CI/CD unless an authoritative entry exists or a temporary exception is recorded and expired automatically.
- Audit: weekly list of new models, changed hashes, and unresolved lineage gaps.
Identity Is The Control Plane. Permissions Define The Blast Radius.
Tokens, service accounts, and vendor clients are the attack surface. Identity controls must be measurable and contractually enforceable.
Quantify identity posture
- Identity hygiene rate: percent of model-related identities meeting MFA, least privilege, and rotation policies. Target: 100% for internal service accounts; for vendors, set phased targets (e.g., 80% within 3 months, 95% within 6 months) and require roadmap evidence for the remainder.
- Token risk metric: count of unscoped or perpetual tokens with access to production models. Target: zero for production access; where vendors cannot immediately remove legacy tokens, require compensating controls and a remediation timeline.
Practical controls to impose
- Vendor identity program: quarterly attestation of service accounts, OAuth clients, and short-lived credential use. Revoke stale credentials and require just-in-time access patterns.
- Contract clauses: require vendors to support short-lived credentials, role scoping, and cross-tenant logs to your SIEM or aggregator where technically feasible.
- Technical enforcement: identity governance for role attestation and automated revocation on contract lapse.
Tabletop Rehearsals Show If You Can Move Fast Enough
A real incident is legal, procurement, and communications as much as technical. Rehearsals reveal where authority and evidence are missing.
What to measure and how often
- Exercise cadence: quarterly full-scenario exercises for high-impact models/vendors; semiannual for lower tiers.
- Playbook coverage: percent of production models with an assigned incident playbook and RTO/RPO. Target: 95% for high-impact models within 90 days; phased for others.
Run meaningful drills
- Scenarios: model theft, data poisoning, prompt injection, vendor compromise. Force procurement and legal into decision roles.
- Evidence requirements: define lineage, telemetry, and vendor logs needed for investigations and contractually require retention timelines.
- Output: auditable post-exercise reports and tracked remediation to closure.
Example scenario (illustrative): a regional service provider that raised inventory coverage from 40% to 90% reduced time-to-isolation in drills from multiple days to under 6 hours. Use similar tabletop measurements to quantify operational improvement.
SBOMs, Provenance, And Secure Defaults—But Expect Phasing
Model provenance matters like software SBOMs do. Do not demand immediate perfection. Map vendor maturity and apply tiered requirements.
Targets with caveats
- Model-provenance coverage: require provenance records for all externally supplied models where vendor capability exists. Set phased deadlines: 60% within 90 days for tier-1 vendors, 95% in 6 months, and require a remediation plan for the remainder.
- Secure defaults: insist on deny-by-default network and minimal data access for new vendor deployments. For legacy deployments, require compensating controls and a timeline for migration.
Procurement pulls it all together
- Contract language: require inventory registration, provenance/SBOM where available, identity inventory, named technical POC, and escalation SLAs. Do not rely on informal attestations.
- Evidence SLAs: aim for 24-hour incident notification and 72-hour lineage artifact delivery where technically possible. Where vendors cannot meet these windows, require a documented exception and an agreed improvement plan.
- Audit rights: embed rights to review logs, request artifacts, and require remediation windows.
Standards note: align with NIST AI risk guidance and emerging industry work on model provenance and SBOM concepts.
Cadence: Weekly Rituals, Quarterly Evidence Packs
Governance decays without a heartbeat. Make it routine and auditable.
Weekly ritual
- Review new model entries, outstanding lineage gaps, token expirations, and identity attestations.
- Owners: CISO or delegated product security lead signs off on critical changes.
- Output: a short, timestamped audit log that feeds the quarterly pack.
Quarterly evidence pack
- Validate inventory coverage, run the vendor identity sweep, verify provenance claims, and present tabletop results.
- Audience: CISO, Procurement, Head of Engineering, Legal. Produce a resilience score and a remediation plan with owners and deadlines suitable for audit and board review.
Prioritize: quick wins first. Start by locking down inventory and identity hygiene for tier-1 production models. Follow with provenance and full playbook coverage.
One Board Test. One Operational Demand.
If you can produce this single packet in under 48 hours, you are operating at a reasonable baseline:
- last week's inventory changes
- current vendor identity attestation status
- remediation list from the most recent tabletop with owners and deadlines
If you cannot, you do not yet own your AI risk.
NightFortress advises organizations on AI risk governance and operational resilience from Arlington, VA. Make governance auditable. Make it routine. Then you can be resilient on purpose.
If you want help assessing your exposure, start with the free AI SMB Risk Index Survey. Five minutes. Immediate baseline score.
For the field guide version of what I publish here each week, pick up a copy of Exposed: Inside Risks and The New Architecture of AI Defense on Amazon.
NightFortress works with executives, founders, and mid-market organizations in Northern Virginia and the DC metro area to assess exposure, govern risk, and build security programs that match the actual threat landscape. Contact us to start a conversation.
The information in this article is for educational and informational purposes only. It is not intended as legal, compliance, or professional cybersecurity advice for any specific organization. Consult qualified professionals before making security or compliance decisions.