On September 22, Reuters reported that Meta had been piloting a "human concierge" feature for its Muse AI agent, involving contractors handling customer inquiries. This revelation sparked internal privacy concerns among Meta employees, leading a vice president to acknowledge the oversight of not initially disclosing contractor involvement in placed calls. The feature, which was being tested internally with Meta employees and not yet with enterprise clients, has since been rolled back. This incident, while specific to Meta, illuminates a broader and critical challenge for companies procuring AI-enabled services: understanding how the service can evolve even when the underlying AI model and its version remain static. For chief compliance officers, audit executives, and third-party risk managers, the fundamental question is whether the AI service currently in use aligns with the factual basis established during its initial approval.
The intricate relationship between AI technology and human oversight in service delivery presents a complex landscape for vendor risk management. As AI systems become more sophisticated and integrated into core business operations, the human element supporting these systems also evolves. This evolution, often occurring without a formal AI model release, can introduce new risks that traditional vendor monitoring protocols might overlook. The Meta Muse AI incident serves as a potent reminder that changes in staffing, access levels, and operational responsibilities within a vendor’s ecosystem can significantly alter the risk profile of an AI service.
The Evolving Human Layer in AI Service Delivery
For several years, regulatory advisor Maja Kurek managed the end-to-end experience of a substantial global network of over 10,000 external mathematics experts who served as the human component of an AI-powered learning product. This experience provided firsthand insight into the complexities of managing human resources in a vendor context. When this network transitioned to a new provider, critical aspects such as data protection, access controls, payment security, and operational continuity had to be meticulously managed to ensure a seamless handover. The paramount concerns revolved around who had access to what information, the redistribution of responsibilities, and the uninterrupted continuation of work and payment processes. While an expert network differs from a typical enterprise AI vendor, Kurek’s experience underscored a crucial lesson: the importance of tracking the role rather than solely the roster of individuals involved.
This perspective is vital because the human layer supporting AI services is not static. It can be dynamic, with individuals’ roles expanding, their access privileges being modified, or their responsibilities being redefined, all without a corresponding update to the AI model itself. For instance, an AI service initially approved based on external reviewers sampling redacted outputs might later see the same individuals resolving customer cases using full, unredacted records. In such a scenario, the AI model and its version remain unchanged, but the expanded access and potentially altered authority necessitate a re-evaluation of privacy, security, and customer-protection controls.
Beyond Model Updates: Detecting Service Transformations
While contractual clauses mandating notification of AI model changes are valuable, they represent only one facet of service evolution. Elaine F. Duffus and Aoife May, in a recent article for Corporate Compliance Insights on bank AI governance, rightly listed model change notifications among essential vendor terms. However, a model release is far from the only way an approved AI service can transform.
The practical challenge for compliance and risk management teams lies in ensuring that the current operational reality of an AI service continues to align with the factual underpinnings of its initial approval. This requires a shift in focus from purely technical specifications to the operational context in which the AI service is deployed and supported by human actors.
Staff turnover within a vendor organization is a normal occurrence and should not necessitate a full re-evaluation of every approved service. Initial vendor diligence establishes the fundamental acceptability of a service. The ongoing task, however, is to detect when the specific conditions that informed that initial decision are no longer true. This proactive monitoring is essential for maintaining regulatory compliance and mitigating emergent risks.
A Gartner survey of third-party relationship owners revealed a significant gap in risk communication: only 52% reliably shared observed changes in the scope of a relationship with compliance departments. Relationship owners, often the frontline observers of such shifts, may lack clarity on which changes are material to control functions. This highlights the inadequacy of a general "tell us if anything changes" instruction. Instead, business owners require more concrete triggers for reporting.
Establishing a Robust Framework for Tracking Role-Based Changes
To address these evolving risks, a more granular approach to documenting and monitoring AI services is required. The National Institute of Standards and Technology’s (NIST) AI Risk Management Framework (AI RMF) playbook, specifically under MANAGE 3.1, advocates for regular monitoring of third-party AI risks. It also emphasizes the utility of documenting technologies, personnel, and other resources to aid risk management. However, it does not explicitly define which personnel-related facts are critical to retain.
For external human roles that materially support an AI service, a more detailed documentation strategy is advisable at the point of approval. This strategy should capture four key facts about the role:
- The specific function performed by the human role: Clearly define the tasks and responsibilities undertaken by individuals in this capacity.
- The scope of data access granted: Detail the types of data these individuals can access, including sensitivity levels and any anonymization or redaction protocols.
- The level of authority granted: Specify the decisions these individuals can make, their autonomy, and any escalation paths.
- The reporting structure and oversight: Outline who the individuals report to and the mechanisms for supervision and quality assurance.
Vendors should be contractually obligated to confirm these facts in writing. For higher-risk services, the agreement should stipulate the evidence required for subsequent checks, such as updated role descriptions, summaries of access controls, or relevant assurance reports pertaining to the specific service. If a vendor cannot confirm a material fact, this discrepancy must be documented and escalated to the relevant risk owner. A missing confirmation should not be interpreted as an indication that nothing has changed.
Analyzing the Impact of Material Changes
A crucial step in managing these evolving risks is to consistently ask: "If this change had been in effect on Day One, could it reasonably have altered the initial approval decision or the controls associated with it?" Changes that do not meet this threshold can typically be managed as operational matters. However, those that do warrant immediate escalation to the appropriate risk management team.
For example, granting broader access to sensitive data would fall under the purview of privacy or security teams. If a role gains new authority to act on behalf of a customer, this necessitates legal review and an assessment of potential customer impact. Similarly, if human support becomes indispensable or shifts to a different provider, it triggers a review of business continuity and exit plans.
In most instances, organizations can revisit and re-approve only the affected component of the service rather than initiating a complete vendor reassessment. This targeted approach streamlines the risk management process.
Legal and procurement teams play a vital role in aligning vendor notice obligations with these documented facts. Existing contractual clauses regarding incidents, model releases, or named subprocessors might not adequately capture changes in human roles. It is essential to negotiate clear agreements on which material changes necessitate notification, moving beyond the vendor’s sole definition of a "model update."
During periodic reviews, business owners should proactively inquire about any external roles that have gained materially broader access or authority, experienced a shift in their delivery arrangement that impacts risk, or become significantly more critical to service delivery. Answers should be corroborated against agreed-upon evidence. This oversight function should ideally be integrated into existing third-party risk, privacy, procurement, or operational risk processes, rather than creating a separate, standalone AI governance workflow.
Empowering Internal Audit with Testable Mismatches
The Institute of Internal Auditors’ third-party topical requirement, effective September 15, establishes a baseline for assessing third-party governance, risk management, and control processes when internal audit functions provide assurance on third-party risk. For AI services selected for review, internal audit can be equipped with two focused questions:
- Does the service currently delivered align with the material facts documented at the time of approval?
- Where discrepancies exist, did the change reach the appropriate internal owner and result in a documented decision?
Returning to the hypothetical example: if the approval record states an external role samples redacted outputs, but the current service utilizes individuals in that role to resolve customer cases using full records, internal audit can investigate. They can ascertain when this change occurred, who reviewed the expanded access, and what organizational decision was made. This approach allows internal audit to test the integrity of the approval process without needing to reverse-engineer the AI model or obtain the vendor’s complete workforce roster.
While Meta’s employees had internal visibility into the testing of their AI agent, enterprise customers often lack such direct insight. The approval record, therefore, provides organizations with a concrete benchmark against which to test the ongoing reality of the services they are consuming. This focus on the documented facts and their alignment with current operations, rather than solely on model versions, offers a more robust and practical approach to managing the evolving risks associated with AI-enabled services. The dynamic nature of human roles within these vendor ecosystems demands a continuous, role-centric monitoring strategy to ensure that the promise of AI is delivered responsibly and securely.
