Companies across the globe are rapidly formalising the role of artificial intelligence within their hierarchies, listing AI agents on organisational charts and treating them as legitimate team members. This trend, which has accelerated over the past year, reflects widespread corporate enthusiasm for automation and digital transformation. Yet a growing body of academic research suggests that organisations may be moving far faster than their understanding of the technology's risks, creating vulnerabilities that could ultimately sabotage the very productivity improvements they seek.
Boston University professor Emma Wiles, who specialises in the intersection of AI and workforce dynamics, discovered this phenomenon at an industry conference where human resources executives touted the employment of AI agents as a competitive advantage. Intrigued by the approach, Wiles partnered with researchers from Boston Consulting Group to examine how this practice actually plays out in real corporate environments. Their findings proved sobering: managers systematically underperform when overseeing AI-generated work compared to work attributed to human colleagues.
The research centred on a deceptively simple experimental design. Managers across dozens of companies were asked to review documents containing deliberate errors within a twenty-minute window. The critical variable was attribution: some managers were told the documents came from AI employees, others from AI tools, and still others from human workers. While attribution generally made little difference in review rigour, a striking pattern emerged among companies that had formalised AI positions on their organisational structure. These managers caught substantially fewer mistakes when reviewing AI work, despite identical error rates across all conditions.
Wiles attributes this oversight gap to a fundamental shift in how managers conceptualise responsibility. When supervising human subordinates, managers typically embrace personal accountability for their teams' outputs, understanding that errors reflect on their own management. Similarly, when reviewing work from an AI tool, managers often feel implicitly responsible for the technology's performance. However, managers of anthropomorphised AI agents—those listed as "teammates" or "peers" on company rosters—appear to experience a psychological distance from accountability. They unconsciously delegate responsibility elsewhere: to the technology team, to senior executives who mandated the AI initiative, or to the AI system itself.
This accountability gap represents just one layer of a much deeper problem. As companies race to embed AI into operational decisions, researchers are cataloguing a troubling catalogue of second-order effects that most organisations appear to overlook entirely. The shortcomings extend far beyond simple human error in document review; they touch on fundamental questions about how algorithms make consequential business judgments and how bias embeds itself within automated systems.
One well-documented yet underappreciated vulnerability involves how AI systems evaluate other AI-generated content. Research from 2025 demonstrates that large language models systematically favour outputs produced by other AI systems over comparable work generated entirely by humans. This bias manifests across multiple corporate functions, most notably in recruitment. Ohio State University operations professor Jane Yi Jiang and her collaborators found that resume screening algorithms consistently rank AI-assisted resumes above human-written ones of equivalent quality. When recruiting firms learned of this bias, some inquired about remediation strategies. Yet Jiang emphasises that resume evaluation represents merely one visible manifestation of a broader, largely invisible problem: companies are adopting language models at breakneck speed with minimal consideration of downstream consequences, biases, or unintended interactions.
The implications extend into strategic business territory. Some companies now deploy AI to answer consequential questions about pricing strategy and site selection for new facilities. These decisions rest on fundamentally different assumptions about human behaviour than those embedded in actual market dynamics. Where humans often pursue cooperative outcomes and mutually beneficial arrangements, the mathematical frameworks underlying many AI models default to game-theoretic "rationality" that prioritises aggressive competitive positioning. An AI system might recommend that a company aggressively undercut competitors on price, a strategy that sounds rational in isolation but frequently triggers destructive price wars that damage all participants. University of Maryland doctoral candidate Jiannan Xu notes that most large language models overestimate human rationality, leading to strategic recommendations that sound logical but produce collective negative outcomes.
The scope of potential problems remains poorly mapped. Wiles emphasises that researchers are likely identifying only a fraction of the issues introduced by widespread AI deployment. The phrase "unknown unknowns" captures the epistemic challenge: companies cannot easily identify problems they have not yet encountered or theorised. Beyond the documented risks—AI bias against protected groups, confident generation of false information, unintended disclosure of confidential data—lie countless interaction effects and emergent properties that only become apparent as systems operate at scale in complex organisational environments.
Survey data provides quantitative scope to the phenomenon. Wiles and colleagues surveyed more than one thousand corporate managers, finding that approximately one-third work for organisations that formally classify AI as a "teammate or employee," while nearly one-quarter report that their companies include AI agents on official organisational charts. One manager interviewed for the study referred to an AI system as "Scout," describing it as an equivalent peer within the team structure. This language reflects a genuine conceptual shift in how organisations are categorising automated systems, with profound implications for governance, accountability, and risk management.
The fundamental challenge identified by researchers concerns the absence of established management frameworks for anthropomorphised AI. Over centuries, business leaders and scholars have developed sophisticated practices for managing human subordinates, practices refined through countless failures and iterations. These systems rest on psychological principles, accountability structures, and cultural norms that have proven robust across diverse contexts. However, the psychology of managing AI agents that possess human characteristics—that appear to have names, roles, and team membership—operates according to different rules. Managers instinctively adjust their behaviour based on whether they perceive an entity as human or machine, yet they often fail to recognise these adjustments or understand their consequences.
Wiles' conclusion reflects appropriate caution: organisations are "going out there blind," operating without adequate understanding of how management practices must adapt to accommodate AI agents embedded within traditional hierarchies. The risks are not intrinsic to the technology itself but rather emerge from the gap between the speed of AI adoption and the deliberate, studied approach required to implement it responsibly. Companies pursuing competitive advantage through AI integration must confront an uncomfortable reality: their enthusiasm for deployment may be outpacing their capacity to manage the systems they are creating.
For Southeast Asian businesses considering similar AI integration, the stakes are particularly high. Regional companies often operate in contexts characterised by rapid expansion, competitive pressure from multinational enterprises, and pressure to demonstrate technological sophistication. Yet the research from Wiles, Jiang, and their collaborators suggests that moving faster than institutional wisdom can accommodate creates concentrated risk. The path forward requires deliberate investment in understanding how these systems perform in practice, establishing clear accountability structures, and resisting the temptation to outsource responsibility for AI decisions to the technology teams or to the algorithms themselves.
