Who benefits from AI assistance, and what does adoption measure?

Experience, task boundaries and the distinction between use and improved performance

This retrospective review considers source versions available by 5 June 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: AI adoption; expertise; heterogeneous effects; productivity; human AI collaboration

1. The question behind adoption

Who benefits when people begin using AI assistance, and what does a measure of adoption actually establish? Access to a tool, use of its suggestions and improved performance are different observations. An organisation can distribute licences without changing how tasks are completed. Users can engage frequently with a system while gaining little on the outcome that matters. A measured performance improvement can also occur in a controlled experiment without establishing sustained use in the workplace.

The distinction becomes more important when treatment effects differ across people and tasks. An average can conceal gains concentrated among less experienced workers, benefits on one task and losses on another, or a change in speed accompanied by a change in quality. The question is therefore not answered by a single adoption percentage or a general statement that AI raises productivity. The relevant population, intervention and outcome need to remain visible.

This review examines experimental and field evidence on heterogeneous effects, the interpretation of assistance and the measurement of subsequent independent performance. It also considers earlier algorithm-advice research to clarify why willingness to rely on a system is separate from its effect on judgement. The focus is on empirical interpretation rather than a proposed adoption model, staffing recommendation or forecast of employment effects.

2. Review method

Targeted searches on 21 October 2026 covered generative AI productivity, experience, expertise, human–AI combinations, developer productivity, algorithm aversion, algorithm appreciation and unassisted performance. 14 sources were selected to contrast populations, task designs and endpoints. These include randomised experiments, a workplace field study, a published meta-analysis and foundational behavioural studies. Selection was purposive, with no exhaustive database search, independent screening or new meta-analysis.

Full-text inspection was concentrated on the customer-support author manuscript, the experienced-developer experiment, the two algorithm-advice papers and the creativity experiment [2] [4] [8] [9] [10]. The consulting article's task and design sections were also inspected. Other findings are constrained to author or publisher summaries. The source-access register identifies this variation; availability of a full paper is not described as independent replication of its analyses.

The synthesis keeps several distinctions explicit. Assigned access differs from self-selected use. A worker's experience in a profession differs from experience with a particular repository or AI interface. A rated output differs from task completion time and from later unaided capability. Differences across studies can suggest where transfer requires caution, but do not identify a causal moderator when many design features change together.

3. Exposure, use and performance

An access experiment estimates the effect of being assigned the opportunity to use assistance under the tested conditions. Participants may ignore, edit or selectively use suggestions. That behaviour is part of the intervention's observed result. Comparing only participants who used the tool heavily can change the question, because intensity of use may reflect task difficulty, prior ability or early experience of the tool. A correlation between use and performance does not by itself estimate the effect of greater use.

A workplace adoption measure may instead record whether a person used a service at least once during a period. Such a measure describes reach. It does not capture the share of work affected or how much of an output came from the tool. A measure of frequent activity describes another dimension, but request counts can rise because a system is helpful or because it requires repeated correction. The connection to productive use needs observation of the work and its quality.

Noy and Zhang's writing experiment reports effects on time and rated output, and subsequent self-reported use of ChatGPT [1]. These outcomes illustrate the distinction between an experimental performance result and later behaviour. The follow-up use measure does not directly establish the quality of every later workplace task. Equally, a laboratory gain remains evidence about the assigned writing tasks even if some participants subsequently decide not to use the tool.

4. Expertise is task-specific

4.1 Customer support

Brynjolfsson, Li and Raymond examine a conversational assistant deployed to customer-support agents. Their revised author account reports greater improvements among less experienced and lower-performing workers [2]. The system supplied suggestions while agents retained responsibility for the conversation and could edit or ignore them. The intervention therefore combined model output with an existing service workflow and human discretion.

The manuscript distinguishes tenure from a skill index based on prior performance, and its aggregate productivity outcome is reported at the agent-month level because some quality information is available only at that level [2]. These design details matter when interpreting heterogeneity. “Less skilled” refers to the operational measure in that setting; it is not a universal classification of a person's intelligence or professional potential. A different skill measure could define different comparison groups.

The finding does not imply that all expertise is dispensable. The assistance may make some task-relevant patterns more available to workers who have not yet acquired them through experience. Establishing that mechanism requires the study's own analyses and remains bounded by the system and tasks observed. It does not show that a general-purpose model can replace all knowledge held by the most experienced employees.

4.2 Consulting and task boundaries

Dell'Acqua and colleagues randomly assigned consultants to no AI access, GPT-4 access or access with a prompt-engineering overview after a baseline assessment. The March 2026 report distinguishes tasks within the assessed capability frontier from a selected task outside it [3]. AI improved performance on the former and reduced correctness on the latter. Professional status alone therefore did not determine whether assistance was beneficial.

The relevant expertise can include knowing the domain, understanding the task and judging when a plausible response is wrong. These capacities need not vary together. A person can be experienced in a profession while unfamiliar with the model's failure modes, or skilled at operating an interface while lacking the knowledge needed to verify the answer. The reviewed experiments do not collapse those dimensions into one universally sufficient skill score.

5. Coding studies with different outcomes

Peng and colleagues' controlled experiment asked developers to implement an HTTP server in JavaScript and reports faster completion with GitHub Copilot [6]. Becker and colleagues examine experienced contributors completing real tasks in repositories they know well and report longer completion time when early-2025 AI tools were allowed [4]. Both concern software development, but their task populations and operating conditions differ substantially.

The experienced-contributor experiment involved 16 developers and 246 tasks. Tasks were defined before random assignment to AI-allowed or AI-disallowed conditions. The outcome included self-reported implementation time before and after pull-request review; screen recordings supplied additional process evidence. The task count does not turn the developer sample into 246 independent people. Repository familiarity and review standards also form part of the setting [4].

The authors compare participants' expected and perceived speed gains with observed time, finding a divergence between them [4]. This supplies a concrete reason to distinguish subjective usefulness from measured completion time. It does not establish that user experience is irrelevant or that all contemporary coding tools slow work. The tools correspond to a specified early-2025 period, and later models or different workflows require their own evaluation.

A direct comparison of the two headline percentages would be misleading. The studies use different tasks, participants, interfaces and quality processes. The contrast establishes that the sign of an effect can differ across settings described by the same broad occupational label. It does not isolate which single difference caused the change. Such identification would require a design that varies that feature while controlling the others or a sufficiently supported comparative synthesis.

6. Human–AI combinations and the choice of baseline

Vaccaro, Almaatouq and Malone's meta-analysis distinguishes performance of a human–AI combination relative to humans alone from performance relative to the better of human or AI alone [5]. A combination can improve on the human baseline while still falling short of the strongest standalone performer. The word “augmentation” can therefore refer to a benefit without demonstrating synergy. The comparator must be stated whenever a result is described as better.

This has practical consequences for interpreting demonstrations. A team using AI can outperform an unsupported individual, but the comparison changes more than model access if team size or task allocation also differs. Conversely, a standalone model evaluation may omit the costs of preparing inputs and checking outputs. The appropriate baseline follows the question: improvement over current work, maximum achievable performance and the value of a particular collaboration arrangement are related but distinct objectives.

The Cybernetic Teammate working-paper summary reports a preregistered product-innovation experiment with 776 professionals, crossing individual or team work with AI access. It reports that individuals using AI matched the performance of teams without it and that proposals became more balanced across functional orientations [7]. These are findings about the defined innovation challenges and evaluation. They do not establish that an individual with AI can replace every function of an enduring team.

A team also performs coordination, learning and accountability over time. A short product-innovation task may assess the resulting proposal without capturing all of those properties. This does not diminish the task finding; it identifies the boundary of the comparison. Claims about sustained organisational substitution require observations of the responsibilities and time horizon that the broader claim includes.

7. Reliance, trust and observed mistakes

Dietvorst, Simmons and Massey investigate reactions to algorithmic forecasting. Their experiments varied whether participants saw the algorithm, a human, both or neither perform before choosing whose forecasts would determine incentives. Seeing algorithmic errors reduced reliance even where the algorithm outperformed the human comparator [8]. The design concerns statistical forecasts and observed performance feedback, rather than a generative assistant in an open-ended workplace task.

Logg, Minson and Moore find greater adherence to advice described as algorithmic in several estimation tasks, with important boundaries involving own judgement and expertise [9]. Together, the papers caution against a universal claim that people simply distrust or prefer algorithms. Task, framing, experience and the available comparator affect the observed choice. Their findings do not supply a single adoption coefficient for modern language-model services.

Reliance also needs a performance interpretation. Greater trust is helpful only when it improves the use of information relative to the task. Blindly accepting a correct suggestion and blindly accepting an incorrect one can look identical in a use log. Appropriate reliance requires a relationship between the quality of the suggestion and the user's response. A system can have high acceptance rates while users fail to detect its consequential mistakes.

The comparison with a person's own judgement introduces another boundary. A participant choosing between personal and algorithmic estimates may respond differently from one choosing between two external advisers. Organisational adoption can contain both situations: users may treat the system as a collaborator on their own work or as a substitute for someone else's advice. The behavioural evidence supports attention to that distinction without establishing which framing should be preferred in every setting.

8. Assisted performance and learning

Kumar and colleagues' creativity experiments assign different forms of assistance during exposure rounds and subsequently test all groups without assistance. The tasks distinguish divergent idea generation from convergent association. The study does not find consistent improvement in later unaided performance, and some comparisons favour the control [10]. This is evidence about a short exposure-and-test sequence, not a demonstrated lifelong decline in creative ability.

The design separates a tool's contribution to an immediate deliverable from changes in the user's capability. A person can produce a stronger answer with assistance without learning how to produce it independently. Conversely, a learning activity can temporarily take longer because it requires the learner to work through a problem. Completion speed alone cannot establish the educational effect of an intervention. A study needs an endpoint that matches the capability claim.

Doshi and Hauser report larger gains in story ratings among writers with lower baseline divergent-association scores under their constrained AI-idea intervention [11]. That baseline is a specific measure, not a complete ranking of creative expertise. Erwin, Tran and Koutstaal's design-ideation study finds that different divergent-thinking measures contribute differently to performance [12]. This measurement evidence limits the assumption that one baseline score exhaustively describes the capacities relevant to every creative task.

Training a user to operate the interface is also different from teaching the underlying domain. A person may learn how to obtain a fluent response while remaining unable to verify its factual or strategic content. Measuring independent capability therefore requires a suitable test after assistance is removed, with a defined delay and task relation. Neither satisfaction with a training session nor the quality of a model-assisted final submission answers that question by itself.

9. Estimating heterogeneity without inventing moderators

An average treatment effect is an average over the study's participants and tasks. Changing their distribution can change the average even if each subgroup's effect remains the same. This is an aggregation property. It explains why a result from one workforce cannot be transferred mechanically to another with a different task mix. The necessary weights must be observed or stated as assumptions; their appearance in a calculation does not establish them empirically.

Subgroup analysis needs care. A study can find a statistically significant effect in one group and an insignificant effect in another without establishing that the two effects differ. The relevant comparison is the difference between the effects and its uncertainty. Small subgroups can also produce unstable estimates. The present review avoids turning isolated subgroup patterns into a new general rule about who should or should not use AI.

Experience is especially vulnerable to ambiguous coding. Years in an occupation, months in a role, familiarity with a repository and prior use of a particular interface are different variables. A source that measures one should not be reported as if it measured all. The developer and support studies illustrate this diversity of operational definitions. A synthesis must preserve it before claiming a common expertise effect.

Cross-study differences are also confounded by design. If a newer model is tested on a different task with a different population, a better result cannot be attributed solely to model improvement. Similarly, a negative result with experienced users cannot establish that experience caused the loss if the task and environment differ from the positive study. Direct comparisons or appropriately controlled designs are required to identify such relationships.

9.1 Measurement after adoption

A live rollout creates selection questions that a random access experiment can partly avoid. Early users may be unusually motivated, face different tasks or have more supportive managers. Comparing them with nonusers can combine those differences with the tool's effect. Comparing the same person's performance before and after adoption can also capture other changes over time. The empirical strategy must address the relevant counterfactual rather than assuming that use itself supplies one.

Outcome definitions can change when a tool makes new work possible. Counting tasks completed can rise because work is divided into smaller pieces. Counting text can rise because the same answer becomes more verbose. Measuring elapsed time can miss changes in the accepted scope. Stable task definitions and explicit quality criteria improve interpretability, while a separate account of newly enabled work preserves benefits that a fixed task comparison might omit.

Self-reports remain useful for experiences that are difficult to observe directly, such as perceived effort or satisfaction. They should be reported as those outcomes. The experienced-developer experiment demonstrates why a perception of speed should not be substituted for recorded completion time [4]. An evaluation can retain both and investigate their divergence, rather than discarding one or assuming they must agree.

10. Transfer to marketing

Marketing work contains tasks resembling several reviewed settings, including writing, ideation, research synthesis and coordination. Resemblance does not establish an equal effect. A communication may need to satisfy brand, factual and market-specific requirements absent from a generic writing task. An experienced marketer's context can be distributed across documents and tacit knowledge. The relevant evaluation therefore concerns the actual assignment and the intended form of assistance.

A task-specific comparison can measure accepted output, review effort and elapsed time while retaining baseline expertise and exposure. The point is to identify who benefits on which work, rather than infer success from licence distribution. If the organisation also claims skill development, an independent test addresses that additional endpoint. If it claims better commercial outcomes, production ratings alone remain insufficient.

The reviewed evidence does not establish that novice marketers always benefit more, that experienced marketers should avoid AI, or that coaching interfaces necessarily produce learning. It supports a more precise observation: treatment effects depend on the intervention, task and population, and different outcomes can move differently. A favourable result requires those details to remain attached when it is used to inform another setting.

Cognitive-offloading research distinguishes external support for task execution from changes in internal capability. [13] Cognitive-forcing experiments show that interface design can affect overreliance. Experience and adoption frequency therefore do not fully specify the assistance condition; the opportunity and incentive to check the system also matter. [14]

11. Limitations and conclusion

The evidence base spans different technologies, years and task populations. Earlier algorithm-advice research concerns forecasting rather than current generative systems. Several contemporary findings were consulted through abstracts or author summaries, while full-text appraisal was concentrated on selected central reports. No new causal estimate or meta-analysis is calculated. The review cannot quantify a universal effect of expertise, adoption intensity or training on marketing performance.

AI access, reliance and performance are separate empirical quantities. Existing studies report gains, losses and heterogeneous effects under specified conditions, while learning studies distinguish immediate assisted output from later independent capability. The appropriate conclusion concerns evaluation design: define the task and assistance, retain the comparator, measure the outcome that matters and preserve variation across people and work. An adoption count establishes use; a productivity or capability claim needs its own evidence.

Source access and appraisal register

The additional comparative sources in references 13–14 are used for the specific distinctions cited in the discussion. Access is limited to previously inspected primary-source abstracts, publication records or manuscripts. Their inclusion does not imply a fresh full-text risk-of-bias appraisal or direct validation in the application discussed here.

Full texts were inspected for [2], [4], [8], [9] and [10]; selected design, measurement and task sections were inspected for [3] and [12]. Reference [1] was consulted through the published abstract. Reference [5] was consulted through indexed publisher and abstract summaries after direct journal retrieval was unavailable. Reference [6] was consulted at author-abstract level, and [7] through the institutional working-paper abstract. Reference [11] was consulted through indexed primary-publication abstract and methods information. Supplementary datasets and analyses were not independently reproduced. The customer-support HTML displays a later internal manuscript date than its repository version label, so the review avoids treating all version-specific numerical details as interchangeable.

References