What does the build-versus-buy decision omit about AI maintenance?

Production readiness, dependency ownership and the lifecycle of an AI application

This retrospective review considers source versions available by 28 April 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: AI maintenance; build versus buy; technical debt; production readiness; model documentation

1. The research question

When does ownership of an AI application justify responsibility for its maintenance? Build-versus-buy discussions often compare the cost of producing an initial demonstration with the subscription price of an existing service. Those quantities cover different scopes. A demonstration establishes that selected inputs can produce a useful-looking output. A maintained service must keep working as its inputs, dependencies and users change. The comparison becomes meaningful only when both alternatives are evaluated over a common period and against the same service requirements.

The choice also has more than two forms. Configuring a supplied application, integrating an external model, adapting a model and operating a complete inference service distribute responsibility differently. Custom code can depend heavily on a hosted model. A purchased application can still require substantial internal work to prepare data and evaluate outputs. The label “bespoke” therefore says little about which capabilities or costs the organisation actually owns.

This review asks what machine-learning engineering research and technical governance frameworks contribute to this decision. It concentrates on the service lifecycle, dependency boundaries and evidence of production readiness. It does not rank current software vendors or estimate the return on a particular procurement. Marketing applications provide a useful context because changes to product information, approved claims and campaign requirements can affect a system even when the model and source code remain unchanged.

2. Review method

Targeted searches on 21 October 2026 examined technical debt, production readiness, machine-learning software engineering, model and dataset documentation, underspecification and AI risk management. Twelve sources were selected to connect the initial implementation decision with the continuing operating obligation. Original research papers, author summaries and official technical guidance formed the evidence base. Selection was purposive; no exhaustive search, duplicate screening or quantitative synthesis was undertaken.

The engineering papers predate some current generative applications. They are used for the mechanisms and system properties they discuss, rather than as direct estimates of maintenance costs for contemporary language-model products. The retrieved sources contain both empirical observations and proposed documentation practices. These types of evidence have different force. A reporting framework can identify information that should be recorded without proving that adoption of the framework causes a specified financial benefit.

Where a research source was consulted at abstract or author-summary level, the account remains at that level. No percentage of project failures, universal maintenance multiplier or comparison of current vendor prices is inferred. The synthesis instead examines which assumptions a fair comparison would need to make explicit and which published findings challenge the idea that model acquisition resolves the surrounding engineering work.

3. What is being acquired?

3.1 The component and the application

A trained model is one component of an application. A working application additionally has a way to obtain inputs, control access, call the model, interpret its output and deliver the result to another person or system. Some components may be supplied together; others may be assembled internally. Comparing alternatives requires an inventory of those boundaries. Otherwise a nominally inexpensive option may omit work that appears in the competing proposal.

Amershi and colleagues' case study of Microsoft teams identifies difficulties in data versioning, the skills required for model customisation and reuse, and the entanglement of AI components [3]. The observation is important for interpreting modularity claims. A well-defined API can simplify the software interface while leaving performance dependent on changes elsewhere in the workflow. An integration can remain syntactically valid even when the quality of its output has changed.

For a marketing knowledge assistant, for example, the output depends on which source documents are available and how a question is interpreted. Adding an approved document can improve one class of answers while changing retrieval for another. This is a structural illustration of the dependency problem, not a report of a measured failure rate. The relevant test is whether the assembled service performs its intended tasks after the change, rather than whether one component still passes its own isolated checks.

3.2 Ownership and control

Ownership of application code does not imply control of every dependency. Model weights, hosting, identity systems, search indexes and third-party APIs can remain outside the code owner's control. Conversely, buying a managed service can transfer some operational responsibilities while leaving the customer accountable for its own data preparation, user permissions and acceptance decisions. Both arrangements combine internal and external obligations.

The economic question is therefore conditional on the service boundary. If a supplier maintains the inference layer, the internal team may still need to detect regressions in business outputs. If the organisation operates the model itself, it additionally needs the capacity to maintain that layer. Neither arrangement can be evaluated by counting only the components visibly written by the organisation. The comparison must include the resources needed to sustain the complete intended service.

4. Maintenance mechanisms documented in the literature

Sculley and colleagues describe technical debt arising from data dependencies, feedback loops, configuration and other system interactions [1]. Their account challenges the inference that a small amount of model code implies a small maintenance obligation. It does not supply a universal cost ratio. The relevant contribution is an explanation of how dependencies can accumulate outside the model and make later changes more difficult to reason about.

Breck and colleagues propose a rubric of tests and monitoring needs for production readiness [2]. This shifts attention from a demonstration score to observable service properties. A system can perform well on a fixed evaluation set while lacking a way to detect that live inputs have changed. It can also have monitoring that identifies a problem without a usable recovery procedure. The existence of one production control should not be treated as evidence that the rest of the lifecycle is covered.

Baylor and colleagues describe TFX as an implemented production machine-learning platform spanning the surrounding pipeline, rather than only model training [11]. This is evidence from a particular engineering setting. It does not establish a universal preference for buying or building. Its relevance is the system boundary: a model sits within a service that also depends on data handling, validation and deployment. An evaluation confined to model acquisition leaves those components outside the comparison even though the service requires them.

Maintenance also has a knowledge component. A system may have been built around assumptions that are understood by one developer but absent from its documentation. Later maintainers then see the current implementation without the reasons behind it. A useful service record links requirements, accepted examples and known constraints to the implementation. This permits a change to be assessed against the original purpose, while leaving room to revise that purpose when the business requirement actually changes.

5. Data and model documentation

5.1 Dataset scope

Datasheets for Datasets proposes documenting a dataset's motivation, composition, collection and recommended uses [5]. For an organisational application, the corresponding questions extend beyond file presence. A document can be current, superseded, incomplete or valid only for a particular market. If these distinctions are lost when material enters a retrieval system, the application may return an answer based on a source that was never intended for that use.

The documentation proposal is not proof that a documented dataset is accurate or representative. It makes the basis of a judgement more inspectable. A data inventory can reveal that certain customer groups or product categories are absent, but cannot manufacture observations about them. This matters in procurement because a vendor can demonstrate functionality on a supplied corpus without establishing that the corpus is adequate for the organisation's full task population.

5.2 Model scope

Model Cards proposes structured reporting of intended uses, evaluation procedures and performance across relevant conditions [4]. Such information can help a purchaser understand what a model has actually been tested to do. The card itself is a record, not a warranty that an application assembled around the model will perform equally well. Application-specific prompting, retrieval, tools and review processes change the context in which the model operates.

Versioning is essential to the connection between documentation and behaviour. A test record for one model version does not automatically describe a replacement. Likewise, an unchanged model used with a new retrieval index or a revised instruction can constitute a new application configuration. A reproducible service record identifies the combination that was evaluated, rather than attaching all historical results to a product name that persists across revisions.

6. Retrieval and customisation

Lewis and colleagues examine a retrieval-augmented generation architecture that combines parametric and non-parametric memory for knowledge-intensive tasks [9]. The work establishes a technical route for bringing external material into generation. It does not show that every organisational knowledge problem requires model training, nor that adding retrieval automatically makes all responses correct. The retrieval corpus, selection mechanism and generation process remain parts of the system being evaluated.

This distinction affects the build decision. A requirement may concern access to changing factual material rather than a new model capability. Another may concern a specialised classification or formatting task. The fact that both are described as custom AI does not establish that the same implementation is appropriate. A comparative evaluation should identify the requirement that a simpler configuration fails to meet and then assess whether the additional component resolves that failure.

Maintenance follows the chosen intervention. Updating a document store requires a process for changes to the source material and index. Adapting model parameters introduces another set of artefacts and evaluation obligations. Building a bespoke interface introduces interaction and accessibility requirements. These responsibilities can coexist. A proposal that describes only the most novel technical component leaves the service boundary incomplete.

The review cannot infer a universal hierarchy in which configuration should always precede custom development. Some requirements may rule out available managed services at the outset. The evidential requirement is narrower: alternatives should be compared against explicit needs, using the simplest credible comparator that can serve those needs. A technically sophisticated solution has not demonstrated an advantage merely by containing more components.

7. Testing beyond a single aggregate score

D'Amour and colleagues show that predictors with similarly strong held-out performance can behave differently in deployment domains, describing underspecification as a source of instability [6]. This finding concerns the relationship between selection criteria and deployment behaviour. It warns against treating one aggregate score as a complete specification of suitability. The exact failure mechanisms in a language-model application still require direct examination.

HELM's multi-metric evaluation approach reinforces the distinction between performance dimensions [10]. A service can become faster while less reliable on an important class of request. It can improve average answer quality while changing the distribution of errors. A procurement comparison should preserve those dimensions, particularly when one failure type makes an output unusable regardless of its score on other criteria.

Evaluation sets also require lifecycle management. A fixed set supports regression testing because versions can be compared on common examples. However, a set that never changes may cease to reflect the work users actually request. Adding new examples makes coverage more relevant but changes the benchmark. Reporting both the stable comparison set and new challenge cases can preserve continuity without presenting yesterday's workload as an exhaustive description of today's use.

A quality change can originate outside the model. Updated product information may remove a previously correct response. A changed downstream schema can cause valid prose to fail a system interface. A new group of users can ask questions that were absent from the original test. The service boundary should therefore include tests of data preparation, integration and user-facing behaviour. Model-only benchmarks address only part of that boundary.

8. Responsibility across the lifecycle

NIST's AI Risk Management Framework is a voluntary framework for incorporating trustworthiness considerations across AI design, development, use and evaluation [7]. The Generative AI Profile provides a companion treatment for generative systems [8]. These documents help structure an inventory of risk-management activity. They do not certify a service or establish that one sourcing arrangement is safer or more economical than another.

Responsibility needs to remain identifiable when work crosses organisations. A supplier may monitor uptime while the customer monitors business accuracy. A model provider may publish a release change while an application vendor decides when to adopt it. An internal owner may need to decide whether a degraded service should be paused. The operational question is whether these roles fit together, including the intervals in which an incident is detected but not yet resolved.

Evidence of a recovery path differs from evidence of a prevention measure. An evaluation suite may reduce the chance that a known regression reaches users. A rollback procedure supports recovery after a problematic release. A manual alternative may sustain service when the model route is unavailable. Each covers a different event and can have a different cost. Describing them together as “governance” can obscure which capabilities have actually been demonstrated.

9. Total cost and exit conditions

The FinOps Framework places technology-cost management in a business-value context [12]. A build-versus-buy comparison needs a consistent accounting window covering implementation, integration, operation and changes required to sustain the service. A subscription and internal engineering expenditure can be compared only after their included activities are specified. Moving work to another department changes its accounting location without necessarily removing resources consumed. This is an accounting distinction, not an observed estimate of the savings from either route.

Uncertainty should be visible in the comparison. Future demand, required service level and the frequency of changes may be unknown. These inputs can be varied in a scenario analysis, while keeping assumptions separate from observed costs. A break-even calculation is conditional on those inputs. It is not evidence that demand will reach the level at which the preferred route becomes cheaper, or that output quality will remain constant as the workload expands.

Exit is another lifecycle event. Moving to another provider can require transferring data, rebuilding integrations and re-evaluating outputs. Possession of source code may make some of this easier while leaving other dependencies intact. The economically relevant question is what must be reconstructed to sustain the service elsewhere. A documented export format is useful evidence about portability, but a successful migration test provides a different and stronger observation about the assembled application.

These considerations also apply to services built with coding assistants. Faster code production can change the initial development effort without removing the need to specify requirements and verify behaviour. If the maintenance burden is concentrated in the surrounding data, integration or operational processes, reducing code-writing time addresses only one component. The reviewed literature does not establish that coding acceleration makes every custom application cheaper over its lifecycle.

10. Limitations and conclusion

The evidence base is selective and heterogeneous. Several papers concern traditional machine-learning systems rather than current generative applications, and most were inspected at abstract or author-summary level. Their relevance is conceptual and technical; no direct maintenance-cost estimate for marketing organisations was extracted. Proposed documentation and governance frameworks are distinguished from empirical evaluations, and the review contains no original procurement experiment or longitudinal service audit.

The build-versus-buy decision is best understood as a comparison of sustained service arrangements. Existing research documents dependencies, evaluation limitations and production obligations that can persist after a model is acquired or a demonstration is complete. Custom development can be justified by requirements that available alternatives do not meet, but that justification must cover the full service and its maintenance. Ownership of code, model capability and operational control are separate properties. A credible comparison states which of them changes and measures the cost and performance of the resulting arrangement.

Source access and appraisal register

References [1], [2] and [3] were inspected through original research or proceedings summaries. References [4], [5], [6], [9] and [10] were consulted at author-abstract level. NIST references 7–8 were assessed through official publication overviews, not a complete control mapping. The original TFX research record [11] and dated FinOps Framework poster [12] replace rolling engineering and cost guidance. No current-system procurement test or independent full-text appraisal is claimed.

References