Developer Reliance and Verification in LLM-Assisted Software Development: A Systematic Literature Review
Abstract
Generative AI coding assistants have rapidly penetrated daily software engineering workflows. Yet, software engineering literature frequently examines developer reliance and code verification in isolation. Adhering to PRISMA 2020 guidelines and MMAT appraisal, this review systematically synthesizes 12 empirical studies (2025–2026) alongside four methodological benchmarks.
Crucially, existing empirical findings directly conflict: one cluster of experiments reveals that heavy reliance breeds developer complacency and inflates defect escape rates, whereas another cluster demonstrates that developers review machine output with the same rigorous scrutiny given to human peer code.
1. Introduction
The rollout of generative code engines has reshaped how software practitioners and computer science students write software. Broad landscape studies catalog hundreds of papers since 2022 charting LLM adoption across different stages of development [1]. Controlled trials highlight major speedups on self-contained programming assignments when using tools like GitHub Copilot [2]. Simultaneously, security audits caution that automated engines frequently emit vulnerable or buggy code snippets [3].
This tension between developer speed and code safety raises two questions that are seldom studied in tandem: How much do programmers actually trust code produced by LLMs, and how thoroughly do they inspect that code before merging it? A critical review by Baltes et al. noted that software engineering research often leans on a simplistic behavioral proxy—whether someone clicks "accept" on a suggestion—instead of examining the calibrated trust frameworks long established in human-computer interaction [4]. This construct conflation clouds comparisons across empirical findings.
This review targets that specific gap. Rather than reviewing every dimension of AI-assisted engineering—productivity, software quality, and testing all at once—we restricted our scope to developer reliance, code verification, and their direct behavioral relationship.
This focus stems from an identifiable split in the published record. Our systematic screening surfaced two distinct literatures. On one side are qualitative and survey-based papers detailing interpersonal trust, developer sentiment, and perceived tool helpfulness [9], [10], [12]. On the other side are observational experiments tracking physical verification routines using eye trackers and IDE interaction monitors [15], [17]. Experiments rarely measure psychological trust and objective inspection actions together.
Where papers do measure both, their conclusions diverge. Some trials show that trusting an assistant degrades review thoroughness [11], [18]. Others find that engineers sustain baseline review diligence regardless of tool use [12], [19]. Prior secondary reviews focus on developer adoption [5] or general productivity [6]. To date, no literature review has systematically evaluated the intersection of reliance and code verification.
1.1 Research Objectives and Questions
Software engineering studies frequently treat suggestion acceptance rates as an unproblematic proxy for calibrated situational trust, while treating self-reported review practices as equivalent to verified defect discovery. Comparing studies without accounting for these measurement differences leads to misleading generalizations. This review synthesizes empirical findings across four targeted research questions:
1.2 Contributions
- Joint PRISMA-Guided Synthesis: We provide an empirical synthesis focused specifically on developer reliance and verification, distinguishing between studies that measure calibrated trust versus simple suggestion acceptance.
- Delineation of Empirical Discordance: We map the empirical contradiction in RQ3, explaining why current study designs—split between self-reports and small-scale lab telemetry—cannot resolve whether reliance degrades verification.
- Moderator Hierarchy: We evaluate task stakes, developer seniority, and provenance disclosure as candidate moderators, ranking each by the strength of its underlying experimental evidence.
- Conceptual Map and Testable Protocol: We synthesize candidate interactions into an explicit conceptual framework (Section 5.1) and outline an executable 2×2 factorial protocol (Section 6) to test these variables directly.
2. Theoretical Background and Related Work
2.1 LLM-Assisted Software Development
Generative language models trained on massive source code repositories can generate complex code blocks, complete multi-line statements, and explain existing logic from natural language prompts. Commercial IDE extensions—such as GitHub Copilot and conversational interfaces like ChatGPT—have shifted from experimental novelties into standard industry utilities across diverse engineering workflows [1], [6]. Because this adoption happened so quickly, understanding how developers supervise automated suggestions has become a pressing software reliability challenge.
2.2 Developer Reliance and Trust
Researchers define reliance and trust in inconsistent ways. Many investigations rely on single-item behavioral counts, such as how often a user accepts autocomplete prompts. Others adapt interpersonal trust models based on perceived capability, integrity, and helpfulness [10].
As Baltes et al. noted, treating autocomplete acceptance as an equivalent indicator of human trust is an empirical error: a programmer might accept an inline completion just to avoid typing repetitive boilerplate, intending to refactor or delete it moments later [4]. Conflating that shortcut with psychological trust compromises construct validity. Grounding software engineering research in established models of human-automation reliance, such as the framework developed by Lee and See, provides a more grounded theoretical lens for interpreting current studies [4].
2.3 Verification of AI-Generated Code
In this review, verification refers to the active steps a programmer takes to read, test, and correct code suggestions before checking them in. While related to formal software testing, verification includes cognitive inspection, inline code adjustments, and exploratory debugging in an IDE.
Code produced by generative models contains unique defect categories. An empirical taxonomy by Tambon et al. revealed that LLMs produce structural bug types—such as hallucinated API methods, incorrect assumptions about execution environments, and silent logic omissions—that rarely occur in code written by humans [14]. Because these flaws are often syntactically clean and visually convincing, auditing AI-authored snippets demands distinct verification strategies rather than standard peer-review habits.
3. Systematic Review Methodology
This study follows the reporting structure outlined in the PRISMA 2020 statement [7].
3.1 Databases and Search Strategy
Our search targeted six primary academic repositories: IEEE Xplore, ACM Digital Library, Scopus, Web of Science, SpringerLink, and ScienceDirect, while using Google Scholar and arXiv for discovery and citation snowballing. We constructed search queries by combining three core keyword groups:
Technology: "large language model" OR LLM OR "generative AI" OR "AI coding assistant" OR Copilot
Population: developer OR programmer OR "software engineer" OR "code review"
Construct: reliance OR trust OR "over-reliance" OR verification OR "error detection"
Queries combined using: (Technology) AND (Population) AND (Construct).
3.2 Search Period
The search period covers January 1, 2023, through September 2026, capturing the era of mainstream LLM coding assistant adoption. Foundational theoretical frameworks published before this window are cited as background context.
3.3 Inclusion and Exclusion Criteria
- Inclusion: Empirical papers (qualitative, quantitative, or mixed-methods) assessing human developers working with generative code tools; studies that explicitly measure or evaluate reliance, trust, verification, or defect detection; published in English from 2023 onward with accessible full text.
- Exclusion: Studies evaluating legacy, non-LLM program synthesizers; benchmark papers assessing raw model accuracy without human participants; opinion pieces, blogs, and vendor white papers without an empirical methodology; duplicate records or preprints superseded by peer-reviewed versions.
- Protocol Deviation: During screening, we excluded studies where an LLM checked its own outputs without human involvement, as our research questions focus strictly on human verification behavior.
3.4 Screening Process
Retrieved records were deduplicated, screened by title and abstract, and evaluated in full text against the eligibility criteria. Backward and forward citation snowballing was performed across all eligible studies. Screening was carried out by a single reviewer. Best practices under PRISMA suggest dual independent screening with inter-rater agreement metrics (e.g., Cohen's kappa), and this single-reviewer setup is noted as a threat to validity in Section 7.
3.5 PRISMA Flow (Figure 1)
3.6 Quality Assessment (Table 2)
We appraised all 12 primary studies using the Mixed Methods Appraisal Tool (MMAT), 2018 version [8]. Each applicable criterion was scored as Yes, No, or Can't Tell. In accordance with MMAT guidelines, we do not calculate an aggregate numeric score.
| Study | Design Category | Criteria: Yes | Criteria: Can't Tell | Criteria: No |
|---|---|---|---|---|
| Sabouri et al. [9] | Mixed methods | 4 / 5 | 1 / 5 | 0 / 5 |
| Wang et al. [10] | Mixed methods | 3 / 5 | 2 / 5 | 0 / 5 |
| Perry et al. [11] | RCT | 2 / 5 | 3 / 5 | 0 / 5 |
| Klemmer et al. [12] | Qualitative | 5 / 5 | 0 / 5 | 0 / 5 |
| Alami et al. [13] | Qualitative | 4 / 5 | 1 / 5 | 0 / 5 |
| Tambon et al. [14] | Quantitative descriptive | 2 / 5 | 3 / 5 | 0 / 5 |
| Tang et al. [15] | Non-randomized quantitative | 2 / 5 | 3 / 5 | 0 / 5 |
| Zhong et al. [16] | Not scored (insufficient detail) | — | — | — |
| Khojah et al. [17] | Non-randomized quantitative | 2 / 5 | 3 / 5 | 0 / 5 |
| Prather et al. [18] | Non-randomized quantitative | 2 / 5 | 3 / 5 | 0 / 5 |
| Kazemitabaar et al. [19] | RCT | 3 / 5 | 2 / 5 | 0 / 5 |
| Yu & Moon [20] | Qualitative | 3 / 5 | 2 / 5 | 0 / 5 |
Table 2. MMAT quality appraisal summary. "Can't Tell" scores reflect incomplete methodological reporting in short-paper conference formats rather than confirmed empirical flaws.
3.7 Data Extraction and Synthesis
We extracted data on participant backgrounds, experimental tasks, tools evaluated, reliance metrics, verification tracking methods, primary empirical findings, and identified limitations. Given the wide methodological variation among qualitative interviews, observational eye-tracking studies, and controlled experiments, a statistical meta-analysis was inappropriate. We instead conducted a narrative synthesis structured around agreement, divergence, and methodology across research questions.
4. Results
Table 1. Included primary empirical studies (n=12)
| Study ID | Reference & Venue | Design Type | Sample (N) | Addressed RQs |
|---|---|---|---|---|
| [9] | Sabouri et al., ICSE 2025 | Mixed methods | 29 survey + 10 observed | RQ1, RQ2 |
| [10] | Wang et al., FAccT 2024 | Mixed methods | 17 interviews + 12 probe | RQ1 |
| [11] | Perry et al., CCS 2023 | RCT | 47 developers | RQ1, RQ3 |
| [12] | Klemmer et al., CCS 2024 | Qualitative | 27 interviews (+190 posts) | RQ1, RQ3 |
| [13] | Alami et al., TOSEM 2026 | Qualitative | 20 practitioners | RQ1, RQ2 |
| [14] | Tambon et al., EMSE 2025 | Quant. descriptive | 333 bugs; 50+ devs | RQ2 |
| [15] | Tang et al., VL/HCC 2024 | Non-randomized quant. | 28 developers | RQ1-RQ4 |
| [16] | Zhong et al., arXiv 2026 | Conceptual review | Agentic review model | RQ2 |
| [17] | Khojah et al., ASE 2026 | Non-randomized quant. | 32 developers | RQ2, RQ4 |
| [18] | Prather et al., ICER 2024 | Non-randomized quant. | 21 students | RQ3, RQ4 |
| [19] | Kazemitabaar et al., CHI 2023 | RCT | 69 novice learners | RQ3, RQ4 |
| [20] | Yu & Moon, AIES 2026 | Qualitative synthesis | Junior-to-senior path | RQ4 |
4.1 RQ1: Conceptualization and Measurement of Reliance
Across the surveyed papers, reliance is operationalized in four distinct ways that are rarely reconciled.
First, Sabouri et al. investigated developers' stated criteria for evaluating code suggestions, identifying code readability and perceived correctness as primary factors [9]. However, they uncovered an intention-behavior gap: what engineers said made code trustworthy did not match their real-time evaluation behavior during active programming [9].
Second, Wang et al. framed trust through interpersonal relationships, measuring perceptions of model capability, behavioral predictability, and perceived benevolence [10]. Their findings show that trust fluctuates with task difficulty and real-world consequences, rather than remaining static [10].
Third, Perry et al. adopted an outcome-based metric, defining trust as a participant's post-task confidence in the security of their submitted solution [11]. This metric closely resembles the acceptance proxies criticized by Baltes et al. [4] for overlooking cognitive trust calibration.
Fourth, qualitative investigations uncover a divide between stated skepticism and actual use. Klemmer et al. found that professional engineers openly distrust LLM output on security-critical problems, yet regularly use AI tools on those same tasks [12]. Alami et al. observed that while developers appreciate the lower emotional stress of receiving automated review comments instead of human critique, the high volume of generic AI suggestions increases cognitive load [13]. Engineers require higher confidence in a bot's domain competence before committing its suggested edits [13].
Overall, existing work shows that trust in AI assistants is dynamic and context-dependent, though it remains loosely coupled to developers' actual verification behaviors.
4.2 RQ2: Verification Methods and Behavior
Research on code verification focuses on error characteristics, observational telemetry, and self-reported inspection routines.
Tambon et al. established baseline targets for verification by identifying ten recurring defect patterns in LLM-generated code [14]. These range from incorrect framework usage to subtle boundary conditions that practitioners find difficult to catch during visual inspections [14].
Observational studies indicate that verification patterns depend on whether a developer knows code was generated by an AI. Tang et al. logged IDE telemetry and eye-tracking fixations, discovering that developers often do not realize code was suggested by an LLM unless the interface explicitly labels it [15]. When notifications were absent, developers rarely verified code systematically [15]. When suggestions were explicitly labeled as AI-generated, participants switched more frequently between comments and code and rewrote risky segments [15].
Khojah et al. confirmed this dynamic using an eye-tracking experiment where code provenance labels were manipulated [17]. Developers spent more time reading identical code when it had an AI badge, although visual dwell time did not necessarily increase defect discovery rates [17].
Conversely, studies by Klemmer et al. [12] and Alami et al. [13] rely on self-reported descriptions of review rigor without validating whether participants caught seeded bugs. Zhong et al. argued that as autonomous coding agents create more pull requests, verification will shift from human inspection to joint human-AI reviews [16].
4.3 RQ3: The Reliance-Verification Relationship
The literature reveals an unresolved contradiction regarding whether tool reliance degrades verification thoroughness.
In the first cluster, reliance correlates with poorer verification. Perry et al. observed that participants with access to an AI assistant wrote significantly more insecure code while reporting higher confidence in their solutions [11]. Prather et al. identified a similar dynamic among novice programmers: students with weaker metacognitive monitoring accepted incorrect suggestions uncritically, compounding logic errors [18].
In the second cluster, reliance does not harm verification. Kazemitabaar et al. tracked novice programmers using code generators and found that access to automated suggestions did not undermine their ability to modify or debug code on subsequent manual tasks [19]. Similarly, experienced engineers in Klemmer et al.'s qualitative study reported reviewing AI code with the same scrutiny applied to human pull requests [12].
Three methodological differences explain this divergence:
- Task Stakes: Perry et al. assigned open-ended, security-sensitive programming problems [11], whereas Kazemitabaar et al. used bounded, introductory programming exercises [19].
- Participant Seniority: Prather et al. observed early-stage computing students [18], while Klemmer et al. interviewed experienced industry engineers [12].
- Measurement Ground Truth: Studies finding degraded verification evaluated objective code correctness against seeded test suites [11], [18], whereas studies finding no degradation relied on self-reported inspection claims [12] or proxy transfer tasks [19].
Tang et al.'s work suggests an additional factor: because developers frequently miss that code was machine-generated unless an alert is present, divergent findings across studies may reflect how clearly each experiment highlighted code provenance [15].
4.4 RQ4: Moderating Factors
Three factors appear to moderate the relationship between reliance and verification:
- Task Stakes: High-stakes, security-critical assignments [11] show wider trust-accuracy mismatches than lower-stakes learning problems [19]. Because task types, participant pools, and outcome metrics differ across these studies, task stakes remains a plausible hypothesis rather than a confirmed cause.
- Developer Seniority: Novice developers struggle with metacognitive validation and fall into passive acceptance [18], whereas experienced engineers report applying structured verification habits [12], [13]. However, no single study has directly compared novices and senior engineers on identical tasks.
- Provenance Disclosure: Explicit provenance labels offer the clearest experimental evidence of an effect. Directly manipulating provenance indicators changes how engineers inspect code in IDEs [15], [17]. Yu and Moon caution that uncritical reliance among junior programmers could stunt their progression to senior engineering roles [20], highlighting the need to understand how provenance labels influence habits over time.
4.5 Cross-Cutting Synthesis
Current empirical work suffers from a recurring division: studies that measure reliance through nuanced psychological frameworks rely on surveys or interviews without tracking code-level verification [9], [10], [12], while studies that record code-level telemetry reduce reliance to simple acceptance rates [11], [15]. Aside from Tang et al. [15], few investigations measure trust calibration and verification telemetry within the same study.
5. Discussion
5.1 Hypothesized Conceptual Framework (Figure 3)
Claims that AI assistance degrades verification rigor rest on comparisons across studies with different tasks, metrics, and participant cohorts. To support future empirical work, Figure 3 organizes these candidate relationships into an explicit framework.
5.2 Points of Agreement
- Trust is Contextual: Trust in automated assistants is not a static user trait; it shifts based on task complexity, unfamiliarity with the language, and domain risks [9], [10].
- Provenance Badges Change Inspection Behavior: Explicitly labeling code as machine-generated changes how engineers visually read and edit that code [15], [17].
- AI Defects Require Targeted Scrutiny: LLM-generated bugs frequently look like clean code, meaning standard manual review methods may miss them [14].
5.3 Contradictions and Candidate Explanations (Table 3)
| Cluster | Representative Papers | Primary Findings | Candidate Explanations |
|---|---|---|---|
| Degraded Verification | Perry et al. [11]; Prather et al. [18] | Higher confidence paired with worse security and defect detection | High-stakes tasks; objective defect measurement; novice metacognitive struggles |
| Maintained Verification | Kazemitabaar et al. [19]; Klemmer et al. [12] | No measurable drops in code-fixing skill; self-reported review rigor preserved | Bounded tasks; senior developers; reliance on self-reported surveys |
Table 3. Synthesis of the RQ3 divergence and candidate explanations.
5.4 Practical Implications
- For Developers: Programmers should mentally categorize AI suggestions as requiring distinct review strategies, as knowing code came from an AI measurably improves inspection behavior [15].
- For Empirical Researchers: The field needs factorial study designs that measure both psychological trust and objective code-level telemetry in the same subjects.
- For Tool Designers: Current IDE extensions offer few visual aids to support code inspection [10]. Persistent provenance tags in diff viewers could help developers remember to review machine output critically [15].
6. Research Gaps and Future Research Agenda
Table 4. Evidence-based research gap matrix
| Research Area | Current Knowledge | Unresolved Questions | Recommended Future Study |
|---|
6.1 Proposed Factorial Study Protocol
To resolve the contradiction in RQ3, we outline an executable research protocol. This design has not yet been run; it translates the gaps in Table 4 into a concrete study.
6.2 Experimental Design: We propose a 2×2 mixed factorial design:
• Between-Subjects Factor 1 (Developer Seniority): Novices (undergraduate students with under one year of programming experience) vs. Experienced Engineers (industry practitioners with at least three years of professional experience).
• Between-Subjects Factor 2 (Provenance Labeling): Explicit AI Badge (visual indicators tagging generated suggestions) vs. Masked Provenance (suggestions presented without author labels).
• Within-Subjects Factor (Task Stakes): A high-stakes security patch vs. a low-stakes helper function.
Participants complete both task conditions in counterbalanced order, preventing disclosure carryover effects while allowing within-subject comparisons of task stakes.
6.3 Participants:
• Novice Cohort: Upper-division computer science undergraduates who have completed introductory algorithms and at most one internship.
• Experienced Cohort: Professional software developers with three or more years of industry experience and prior experience with AI coding tools.
• Sample Size: Determined through an a priori power calculation for mixed ANOVA models (targeting f=0.25, α=0.05, power=0.80).
• Research Ethics: Requires institutional ethics board review, informed consent, and anonymized data management.
6.4 Materials and Seeded Defects: Tasks are drawn from a matched-difficulty problem repository. Each problem contains a seeded defect based on Tambon et al.'s taxonomy [14] (such as an incorrect boundary condition or hallucinated API call), providing an objective ground truth for scoring verification success.
• High-Stakes Task: A security-sensitive routine (e.g., input sanitization in an authentication flow) where missed defects create an exploitable vulnerability.
• Low-Stakes Task: A string parsing or data transformation utility matched for cyclomatic complexity.
The underlying code suggestions remain identical across conditions; only the presence of the provenance badge changes.
6.5 Independent Variables:
• Developer Seniority: Evaluated via a pre-task screening questionnaire.
• Provenance Labeling: Controlled via the custom IDE extension (AI-generated badge vs. no label).
• Task Stakes: Controlled via assignment scenarios, with execution order counterbalanced.
6.6 Dependent Variables and Instrumentation:
• Trust and Reliance: Situational trust measured before and after tasks using an adapted inventory (capturing perceived competence and predictability per Wang et al. [10]). Behavioral reliance is measured via suggestion acceptance rates, latency before accepting suggestions, and the frequency of manual prompt edits.
• Verification Telemetry: Monitored via IDE interaction logs, including AST diff inspections, comment-to-code switches [15], test runs, and manual edits. Eye-tracking hardware measures fixation duration on lines containing seeded bugs.
• Verification Performance: Defect escape rate (whether the seeded bug was committed) and test coverage changes.
6.7 Procedure: Following onboarding, participants complete a brief IDE tutorial. They then complete both programming assignments under their assigned condition while telemetry and eye-tracking data are recorded. Brief post-task interviews capture their reasoning on acceptance and review decisions.
6.8 Analysis Plan: Data will be analyzed using mixed-effects models, treating seniority and provenance labeling as between-subjects factors, task stakes as a within-subjects factor, and participant identity as a random effect. The primary test is the three-way interaction: evaluating whether the divergence between Path A and Path B depends on the interaction between seniority, task stakes, and provenance labels.
7. Threats to Validity
• Search Methodology Limitations: Candidate studies were identified through targeted searches and citation snowballing rather than automated multi-database queries. Running full query exports may identify additional candidate papers.
• Single-Reviewer Screening: Paper selection and quality appraisals were conducted by a single reviewer without an inter-rater reliability metric, deviating from PRISMA guidelines.
• Database Access: Varied institutional database access may limit access to specific publisher repositories during full protocol replication.
• Publication Bias: Controlled trials with null results regarding AI tool reliance may be underrepresented in the literature.
• Tool Evolution: Older papers evaluated early code engines (such as original Codex models [11], [19]), which may not reflect how developers interact with newer models.
• Study Heterogeneity: The included papers span diverse qualitative, survey, and observational designs, preventing statistical meta-analysis.
• Quality Appraisal Data Gaps: Several short conference papers lacked the reporting detail needed to evaluate MMAT confound controls, leading to "Can't Tell" evaluations.
• Language Restrictions: Excluding non-English publications may have omitted relevant empirical studies.
8. Conclusion
This systematic literature review synthesized 12 primary empirical investigations examining developer reliance and verification behavior in LLM-assisted programming. Reliance is operationalized inconsistently across existing studies, often relying on simple acceptance rates instead of calibrated trust. Code verification remains understudied, relying primarily on small-cohort observational experiments.
Crucially, empirical results on the relationship between reliance and verification directly conflict: some studies find that automated assistance causes complacency and higher defect rates, while others show that verification rigor is preserved. While provenance disclosure currently has the strongest experimental support as a moderator, existing evidence cannot resolve this divergence.
The primary gap in this field is the absence of studies that measure psychological trust and code-level verification in the same participants. Conducting factorial experiments that manipulate task stakes, developer experience, and provenance labeling will help determine when automated tools support software engineering and when they introduce silent reliability risks.
References
- X. Hou et al., "Large language models for software engineering: A systematic literature review," ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 1–79, 2024.
- S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, "The impact of AI on developer productivity: Evidence from GitHub Copilot," arXiv:2302.06590, 2023.
- H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, "Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions," Commun. ACM, vol. 68, no. 2, pp. 96–105, 2025.
- S. Baltes, T. Speith, B. Chiteri, S. Mohsenimofidi, S. Chakraborty, and D. Buschek, "On the need to rethink trust in AI assistants for software development: A critical review," IEEE Trans. Softw. Eng., vol. 52, no. 4, pp. 1265–1281, Apr. 2026.
- S. Ferino, R. Hoda, J. Grundy, and C. Treude, "Novice developers' perspectives on adopting LLMs for software development: A systematic literature review," ACM Trans. Softw. Eng. Methodol., 2026.
- A. Mohamed, M. Assi, and M. Guizani, "The impact of LLM-assistants on software developer productivity: A systematic review and mapping study," ACM Trans. Softw. Eng. Methodol., 2026.
- M. J. Page et al., "The PRISMA 2020 statement: An updated guideline for reporting systematic reviews," BMJ, vol. 372, art. n71, 2021.
- Q. N. Hong et al., "The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers," Educ. Inf., vol. 34, no. 4, pp. 285–291, 2018.
- S. Sabouri, P. Eibl, X. Zhou, M. Ziyadi, N. Medvidovic, L. Lindemann, and S. Chattopadhyay, "Trust dynamics in AI-assisted development: Definitions, factors, and implications," in Proc. 2025 IEEE/ACM 47th Int. Conf. Software Engineering (ICSE), 2025, pp. 1678–1690.
- R. Wang, R. Cheng, D. Ford, and T. Zimmermann, "Investigating and designing for trust in AI-powered code generation tools," in Proc. 2024 ACM Conf. Fairness, Accountability, and Transparency (FAccT '24), 2024, pp. 1475–1493.
- N. Perry, M. Srivastava, D. Kumar, and D. Boneh, "Do users write more insecure code with AI assistants?" in Proc. 2023 ACM SIGSAC Conf. Computer and Communications Security (CCS '23), 2023, pp. 2785–2799.
- J. H. Klemmer et al., "Using AI assistants in software development: A qualitative study on security practices and concerns," in Proc. 2024 ACM SIGSAC Conf. Computer and Communications Security (CCS '24), 2024, pp. 2726–2740.
- A. Alami, N. Cassee, T. R. Silva, E. Paja, and N. A. Ernst, "Engagement in code review: Emotional, behavioral, and cognitive dimensions in peer vs. LLM interactions," ACM Trans. Softw. Eng. Methodol., 2026.
- F. Tambon, A. Moradi-Dakhel, A. Nikanjam, F. Khomh, M. C. Desmarais, and G. Antoniol, "Bugs in large language models generated code: An empirical study," Empir. Softw. Eng., vol. 30, art. 65, 2025.
- N. Tang, M. Chen, Z. Ning, A. Bansal, Y. Huang, C. McMillan, and T. J.-J. Li, "Developer behaviors in validating and repairing LLM-generated code using IDE and eye tracking," in Proc. 2024 IEEE Symp. Visual Languages and Human-Centric Computing (VL/HCC), 2024, pp. 40–46.
- S. Zhong, S. Noei, and B. Adams, "Human-AI synergy in agentic code review," arXiv:2603.15911, 2026.
- R. Khojah, F. Gomes de Oliveira Neto, M. Mohamad, J. Frattini, and P. Leitner, "Same scrutiny, more time: Eye tracking insights into reviewing LLM-labelled code," in Proc. 41st IEEE/ACM Int. Conf. Automated Software Engineering (ASE), 2026.
- J. Prather et al., "The widening gap: The benefits and harms of generative AI for novice programmers," in Proc. 2024 ACM Conf. Int. Computing Education Research (ICER), 2024, pp. 469–486.
- M. Kazemitabaar, X. Hou, A. Z. Henley, B. J. Ericson, D. Weintrop, and T. Grossman, "Studying the effect of AI code generators on supporting novice learners in introductory programming," in Proc. 2023 CHI Conf. Human Factors in Computing Systems, 2023, art. 455.
- S. Yu and T. Moon, "Who will become the next senior? How generative AI erodes the development pathway in software engineering," in Proc. 2026 AAAI/ACM Conf. AI, Ethics, and Society (AIES), 2026.
Appendix A: Publication Distribution (2025–2026)
This appendix provides the empirical distribution of the 16 records synthesized in this review, covering the 12 primary empirical studies and the 4 domain-specific background/methodological benchmarks published during the targeted 2025–2026 surveillance window.
Table A1. Temporal Distribution of Evaluated Studies (N = 16)
| Year | Primary Empirical Investigations (n=12) | Background / Methodological Benchmarks (n=4) | Total Evaluated Records |
|---|---|---|---|
| 2023 | Perry et al. [11]; Kazemitabaar et al. [19] | Peng et al. [2] | 3 |
| 2024 | Wang et al. [10]; Tang et al. [15]; Prather et al. [18] | Hou et al. [1] | 4 |
| 2025 | Sabouri et al. [9]; Tambon et al. [14] | Pearce et al. [3] | 3 |
| 2026 | Alami et al. [13]; Zhong et al. [16]; Khojah et al. [17]; Yu & Moon [20] | Baltes et al. [4] | 5 |
