🔍 Read the full analysis: Why ChatGPT’s Data Science Benefits For Consultants May Not Last on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. After ChatGPT was removed, researchers found no evidence that the users performed better on technical questions without it.
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT improved their scores on coding, statistics and prediction tasks, but researchers found no evidence of better performance when participants later answered technical questions without the chatbot. Published in Nature Human Behaviour on October 26, 2026, the study tests whether generative AI can help knowledge workers do unfamiliar work—and whether that boost remains when the tool is gone.
The trial allocated 986 consultants to a ChatGPT group or a control group trained to use Google and sites including Stack Overflow and Khan Academy. A total of 487 participants submitted work on both assigned tasks, forming the main analysis sample. The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. Participants completed two 90-minute tasks drawn from coding, statistics and prediction.
Compared with the control group, ChatGPT users scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. They also finished coding and prediction faster on average, by about nine and 14 minutes respectively; the researchers reported no significant time difference on statistics. The tasks were designed so that ChatGPT could guide users but could not independently answer every question correctly.
The researchers compared participants’ work with that of 44 BCG data scientists, who completed the tasks without AI. The ChatGPT group as a whole remained below the data scientists’ average on all three tasks. After the tasks, participants answered five technical questions without ChatGPT. The authors report that the ChatGPT group did not outperform the control group on that follow-up. They also report increased confidence among ChatGPT users that the system could solve problems it could not; the authors describe that finding as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
A Boost That Depends on the Tool
The findings distinguish doing a task with AI from acquiring a skill that can be used independently. For employers, that difference could affect how they evaluate training, staffing and the quality of work produced with generative AI. A worker may complete a technical assignment more effectively with a chatbot without being able to repeat the work unaided.
The study does not show that ChatGPT has no value as a workplace tool. It found sizable score gains on the tasks tested, and some participants completed work faster. But the results caution against treating those gains as proof that employees have been permanently upskilled. Businesses using AI-supported work may still need human review and appropriate expertise, especially where correct analysis matters.
The confidence result also points to a possible concern: users may become more assured of the tool’s ability than its performance warrants. Because the authors call this analysis exploratory, it should not be treated as a settled finding about all ChatGPT users or work settings.
Products Worth Considering
DJI Mini 3 Drone with DJI RC and 3‑Axis Mechanical Gimbal
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
Samsung 27-inch Odyssey OLED G6 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The experiment was preregistered on March 13, 2024, before data collection. It ran at BCG in late March and early April of that year. Participants had at least a bachelor’s degree, and the authors say most reported no coding experience. Each participant completed two of three tasks: cleaning and combining datasets with Python; evaluating a statistical model and working with probabilities; or building a model related to the predictability of international football matches.
BCG data scientists developed the tasks with OpenAI’s Economic Impacts Research team. The researchers’ design aimed to make ChatGPT useful as a guide without allowing it to provide correct answers to every question on its own. The expert benchmark consisted of 44 BCG data scientists working without AI. The study’s results therefore concern a particular group, set of tasks and version of the technology, rather than every workplace use of generative AI.
The paper’s title refers to the temporary nature of the measured performance gain. In the authors’ framing, AI can extend what a worker can do while it is available, much as an exoskeleton assists its wearer. The follow-up test was included to examine whether that advantage carried over after access was removed.
“The authors report that they “find no evidence” that the ChatGPT group did better on the technical questions answered without the tool.”
— The study authors
Limits of the Skill Follow-Up
The follow-up consisted of five technical questions and does not establish whether longer or different training could produce lasting gains. The reported lack of evidence for an advantage is not proof that no participant learned anything; it means the study did not find that the ChatGPT group performed better than the control group on this test.
The findings are also bounded by the BCG participants, the tasks, the study period and GPT-4 as used in the trial. The source material does not establish how the results would generalize to other occupations, newer AI systems, longer-term use or workplace settings with different training and oversight. The confidence finding was exploratory, and its broader meaning remains uncertain.
What Employers Should Watch
The study reports completed trial results rather than a forthcoming experiment or scheduled company action. For employers, the immediate question is how to distinguish AI-assisted output from independent competence when assigning work and evaluating training. The findings support tracking accuracy and review needs alongside speed and task completion, rather than assuming that using a tool has taught the underlying skill.
Further research would be needed to test whether more extensive instruction, repeated practice or different AI tools lead to durable unaided performance. Until then, the evidence here supports a narrower conclusion: ChatGPT improved results on selected tasks at BCG, while the study found no evidence that the advantage carried over to its brief test without ChatGPT.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. The researchers found no evidence of better results for the ChatGPT group on a later technical test completed without the tool.
How much higher were the ChatGPT group’s task scores?
Compared with the control group, ChatGPT users scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction in this trial.
Did the ChatGPT group perform as well as data scientists?
Not overall. The group’s average remained below the benchmark set by 44 BCG data scientists, who completed the tasks without AI.
Does the study prove ChatGPT cannot teach lasting skills?
No. The study found no evidence of an advantage on its five-question follow-up test without ChatGPT. That result does not establish what longer training or other forms of practice might achieve.
Source: ThorstenMeyerAI.com
