Abstract
The ability to maximize rewards and minimize the costs of obtaining them is vital to making advantageous explore/exploit decisions. Exploratory decisions are theorized to be greater among individuals with attention-deficit/hyperactivity disorder (ADHD), potentially due to deficient catecholamine transmission. Here, we examined the effects of ADHD status and methylphenidate, a common ADHD medication, on explore/exploit decisions using a 6-armed bandit task. We hypothesized that ADHD participants would make more exploratory decisions than controls, and that MPH would reduce group differences. On separate study days, adults with (n = 26) and without (n = 23) ADHD completed the bandit task at baseline, and after methylphenidate or placebo in counter-balanced order. Explore/exploit decisions were modeled using reinforcement learning algorithms. ADHD participants made more exploratory decisions (i.e., chose options without the highest expected reward value) and earned fewer points than controls in all three study days, and methylphenidate did not affect these outcomes. Baseline exploratory choices were positively associated with hyperactive ADHD symptoms across all participants. These results support several theoretical models of increased exploratory choices in ADHD and suggest the unexplained variance in ADHD decisions may be due to less value tracking. The inability to suppress actions with little to no reward value may be a key feature of hyperactive ADHD symptoms.
Similar content being viewed by others
Log in or create a free account to read this content
Gain free access to this article, as well as selected content from this journal and more on nature.com
or
References
Wehmeier PM, Schacht A, Barkley RA. Social and emotional impairment in children and adolescents with ADHD and the impact on quality of life. J Adolesc Health. 2010;46:209–17.
Sobanski E, Bruggemann D, Alm B, Kern S, Philipsen A, Schmalzried H, et al. Subtype differences in adults with attention-deficit/hyperactivity disorder (ADHD) with regard to ADHD-symptoms, psychiatric comorbidity and psychosocial adjustment. Eur Psychiatry. 2008;23:142–9.
APA. Diagnostic and statistical manual of mental disorders: DSM-5. Washington, DC: American Psychiatric Association; 2013.
Biederman J, Mick E, Faraone SV. Age-dependent decline of symptoms of attention deficit hyperactivity disorder: Impact of remission definition and symptom type. Am J Psychiatry. 2000;157:816–8.
Moffitt TE, Houts R, Asherson P, Belsky DW, Corcoran DL, Hammerle M, et al. Is adult ADHD a childhood-onset neurodevelopmental disorder? Evidence from a four-decade longitudinal cohort study. Am J Psychiatry. 2015;172:967–77.
Simon V, Czobor P, Balint S, Meszaros A, Bitter I. Prevalence and correlates of adult attention-deficit hyperactivity disorder: meta-analysis. Br J Psychiatry. 2009;194:204–11.
Wilens TE, Biederman J, Faraone SV, Martelon M, Westerberg D, Spencer TJ. Presenting ADHD symptoms, subtypes, and comorbid disorders in clinically referred adults with ADHD. J Clin Psychiatry. 2009;70:1557–62.
Willcutt EG, Pennington BF, Olson RK, Chhabildas N, Hulslander J. Neuropsychological analyses of comorbidity between reading disability and attention deficit hyperactivity disorder: In search of the common deficit. Dev Neuropsychol. 2005;27:35–78.
Boonstra AM, Oosterlaan J, Sergeant JA, Buitelaar JK. Executive functioning in adult ADHD: a meta-analytic review. Psychol Med. 2005;35:1097–108.
Hervey AS, Epstein JN, Curry JF. Neuropsychology of adults with attention-deficit/hyperactivity disorder: a meta-analytic review. Neuropsychology. 2004;18:485–503.
Sonuga-Barke EJS. The dual pathway model of AD/HD: an elaboration of neuro-developmental characteristics. Neurosci Biobehav Rev. 2003;27:593–604.
Tripp G, Wickens JR. Neurobiology of ADHD. Neuropharmacology. 2009;57:579–89.
Luman M, Tripp G, Scheres A. Identifying the neurobiology of altered reinforcement sensitivity in ADHD: a review and research agenda. Neurosci Biobehav Rev. 2010;34:744–54.
Jackson JN, MacKillop J. Attention-deficit/hyperactivity disorder and monetary delay discounting: a meta-analysis of case-control studies. Biol Psychiatry Cogn Neurosci Neuroimaging. 2016;1:316–25.
Luman M, Oosterlaan J, Sergeant JA. The impact of reinforcement contingencies on AD/HD: a review and theoretical appraisal. Clin Psychol Rev. 2005;25:183–213.
Aarts E, van Holstein M, Cools R. Striatal dopamine and the interface between motivation and cognition. Front Psychol. 2011;2:163.
Fusar-Poli P, Rubia K, Rossi G, Sartori G, Balottin U. Striatal dopamine transporter alterations in ADHD: pathophysiology or adaptation to psychostimulants? A meta-analysis. Am J Psychiatry. 2012;169:264–72.
Volkow ND, Wang GJ, Kollins SH, Wigal TL, Newcorn JH, Telang F, et al. Evaluating dopamine reward pathway in ADHD clinical implications. J Am Med Assoc. 2009;302:1084–91.
Volkow ND, Wang GJ, Newcorn JH, Kollins SH, Wigal TL, Telang F, et al. Motivation deficit in ADHD is associated with dysfunction of the dopamine reward pathway. Mol Psychiatry. 2011;16:1147–54.
Challman TD, Lipsky JJ. Methylphenidate: Its pharmacology and uses. Mayo Clin Proc. 2000;75:711–21.
Faraone SV, Spencer T, Aleardi M, Pagano C, Biederman J. Meta-analysis of the efficacy of methylphenidate for treating adult attention-deficit/hyperactivity disorder. J Clin Psychopharmacol. 2004;24:24–29.
Castells X, Blanco-Silvente L, Cunill R. Amphetamines for attention deficit hyperactivity disorder (ADHD) in adults. Cochrane Datab Syst Rev. 2018:CD007813.
Coghill DR, Seth S, Pedroso S, Usala T, Currie J, Gagliano A. Effects of methylphenidate on cognitive functions in children and adolescents with attention-deficit/hyperactivity disorder: evidence from a systematic review and a meta-analysis. Biol Psychiatry. 2014;76:603–15.
Pievsky MA, McGrath RE. Neurocognitive effects of methylphenidate in adults with attention-deficit/hyperactivity disorder: a meta-analysis. Neurosci Biobehav Rev. 2018;90:447–55.
Cohen JD, McClure SM, Yu AJ. Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos Trans R Soc B. 2007;362:933–42.
Sutton RS, Barto AG. Reinforcement learning: an introduction. Cambridge, Mass.: MIT Press. xviii; 1998. 322 p.
Graybiel AM. Habits, rituals, and the evaluative brain. Annu Rev Neurosci. 2008;31:359–87.
Daw ND, Doya K. The computational neurobiology of learning and reward. Curr Opin Neurobiol. 2006;16:199–204.
Ziegler S, Pedersen ML, Mowinckel AM, Biele G. Modelling ADHD: a review of ADHD theories through their predictions for computational models of decision-making and reinforcement learning. Neurosci Biobehav Rev. 2016;71:633–56.
Frank MJ, Santamaria A, O’Reilly RC, Willcutt E. Testing computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorder. Neuropsychopharmacology. 2007;32:1583–99.
Sagvolden T, Johansen EB, Aase H, Russell VA. A dynamic developmental theory of attention-deficit/hyperactivity disorder (ADHD) predominantly hyperactive/impulsive and combined subtypes. Behav Brain Sci. 2005;28:397-+.
Seeman P, Madras B. Methylphenidate elevates resting dopamine which lowers the impulse-triggered release of dopamine: a hypothesis. Behav Brain Res. 2002;130:79–83.
Hauser TU, Fiore VG, Moutoussis M, Dolan RJ. Computational psychiatry of ADHD: neural gain impairments across marrian levels of analysis. Trends Neurosci. 2016;39:63–73.
Hauser TU, Iannaccone R, Ball J, Mathys C, Brandeis D, Walitza S, et al. Role of the medial prefrontal cortex in impaired decision making in juvenile attention-deficit/hyperactivity disorder. JAMA Psychiatry. 2014;71:1165–73.
Sethi A, Voon V, Critchley HD, Cercignani M, Harrison NA. A neurocomputational account of reward and novelty processing and effects of psychostimulants in attention deficit hyperactivity disorder. Brain. 2018;141:1545–57.
Conners CK, Erhardt D, Sparrow E. The Conners adult ADHD rating scale (CAARS). Multi-Health Systems, Inc: Toronto; 1998.
Epstein J, Johnson DE, Conners CK. Conners’ adult ADHD diagnostic interview for DSM-IV (CAADID). MHS: New York; 2001.
Sheehan D, Janavas J, Harnett-Sheehan K, Sheehan M, Gray C. MINI International Neuropsychiatric Interview (English Version 6.0.0). Tampa, FL: University of South Florida College of Medicine; 2009.
Daw ND, O’Doherty JP, Dayan P, Seymour B, Dolan RJ. Cortical substrates for exploratory decisions in humans. Nature. 2006;441:876–9.
Gittins JC, Jones DH. A dynamic allocation index for the sequential design of experiments, In: Gani JM, editor. Progress in statistics. Amsterdam: North Holland; 1974.
Gittins JC. Multi-armed bandit allocation indices. Wiley-Interscience series in systems and optimization. Chichester; New York: Wiley; 1989. p. 252.
Berry DA, Fristedt B. Bandit problems: sequential allocation of experiments. Monographs on statistics and applied probability. New York: Chapman and Hall, London; 1985.
Whittle P. Restless bandits: activity allocation in a changing world celebration of applied probability. J Appl Probab. 1988;25A:287–98.
Pearson JM, Hayden BY, Raghavachari S, Platt ML. Neurons in posterior cingulate cortex signal exploratory decisions in a dynamic multioption choice task. Curr Biol. 2009;19:1532–7.
Jepma M, te Beek ET, Wagenmakers E-J, van Gerven JMA, Nieuwenhuis S. The role of the noradrenergic system in the exploration-exploitation trade-off: a psychopharmacological study. Front Hum Neurosci. 2010;4:1–13.
Addicott MA, Pearson JM, Wilson J, Platt ML, McClernon FJ. Smoking and the bandit: a preliminary study of smoker and nonsmoker differences in exploratory behavior measured with a multiarmed bandit task. Exp Clin Psychopharmacol. 2013;21:66–73.
Brainard DH. The psychophysics toolbox. Spat Vis. 1997;10:433–6.
Addicott MA, Schechter JC, Sapyta JJ, Selig JP, Kollings SH, Weiss MD. Methylphenidate increases willingness to perform effort in adults with ADHD. Pharmacol Biochem Behav. 2019;183:14–21.
Jepma M, Nieuwenhuis S. Pupil diameter predicts changes in the exploration-exploitation tradeoff: evidence for the adaptive gain theory. J Cogn Neurosci. 2011;23:1587–96.
Anderson BDO, Moore JB. Optimal filtering. Englewood Cliffs, NJ: Prentice-Hall, 1979.
Pietrzak RH, Mollica CM, Maruff P, Snyder PJ. Cognitive effects of immediate-release methylphenidate in children with attention-deficit/hyperactivity disorder. Neurosci Biobehav Rev. 2006;30:1225–45.
Linssen AMW, Sambeth A, Vuurman EFPM, Riedel WJ. Cognitive effects of methylphenidate in healthy volunteers: a review of single dose studies. Int J Neuropsychopharmacol. 2014;17:961–77.
Kofler MJ, Rapport MD, Sarver DE, Raiker JS, Orban SA, Friedman LM, et al. Reaction time variability in ADHD: a meta-analytic review of 319 studies. Clin Psychol Rev. 2013;33:795–811.
Tamm L, Narad ME, Antonini TN, O’Brien KM, Hawk LW, Epstein JN. Reaction time variability in ADHD: a review. Neurotherapeutics. 2012;9:500–8.
Losier BJ, McGrath PJ, Klein RM. Error patterns on the continuous performance test in non-medicated and medicated samples of children with and without ADHD: a meta-analytic review. J Child Psychol Psychiatry Allied Discip. 1996;37:971–87.
Aston-Jones G, Cohen JD. Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. J Comp Neurol. 2005;493:99–110.
Addicott MA, Pearson JM, Sweitzer MM, Barack DL, Platt ML. A primer on foraging and the explore/exploit trade-off for psychiatry research. Neuropsychopharmacology. 2017;42:1931–9.
Beeler JA, Daw N, Frazier CRM, Zhuang XX. Tonic dopamine modulates exploitation of reward learning. Front Behav Neurosci. 2010;4:170.
Rutledge RB, Lazzaro SC, Lau B, Myers CE, Gluck MA, Glimcher PW. Dopaminergic drugs modulate learning rates and perseveration in Parkinson’s patients in a dynamic foraging task. J Neurosci. 2009;29:15104–14.
Chakroun K, Mathar D, Wiehler A, Ganzer F, Peters J. Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making. bioRxiv, 2019:706176.
Kollins SH, English J, Robinson R, Hallyburton M, Chrisman AK. Reinforcing and subjective effects of methylphenidate in adults with and without attention deficit hyperactivity disorder (ADHD). Psychopharmacology. 2009;204:73–83.
Kollins SH, Schoenfelder E, English JS, McClernon FJ, Dew RE, Lane SD. Methylphenidate does not influence smoking-reinforced responding or attentional performance in adult smokers with and without attention deficit hyperactivity disorder (ADHD). Exp Clin Psychopharmacol. 2013;21:375–84.
Sweitzer MM, Kollins SH, Kozink RV, Hallyburton M, English J, Addicott MA, et al. ADHD, smoking withdrawal, and inhibitory control: results of a neuroimaging study with methylphenidate challenge. Neuropsychopharmacology. 2018;43:851.
Mitchell JT, Zylowska L, Kollins SH. Mindfulness meditation training for attention-deficit/hyperactivity disorder in adulthood: current empirical support, treatment overview, and future directions. Cogn Behav Pract. 2015;22:172–91.
Author information
Authors and Affiliations
Contributions
MAA, principal investigator, designed and conducted the study, organized and analyzed the data, and wrote the majority of the manuscript. JMP provided support for the reinforcement learning algorithms and interpretation of the results. JCS and JJS performed clinical evaluations of the ADHD participants. SHK and MDW guided the development of the protocol and provided study oversight. All authors approved the final manuscript before submission.
Corresponding author
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Addicott, M.A., Pearson, J.M., Schechter, J.C. et al. Attention-deficit/hyperactivity disorder and the explore/exploit trade-off. Neuropsychopharmacol. 46, 614–621 (2021). https://doi.org/10.1038/s41386-020-00881-8
Received:
Revised:
Accepted:
Published:
Version of record:
Issue date:
DOI: https://doi.org/10.1038/s41386-020-00881-8
This article is cited by
-
A review on exploration–exploitation trade-off in psychiatric disorders
BMC Psychiatry (2025)
-
Hyperactive ADHD symptoms are associated with increased variability in thought content in less constrained contexts
Scientific Reports (2025)
-
Comparative behavioural and neurofunctional analysis of executive functions in adults with obsessive-compulsive disorder and attention-deficit/hyperactivity disorder using a virtual reality task
Virtual Reality (2025)
-
Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder
Translational Psychiatry (2024)
-
How does apathy impact exploration-exploitation decision-making in older patients with neurocognitive disorders?
npj Aging (2023)
