Advertisement

Preventing misestimation of transcranial magnetic stimulation motor threshold with MTAT 2.0

  • Lari M. Koponen
    Affiliations
    Centre for Human Brain Health, School of Psychology, University of Birmingham, Birmingham, UK
    Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
    Search for articles by this author
  • Angel V. Peterchev
    Correspondence
    Corresponding author.
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
    Department of Biomedical Engineering, Duke University, Durham, NC, USA
    Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
    Department of Neurosurgery, Duke University, Durham, NC, USA
    Search for articles by this author
Open AccessPublished:August 05, 2022DOI:https://doi.org/10.1016/j.brs.2022.07.057

      1. Introduction

      We discovered situations in which the popular transcranial magnetic stimulation (TMS) motor threshold (MT) assessment software tool MTAT 2.0 [
      • Awiszus F.
      • Borckardt J.J.
      TMS motor threshold assessment tool (MTAT 2.0).
      ] can produce a large misestimation of the MT. We describe the issue, study it computationally, and suggest ways to address it.
      MTAT 2.0 estimates the MT with a maximum likelihood parametric estimation by sequential testing (ML-PEST) algorithm [
      • Awiszus F.
      TMS and threshold hunting.
      ]. MTAT is known to produce accurate estimates of the MT—within 5% of the maximum stimulator output (MSO)—with just 14–30 pulses [
      • Awiszus F.
      Fast estimation of transcranial magnetic stimulation motor threshold: is it safe?.
      ,
      • Julkunen P.
      Mobile application for adaptive threshold hunting in transcranial magnetic stimulation.
      ], much less than the number of pulses required with the conventional approach based on the relative frequency of responses, which requires typically around 75 pulses [
      • Tranulis C.
      • Guéguen B.
      • Pham-Scottez A.
      • Vacheron M.N.
      • Cabelguen G.
      • Costantini A.
      • et al.
      Motor threshold in transcranial magnetic stimulation: comparison of three estimation methods.
      ].
      During a pilot TMS experiment with a healthy 31-year-old male participant, we observed substantial underestimation of the MT with MTAT. After our default procedure of 30 pulses, the estimated resting MT was about 50% MSO instead of the expected value of about 70% MSO, which we knew a priori for the participant. During this failed estimation process, the stimulator output remained always well below the true MT, and the algorithm converged to its usual narrow estimation range centered on the underestimated MT value. There was no obvious explanation to this anomalous estimation process in either the TMS or electromyography (EMG) systems. There was, however, an obvious lack of motor evoked potentials (MEPs) at the estimated MT.

      2. Methods

      To analyze this issue, we developed a method to test the MTAT 2.0 software with virtual subjects, and we were able to recreate the observed behavior for some of our virtual subjects. First, we generated a set of 100 virtual subjects with a stochastic statistical model of MEPs [
      • Goetz S.M.
      • Alavi S.M.M.
      • Deng Z.-D.
      • Peterchev A.V.
      Statistical model of motor-evoked potentials.
      ]. For each virtual subject, we simulated a set of 100 possible MEP responses at each stimulator output from 0 to 100% MSO. The set of virtual subjects and their MEP responses is included in the supplementary material. For each subject, we defined the ground truth MT as the lowest intensity at which at least 50 out of the 100 simulated MEPs had a peak-to-peak amplitude of at least 50 μV. The mean MT of the virtual subjects was 66% MSO (range 44–84% MSO).
      To automate the use of MTAT 2.0 for running many trials in the virtual subjects, we implemented a screen scraping software, leveraging an auto clicker and keyboard (Java Abstract Window Toolkit) combined with optical character recognition (Computer Vision Toolbox, R2018a, MATLAB) to read the MTAT output. With this software setup, we ran 200 MT estimations for each virtual subject, for a total of 20000 MT estimates. Half of the estimations were run with the default MTAT mode “threshold estimation in the range from 20% to 80% stimulator output” and the other half were run in the “refinement of a raw threshold estimate” mode. For the second half, the raw threshold estimates were generated by adding a uniform random offset of 5–15% MSO to the true MT, which corresponds approximately to the intensity typically used to localize the motor hotspot before the MT estimation.
      We chose to terminate each MT estimation trial at 30 TMS pulses, since this appears to be the most conservative recommended value for this ML-PEST implementation. In the default mode, MTAT 2.0 shows the MT estimate in red after 12 pulses, yellow after 16 pulses, and green (accompanied with a beep) after 30 pulses. It has been suggested that 14 pulses are sufficient for the true MT to be inside the 95% confidence interval (CI) [
      • Awiszus F.
      Fast estimation of transcranial magnetic stimulation motor threshold: is it safe?.
      ], and that 20 pulses produce a sufficiently accurate estimate for all subjects [
      • Awiszus F.
      On relative frequency estimation of transcranial magnetic stimulation motor threshold.
      ]. Finally, another app based on ML-PEST specifies 14 pulses as the minimum and 30 pulses as the upper limit for sampling, and 20 pulses are used in its demonstration [
      • Julkunen P.
      Mobile application for adaptive threshold hunting in transcranial magnetic stimulation.
      ].

      3. Results

      In 95% of the trials, after 30 TMS pulses, the MT estimate was between −4 and + 3% MSO and −3 and +3% MSO of the ground truth for the default mode and the mode with an initial guess, respectively.
      In the default mode, however, the MT estimation error varied greatly across the virtual subjects (Fig. 1ab). For 21 subjects, the maximum absolute error was over 5% MSO. For the worst-case virtual subject, the 95% CI was from −24 to +1% MSO and the full range of estimates was from −30 to +2% MSO for the default MTAT mode. The mechanism of MT underestimation in this subject is illustrated in Fig. 1cd, which shows that MEPs occurred at an intensity well below the true MT in about 10% of the samples. Such MEP distributions are indeed observed in experiments; for example, excluded participant #13 in Ref. [
      • Kallioniemi E.
      • Awiszus F.
      • Pitkänen M.
      • Julkunen P.
      Fast acquisition of resting motor threshold with a stimulus–response curve – possibility or hazard for transcranial magnetic stimulation applications?.
      ], shown in their supplementary data, had a similarly wide range of MEP responses. When an MEP occurs for a stimulator output well below the true MT at the beginning of the estimation sequence, the MTAT algorithm will only sample low stimulator outputs even if very few later MEPs are observed. For example, in the purple curve in Fig. 1c, the first pulse at 45% MSO resulted in an MEP, followed by only one more MEP on the third pulse. After the suggested minimum number of 14 pulses [
      • Awiszus F.
      Fast estimation of transcranial magnetic stimulation motor threshold: is it safe?.
      ], MTAT reported an MT estimate of 45.40% MSO with a CI of 42.92–47.79% MSO, which is more than 23% MSO below the true MT value of 71% MSO. After 30 pulses, the algorithm reported an MT estimate of 52.71% MSO with a CI of 50.90–54.47% MSO, still more than 16% MSO below the true MT value. This behavior is similar to what we observed during our pilot experiment.
      Fig. 1
      Fig. 1MT estimation error of MTAT 2.0 in a computational study with 100 virtual subjects. (a) MT estimation error range across 100 trials in each of the 100 virtual subjects after 30 TMS pulses in each of two MTAT 2.0 modes: (red lines) default mode with fixed initial intensity at 45% MSO, and (blue lines) “refinement of a raw threshold estimate” starting with an initial intensity of 5–15% MSO above the true MT, corresponding to typical intensities used to localize the motor hotspot. Light red and light blue lines denote trials with 2–8 MEPs during the last 10 pulses. The subjects are ordered by their error range in the respective modes. (b) Corresponding scatterplot of the estimation error for each trial versus the difference between the initial intensity and the true MT. (c) 100 trials of MTAT MT estimation in the default mode for the virtual subject with the largest error range and a true MT of 71% MSO (horizontal green line). Purple curve shows an instance of a sampling sequence that resulted in inaccurate MT estimation; error bars are the MT CI reported by MTAT after each pulse; yellow stars denote MEP generation; and gray curves depict the other 99 trials. (d) Histogram of the MT estimates from the 100 trials in default mode shown in (c) (red bars) and the results for the same subject if run with an initial guess (blue outline). (e) 100 trials of the subject with the largest error range for MTAT's “refinement of a raw threshold estimate” mode with initial guesses 5–15% MSO above the true MT. (f) Histogram of the MT estimates from the 100 trials with initial guess shown in (e) (blue bars) and the results for the same subject if run in the default mode (red outline). The color convention in the histograms matches that in (a) and (b).
      The use of an initial guess in the “refinement of a raw threshold estimate” MTAT mode removed very large errors and reduced the overall variability of the MT estimates (Fig. 1ab), with the worst-case subject for this mode having 95% CI of −5 to +7% MSO (full range −7 to +9% MSO, Fig. 1ef). Nonetheless, even with the initial guess, 14 virtual subjects had greater than 5% MSO error for their worst-case MT estimate.
      Another important observation is that anomalous MT estimation can be identified from the apparent shortage or excess of MEPs during the MT estimation. In the default mode, 1.7% of all trials had 0 or 1 MEPs during the last 10 pulses (range 0–12% across subjects). Rejecting all such trials removed most misestimations and reduced the maximum MT estimation error from −30% MSO to −13% MSO and the number of subjects with estimation error >5% MSO from 21 to 16. With an initial MT guess, 1.5% of all trials had 9 or 10 MEPs during the last 10 pulses (range 0–7% across subjects). Rejecting these trials did not reduce the full range of misestimation (−7 to +10% MSO) but decreased the number of subjects with greater than 5% MSO error from 14 to 13.

      4. Conclusion

      In some cases when the initial TMS intensity is far from the true MT, the default mode (“threshold estimation in the range from 20% to 80% stimulator output”) of the MTAT 2.0 ML-PEST algorithm can produce a large misestimation of the MT for particular subjects and/or muscles. To mitigate this issue, we recommend inputting the TMS intensity used for motor hotspot localization as an initial guess with the MTAT mode “refinement of a raw threshold value.” Doing so removed all MT misestimates larger than 10% MSO in our simulations.
      In our simulations the estimation results were quantified after a fixed number of 30 pulses. With a good initial estimate, MTAT may converge to the final estimate with fewer pulses; in our results there was little improvement after about 25 pulses, although we did not characterize this.
      For additional protection against misestimation, we also recommend rejecting MT estimates with either a very low or a very high number of MEPs during the last 10 pulses (reported in the top 10 rows of column “TMSIE” in MTAT). In particular, over the last 10 pulses, the number of MEPs should be close to 5; observing 0, 1, 9, or 10 responses means that the MT estimate is likely inaccurate. Indeed, such validation based on the 5-out-of-10-responses criterion has been applied when using MTAT [
      • Julkunen P.
      • Säisänen L.
      • Danner N.
      • Niskanen E.
      • Hukkanen T.
      • Mervaala E.
      • et al.
      Comparison of navigated and non-navigated transcranial magnetic stimulation for motor cortex mapping, motor threshold and motor evoked potentials.
      ]. Whilst these solutions are not perfect, in our simulations their combination produced MT estimates accurate within 10% MSO for all the virtual subjects and within 5% MSO for 87% of the virtual subjects.
      A longer-term solution would be to develop a method that can assess the goodness of its MT estimate and adjust the number of pulses accordingly. Such solution would also help reduce other sources of uncertain MT estimates not suppressed by adding an initial guess. For example, for subjects and/or muscles with shallower input–output curves, the standard amount of 30 pulses produces a broad, near-uniform MT estimate distribution whose width exceeds the desired 5% MSO accuracy even with a good initial guess.
      Finally, in our simulations the virtual subjects were extrapolated from a model fitted to experimental data for the right first dorsal interosseus muscle of 12 healthy volunteers [
      • Goetz S.M.
      • Alavi S.M.M.
      • Deng Z.-D.
      • Peterchev A.V.
      Statistical model of motor-evoked potentials.
      ]. The extreme cases of this distribution might exaggerate certain features of the MEP generation probability and fail to capture others. To develop improved MT estimation methods, more open data on experimentally recorded MEPs is needed for model tuning and validation.

      Ethics statement

      The experiment was approved by Institutional Review Board of Duke University Health System and the participant provided their informed consent.

      Declaration of competing interest

      The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Lari M. Koponen is an inventor on patents on TMS technology and has received patent royalties from Nexstim. Angel V. Peterchev is an inventor on patents on TMS technology and has received research funding, travel support, patent royalties, consulting fees, equipment loans, hardware donations, and/or patent application support from Rogue Research, Magstim, MagVenture, Neuronetics, BTL Industries, Advise Connect Inspire, and Ampa.

      Acknowledgments

      Research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under Award Numbers RF1MH114268 and R01MH111865 , and by the European Union's Horizon 2020 research and innovation program under Marie Sklodowska-Curie grant agreement No. 101027633. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

      Appendix A. Supplementary data

      The following are the Supplementary data to this article:

      References

        • Awiszus F.
        • Borckardt J.J.
        TMS motor threshold assessment tool (MTAT 2.0).
        2011
        • Awiszus F.
        TMS and threshold hunting.
        in: Paulus W. Tergau F. Nitsche M.A. Rothwell J.G. Ziemann U. Hallett M. Transcranial Magn. Stimul. Transcranial Direct Curr. Stimul. Elsevier, 2003: 13-23https://doi.org/10.1016/S1567-424X(09)70205-3
        • Awiszus F.
        Fast estimation of transcranial magnetic stimulation motor threshold: is it safe?.
        Brain Stimul. 2011; 4: 58-59https://doi.org/10.1016/j.brs.2010.09.004
        • Julkunen P.
        Mobile application for adaptive threshold hunting in transcranial magnetic stimulation.
        IEEE Trans Neural Syst Rehabil Eng. 2019; 27: 1504-1510https://doi.org/10.1109/TNSRE.2019.2925904
        • Tranulis C.
        • Guéguen B.
        • Pham-Scottez A.
        • Vacheron M.N.
        • Cabelguen G.
        • Costantini A.
        • et al.
        Motor threshold in transcranial magnetic stimulation: comparison of three estimation methods.
        Clin Neurophysiol. 2006; 36: 1-7https://doi.org/10.1016/j.neucli.2006.01.005
        • Goetz S.M.
        • Alavi S.M.M.
        • Deng Z.-D.
        • Peterchev A.V.
        Statistical model of motor-evoked potentials.
        IEEE Trans Neural Syst Rehabil Eng. 2019; 27: 1539-1545https://doi.org/10.1109/TNSRE.2019.2926543
        • Awiszus F.
        On relative frequency estimation of transcranial magnetic stimulation motor threshold.
        Clin Neurophysiol. 2012; 123: 2319-2320https://doi.org/10.1016/j.clinph.2012.04.014
        • Kallioniemi E.
        • Awiszus F.
        • Pitkänen M.
        • Julkunen P.
        Fast acquisition of resting motor threshold with a stimulus–response curve – possibility or hazard for transcranial magnetic stimulation applications?.
        Clin Neurophysiol Pract. 2022; 7: 7-15https://doi.org/10.1016/j.cnp.2021.10.005
        • Julkunen P.
        • Säisänen L.
        • Danner N.
        • Niskanen E.
        • Hukkanen T.
        • Mervaala E.
        • et al.
        Comparison of navigated and non-navigated transcranial magnetic stimulation for motor cortex mapping, motor threshold and motor evoked potentials.
        Neuroimage. 2009; 44: 790-795https://doi.org/10.1016/j.neuroimage.2008.09.040