Preventing misestimation of transcranial magnetic stimulation motor threshold with MTAT 2.0

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.


Introduction
We discovered situations in which the popular transcranial magnetic stimulation (TMS) motor threshold (MT) assessment software tool MTAT 2.0 [1] can produce a large misestimation of the MT. We describe the issue, study it computationally, and suggest ways to address it.
MTAT 2.0 estimates the MT with a maximum likelihood parametric estimation by sequential testing (ML-PEST) algorithm [2]. MTAT is known to produce accurate estimates of the MTdwithin 5% of the maximum stimulator output (MSO)dwith just 14e30 pulses [3,4], much less than the number of pulses required with the conventional approach based on the relative frequency of responses, which requires typically around 75 pulses [5].
During a pilot TMS experiment with a healthy 31-year-old male participant, we observed substantial underestimation of the MT with MTAT. After our default procedure of 30 pulses, the estimated resting MT was about 50% MSO instead of the expected value of about 70% MSO, which we knew a priori for the participant. During this failed estimation process, the stimulator output remained always well below the true MT, and the algorithm converged to its usual narrow estimation range centered on the underestimated MT value. There was no obvious explanation to this anomalous estimation process in either the TMS or electromyography (EMG) systems. There was, however, an obvious lack of motor evoked potentials (MEPs) at the estimated MT.

Methods
To analyze this issue, we developed a method to test the MTAT 2.0 software with virtual subjects, and we were able to recreate the observed behavior for some of our virtual subjects. First, we generated a set of 100 virtual subjects with a stochastic statistical model of MEPs [6]. For each virtual subject, we simulated a set of 100 possible MEP responses at each stimulator output from 0 to 100% MSO. The set of virtual subjects and their MEP responses is included in the supplementary material. For each subject, we defined the ground truth MT as the lowest intensity at which at least 50 out of the 100 simulated MEPs had a peak-to-peak amplitude of at least 50 mV. The mean MT of the virtual subjects was 66% MSO (range 44e84% MSO).
To automate the use of MTAT 2.0 for running many trials in the virtual subjects, we implemented a screen scraping software, leveraging an auto clicker and keyboard (Java Abstract Window Toolkit) combined with optical character recognition (Computer Vision Toolbox, R2018a, MATLAB) to read the MTAT output. With this software setup, we ran 200 MT estimations for each virtual subject, for a total of 20000 MT estimates. Half of the estimations were run with the default MTAT mode "threshold estimation in the range from 20% to 80% stimulator output" and the other half were run in the "refinement of a raw threshold estimate" mode. For the second half, the raw threshold estimates were generated by adding a uniform random offset of 5e15% MSO to the true MT, which corresponds approximately to the intensity typically used to localize the motor hotspot before the MT estimation.
We chose to terminate each MT estimation trial at 30 TMS pulses, since this appears to be the most conservative recommended value for this ML-PEST implementation. In the default mode, MTAT 2.0 shows the MT estimate in red after 12 pulses, yellow after 16 pulses, and green (accompanied with a beep) after 30 pulses. It has been suggested that 14 pulses are sufficient for the true MT to be inside the 95% confidence interval (CI) [3], and that 20 pulses produce a sufficiently accurate estimate for all subjects [7]. Finally, another app based on ML-PEST specifies 14 pulses as the minimum and 30 pulses as the upper limit for sampling, and 20 pulses are used in its demonstration [4].

Results
In 95% of the trials, after 30 TMS pulses, the MT estimate was between À4 and þ 3% MSO and À3 and þ3% MSO of the ground truth for the default mode and the mode with an initial guess, respectively.
In the default mode, however, the MT estimation error varied greatly across the virtual subjects (Fig. 1ab). For 21 subjects, the maximum absolute error was over 5% MSO. For the worst-case virtual subject, the 95% CI was from À24 to þ1% MSO and the full range of estimates was from À30 to þ2% MSO for the default MTAT mode. The mechanism of MT underestimation in this subject is illustrated in Fig. 1cd, which shows that MEPs occurred at an intensity well below the true MT in about 10% of the samples. Such MEP distributions are indeed observed in experiments; for example, excluded participant #13 in Ref. [8], shown in their supplementary data, had a similarly wide range of MEP responses. When an MEP occurs for a stimulator output well below the true MT at the beginning of the estimation sequence, the MTAT algorithm will only sample low stimulator outputs even if very few later MEPs are observed. For example, in the purple curve in Fig. 1c, the first pulse at 45% MSO resulted in an MEP, followed by only one more MEP on the third pulse. After the suggested minimum DOI of original article: https://doi.org/10.1016/j.brs.2022.08.021.

Contents lists available at ScienceDirect
Brain Stimulation j o u r n a l h o m e p a g e : h t t p : / / w w w . j o u r n a l s . e l s e v ie r . c o m / b r a i n -s t i m u l a t i o n The use of an initial guess in the "refinement of a raw threshold estimate" MTAT mode removed very large errors and reduced the overall variability of the MT estimates (Fig. 1ab), with the worstcase subject for this mode having 95% CI of À5 to þ7% MSO (full range À7 to þ9% MSO, Fig. 1ef). Nonetheless, even with the initial guess, 14 virtual subjects had greater than 5% MSO error for their worst-case MT estimate. Another important observation is that anomalous MT estimation can be identified from the apparent shortage or excess of MEPs during the MT estimation. In the default mode, 1.7% of all trials had 0 or 1 MEPs during the last 10 pulses (range 0e12% across subjects). Rejecting all such trials removed most misestimations and reduced the maximum MT estimation error from À30% MSO to À13% MSO and the number of subjects with estimation error >5% MSO from 21 to 16. With an initial MT guess, 1.5% of all trials had 9 or 10 MEPs during the last 10 pulses (range 0e7% across subjects). Rejecting these trials did not reduce the full range of misestimation (À7 to þ10% MSO) but decreased the number of subjects with greater than 5% MSO error from 14 to 13.

Conclusion
In some cases when the initial TMS intensity is far from the true MT, the default mode ("threshold estimation in the range from 20% to 80% stimulator output") of the MTAT 2.0 ML-PEST algorithm can produce a large misestimation of the MT for particular subjects and/or muscles. To mitigate this issue, we recommend inputting the TMS intensity used for motor hotspot localization as an initial guess with the MTAT mode "refinement of a raw threshold value." Doing so removed all MT misestimates larger than 10% MSO in our simulations.
In our simulations the estimation results were quantified after a fixed number of 30 pulses. With a good initial estimate, MTAT may converge to the final estimate with fewer pulses; in our results there was little improvement after about 25 pulses, although we did not characterize this.
For additional protection against misestimation, we also recommend rejecting MT estimates with either a very low or a very high number of MEPs during the last 10 pulses (reported in the top 10 rows of column "TMSIE" in MTAT). In particular, over the last 10 pulses, the number of MEPs should be close to 5; observing 0, 1, 9, or 10 responses means that the MT estimate is likely inaccurate. Indeed, such validation based on the 5-out-of-10-responses criterion has been applied when using MTAT [9]. Whilst these solutions are not perfect, in our simulations their combination produced MT estimates accurate within 10% MSO for all the virtual subjects and within 5% MSO for 87% of the virtual subjects.
A longer-term solution would be to develop a method that can assess the goodness of its MT estimate and adjust the number of pulses accordingly. Such solution would also help reduce other sources of uncertain MT estimates not suppressed by adding an initial guess. For example, for subjects and/or muscles with shallower inputeoutput curves, the standard amount of 30 pulses produces a broad, near-uniform MT estimate distribution whose width exceeds the desired 5% MSO accuracy even with a good initial guess.
Finally, in our simulations the virtual subjects were extrapolated from a model fitted to experimental data for the right first dorsal interosseus muscle of 12 healthy volunteers [6]. The extreme cases of this distribution might exaggerate certain features of the MEP generation probability and fail to capture others. To develop improved MT estimation methods, more open data on experimentally recorded MEPs is needed for model tuning and validation.

Ethics statement
The experiment was approved by Institutional Review Board of Duke University Health System and the participant provided their informed consent.