Brain–computer interface control with artificial intelligence copilots

1. Hochberg, L. R. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006).

   Google Scholar 

2. Gilja, V. et al. Clinical translation of a high-performance neural prosthesis. Nat. Med. 21, 1142–1145 (2015).

   Google Scholar 

3. Pandarinath, C. et al. High performance communication by people with paralysis using an intracortical brain-computer interface. eLife 6, e18554 (2017).

   Google Scholar 

4. Hochberg, L. R. et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375 (2012).

   Google Scholar 

5. Collinger, J. L. et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–564 (2013).

   Google Scholar 

6. Wodlinger, B. et al. Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J. Neural Eng. 12, 016011 (2015).

   Google Scholar 

7. Aflalo, T. et al. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348, 906–910 (2015).

   Google Scholar 

8. Edelman, B. J. et al. Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci. Robot. 4, eaaw6844 (2019).

   Google Scholar 

9. Reddy, S., Dragan, A. D. & Levine, S. Shared autonomy via deep reinforcement learning. In Proc. Robotics: Science and Systems https://doi.org/10.15607/RSS.2018.XIV.005 (RSS, 2018).

10. Laghi, M., Magnanini, M., Zanchettin, A. & Mastrogiovanni, F. Shared-autonomy control for intuitive bimanual tele-manipulation. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids) 1–9 (IEEE, 2018).

11. Tan, W. et al. On optimizing interventions in shared autonomy. In Proc. AAAI Conference on Artificial Intelligence 5341–5349 (AAAI, 2022).

12. Yoneda, T., Sun, L., Yang, G., Stadie, B. & Walter, M. To the noise and back: diffusion for shared autonomy. In Proc. Robotics: Science and Systems https://doi.org/10.15607/RSS.2023.XIX.014 (RSS, 2023).

13. Peng, Z., Mo, W., Duan, C., Li, Q. & Zhou, B. Learning from active human involvement through proxy value propagation. Adv. Neural Inf. Process. Syst. 36, 20552–20563 (2023).

    Google Scholar 

14. McMahan, B. J., Peng, Z., Zhou, B. & Kao, J. C. Shared autonomy with IDA: interventional diffusion assistance. Adv. Neural Inf. Process. Syst. 37, 27412–27425 (2024).

15. Shannon, C. E. Prediction and entropy of printed English. Bell Syst. Tech. J. 30, 50–64 (1951).

16. Karpathy, A., Johnson, J. & Fei-Fei, L. Visualizing and understanding recurrent networks. In International Conference on Learning Representations https://openreview.net/pdf/71BmK0m6qfAE8VvKUQWB.pdf (ICLR, 2016).

17. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (2019).

18. Gilja, V. et al. A high-performance neural prosthesis enabled by control algorithm design. Nat. Neurosci. 15, 1752–1757 (2012).

    Google Scholar 

19. Dangi, S., Orsborn, A. L., Moorman, H. G. & Carmena, J. M. Design and analysis of closed-loop decoder adaptation algorithms for brain-machine interfaces. Neural Comput. 25, 1693–1731 (2013).

    MathSciNet  Google Scholar 

20. Orsborn, A. L. et al. Closed-loop decoder adaptation shapes neural plasticity for skillful neuroprosthetic control. Neuron 82, 1380–1393 (2014).

    Google Scholar 

21. Silversmith, D. B. et al. Plug-and-play control of a brain–computer interface through neural map stabilization. Nat. Biotechnol. 39, 326–335 (2021).

    Google Scholar 

22. Kim, S.-P., Simeral, J. D., Hochberg, L. R., Donoghue, J. P. & Black, M. J. Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J. Neural Eng. 5, 455 (2008).

    Google Scholar 

23. Sussillo, D. et al. A recurrent neural network for closed-loop intracortical brain–machine interface decoders. J. Neural Eng. 9, 026027 (2012).

    Google Scholar 

24. Sussillo, D., Stavisky, S. D., Kao, J. C., Ryu, S. I. & Shenoy, K. V. Making brain–machine interfaces robust to future neural variability. Nat. Commun. 7, 13749 (2016).

    Google Scholar 

25. Kao, J. C. et al. Single-trial dynamics of motor cortex and their applications to brain-machine interfaces. Nat. Commun. 6, 7759 (2015).

    Google Scholar 

26. Kao, J. C., Nuyujukian, P., Ryu, S. I. & Shenoy, K. V. A high-performance neural prosthesis incorporating discrete state selection with hidden Markov models. IEEE Trans. Biomed. Eng. 64, 935–945 (2016).

    Google Scholar 

27. Shenoy, K. V. & Carmena, J. M. Combining decoder design and neural adaptation in brain-machine interfaces. Neuron 84, 665–680 (2014).

    Google Scholar 

28. Lawhern, V. J. et al. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J. Neural Eng. 15, 056013 (2018).

    Google Scholar 

29. Forenzo, D., Zhu, H., Shanahan, J., Lim, J. & He, B. Continuous tracking using deep learning-based decoding for noninvasive brain–computer interface. PNAS Nexus 3, pgae145 (2024).

    Google Scholar 

30. Pfurtscheller, G. & Da Silva, F. L. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin. Neurophysiol. 110, 1842–1857 (1999).

    Google Scholar 

31. Olsen, S. et al. An artificial intelligence that increases simulated brain–computer interface performance. J. Neural Eng. 18, 046053 (2021).

    Google Scholar 

32. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. Proximal policy optimization algorithms. Preprint at https://arxiv.org/abs/1707.06347 (2017).

33. Liu, S. et al. Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In 18th European Conference 38–55 (ACM, 2024).

34. Golub, M. D., Yu, B. M., Schwartz, A. B. & Chase, S. M. Motor cortical control of movement speed with implications for brain-machine interface control. J. Neurophysiol. 112, 411–429 (2014).

    Google Scholar 

35. Sachs, N. A., Ruiz-Torres, R., Perreault, E. J. & Miller, L. E. Brain-state classification and a dual-state decoder dramatically improve the control of cursor movement through a brain-machine interface. J. Neural Eng. 13, 016009 (2016).

    Google Scholar 

36. Kao, J. C., Nuyujukian, P., Ryu, S. I. & Shenoy, K. V. A high-performance neural prosthesis incorporating discrete state selection with hidden Markov models. IEEE Trans. Biomed. Eng. 64, 935–945 (2017).

    Google Scholar 

37. Stieger, J. R. et al. Mindfulness improves brain–computer interface performance by increasing control over neural activity in the alpha band. Cereb. Cortex 31, 426–438 (2021).

    Google Scholar 

38. Stieger, J. R., Engel, S. A. & He, B. Continuous sensorimotor rhythm based brain computer interface learning in a large population. Sci. Data 8, 98 (2021).

    Google Scholar 

39. Edelman, B. J., Baxter, B. & He, B. EEG source imaging enhances the decoding of complex right-hand motor imagery tasks. IEEE Trans. Biomed. Eng. 63, 4–14 (2016).

    Google Scholar 

40. Scherer, R. et al. Individually adapted imagery improves brain-computer interface performance in end-users with disability. PLoS ONE 10, e0123727 (2015).

    Google Scholar 

41. Millan, J. d. R. et al. A local neural classifier for the recognition of EEG patterns associated to mental tasks. IEEE Trans. Neural Netw. 13, 678–686 (2002).

    Google Scholar 

42. Huang, D. et al. Decoding subject-driven cognitive states from EEG signals for cognitive brain–computer interface. Brain Sci. 14, 498 (2024).

    Google Scholar 

43. Meng, J. et al. Noninvasive electroencephalogram based control of a robotic arm for reach and grasp tasks. Sci. Rep. 6, 38565 (2016).

    Google Scholar 

44. Jeong, J.-H., Shim, K.-H., Kim, D.-J. & Lee, S.-W. Brain-controlled robotic arm system based on multi-directional CNN-BiLSTM network using EEG signals. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 1226–1238 (2020).

    Google Scholar 

45. Zhang, R. et al. NOIR: neural signal operated intelligent robots for everyday activities. In Proc. 7th Conference on Robot Learning 1737–1760 (PMLR, 2023).

46. Jeon, H. J., Losey, D. P. & Sadigh, D. Shared autonomy with learned latent actions. In Proc. Robotics: Science and Systems https://doi.org/10.15607/RSS.2020.XVI.011 (RSS, 2020).

47. Javdani, S., Bagnell, J. A. & Srinivasa, S. S. Shared autonomy via hindsight optimization. In Proc. Robotics: Science and Systems https://doi.org/10.15607/RSS.2015.XI.032 (RSS, 2015).

48. Newman, B. A. et al. HARMONIC: a multimodal dataset of assistive human-robot collaboration. Int. J. Robot. Res. 41, 3–11 (2022).

49. Jain, S. & Argall, B. Probabilistic human intent recognition for shared autonomy in assistive robotics. ACM Trans. Hum. Robot Interact. 9, 2 (2019).

    Google Scholar 

50. Losey, D. P., Srinivasan, K., Mandlekar, A., Garg, A. & Sadigh, D. Controlling assistive robots with learned latent actions. In 2020 IEEE International Conference on Robotics and Automation (ICRA) 378–384 (IEEE, 2020).

51. Cui, Y. et al. No, to the right: online language corrections for robotic manipulation via shared autonomy. In Proc. 2023 ACM/IEEE International Conference on Human-Robot Interaction 93–101 (ACM, 2023).

52. Karamcheti, S. et al. Learning visually guided latent actions for assistive teleoperation. In Proc. 3rd Conference on Learning for Dynamics and Control 1230–1241 (PMLR, 2021).

53. Chi, C. et al. Diffusion policy: visuomotor policy learning via action diffusion. Int. J. Rob. Res. https://doi.org/10.1177/02783649241273668 (2024).

54. Brohan, A. et al. RT-1: robotics transformer for real-world control at scale. In Proc. Robotics: Science and Systems https://doi.org/10.15607/RSS.2023.XIX.025 (RSS, 2023).

55. Brohan, A. et al. RT-2: vision-language-action models transfer web knowledge to robotic control. In Proc. 7th Conference on Robot Learning 2165–2183 (PMLR, 2023).

56. Nair, S., Rajeswaran, A., Kumar, V., Finn, C. & Gupta, A. R3M: a universal visual representation for robot manipulation. In Proc. 6th Conference on Robot Learning 892–909 (PMLR, 2023).

57. Ma, Y. J. et al. VIP: towards universal visual reward and representation via value-implicit pre-training. In 11th International Conference on Learning Representations https://openreview.net/pdf?id=YJ7o2wetJ2 (ICLR, 2023).

58. Khazatsky, A. et al. DROID: a large-scale in-the-wild robot manipulation dataset. In Proc. Robotics: Science and Systems https://doi.org/10.15607/RSS.2024.XX.120 (RSS, 2024).

59. Open X-Embodiment Collaboration. Open X-Embodiment: robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA) 6892–6903 (IEEE, 2024).

60. Willett, F. R. et al. A high-performance speech neuroprosthesis. Nature 620, 1031–1036 (2023).

    Google Scholar 

61. Leonard, M. K. et al. Large-scale single-neuron speech sound encoding across the depth of human cortex. Nature 626, 593–602 (2024).

    Google Scholar 

62. Card, N. S. et al. An accurate and rapidly calibrating speech neuroprosthesis. N. Engl. J. Med. 391, 609–618 (2024).

    Google Scholar 

63. Sato, M. et al. Scaling law in neural data: non-invasive speech decoding with 175 hours of EEG data. Preprint at https://arxiv.org/abs/2407.07595 (2024).

64. Kaifosh, P., Reardon, T. R. & CTRL-labs at Reality Labs. A generic non-invasive neuromotor interface for human–computer interaction. Nature https://doi.org/10.1038/s41586-025-09255-w (2025).

65. Zeng, H. et al. Semi-autonomous robotic arm reaching with hybrid gaze-brain machine interface. Front. Neurorobot. 13, 111 (2019).

    Google Scholar 

66. Shafti, A., Orlov, P. & Faisal, A. A. Gaze-based, context-aware robotic system for assisted reaching and grasping. In 2019 International Conference on Robotics and Automation 863–869 (IEEE, 2019).

67. Argall, B. D. Autonomy in rehabilitation robotics: an intersection. Annu. Rev. Control Robot. Auton. Syst. 1, 441–463 (2018).

    Google Scholar 

68. Nuyujukian, P. et al. Monkey models for brain-machine interfaces: the need for maintaining diversity. In Proc. 33rd Annual Conference of the IEEE EMBS 1301–1305 (IEEE, 2011).

69. Suminski, A. J., Tkach, D. C., Fagg, A. H. & Hatsopoulos, N. G. Incorporating feedback from multiple sensory modalities enhances brain-machine interface control. J. Neurosci. 30, 16777–16787 (2010).

    Google Scholar 

70. Kaufman, M. T. et al. The largest response component in motor cortex reflects movement timing but not movement type. eNeuro 3, ENEURO.0085–16.2016 (2016).

    Google Scholar 

71. Dangi, S. et al. Continuous closed-loop decoder adaptation with a recursive maximum likelihood algorithm allows for rapid performance acquisition in brain-machine interfaces. Neural Comput. 26, 1811–1839 (2014).

    Google Scholar 

72. Fitts, P. M. The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47, 381 (1954).

    Google Scholar 

73. Gramfort, A. et al. MNE software for processing MEG and EEG data. NeuroImage 86, 446–460 (2014).

    Google Scholar 

74. Lee, J. Y. et al. Data: brain–computer interface control with artificial intelligence copilots. Zenodo https://doi.org/10.5281/zenodo.15165133 (2025).

75. Lee, J. Y. et al. kaolab-research/bci_raspy. Zenodo https://doi.org/10.5281/zenodo.15164641 (2025).

76. Lee, J. Y. et al. kaolab-research/bci_plot. Zenodo https://doi.org/10.5281/zenodo.15164643 (2025).

Download references