GPT has entered the chat: within Synchron’s BCI-OpenAI integration

Last week Synchron announced an integration with OpenAI’s ChatGPT to include prompt-based interactions with their brain-computer interface.

In this application, ChatGPT generates input by asking users, whose implanted BCIs enable navigation and selection of answers. As Synchron founder and CEO Dr. Tom Oxley, described Bloombergthe BCI restores people’s ability to choose how they respond, while AI converts various inputs into a context-based experience.

According to CNBCSynchron’s user Mark, a person living with ALS, says the feature has saved him time and energy and taken the pressure off by giving him choices about how to communicate through the platform.

In addition to restoring “faster response times, natural conversation speeds, and SMS interaction” today, this integration will enable Synchron to bring sound, vision, and video to their users for multimodal experiences in the coming months.

I spoke with Oxley about the current limitations, opportunities, and challenges of large language model (LLM) integration and what this work means for the BCI field.

BCI as port of choice

Synchron’s mission is to restore autonomy to their users. Oxley explained that their focus with this integration is on restoring the ability to interact with digital environments more naturally for people with severe mobility limitations in their upper extremities.

“Generative AI takes input from the digital world and generates prompts, and then we’re on the other side. Our deep learning models make predictions about what the brain wants to do, so the BCI is the port of choice and restores autonomy in decision-making.”

The BCI use case in the demo involves controlling a platform via navigation and selection of prompts displayed on an iPad app, which is built Apple’s iOS accessibility features. Oxley says that the GPT-4o announced in Maythat works in real time through audio, images and text, represents a frontier beyond text.

“The reason we did this is because the multimodal ‘4o’ is slightly different, as it uses input from the environment that behaves like an extension of the user’s brain. It can see, hear and receive text input, so when the user starts the directions, it gets a real-time stream of information about everything happening in its environment.”

Give a personal touch

While this integration is currently a feature that can be turned on or off, Oxley noted, “We are now devoting significant resources to new methods of interacting with generative AI.”

He says the text-based feature is just the beginning, with vision, audio and eventually video waiting in the wings for additional use cases: “As this grows with different inputs, we’ll be building interaction methods that are simple, but that unlock access to technology that requires us to use our hands.”

These can range from recording conversational context from family members and caregivers to powering calendar integrations and other practical applications, including ultimately connecting to other consumer devices. But Oxley says the immediate focus is determined by their users.

“A big problem we hear from our patient population is that they are isolated because they can’t have conversations at a natural rate. So the goal here is to maintain a level of conversation, in dialogue with friends, family and caregivers. The AI ​​bridges the gap to keep up.”

Another key part of this integration is a feature Synchron calls emotional categorization, to provide users with a context-based set of choices. In the demo example, a doctor’s question about back pain generates several cues that include different positive, neutral, or negative responses.

“When you or I are asked a question or have a conversation, we decide how to respond based on how we feel. For our users, we’ve now also built some of our own layers on top of it, including these emotional categorizations for selecting words and verbal cues.”

Strategic outlook: clear vision, long range

Oxley believes multimodal LLM represents the future of BCI interactions. But he takes a pragmatist view, pointing to several variables that will shape the product roadmap, business partnerships and other commercial strategies in the coming years.

Non-exclusive

This integration does not represent an exclusive partnership between Synchron and OpenAI. “This is such a rapidly evolving field, so we remain agnostic to whichever API we use, based on what is in the best interest of our users at any time.”

Privacy first

Synchron has repeatedly stated that it will not share brain data with OpenAI. Given the nature of this technology, I asked if the platform will remember users over time. “That’s the goal, but it poses some privacy challenges,” he said.

Speaking about the technical work required to configure the encryption between device, local network and cloud, he added: “We are working on what that looks like in our first commercial offering. But the product vision is to build that user knowledge base locally and protect it. We set the bar very high when it comes to privacy.”

Playing for long periods of time

Oxley “Honestly [acknowledged] that we have debated internally about whether we should do this at all,” citing costs, time, risks, perceptions and unknowns. Ultimately, GPT’s 4o multimodal capabilities provided the impetus moving forward, to better connect users like Mark to digital systems today and unlock broader applications in tomorrow’s rapidly evolving consumer technology ecosystem.

Reflections and ripples

As a regular user of several popular LLMs, I admitted to Oxley that I shrugged my shoulders when I first saw the press release. Despite knowing full well that the out-of-the-box GPT I use is significantly different from this application, I still had trouble seeing its significance. Several neurotechnology executives have privately voiced similar criticisms.

But with technology in its early stages, it’s good to have more questions than answers. As a skeptical optimist, here are some summary reflections and lingering thoughts on what LLM could mean for user applications in commercial neurotech markets.

Burden of interpretation

Normal communication is something that most of us do without thinking and therefore take for granted: analyzing a question for intent, gathering a broader context based on external cues and internal feelings, and processing that into an answer, and all that at conversational pace.

What healthy people like me can’t fathom, however, is how this can be a hurdle or even a barrier for people living with various forms of disability, including early BCI users. If you’ve struggled to use a remote to browse Netflix, imagine all the communication required to do so.

By removing even some of the burden of interpretation and other friction for users, LLM can significantly improve the usability of BCI technology and lead to new use cases. Basic functions are a good starting point. I’m curious to see what ideas and functions emerge from end users, patient interest groups, healthcare provider circles, registries and trial participants.

Form factor

A common category of questions I receive is about minimally invasive versus non-invasive neural interfaces. In this case, the question might be, “Why is an implanted device needed to integrate LLM at all?”

Aside from the several critical differences between implanted and non-invasive devices, the question in this case is a good one. Cognixion, Augmental, Synaptrix and many others are commercializing wearable or non-invasive means to decode user intent into digital signals, in some cases with expandability, cost and security benefits.

So while Synchron’s integration with OpenAI is fairly prominent, the truth is that LLM is modality agnostic. The result: Expect several other neurotech players to adopt similar and different approaches to leverage rapid automation and other LLM features in the coming months.

Towards social robotics

Thanks to Musk’s comments, enough ink has already been spilled on BCI as a gateway to transhumanism, a topic that Oxley shrugged off. But a separate area requiring speculation is the role that semi-autonomous AI agents could play in future BCI markets where human privacy and security can be protected and LLM reliability ensured.

In addition to responsive prompts, there is also tremendous value to be unlocked for these users (and others) through small automations. If personalized agents could become capable and, again, reliable, they could help with check-in, communicate with the medical team, collect patient-reported results, and potentially handle administrative work, from managing appointments, medications , paying bills, dealing with insurers or disability agencies, initiating interpersonal requests to caregivers, for example to charge a device or signal other needs, and so on.

These examples are all text-based; the possibilities for integrating audio, cameras, sensors, wearables and other inputs (e.g. BCI-derived neural data for additional passive sensing) could yield applications arguably beyond what we can even imagine today. To be clear, all indications suggest that this level of reliability with LLM will likely last for years. But Synchron’s current prototype, with a laser focus on user privacy and simple value-added features, opens the door for these possibilities to take shape in the future.

Widget vs platform

Some think the frothy implantable BCI space should be taken into account, driven by the hype of Synchron’s main competitor. Just this week, Neuralink’s valuation reached one reported $8 billionwhich represents a larger size than all active US neuromodulation markets last year ($6.6 billion). LSI’s recent analysis.

The hype factor is real, but this only makes sense if you view BCI as an enabling platform technology. Synchron’s integration provides a keyhole overview. Think of it this way: if a commercially available API can significantly increase the user functionality of a Class III medical device – during active clinical trials – then BCI is not just a point solution, but an enabling technology whose features and functions are tailored to the market . (and not necessarily supervisors.)

This integration is the latest example of how today’s BCI companies are building publicly. What seems like relatively limited use cases and target markets will expand because they can only do so, given the pace of innovation in AI, ML, materials science, surgical innovations, consumer technology and beyond.