As ChatGPT continues to ricochet through the news cycle, media outlets are surely on the hunt for new angles they can present to the public in order to keep this story in motion. Among other threads, one that has gained some traction is the question of risks to privacy and security presented by these new systems.
Last week, a number of US outlets reported on data leaks at Samsung, in which three employees (in separate incidents) apparently entered confidential company information into ChatGPT. According to reports, in one case, an employee tried using ChatGPT to help debug code, another to optimize code, and a third to have it produce a summary of meeting notes.
This story was first reported in the Korean edition of The Economist (in Korean), and picked up by US outlets a week later. The interesting thing is that it is unclear how The Economist broke this story. It seems they were able to confirm it by talking to one of the employees involved, but they also have a quote from a Samsung Electronics official saying “It is difficult to confirm because it is an internal matter of the company” (which I have translated into English using Google Translate).
The main angle that most coverage seems to have taken on this story is that the information that Samsung employees entered into the system could potentially be incorporated into subsequent iterations of ChatGPT, thus representing a kind of data breach. Interestingly, OpenAI’s policy on this differs depending on whether one is using the API or the web interface. For the web interface, the company claims the right to use any text you enter for model training, unless you specifically opt out. With the API, by contrast, OpenAI will not use your queries for training, unless you specifically opt in. In either case, the company does reserve the right to store queries for 30 days, so that it can audit the system for abuse or misuse, in collaboration with a third party.
Although the news coverage was not specific on this point, one might infer from the US reporting that this event was somehow discovered by other users extracting confidential information from ChatGPT. This, however, seems highly unlikely. It is certainly true that models are capable of “memorizing” information that is in the training data. In principle, under the right conditions, a model might perfectly reproduce text that it has previously seen, if given the right prompt. Actually hacking the model in this way, however, in order to extract specific information that might be in there, would seem to be extremely difficult. First of all, one would likely need to have a good idea of at least part of the text that had been entered. Second, it would be more or less impossible to know whether any text that was returned was the actual text that had been previously seen, or just something that was randomly generated. Finally, it seems unlikely that one single copy of a piece of code would be sufficient for it to be memorized and recoverable in this way (even if it were to be utilized in training, which is by no means guaranteed). Ultimately, it seems far more likely that The Economist picked up on some details about one of these incidents, and was able to develop the story by talking to people (the old fashioned way).
Regardless, it does seem at this point like entering confidential or sensitive information into ChatGPT remains a generally bad idea. Even if OpenAI officially promises not to use it, that information will be stored on their servers for some amount of time, which potentially exposes that data to being seen or hacked.1 Indeed, in a separate incident, personal information of ChatGPT users (name, partial credit card information, etc.), was recently exposed due to a security flaw. This seems to have not specifically been OpenAI’s fault per se (rather, it was a problem with a widely used open source python package), but it is a useful reminder than OpenAI’s systems are potentially vulnerable, just like any others. Just this week, in fact, OpenAI also introduced a new bug bounty program, which will hopefully encourage people to devote effort to finding potential vulnerabilities so that they can be patched.
The Samsung incident makes for a quick and easy story, but in all likelihood we’ll never hear about similar incidents from different contexts, especially as it becomes increasingly unclear whether a system you are using might be linked to a GPT-like model. Between partnerships with Bain Capital, Khan Academy, DuoLingo, and many others, it’s clear that OpenAI is aggressively trying to find ways to deploy models in all kinds of contexts, and it seems overwhelming likely that we will increasingly encounter systems that depend on an OpenAI backbone, whether they are transparent about that or not. Indeed, it seems like there is a real possibility of GPT-4, or some similar system, becoming an integral part of the technical infrastructure powering much of the modern web, in a way that potentially makes it a major vulnerability and technical dependency, despite being so poorly understood.