The OpenAI Library

The New York Times recently ran a brief article about the reading room in OpenAI's office in San Francisco. The article was heavy on images and light on text, but the overall theme was the tension between the company's GPT models—which have been trained on vast swaths of human culture, and are therefore able to regurgitate, remix, and approximate it—versus embracing the design and aesthetic trappings of a traditional library reading room.

The author mentions a handful of books that could be found in the OpenAI library, but many more were clearly visible in the photographs that accompanied the article. Given that many of the books were apparently suggested by OpenAI employees, I was curious what works could be found there, and decided to see how many I could identify. To make this slightly more worthwhile, I also decided to use this as an excuse to play around with GPT-4o's computer vision capabilities.

Unsurprisingly, this turned out to be much more work than I had anticipated. But once I had started, it felt like it was too late to turn back, so I just pushed on, and was ultimately able to identify over 200 different books shown in the images.

GPT-4o was undeniably helpful for this task, although not as helpful as I had hoped. In the best cases, I could give it an image, and it would return a perfect numbered list of the books shown in the photo, along with the full author names. Unfortunately, the reliability was such that this could never be simply taken at face value. Moreover, in almost no case was it able to correctly identify every single book in an image, except when it was a section that contained only one or two books with clearly legible (and unambiguous) titles. On the other hand, there were also some cases in which it was able to reconstruct a title and author from a book spine that was only partially visible, which I was then able to verify.

As an example, here is a segment of an image from the original article, in which I was able to manually identify all ten books, using a combination of what I can see and searches on Google or Amazon. Without any attempt to adjust the image or tune the prompt (for which I just used "Please identify the books in this image"), GPT-4o correctly identified the titles and full author names for seven of the ten books. For the other three, it came up with a title that was close, but not correct. In each of these cases, the title corresponded to a real book, and GPT-4o also returned the authors of the incorrect book (not the ones shown in the image).

This seemed overwhelmingly to be the pattern. Almost every book and author combination returned by the model did correspond to a real published book, just not necessarily the one shown in the image. The only case I identified in which it seems to have just made up a book was one it suggested, called The Prints of Japan by Frank Lloyd Wright, which seems not to exist. However, there is a real book by Wright called The Japanese Print: An Interpretation, as well as a different book (by another author) called The Prints of Japan. There were, of course, also many cases where it returned a title without an author (especially for art books with quite generic titles), and many cases in which it simply skipped a book, returned things slightly out of order, or made up something entirely. In a few cases, it returned several incorrect titles and author pairs in a row, which seemed to bear no particular relation to the image at all.

For those cases where GPT-4o returned a title and author that clearly matched the spine, it was then quite easy to verify by looking at the original image, especially for books that I at least vaguely knew (e.g., classics). Unfortunately, almost every title returned needed to be at least minimally verified, and many required some additional work. In some ways, the biggest time saving that came from GPT-4o was simply just not having to manually type out the words shown in the image.

Reflecting on how I personally carried out the task of identifying a book, the text that can be observed of the title and author are obviously important. However, I also frequently made use of additional cues. In particular, it was clear that at least in some cases GPT-4o was relying on calling an OCR tool to extract text from the images. It's unclear if it was also using other characteristics from the image to come up with an answer, or only using the OCR output. By contrast, I found that something as subtle as the font used for the text on the spine was an incredibly helpful cue for matching to the correct book in the case of ambiguous titles.

Other minor cues that were sometimes helpful included being able to see the publisher logo (such as Penguin), and being able to see the shape of book's title or an author's name (e.g, the relative length of first and last name), even if the text itself was not legible. In rare cases, I was able to immediately identify a book from its spine based on personal knowledge, without even needing to be able to read the text. Being able to find a photograph of a book's spine online could also be very helpful for verification. In many cases, it was easier to find a photograph of the front of a book, rather than the spine, but if the font matched between those two, that was still very helpful.

Of the books that could be identified from these photos, very few are surprising. There appear to be rough sections for things like history, biography, art, theater, and leadership or management. Stuart Russell is there of course, as is Hans Moravec and Douglas Hofstadter. In fiction, there are classics, like David Copperfield, along with many genre works, like Fear Nothing by Dean Koontz, including a fair bit of classic science fiction (e.g., Philip K. Dick, W. J. Stuart, Frank Herbert).

The most well represented authors are those that have a boxed set or series included, such as Diana Gabaldon's Outlander Series, J. R. R. Tolkien's Lord of the Rings, James Lovegrove's Firefly novels, and Nassim Nicholas Taleb's "Incerto". There also appeared to be four copies of Profiles of the Future: An Inquiry into the Limits of the Possible by Arthur C. Clarke, who is also represented by 2001: A Space Odyssey.

For me, there are two books that stand out as interesting or surprising. One is an apparently self-published book called AI and the Human Mind, written under the pen name "B. K.", which I am tempted to check out. The second is The Mad Ramblings of an Unaligned AGI by "Openai Library", which I assume is some sort of internal joke, as I can't seem to find any information about it online.

This is obviously not a complete list, as there are many books which could not be easily identified, and presumably, many in the reading room that were not pictured in photographs by the New York Times. Nevertheless, here is the table of books that I was able to extract: [tsv file]