The New York Times recently ran a brief article about the reading room in OpenAI's office in San Francisco. The article was heavy on images and light on text, but the overall theme was the tension between the company's GPT models—
which have been trained on vast swaths of human culture, and are therefore able to regurgitate, remix, and approximate it—
versus embracing the design and aesthetic trappings of a traditional library reading room.
The author mentions a handful of books that could be found in the OpenAI library, but many more were clearly visible in the photographs that accompanied the article. Given that many of the books were apparently suggested by OpenAI employees, I was curious what works could be found there, and decided to see how many I could identify. To make this slightly more worthwhile, I also decided to use this as an excuse to play around with GPT-4o's computer vision capabilities.
Unsurprisingly, this turned out to be much more work than I had anticipated. But once I had started, it felt like it was too late to turn back, so I just pushed on, and was ultimately able to identify over 200 different books shown in the images.
GPT-4o was undeniably helpful for this task, although not as helpful as I had hoped. In the best cases, I could give it an image, and it would return a perfect numbered list of the books shown in the photo, along with the full author names. Unfortunately, the reliability was such that this could never be simply taken at face value. Moreover, in almost no case was it able to correctly identify every single book in an image, except when it was a section that contained only one or two books with clearly legible (and unambiguous) titles. On the other hand, there were also some cases in which it was able to reconstruct a title and author from a book spine that was only partially visible, which I was then able to verify.
As an example, here is a segment of an image from the original article, in which I was able to manually identify all ten books, using a combination of what I can see and searches on Google or Amazon. Without any attempt to adjust the image or tune the prompt (for which I just used "Please identify the books in this image"), GPT-4o correctly identified the titles and full author names for seven of the ten books. For the other three, it came up with a title that was close, but not correct. In each of these cases, the title corresponded to a real book, and GPT-4o also returned the authors of the incorrect book (not the ones shown in the image).
![A photograph of ten books from the OpenAI library](../../img/openai-library/books.png)
This seemed overwhelmingly to be the pattern. Almost every book and author combination returned by the model did correspond to a real published book, just not necessarily the one shown in the image. The only case I identified in which it seems to have just made up a book was one it suggested, called The Prints of Japan by Frank Lloyd Wright, which seems not to exist. However, there is a real book by Wright called The Japanese Print: An Interpretation, as well as a different book (by another author) called The Prints of Japan. There were, of course, also many cases where it returned a title without an author (especially for art books with quite generic titles), and many cases in which it simply skipped a book, returned things slightly out of order, or made up something entirely. In a few cases, it returned several incorrect titles and author pairs in a row, which seemed to bear no particular relation to the image at all.
For those cases where GPT-4o returned a title and author that clearly matched the spine, it was then quite easy to verify by looking at the original image, especially for books that I at least vaguely knew (e.g., classics). Unfortunately, almost every title returned needed to be at least minimally verified, and many required some additional work. In some ways, the biggest time saving that came from GPT-4o was simply just not having to manually type out the words shown in the image.
Reflecting on how I personally carried out the task of identifying a book, the text that can be observed of the title and author are obviously important. However, I also frequently made use of additional cues. In particular, it was clear that at least in some cases GPT-4o was relying on calling an OCR tool to extract text from the images. It's unclear if it was also using other characteristics from the image to come up with an answer, or only using the OCR output. By contrast, I found that something as subtle as the font used for the text on the spine was an incredibly helpful cue for matching to the correct book in the case of ambiguous titles.
Other minor cues that were sometimes helpful included being able to see the publisher logo (such as Penguin), and being able to see the shape of book's title or an author's name (e.g, the relative length of first and last name), even if the text itself was not legible. In rare cases, I was able to immediately identify a book from its spine based on personal knowledge, without even needing to be able to read the text. Being able to find a photograph of a book's spine online could also be very helpful for verification. In many cases, it was easier to find a photograph of the front of a book, rather than the spine, but if the font matched between those two, that was still very helpful.
Of the books that could be identified from these photos, very few are surprising. There appear to be rough sections for things like history, biography, art, theater, and leadership or management. Stuart Russell is there of course, as is Hans Moravec and Douglas Hofstadter. In fiction, there are classics, like David Copperfield, along with many genre works, like Fear Nothing by Dean Koontz, including a fair bit of classic science fiction (e.g., Philip K. Dick, W. J. Stuart, Frank Herbert).
The most well represented authors are those that have a boxed set or series included, such as Diana Gabaldon's Outlander Series, J. R. R. Tolkien's Lord of the Rings, James Lovegrove's Firefly novels, and Nassim Nicholas Taleb's "Incerto". There also appeared to be four copies of Profiles of the Future: An Inquiry into the Limits of the Possible by Arthur C. Clarke, who is also represented by 2001: A Space Odyssey.
For me, there are two books that stand out as interesting or surprising. One is an apparently self-published book called AI and the Human Mind, written under the pen name "B. K.", which I am tempted to check out. The second is The Mad Ramblings of an Unaligned AGI by "Openai Library", which I assume is some sort of internal joke, as I can't seem to find any information about it online.
This is obviously not a complete list, as there are many books which could not be easily identified, and presumably, many in the reading room that were not pictured in photographs by the New York Times. Nevertheless, here is the table of books that I was able to extract: [tsv file]
Book | Author(s) |
---|---|
2001: A Space Odyssey | Arthur C. Clarke |
A Breath of Snow and Ashes | Diana Gabaldon |
A Concise History of Modern Painting | Herbert Read |
A General's Life: An Autobiography | Omar N. Bradley |
A Guide to Drawing | Daniel M. Mendelowitz |
A Hero for Our Time | Ralph G. Martin |
A History of Private Life: From Pagan Rome to Byzantium | Paul Veyne (editor) |
A Jefferson Profile: As Revealed in His Letters | Edwin Morris Betts (editor) |
A Little Life | Hanya Yanagihara |
A Most Wanted Man | John le Carré |
A Short History of Painting | [unknown author] |
A Step from Heaven | An Na |
Advent of Modernism: Post Impressionism & North American Art | Peter Morrin, Judith Zilczer, and William C. Agee |
Albrecht Durer Drawings and Water Colors | Edmund Schilling |
All Other Nights | Dara Horn |
All the Light We Cannot See: A Novel | Anthony Doerr |
Amatka | Karin Tidbeck |
American Art of the 20th Century | Sam Hunter and John Jacobus |
American Kingpin: The Epic Hunt for the Criminal Mastermind Behind the Silk Road | Nick Bilton |
American Landscape Painting | Wolfgang Born |
American Painting: History and Interpretation | Virgil Barker |
American Prometheus: The Triumph and Tragedy of J. Robert Oppenheimer | Kai Bird and Martin J. Sherwin |
An Echo in the Bone | Diana Gabaldon |
Ancient World in Your Pocket: Over 3,000 Essential Facts Hardcover | Paul Bahn |
Antifragile | Nassim Nicholas Taleb |
Art Treasures in Russia: Monuments, Masterpieces, Commissions and Collections | Prince Dimitri Obolensky |
Art in Every Day Life | Harriet and Vetta Goldstein (Fourth Edition) |
Art of Asia | Helen Rubissow |
ArtTalk | Rosalind Ragans |
Artificial Intelligence and the Human Mind | B. K. |
Asylum | Patrick McGrath |
Atlas of Medieval Europe | [unknown author] |
Bad Blood: Secrets and Lies in a Silicon Valley Startup | John Carreyrou |
Ball Four: My Life and Hard Times Throwing the Knuckleball in the Big Leagues | Jim Bouton |
Between Sittings: An informal autobiography of Jo Davidson | Jo Davidson |
Biographic Hemingway | Sophie Collins |
Birdscape | Robert Burton and Bruce Pearson |
Bloodmoney: A Novel of Espionage | David Ignatius |
Born a Crime: Stories from a South African Childhood | Trevor Noah |
Carpaccio: Paintings and Drawings: Complete Edition | Jan Lauts |
Charles Whistler's Omnium Gatherum | Philip Godlee (Penguin Books) |
Children of the Lamp: The Eye of the Forest | P. B. Kerr |
Chronicle of the World | Jerome Burne |
Cosmos | Carl Sagan |
Crucial Conversations: Tools for Talking When Stakes Are High | Kerry Patterson, Joseph Grenny, Ron McMillan, and Al Switzler |
Cryptonomicon | Neal Stephenson |
David Copperfield | Charles Dickens |
Do Androids Dream of Electric Sheep? | Philip K. Dick |
Downbelow Station | C. J. Cherryh |
Dragonfly in Amber | Diana Gabaldon |
Drums of Autumn | Diana Gabaldon |
Dune | Frank Herbert |
Eight Famous Elizabethan Plays (Modern Library) | [unknown author] |
Elizabeth and Mary | Jane Dunn |
Encyclopedia of Exploration | Ray Howgego |
Ender's Game | Orson Scott Card |
Endgame | John Mauldin and Jonathan Tepper |
Endurance: Shackleton's Incredible Voyage | Alfred Lansing |
Etruscans: Italy's Lovers of Life | Dale M. Brown |
Eve & Adam | Michael Grant and Katherine Applegate |
Everlasting | Alyson Noel |
Fear Nothing | Dean Koontz |
Fellside | M. R. Carey |
Ficciones | Jorge Luis Borges |
Firefly: Big Damn Hero | James Lovegrove |
Firefly: Generations | Tim Lebbon |
Firefly: Life Signs | James Lovegrove |
Firefly: The Ghost Machine | James Lovegrove |
Firefly: The Magnificent Nine | James Lovegrove |
Flesh in the Age of Reason: The Modern Foundations of Body and Soul | Roy Porter |
Fooled by Randomness | Nassim Nicholas Taleb |
Forbidden Planet | W. J. Stuart |
François Truffaut: The Adventures of Antoine Doinel | Anne Gillain |
Frontlines: Snapshots of History | Nicholas Moore and Sidney Weiland (editors) |
Get in the Game: 8 Elements of Perseverance That Make the Difference | Cal Ripken Jr. |
Good to Great: Why Some Companies Make the Leap...And Others Don't | Jim Collins |
Great Battles of the Civil War | Swafford Johnston |
Great Prints & Printmakers | Herman J. Wechsler |
Half of a Yellow Sun | Chimamanda Ngozi Adichie |
Half the Sky: Turning Oppression into Opportunity for Women Worldwide | Nicholas D. Kristof and Sheryl WuDunn |
Heretics in the Middle Ages | Martin Erbstosser |
High Output Management | Andrew S. Grove |
Homegoing | Yaa Gyasi |
How to Change Your Mind | Michael Pollan |
Human Compatible: Artificial Intelligence and the Problem of Control | Stuart Russell |
I Am a Strange Loop | Douglas Hofstadter |
Ibsen: A biography | Michael Leverson Meyer |
Identical | Scott Turow |
Infinite Jest | David Foster Wallace |
Inside U.S.A. | John Gunther |
Intent to Harm | Stan Washburn |
Into Thin Air | Jon Krakauer |
Irenicon | Aidan Harte |
Jerusalem, Sacred City of Mankind: A History of Forty Centuries | Teddy Kollek and Moshe Pearlma |
Kafka on the Shore | Haruki Murakami |
King Arthur's Place in Prehistory: The Great Age of Stonehenge | W. A. Cummins |
Kingdom Keepers IV | Ridley Pearson |
Kiss of Broken Glass | Madeleine Kuderick |
Lean In: Women, Work, and the Will to Lead | Sheryl Sandberg |
Life with Picasso | Françoise Gilot and Carlton Lake |
Liftoff | Eric Berger |
Lladro: The Magic World of Porcelain | Ricardo Martin (editor) |
Lolita | Vladimir Nabokov |
Mesopotamia: The Mighty Kings (Lost Civilizations) | Dale M. Brown |
Mind Children: The Future of Robot and Human Intelligence | Hans Moravec |
Mindset: The New Psychology of Success | Carol S. Dweck |
Modern Drawings (The Museum of Modern Art) | [unknown author] |
Multipliers: How the Best Leaders Make Everyone Smarter | Liz Wiseman |
National Geographic Concise History of the World: An Illustrated Time Line | Neil Kagan |
Navajo: Portrait of a Nation | Joel Grimes |
Neuromancer | William Gibson |
Never Cry Wolf | Farley Mowat |
Never Never | James Patterson |
New Voices in the American Theatre (Modern Library) | [unknown author] |
Night Owl | Andrew Mayne |
Nimble, Focused, Feisty | Sara Roberts |
Nine Lives | Wendy Corsi Staub |
Objets D'Art | [unknown author] |
Old Man's War | John Scalzi |
On Tyranny | Timothy Snyder |
Open Leadership | Charlene Li |
Operation Nemesis: The Assassination Plot that Avenged the Armenian Genocide | Eric Bogosian |
Outlander | Diana Gabaldon |
People and Politics: The years 1960-1975 | Willy Brandt |
Persians: Masters of the Empire (Time-Life Books) | [unknown author] |
Playing with Boys: A Novel | Alisa Valdes-Rodriguez |
Portrait of an Artist: A Biography of Georgia O'Keeffe | Laurie Lisle |
Power, Faith, and Fantasy | Michael B. Oren |
Profiles of the Future [four copies] | Arthur C. Clarke |
Puerto Vallarta Squeeze | Robert James Waller |
Radical Candor | Kim Scott |
Restoration Plays | Brice Harris (editor) |
Romanesque Art | George Zarnecki |
Rome: Echoes of Imperial Glory | Dale M. Brown |
Royal Panoply: Brief Lives of the English Monarchs | Carolly Erickson |
Salt Fish Girl | Larissa Lai |
Sapiens: A Brief History of Humankind | Yuval Noah Harari |
Secret Sanction | Brian Haig |
Shantaram: A Novel | Gregory David Roberts |
Sinners and Saints | Eileen Dreyer |
Skeleton Man | Tony Hillerman |
Skin in the Game | Nassim Nicholas Taleb |
Smokin' Seventeen | Janet Evanovich |
Snare of Serpents | Victoria Holt |
Soldiers: A History of Men in Battle | John Keegan and Richard Holmes |
Southern Cross | Patricia Cornwell |
Sprint | Jake Knapp |
Stonehenge and its Mysteries | Michael Balfour |
Story of the Royal Family | Don Coolican |
Sumer: Cities of Eden (Time-Life Books) | [unknown author] |
Synners | Pat Cadigan |
Tell It To The King | Larry King |
The 15 Commitments of Conscious Leadership | Jim Dethmer, Diana Chapman, and Kaley Klemp |
The Art of War in the Western World | Archer Jones |
The Art of the Renaissance | Peter and Linda Murray |
The Ascent of Man | Jacob Bronowski |
The Baker's Wife | Erin Healy |
The Bed of Procrustes | Nassim Nicholas Taleb |
The Beginning of Infinity | David Deutsch |
The Black Swan | Nassim Nicholas Taleb |
The Bridges of Madison County | Robert James Waller |
The Change Masters | Rosabeth Moss Kanter |
The Chronicles of Vladimir Tod -- Twelfth Grade Kills | Heather Brewer |
The Complete Etchings of Goya | Aldous Huxley |
The Courage to be Disliked | Ichiro Kishimi and Fumitake Koga |
The Culture Code | Daniel Coyle |
The Cut | George Pelecanos |
The Dark Forest (The Three-Body Problem Series, Part 2) | Cixin Liu |
The Encyclopedia of World History | [unknown author] |
The Expanding World | Esmond Wright (editor) |
The Fellowship of the Ring | J.R.R. Tolkien |
The Female Man | Joanna Russ |
The Fiery Cross | Diana Gabaldon |
The Fortunes | Peter Ho Davies |
The Golden Compass | Philip Pullman |
The Great Believers | Rebecca Makkai |
The Great Stagnation | Tyler Cowen |
The Greatest Generation | Tom Brokaw |
The Handmaid's Tale | Margaret Atwood |
The Hobbit | J.R.R. Tolkien |
The Holy Land (Time-Life Books) | [unknown author] |
The Iliad | Homer |
The Joy of Mixology | Gary Regan |
The Just City | Jo Walton |
The Likeness | Tara French |
The Long Tomorrow | Leigh Brackett |
The Mad Ramblings of an Unaligned AGI | Openai Library |
The Magicians: A Novel | Lev Grossman |
The Making of Asian America | Erika Lee |
The Measure of All Minds | Jose Hernandez-Orallo |
The Ministry for the Future | Kim Stanley Robinson |
The Most Human Human: What Artificial Intelligence Teaches Us About Being Alive | Brian Christian |
The New Painting Impressionism 1874-1886 | Charles F. Moffet |
The Only Way I Know | Cal Ripkin Jr. |
The Origin of Consciousness in the Breakdown of the Bicameral Mind | Julian Jaynes |
The Plains Indians Hardcover | Colin F. Taylor |
The Precipice [two copies] | Toby Ord |
The Prints of Japan | Frank A Turk |
The Program | Stephen White |
The Ramblers: A Novel | Aidan Donnelley Rowley |
The Return of the King | J.R.R. Tolkien |
The Revolt of the Public | Martin Gurri |
The Scar Boys | Len Vlahos |
The Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter | Joseph Henrich |
The Ship Who Sang | Anne McCaffrey |
The Sixteen Pleasures | Robert Hellenga |
The Soldier Kings: The House of Hohenzollern | Walter Henry Nelson |
The Soong Dynasty | Sterling Seagrave |
The Sound of Your Voice, Only Really Far Away | Frances O'Roark Dowell |
The Spirit Catches You and You Fall Down | Anne Fadiman |
The Story of Man | Cyril Aydon |
The Times Concise Atlas of World History | Geoffrey Barraclough |
The Tomb of Tutankhamen Hardcover | Howard Carter |
The Two Towers | J.R.R. Tolkien |
The U.S. Marine Corps Story | J. Robert Moskin |
The Unquiet | John Connolly |
The Vacant Chair: The Northern Soldier Leaves Home Hardcover | by Reid Mitchell |
The Walls | Hollie Overton |
Thomas Jefferson Architect | William L. Beiswanger (Chrysler Museum of Art and Yale University Press) |
Timelines of the Arts and Literature | David Brownstone and Irene Franck |
True Compass | Edward M. Kennedy |
Truth about History | Robert Dolezal |
Tuesdays with Morrie: An Old Man, a Young Man, and Life's Greatest Lesson | Mitch Albom |
Turning the Tide of War | Tim Newark |
Twisted | Jonathan Kellerman |
U.S. Camera 1960 | Tom Maloney (editor) |
Ulysses S. Grant: The Unlikely Hero | Michael Korda |
Villages | John Updike |
Voyager | Diana Gabaldon |
Warpaths | John Keegan |
Weak Courts, Strong Rights | Mark Tushnet |
When Breath Becomes Air | Paul Kalanithi |
Where Is Joe Merchant? | Jimmy Buffett |
Winslow Homer: American Artist, His World and His Work | Albert Ten Eyck Gardner |
Woman at Point Zero | Nawal El Saadawi |
Written in My Own Heart's Blood | Diana Gabaldon |
Written on Her Heart | Alan Maki |