What Is ChatGPT Doing... and Why Does It Work?

by Stephen Wolfram

Nobody expected this—not even its creators: ChatGPT has burst onto the scene as an AI capable of writing at a convincingly human level. But how does it really work? What's going on inside its "AI mind"? In this short book, prominent scientist and computation pioneer Stephen Wolfram provides a readable and engaging explanation that draws on his decades-long unique experience at the frontiers of science and technology. Find out how the success of ChatGPT brings together the latest neural net technology with foundational questions about language and human thought posed by Aristotle more than two thousand years ago.

Genres: Artificial Intelligence, Science, Nonfiction, Technology, Computer Science, Mathematics, Programming

143 pages, Kindle Edition

Published March 10, 2023

About the author

Stephen Wolfram is the founder & CEO of Wolfram Research, creator of Mathematica, Wolfram|Alpha & Wolfram Language, author of A New Kind of Science and other books, and the originator of Wolfram Physics Project.

Reviews

[Adam · Rating 3 out of 5]

Disappointing. The first half is a good, though very basic introduction to large language models and chatgpt. The second half is so populated with self promotion of the wolfram language and wolfram alpha that its pretty unreadable. At least the title should have been more transparent that it's mostly a book about ideas and visions on how chatgpt can work with wolfram software. The author desperately tries to show that wolfram software is not made obsolete by large language models. Honestly, I expected more from this book.

[Stergios Stergiou · Rating 2 out of 5]

Not particularly useful

There is little information that is ChatGPT-specific. The author seems to be more interested about striking a deal between his company and OpenAI.

[Liedzeit Liedzeit · Rating 4 out of 5]

The title of Wolfram’s book on ChatGPT is interesting in itself. Because it implies that while we know what ChatGPT is doing it is a totally different question why it works. And in a sense we do not know that (and having just seen Sam Altman the CEO of OpenAI in conversation with Lex Friedman they also do not know. Instead of developing a new tool they created, he says, a new science, which now has to be explored).

ChatGPT uses the data it has been trained upon to predict the next word. This part of the book is very accessible. Wolfram compares it with the occurrence of characters in English language and how you can come up with actual sentences by using statistics and he uses the example of simple image recognition. There are weighs attached to the candidates and some randomness to make it more interesting.


So in principle it is clear what ChatGPT is doing. But if at the end there are only strings put together by statistics and then by reinforcement, how does meaning arise. And the fact that meaning arises is evident to anyone who has ever played with Chat.

What goes on in the brain of ChatGPT? “... it’s complicated in there, and we don’t understand it—even though in the end it’s producing recognizable human language.”

And: “It has to be emphasized again that (at least so far as we know) there’s no ‘ultimate theoretical reason’ why anything like this should work.”

It is, says Wolfram, a surprising scientific discovery “that somehow in a neural net like ChatGPT’s it’s possible to capture the essence of what human brains manage to do in generating language.”

One conclusion Wolfram comes up with is that the writing of an essay is a computationally shallower problem than we thought. And: “So how is it, then, that something like ChatGPT can get as far as it does with language? The basic answer, I think, is that language is at a fundamental level somehow simpler than it seems”.

Language then is, it seems, not nearly as complicated as e.g. Chomsky thought. The syntax of (a particular) language and also the laws of syllogism for example were found by Chat implicitly during the task to create semantically correct meaningful texts.

Of course, Wolfram thinks that the next big step is to combine the power of ChatGPT with Wolfram technology. But I think while that might make a better version right now in the long run Chat will not need access to any other sources except what is already available.

[Craig Hunter · Rating 2 out of 5]

Largely an ad about Wolfram Alpha

Some details on ChatGPT architecture, but the gist seems to be that the author doesn't know why ChatGPT works. Then he goes into an ad for Wolfram Alpha being a good complement for ChatGPT as ChatGPT doesn't do math well. Readable, but I'm still looking for something that the title claims to address. Overall, a waste of $8.

[Khan · Rating 2 out of 5]

I have a lot of respect for Stephan Wolfram but this is a shallow read where the reader is left with very little in terms of understanding how ChatGPT works and the implication of ChatGPT. I don't think this was even worth writing let alone reading but AI enthusiasts might possibly disagree with me. If you're interested, you must understand calculus and basic machine learning terms to be able to comprehend what he is saying. Along with a general understanding of neural nets or you will be deeply confused. Nevertheless, I was surprised given how recent ChatGPT has released that a book about it has also been released. It seems like a marketing ploy to capture the immediate curiosity of people wondering what chatGPT is doing but failing to deliver substantive detail on its inner workings.

[Pablo Mejia · Rating 5 out of 5]

Wolfram es uno de los divulgadores científicos que más admiro y me pareció pertinente buscar contenido de alta calidad para un tema de tanta actualidad como Chat GPT.

Este libro corto resulta útil para entender la mecánica de "continuación razonable " que guía la arquitectura de este algoritmo, que utiliza métodos probabilisticos para unir frases y/o palabras. Su funcionamiento es impresionante dado que es capaz de arrojar frases con sentido semántico y en gran cantidad de casos, respuestas veraces. Para Wolfram, este hecho demuestra que los algoritmos cada vez convergen hacia niveles de desempeño cercano al de los seres humanos.

Sin embargo, los ejemplos al final de interacción con el "prompt" muestran que el algoritmo privilegia coherencia sobre exactitud. El modelo de Chat GPT se equivoca en tareas sencillas de matemáticas, de geografía, de economía, etc.

Lo importante de ver los avances de las arquitecturas de redes neuronales aplicadas a textos y palabras, que ya habían sido probadas con imágenes, demuestra que los seres humanos tenemos una estructura para escribir, pensar y destilar conocimiento . En palabras de Wolfram, sugiere que escribir no es tan difícil entonces, pero no desconoce lo impresionante del algoritmo desarrollado por OpenAI.

Wolfram explora el paso a paso de como funcionan estos algoritmos utilizando versiones anteriores de Chat GPT y lo conecta con uno de sus frentes de estudio: la irreducibilidad computacional. Esto quiere decir que los modelos que aprenden basados en millones de ejemplos no tienen una regla única que explique su forma de predecir que una imagen obedece a un perro o a un gato, lo que muchas veces los convierte en "cajas negras". Bajo estos escenarios, es clave ser cauto y prudente en el uso de estas herramientas.

El libro también tiene un objetivo comercial y tiene que ver con la integración anunciada entre el lenguaje de programación Wolfram Alpha con Chat GPT. Esto es muy emocionante dado que podría corregir muchas de las fallas actuales del modelo y acercarnos a una producción acelerada de conocimiento con una curaduria cientifica.

Habrá que ver como los seres humanos seguimos conviviendo con estas herramientas, pues a diferencia de las "teorías del todo" de Wolfram y su "Ruliad", me rehuso a pensar que en el mundo practico las máquinas reemplacen la creatividas de los seres humanos.

Como decimos en Colombia: "hecha la regla, hecha la trampa".

[Jeevan · Rating 4 out of 5]

A short and well written "book" or more like a lengthy blog post, that goes into the basics of ChatGPT, neural nets and hits the nail on the un-explainability of why any of this setup works :) The author does refer to Wolfram materials quite a lot and makes a passing argument that integrating the likes of ChatGPT with Wolfram components will make it more complete. I highly recommend reading this, regardless of where your knowledge levels are in this space.

[Doug · Rating 3 out of 5]

Raises interesting questions

Having some background in how neural nets and back-propagation work this was super quick and gave me some new concepts. I would have liked to have heard more about how it turns a question into an answer, and specifically how an AI "hallucination" occurs.

[Fin Moorhouse]

"Stephen Wolfram" sure loves unnecessary "inverted commas"

[Artur Coelho · Rating 4 out of 5]

Provavelmente, apenas os engenheiros da OpenAI sabem como é que o ChatGPT realmente funciona. Stephen Wolfram procura uma aproximação, pegando no seu saber como criador do Wolfram Alpha e do seu interesse no lado técnico da IA generativa. O livro é uma excelente e concisa introdução à tecnologia, mostrando como a conjugação de estatística e probabilidades, acesso a quantidades elevadas de dados, escalagem de parâmetros de treino e análise, e a possibilidade trazida pelos transformadores de criar tokens numéricos para qualquer tipo de dados permite às redes neuronais gerar informação. Vindo da computação mais clássica, Wolfram surpreende-se com a capacidade dos LLMs em gerar texto coerente, levantando a ideia que características humanas complexas, como a linguagem, são computacionalmente intensivas mas não inatingíveis.

[Kaitlyn Kwan · Rating 4 out of 5]

Really accessible! Lots of things I hadn’t thought through myself when it comes to semantics and meaning within sentences. The last bit basically being a Wolfram Alpha ad was kinda weird though.

[Julius · Rating 3 out of 5]

Este libro intenta responder a la difícil pregunta de cómo funciona el famoso ChatGPT, e intentar explicarlo de manera asequible para personas, en principio, poco técnicas.

En mi opinión, fracasa en su intento, y en ningún momento me he sentido muy metido en esta obra. Al igual que digo en muchas ocasiones, creo que esta explicación tendría mucho más sentido en un formato de vídeo. El libro consta de numerosas gráficas, diseños de neuronas, o mapas de calor, muy mal editadas y en blanco y negro. Las figuras resultan muy pequeñas y apenas se ven.

Al principio, el libro se centra en explicar cómo funcionan las redes neuronales, para poco a poco pasar al entrenamiento del lenguaje natural y el uso de tokens que han hecho famoso a ChatGPT. Sin embargo, me parece que este salto es muy brusco, no logro una fluidez en la explicación, y creo que el autor está excesivamente centrado en hacer un paralelo entre lo que ha hecho la empresa de ChatGPT (OpenAI) y lo que puede hacer la propia empresa del autor (MathWorks).

Desde luego, no es un libro fácil, ni se lo recomendaría a nadie plenamente convencido de querer darle una oportunidad a esta comparativa entre el lenguaje Matlab y ChatGPT. Para mi gusto, hay material didáctico mucho mejor en la Red, de sobra.

Por todo ello, un aprobado: 2,5 estrellas.

[Pramod Biligiri · Rating 4 out of 5]

This was a nice and understandable introduction to ChatGPT, by the noted scientist Stephen Wolfram who is also the creator of Mathematica software. Given that there would be a lot of maths underpinning the workings of ChatGPT, Wolfram does a good job not overloading you with the details. He uses simplified models and lots of diagrams to convey concepts. The "chapters" too are short and concise rather than taking on too much in one shot. At the same time, it doesn't feel so watered down that you begin to wonder whether you are learning anything at all. Overall, high school level mathematical knowledge should be sufficient to understand this book, as long as you are willing to pause at times to re-read some sections carefully, and look up any concept which you cannot recollect with ease. The book is also available to read online for free - https://writings.stephenwolfram.com/2...

There are a few philosophical musings at the end, like whether the capabilities of ChatGPT mean that language and writing essays are "computationally" easier than we believed. But the bulk of the book is about the workings of ChatGPT itself, and doesn't get into any larger questions about Artificial Intelligence. Another thing I liked about this book is that at many places he admits that we just don't know why certain aspects of ChatGPT work, in any deep theoretical sense. This extends to some aspects of the training of ChatGPT as well. He repeatedly labels these bits of arbitrary wisdom as "Neural Net Lore" which I thought was so appropriate! These kind of tidbits can only be interspersed by someone well versed in the field, and hence might make this book stand out from other introductions to the topic.

I found the first few chapters to be the best, where Wolfram clarifies what ChatGPT is (and by implication what *it is not*). As he repeatedly exclaims, it's amazing how a piece of software that only tries to generate the most likely next word (or 'token') can accomplish such a wide range of tasks so well. The examples he takes up here in terms of how many words the English language has, how much text there is out there on the Web, and how the probability aspects of these quantities work are very illuminating. Since Wolfram is a trained physicist, he is also able to articulate well what a "model" is. Elsewhere in the book, he points out how our understanding of ChatGPT is nowhere close to our understanding of how physical laws work. Also key here is the concept of "embeddings", without which any input data wouldn't be expressible as numbers in the first place! Overall, I came away awestruck that when ChatGPT is generating output, it is not even repeatedly scanning (called "looping" in programming terms) any piece of its internal data, unlike almost all programs!

The relatively hard parts of the book are in the middle, where he tries to explain what a "neural network" is with the common example of getting a computer to recognize digits (0-9) . For someone encountering the concept for the first time, the idea of weights, connections and how "training" happens to set up these weights and connections may not be very clear. You might benefit from reading animation-based explanations, like this one on YouTube: "But what is a neural network?". The image recognition task felt like an unexpected detour from the text based problems he had been describing thus far. It didn't tie back cleanly to the rest of the book. Then there are sections dealing with the inner working of ChatGPT, which I thought were the weakest; the concept of "attention" and "attention heads" wasn't clear to me from his explanation. Instead of the image recognition task, he could have shown how the weights for each word (i.e., the list of numbers associated with each word) is arrived at from a corpus of text with an example.

Towards the end, he looks for potential patterns in the inner workings of ChatGPT. Of course, there isn't much conclusive evidence to show here. But it's like a sneak peek into what scientists might be trying to figure out. Or, to end with a quote from the book itself: "Perhaps we’re just looking at the wrong variables (or wrong coordinate system) and if only we looked at the right one, we’d immediately see that ChatGPT is doing something “mathematical-physics-simple” like following geodesics. But as of now, we’re not ready to empirically decode from its internal behavior what ChatGPT has “discovered” about how human language is put together"

[Carlos · Rating 5 out of 5]

Demystifies ChatGPT for the layman. Well somewhat; nobody actually understands the full extent of its workings.


Wolfram makes the technical aspects accessible at a very coarse grained level which tear off ChatGPT’s all knowing oracle mask, especially for someone with no background in the field. Although not exactly a philosophical book, some of the key takeaways for me are rather less than technical than high level idea about the nature of human language and what it actually means to be “intelligent”.

A few of these takeaway points:

1. The fact such simple neural nets capture human-like language suggests simple laws of human language might exist, simple rules leading to complex outputs. It is not that ChatGPT is so smart that it can understand human language, it’s that human language is shockingly “simple”

2. ChatGPT’s trainability actually comes at the expense of its ability to perform complex computation. “Learning” involves general pattern recognition in complexity. Computing on the other hand is “calculating” each individual step in the complex system without generalization into patterns. The exact reason why ChatGPT has learned human like language so well, is the exact reason it can’t… well… reason.

3. ChatGPT has captured syntactic and semantic structures in language- this means it can makes proper sentences with the logic embedded language. This means it makes sentences that are “correct” but that have no reference to our actual contingent World. For example: “The elephant lives on the moon”. It is not meaningfully talking or reasoning “about” the world. Linking it with plugins like Wolfram Language would be a step to link it to reality and give it computational tools.

I’m looking forward to rereading it soon! And no this summary was not produced by ChatGPT, despite its lack of true reasoning and computation, I must in all humility accept it would have created a much better summary than this lol… recommended!

[Conner Walstad · Rating 3 out of 5]

“What Is ChatGPT Doing... and Why Does It Work?” is a book by Stephen Wolfram which goes into the basics of ChatGPTand other AI’s work. I picked up this book because Stephen Wolfram is a person that I knew of because of my familiarity with his Wolfram Alpha program which is effectively an AI for Mathematics. I enjoyed reading this book to an extent but it almost felt like it was teaching me more about how these types of AI engines work rather than exactly what ChatGPT was doing which felt quite misleading. I feel like there is also a knowledge gap where the author talks a lot about mathematical reasoning for things happening that might not be fully understood by someone who is interested in the more computer science aspects of artificial intelligence. With all the talk about ChatGPT in the media lately I wanted to know more about how it works but feel like I am more equipped to program a neural network than understand ChatGPT. It felt like almost the entire time the author was guessing how the program worked based off of various tests done on earlier editions of the AI and never gave the reader much more than mere speculation. He said several times that he doesn’t know how it has gotten so good at what it does and his only explanation was that they must have put the program through a lot of training. Later in the book he starts comparing his Wolfram Alpha program to ChatGPT and explains how they could work together and it almost seems like he could’ve ended this book earlier to make it actually applicable to the title. Overall, I can’t bash on this book too much since it was still fairly interesting and well explained but I was left wanting more than I was given. I’d recommend this to anyone wanting to learn more about how artificial intelligence works and some of the various intricacies covered within but don’t expect to find a super in depth review like the title may lead you to believe.

[Khurram · Rating 3 out of 5]

Computational language Vs Chat GPT

An informative book. It was a very good refresher into neural networks, which I studied over 20 years ago. It is amazing how far things have moved in technology.

The problem I had I this book though I did grasp the concepts the pictures of the symbolism had me more confused. I do like that the book, in a way, promotes the human to AI to human concept, but towards the end, it starts pushing Wolfram|Alpha as a way to boost Chat GPT in light of the the mistakes that Chat GPT makes.

I did get a better understanding of Chat GPT and neural networks after reading this book. I will say this is not a not a book for beginners. A decent understanding of IT is necessary. Though it is a year since this book has been published so some bugs/mistakes might already have been fixed.

[Pete · Rating 3 out of 5]

What Is ChatGPT Doing… and Why Does It Work? (2023) by Stephen Wolfram does what the title suggests.

The book is an impressively short and upbeat introduction to ChatGPT. Wolfram is an incredibly smart guy and in making an effort to write a short and comprehensible introduction to large language models (LLMs), neural nets and transformers he’s done a great job. The first two thirds of the book is well worth a read.

The book switches later to compare Wolfram Alpha to ChatGPT. Wolfram makes the point that ChatGPT is very poor at many calculations. He makes the point that a symbiosis of the two would be incredibly useful.

What is ChatGPT Doing isn’t a bad book. Getting a better, deeper introduction to LLMs would require some code.

[Justin Curry · Rating 5 out of 5]

Short easy read that communicates the big ideas of how LLMs work, while giving a taste of the mathematics underpinning these models. Highly recommended Intro text. As insufferable as Wolfram can be at times, this long blog post (available online for free in color, the book is a bit of a crappy black and white reproduction) is a service to the community. As a math professor who teaches basic data science this was a nice expositional account that I recommend to students.

[mar234 · Rating 4 out of 5]

A short and well-written introduction to machine learning and AI models. In this book Wolfram explains the fundamentals of an AI model in an easy to understand, yet still technical language. Entertaining and full of knowledge—although, often times, apparently no one knows why and how some things work—it's the perfect book to read for someone looking to get into the topic of machine learning, as well as for the casual, curious reader.

[Caroline · Rating 3 out of 5]

A lot was over my head but a good handful was fairly easy to follow. Definitely an interesting read if you want to try to have a basic understanding about this platform. I’m also the kinda person who is curious about how/why things work the way they do so I’m sure most would find this a little boring

[Christy Matthews · Rating 5 out of 5]

Short and sweet. Wolfram (of Wolfram Alpha) gives a high level overview of how Chat GPT and other Ai language creation tools work. He also highlights flaws, especially pertaining to precise calculations and how a tool like Wolfram Alpha could be integrated into Chat GPT to increase accuracy. And to presumably generate more income for the author.

[DK · Rating 3 out of 5]

For specialists, not everyday readers

While some of the concepts put forth in this book are generally understandable, most of it is likely beyond the knowledge level of most readers. Readers with a background beyond basic level understanding of AI, mathematics, computer science or similar topics may get excited over Wolfram's work.

While Stephen Wolfram may be an accomplished scientist, I find his writing style used in this work lacking in communicating his ideas to common readers. It left me wishing he'd hired a graduate student or ghost writer to make it more accessible. Additionally, the writing style vacillates between stuffy academic writing to embarrassingly informal overuse of what I perceive to be "air quotes" and a ridiculous number of sentences beginning with "OK, so".

Lastly, the inclusion of this poorly written ebook in Amazon's Kindle Unlimited program comes off of as a self promotional scheme--which in itself I have no issue-- to boost its ranking. Ultimately, this appears to be an effort to promote his other projects at Wolfram Research.

BOTTOM LINE: HIRE A PROFESSIONAL WRITER TO REWRITE THIS BOOK! STILL useful, but tedious.

[Marius Balaj · Rating 3 out of 5]

The book explains things well at the beginning but turns into ads later on.

[Coco Tsai · Rating 5 out of 5]

Wolfram provided a good overview to understand LLM method and limitations. Easy to follow for those without hands-on experience like me

[Ferhat Culfaz · Rating 4 out of 5]

Very simple and clear explanation of how chatGPT works. Still a bit involved for layman, but straightforward for practitioners of machine learning but outside of their domain.

[Jonathan · Rating 3 out of 5]

Unless you have been living under a rock for the past six months, the chances are you have heard of ChatGPT.

ChatGPT has not just changed the world we live in by bringing artificial intelligence into the hands of the general population. It has defined a new era in which we live.

In April 2023, it was reported that 10 million queries were requested each day. That might not seem like a lot, but it only took OpenAI (the company behind ChatGPT) 5 days to reach 1 billion users. To put that into context, tech giant Instagram (who took the second least amount of time to reach this figure) took two months!

ChatGPT is not just the most common form of AI that is used - but also provides the backbone for other AI services out there through the use of its API.

But how does it work?

How can it interact with you and provide such in-depth information with such a high degree of accuracy?

Spoiler alert - whilst we may know the answer to some of these questions (this book tries to answer), we don’t fully understand, and perhaps we never will.

This is a highly recommended book, which, although at times does become a bit technical, is positioned in a way that would allow anyone interested in technology to get something.

[Michael Nguyen · Rating 3 out of 5]

Slightly difficult book. Good to help understand how it works. He references his own computer programming language "Wolfram Alpha" too much though.

[RK Byers · Rating 2 out of 5]

nice idea

gave me a basic understanding of the logic behind the thing but in application, WAY dangerous! 1/2 the information you currently get on ChatGPT is wrong, some of it grossly so.

[Diogo · Rating 3 out of 5]

Gostei de conhecer melhor o funcionamento matemático e a reflexão filosófica do que se figura a língua e comunicação e a forma como depois se traduz em modelos de linguagem. Todavia, parecia que o autor estava sempre a vender o seu peixe ao invés de se focar no que se pressupõe a fazer.

[Christian Souza · Rating 3 out of 5]

Esse será talvez o único livro que importa ler quando se interessa em saber realmente como o ChatGPT funciona. Tudo o mais que for escrito sobre o assunto a partir daqui será redundante e, sem dúvida, superficial.

Por que Stephen Wolfram é a melhor pessoa no mundo para tentar nos explicar o quase inexplicável: afinal, como funciona esse ChatGPT? Como é que esse milagre pôde acontecer?

Porque Stephen passou os últimos 43 anos de sua vida estudando as redes neurais, que são o cérebro do ChatGPT. Mesmo assim, ele se mostra impressionado com os resultados gerados por ele. Isso significa que estamos realmente em um momento revolucionário da tecnologia e não há como prever aonde chegaremos a partir de agora.

No entanto, não posso dar mais do que 3 estrelas, pois o livro sofre do mal comum aos livros de Computação: não foi tão bem escrito e foi muito mal traduzido (para o Português de Portugal, no caso).