What do you think?
Rate this book
An Introduction to Information Theory: Symbols, Signals and Noise
Behind the familiar surfaces of the telephone, radio, and television lies a sophisticated and intriguing body of knowledge known as information theory. This is the theory that has permitted the rapid development of all sorts of communication, from color television to the clear transmission of photographs from the vicinity of Jupiter.
To give a solid introduction to this burgeoning field, J. R. Pierce has revised his well-received 1961 study of information theory for a second edition. Beginning with the origins of the field, Dr. Pierce follows the brilliant formulations of Claude Shannon and describes such aspects of the subject as encoding and binary digits, entropy, language and meaning, efficient encoding, and the noisy channel. He then goes beyond the strict confines of the topic to explore the ways in which information theory relates to physics, cybernetics, psychology, and art.
Mathematical formulas are introduced at the appropriate points for the benefit of serious students. A glossary of terms and an appendix on mathematical notation are provided to help the less mathematically sophisticated.
J. R. Pierce worked for many years at the Bell Telephone Laboratories, where he became Director of Research in Communications Principles. His Introduction to Information Theory continues to be the most impressive nontechnical account available and a fascinating introduction to the subject for lay readers.
To give a solid introduction to this burgeoning field, J. R. Pierce has revised his well-received 1961 study of information theory for a second edition. Beginning with the origins of the field, Dr. Pierce follows the brilliant formulations of Claude Shannon and describes such aspects of the subject as encoding and binary digits, entropy, language and meaning, efficient encoding, and the noisy channel. He then goes beyond the strict confines of the topic to explore the ways in which information theory relates to physics, cybernetics, psychology, and art.
Mathematical formulas are introduced at the appropriate points for the benefit of serious students. A glossary of terms and an appendix on mathematical notation are provided to help the less mathematically sophisticated.
J. R. Pierce worked for many years at the Bell Telephone Laboratories, where he became Director of Research in Communications Principles. His Introduction to Information Theory continues to be the most impressive nontechnical account available and a fascinating introduction to the subject for lay readers.
306 pages, Paperback
First published November 1, 1961
374 people are currently reading
1788 people want to read
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 50 reviews
June 6, 2011
An excellent introduction to the new and complicated science of communication. It explores topics of entropy, information as bits, noisy channels and other technical issues with plenty of examples, elaboration and analogy.
Negatives: Though the math and equations never get beyond basic algebra and physics, some of the equations and the relationships between them lack clarity. Furthermore, due to the age of the text and despite some recent updates to particular chapters, one is never sure how out of touch some of the referenced technology. For example, there is no mention of the internet or cell phones which have tackled newer challenges in communication and information theory.
Positives: Clearly, J Pierce put a lot of thought in the order of the content starting with a prolonged discussion on entropy in both statistical physics and information theory. This is a great alternative to reading a text book on the subject. The reader will be surprised by the author's sense of humor, appreciation of the difficult of the subject, and his extensive knowledge on related but different fields.
On one pass, I think I understood 80% of it and retained with fluency about 30%. It is certainly worth another read.
Negatives: Though the math and equations never get beyond basic algebra and physics, some of the equations and the relationships between them lack clarity. Furthermore, due to the age of the text and despite some recent updates to particular chapters, one is never sure how out of touch some of the referenced technology. For example, there is no mention of the internet or cell phones which have tackled newer challenges in communication and information theory.
Positives: Clearly, J Pierce put a lot of thought in the order of the content starting with a prolonged discussion on entropy in both statistical physics and information theory. This is a great alternative to reading a text book on the subject. The reader will be surprised by the author's sense of humor, appreciation of the difficult of the subject, and his extensive knowledge on related but different fields.
On one pass, I think I understood 80% of it and retained with fluency about 30%. It is certainly worth another read.
October 13, 2017
Terrific and short introduction to the subject via an incredibly precise and clear writing. As advertised it is intended for a general audience, but it is not a dumbed-down pop-sci book - you'll need a fair amount of grit to get through it.
Perhaps the best takeaway from the book is clearing of the nonsense and confusion that permeates 95% of "entropy" discussions in print and digital media. In one of the early chapters Pierce summarizes and contrasts the usage of the term in physics and communication theory. In physics entropy is associated with irreversibility of certain processes, and if entropy increases available free energy decreases. In statistical mechanics an increase of entropy is interpreted as increase in disorder and a decrease in our knowledge (think of gas filling up the second chamber of a 2-chamber container once the separation is removed).
Communication theory is all about message sources, message receivers, and amount of information in the messages. In that context entropy is the amount of uncertainty associated with the message that a receiver can get. The higher the information content of the message, the higher the uncertainty as to which message a producer will send, and the larger the entropy.
In the main body of the book Pierce takes us from Fourier analysis and linear circuits to ergodic sources (thank you Kolmogoroff and Wiener), efficient message encoding (e.g. Huffman codes, hyperquantization, vocoders etc), sampling theorem and of course Claude Shannon, noisy channels and channel capacity (via brilliant hyperspaces-inspired geometrical proof).
At the tail end Pierce serves us a delicious treat by bringing information theory and physics together. He notes that while information theory is fundamentally a math theory, in the end communication must happen in our physical reality with imposes unavoidable limitations. He proceeds to construct a single-molecule variant of Maxwell demon where we can seemingly escape 2nd law of thermodynamics, if we know just one bit of information (which of course has certain communication-theory entropy). But Johnson noise (electromagnetic noise emitted by any non-zero temperature body) incurs an unavoidable energy cost on transmitting that message. In Pierce's construction we need just 1 bit of information, and then we can reduce the statistical-mechanics entropy of the system and get free energy, but that free energy is exactly equal to the energy we would need to transmit that 1 bit over a fundamentally thermally-noisy channel! Tasty!
Note, the book is not nearly as dry as I make it sound, and it has plenty of excursions into natural language and meaning, art, “cybernetics”, and psychology. I love it how Pierce sprinkles opinionated pearls here and there, for example contrasting skepticism of general public (a masked confusion) vs skepticism of scientists (clarity and avoidance of nonsense). You’ll also get a bonus discussion of James Joyce and Zipf’s law plus Mozart and rule-based random waltzes generation. Finally you probably wouldn’t be too surprised to find out that stochastic art is often preferable to mediocrity. Hoo-ah, what’s not to love!
Perhaps the best takeaway from the book is clearing of the nonsense and confusion that permeates 95% of "entropy" discussions in print and digital media. In one of the early chapters Pierce summarizes and contrasts the usage of the term in physics and communication theory. In physics entropy is associated with irreversibility of certain processes, and if entropy increases available free energy decreases. In statistical mechanics an increase of entropy is interpreted as increase in disorder and a decrease in our knowledge (think of gas filling up the second chamber of a 2-chamber container once the separation is removed).
Communication theory is all about message sources, message receivers, and amount of information in the messages. In that context entropy is the amount of uncertainty associated with the message that a receiver can get. The higher the information content of the message, the higher the uncertainty as to which message a producer will send, and the larger the entropy.
In the main body of the book Pierce takes us from Fourier analysis and linear circuits to ergodic sources (thank you Kolmogoroff and Wiener), efficient message encoding (e.g. Huffman codes, hyperquantization, vocoders etc), sampling theorem and of course Claude Shannon, noisy channels and channel capacity (via brilliant hyperspaces-inspired geometrical proof).
At the tail end Pierce serves us a delicious treat by bringing information theory and physics together. He notes that while information theory is fundamentally a math theory, in the end communication must happen in our physical reality with imposes unavoidable limitations. He proceeds to construct a single-molecule variant of Maxwell demon where we can seemingly escape 2nd law of thermodynamics, if we know just one bit of information (which of course has certain communication-theory entropy). But Johnson noise (electromagnetic noise emitted by any non-zero temperature body) incurs an unavoidable energy cost on transmitting that message. In Pierce's construction we need just 1 bit of information, and then we can reduce the statistical-mechanics entropy of the system and get free energy, but that free energy is exactly equal to the energy we would need to transmit that 1 bit over a fundamentally thermally-noisy channel! Tasty!
Note, the book is not nearly as dry as I make it sound, and it has plenty of excursions into natural language and meaning, art, “cybernetics”, and psychology. I love it how Pierce sprinkles opinionated pearls here and there, for example contrasting skepticism of general public (a masked confusion) vs skepticism of scientists (clarity and avoidance of nonsense). You’ll also get a bonus discussion of James Joyce and Zipf’s law plus Mozart and rule-based random waltzes generation. Finally you probably wouldn’t be too surprised to find out that stochastic art is often preferable to mediocrity. Hoo-ah, what’s not to love!
September 20, 2018
It's always fun to delve into something written in the '80s. It was a nice attempt from Mr. Pierce to give an intro into communication theory. That's not an easy task since communication theory is a mathematical as well.
Let's put it as, encoding was an average, with a little be too many binary bits...although, let us admit, that from point of view of mathematical readiness, the channel was perhaps too noisy. But, the message came through.
Generally speaking, there was some intentional degree of obfuscation that did some part appear pretty Cimmerian, and prolonged reading and comprehending a bit longer, yet the read is enjoyable.
Let's put it as, encoding was an average, with a little be too many binary bits...although, let us admit, that from point of view of mathematical readiness, the channel was perhaps too noisy. But, the message came through.
Generally speaking, there was some intentional degree of obfuscation that did some part appear pretty Cimmerian, and prolonged reading and comprehending a bit longer, yet the read is enjoyable.
July 18, 2011
A gentle yet solid introduction to information theory.
November 11, 2022
Well written review of information theory. A bit dated, but overall demonstrates what information theory is and its applications.
January 16, 2013
I learned a bit and enjoyed the read. Another reviewer said it was a 'gentle thorough' introduction to the topic. I can't speak to the thorough, not my field, but it was gentle. Pierce has a sense of humor, and by golly, I believed that I could do the math if I took a real course in information theory.
What made it truly interesting to me was the date of revision, 1980, before the big revolution in IT. In 1990 I still had to create a program in order to use a program. Reading this book was a bit nostalgic.
What made it truly interesting to me was the date of revision, 1980, before the big revolution in IT. In 1990 I still had to create a program in order to use a program. Reading this book was a bit nostalgic.
March 23, 2012
Interesting with a lot of applications. With moments somewhat old fashioned.
January 12, 2019
Pierce offers here both a philosophical and mathematical account of first principles of information theory at the same time as he sketches an historical overview of the field which veers as much towards hagiography of Shannon as Pierce will allow himself to veer, given his stark plain-spokenness. Pierce doesn't offer a new approach to information theory, but he is masterful at presenting a concise synopsis of and introduction to the problems and quandaries of the field, foremost among which seem to be the two eternal conundrums of the information theorist, namely, how to pack as much information into as few bits as possible and, second, how to use good old science and ingenuity to outwit the eternal foe of information theory: noise. Pierce is fun to read because you can easily imagine a whole tradition of American language-users from the Puritans to Hemingway reading and endorsing what seem to be Pierce's practical potions for avoiding wastefulness with words and casting out confounded Satanic noise. So, where others may come to information theory and find a dry, dull science, in Pierce I found the standard of that uniquely Protestant and uniquely American insistence on economy of language being raised in another arena beyond literature. And it made me proud.
August 23, 2014
What exactly is an introduction? At what point do you say that a book is far too technical and the material is better fit for people with intermediate experience? Of course, it all depends. With that said, I think this book does still qualify as an introduction to information theory, but it really pushes the limit. Perhaps another way to say it is that this book is better fit for students in a college course, not casual readers with a passing interest in information theory. I definitely found it interesting and I got a lot out of it, but I also consider this one of the tougher books I've read outside of my field. In total, it took me about 18 months to read the first 180 pages (while waiting on my clothes down at the laundromat), until I decided a couple weeks ago to just suck it up and finish it.
In the beginning of chapter 6 Pierce acknowledges the difficulty of the first chapters as necessary preparation for the rest of the book. Those first chapters are dense, technical, and boring. Probably too boring for the average reader. The sheet amount of mathematical notation was just made it so much more intimidating every time I picked it up. Eventually I decided on the strategy of ignoring whatever I didn't understand on the first pass, and that seemed to work fine. Pierce explains the ideas thoroughly, and the equations provided just offer a useful restatement of the same ideas for the mathematically inclined. I wish I'd known that when I started. He even says exactly this in the Appendix ("On Mathematical Notation"):
"The reader will find a fairly liberal use of mathematical notation in this book, including a number of equations. This many incline him to say that this book is full of mathematics.
Of course it is. Communication theory is a mathematical theory, and, as this book is an exposition of communication theory, it is bound to contain mathematics. The reader should not, however, confuse the mathematics with the notation used. The book contain just as much mathematics and not include one symbol or equality sign."
So there, do your best and get what you can. The above quote also illustrates his writing style. I wasn't sure what to make of it at first, but it grew on me. He has a dry wit, which I enjoy but I know so many people will hate.
If you have a passing interest in communication or information theory, you might want to start off with a more accessible text. If you have a fairly strong background in math or you're willing to make up the difference with effort, then this is probably the best introductory book on the topic.
In the beginning of chapter 6 Pierce acknowledges the difficulty of the first chapters as necessary preparation for the rest of the book. Those first chapters are dense, technical, and boring. Probably too boring for the average reader. The sheet amount of mathematical notation was just made it so much more intimidating every time I picked it up. Eventually I decided on the strategy of ignoring whatever I didn't understand on the first pass, and that seemed to work fine. Pierce explains the ideas thoroughly, and the equations provided just offer a useful restatement of the same ideas for the mathematically inclined. I wish I'd known that when I started. He even says exactly this in the Appendix ("On Mathematical Notation"):
"The reader will find a fairly liberal use of mathematical notation in this book, including a number of equations. This many incline him to say that this book is full of mathematics.
Of course it is. Communication theory is a mathematical theory, and, as this book is an exposition of communication theory, it is bound to contain mathematics. The reader should not, however, confuse the mathematics with the notation used. The book contain just as much mathematics and not include one symbol or equality sign."
So there, do your best and get what you can. The above quote also illustrates his writing style. I wasn't sure what to make of it at first, but it grew on me. He has a dry wit, which I enjoy but I know so many people will hate.
If you have a passing interest in communication or information theory, you might want to start off with a more accessible text. If you have a fairly strong background in math or you're willing to make up the difference with effort, then this is probably the best introductory book on the topic.
July 2, 2017
_Symbols, Signals and Noise_ lies somewhere in the aether between textbooks and popular science. It is certainly more easy to read than many examples of the former, with careful explanations in plain language of the phenomena being discussed, and intuitive explanations of any important equations or discoveries, yet it is also certainly a deeper treatment than a swathe of modern books which present only gaudy tidbits of information and some patronising examples. Pierce stresses that his book would be empty without maths, and aims to fill in the gaps in laymen's understanding rather than skim over them -- even if he does omit proofs in many cases.
Most notable about the book is its readable style. Rarely does treatment of a scientific topic have this narrative pull which keeps you reading, or such a sense of an author's voice. It is a pleasure to learn from this character, whose intuitive explanations, scientific modesty and occasional spot of mild humour all work to educate you about a deep and broad topic, both in its core components as established by Shannon and in its applications to fields as diverse as linguistics, psychology and physics.
The book is not without flaws, but these are for the most part textual. The terms 'information theory' and 'communication theory' are both used, seemingly interchangeably throughout the book, with no real explanation of why this is so. There are a number of typographical errors in unfortunately critical areas -- numbers which are mistyped, symbols which are exchanged in the middle of important explanations. I entertained for a while that the book itself was demonstrating a form of error correction -- the redundancy in the text being sufficient to allow the detection and correction of errors. In his chapter on cybernetics, Pierce discusses for a while computing and predictive programs, and his treatment here has dated quite amusingly, not that it is any fault of his that the history of computation has changed since the book's publication, or that problems he saw unsolved have been tackled with great energy.
Certainly a great read for anyone with an interest in the topic, and I daresay a valuable guide to understanding for an undergraduate in communications or computing, or a professional moving from another field.
August 4, 2018
Several conversations with co-workers convinced me that I should try to get at least a passing familiarity with information theory, if only for my own benefit. I chose this particular book, honestly because it was easily available, with decent reviews, and comparatively inexpensive next to textbooks on the subject. The author goes to solid lengths to avoid dropping too much math on the reader, and when it is unavoidable, does a great job getting the interpretation across. While a small chunk of the book was material I was already familiar with (namely Fourier transforms and their properties), the rest was almost entirely new to me. Over all, I can say I'm quite happy with the book.
There are a handful of chapters, specifically 10-13, where the author departs from the nuts-and-bolts of information theory to cover more of the applications. Some of these are quite concrete and instructive, like the chapter on information theory in physics. Others, well.. not so much. They were entertaining to read, the author waxes poetic on the ideas of cybernetics (with a meaning pulled from the 1960s that the reader will likely have to acclimate to) and art, and not without some skill, just don't expect more coverage of technical topics there.
I've been meaning to double-back on the first handful of chapters while taking notes, as I'd like to get the specifics to really sink in, and follow it up with one of Shannon's original papers, which I'm told I should now have enough context to really appreciate. All-in-all, Pierce's introduction was a clear and entertaining entry point to the field. I can recommend this book to anyone who has a passing interest in the idea of information theory, but doesn't want to dive head-first into the academic literature.
There are a handful of chapters, specifically 10-13, where the author departs from the nuts-and-bolts of information theory to cover more of the applications. Some of these are quite concrete and instructive, like the chapter on information theory in physics. Others, well.. not so much. They were entertaining to read, the author waxes poetic on the ideas of cybernetics (with a meaning pulled from the 1960s that the reader will likely have to acclimate to) and art, and not without some skill, just don't expect more coverage of technical topics there.
I've been meaning to double-back on the first handful of chapters while taking notes, as I'd like to get the specifics to really sink in, and follow it up with one of Shannon's original papers, which I'm told I should now have enough context to really appreciate. All-in-all, Pierce's introduction was a clear and entertaining entry point to the field. I can recommend this book to anyone who has a passing interest in the idea of information theory, but doesn't want to dive head-first into the academic literature.
June 10, 2020
This book is a classic. It describes "Information Theory" as handed down by Shannon, in that respect it is above being dated since it is merely right. If you want an understandable description of information theory with no hype (the book was originally written in the 60s), this is a great book. The author was responsible for the images transmitted back from Jupiter and other deep space probes, so he is both authentic in his excitement for what Shannon discovered, and for what it can do for anyone working with real signals in real noise.
Pierce’s musing on information theory and psychology and art are interesting, but he ends with a chapter reminding the reader of the limitations of information theory proper. I suppose in the 1960s when the book was written, people might have sprinkled “information theory” and “cybernetics” over anything they wanted to make sound modern, much in the way artificial intelligence is used today.
If you replace "cybernetics" with "artificial intelligence" or "deep learning" and transport yourself to 1980 when the Dover edition was released, (remembering that the space shuttle used IBM 360 computers with magnetic core memory), then the author’s warnings about hype seem prescient.
Pierce gives a very accessible description of Information Theory and how it is applied it to real problems in communications and signal processing.
Pierce’s musing on information theory and psychology and art are interesting, but he ends with a chapter reminding the reader of the limitations of information theory proper. I suppose in the 1960s when the book was written, people might have sprinkled “information theory” and “cybernetics” over anything they wanted to make sound modern, much in the way artificial intelligence is used today.
If you replace "cybernetics" with "artificial intelligence" or "deep learning" and transport yourself to 1980 when the Dover edition was released, (remembering that the space shuttle used IBM 360 computers with magnetic core memory), then the author’s warnings about hype seem prescient.
Pierce gives a very accessible description of Information Theory and how it is applied it to real problems in communications and signal processing.
December 15, 2016
I meant to buy this years ago when I saw it on the shelf at Borders(!) and I regret not doing so. This is another great example of technical writing that moves seamlessly between highly mathematical and conversational tones.
Proofs aren't included here, opting instead to explore a breadth of applications. After building the foundational concepts and vocabulary, Pierce leads us into relationships between information theory and physics, psychology, and even art. I was surprised at the how well the discussion about computers held up thirty-five years later, but then, this isn't just any pop-science writer: Pierce named the transistor and led the team that first built it.
If you want a mathematical basis for understanding how communication works and the limits of what is possible, this is an excellent starting point.
Proofs aren't included here, opting instead to explore a breadth of applications. After building the foundational concepts and vocabulary, Pierce leads us into relationships between information theory and physics, psychology, and even art. I was surprised at the how well the discussion about computers held up thirty-five years later, but then, this isn't just any pop-science writer: Pierce named the transistor and led the team that first built it.
If you want a mathematical basis for understanding how communication works and the limits of what is possible, this is an excellent starting point.
July 13, 2008
In my opinion, this text's greatest strength and weakness are the same: it is mathematically very non-rigorous. This allows for the material to be very introductory and accessible, but it also tends to several holes and hand waving arguments (as well as a good deal of extremely simplistic background information that can be tedious if you are familiar with mathematics or science). Pierce's writing style is extremely conversational and the text is extremely clear and easy to follow. I actually found it to be a delightful read! It covers a broad array of topics and for it's age only seems slightly outdated. We'll see how good of an introduction to information theory it provides after I tackle a more mathematical text!
March 2, 2018
The author describes the history of information theory (before it was coined) starting with Alexander Bell, Morse, Binary / Boolean. He explains in a very straight forward and retail manner, concepts such as ergodicity and the units of measurement known as bits and the mathematics behind calculating bits per message.
Information theory is a hammer that allows almost any discipline and topic to be a nail. In the 50+ years since this book was written, information theory has been applied to economics, evolution, medicine, and remains the backbone of computing. Some argue it is the most robust unifying theory we know.
Great introduction for those wanting to learn about this very important concept.
Information theory is a hammer that allows almost any discipline and topic to be a nail. In the 50+ years since this book was written, information theory has been applied to economics, evolution, medicine, and remains the backbone of computing. Some argue it is the most robust unifying theory we know.
Great introduction for those wanting to learn about this very important concept.
December 5, 2017
This was a bit of a tough read for me. I appreciated the author's conversational tone, but some of the writing was difficult to parse. And when it came to the maths, I am woefully unprepared to ingest it. However, there were a few morsels I was able to extract, such as information theory's variation on the idea of entropy, that will lead me to explore other books on this topic that I hope are more approachable for me. I may even brush up on my maths so I stand a chance of interpreting future texts' use of it to explain these theories.
December 10, 2016
We are in 1980, Pierce is showing off Information Theory to layperson like myself. You may very much appreciate his introductory discourse on mathematical models and "informed ignorance" (his is humble but setting the boundaries of Shannon’s work). The text is warm and he dares to knit together language, cybernetics, psychology and art. Low entropy everywhere and exciting sometimes.
April 25, 2013
Good recap from my RF engineering days where Nyquist and Shannon's work were often applied to sat-com ...I was hoping for a bit more on the metaphysical front but that was probably unrealistic given the chapters listed in the TOC.
December 24, 2015
Beautiful explanation of various concepts related to information theory. Instead of seeking to be technical and rigorous, the author aims to deliver a good intuition behind solid ideas. Worth reading even though some of the discussions at the end of the book is outdated.
July 9, 2017
Good introduction to information theory.
I'm quite certain I'd have been more benefited by this book if I had taken the time to also do the math, but it was a worthy read nonetheless.
I'm quite certain I'd have been more benefited by this book if I had taken the time to also do the math, but it was a worthy read nonetheless.
March 18, 2018
A surprisingly accessible yet fairly in depth introduction to Claude Shannon’s Information Theory, getting both into the nitty gritty details, while also giving a broad overview of it’s usage within a broad range of fields...
August 20, 2009
Classic intro text, recommended by Chris Willmore. I read it mostly at the kitchen table in 1998. Useful stuff!
April 18, 2010
Terrific introduction to information theory. Not only gave clear and concise examples, but provided numerous applications of which information theory is apparent.
May 24, 2012
A good intro to IT. Recommended as a first.
May 31, 2021
Maybe more of a 3.5. A solid not-very-technical introduction, which also explores the relation between Information Theory and other disciplines. Well-explained and easy to read.
July 21, 2014
An interesting introduction to information theory, with philosophical discussion of language and data.
October 26, 2017
Great introduction to the field. Wide ranging and no more math than necessary.
July 2, 2025
The book is a remarkable historical artifact, a crystal-clear window into the nascent moments of a field that would eventually underpin our digital world. Reading it now, in an age dominated by the seemingly magical abilities of artificial intelligence, is to journey back to the very bedrock of information science, an era when the foundational principles were being laid with mathematical precision, long before the heuristics and practical complexities of modern applications took center stage. The book is a testament to the idea that, at its core, was born as a rigorous mathematical discipline, a concept so mathematically pristine that the author feels compelled to repeat it throughout the book emphatically.
Superficially, the book explores the foundational concepts of information theory as outlined by Claude Shannon. For modern readers, the book serves as both an educational primer on the theoretical underpinnings of contemporary technologies and a historical snapshot that showcases the earliest promises of a field that gave birth to the digital age.
The concept of *entropy*, borrowed from thermodynamics, is central to this framework, representing the uncertainty or randomness in a message source. The book explains how entropy sets fundamental limits on data compression and transmission, introducing readers to the source coding theorem and the channel capacity theorem. These ideas reveal the theoretical limits of efficient communication, such as the maximum amount of data that can be compressed without loss or transmitted reliably over a noisy channel. The accessible prose demystifies these abstract concepts and makes them approachable for readers without advanced mathematical training.
The stochastic and statistical concepts presented in the book were designed for optimal communication protocols; however, in hindsight, the same probability and regularity concepts planted the seeds for modern machine learning and artificial intelligence, particularly transformer models and neural networks. For instance, the idea that natural language exhibits predictable statistical patterns—quantified through measures like *conditional entropy*—is a cornerstone of how transformers exploit context to predict the next word in a sequence. Similarly, error-correcting codes and mutual information suggest the optimization techniques employed in training neural networks to minimize loss. While the author could not have foreseen the computational scale of modern AI, his explanations reveal the theoretical lineage connecting Shannon’s work to contemporary innovations.
For the contemporary reader, the value of concepts that optimized bandwidth for radios or telegraph machines extends beyond a mere academic understanding of foundational concepts. The historical document spotlights the original promise of the field against the backdrop of its subsequent evolution. The early vision was one of elegant, mathematical certainty around communication's fundamental limits. The emergent reality, however, has been shaped by a host of heuristic and practical methods. While the core principles of entropy and channel capacity remain inviolable, the path to achieving them in complex, real-world scenarios has led to the development of sophisticated algorithms and massive computational models that were unimaginable and perhaps inevitable. This book is a powerful reminder of the enduring interplay between pure theory and the messy, emergent properties of its application that have ultimately reshaped our world.
Superficially, the book explores the foundational concepts of information theory as outlined by Claude Shannon. For modern readers, the book serves as both an educational primer on the theoretical underpinnings of contemporary technologies and a historical snapshot that showcases the earliest promises of a field that gave birth to the digital age.
The concept of *entropy*, borrowed from thermodynamics, is central to this framework, representing the uncertainty or randomness in a message source. The book explains how entropy sets fundamental limits on data compression and transmission, introducing readers to the source coding theorem and the channel capacity theorem. These ideas reveal the theoretical limits of efficient communication, such as the maximum amount of data that can be compressed without loss or transmitted reliably over a noisy channel. The accessible prose demystifies these abstract concepts and makes them approachable for readers without advanced mathematical training.
The stochastic and statistical concepts presented in the book were designed for optimal communication protocols; however, in hindsight, the same probability and regularity concepts planted the seeds for modern machine learning and artificial intelligence, particularly transformer models and neural networks. For instance, the idea that natural language exhibits predictable statistical patterns—quantified through measures like *conditional entropy*—is a cornerstone of how transformers exploit context to predict the next word in a sequence. Similarly, error-correcting codes and mutual information suggest the optimization techniques employed in training neural networks to minimize loss. While the author could not have foreseen the computational scale of modern AI, his explanations reveal the theoretical lineage connecting Shannon’s work to contemporary innovations.
For the contemporary reader, the value of concepts that optimized bandwidth for radios or telegraph machines extends beyond a mere academic understanding of foundational concepts. The historical document spotlights the original promise of the field against the backdrop of its subsequent evolution. The early vision was one of elegant, mathematical certainty around communication's fundamental limits. The emergent reality, however, has been shaped by a host of heuristic and practical methods. While the core principles of entropy and channel capacity remain inviolable, the path to achieving them in complex, real-world scenarios has led to the development of sophisticated algorithms and massive computational models that were unimaginable and perhaps inevitable. This book is a powerful reminder of the enduring interplay between pure theory and the messy, emergent properties of its application that have ultimately reshaped our world.
April 5, 2022
This book is a perfect introduction to information theory and its applications, and the significance of Shannon's discoveries. I didn't finish the book though because a few chapters were either irrelevant or too technical for me.
"Entropy is a wonderful and universal measure of amount of information."
Chapter 2: The Origins of Information Theory
information theory as communication theory; challenges (encode/decode signal, noise); bandwidth (range of symbols used to encode signal); channel capacity (amount of signal that can reliably be transmitted in unit time);
Noisy-channel: The objective of communication is not to reproduce the message, but to predict the message.
Chapter 4: A Mathematical Model
a mathematical model to produce English text (random words, unigram, bigram, trigram); finite-state automata with limited capabilities; ergodic source of text
Chapter 4: Encoding and Binary Digits
symbols and bits; encode text in terms of characters (28 characters --> 5 bits per character --> 27 bits per words considering that the average word length is 5.5 characters) or words (14 bits per word assuming 16K vocabulary);
Chapter 5:Entropy
Entropy is the average bits per symbol needed to encode a message. A high entropy encoding corresponds to efficient encoding because of the less number of bits required to encode messages and thus leads to high surprise at decode time.
the fundamental theorem of noiseless channel (probable messages vs. possible messages); Huffman coding; entropy of a message source vs. entropy of a message; entropy vs. channel capacity
Chapter 6: Language and Meaning
language is chief form of communication but imperfect; semantics (ambiguity, non-compositionality); pragmatics
Chapter 7: Efficient Encoding
text (characters) vs. audio (characters, pitch, volume, stress) vs. video (image); remove redundancy
Chapter 12: Information Theory and (Human) Psychology
the entropy of a task is directly proportional to its level of difficulty; however, the rate of change in difficulty slows down with increase in complexity of the task; humans are good at complexity but only fair at speed; Zipf's law (principle of least effort)
Chapter 13: Information Theory and Art
high entropy art (sounds dull and unfamiliar) vs. low entropy art (too familiar and uninteresting); probably the reason why classical Indian music uses 7 notes and Western music uses 7+5 notes; for audience to appreciate a piece of art, it should strike the right balance between order and randomness
"Entropy is a wonderful and universal measure of amount of information."
Chapter 2: The Origins of Information Theory
information theory as communication theory; challenges (encode/decode signal, noise); bandwidth (range of symbols used to encode signal); channel capacity (amount of signal that can reliably be transmitted in unit time);
Noisy-channel: The objective of communication is not to reproduce the message, but to predict the message.
Chapter 4: A Mathematical Model
a mathematical model to produce English text (random words, unigram, bigram, trigram); finite-state automata with limited capabilities; ergodic source of text
Chapter 4: Encoding and Binary Digits
symbols and bits; encode text in terms of characters (28 characters --> 5 bits per character --> 27 bits per words considering that the average word length is 5.5 characters) or words (14 bits per word assuming 16K vocabulary);
Chapter 5:Entropy
Entropy is the average bits per symbol needed to encode a message. A high entropy encoding corresponds to efficient encoding because of the less number of bits required to encode messages and thus leads to high surprise at decode time.
the fundamental theorem of noiseless channel (probable messages vs. possible messages); Huffman coding; entropy of a message source vs. entropy of a message; entropy vs. channel capacity
Chapter 6: Language and Meaning
language is chief form of communication but imperfect; semantics (ambiguity, non-compositionality); pragmatics
Chapter 7: Efficient Encoding
text (characters) vs. audio (characters, pitch, volume, stress) vs. video (image); remove redundancy
Chapter 12: Information Theory and (Human) Psychology
the entropy of a task is directly proportional to its level of difficulty; however, the rate of change in difficulty slows down with increase in complexity of the task; humans are good at complexity but only fair at speed; Zipf's law (principle of least effort)
Chapter 13: Information Theory and Art
high entropy art (sounds dull and unfamiliar) vs. low entropy art (too familiar and uninteresting); probably the reason why classical Indian music uses 7 notes and Western music uses 7+5 notes; for audience to appreciate a piece of art, it should strike the right balance between order and randomness
November 10, 2024
Fascinating - as a historical item.
The thing you have to remember when reading this is that it's a quick update in 1980 of a book from 1961. There's nothing in here on the engineering side that should come as any surprise to a bright and interested EE-aware teenager. And that's what's so interesting!
We get the historical background - Nyquist, Shannon, Huffman, the usual culprits, but all presented as though this stuff is leading-edge new (which of course it was, at one time)! Get a quick summary of the point that Information Theory, in a nutshell, is about removing inefficient redundancy (compression) then adding efficient redundancy (modulation and channel codes).
What's interesting, then, is all this leaves out! Lossless compression is limited to Huffman, we don't even hear about arithmetic coding or universal coders like LZW. There's not yet any clear understanding of lossless vs lossy compression, and modeling as a stage before compression is still basically about trying to do something based on, eg the structure of the vocal tract (for speech compression) or various somewhat lame and limited RLE/delta compression schemes for images or video. No feeling yet for the sweet spot that uses some elements of perception (think YUV 4:2:0 for images or video, or masking for audio) yoked to more statistics-based compression schemes.
Viterbi is mentioned, but perhaps the full import was not yet realized (we get a name, but no details; probably at the time of 1980 the only use case that could justify the cost of Viterbi was deep space communication). No CDMA yet, no OFMDA, not even yet much understanding of the importance of noise models (eg Rayleigh fading) other than Additive White Gaussian. And of course not even an inkling of MIMO!
Very much a different time! I must try to read something similar from around 2000 to see how much the world has moved on -- and yet is still so far from now!
The thing you have to remember when reading this is that it's a quick update in 1980 of a book from 1961. There's nothing in here on the engineering side that should come as any surprise to a bright and interested EE-aware teenager. And that's what's so interesting!
We get the historical background - Nyquist, Shannon, Huffman, the usual culprits, but all presented as though this stuff is leading-edge new (which of course it was, at one time)! Get a quick summary of the point that Information Theory, in a nutshell, is about removing inefficient redundancy (compression) then adding efficient redundancy (modulation and channel codes).
What's interesting, then, is all this leaves out! Lossless compression is limited to Huffman, we don't even hear about arithmetic coding or universal coders like LZW. There's not yet any clear understanding of lossless vs lossy compression, and modeling as a stage before compression is still basically about trying to do something based on, eg the structure of the vocal tract (for speech compression) or various somewhat lame and limited RLE/delta compression schemes for images or video. No feeling yet for the sweet spot that uses some elements of perception (think YUV 4:2:0 for images or video, or masking for audio) yoked to more statistics-based compression schemes.
Viterbi is mentioned, but perhaps the full import was not yet realized (we get a name, but no details; probably at the time of 1980 the only use case that could justify the cost of Viterbi was deep space communication). No CDMA yet, no OFMDA, not even yet much understanding of the importance of noise models (eg Rayleigh fading) other than Additive White Gaussian. And of course not even an inkling of MIMO!
Very much a different time! I must try to read something similar from around 2000 to see how much the world has moved on -- and yet is still so far from now!
Displaying 1 - 30 of 50 reviews