Information Theory Basics (Simplified)
George Legrady
-
-
-
Information Theory has influenced the development of the digital world - the computer
Claude Shannon, mathematician from Bell Labs developed the theory to
get rid of noise from telephone communication (which at the time was very
noisy)
The goal of the theory:
1) how to send a maximum amount of information in a given channel
2) how to measure the capacity of a given channel to carry information
This theory came out of math and engineering but became influencial as
it could apply to all kinds of human communication
-
-
-
-
The smallest unit of information in the digital worlds is a "bit"
or byte.
(short for binary digit)
A bit is like a switch - it is either on or its off: [1] or [0]
10011101: 8 bit
1011010110101001:16 bit
-
-
-
-
-
-
-
-
In all communication, there is a sender, a message/communication and
a receiver.
The sender encodes a message, the receiver decodes it.
Meaning of a message is greatly dependent on the culture in which it is
transmitted.
The sender and receiver both know the code used to encode and decode the
message as it is learned through cultural training)
Between the sender, the message and receiver, noise gets in the way and
complicates the process.
A noiseless communication does not exist. There always is some kind of
noise entering the communication.
The goal of Information Theory is to reduce the noise to make the communication
more effective
Engineers try to remove the noise to get pure signal
Many artists, cultural theorists usually argue that it is the noise that
makes the signal interesting
-
-
-
Noise is unintended, unplanned, extra information that enters the communication
process
Example of noise include distortion, feedback, anything that breaks down
the signal
Noise can enter the process from the sender, the communication or the
receiver
Noise can be considered as limiting but also as increasing information:
1) Noise in the system limits the amount of desired communication byt
the sender,
2) but also adds new kinds of information (which may make the communication
interesting)
-
-
-
-
-
Shannon and Weaver (the authors of Information Theory) redefined the
meaning of "information": Information must not be confused with
meaning!
Information in Information Theory does not necessarily refer to meaningful
content
Information is a measure of the predictability of the signal: If I meet
you on the street, I can expect that we will say "Hi" to each
other.
I can predict that there will be communication, so "the predictability
of the signal is high!"
In the design of a telephone system, the critical factor is the number
of signals it can carry. What people actually say is irrelevant
-
-
-
-
-
-
-
Something is redundant if I already know it.
Redundancy does not give new information but is essential to communication
Redundancy provides a way to identity errors in the communication through repetition.
(We repeat ourselves on a bad telephone line)
The English language is about 50% redundant - that means we can delete about 50% of the words and still have usable communication (you me go have drink now)
The opposite of redundancy is entropy or randomness (noise)
Noise usually means lots of new information (as opposed to planned information) but unable to make sense of it:
a) how ya doin, see you later
b) jpe us fpom, drr upi ;syrt (the same text as in (a) but with 1 keyboard character to right)
(b) has lots more new information but very difficult to understand unless one has the code to read it
Entropy: Conditional probability - a balance between random values of 0 and 1
-
-
-
Subculture behavior - specialized communication recognized by members through repetition of conventions
CODE: a system of meaning common to members of a culture
it consists of signs and conventions that determine in what contexts these signs are used: Dress code, behavior code, alphabet, music taste, eating behavior, etc.