If you're seeing this message, it means we're having trouble loading external resources on our website.

If you're behind a web filter, please make sure that the domains *.kastatic.org and *.kasandbox.org are unblocked.

### Course: Computer science theory>Unit 3

Lesson 2: Modern information theory

# Measuring information

How can we quantify/measure an information source? Created by Brit Cruise.

## Want to join the conversation?

• I think that for sending the 'poker hand' information there must be a better algorithm because unlike the word, the order does not matter. Is there some algorithm that can do this?
• So here's how you could use less space.....
There are 52 C 5 possible hands i.e. 52!/(5!*47!)=2598960
So for all the possible hands sort them in order first by suit (alphabetically clubs,diamonds,hearts, spades), then by rank (Ace to King). Assign the 1st of the sorted hands 1, then 2nd 2.... etc. until you have numbered all the hands.
i.e. 1st hand would be Ac,2c,3c,4c,5c, 2nd hand would be Ac,2c,3c,4c,6c,.... last hand (2598960th hand) would be 9s,10s,Js,Qs,Ks
So we now have a table of hands numbered from 1 to 2598960.

If both sides have this table we can just send the number in the table and both sides will know which hand is being referred to.

Note:
log_2(2598960) approximately equals 21.3095

Compare this to sending log_2(52)=5.7004 bits per card for 5 cards
for a total of 28.50219 bits

We saved 28.5022-21.3095= 7.1927 bits

Here's a bit more details as to where the savings came from:
-We save 0.29 bits by not having duplicate cards
52^5/(52*51*49*48*47)=1.21909
log_2(1.21909)=0.28580
-We save 6.91 bits by not caring about the order
5! permutations= 120
log_2(120)=6.90689

Hope this makes sense
• About the poker hand, what about sending message of card's number and suit separately? For example first question about color, red or black. Then, about the suit. Then about the number? Would it be the same as randomly dividing the shuffled deck?
• Umm lets count
Color: R/B - one bit
Suit: this is another one bit because two suits were ruled out by the earlier step
Number: Is it higher than 7? one bit
Is it higher than 4/11 one more bit
Higher than 2/6/10/12
And one more bit to make the final guess
That adds up to 6 bits, the same as Brit said I think.
• So it's the board game "Guess Who?"
• Pretty much. In the words of Sherlock Holmes (the books, not the horribly romanticized violent movies) "When every possibility is ruled out, whatever remains, however improbable, must be the truth."
• There's been a lot of talk about predictability and redundancy in this Q&A, and as I was thinking about it, a thought hit me: Are predictability and redundancy really all that different; that is, are they really two related phenomena, and that something is predictable because of redundancy.

Of course, now that I write this, I begin to have my doubts. After all, its one thing to say the same thing twice, its another to give the necessary clues for math, logic, and science to deduce with.

A counter argument, I feel, can be found in the example of the Periodic Table of the Elements, which is a high-redundancy communication (according to The Way Things Work; The Book of the Computer), which nevertheless seem to me as being filled with a lot more "clues" and less repetitions.

I guess what I am trying to say is, "Does redundancy encompass clues, or repetition only?"

What do you think.
• After reading "A Mathematical Theory of Communication" a bit in Research, I know that repetition leads to probability. To find the probability of letters, we must first find repetitions. Redundancy, however, means something extra, something not needed. Although repetition and redundancy have the same literal meaning, they have different connotations. Since redundancy or too much of something is not usually found in English text (over-describing something or repeating something until it becomes unnecessary to do so any more is looked down upon in English texts), I would say that only repetition leads to probability, not redundancy.

Does this answer your question? I know it's completely different from my previous one.
• I have a question to do with the second example, the one with the letters. If you are sending the letters in a series of ones and zeros, with absolutely no other symbols or breaks (that would signify something), don't you have to have more than 28.2.
Suppose you give the first sixteen letters of the alphabet values with 1s and .0s.
Such as a = 0000
b= 0001
c=0010
d=0011
and so on. When you hit p you will run out of possibilities and have to use five symbol ones. Any five symbol signal you devise will be one of the four symbol ones with a 1 or 0 tacked on the end (or the beginning, depending how you think about it).
If Alice sends six of these signals in a row, all the letters will run into each-other. If she sends a five-bit letter followed by a four-bit letter it will be impossible to tell from a four-bit letter followed by a five-bit one. If she pauses between her signals, that would be like another symbol (she could then choose from 1 0 or off).
Supposing Bob sent messages in between hers, asking questions, we would be back down to 4.7 per word, but Bob would be wasting a lot of time encoding each of his questions letter by letter. It would still be more efficient for Alice to encode each letter into a five-bit symbol and Bob recognize that each five-bits represented one word.
Also, won't Alice have to send additional information at the beginning of each message clarifying if she's sending words or numbers or cards or whatever else?
• Thanks, I'm answering this so I could find useful questions from my profile page
(1 vote)
• At It says that information isn't always random, so couldn't one arrange the most common or predictable information so that it takes fewer bits to transmit? Like at where the number of questions can be at least four but at most five, the most common letters could be arranged to take only four and the least common to take five bits so the average power of two could be reduced, leading to a more streamlined information transmission system.
• This is actually what Morse Code does, so yes, this might be the "next step"
• Can someone explain what a logarithm is to me? I'm only in Algebra, and I haven't learned that yet. What kind of a function is it?
• Did anyone else notice that at the end, the alphabet has x and y switched?
• Hahah, no, I did not notice! You must have been paying great attention!
(1 vote)
• What are the 5 cards in a poker hand?