My #3 Son was reading Poe's The Gold Bug and so he and I were discussing solving crypto puzzles. My son mentioned the letter frequencies Poe gave and I said Poe was wrong. So I asked him if he was sure he remembered it correctly. My son said he was sure he remembered it correctly and got the book out to prove it. Of course I had read Poe many years ago, but I didn't remember the letter frequencies Poe gave. Here are the letter frequencies Poe gave in order from most common to least as:
e t a o i n s h r d l u c m f w y p v b g k q j x z
and like any good crypto puzzle solver let me put them one on top of the other:
e a o i d h n r s t u y c f g l m w b k p q x z
e t a o i n s h r d l u c m f w y p v b g k q j x z
Do any of you have an explanation for the differences at least among the most common letters? Was he trying to make it harder for his readers to solve the crypto puzzles he often published?
I speculate that Poe didn't have access to a good frequency table and generated his own, perhaps from an atypical sample, or from too small a sample size.
SBP · January 1, 2009 11:03 PM
Why doesn't Poe include "j" and "v"?
Anonymous · January 1, 2009 11:07 PM
The frequencies he gives are from his own cryptograms, none of which decodes to "vajayjay," because it was old-timey days.
guy on internet · January 2, 2009 01:21 AM
SBP,
You would have a point if the letters "in error" came at the end of the table.
However misplacing t d and h so significantly and t especially seems intentional.
It might be interesting to scan a number of texts to see if it is possible to find one with that frequency pattern. Poe may have hidden a cryptogram in his letter frequency chart.
Wasn't Poe an accomplished cryptographer who claimed that no one in the world could design a cipher he couldn't crack? He apparently succeeded when he challenged readers of his newspaper column to try to stump him.
It seems more likely that he gave bad information on purpose. Perhaps he didn't want to give away too much information. It might generate interest in the puzzles but not give people an absolute key to solving them. The rest of his readers, who still didn't go in for the puzzles, would be none the wiser but could still enjoy the story with its internal logic.
No mystery. Poe wrote 150 years ago. English word use was different then. And Poe was just one person, not the English Speaking Peoples. His table might have been right for the United States of his day.
An interesting test would be a table made from Poe's writings. That would be English as he used it.
Type would have still been set by hand for Poe. Mark Twain famously invested in early typesetting machines and lost a fortune.
K · January 2, 2009 11:50 AM
Intriguingly, this is the ranking for a book cipher which uses the first letter of each word.
The ranking for first letters in English is
t o a w b c d s f m r h i y e g l n o u j k
whereas that for every letter in English is as you quoted.
If you multiply the numerical frequencies for first letters by the numerical frequencies for every letter, the ranking will be close to that given by Poe.
Classical Latin uses j and i interchangeably, and the same for u and v.
I wonder if Poe was working from a Latin text?
SBP · January 2, 2009 02:00 PM
The Wikipedia page doesn't give a table for Latin, but the Italian table has "t" at 5.62% and the Spanish table has "t" at 4.63%, both significantly lower than the 9.06% for English.
SBP · January 2, 2009 02:08 PM
Kluber gives the percentages for Latin as:
I - 10.1 M - 3.4 V - 0.7
E - 9.2 C - 3.3 X - 0.6
U - 7.4 P - 3.0 H - 0.5
T - 7.2 L - 2.1 J - 0
A - 7.2 D - 1.7 K - 0
S - 6.8 G - 1.4 Y - 0
R - 6.8 Q - 1.3 Z - 0
N - 6.0 B - 1.2
O - 4.4 F - 0.9
Alex Green · January 2, 2009 03:47 PM
Poe's table is probably based on a pretty small sample size. Note that in
eaoi(dhnrstuy)(cfglmw)(bkpqxz)
the three parenthesized groups are alphabetized. My guess is that this means they all had the same (or statistically indistinguishable) counts in his sample, so he just listed them in order. If this is true, then only the T has really shifted very much.
The Anti-Idiotarian Rottweiler
Don?t bang your head!
Simon Singh's terrific "The Code Book" has a crypto problem at the end. He offered money to the first person to solve it. Someone did.