Moshe Rubin’s Cryptologia Chaocipher article, now viewable online…

by ⋅ 11 Comments

Moshe Rubin just emailed me to let me know that his extensive October 2011 Cryptologia article “John F. Byrne’s Chaocipher Revealed: An Historical and Technical Appraisal” (vol. 35 issue 4, pp.328-379 [!!!]) can currently be viewed and downloaded for free from Taylor & Francis (who publish Cryptologia), via the “Download full text” button there.

If (like me) you’re into both the social and technical aspects of historical cryptography, it’s a cracking old read, covering both Byrne’s life and his numerous attempts to get the US military to accept his “Chaocipher” invention. Yet Moshe’s article is far from all ra-ra-pro-Byrne stuff: it also makes clear…
* the system’s inherent fragility (because each step changed the state of the two rotors, it suffered from near-worst-case error propagation);
* Byrne’s cryptographic inexperience (the way that he proposed concealing the indicator settings was far from secure); and
* Byrne’s cryptologic naivety (he believed that the flat letter distribution of the ciphertext made it explicitly unbreakable).

If you’ve read Ratcliff’s “Delusions of Intelligence” (a book the GCHQ Historian recommended I read, thanks for that!), you’ll know that this last mindset was precisely what the various German agencies using the Enigma machine suffered from: and if Chaocipher had been extensively used by the Allies in WW2, who’s to say that Hitler’s fragmented array of codebreaking agencies wouldn’t have eventually found a way of breaking into it, just as they did with virtually all the Allies’ low-to-medium-echelon ciphers?

One thing that strikes me most about the whole saga is that even though Byrne (who sometimes wrote under the anagrammatic pseudonym “J. F. Renby”, I was amused to see) seems to have envisaged Chaocipher as an expensive-to-build set of mechanical rotors, I think it is actually very easy to use with two Scrabble alphabets arranged in horizontal rows. (OK, Scrabble wasn’t devised until the 1930s, but my basic point still stands regardless). All the sliding operations (zenith / nadir, etc) then become immediately straightforward, arguably far more so than if you were using a machine to do the same.

Regardless of whether or not Scrabble tiles are the best way to Chaocipherify your plaintext, I’d argue that what sets Byrne’s cryptographic ideas apart most is the way he conceptualized his crypto system in terms that mesh peculiarly well with modern computer science: in fact, it’s quite hard to describe it at all without lapsing into contemporary CompSciSpeak. It’s almost as if Byrne were projecting himself forward into a software world: but then again, one of the chapters of his autobiography was SciFi, so perhaps the future was where he felt most at home! 🙂

Posted in: Chaocipher

Chaocipher updates…

by ⋅ 3 Comments

Like hourly buses on a wet winter morning, here’s a pair of Chaocipher pages that arrived at my stop one after the other, both discussing how to break John Byrne’s Exhibit 1, and both strongly recommended reading for those interested in the Chaocipher.

First to arrive was Carl Scheffler’s page on Exhibit 1 (but you might perhaps want to read his introductory page on the Chaocipher first, complete with nice coloured disk diagrams). By looking for long sequences of repeated symbols, he managed to reduce the staggeringly-large search space down to a mere ~457,000 permutations to check: in fact, he further managed to reduce the space to only 444 permutations, which would probably be achievable even without the aid of computers. Furthermore, once he had discovered the initial ring state, Carl went on to reverse engineer the keyphrase used to set the disks up (‘THINKTHINK’, with the sequence of letters applied to the disks with the pattern LLRLLRLRRLR). He has a further page planned on Exhibit 4 – I’ll let you know when he posts this.

Subsequently, Moshe Rubin’s near-definitive update on Exhibit 1 turned up. As usual, Moshe’s 12-page PDF manages to answer more or less every question you find yourself asking along the way (though admittedly he doesn’t yet know to whom Byrne’s enciphered “CORDIALTHANKSTOLO” was referring). From this, you can also see that Byrne used ‘Q’ and ‘W’ for ‘,’ and ‘.’ (plus ‘Z’ for ‘end-of-line’), hence the plaintext begins “ALLGOODQQUICKBROWNFOXESJUMPOVERLAZYDOGTOSAVETHEIRPARTYW“.

Incidentally, though the idea of encoding punctuation as rarely-used letters is a well-known cipher trick, I find the historical question of when this mechanism was first used particularly intriguing. This is because I’ve long wondered whether the “am” letterpair frequently found at Voynich line-ends might also encipher a rare letter (such as ‘X’). True, there are some Milanese ciphers with letters for scribal abbreviations and contractions (the 1450 cipher for Tristano Sforza enciphers ‘-9’, while the 1455 cipher for Ludovico Petronio Senen has a cipher for ‘subscriptio’), but these seem to belong to a quite different family. I can’t see this in Kahn or al-Qalqashandi, so… what was the earliest cipher to replace punctuation with rarely used letters?

Posted in: Chaocipher, Voynich Manuscript

The Chaocipher got Slashdotted!

by ⋅ 3 Comments

The Internet is a strange thing, a virtual photographer’s jacket crammed with countless pockets of enthusiasts. For example, you beautiful cipher mysteries fans circulate within one bijou (but nicely-appointed) pocket, while the massed legions of Slashdot fans have a Tardis-style hyperzoom lens pocket all of their own. But… what would happen if these two worlds collided?

A chance to find out came in December 2009, when Edith Sherwood’s The-Voynich-Manuscript-was-made-by-Leonardo-da-Vinci-so-it-was website got picked up by Slashdot. From the 4900 overspill visits Cipher Mysteries got at the time, I estimated that she must have had “(say) 30000 or more” visits. This was probably about right, because in the few days since the same thing happened to Cipher Mysteries last weekend, its visit counter has lurched up by 38,000+. The onslaught started on Saturday night, when at its peak the Cipher Mysteries server was getting a new visitor roughly every second. By late Sunday, however, the story had finally slid off the bottom of the Slashdot front page (which only ever lists the ten most recent news items), at which point the tsunami turned into merely a large river. 🙂

According to the server logs, my Slashdotted Chaocipher page was read in 132 countries (USA 52%, Canada 8%, UK 7.5%, Australia 5.4%, etc), while US Slashdotters were mainly from California, Texas, New York, Washington, followed by another long tail. And OK, I know it’s a biased sample, but it was nice to see Internet Explorer in less than 8% of the browsers. One long-standing stereotype did fall by the wayside, though: there was a relative absence of trolls leaving snarky comments. Might Slashdot be *gasp* growing up? 😉

Actually, the nicest thing about the whole episode for me was that Moshe Rubin’s brother in Florida was unbelievably impressed when he saw Moshe’s name pop up on Slashdot. I know it’s only a small thing, but I’m really pleased for the guy, he deserves credit for his hard work and persistence bringing the Chaocipher out into the light.

* * * * * * *

Some quick follow-up thoughts on the Chaocipher…

It strikes me that Byrne’s neologism “Chaocipher” was remarkably prescient for 1918, because the whole idea of “chaos theory” – as per Wikipedia, “the behavior of dynamical systems that are highly sensitive to initial conditions“, AKA ‘the butterfly effect’ – had not long before that been started by Henri Poincaré. The French mathematician had shown that the classical three-body problem sometimes yielded tricksy outcomes that never converged (i.e. to a collision) nor diverged (i.e. to increasing distance from each other), but where the three bodies were somehow trapped in a dynamically constrained yet utterly mad-looking (OK, he actually said ‘nonperiodic’) manner. Yet after this promising beginning in the 1880s, the ‘chaos’ concept’s journey onwards was a particularly arduous (and non-obvious) one: even though people noticed the signatures of this odd behaviour in many different contexts, they had no comfortable vocabulary to describe it until well after Benoit Mandelbrot and Edward Lorenz in the 1960s.

And so I find it neatly uncanny that the Chaocipher appropriates the “chaos” word 50 years earlier than it should, while at the same time exactly demonstrating the properties that contemporary mathematicians now ascribe to it (i.e. “deterministic chaos”). As the cipher’s twizzling steps subtly mangle the order of the letters on the two rotors, both the error propagation and the cipher system complexity sharply ramp up over time, in a (quite literally) chaotic way: to my eyes, Byrne’s Chaocipher is no less artful and pleasing than any Mandelbrot set I’ve ever seen. However, because its mechanism was not disclosed until this year (2010), it is perhaps best thought of part of the secret history of applied chaos: by way of comparison, the earliest paper on “chaotic cryptography” I’ve found was Baptista’s “Cryptography with chaos” in Physics Letters A (1998) [mentioned online here].

So, it might be that as the full story behind the Chaocipher emerges from Byrne’s papers, we’ll discover that he cleverly applied Poincaré’s and Hadamard’s ideas to cryptography: but – between you and me –  I somehow doubt that this is what really happened. In my mind, there’s something both ham-fistedly mathematical and deviously mechanical about the Chaocipher, that makes its mongrelly combination of Alberti’s cipher wheel and movable circular type something that could (in principle, at least) have been devised any time since about 1465. All the same, I think that the single aspect of the Chaocipher that most makes it resemble an out-of-place artifact is that it is a pure algorithm made solid – a bit like a programming hack devised by someone who had never seen a computer. Perhaps programming is closer to carpentry than we think!

Without doubt, the Chaocipher lies just outside the rigid mathematical confines of the cipher development path laid down by the sequence of crytographers since Alberti: and so for me, the most inspiring lesson to be learned from it is that genius need take only a single step sideways to become utterly unrecognizable to the mainstream. Thinking again about the Voynich Manuscript’s cipher, might that too merely stand a single conceptual step beyond our tightly-blinkered mental range? Furthermore, might that also ultimately turn out to be part of the same secret history of applied chaos? It’s certainly an interesting thought…

Posted in: Chaocipher, Voynich Manuscript ⋅ Tagged: , , , ,

The Chaocipher revealed!

by ⋅ 42 Comments

“The Chaocipher” is a devious cipher system invented in 1918 by John F. Byrne: allegedly, it was so complex that nobody could crack his challenge ciphertexts (even with the plaintext to refer to!), yet was so simple that its mechanism was claimed to comprise only two rotating disks small enough to fit in a cigar box, and could be operated by a ten-year-old (admittedly a diligent, determined and well-practised one) to encipher and decipher texts.

Hence, the Chaocipher’s long-standing mystery revolved around three questions:

  1. Was the Chaocipher for real? (i.e. could something so simple really produce such tricksy ciphertext)?
  2. Was it more secure than, say, the Enigma machine?
  3. More to the point, is the Chaocipher actually an unbreakable cipher?

As of a few years ago, only three people knew the Chaocipher’s secrets – John Byrne Jr (the inventor’s son), and two Cryptologia editors (who saw it in 1990 but were sworn to silence). Yet as Chaucer noted, time and tide wait for no man (not even Cryptologia editors) – so there was a very real (and growing) possibility that the secrets of the Chaocipher might somehow get lost forever.

Hence last August, Moshe Rubin – who CM readers may well recall as the zesty Israeli software / crypto guy who not long before had set up the Chaocipher Clearing House website – decided to try to contact John Jr before it was too late, and so cold-called his way through the list of Byrnes living in Vermont. Before long, Moshe found himself in contact with Patricia Byrne (John Jr’s wife) from whom he discovered the sad news that her husband had passed away a year or two previously.

However, because Pat Byrne was already looking for a buyer for her husband’s cryptological material, Moshe put her in contact (via David Kahn) with David D’Auria, the chairman of the National Cryptological Museum’s Acquisitions Commitee. Somewhat surprisingly, after a couple of months Pat Byrne very generously decided to donate the whole set to the NCM, a terrific gesture which I (for one) highly appreciate (and I hope that you do too!)

And so it came to be that Moshe Rubin found himself allowed what he describes as “preview access to some of the material“.  Although he found that the precise setup John Byrne Sr had employed was not immediately obvious from the material to hand, Moshe burnt a load of midnight oil (is elbow grease more or less inflammable?) before finally managing to reconstruct the original algorithm in all its subtly obfuscatory glory.

Just as Byrne had described, his Chaocipher used two rotors (with the plaintext alphabet on the right rotor and the ciphertext alphabet on the left rotor) BUT with both alphabets altered slightly (let’s call this process ‘twizzling’, for want of a better word) after processing each letter. I’ve hacked together a 30-second Chaocipher animation on YouTube to try to demonstrate Byrne’s twizzlification…

Rather than go through the fine details here, I’m happy to refer you to Moshe’s detailed (and very readable) description of the process here: the only significant difference between my video and his text is that because the rotors mesh (and hence physically rotate in opposite directions to each other), the numbering sequence on each rotor is reversed relative to the other – i.e. even though #1 is at the top of each rotor, #2 and #3 proceed clockwise on the right (plaintext) rotor but anticlockwise on the left (ciphertext) rotor. Whereas in his text, both numbering systems run in parallel to each other (which might confuse you, it certainly confused me a little).

Of course, the obvious practical weakness of the Chaocipher is that any errors in enciphering, transmission, and deciphering get near-irreversibly propagated through the rest of the message: which probably makes the whole system too fragile to use in wartime, however cryptographically secure it may be (and, answering the second question above, I suspect that it may well prove to be more complex than Enigma, for it really is quite a fiendish system).

But is it (practically) unbreakable? Well, the obvious answer would be that if it has now been released into the wild, you’d have thought someone in a three-letter-agency (or GCHQ, naturally) would have worked out a clever way in. However, I’m not 100% sure that has happened yet… so, interesting times.

All credit to Moshe Rubin, then, for his persistence and hard work bringing this cipher mystery into the light: he has a Cryptologia paper coming up, and plenty more work to do over coming months (or years?) fleshing out the behind-the-scenes story from the stack of Byrne’s papers now in the NCM. It’s a fascinating slice of cipher history, and I wish him the very best of luck with the inevitable book and selling the movie rights! 😉

* * * * * * *

Update: I’ve added a follow-on Chaocipher post here, discussing the intriguing parallels between the Chaocipher and chaos theory…

Posted in: Chaocipher ⋅ Tagged: ,

All about the Chaocipher…

by ⋅ 8 Comments

I’m getting a bit cheesed off with the Internet: every time I do a search for anything Cipher Mysteries-ish, it seems that half Google’s hits are for ghastly sites listing “Top 10 Unsolved Mysteries” or “10 Most Bizarre Uncracked Codes“. Still, perhaps I should be more grateful to the GooglePlex that I’m not getting “Top 10 Paris Hilton Modesty Tips” and its tawdry ilk.

Realistically, there is only one uncracked code/cipher listing on the web from which all the rest get cut-and-pasted: Elonka’s list of famous unsolved codes and ciphers. But Elonka Dunin has long since moved on (coincidentally, she went from cryptography into computer game production at about the same time that I made the reverse journey), which is perhaps why all of these lists look a bit dated. Perhaps I should do my own list soon (maybe, if I had the time).

Happily, Elonka did manage to nail most of the usual suspects: the Beale Papers, the Voynich Manuscript, Dorabella, Zodiac Killer, d’Agapeyeff, Phaistos Disk, and so on… each typically a piece of ciphertext which we would like to decipher in order to crack a historical mystery. However, one of the items on her list stands out as something of an exception.

For John F. Byrne’s 1918 “Chaocipher”, we have a description of his device (the prototype fitted in a cigar box, and allegedly contained two wheels with scrambled letters), and a fair few examples of both Chaocipher ciphertext and the matching plaintext. So, the mystery isn’t so much a whodunnit as a howdunnit. Though a small number of people are in on the secret mechanism (Lou Kruh, for one), Byrne himself is long dead: and the details of how his box of tricks worked have never been released into the public domain.

Was Byrne’s Chaocipher truly as unbreakable as he believed, or was it no more than the grand delusion of an inspired cryptographic outsider? This, really, is the mystery here – the everything-or-nothing “hero-or-zero” dramatic tension that makes it a good story. Yet hardly anybody knows about it: whereas “Voynich” gets 242,000 hits, “Chaocipher” only merits 546 hits (i.e. 0.0022% as much).

Well, now you know as well: and if you want to know a little more about its cryptography, I’ve added a Chaocipher page here. But the real site to go to is Moshe Rubin’s “The Chaocipher Clearing House“, which is so new that even Google hasn’t yet found it (Moshe emailed me to tell me about it, thanks!) Exemplary, fascinating, splendid – highly recommended. 🙂

OK, enough of the raw factuality, time for the obligatory historical riff. 🙂

I’m struck by the parallels between John Byrne’s device and Leon Battista Alberti’s cipher wheel. Both men seem to have caught the leading edge of a wave and tried to harness its power for cryptography, and made high-falutin’ claims as to their respective cipher systems’ unbreakability: whereas Alberti’s wave was mathematical abstraction, Byrne’s wave was (very probably) algorithmic computing.

Circa 1920, this was very much in the air: when J. Lyons & Co. hired the mathematician J.R.M. Simmons in 1923, the company was thinking about machines, systems, and operational management: mathematical calculators were absolutely de rigeur for them. The first Enigma machines were constructed in the early 1920s (and used in a commercial environment), and there were doubtless many other broadly similar machines being invented at the same time.

Do I think that there was anything unbreakable in Byrne’s box? No, not really: the real magic in there was most likely a programmatic mindset that was cutting-edge in 1918, but might well look somewhat simplistic nearly a century later. But I could be wrong! 😉

Posted in: Beale Ciphers, Chaocipher, D'Agapeyeff Cipher, Dorabella Cipher, Phaistos Disc, Voynich Manuscript, Zodiac Killer Ciphers ⋅ Tagged: , ,

The Chaocipher

In 1918, John F. Byrne devised a novel cipher system (“The Chaocipher“) that he claimed produces the “annihilation of order and design in written language” – an indecipherable cipher. Though “crude”, Byrne’s working model fitted inside a tiny box that had contained “fifty small Havana cigars”. However, even though he apparently managed to get his invention looked at by all the great American cryptographers / cryptanalysts of the day, it was never used by the US (or, in fact, by anyone else). So – was this a huge missed opportunity or merely a grand delusion?

The Chaocipher certainly existed – you can see one of the ciphertext/plaintext pairs below (#3), and there is a complete set of ct/pt pairs on Moshe Rubin’s site. Though we can tell from this that it is certainly a clever system, the details of the Chaocipher’s mechanics are still not in the public domain, and hence it is not known whether it is truly as unbreakable as Byrne believed.

What might be inside Byrne’s cigar box? A series of articles in Cryptologia and elsewhere have helped to rule out most of the more exotic possibilities that have been suggested over the years: much as you’d expect from a pre-Computer Era device, it now seems most likely to have consisted simply of an ingenious set of rotors, where the internal rotor positions retain some kind of state information between processed characters.

As an example of this kind of thing, Moshe Rubin has proposed that (for example) the state of the Chaocipher might hold a kind of variable offset (say, +1, +2, or +3) that is affected in some non-obvious way by each character.

My own key observation is simply that, even though Byrne’s Chaocipher’s prototype fitted in a tiny cigar box, a production unit would have been far more complex: and so the prototype probably required the user to do some of the work by hand that the final machine would have done automatically. I would therefore predict that the prototype required that the encipherer read some figure from a dial or rotor and advance a different rotor by hand according to that figure. That is, perhaps the rotors in the prototype were not physically connected but were only algorithmically connected: a system that could be followed by a (literate & numerate) “ten-year-old”… or rather, by that particular sequential device which did not yet exist but which was just beginning to grow in the collective scientific imagination, the Computer.

As a specific example, perhaps each character spins one cipher rotor by a certain amount to produce a character and a separate rotor by a different amount to produce an offset: and the user then advances the first cipher rotor by that offset by hand. Though this seems trivial to us now, I suspect that in 1918 it would have appeared radical and edgy. In C:-

#define ROTOR_A_SIZE 26
#define ROTOR_B_SIZE 23   // or whatever, probably non-coprime with 26?
char rotor_A[ROTOR_A_SIZE] = { ‘Q’, ‘F’, ‘G’, …etc… };
signed char rotor_B[ROTOR_B_SIZE] = { +1, +3, +2, +3, +1, +3, +1, …etc… };
int A_state = 0; // in range [0…ROTOR_A_SIZE-1];
int B_state = 0; // in range [0…ROTOR_B_SIZE-1];
char chaocipher_encipher(char input)
{
   int a = (int) (input – ‘A’);  // convert letter to rotor position
   A_state = (A_state + rotor_B[B_state]) % ROTOR_A_SIZE;
   B_state = (B_state + rotor_B[B_state]) % ROTOR_B_SIZE;
   return rotor_A[A_state];
}

How you’d decipher this kind of thing is another matter. But I leave that to the reader to work out. 🙂

The best online resource for the Chaocipher is without doubt Moshe Rubin’s “Chaocipher Clearing House“, which includes a scan of Chapter 21 of Byrne’s memoirs “Silent Years” (with Albert Einstein’s marginalia!) As is so often the case with cipher history, Wikipedia’s entry is embarrassingly poor, not even meriting a link here. 🙁

Plaintext #3

THEHISTORYOFWARTEEMSWITHOCCASIONSWHERETHEINTERCEPT
IONOFDISPATCHESANDORDERSWRITTENINPLAINLANGUAGEHASR
ESULTEDINDEFEATANDDISASTERFORTHEFORCEWHOSElNTENTIO
NSTHUSBECAMEKNOWNATONCETOTHEENEMTHEHISTORYOFWARTEE
MSWITHOCCASIONSWHERETHEINTERCEPTIONOFDISPATCHESAND
ORDERSWRITTENINPLAINLANGUAGEHASRESULTEDINDEFEATAND
DISASTERFORTHEFORCEWHOSElNTENTIONSTHUSBECAMEKNOWNA
TONCETOTHEENEMTHEHISTORYOFWARTEEMSWITHOCCASIONSWHE
RETHEINTERCEPTIONOFDISPATCHESANDORDERSWRITTENINPLA
INLANGUAGEHASRESULTEDINDEFEATANDDISASTERFORTHEFORC
EWHOSElNTENTIONSTHUSBECAMEKNOWNATONCETOTHEENEMTHEH
ISTORYOFWARTEEMSWITHOCCASIONSWHERETHEINTERCEPTIONO
FDISPATCHESANDORDERSWRITTENINPLAINLANGUAGEHASRESUL
TEDINDEFEATANDDISASTERFORTHEFORCEWHOSElNTENTIONSTH
USBECAMEKNOWNATONCETOTHEENEMTHEHISTORYOFWARTEEMSWI
THOCCASIONSWHERETHEINTERCEPTIONOFDISPATCHESANDORDE
RSWRITTENINPLAINLANGUAGEHASRESULTEDINDEFEATANDDISA
STERFORTHEFORCEWHOSElNTENTIONSTHUSBECAMEKNOWNATONC
ETOTHEENEM

Ciphertext #3

ODHSTOCOCPBHRSLTANURICIAVZDQWOCPRIWFLQXQPBGRNSJKZY
RHONXXQHRTVNHNCOXOQQLOUNFBWDGSRHBESVACZKKCXQKEVTOV
QBFLBNNAYBYGMNUIUEXTNVIJDLBQTIISKPPQVMFFBBMPMHSPSX
RILKIJDTNCHBXOBLYVVFTPPTGNNJVFLOCLXREZMNXZTUUWLGSW
UEJHYKRWVBQSVKVLPGBEVOQKPNVLLWABXRDZODVGBCRJOEXSHB
TLXCRJJUKACMVTEFENSRXYTOLPLEGGRZRTNFOGGABNLVAKMSKP
KTDIBFTWDFREWOSUABUYIGRSUQCAINGKSBRKWYVVSQLFGGMGVJ
MDFAZDFQXMSEGIVVVQMDWHFJPWAGPAMANHSFYYLFYBFIXQLHTF
KKEFVEAKUIXMSXSZQNPLDVPHAFLQNZRXGRHXZEBWQPYHISNYTL
FUHFSMKPOWDGFDLYQRHIVVUJHFBISOPFKKLZTAYYVAGNTQQECS
FDBVTTMAPOMFNSFLTNMUPTUHJPUORZTISYCMQXEPTKFBSXUJAA
PGZNORYXNUGTUESBABJZVTMDLRYUPSAWCUQXPWEIYGFCCPYNLR
PMYZSLMHAROCPYFNQDDVLEGPISKXASBHMLANCJAUKMYWRSUWNR
IOCHZTXEGWWOROPGJQIGNHJLWDUNRPOGLOWGGREHMFDPVCTKQP
YSAOBNXXUCJGVJEIZCPGEKWHGUKVNJLWQMFGHZKZRPKKDQIKNO
LKTMPKVBZMYQRSPEASAZBNKGQYPWVJEPNQOWRQFBSKKYCJOYFC
PRVJBYYGFSBQEUKIYJELJZKPHHLSXKNHXWBOBIBRGACMUVTZYT
QNJWCJFFSVNIXPBJCSUUOUDSWAGODEXPBXKNSGQVZYKCJLPEKX
PXSREXQLKY

Declassified NSA documents…

by ⋅ 13 Comments

Hans Jahr very kindly left a comment here on Cipher Mysteries recently, pointing me at a treasure trove of declassified NSA documents relating to William Friedman.

Some of these had already been declassified by different NSA mechanisms: but even so, there’s simply so much new material to go through here that it’s enough to make your head spin.

Of course, I’ve started trying to go through these (e.g. there’s not that much new on the Voynich Manuscript, but there’s a lot on the role of Typex II in the post-WW2 years), but it’s going to take a while. Please feel free to browse and search yourselves, and let me know if you stumble onto anything interesting. 🙂

Incidentally, given that Friedman evaluated the Chaocipher in 1942, there may well be some Chaocipher-related correspondence in there somewhere for Moshe Rubin to find. (There’s a large Excel spreadsheet briefly listing out all the items).

Speaking of whom, Moshe Rubin also very kindly dropped me a email about an NSA page containing recently declassified Beale Papers documents.

As you’d expect, many have dated badly and many others are of little use, but that still leaves many other interesting things in the heap. Browse and enjoy; and if you do happen to find something about four miles from Buford’s Tavern… 🙂

More generally, though, the NSA has posted up a list of declassified topics: this includes not only Friedman and Beale, but also VENONA, the death of John F. Kennedy, UFOs, etc etc.

Yet personally, I’m more interested by this unbelievably long list of declassified code/cipher stuff, which includes items relating to just about every country that ciphered anything in the first half of the 20th century. It took me half an hour to speed read through the titles alone!

I picked out a few choice items, firstly on Double Transposition (more on that in a few days’ time):-

NR 1479 CBKH37 6021A 19401200 TRAINING PAMPHLET NO. 40 DOUBLE TRANSPOSITION CIPHERS NUMERICAL SOLUTION
NR 1871 CBLI66 4262A 19340000 GENERAL SOLUTION FOR THE DOUBLE TRANSPOSITION CIPHER
NR 2017 CBLJ73 6206A 19450212 GERMAN POW REPORTS ON DOUBLE TRANSPOSITION CIPHER SYSTEM USED BY AMERICAN
NR 4608 ZEMA36 13947A 19430200 SIGRED-2 INSTRUCTIONS FOR USING DOUBLE TRANSPOSITION CIPHER

Next, some documents on secret writing (at least one person here likes that 🙂 ):

NR 3464 CBQM37 4000A 19200000 SECRET INKS
NR 3465 CBQM37 4852A 19430109 POSTAGE STAMP CODE – ANTHONY E. SCOTTINO CASE
NR 3468 CBQM37 863A 19431129 SECRET WRITING
NR 3469 CBQM37 896A 19430326 WORKSHEETS OF THE LAB BRANCH ON SECRET INK TESTS
NR 3470 CBQM37 897A 19430601 MISCELLANEOUS PAPERS ON POW CENSORSHIP AND SECRET WRITING
NR 3471 CBQM37 898A 19440331 LETTERS CONTAINING SECRET WRITING CONDEMNED BY LAB AFTER EXAMINATION

There’s a report on the Zimmerman Telegram:-

NR 1872 CBLI66 4263A 19380000 ZIMMERMAN TELEGRAM OF JANUARY 16, 1917 AND ITS CRYPTOGRAPHIC BACKGROUND

A few pages I’ll try to get around to looking at before very long:-

NR 3552 CBSC73 6101A 19430500 SECURITY OF ALLIED CIPHERS
NR 3857 ZEMA109 46120A 19431113 UNITED KINGDOM “EXAMPLES OF OPERATIONAL EXPERIENCE IN THE USE OF PIGEONS BY THE R.A.F.”
NR 4660 ZEMA42 4410A 19420120 INTERCEPT OF CARRIER PIGEON MESSAGE

And finally, a fair few photos:-

NR 4181 ZEMA169 41048A 19170000 PHOTOGRAPH. WILLIAM F. FRIEDMAN, SIS, ASA, NSA
NR 4195 ZEMA169 41111A 19420000 PHOTOGRAPH: BRIG. JOHN H. TILTMAN, UK
NR 4693 ZEMA46 41289A 19450000 PHOTOGRAPHS: EQUIPMENT – BOMBE

Posted in: Historical Ciphers

John Tiltman’s 1968 Voynich paper…

by ⋅ 36 Comments

Here’s a particularly interesting Voynich Manuscript paper declassified by the NSA in 2002 (but only recently released as a PDF scan) – so many thanks to the ever-vigilant Moshe Rubin (of Chaocipher Clearing House fame) for pinging me with a link to it, much appreciated! 🙂

Of course, Brigadier John Tiltman’s The Voynich Manuscript – “The Most Mysterious Manuscript in the World” is in many ways no more than an introduction to the VMs (and we have far better scans nowadays, so its pages 15..46 need not really concern us): yet all the same, it does contain plenty of incidental meaty morcels for modern Voynichologists to lightly dine upon. For instance:-

  • “Newbold’s solution left a legacy of ill-feeling which persisted for many years and which I found reflected in a letter which Charles Singer wrote to me in 1957.“
  • Plate 5 [f56r] – “has been identified as some sort of Bindweed (Convulvulus)“
  • Plate 6 [f4r] – “seems to me a fairly natural representation of cross-leaved Heath (Erica)“
  • “All the zodiacal drawings carry the name of the month in the centre in a later hand and in readable script though the language has been disputed.“
  • Tiltman was introduced to the VMs in 1950 by William Friedman (which to me reads a bit like a cryptologic version of Cluedo), and started by looking at some of Quire 20’s starred paragraphs.
  • “Charles Singer, in a letter to me, put the date at late 16th Century. Professor Panoffsky [sic] and the keeper of the manuscripts at the Cambridge Library both independently insisted on a date within 20 years of 1500 A.D., and the manuscript as we have it may be a copy of a much earlier document.”
  • Tiltman discusses Cave Beck’s 1657 universal language at reasonable length (pp.9-10, Plates 21-23), given that Friedman had told him that this was the kind of thing he believed the VMs to be. However, Tiltman was plainly far from convinced.
  • In 1957, Tiltman showed photostats of the VMs to various medieval herbal specialists. One of these was Dr. T. A. Sprague in Cheltenham, who – having spent many years on beautiful, well-annotated herbals – found that its “awful pictures” made him “more and more agitated“.

Yet the real substance of the paper arguably lies in the summary of Voynichese’s properties which Tiltman passed back to Friedman in 1951 (remember that Tiltman was arguably the 2oth Century’s greatest non-machine cipher cracker). Tiltman’s own transcription reads Voynichese as being comprised of 17-odd symbols, several of which have second variant forms. The reproduction of Plate 17 is far from crystal clear, but I’m reasonably sure the following is what Tiltman means (converted to EVA), though I’m not 100% sure about “2” because it seems to overlap with “R” and/or “S”:-

Tiltman:     D H E G 8 4 O A L S 2 R I C  T  DZ  HZ
EVA:         k t l y d q o a n s ? r i e ch ckh cth
2nd variant: f p m                       sh cfh cph

Though most of the paper is dryly factual (though written in an accessible style), Tiltman managed to sneak his own summary into page 9 – “My analysis revealed to me a cumbersome mixture of different types of substitution.” Of course, this is exactly my conclusion too… but regular Cipher Mysteries readers knew that anyway, so I won’t press the point. 🙂

So, the rest of this post contains Tiltman’s notes (but with the notation converted to EVA), with a few light annotations from me. Note that Peter Long mentioned was a senior figure at GCHQ who corresponded extensively about the VMs (he died in 1999): he’s also mentioned on p.216 of Kennedy and Churchill’s book, though mostly in regard to the “K:D:P” (Kelley/Dee/Pucci) theory of Long’s nephew Tim Mervyn, who has continued his uncle’s Voynichological interest. Enjoy!

* * * * * * * *

(a) Following are some notes on the common behaviour of some of the commonly occurring symbols. I would like to say that there is no statement of opinion below to which I cannot myself find plenty of contradiction. I am convinced that it is useless (as it is certainly discouraging) to take account at this stage of rare combinations of symbols. It is not even in every case possible say what is a single symbol and what is not. For example, I am not completely satisfied that the commonly occurring <a> has not to be resolved into <ci> or possibly <oi>. I have found no punctuation at all.

(b) <ckh/cfh> and <cth/cph>appear to be infixes of <k/f> and <t/p> within <ch>. The variant symbol represented by <m> appears most commonly at the end of a line, rarely elsewhere.

(c) Paragraphs nearly always begin with <k/f> or <t/p>, most commonly in the second variant forms [i.e. <f>/<p>], which also occur frequently in words in the top lines of paragraphs where there is some extra space. [Also known as “Neal Keys”.]

(d) <y> occurs quite frequently as the initial symbol of a line followed immediately by a combination of symbols which seem to be happy without it in any part of a line away from the beginning. Otherwise it occurs chiefly before spaces very frequently preceded immediately by <d>. Hence my belief that these two have some separative or conjunctive function. (I have to admit, however, the <y> also seems sometimes to take the place of <o> before <k/f> or <t/p> (though rarely, if ever, after <q>); this is particularly noticeable in some of the captions to illustrations in the astronomical section of the manuscript – these most commonly begin <ok>/<of> or <ot>/<op> and it is here that we occasionally see <yk>/<yf> or <yt>/<yp>.)

(e) I have tried, for convenience of handling, to divide words into what I call “roots” and “suffixes.” This arrangement is show in the bottom of Plate 17. [Which is, as best as I can tell, the following table:-]

Roots: <ok>/<of>, <ot>/<op>, <qok>/<qof>, <ch>/<sh>, <s>, <d>, 2 [??] <lk>/<lf>
Suffixes: either (i)  <e>, <ee>, <eee> followed by <y> or <dy>
            or   (ii) one or more of the following -
                <an> <ain> <aiin> <aiiin>
                <ar> <air> <aiir> <aiiir>
                <al> <ail> <aiil> <aiiil>
                <or> <ol>

Regarding the second type of suffix, some of the combinations are so rare that I have been uncertain whether to take any account of them at all. Some are very common indeed. It seems to me that each of these combinations beginning <a> has its own characteristic frequency which it maintains in general throughout the manuscript and independent of context except in cases where two or more <a> groups are together in series, as referred to later). These <a> groups e.g. <ar> or <aiin>, frequently occur attached directly to “roots,” particularly <od>, <ot>, <d> and <s>.  <okaiin>, <qokaiin> and <daiin> rank high among the commonest words in the manuscript.

(f) There are however many examples of 2, 3, 4 or even 5 <a> groups strung together on end with or without spaces between them. When this occurs, there appears to be some selective preference. For example, <ar> is very frequently doubled, i.e. <ar ar>, whereas <aiin> which is generally significantly commoner, is rarely found doubled. Perhaps the commonest succession of three of these groups is <ar ar al>. <al> very frequently follows <ar>, but <ar> hardly ever follows <al>.

(g) <o>, which has a very common and very definite function in “roots,” seems to occur frequently in “suffixes” in rather similar usage to <a>, but nearly always as <or> and <ol>. <or aiin> is very common.

(h) The behaviour of the <a> (and <o>) groups has suggested to me that they may in fact constitute some form of spelling. It might be, for instance, that the manuscript is intended to demonstrate some very primitive universal language and that the author was driven to spell out the ends of words in order to express the accidence [i.e. the inflectional morphology] of an inflected language. If all the possible <a> and <o> combinations can occur, then there are 24 possibilities. They may, however, be modified or qualified in some way by the prefixed symbols <k>/<f>, <p>/<t>, <ok>/<of>, <op>/<ot>, <ch>/<sh>, <s>, <d>, 2 [??], etc., and I have not so far found it possible to draw a line anywhere. This, coupled with ignorance of the basic language, if any, makes it difficult to make any sort of attempt at solution, even assuming that there is spelling.

(i) <l>, usually preceded by <a> or <o>, is very commonly followed by <k>/<f>, much less commonly by <t>/<p>, with or without a space between. In this connection, I have become more and more inclined to believe that a space, though not intended to deceive, must not necessarily be regarded as a mark of division between two words or concepts.

(j) Speaking generally, each symbol behaves as if it had its own place in an “order of precedence” within words; some symbols such as <o> and <y> seem to be able to occupy two functionally different places.

(k) Some of the commoner words, e.g. <okeed>, <okeedy>, <qokeedy>, <odaiin>, <okar>, <okal>, <daiin>, <chedy> occur twice running, occasionally three times.

(l) I am unable to avoid the conclusion that the occurrence of the symbol <e> up to 3 times in one form of “suffix” and the symbol <i> up to 3 times in the other must have some systematic significance.

(m) Peter Long has suggested to me that the <a> groups might represent Roman numerals. Thus <aiin> might be IIJ, and <ar ar al> XXV, but this, if true, would only present one with a set of numbered categories which doesn’t solve the problem. In any case, though it accounts for the properties of the commoner combinations, it produces many impossible ones.

(n) The next three plates show pages where the symbols occur singly, apparently in series, and not in their normal functions. The column of symbols at the left in Plate 18 [i.e. the vertical column at the left edge of f49v] appears to show a repeating cycle of 6 or 7 symbols <t> <o> <r> <y> <e> <o> <k> <s>. In Plate 19 [i.e. f57v] the succession of symbols in the circles must surely have some significance. One circle has the same series of 17 symbols repeated 4 times, an interesting column of symbols. Plate 20 [i.e. f66r] also has an interesting column of symbols. In all three there are symbols which rarely, if ever, occur elsewhere.

(o) My analysis, I believe shows that the text cannot be the result of substituting single symbols for letters in the natural order. Languages simply do not behave in this way. If the single words attached to stars in the astronomical drawings, for instance, are really, as they appear to be, captions expressing the names or qualities of those stars, there can hardly be any form of transposition system involved. And yet I am not aware of any long repetition of more than 2 or 3 words in succession, as might be expected for instance in the text under the botanical drawings.

Posted in: Voynich Manuscript ⋅ Tagged:

Cipher Mysteries – 100,000 visit milestone!

by ⋅ 10 Comments

The Cipher Mysteries stats to date: 568 posts – 16 static pages – 100,155 visits – roughly 190,000 page loads – 1775 comments – 115 subscribers. Thank you all very much for your continued interest, support, comments (both appreciative and snarky), tweets, and off-line posts and notes (always interesting). Just so you know, most referrals continue to come from Google with only one Slashdot-style traffic superspike (2nd-3rd December 2009, 5000 visits in one day), while the Akismet anti-spam plugin has caught 16,151 spam messages with only 9 false positives (mainly comments from hotmail.com).

It’s also nice to see more cipher mystery bloggers popping up: a big tip of the hat to Elmar Vogt, Elias Schwerdtfeger, Julian Bunn, Diane O’Donovan, Rich SantaColoma, Moshe Rubin, etc (in no particular order). Good luck to all of them (blogging is surprisingly hard work).

As for Cipher Mysteries itself, there’s plenty I’d like to fix: the front page needs a makeover to make it more useful for first-time visitors; the web hosting performance has slowly worsened (but moving such a large database to a new host is proving time-consuming and difficult); and I have a slow-burn plan to get the visit count to 1,000,000 and beyond. But I’m busier than ever in my real-world work, so all in good time, eh?

I have several weeks’ worth of draft posts to finish, and am still working away on an entirely new historical cipher mystery that I suspect will ultimately be more surprising and revealing than the Voynich Manuscript: but I’ll leave that as a ncie surprise for another day. Thanks you all again! 🙂

Posted in: Historical Ciphers, Voynich Manuscript ⋅ Tagged:

D’Imperio’s “Elegant Enigma”, now a downloadable PDF!

by ⋅ 11 Comments

A highly surprising message just arrived here at Cipher Mysteries from our Chaocipher Clearing House chum Moshe Rubin:-

Is your readership aware that NSA has placed the entire text [of Mary D’Imperio’s “An Elegant Enigma”] on its site?

I couldn’t find the link on your site so here it is!

This is a great find, highly recommended for all Voynich researchers – if you haven’t read it already, download it straight away! Having said that, even though it took me six attempts to download it completely (doubtless they were tracert’ing me to see if I just happened to be a terrorist, bless ’em), I did get it all in the end. But please let me know if this happens to get removed (which is always possible).

Incidentally, this is just one of a number of cryptological history publications the NSA has kindly made available on its website. The 2007 article by John Clabby on Brigadier John Tiltman (“A Giant Among Cryptanalysts“) is also well worth looking at (though not nearly so essential as D’Imperio, naturally).

Posted in: Voynich Manuscript ⋅ Tagged: , ,