Recruited to Bletchley Park by Alan Turing, whom he had met at Princeton University, New Jersey, in the late 1930s, Wylie arrived at the Buckinghamshire estate in February 1941. He joined Turing's section, Hut 8, which was working on the German navy's Enigma encryption device, and became head of the crib subsection, working to identify words or phrases expected to form part of encrypted texts (such as weather conditions or the locations of allied attacks) as a key to deciphering them.
Hugh Alexander, who succeeded Turing as section head, said that "except for Turing, no one made a bigger contribution to the success of Hut 8 than Wylie; he was easily the best all-rounder in the section, astonishingly quick and resourceful".
In autumn 1943, Wylie joined the team that was dedicated to cracking the teleprinter cipher machine, known as "Tunny". Their work ultimately enabled the allies to decipher all messages relayed from Hitler to his frontline generals. Wylie would later state that "the breaking of the Enigma machine ciphers is invariably cited as the great achievement of the Bletchley Park codebreakers. But the breaking of the German enciphered teleprinter traffic was ... far greater."
Born in Oxford to Sir Francis and Lady Kathleen Wylie, he was educated at the city's Dragon school and then Winchester college, and won a scholarship to New College, Oxford, where he studied mathematics and classics. In 1934 he went to Princeton, obtaining a PhD in topology in 1937, then carried out post-doctoral research at Aberdeen University, during which time he also represented Scotland in international hockey.
His time at Bletchley Park was not solely spent breaking German codes. He became president of the dramatic club, indulging his love of theatre. After his transfer to work on Tunny, he met Odette Murray, a Wren based in the section, and the pair married in 1944. After the war, Wylie went to Trinity Hall, Cambridge, where he was a fellow. With Peter Hilton, he co-authored Homology Theory: An Introduction to Algebraic Topology (1960), and he was consulted by the pioneering DNA researchers James Watson and Francis Crick over whether the two helices could move apart or would need to unravel: "unravel," came the answer.
One of his former students, the computer pioneer Norman Sanders, recalls Wylie as a particularly gifted teacher: "He lectured on linear algebra, a somewhat abstract subject for many. But his lectures were miracles of clarity and 50-odd years later I still remember his little diagrams of the undiagrammable with vast amusement. Mathematics is not perhaps the most hilarious of subjects, but in Shaun's hands -- and voice -- it was pretty close."
In 1958, Wylie became chief mathematician at GCHQ, the UK signals intelligence organisation, in Cheltenham, another culture of secrecy that left even his close family in the dark as to his daily travails.
After stepping down from GCHQ in 1973 he taught mathematics and Greek at Cambridgeshire high school for boys for seven years. He was elected an honorary fellow at Trinity Hall in 1980.
Never one to rest on his laurels, he had a full and active retirement. He was a founder member of the Social Democratic party and an active supporter of the Liberal Democrats. Like many Bletchley Park veterans, he was also a crossword enthusiast, going as far as to become a setter for the Times's Listener crosswords under the nom de plume Petti -- "wyliecoat" being an old Scottish term for petticoat.
In 1999, he and Odette were interviewed for Channel 4's Station X series about Bletchley. Wylie contributed to the burgeoning Enigma literature through interviews and with his own essay, Breaking Tunny and the Birth of Colossus (2001). Asked whether he had any regrets, he simply said: "I wish I'd been a better mathematician."
He was predeceased by Odette and his eldest son, Keith, and is survived by his daughter, Rowan, and sons Malcolm and Bartow.
Shaun Wylie, codebreaker and mathematician, born 17 January 1913; died 2 October 2009.
28 October 2009 © Guardian Newspapers Limited