Mr. Gleason, a professor emeritus at Harvard University perhaps best known for helping solve a key component of "Hilbert's Fifth" problem, died Friday of complications following surgery at Mount Auburn Hospital in Cambridge. He was 86.

Born in Fresno, Calif., Andrew Mattei Gleason moved with his family to New York while he was in high school. After graduating from Yale University in 1942, he enlisted in the Navy and served as a cryptanalyst during World War II, seeking to break Japanese and German codes. He reenlisted in 1950 to serve as a code breaker in the Korean War until 1953.

"He was a mathematician, and many mathematicians were code breakers, because it took a mathematical mind," said his wife, whom he married in 1959.

In 1946, Mr. Gleason was appointed as a Junior Fellow of the Society of Fellows, a select group of young scholars who are given three-year fellowships to pursue their studies without formal requirements at Harvard.

He set about solving Hilbert's Fifth, a problem mathematician David Hilbert formulated in 1900. Mr. Gleason solved a key aspect of the problem with three others, winning the Newcomb Cleveland Prize from the American Association for the Advancement of Science in 1952.

By solving the problem, Mr. Gleason and his team made a tremendous advance in the understanding of symmetries, which are the basis of Hilbert's Fifth, said Clifford H. Taubes of Belmont, former chair of the Mathematics Department at Harvard.

"His biggest contribution was to solve this problem. His work gave one big step," said Taubes. "Solving the problem said things were simpler than they could have been. Now [other mathematicians] don't have to worry about these other cases . . . other people can focus on other parts of symmetry."

In 1969, Mr. Gleason was named the Hollis professor of Mathematics and Natural Philosophy, an endowed chair at Harvard.

He also served as head fellow of the Society of Fellows, helping select junior fellows from 1989 to 1996. He taught mathematics at Harvard until he retired in 1992.

Mr. Gleason also worked on developing new mathematics curricula for students, particularly calculus courses.

"He loved working with children; he was always engaged in math curriculum reform. He was interested in how children thought. He wanted children to understand how mathematics worked," his wife said.

Ethan Bolker of Newton took an abstract algebra class with Mr. Gleason in the fall of 1956 at Harvard.

"One of the things he was famous for as a mathematician was solving Hilbert's Fifth, but it wasn't at all what made him a good teacher," said Bolker, a professor of mathematics and computer science at the University of Massachusetts-Boston.

Mr. Gleason was Bolker's professor through graduate school and his thesis adviser.

"I got to be friends with Andy for the last 30 years. Anything I was teaching we could talk about. I spend a morning a week in an elementary school in Boston, and he and I would talk about what first-, second-, and third-graders could do," Bolker said. "He was very kind and very humane. He approached everything with mathematics and mathematical metaphors."

He also had a passion for astronomy. "He loved looking at the stars. He knew every star in the sky and could tell you their names," his wife said. "Early on, he was planning on becoming an astronomer but then he learned how cold it was" to sit outside and watch.

In 2006, Mr. Gleason sailed along the coast of Turkey to see a solar eclipse and also traveled to the United Kingdom, France, and Kenya to view celestial events, his wife said.

In addition to his wife, Mr. Gleason leaves three daughters; Katherine of New York, Pam of Wagener, S.C., and Cynthia of Framingham; and a sister, Anne Eudey of Walnut Creek, Calif.

By Caitlin Castello

October 20, 2008 © Copyright 2008 Globe Newspaper Company.