He had suffered a long illness, his family said.
Professor Birkhoff, who explored the rarefied regions of the abstract in modern algebra, also carried out research on scientific computing, reactor theory, differential equations and the history of mathematics. He retired in 1981 as George Putnam Professor emeritus, having held the chair since 1969.
Born in Princeton, N.J., he was the son of an eminent Harvard mathematician, George D. Birkhoff. From his father and Harvard associates Garrett Birkhoff acquired his mathematical footing and the building blocks for his own theoretical constructs.
After graduating with a B.A. from Harvard College in 1932, he began to teach at the university in 1936 without ever receiving master's or doctoral degrees.
His early research was devoted to a fledgling branch of algebra based on what is now known as the lattice theory. His book by that name demonstrated the highly abstract theory of structure and relative magnitude. First published in 1940, it was reissued in a third edition in 1967.
He left a legacy of other pioneering texts for his students. They included ''Survey of Modern Algebra,'' with Saunders MacLane (1941, fourth edition 1977); ''Hydrodynamics'' (1950, revised edition 1960); ''Jets, Wakes and Cavities,'' with E. Zarantello (1957), and ''Ordinary Differential Equations,'' with G. C. Rota (1962, third edition 1978).
Also, ''Algebra,'' with Dr. Mac Lane (1967, revised edition 1979); ''Modern Applied Algebra,'' with Thomas Bartee (1970); ''Source Book in Classical Analysis'' (1973), and ''Numerical Solution of Elliptic Problems'' (1987).
Many of his books are considered classics and remain in print. At his death he was working on a history of mathematics at Harvard.
His investigations of fluid dynamics derived from his experience in World War II when he mathematically analyzed the explosion of bazooka charges and phenomena associated with air-launched missiles entering water. In recent years he served as a visiting professor at Vanderbilt, Carnegie-Mellon, Purdue and Drexel Universities and Georgia Tech.
The author of some 200 research papers, he was Phi Beta Kappa Lecturer in 1985-86.
Professor Birkhoff is survived by his wife, Ruth Collins Birkhoff; a son, John, of Englewood, N.J.; two daughters, Ruth, of Carlisle, Mass., and Nancy Asher of Water Mill; and five grandchildren.
By WOLFGANG SAXON, November 28, 1996 © The New York Times Company