The need for careful analysis of all assumptions that go into a mathematical model, and a corresponding willingness to investigate the predicted output of a model vs. what is actually observed, is sine qua non for all mathematics modelers.
I mention this because I just heard of the death of Ida Hoos - someone whom I was unfamiliar with, but who published frequently on the potential problems with mathematical modeling in the social sciences.
From the 5/5/2007 NYTimes obit by Katie Hafner:
...Dr. Hoos, a sociologist, was widely recognized as an outspoken critic of systems analysis, which came to prominence after World War II. The approach used mathematical models to perform cost-benefit analyses and risk assessments on complex technologies like radar systems and military aircraft. With the concept strengthening in the 1950s and ’60s, when the use of computers to assess technology grew more popular, she wrote widely on a need to balance it with other considerations like effects on the work force. “A kind of quantomania prevails in the assessment of technologies,” Dr. Hoos wrote in 1979 in the journal Technological Forecasting and Social Change. “What cannot be counted simply doesn't count, and so we systematically ignore large and important areas of concern.” Dr. Hoos urged national decision makers to take such assessments “with a large measure of skepticism lest they lead us to regrettable, if not disastrous, conclusions.” Harold A. Linstone, emeritus professor of systems science at Portland State University and longtime editor in chief of Technological Forecasting and Social Change, said Dr. Hoos was in many ways the intellectual conscience in the field of technology assessment. “She basically pointed out that in a lot of complex social and technical systems, a reliance on these systems analysis approaches couldn’t always do the job,” Dr. Linstone said. “She would not accept the superficial answers or phony arguments.” Dr. Hoos also questioned the usefulness of systems analysis when evaluating public policy. Her 1972 book, “Systems Analysis in Public Policy: A Critique,” cast a critical eye on the prevailing methods for evaluating education, waste management and health care. “These technical-think-tank types were riding high,” and Dr. Hoos “wasn’t averse to pointing out that the king was naked,” said Louis Feldner, an engineer who worked with her on several technical committees over the years. “And she was respected for it.”
All administrations, regardless of party affiliation, use mathematical models to set their agendas, frame their policies, and govern this country. More citizens should be aware of both the extreme care that must be taken in using certain models, and the potential pitfalls of modeling if something is amiss in the assumptions that go into the model - often times this is much harder to spot than problems in analysis or computation. What’s worse is ignoring potential conflicting effects because they are not amenable to mathematization, leading to Hoos’ statement What cannot be counted simply doesn’t count, and so we systematically ignore large and important areas of concern if something is inherently "uncountable."
Certainly today, with the explosive world-wide growth of the discussion of climate models, I read much less about the social effects of climate change than I do about the raging scientific debates. These debates are usually misnamed - they not debates, really, but often self-described "facts"slung from one side to the other. Ida Hoos should be essential reading for anyone attempting to understand the human side of the modeling process, and especially for those who need to make public policy decisions based on these models.
To see whether Ida Hoos’ ideas have made their way into current think-tank modeling efforts, I recommend a for-profit group such as 12 Manage: Rigor and Relevance in Management , where you will find some cost-benefit analyses and much more
For an academic approach, see The Institute for Complex Additive Systems Analysis at New Mexico State. The institute is a "a cooperative alliance among academia, industry, and government" with a focus on understanding "the additive effects—or unintended consequences—of efficient design in interdependent systems of systems…The Institute’s research is characterized by the study of dynamical systems, control theory, mathematical physics, and economics using the tools of theoretical analysis, modeling, and simulation."
Thanks to philosopher and colleague Mike Kerlin of La Salle U. for informing be about Ida Hoos.