Re: [Discuss-sudbury-model] scientific backing of Sudbury

From: Mike Weimann <mike.weimann_at_gmx.net>
Date: Tue Mar 14 05:28:00 2006

Hi,

Mimsy Sadofsky schrieb:
> Try the following books from Sudbury Valley School Press: "Legacy of
> Trust" and "Pursuit of Happiness". Mimsy

I have read the latter book and had some criticism on it, for instance,
that it "actually had almost no scientific/academic method underlying".
This was rejected by Daniel Greenberg on the IDEC mailing list. He spoke
of my "ill-conceived accusations" and he said that I owe a "detailed
explanation of such a dismissal of a massive study".

I have written down some arguments and want to share it with the readers
on this list. There were some threads of discussion about "science" in
January, so I thought this might be of interest. If you like, I can copy
Daniel Greenberg's criticism from the IDEC mailinglist here as well.
Just let me know.

Who of the readers here has read the book "The Pursuit of Happiness"?

Mike

---------copy from: IDEClistserve_at_yahoogroups.com ----------

Dear readers on this list,

I want to reply to Daniel Greenberg's accusation that I made
"unsupported attacks" on his book "The Pursuit of Happiness".

The whole "dialogue" began with Laura Stine's mention of a longitudinal
study of former students of Sudbury Valley. I thought there was another
study - but obviously the "Happiness"-book is the one she was thinking
about.

When I read the book last year I always compared what I read with the
standards of analysis of empirical data I was confronted with as a
meteorologist. I have studied and worked in this profession as a
scientist for almost 17 years. In both atmosphere and society you have
to deal with data which can't or almost cannot be put under laboratory
conditions - like e.g. in solid state physics, where you can fix
temperature or pressure or other influencing variables. The way to
overcome this disadvantage is to collect a sufficient quantity of data
and eventually to apply statistical methods. The latter is the only way
to eliminate the disturbing effects which are of no interest to the
investigation. An essential part of the idea of this procedure is that
the disturbing effects are distributed randomly through the whole amount
of data collected. Hence the whole procedure fails if there are unknown
effects which do not average out at about zero. Depending on some
assumptions about the characteristics of the data to be investigated you
have the choice of several statistical tests to prove whether your
conclusions are right (statistically firm) or not. If you are able to
prove a theory with the help of your data, then you are able to predict
the future behavior of the system, which is a central element and aim of
scientific work. In this context it makes no difference if we talk about
rats or human beings, you have to investigate both in a similar way, but
the number of unknown parameters is higher for the latter (among other
reasons because of reduced ability to intrude on a human being).

For instance you might be interested in the relationship between solar
radiation and the increase in surface temperature. To find out a general
function connecting both quantities you should measure both very often,
and then apply statistics - and nevertheless you may come to a wrong
result because you didn't take into consideration the humidity which
influences the temperature significantly through the effect of evaporation.

Daniel Greenberg and his friends - and this is one of my criticisms of
their study - underestimate or neglect some parameters of this type of
relevance. On page 32 - not hidden, but not on page 1 or 2 where, from
my point of view, this should have been mentioned - they concede "our
former students are somewhat skewed towards being more middle-class,
more suburban, more New England and more educated". Daniel Greenberg et
al. therefore (correctly) didn't expect a correlation between the
breakdown of major job categories for the general population (of the
USA) and the population of their alumni. But despite this awareness they
than proceed with the statement, "the distributions are more disparate
than we expected. [...] a much higher percentage of alumni than the
general population were in fields such as Management; Business and
Financial; Computer and Mathematical; Education; Training and Library;
Arts and Design; and Entertainment and Media." Etcetera.

There is no explanation for this or conclusion drawn from it. They
simply say: "more than we expected". If I had done this study I would
have emphasized this particularity of my sample. Because of its
importance for general statements about the success of the students I
would have tried to quantify and to specify this effect. They instead
talk about "somewhat skewed". As far as I remember (!, I don't have the
time to check this at the moment - maybe someone who is starting to read
the "Happiness"-book now can correct me after having found something),
as far as I remember they never come back to the influence of this
important middle-class effect in the later parts of the book.

I want to add some more details from this part of the book. The
corresponding diagrams 8 (national breakdown of job categories) and 9
(breakdown of job categories for the respondents of the study) show the
percentages of the respective population holding jobs in each category.
The sum of all categories of the national breakdown is 100%. The sum of
Greenberg's diagram is 156%. (To fit this large number into the page, by
the way, they use a different scale for the axis of ordinate.) They
argue, that some former students "held jobs in more than one category".
Do other US citizens always hold jobs in one category only?

You as a reader should certainly not complain about this, because on
page 20 the authors mention: "Often, the total number of respondents
displayed in a particular figure either exceeds or falls short of the
total number of interviewees (119). The number exceeds 119 when some of
the respondents fall into more than one category being graphed. The
number falls short of 119 when some of the respondents did not provide
information relevant to the categories being graphed."

This is easy to understand - but it results in diagrams of reduced
value. By this argumentation you no longer know how many respondents are
in the graph shown. For example in the chapter Lifestyles (p. 172 ff.)
you see figures 30 - 37 showing the frequency of occurrence of different
activities and preferences of the respondents. The number of answers in
these eight diagrams is 265, 148, 126, 76, 96, 163, 115 and 73. But you
are not told whether the 126 answers, for instance, represent all the
119 different people or perhaps only 30 who have about four interests
each in the respective diagram. Almost all graphs of this type
throughout the whole book lack information about the number of people
involved. In the above mentioned 156% figure (figure 9) you even don't
know if this 156% results from 50, 100, 200 or whatever number of answers.

Another example, which may give another glimpse of my impression of the
book is on p. 178 and p. 182. Diagram 38 shows "Unrealized personal
goals". One of nine categories is financial security. This is mentioned
by only 16 respondents. But you don't know from this, if exactly 16
former students regret this lack of financial security or if actually
others lacked it as well, but didn't like to reply (for whatever
reason). Does this mean, that no others put financial security as a goal
- or does it mean, that they realized this goal? If only the small
number of 16 regret it, you don't know whether this is a result of
having attended Sudbury Valley or of a upper-middle-class family in the
background which contributes to the financial security. You don't know
if it happens at the level of a upper middle class person, who
encounters some financial trouble or of a poor person who really depends
on social welfare support.

On page 182 the number of "Personal goals being realized" is six and
apart from "Health" you see none of the categories from page 178. You
then find the following statement, which is certainly chosen as typical:
"I like the infinite level of possibilities about my life. I like having
four limbs. I really enjoy the fact that I can get up and walk anywhere
in the United States. I can swing from tree to tree. I can be active and
whole. It is very difficult to throw away all of the privileges that I
was born with. I was born with every last privilege that you can
possibly be born with. I was born white, American, and male. I'm still
young. I have the use of all four limbs and all of my brain cells."

I couldn't find an evaluation of this comparison of realized and
unrealized goals in the book. Like this example much of the data is just
anecdotes that have been grouped into categories. As I mentioned in one
of my previous mails, I find these examples rather interesting. But they
don't tell the effect of Sudbury Valley. You always can have the
hypothesis that the same person could have told a similar story after
having attended another school. At least there is no test of statistical
significance.

Another example: On p. 355 in the chapter on "Mobility" Daniel Greenberg
and his friends write: "The alumni loved to travel, and made it a high
priority in their lives. Figure 71 displays their travel experience
around the world." There you find 189 answers distributed over 10
regions in the world. If you disregard USA and Canada 156 cases are
left. If you believe (!) that all 119 have answered this question, you
get an average of 1.3 travel experiences per head. My questions are: Is
1.3 travel experiences a high number? For whom? For US citizens? For
suburban, middle-class people in New England? (Again: Were there really
119 people who responded or actually only 50?) How should parents or
other people judge the quality of Sudbury Valley with the help of these
figures?

Doesn't everybody "love to travel"? Did the former students travel
because of their social background / class or because of Sudbury Valley?
- No answer to this. The authors don't even ask the question.

What I missed in the book was a working hypothesis - connected to
falsifiability (http://en.wikipedia.org/wiki/Falsifiability). "In
scientific inquiry any theory that is not falsifiable is unscientific."
The question of the "Happiness"-book is (p. 7): "What kind of adults
develop from children who have been trusted to determine the course of
their own education?"

This question doesn't fulfill the criteria of falsifiability. Number
one: there are many adults of the same kind who have come through the
traditional system. Number two: there is no proof that all students from
Sudbury Valley develop into this "kind of adult". How is this "kind of
adult" finally described, other than by the high number of quotations
from carefully selected extracts from interviews?

In the opening chapter, Overview, p. 12, the answer is given already:
"Reading this material, one cannot help but come away with the feeling
that these people are an interesting, thoughtful, articulate, and varied
group, who lead purposeful lives filled with challenge, hard work,
excitement, and meaning - and who are comfortable with themselves and
with the rapidly changing world in which they live."

I can accept this sentence. But the question here is about the
scientific character of the study.

The authors of the book are Daniel Greenberg, Mimsy Sadofsky and Jason
Lempka. I have been wondering why the original authors' names are not
given? Daniel Greenberg wrote, "the study was conducted by people who
were at arms length to the school, and not associated with it in any
capacity." Why don't the authors honour these people by mentioning their
names? Did they only do the phone interviews and set up the tables and
graphs? Who decided about the numerous quotations and who wrote the
comments between the quotes and the graphs?

On page 319 in the chapter "Major Influences" you can read, that the
fact the respondents "were aware that this study was being sponsored by
the school may have brought their time at school to the forefront of
their minds, but the intensity and clarity of their comments seem to
reveal sincere, deep-seated feelings that go well beyond casual
recollections." The related diagram contains 133 answers in 5
categories; 73 of these answers are connected to the Major Influence
"Attending SVS". (You are not told whether the total number of
respondents to this question was 73 or what else.) The description of
this "result" with words like "the intensity of their comments" (without
any objective measure for this intensity) and "seem to reveal sincere
feelings" is not what one should expect from a serious survey analysis.
I am not familiaar with the terminology in English, but if you are in a
field of social science and empirical data analysis you should take as
much care as possible not to have an interference with the behavior of
the investigated people. You should avoid intrusive methods or at least
combine them with unobtrusive components (Nicht-reaktive Verfahren). An
interview by phone conducted by people who are labeled as being
sponsored by Sudbury Valley itself is a feature of the whole study that
can be called a constructional flaw.

I would be very interested to see any other review of this book, in
particular in a journal which has some reputation. Or maybe Daniel
Greenberg can name some of the "many, many readers who took the trouble
to study the book carefully, [of which not one] complained about
unspecified cohorts - precisely because there were no such instances".

Mike Weimann

-- 
Mike Weimann, Winsstraße 4, 10405 Berlin
http://kraetzae.de | http://idec2005.org
Received on Tue Mar 14 2006 - 05:27:27 EST

This archive was generated by hypermail 2.2.0 : Mon Jun 04 2007 - 00:03:14 EDT