Analyzing research design: Being treated artificially

In the world of quantitative research, there are true experimental and quasi-experimental designs.  It is extremely rare, if not impossible, to have a true experimental research project in the realm of education.  Simply, to be considered true experimental, the participants have to be completely random, from anywhere, pooled from everyone available.  As you can see, research like this is nearly impossible when you want to test one treatment against another in a quantitative educational research project.  For us, there are all sorts of regulations in place at the university, district, class, and student level.  Each one has its own barriers, and each one can thwart a research project at any time.

You will more often see the quasi-experimental design.  Here, researchers use several classes.  These can be whole class units or a class split into groups.  The effect is similar in that one group will get one treatment; another group will get another treatment or be a control group.  There could be just two groups or more than two depending on the number of treatments being measured.

While this design has a purpose, I am, personally, not fond of the two group (control and treatment group) types of research designs in writing research for two reasons:

1.) Any time you explicitly teach a topic or genre, students are bound to do better than when something is not explicitly taught.  These designs often pit an implicit teaching design versus an explicit teaching design.  

For example, the research may use how a teacher normally teaches something versus the new treatment that the researchers want to try out.  When the researchers come in and teach something, or train the teacher how to teach it, more often than not the treatment group exceeds the control group in results.  

Logically, any time you explicitly teach something, or add an additional factor of the research teaching students (thus adding interest only because someone other than the teacher is teaching), or additional training for the teacher (meaning he or she may present the material with more zeal), the treatment group already has more going for it than the control group.  This is a validity problem inherent to all such research like this, no matter how many steps are done to try to minimize the effect.

A slight variation, and only slightly better in design, is when more than one treatment is present to find out which treatment works better.  Again, if there is a control group, they will almost always come in behind the treatment groups.  A researcher might find out which of the treatments is better but that only tells us which is better among those chosen for study.  A different treatment may be far superior but was not part of the research design.  While I don’t see this as a failing on the part of the researchers, it is a limitation of deciding which is treatment might be better.

2.) Most of these research projects have short treatment times, such as a few weeks or a month.  Students can learn and retain information in the short term better than in the long term.  If these treatments are then not used, will the students remember them six months from the treatment or a year later?  Again, it is rare for a research group to go back a year later to find out if the students remember the treatment.  If they do go back or are planning to go back (in the event of preliminary data), the timeline is noted in the research methodology.  

The flaws in research design are just one reason that reading, analyzing, and questioning the different sections of a research article is important.