lederr

the multitude of factors affecting student achievement (hint, it’s not just teachers…)

In Education, Education advocacy, Pedagogy, School reform on Friday, 25 January 2013 at 09:15

Are teachers the most important factor affecting student achievement?

By: Stephen Sawchuck

This has become the default first sentence of many speeches and reports on teacher quality. Recently, it’s become common to clarify that teachers are the most important “school-based” factor in learning—a critical qualification, given that factors external to schools exert more influence overall on student achievement than any factors inside the school.

A famous 1966 study by James Coleman found that background characteristics such as race, parental achievement levels, and family income swamped most other factors studied as determinants of student test scores. Decades of research have confirmed this study’s general findings, with a 1999 paper estimating that 60 percent of variation in student achievement was attributable to such background characteristics. [1]

Researchers have been unable to link a significant share of the variation in student achievement—as much as 25 percent—to any particular input. Of the remaining share, attributable to what happens within school, researchers have linked most of that variation to teachers.

It is difficult to cite an exact figure on what percent of the variation in achievement observed is attributable to differences in teacher effectiveness. Three economists in 1998 estimated that at least 7.5 percent of the variation in student achievement resulted directly from teacher quality and added that the actual number could be as high as 20 percent.[2]

Researchers have found that school-based factors, including teaching, are more influential in math than in reading. A 1999 paper puts all in-school factors, including school-, teacher-, and class-level factors, at approximately 21 percent of the variation in 10th grade mathematics achievement. It further estimated that 8.5 percent was directly due to teacher effectiveness.[3]

Some researchers warn that other important factors that potentially affect achievement— such as the effect of principals and other administrators, and the interaction of teachers with the curriculum—have not been as carefully studied as teacher quality.[4]

It can be said:

Research has shown that the variation in student achievement is predominantly a product of individual and family background characteristics. Of the school factors that have been isolated for study, teachers are probably the most important determinants of how students will perform on standardized tests.

Are value-added estimations reliable or stable?

Value-added modeling measures individual students’ performance on tests over time, using prior test scores to predict future outcomes. Statistical controls attempt to screen out factors such as race, family background, and the effect of peers, so as to attribute the remaining variation in student academic outcomes to schools and teachers.

At the level of individual teachers, such estimates vary considerably, pointing to differences in teachers’ levels of skill.

Some scholars say that of the measures of teacher effectiveness studied so far, value-added appears to be among the most promising. In one study, for instance, researchers used value-added estimates of teachers to predict the student-achievement patterns of some 3,000 students in 78 classrooms, and then randomly assigned teachers to these classrooms. The value-added models, while not perfect, were significant predictors of actual outcomes. [1]

At the same time, researchers have discovered that a host of factors contribute to measurement error in these estimates. These problems include the nonrandom assignment of students and teachers to schools and classrooms[2]; different effect sizes or results based on the statistical models used[3]; differences in the tests that supply the underlying data[4]; the seeming instability of estimates of particular teachers from year to year[5]; and the fade-out of teacher effects.[6]

In general, the variance in year-to-year estimates of individual teachers’ performance could indicate measurement error. Some of these problems, like the problem of tracking and instability in the estimates, seem to be ameliorated by using additional years of student data for each teacher, though researchers continue to debate this issue.[7]

The implications of these problems, both for policy and for research, are difficult to parse, and policy experts continue to debate the use of value-added as a component of teacher evaluations and for other purposes.

Finally, the research on teacher quality suggests that other school factors may affect how effective teachers appear to be in these types of calculations. One study found that up to a quarter of the estimate of an individual teacher’s value-added score depended on whether teachers were a good “match” for a particular school. Its author postulated that such factors as whether the teacher’s teaching philosophy meshed with the school’s culture and the choice of curricula might contribute to this match effect.[8]

Many teacher groups argue that value-added measures fail to take into account the considerable role of school and district leadership. Researchers are still investigating the role of principals as distinct from teachers, but it is difficult to disentangle the two.

Teachers’ peers may also influence their effectiveness. At least one study has found that a teacher appears to improve when surrounded by more-effective colleagues. [9] But a second paper looking at this question found no consistent evidence that teachers who hold National Board Certification, an independent honor that teachers go through a rigorous process to obtain, have an impact on the effectiveness of their peers.[10]

For a longer discussion of the issue of value-added measurement and its place in policy, see Harris 2011.

It can be said:

Value-added models appear to pick up some differences in teacher quality, but they can be influenced by a number of factors, such as the statistical controls selected. They may also be affected by the characteristics of schools and peers. The impact of unmeasured factors in schools, such as principals and choice of curriculum, is less clear.

What are the differences in achievement between students who have effective or ineffective teachers for several years in a row?

Scholars have expressed the variation among teachers in many ways. An early paper on value-added dating from 1992, for instance, suggested that teachers near the top of the performance curve in a district could get an additional year’s worth of growth out of students compared with the poorest-performing teachers.[1] They have also found that the variation in teacher quality is primarily within schools rather than between them, meaning that almost all schools have both effective and ineffective teachers.

Despite much rhetoric from advocacy groups, the question of whether it’s possible to dramatically change outcomes for struggling students by assigning them to several effective teachers in a row remains an open one.

One of the most influential studies to assert this dates from 1996. Drawing from mathematics test data on Tennessee students linked to teachers over a four-year period, the study separated teachers by estimated effectiveness into quintiles. The researchers then tracked those students’ progression from 3rd through 5th grade.

They found that students assigned to three years of effective teachers outscored comparable students with three ineffective teachers by up to 50 percent points. The authors said that the effects of having several excellent teachers in a row accumulated over time.[2]

More recent findings, however, have called into question the assumption that teacher effects can simply be added together year after year.

A number of researchers have highlighted the “fade out” or depreciation of teacher effects over time,[3] a phenomenon that has appeared in a number of studies using value-added modeling. What this means is that impact of a teachers’ instruction on this year’s scores does not seem to persist when those same students move on to other grades. One 2008 study, for instance, found that teacher effects from one year had only half their impact in the next year.[4]

Second, as noted earlier, estimates of individual teachers’ effects vary from year to year, and are more volatile for smaller cohorts of students or when fewer years of data are incorporated into the estimates.

Therefore, policies that pair the best teachers with underserved students would need to identify teachers who consistently produce strong gains and ensure that such gains compound over time, if those initiatives are to have a lasting effect on student scores. Such policies have not been tested at scale.

It can be said:

Some teachers produce stronger achievement gains among their students than others do. However, estimates of an individual teacher’s effectiveness can vary from year to year, and the impact of an effective teacher seems to decrease with time.  The cumulative effect on students’ learning from having a succession of strong teachers is not clear.

Do teacher characteristics such as academic achievement, years of experience, and certification affect student test scores?

Most of these characteristics have been examined using large sets of data in which teachers are linked to student scores. Several of the characteristics do indeed bear a relationship to student achievement, but in general, scholars say their effects tend to be somewhat weak or inconsistent across studies.

Thus, on average, such characteristics matter. But there are plenty of cases in which teachers with advanced degrees, extensive experience, or specialized credentials are not noticeably more effective than their peers, and there are likewise many effective teachers without such credentials.

The basic problem can be summed up by the conclusion of a study dating from 2005. It found large differences among teachers in terms of their value-added impact on student achievement and went on to note that this variation was not readily explained by factors such as graduate degrees or experience after the first few years in the profession.[1] A 1999 paper, meanwhile, estimated that only 3 percent of the variation in student achievement could be traced back to measurable teacher characteristics, such as teachers’ academic profiles and degrees.[2]

Of these measures, there is good evidence that teachers gain in effectiveness with additional years on the job.[3] In general, value-added analyses show early career experience pays off in effectiveness steadily through at least the fifth year. This effect appears to be more consistent for elementary and middle schools than for high schools.[4] In addition, the impact of experience appears to be stronger than that of most other teacher characteristics.[5]

Reviews of the empirical research on credentials, in general, point to consensus that teachers’ math content knowledge seems to improve students’ test scores in that subject.

One study found this connection at the 1st and 3rd grade levels using a specially constructed measure of pedagogical content knowledge.[6] A second study looking at Florida test data found links between content-focused professional-development credits in math and secondary math achievement.[7] And one study found slight boosts in achievement for middle and high school students taught by teachers with an undergraduate or graduate degree in mathematics.[8]

Information on other content areas is sparse, but one study found a link between teachers’ holding a bachelor’s degree in science and student achievement in that subject. [9]

Studies are mixed on the attainment of advanced degrees and elementary-level student achievement; some studies show positive correlations, others negative ones. On balance the link to achievement is likely tenuous at best.[10] Several studies have found that entering teaching with a master’s degree of any kind does not boost achievement, relative to not holding such a degree, nor does earning such degrees seem to improve outcomes at the elementary level.[11]

Licensure test scores seem to matter more for math than for other subjects. They consistently appear linked to improved student achievement in that subject, at both the elementary level and at the high school level for algebra and geometry. Findings are mixed for other subjects.[12]

As for certification, one study found that students taught by teachers with any sort of certification outperformed those without certification or who were certified out of field.[13] Another found that those taught by teachers with standard certification outperformed uncertified teachers or those with nonstandard certifications.[14]

Much of the information on teacher certification also seems to find benefits primarily for math. At the high school level, teachers with subject-specific credentials in math tended to boost students’ scores more than those teachers who were not certified in that subject.[15]

Scholars have noted that teachers may affect learning more in mathematics, which tends to be taught exclusively in school, than in reading.[16]

Scholars have spent much time analyzing the effects of National Board Certification, but these numerous studies have mixed findings. In a review of the literature, a National Research Council panel concluded that evidence supports the notion that students taught by national-board-certified teachers on average have higher scores than those not taught by such teachers. But it said the evidence doesn’t support the idea that the process itself makes teachers better at their craft.[17]

Teachers who enter the profession with specific sets of cognitive and noncognitive skills also on average seem to be slightly more effective than those who do not.[18]

The policy implications of these findings are, again, hard to parse. Generally speaking, the policy question concerns whether investing in certain teacher characteristics, (by paying a premium for teachers who hold National Board Certification or a master’s degree, for instance), are cost effective relative to other possible investments.

For a longer summary of the research literature on credentials, see Goe 2007.

It can be said:

Teachers improve in effectiveness at least over their first few years on the job. Characteristics such as board certification, and content knowledge in math sometimes are linked with student achievement. Still, these factors don’t explain much of the differences in teacher effectiveness overall.

Does merit pay for teachers produce better student achievement or retain more-effective teachers?

Performance-pay policies have been tried at many different points in the last several decades. Most offer monetary bonuses to teachers who boost student scores, participate in professional development, or meet other criteria, but they do not change base pay.

The literature on performance pay is vast, and a full review lies outside the scope of this paper. Scholars say that the research questions around performance pay are hard to answer in just one study, especially since the questions vary. Do the programs encourage teachers to work harder and make them more effective at raising scores? Do they serve as a recruitment incentive, attracting high-quality teachers, over time changing the composition of the teacher workforce? Until recently, most of the research has focused on only the first question.

Conclusions culled from random-assignment experimental studies, the research “gold standard,” are limited. One review found just nine studies that used a random-assignment or quasi-experimental method to determine whether bonus programs raised scores; some of the studies looked at performance pay in countries outside of the United States. Those studies, in general, showed positive effects, but may not be applicable to the U.S. school system.[1]

In 2010, researchers at the National Center on Performance Incentives at Vanderbilt University released the results of a three-year experimental study on merit pay in Nashville middle schools. Under the experiment, math teachers who increased student scores received a significant bonus of up to $15,000. The study found no effects on achievement outside of 5th grade. Its authors concluded that the program had done little to change teacher practices.[2]

Also in 2010, preliminary results from a random-assignment experiment in Chicago on the Teacher Advancement Program, which includes merit pay as well as other features such as modified professional development, found no effects on achievement or teacher- retention rates. [3]

A handful of other quasi-experimental studies have been mounted to study school reform plans that contain a performance-pay element. Using a method to create “synthetic” comparison schools, one study found apparent benefits for students whose teachers participated in the Teacher Advancement Program, a finding that stands in contrast to the Chicago experiment.[4]

Teachers participating in the pay-for-performance component of Denver’s comprehensive ProComp teacher-compensation plan also appeared to boost achievement under certain conditions. The program requires teachers to set achievement goals with their principals. Those teachers who wrote the highest-quality objectives were associated with higher student achievement in elementary, middle, and high school than teachers who wrote lower-quality objectives.

However, a comparison of schools participating in the pilot program with those not participating found mixed effects from the program.[5]

Factors such as the size of the bonus, the number of teachers permitted to receive it, and the methodologies used to award the pay all seem likely to shape the effects of such programs.

It can be said:

In the United States, merit pay exclusively focused on rewarding teachers whose students produce gains has not been shown to improve student achievement, though some international studies show positive effects. Research has been mixed on comprehensive pay models that incorporate other elements, such as professional development. Scholars are still examining whether such programs might work over time by attracting more effective teachers.

Do students in unionized states do better than students in states without unions?

Many studies have attempted to address the impact of unionization or collective bargaining on student achievement. The question needs careful parsing.

It is true that students now tend to do better in heavily unionized states, like Massachusetts, rather than in those without required bargaining, like Alabama and Mississippi. But this simple correlation provides no information about whether unionization causes these achievement patterns. As the general public isn’t likely to be aware of the difference between correlation and causation, it behooves reporters to explain the difference when reporting on this topic.

Some research has been conducted on the causation question, and, as one 2008 paper summarizing the relevant literature found, results appear to be mixed. The studies tended to use different models and methodologies, choices that impacted their findings, the paper found. For instance, “point in time” studies tended to find positive impacts of unionization on academic achievement, while those looking at student growth over time tended to show negative impacts.[1] See Burroughs 2008 for a longer discussion and bibliography.

The most recent study purports to use a “natural experiment” to compare performance on SAT exams from the period between 1993 and 1999, during which New Mexico had mandatory bargaining, to the period between 1999 and 2003, when bargaining was permissible but no longer mandatory. It compared performance during those time periods to achievement patterns in other states, controlling for factors such as state racial composition, poverty rates, and crime rates.[2] In addition, the author attempted to account for the fact that the change in bargaining laws probably would not have had immediate effects.

The study found that mandatory collective bargaining was correlated with an increase in SAT scores, but a lowering of graduation rates.

Critics of the study noted that federal data show that before the shift in state law, not all local teachers had voted in favor of union representation, raising questions about what phenomena the results actually reflect.

It can be said:

Students tend to do well in some heavily unionized states, but it isn’t possible to conclude that it is the presence or absence of unions that cause that achievement.

About the Author: Stephen Sawchuk is an assistant editor at Education Week, an independent national news organization based in Bethesda, Md. He reports on teacher quality and the teaching profession.

EWA Research Brief: What Studies Say About Teacher Effectiveness

Bibliography

Burroughs, Nathan. 2008. “Arguments and Evidence: Collective Bargaining’s Role in Public Education.” Center for Evaluation & Education Policy, Vol. 6(8)

Clotfelter, Charles T. Helen F. Ladd, and Jacob L. Vigdor. 2007a. “Teacher Credentials and Student Achievement: Longitudinal Analysis with Student Fixed Effects.” Economics of Education Review (December)

—. 2007b. “How and Why Do Teacher Credentials Matter for Student Achievement?” CALDER Working Paper 2. Washington, D.C.: The Urban Institute

—.  2007c. “Teacher Credentials and Student Achievement in High Schools: A Cross-Subject Analysis with Student Fixed Effects.” CALDER Working Paper 11. Washington, D.C.: The Urban Institute

—, and Justin Wheeler. 2007. “High Poverty Schools and the Distribution of Teachers and Principals.’ CALDER Working Paper 1. Washington, D.C.: The Urban Institute

Darling Hammond, Linda, Deborah J. Holtzman, Su Jin Gatlin, and Julian Vasquez Heilig. 2005. “Does Teacher Preparation Matter?” Education Policy Analysis Archives, 13(42)

Glazerman, Steven, and Allison Seifullah. 2010. “An Evaluation of the Teacher Advancement Program (TAP) in Chicago: Year Two Impact Report.” Mathematica Policy Research, 6319-520

Goe, Laura. 2007. “The Link Between Teacher Quality and Student Outcomes: A Research Synthesis.” Washington, D.C.: National Comprehensive Center on Teacher Quality

Goldhaber, Daniel, Dominic J. Brewer, and Deborah J. Anderson. 1997. “A Three-Way Error Component Analysis of Educational Productivity.” Education Economics, Abingdon

Goldhaber, Daniel, and Dominic Brewer. 1999. “Teacher Licensing and Student Achievement.” In C. Finn and M. Kanstoroom, eds.:Better Teachers, Better Schools. Washington, D.C. Thomas B. Fordham Institute.

Goldhaber, Daniel, and Dominic Brewer. 1997. “Evaluating the Effect of Teacher Degree Level on Educational Performance.” In W. Fowler, ed., Developments in School Finance, 1996NCES 97–535. Washington, DC: U.S. Department of Education, National Center for Education Statistics

Hakel, Milton, Judith Anderson Koenig, and Stuart W. Elliot, eds. 2008. Assessing Accomplished Teaching: Advanced Level Certification Programs. Committee on Evaluation of Teacher Certification by the National Board for Professional Teaching Standards. Washington, D.C.: National Research Council

Hanushek, Eric A. 1992. “The Trade-off Between Child Quantity and Quality.” Journal of Political Economics, 100(1), 84-117

Hanushek, Eric A., and Steven G. Rivkin. 2010. “Generalizations About Using Value-Added Measures of Teacher Quality.” American Economic Review: Papers & Proceedings Vol. 100

Hanushek, Eric A., John F. Kain, and Steven G. Rivkin. 1998. “Teachers, Schools, and Academic Achievement.” New York: National Bureau of Economic Research Working Paper No. 6691

Harris, Douglas N. 2011. Value-Added Measures in Education: What Every Educator Needs to Know. Harvard Education Press: Cambridge, Mass.

Harris, Douglas N., and Tim R. Sass. 2007. “Teacher Training, Teacher Quality and Student Achievement.” CALDER Working Paper 3. Washington, DC: The Urban Institute

Harris, Douglas N., and Tim R. Sass. 2009. “The Effects of NBPTS-Certified Teachers on Student Achievement.” Journal of Policy Analysis and Management 28 (1), 55-80

Harris, Douglas N., and Tim R. Sass. 2011. “Teacher Training, Teacher Quality, and Student Achievement.” Journal of Public Economics 95 (2011) 798-812

Hill, Heather, Brian Rowan, and Deborah Loewenberg Ball. 2005. “Effects of Teachers’ Mathematical Knowledge for Teaching on Student Achievement.” American Educational Research Journal 42(2), 371-406

Hudson, Sally. 2010. “The Effect of Performance-Based Teacher Pay on Student Achievement.” Stanford Institute for Economic Policy Research, Discussion Paper 09-023

Jackson, C. Kirabo, and Elias Bruegmann. 2009. “Teaching Students and Teaching Each Other: The Importance of Peer Learning for Teachers.” American Economic Journal: Applied Economics 1(4): 85-108

Jackson, C. Kirabo. 2010. “Match Quality, Worker Productivity, and Worker Mobility: Direct Evidence From Teachers.” New York: National Bureau of Economic Research Working Paper No. 15990

Kane, Thomas, Jonah Rockoff, and Douglas O. Staiger. 2006. “What Does Certification Tell Us About Teacher Effectiveness? Evidence From New York City.” New York: National Bureau of Economic Research Working Paper No. 12155

Kane, Thomas J., and Douglas O. Staiger. 2008. “Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation.” New York: National Bureau of Economic Research Working Paper No. 14607

Koedel, Cory, and Julian R. Betts. 2007. “Re-Examining the Role of Teacher Quality in the Educational Production Function.” Unpublished working paper, University of Missouri

Koedel, Cory, and Julian R. Betts. 2009. “Does Student Sorting Invalidate Value-Added Models of Teacher Effectiveness? An Extended Analysis of the Rothstein Critique.” Unpublished working paper, University of Missouri

Hill, H.C., B. Rowan, and D. Ball. 2005. “Effects of Teachers’ Mathematical Knowledge for Teaching on Student Achievement.”American Educational Research Journal 42(2)

Lindy, Benjamin. 2011. “The Impact of Teacher Collective Bargaining Laws on Student Achievement: Evidence from a New Mexico Natural Experiment.” 120 Yale Law Journal 1130

Nye, Barbara, Spyros Konstantopoulos, and Larry Hedges 2004. “How Large Are Teacher Effects?” Educational Evaluation and Policy Analysis, 26(3)

Paypay, John P. 2011. “Different Tests, Different Answers: The Stability of Teacher Value-Added Estimates Across Outcome Measures.” American Educational Research Journal 48(1), 163-193

Podgursky, Michael J., and Matthew G. Springer. Teacher Performance Pay: A Review. 2006. National Center on Performance Incentives, Working Paper 2006-01

Rice, Jennifer King. 2010. “The Impact of Teacher Experience: Examining the Evidence and Policy Implications.”  CALDER. Washington, D.C.: The Urban Institute

Rivkin, Steven G., Eric A. Hanushek, and John F. Kain. 2005. “Teachers, Schools, and Academic Achievement.” Econometrica 73(2), 417-458

Rockoff, Jonah. 2004. “The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data.” American Economic Review 94(2), 247-252.

Rockoff, Jonah E., Brian A. Jacob, Thomas J. Kane, and Douglas O. Staiger. 2008. “Can You Recognize an Effective Teacher When You Recruit One?” New York: National Bureau of Economic Research Working Paper No. 14485

Rothstein, Jesse. 2009. “Student Sorting and Bias in Value-Added Estimation: Selection on Observables and Unobservables.”Education Finance and Policy 4(4), 537-571

Rothstein, Jesse. 2010. “Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement.” Quarterly Journal of Economics, 125(1): 175–214

Rothstein, Richard. 2010. “How to Fix Our Schools.” Washington D.C.: Economic Policy Institute Issue Brief No. 286.

Sanders, William L. and June C. Rivers. 1996. “Cumulative and Residual Effects of Teachers on Future Academic Achievement.” Knoxville, Tenn.: University of Tennessee Value-Added Research and Assessment Center

Slotnik, William, and Maribeth Smith. 2004. Catalyst for Change: Pay for Performance in Denver. Community Training and Assistance Center

Retrieved from: http://www.ewa.org/site/PageServer?pagename=research_teacher_effectiveness2

Thanks for your comments!

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 114 other followers

%d bloggers like this: