Knowing Tests and Using Data: Stories Behind the Numbers on the Page

  By Scott Hartl

The Harbor School, founded four years ago as an Expeditionary Learning pilot school, currently serves a diverse group of 264 urban middle school students in Dorchester, Massachusetts. A commitment to quality work drives every effort within the school building. Students stay with the same team of teachers from seventh to eighth grades. As part of the graduation criteria, students present their cumulative portfolios before a panel of parents and community members.

In Massachusetts, third through tenth graders take the Massachusetts Comprehensive Assessment System (MCAS), which is a high-stakes, criterion-referenced assessment in English/Language Arts; Mathematics; Science and Technology; and Social Studies and History. Students must score at a benchmark level in tenth grade in order to graduate. The Harbor School eighth graders spend 20 days on standardized tests, including 10 days on the MCAS. In an interview with Harbor School Director Scott Hartl, Web Editor Amy Mednick asked about the effects of testing on schools committed to the Expeditionary Learning approach.

The standards and accountability movement has actually brought us both good and harm. At both a state and national level, standards and accountability measures are intended to address the vicious achievement gaps between urban and suburban schools and between students of various racial and class backgrounds. The intent of the standards movement is also to generate a national commitment to improvement in all of our schools and to link increased resources to smart accountability. As a school leader, I agree with this intent.

The main problem for us as a city school is that the accountability movement is expressed in one overly powerful test that exerts far too much control over our school. It has produced a top-down and cookie-cutter mentality that takes away autonomy at the school level. Another part of the problem is that it has become a highly political process with urban students in Massachusetts fated to failure. Based on the first two years of MCAS performance in Boston, approximately 70 percent of tenth-grade students will not meet the benchmark score required to graduate.

Unfortunately, this emphasis on testing leads to pressures on schools, especially urban schools, that may result in exactly the opposite of what Expeditionary Learning and even the standards movement are about. When districts require detailed and prescribed content coverage, it competes with in-depth curriculum and effective pedagogy. The sense that there are many roads to high standards, and that schools can develop a unique approach toward reaching high standards that reflect their character and mission, is not what is being supported right now. Driven by test preparation, we get into the classic teaching-to-the-test mode, which moves us away from what we believe students should be learning and understanding.

Those pressures place in jeopardy key elements of our commitment to Expeditionary Learning: a focus on in-depth coverage, a project-based approach to learning, portfolios and exhibitions as key elements of the assessment system, and strong attention to character within the curriculum. I believe that Expeditionary Learning schools can flourish in districts with a heavy presence of state and district accountability. It's possible, but it's not easy. It is worth the effort, however. In many Massachusetts urban schools the test pressure has led to an abandonment of interesting or engaging curriculum and a focus on boring and bad teaching and learning in the name of back-to-basics.

How have you found ways to work within the system to continue to move forward and create positive change in the school community?

The Harbor School was born out of a strong belief that, for the urban students and families we serve, Expeditionary Learning presents an organization for a school and an approach to teaching and learning that: allows us to demand high-quality work; exposes our students to life-shaping experiences; builds a commitment to learning; and equips them with the academic background, habits of work and character, and vision for their own future that will put them on the road to school success.

If that is what we believe, then we need to do that work within the context of mandated state standards and tests. When we are planning learning expeditions, the MCAS has to guide our decisions on what to focus on in our curriculum and what to leave out. It is impossible, particularly in social studies and science areas, to cover all that is required. As we make choices, we need to do so guided by a deep understanding of that test.

If you don't know your test, all you can do is a one-shot test preparation approach. On the MCAS reading test, for example, students must demonstrate they can read beyond simple comprehension. Even if each question related to comprehension on the test were answered correctly, students still would not receive a passing score. They must be able to work with metaphor and analogy, and analyze and interpret text. Our teachers know this and they adapt their classes accordingly. In addition, more than 35 questions on the math test are about interpreting data sets. In response to that, starting next year, one-third of the science curriculum at The Harbor School will cover the use of data. We've re-tooled an expedition on whales to look at population dynamics and migration so that we can focus on interpreting data. Our knowledge of the test allows us to make choices about our curriculum.

At the Harbor School, every learning expedition needs to address a core topic or skill outlined on the MCAS and the state standards. Our sixth graders, for instance, spend three months studying the Boston Harbor Islands. They are not on the MCAS test, but the expedition introduces the Harbor School's research and writing process and those skills are central to the MCAS test in Language Arts and Social Studies.

We prepare our students for the tests throughout the year to avoid an in-your-face, last-minute, drill-and-kill test prep. For example, on every test and quiz we give, some of the questions are formatted like MCAS questions. The writing test is heavily rehearsed four times a year with a writing prompt. We use a slow and consistent trickle of exposing them to the format and style they will face on these exams.

Harbor School eighth-grade students ranked second out of 27 open-enrollment middle schools in Boston on their English/Language Arts, Math, and Science MCAS tests and ranked fourth in Social Studies/History in 2000. What do you think contributed to this success?

We spend large amounts of time looking at data on how our students are doing on these tests: What kinds of questions did they do well on? Are there patterns where many students failed to perform? We also confront the harder questions of who is doing well in our school and who isn't. Now we know what students will be facing on the tests and their patterns of performance. But the school community still decides what we cover in our curriculum. Even if something is heavily represented in the MCAS test and our students aren't doing well in that, we will not automatically change our curriculum. It's no surprise that we as a school performed strongest on the writing portions of the English/Language Arts test. That's the portion of the MCAS test that most closely matches our style of teaching and learning. I'm willing to live with lower performance on the Science and History sections of the MCAS because the skills those sections demand place us in conflict with our core beliefs of covering some things in depth rather than many things at a shallow level.

Specifically, how do you look at the data and use them to understand students' needs?

We have gotten important help in looking at the data from an outside consultant. It's helpful to have an outside voice to tell the school some of the hard truths we have to face. We have done a great deal of work to create a computer structure to enter the data for our students, and we have created a consistent format for all the state, national, and school-level tests. When we looked at the Stanford 9 test results over time, we saw that the students' reading scores had actually declined in the first two years. Our consultant asked the faculty, "If the majority of students in your classroom are slightly below grade level, are you pitching your instruction there?" A picture surfaced from teachers: "I've been so overwhelmed with getting everyone up to a certain place, I haven't been thinking about my strongest readers." Emerging out of data on a page are some instructional stories that are giving rise to interventions. Once the data is well organized and cogently presented, that allows us to use them to discuss trajectories of particular students and to identify strategies that might be helpful to teachers.

One student, Lakisha B. [name changed], performed tremendously at our exhibitions and consistently had some of the strongest project work in our school. Visitors were particularly drawn to her work and its quality, and yet, on important measures of reading and writing, her scores were low and declining. Through conversation, it emerged that Lakisha needs a lot of adult help. Her perseverance and her ability to be coached is very high, so her performance on effort-sensitive assessments like project work that goes through multiple drafts is very high. But her ability to respond in test format without adult support exposed her skill deficits. Those conversations about Lakisha's scores led to an awareness that as a faculty we need to attend to basic skills development for all the students; to delineate between effort-sensitive and non-effort sensitive assessment; and to maintain a balance between teacher-supported and on-demand work. The data is only as useful as the conversations that it starts and the interventions that follow.

In many of our schools we know the students really well. We have deep intuition about what works well and what doesn't. I believe that when you work from your intuition there is about an equal chance of doing the right thing with the students and being blinded by your assumptions. In schools that know students really well, data becomes particularly important because it counter-balances the intuitive sense we have about our work.

This article combines excerpts from a presentation Hartl gave at the Expeditionary Learning National Conference held in Denver in April with an interview between Web Editor Amy Mednick and Hartl.

THINGS TESTS DON'T TEST

• creativity
• critical thinking
• motivation
• ambition
• persistence/perseverance
• humor
• attitude
• reliability
• politeness
• enthusiasm
• civic attitudes

Learn to Show What You Know

By Becky Schou

In Maryland, third, fifth, and eighth grades take the performance-based, criterion-referenced Maryland School Performance Assessment Program (MSPAP). Second, fourth and sixth graders take the multiple choice, norm-referenced, Comprehensive Test of Basic Skills (CTBS). Some school districts administer CTBS at the beginning and end of each year at every grade level.

As teacher teams in Maryland finalize learning expedition plans, they are analyzing how often and to what degree students are engaged in instruction and assessment that mirrors the state assessment. Teachers might imbed these projects and testing formats into the learning expedition framework, for example, by adding a school district performance task to a bird migration project or by weaving writing prompts and rubrics into the creation of a field guide on species of the Chesapeake Bay. Their expedition fieldwork on creating the Chesapeake Bay model would include the required skills of data analysis, and summarization. Each of these experiences is not only good instruction, but will help prepare students to take the MSPAP.

This is an example of how Expeditionary Learning schools in Maryland are beginning to use Expeditionary Learning practices to help prepare students to take the state assessments. Imbedding projects and assessments similar to the MSPAP into learning expeditions first began in schools such as Annapolis, Chase, Glen Avenue elementaries, and Middle River Middle, which have been implementing Expeditionary Learning for three or more years. Some schools even began to use reflection and critique protocols and culture-building practices to reduce test anxiety, increase student reflection and perseverance, and develop a spirit of camaraderie and best effort. Each of these schools has experienced growth in both MSPAP and CTBS scores. Annapolis, Glen Avenue, and Middle River have been cited by The Maryland State Department of Education for their improved test scores, and Glen Avenue has been named a National Distinguished Title I School.

When six elementary and three middle schools in Prince George's County joined our region during the 1999-2000 school year, Angela Jolliffe, then the Expeditionary Learning field director for Maryland¹, began conducting bimonthly principals' meetings to refine these practices. At these meetings, Jolliffe, the principals, school designers, and district personnel shared their work, traded new ideas, and resolved common concerns. Additionally, school designers worked with some schools in preparing for and proctoring the May, 2000 MSPAP. This support enabled school designers to become familiar with the test, observe testing conditions, as well as student attitudes and performance.

ANALYZING THE SCORES

The 2000 MSPAP scores were released in November, and each school leadership team and their school designer began analyzing the results and developing an action plan for the following year. This year the schools could count on the additional support from Expeditionary Learning as well as the school district. Soon after the scores were released, I met with representatives from the district's Office of School Improvement to review each individual school's scores as well as the performance of all nine schools.

As we analyzed the school's scores, we saw some clear evidence of Expeditionary Learning's positive effect on test scores. Greenbelt Elementary School experienced a rise in science scores, most likely as a result of their schoolwide science learning expedition. Oakcrest Elementary School saw writing scores increase after a yearlong focus on imbedding writing into expeditions. Each school's data had a unique story to tell and yet we also found that all nine schools had some common needs. We agreed that we could provide each of our schools with specific tools to address these needs by making good use of Expeditionary Learning's practices. These include the performance-based framework of a learning expedition, increasing the use of task-specific rubrics and revision practices, and building positive culture during the testing setting. We developed a draft action plan and timeline of support. The plan focused on listening to and learning from other successful schools, imbedding MSPAP-like projects, tasks, and scoring tools into existing learning expeditions and refining/developing each school's test preparation initiatives to take full advantage of Expeditionary Learning's design principles and core practices.

SHARING THE PLAN

The draft action plan was shared with the principals at our bimonthly meeting in December. Then in January, school designers met with key faculty members in each school to gather input. Additionally, Expeditionary Learning provided the school designers with professional development in the specific components of MSPAP, rubric development, and infusion of MSPAP-like projects and tasks into learning expeditions. The school designers and I continue to meet regularly to review our work with the schools, to examine promising practices throughout the state, and to use this information to improve our own practices.

ENACTING THE PLAN

At February's meeting, principals and school designers met in school teams and updated each other on progress and new ideas. Additionally, I shared a test preparation option organized into a mini-expedition. In this "Learn to Show What You Know" expedition, students engage in research into the format and demands of MSPAP. They conduct fieldwork by interviewing students and teachers who had taken and given MSPAP last year. They engage in acts of service by developing a full value contract in each testing group that includes making a commitment to support and encourage one fellow student during testing. They then bring all of this together in a culminating project--the creation of "A Students' Guide to Success on MSPAP" brochure. Throughout the winter and spring, Expeditionary Learning school designers have assumed an additional responsibility in their schools by devoting some of their time to support the school's test preparation plans.

An additional part of this district-supported effort is the expansion of our staff development options to include opportunities for teachers throughout the district to learn from each other. On a staff development day in late March, Expeditionary Learning teachers in Prince George's County had the opportunity to attend sessions led by area school designers and teachers. Titles of these sessions included: "Using Rubrics and Peer Critique," "Defining A Culture of Quality and Success," "Sailing To Success on MSPAP," and "Imbedding Performance Tasks into Learning Expeditions." Another day like this is being planned in June to support and augment summer institutes.

In Prince George's County, we are working collaboratively to share in the joy and responsibility of teaching the children. It is our belief that this responsibility includes assisting them in learning how to demonstrate their knowledge on MSPAP and CTBS. Through the familiar format of a learning expedition--reflection and critique practices; establishing a culture of support within the testing environment; and using the design principles to prepare and sustain them during the testing experience--we hope to support all students in successfully meeting the challenges of any test they need to take.

Becky Schou is the field director of the MidAtlantic Regional Office of Expeditionary Learning.

¹Angela Jolliffe is currently the field director for the Southeast Regional Office of Expeditionary Learning.

MARYLAND ACTION PLAN (EXCERPT)

Ongoing Support Structures

I. Share with parents and students, at the beginning of the year, which mandated assessments the students will be required to take and what your school is going to be doing to help ensure their success.

II. Provide parents with guidelines and suggestions for assisting their child in being successful. About a month before the test, send a letter home reminding parents about the assessment and reaffirm the importance of their role in supporting their child.

III. Imbed testing format experiences into learning expedition instruction and assessment, so that students gradually become comfortable with the way in which they will need to express their knowledge on the test. These experiences with the test format need not be extensive or "take over" the expedition, but by providing children with the opportunity to see that they express their knowledge in a wide variety of ways, they gain confidence.

IV. Analyze quarterly benchmark test data to review and refine instructional practices.

Standards and Garbage: Deconstructing an Elementary Research Project

By Joanna Leeds and Linnea Krizsan

Wouldn't it be nice, considering the recent focus on standards in Expeditionary Learning, if we could claim that, in planning our expedition, we put standards at the forefront of our minds, that we continually referenced standards, and that we knew them forwards and backwards? We can make no such claim. Instead of standards, we admit to focusing on designing an integrated, engaging, project-based expedition. It seems that every staff meeting, workshop, and professional article pressures us to create curriculum around standards and we do believe that standards must be covered and mastered. We looked first, however, to the goals of our expedition. We focused on creating an expedition that would motivate children to produce quality work. We believed that by going through the process of learning the skills necessary to create criteria-based projects, students would meet and exceed district standards in several content areas.

You might be thinking now, "Oh, these teachers don't have any accountability toward standards because they teach in a charter school." However, our charter with Denver Public Schools specifically states that we are to meet and exceed district standards. We are required to administer state-mandated tests (Colorado State Assessment Program) and we have adopted the Six Trait Writing Framework. We have felt a strong push to address all district standards this year.

We chose "Garbage" as our expedition, hoping it would pique the interest of our two crews of curious eight and nine year olds. Our guiding questions were: What is garbage? What can we learn from studying garbage? What happens to garbage and how does it impact the earth? We wanted to help students look critically at environmental issues and take responsibility for finding solutions by asking the questions: What is the problem? What causes the problem? What impact does our behavior have on the problem? What can we do? Garbage is something we see, handle, and produce daily. But do we know where the trash goes after the garbage collector takes it away? We hoped that by taking an in-depth look at garbage (what nine year old wouldn't love that?) students would develop an awareness of waste management and waste reduction issues. They would learn that managing garbage is more than just taking out the trash, develop possible solutions to the problem, and begin to make informed personal and social choices about the environment. In addition, they would have an engaging topic through which to acquire skills such as gathering information, reading non-fiction, writing outlines, forming paragraphs, editing and revising, writing organized research papers, and creating quality projects.

MODELING AND DOCUMENTING

We began by doing a class research project. By modeling each step and going through it together, students became familiar with the individual phases of research, as well as the process as a whole. The class documented and displayed each step on our "Research Board" --- a documentation panel in our classroom. Then, as students conducted their individual research, they could reference our display if they needed clarification on which step came next or what a certain step should look like. The fact that we had all created a group project, chosen the criteria for the final product, and constructed a rubric together, motivated students to do their best work, while providing them with clear expectations and guidance.

Each time we worked on these projects, you could see children's heads turn toward the Research Board. It became almost a reflex to reference the board if they had a question. Alyson was absent during a week of research work, and she had missed the review on how to do an outline. She relied on the Research Board --- where we had written an outline as a class --- as a reminder of how to do it and what it should look like. Students knew that they could meet the criteria that they themselves had practiced and then set forth in the rubric.

CHOOSING A TOPIC

After students understood the critical steps in the research process, they were ready to brainstorm their own topics. Students chose diverse areas to study. Their inspiration for these projects came from two months of fieldwork and immersion. Students wanted to study landfills, plastics, methane, Earth Ships, Straw Bale houses, solar energy, and artists who only use recycled materials, to name just a few. They were eager to jump into more in-depth research of topics that had piqued their interest during our immersion phase. They completed each step individually, but were supported by the rest of their crew, who were each completing the same step. We reviewed the skills needed for each step, and found that some skills required more practice than others did. For example, they could "brainstorm resources" after the first lesson, but needed to read several nonfiction articles to practice "determining importance" before they felt comfortable working on their own. They also used what they remembered from our class research, and used each other as resources. Alyson was studying The Art Guys, a pair of artists who only use recycled materials in their art. She turned to Jeffrey for help. In the fall, when we all wrote letters to different companies, organizations, and individuals for information on garbage, recycling, environmental policies, and more, Jeffrey had written to The Art Guys and received a book, photographs, and a letter. Alyson used Jeffrey and his information as a resource in her research. Final student projects reflected the new skills they had acquired and demonstrated their motivation to produce accomplished, quality work that met the pre-established requirements.

TOOLS TO IMPROVE

We use peer critiques and the Collaborative Assessment Conference Protocol with everything that we do in our classrooms. These tools help students accomplish quality work in every part of our curriculum. After the first and second drafts of their writing for the research project, as well as during the final days of putting together display boards, groups of students looked at each other's work through different lenses. One group might focus on conclusions, another might struggle with organization of ideas, and still another might look at how the board draws in the viewer and whether the visuals are understandable. We have found that these critique tools highly motivate students to do their best work and to meet the standards that permeate the project. When Taylor's peers read her draft and still really did not know where solar energy came from, she was motivated to add a few paragraphs that explain where solar energy comes from. Taylor wanted her peers to understand her work. When Alyson critiqued Max's board, she noticed how his title really stood out. She realized that her title was more difficult to read. Examining others' work not only helped her offer advice to her peers, but gave her new ideas as well.

EXHIBITION

At the end of every expedition, we have an Exhibition Night when students present their final projects to the community. The format of this year's Exhibition Night was in seminar style. Small groups of students who researched similar topics (for example, Taylor's research on Solar Energy was paired with Max's research on alternative energy and Willa's research on electric cars) presented their work to groups of students, teachers and families and still wanted to learn more. Students gave short presentations of their work and the audience asked questions. This process was extremely exciting and motivating for students. They wanted their display boards to be perfect and their information accurate. Before the "real thing," they were critiqued by their peers and did practice presentations in front of other crews.

Students and their families were motivated by what they learned and still wanted to learn more. Kelly's mom still gives me a weekly report on how many bags of garbage she put out this week in contrast to how many bags her neighbor puts out ("They had three whole bags - for one week!"). Kids who bring "Lunchables" are chastised for buying a product that is so clearly not environmentally friendly. The other day, a guest in our classroom threw away an empty bottle of juice. Kendall fished it out of the garbage, reminding her that at Odyssey, "We recycle these bottles!" Students also feel comfortable with the research skills they know. We are currently working on another research project and although the content is different, the skills are still relevant and useful. Kendall told us he feels more comfortable reading nonfiction. The other day, we saw Baylee and Ellen get out an article they had printed off the Worldwide Web and begin to underline and code the text without needing to be reminded.

FINDING THE STANDARDS

After the expedition, we examined our research projects to outline what standards had been covered for a whole school "curriculum map." We discovered (should we say, we were not surprised?) that the project had covered a wide array of standards in areas of reading, writing, math, and visual arts. We originally thought our expedition would cover more science, such as decomposition, bacteria, and the life cycle. However, as always, expeditions never turn out the way they are originally planned. We used this information to help plan our spring expeditions with a more scientific focus.

Of course, standards must be covered and mastered. But standards do not always have to be the driving force behind curriculum planning. A curriculum that is integrated, engaging, and stresses quality will naturally provide students with work that requires standards to be addressed. Projects should have clear expectations and requirements that are assessed though rubrics. Students should see and understand the rubric before the project is done, so that they may reference it in order to assess on their own what they have accomplished and what they still need to work on. If projects are designed with specific skills, as well as content, in mind, they will undoubtedly meet standards in a natural, authentic, and engaging way.

Joanna Leeds and Linnea Krizsan teach at the Odyssey School in Denver, Colorado.

The Fieldwork Archive