Hi Sara, thanks so much for your interest and question! Here's a response from our TEEMS project member Jeremy Pina:

For assessing student engagement and STEM identity, we're making use of the "Student Attitudes Toward STEM Survey" (S-STEM), the citation for which is: Friday Institute for Educational Innovation (2012). Student Attitudes toward STEM Survey-Upper Elementary School Students, Raleigh, NC: Author. The S-STEM looks at students self-reported feelings of self-efficacy and confidence in the areas of science, math, and technology/engineering. It also focuses on future career orientation and engagement with 21st century learning skills.    For assessing the impact of IE-based instruction on science and engineering student learning outcomes (which I think summarizes what you asked about impact on content and practice), we're developing our own TEEMS assessment instruments, inclusive of a content assessment aligned with NGSS standards, as well as an assessment of far-transfer, critical thinking. In service to these efforts, we're making use of adapted form of Schwartz, Bransford, and Sears's (2005) preparation for future learning framework.     Development of these assessments is ongoing, and we're hoping to help out the IE research community at large by leading the vanguard in utilizing these forms of assessment as means of generating new perspectives about the impact of IE on students engagement and learning. Stay tuned!

 

Hi Christine!

The research we're doing will be looking primarily at the influence of IE instruction on students' STEM identity and on student learning outcomes. In terms of STEM identity, we're interested in looking at self-efficacy and engagement in science, math, and tech/engineering. As for learning outcomes, we're interested in the extent to which IE engineering education might help promote knowledge transfer across contexts and, additionally, the influence that IE might have on facilitating engagement with critical, creative, and higher order thinking. (This is not to say that we're not looking at other variables, such as teachers' fidelity of implementation, as well as potential demographic effects on the outcome data, just that these are our focus areas!)

In our current research project, we're lucky enough to have an opportunity to work with several schools in a county here in western MA. They're helping us by providing population data at the county level from administration, at the teacher level via focus groups and participant surveys, and at the student level by means of learning assessments presented a few times a year, each paired with a longer engineering unit. 

I hope this was clear enough! If you have further follow-up questions, please feel free to post 'em. 

:)

Thanks for your interest! Certainly the use of narrative for teaching engineering creates a very natural connector to other school subjects. In the NSF grant that preceded this one, Through My Window, we encouraged teachers to incorporate the two books in language arts programs, and we were very pleased to see that happen organically, even without formal direction by our team. Teachers were creative in using the stories and characters from the books to introduce new vocabulary, inspire art projects, etc. An unexpected result was that incorporation of the books into language arts topics had particular success in some special education settings. In our current research, our focus is developing engineering units that integrate with science topics. While we have no formal plans for collaborating with English language arts teachers, we hope to provide some suggestions in our curriculum guides and PD. Implementation of our curriculum begins this fall, and we will be observing and soliciting feedback from teachers on how they make connections and share ideas.

Hi Catherine!

Thank you so much for your questions! I'm really excited to see that so many people are interested knowing more about our transfer assessment.

To tell you the truth, I'm not entirely certain what level of specificity you'd prefer! I'm hoping to expand my earlier responses to provide more insight, but I'm also hoping not to get overly technical or theoretical. All of which is to say, I apologize in advance if what follows is convoluted.

So, on the question of "What knowledge do we hope will transfer?":

The transfer paradigm we're working with, the preparation for future learning (PFL) model, is adapted from Bransford & Schwartz's (1999) Rethinking transfer and Schwartz, Bransford, and Sears (2005) Efficiency and innovation in transfer.

In this framework, knowledge transfer is parsed into "transfer out" learning, encompassing the "declarative" and "applicative knowledge" associated with a particular learning experience, and "transfer in" learning, which entails the "interpretive knowledge" associated with the learning experience. It is this latter, the "transfer in" learning, that we hope to focus on most directly by means of our transfer assessment. (Though, again, to be clear, we’re assessing more than just transfer in this project!)

“Transfer in” knowledge comprises a set of skills and dispositions internalized by students that help them to “know with” a particular concept or model, utilizing it as a basis for approaching novel problem-solving scenarios. In engineering, for example, “transfer in” might entail “knowing with” the full conceptual model of an engineering design process, allowing a student to successfully orient themselves to a previously unseen, and possibly unusual, task.

And to your second question of “How do you plan on testing the transfer?”:

This is, of course, the challenge! While the PFL model is itself, at some level, inclusive of assessment framework, there are considerations we’re making to more clearly describe students’ cognitions – most especially their engagement with higher order thinking skills – as they “transfer in” knowledge to new contexts.

Are they, for instance, improving analytical thinking as they learn to better “know with” the engineering design process? Are they employing problem-finding? Are they making use of divergent thinking? Are they developing their critical thinking skills? There are, at present, allusions to all of these in the PFL literature, but few clear appeals to evidence.

We’re fortunate enough to be going into a pilot year in our current study, and this latter aspect of the work is in active discussion with members of the IE community. So, in general, we’re hoping to get more insight into the question of assessment going forward.