Hi Lisa,

Thank you for your thoughtful questioning. I am going to purposefully omit responding to some parts to allow my co-presenters a chance to chime in, especially in areas that are more their expertise, such as the measurement aspects. Please allow them to respond!

In terms of challenges we've encountered so far... where do I begin?! When we wrote the grant we knew it would not be easy, but we certainly didn't anticipate the level of constant challenge we are facing. Creating an accurate algorithm alone that uses natural language processing to read, interpret, and construct human language that is usable has been the most challenging piece. We are certainly employing aspects of the design-thinking process and iteratively going back to the drawing board. There is a lot of modeling, testing, and generalizing occurring! One of the most poignant challenges that I care most about as an educator is the issue around educator identity and ensuring that the computer's generated feedback is appropriate and threaded through all cultural lenses. How do we make the computer provide feedback that is not limiting or biased in identity? For example, we have noticed that many of the teachers' feedback messages are rooted in their cultural or personal preferences. The computer won’t know culture or preference, so how do we make it sound like ANYONE without ignoring race and cultural constructs? We are currently addressing this issue by actively recruiting a diverse range of participants to ensure we capture all types of feedback, given that teachers in different parts of the country, race, religion, etc., all have different lived experiences and varying expectations of what feedback should look like and sound like.

As a professor, I try to personally respond to what my students write. I asked them a lot of open ended questions but I only have time to review a tiny fraction. I have written little emails to students and they have responded with "no other professors has ever been so thoughtful to reach out to me! Thanks!"   I feel bad as it don't spend as much time as I should especially considering the positive reaction to the fact that I am paying attention to them.  So I want to just make it easier for a teacher to do such things.    

Our big goal is to save teachers time. We want more kids to get teacher feedback more often.  In year three we will run a study to see how much time we have saved teachers (if any).  Clearly if we suggest messages to send, that teachers don't want to send, we have wasted their time in reviewing them.   

We also want to understand the engage factor: if teachers get a fixed amount of time, and we asks them to send some messages to their students how many more students can they "touch".  And what is the impact of student engagement if their teacher has recents responded to their written work?

Thank you Hilary and Neil for your responses!

Hi Abby!

Thank you for your feedback! ASSISTments has been around for a while! In fact, when I was a fifth grade teacher back in 2010-2014, I used ASSISTments to help prepare my students for standardized testing. It was a great way to instantly see my students' work and know right away how they performed on what was then more multiple-choice oriented questions. Now, I can imagine the struggle teachers face everywhere with the massive focus on open-response like questions.

In fact, in April 2019, the What Works Clearingouse (WWC) published a federal research review of the ASSISTments Efficacy Trial and found ASSISTments to be one of the most effective interventions in the nation! ASSISTments received the highest standard of WWC, which is what is called "positive effects without reservation.” There are only five interventions in the United States that currently meet that standard.

ASSISTments offers other subject areas, too. See here https://app.assistments.org/FA/f/e?sd=certified&p=1. However, as a free, non-profit entity, capacity has required the team to focus and hone in on one subject area and do it really well. Then, to expand beyond that. Additionally, my focus is on mathematics education, so we decided to stick with that subject for this particular grant. This can and should be applied to other subjects!

Thank you for sharing! It is so great that you are adding to a tool that already exists and is proven!

Thanks Bruce,

ASSISTments has a builder with it so any teacher can create their own problems. I use it in my AI class at WPI. In fact this is where I started to think more about the open-response questions. ASSISTments is really good at scoring math problems with answers like "2x+3" or  "4/5 (which is also .8)" I would randomly review the open response responses in class, this was easily facilitated by ASSISTments. One day I sat down and wrote emails that referred directly to my students responses. The feedback was amazing, students said they had never been given such personal feedback.  That is when my wife Cristina and I began designing this project.  

We are creating the algorithms that should eventually work for any grade.  ASSISTments has content for 4th and 5th grade already so any teacher can start using it and start to get feedback on the computer scorable problems.  This new added feature of DRIVER-SEAT is being developed in MS but should extend to other subjects and grades. 

Hi Quinn,

Thank you for your feedback, especially offering the English teacher perspective. What I meant in that other thread is that we have to be incredibly cognizant of how performance feedback varies across cultures. While some cultures are more direct, others are not. For example, if a teacher's feedback to a student is "You need to be more careful when adding fractions and be sure to use common denominators" this might come across harshly and not be well received or useful to a student whose culture looks at directness in a negative light or personally does not work well with directness. If a teacher's feedback is less direct, for example, "I wonder how you could use common denominators to help you with this task?," some students may misinterpret and not realize they are to go back and try using the suggestion since it wasn't directly stated. In another fashion, while some teachers prefer complete sentences and academic language, others do not share that same value when assessing mathematics. Most importantly, we need to continue to develop a participant pool that is representative of the various communities and cultural groups present in the United States so we can further study how culture, location, etc., plays a role in feedback. I hope I answered your question!

Hi Dave,  

We are glad you liked it and yes the name felt like an accomplishment all on its own!

One note is that we are working to support teachers as they teach . We are concerned about tools that let students "blaze ahead on a quicker timeline", a few reasons include that without a classroom or human to human experience there is a concern that learning can become shallow. This project is intended to support teachers by saving time allowing for them to plan and orchestrate better human to human teaching. 

Dialogue is what the D in DRIVER-SEAT stands for so yes we are moving in the direction of a back and forth between teachers and students. It is tricky since we also want to save teachers time. Your idea to employ education services interesting. 

We definitely think that by allowing teachers to have more time to plan they will be able to enhance their lessons with connections.  

I just watched your video and we should connect and see if ASSISTments would be an interesting tool for some of your consortium teachers to explore. 

Cristina

Training is an interesting issue.  With this project we are only working with teachers who use Engage NY or Open Up resources so for them there was very little training. ASSISTments is a free service developed at our university, Worcester Polytechnic Institute so we are able to put all the practice problems and exit cards from Engage NY, Open Up by Illustrative Math and Utah math into our system. If a teacher is already using that curriculum and has Google Classroom they can use ASSISTments without any training.  

Training is needed if the teacher has to create their own content or needs help finding what they need.  It may also be needed if a teacher does not see the value of using homework as a formative assessment strategy. That awareness is not always present in teachers so a motivational/informational session may be needed. 

You can learn more at www.assistments.org and keep asking questions on this forum.  

Your point about what goes into the feedback messages is important.  We are interested in finding out what type of feedback works best for whom and when. Part of our research flow involves doing randomized controlled trials to see what type of feedback works best. 

Hello Hilary's Dad!  Thanks for checking out our work.  Now I am motivated to get my dad to comment.  Lets see if I can!