1. Hilary Kreisberg
  2. http://www.lesley.edu/center/math-achievement
  3. Director, Center for Mathematics Achievement
  4. Putting Teachers in the Driver’s Seat: Using Machine Learning to Personalize Interactions with Students (DRIVER-SEAT)
  5. https://sites.google.com/view/driversseatlesley/home
  6. Lesley University
  1. Anthony Botelho
  2. Research Scientist
  3. Putting Teachers in the Driver’s Seat: Using Machine Learning to Personalize Interactions with Students (DRIVER-SEAT)
  4. https://sites.google.com/view/driversseatlesley/home
  5. Worcester Polytechnic Institute
  1. Cristina Heffernan
  2. Math Education Specialist, ASSISTments
  3. Putting Teachers in the Driver’s Seat: Using Machine Learning to Personalize Interactions with Students (DRIVER-SEAT)
  4. https://sites.google.com/view/driversseatlesley/home
  5. Worcester Polytechnic Institute
  1. Neil Heffernan
  2. http://www.neilheffernan.net/
  3. Professor
  4. Putting Teachers in the Driver’s Seat: Using Machine Learning to Personalize Interactions with Students (DRIVER-SEAT)
  5. https://sites.google.com/view/driversseatlesley/home
  6. Worcester Polytechnic Institute
Facilitators’
Choice
Public Discussion
  • Icon for: Hilary Kreisberg

    Hilary Kreisberg

    Lead Presenter
    May 12, 2019 | 04:46 p.m.

    Thank you for taking time to watch our video! Please note that we are in Year 1 of our project, so our data is limited at this time. Regardless, we hope to engage with you in meaningful conversations that help our project move forward.

    We are most excited to discuss with you the:

    (1) platform ASSISTments;

    (2) demand for open-response and reasoning based mathematics questions; and

    (3) possibilities and challenges of using machine learning to improve access to effective feedback in mathematics.

  • Icon for: Lisa Miller

    Lisa Miller

    Facilitator
    May 13, 2019 | 08:41 p.m.

    Thank you for sharing your video!  In your initial post, you wrote that you are excited to discuss the "possibilities and challenges of using machine learning to improve access to effective feedback in mathematics".  What are some of the challenges you've encountered and how have you addressed them?  Also, I realize this is only year 1 of the project, however, how do you plan to measure impact?

  • Icon for: Hilary Kreisberg

    Hilary Kreisberg

    Lead Presenter
    May 13, 2019 | 11:06 p.m.

    Hi Lisa,

    Thank you for your thoughtful questioning. I am going to purposefully omit responding to some parts to allow my co-presenters a chance to chime in, especially in areas that are more their expertise, such as the measurement aspects. Please allow them to respond!

    In terms of challenges we've encountered so far... where do I begin?! When we wrote the grant we knew it would not be easy, but we certainly didn't anticipate the level of constant challenge we are facing. Creating an accurate algorithm alone that uses natural language processing to read, interpret, and construct human language that is usable has been the most challenging piece. We are certainly employing aspects of the design-thinking process and iteratively going back to the drawing board. There is a lot of modeling, testing, and generalizing occurring! One of the most poignant challenges that I care most about as an educator is the issue around educator identity and ensuring that the computer's generated feedback is appropriate and threaded through all cultural lenses. How do we make the computer provide feedback that is not limiting or biased in identity? For example, we have noticed that many of the teachers' feedback messages are rooted in their cultural or personal preferences. The computer won’t know culture or preference, so how do we make it sound like ANYONE without ignoring race and cultural constructs? We are currently addressing this issue by actively recruiting a diverse range of participants to ensure we capture all types of feedback, given that teachers in different parts of the country, race, religion, etc., all have different lived experiences and varying expectations of what feedback should look like and sound like.

     
    1
    Discussion is closed. Upvoting is no longer available

    Lisa Miller
  • Icon for: Neil Heffernan

    Neil Heffernan

    Professor and Director of Learning Sciences and Technologies
    May 14, 2019 | 08:23 a.m.

    As a professor, I try to personally respond to what my students write. I asked them a lot of open ended questions but I only have time to review a tiny fraction. I have written little emails to students and they have responded with "no other professors has ever been so thoughtful to reach out to me! Thanks!"   I feel bad as it don't spend as much time as I should especially considering the positive reaction to the fact that I am paying attention to them.  So I want to just make it easier for a teacher to do such things.    

    Our big goal is to save teachers time. We want more kids to get teacher feedback more often.  In year three we will run a study to see how much time we have saved teachers (if any).  Clearly if we suggest messages to send, that teachers don't want to send, we have wasted their time in reviewing them.   

    We also want to understand the engage factor: if teachers get a fixed amount of time, and we asks them to send some messages to their students how many more students can they "touch".  And what is the impact of student engagement if their teacher has recents responded to their written work?

     
    1
    Discussion is closed. Upvoting is no longer available

    Lisa Miller
  • Icon for: Lisa Miller

    Lisa Miller

    Facilitator
    May 15, 2019 | 09:46 p.m.

    Thank you Hilary and Neil for your responses!

  • Icon for: Abby Funabiki

    Abby Funabiki

    Facilitator
    May 13, 2019 | 10:33 p.m.

    Super exciting to see a project focused on helping teachers save time while still providing valuable feedback! Did the platform ASSISTments already exist in other subjects or are you creating it for this project? 

    Also, why did you choose math and do you think this would work in other subjects?

     
    1
    Discussion is closed. Upvoting is no longer available

    Jared O'Leary
  • Icon for: Hilary Kreisberg

    Hilary Kreisberg

    Lead Presenter
    May 13, 2019 | 10:59 p.m.

    Hi Abby!

    Thank you for your feedback! ASSISTments has been around for a while! In fact, when I was a fifth grade teacher back in 2010-2014, I used ASSISTments to help prepare my students for standardized testing. It was a great way to instantly see my students' work and know right away how they performed on what was then more multiple-choice oriented questions. Now, I can imagine the struggle teachers face everywhere with the massive focus on open-response like questions.

    In fact, in April 2019, the What Works Clearingouse (WWC) published a federal research review of the ASSISTments Efficacy Trial and found ASSISTments to be one of the most effective interventions in the nation! ASSISTments received the highest standard of WWC, which is what is called "positive effects without reservation.” There are only five interventions in the United States that currently meet that standard.

    ASSISTments offers other subject areas, too. See here https://app.assistments.org/FA/f/e?sd=certified&p=1. However, as a free, non-profit entity, capacity has required the team to focus and hone in on one subject area and do it really well. Then, to expand beyond that. Additionally, my focus is on mathematics education, so we decided to stick with that subject for this particular grant. This can and should be applied to other subjects!

     
    1
    Discussion is closed. Upvoting is no longer available

    Abby Funabiki
  • Icon for: Abby Funabiki

    Abby Funabiki

    Facilitator
    May 13, 2019 | 11:05 p.m.

    Thank you for sharing! It is so great that you are adding to a tool that already exists and is proven!

  • Icon for: R. Bruce Mattingly

    R. Bruce Mattingly

    Higher Ed Administrator
    May 14, 2019 | 01:47 a.m.

    Your project is fascinating, and I am eager to follow your progress. As a former college mathematics instructor, I'm wondering if the same approach could be used in higher level mathematics courses?

  • Icon for: Neil Heffernan

    Neil Heffernan

    Professor and Director of Learning Sciences and Technologies
    May 14, 2019 | 07:52 a.m.

    Thanks Bruce,

    ASSISTments has a builder with it so any teacher can create their own problems. I use it in my AI class at WPI. In fact this is where I started to think more about the open-response questions. ASSISTments is really good at scoring math problems with answers like "2x+3" or  "4/5 (which is also .8)" I would randomly review the open response responses in class, this was easily facilitated by ASSISTments. One day I sat down and wrote emails that referred directly to my students responses. The feedback was amazing, students said they had never been given such personal feedback.  That is when my wife Cristina and I began designing this project.  

  • Small default profile

    Tracy Manousaridis

    K-12 Administrator
    May 14, 2019 | 07:18 a.m.

    So excited to see AI technology applied to math education.  Any plans to extend to intermediate grades (4th and 5th)? 

    Thank you to the team for developing this!

  • Icon for: Cristina Heffernan

    Cristina Heffernan

    Co-Presenter
    May 14, 2019 | 07:57 a.m.

    We are creating the algorithms that should eventually work for any grade.  ASSISTments has content for 4th and 5th grade already so any teacher can start using it and start to get feedback on the computer scorable problems.  This new added feature of DRIVER-SEAT is being developed in MS but should extend to other subjects and grades. 

  • Icon for: Quinn Burke

    Quinn Burke

    Facilitator
    May 14, 2019 | 06:33 p.m.

    Hilary (and wider team) - really cool video and exciting work! As a former high school English teacher, I used to spend considerable time on offering substantial feedback on student papers that mixed encouragement, constructive criticism, and productive pushback.  Can see such a need for this with math where kids often get so incredibly frustrated....

     

    I have to follow up on a thread here.  You  mention the challenge of trying to develop teachers' feedback messages in their own cultural or personal preferences - can you give an example from your research here?

  • Icon for: Hilary Kreisberg

    Hilary Kreisberg

    Lead Presenter
    May 14, 2019 | 07:10 p.m.

    Hi Quinn,

    Thank you for your feedback, especially offering the English teacher perspective. What I meant in that other thread is that we have to be incredibly cognizant of how performance feedback varies across cultures. While some cultures are more direct, others are not. For example, if a teacher's feedback to a student is "You need to be more careful when adding fractions and be sure to use common denominators" this might come across harshly and not be well received or useful to a student whose culture looks at directness in a negative light or personally does not work well with directness. If a teacher's feedback is less direct, for example, "I wonder how you could use common denominators to help you with this task?," some students may misinterpret and not realize they are to go back and try using the suggestion since it wasn't directly stated. In another fashion, while some teachers prefer complete sentences and academic language, others do not share that same value when assessing mathematics. Most importantly, we need to continue to develop a participant pool that is representative of the various communities and cultural groups present in the United States so we can further study how culture, location, etc., plays a role in feedback. I hope I answered your question!

  • Icon for: Quinn Burke

    Quinn Burke

    Facilitator
    May 14, 2019 | 07:24 p.m.

    It does!  Thanks Hilary -- very interesting work!

     

  • Icon for: Dave Miller

    Dave Miller

    Higher Ed Faculty
    May 15, 2019 | 09:22 a.m.

    What a terrific project, and thanks for sharing it in the Showcase! I like the name, too! The notion of machine learning to assist with responses could be a huge productivity tool for schools, teachers, and to assist with personalized learning - whether it’s for student who need substantive guidance throughout a topic or for the student who wants to blaze ahead on a quicker timeline.  I’m wondering what dimensions this would add to threaded conversations in chat, for students, and expert others (or even a chat bot) and if the data collected could be applied to a push outreach to connect to experts and online support teachers/tutors who might be employed through an education services provide to be immediately responsive. I’m guessing student work could also be connected through AI so that students (and their teachers) could receive push connections that loop in real-world applications of what students just learned about, so as to assist with helping students make connections. I would be interested in connecting with your team for a conversation over the summer, if you’re interested. Thanks for a great project! - Dav

  • Icon for: Cristina Heffernan

    Cristina Heffernan

    Co-Presenter
    May 15, 2019 | 12:50 p.m.

    Hi Dave,  

    We are glad you liked it and yes the name felt like an accomplishment all on its own!

    One note is that we are working to support teachers as they teach . We are concerned about tools that let students "blaze ahead on a quicker timeline", a few reasons include that without a classroom or human to human experience there is a concern that learning can become shallow. This project is intended to support teachers by saving time allowing for them to plan and orchestrate better human to human teaching. 

    Dialogue is what the D in DRIVER-SEAT stands for so yes we are moving in the direction of a back and forth between teachers and students. It is tricky since we also want to save teachers time. Your idea to employ education services interesting. 

    We definitely think that by allowing teachers to have more time to plan they will be able to enhance their lessons with connections.  

    I just watched your video and we should connect and see if ASSISTments would be an interesting tool for some of your consortium teachers to explore. 

    Cristina

  • Icon for: Jennifer Stevens

    Jennifer Stevens

    President & CEO
    May 15, 2019 | 09:46 a.m.

    Thank you for sharing your work with us! I am simply amazed at the thought of AI being used to help busy teachers provide effective, timely feedback to students! In our project, teachers are providing feedback to one another using videos of classroom instruction, and one of the biggest challenges is TIME - time to view, time to reflect on the feedback needed, and time to actually respond.  I know this is one of the reasons teachers do not give this kind of feedback to their students on a regular basis, so again, I am excited to learn that you are working on this issue.

    Question for you - what kind of training was necessary for your teachers to use the technology?  And for the teachers actually writing the feedback, were they given any kind of training on giving effective feedback to students?  

    Jennifer

  • Icon for: Cristina Heffernan

    Cristina Heffernan

    Co-Presenter
    May 15, 2019 | 01:07 p.m.

    Training is an interesting issue.  With this project we are only working with teachers who use Engage NY or Open Up resources so for them there was very little training. ASSISTments is a free service developed at our university, Worcester Polytechnic Institute so we are able to put all the practice problems and exit cards from Engage NY, Open Up by Illustrative Math and Utah math into our system. If a teacher is already using that curriculum and has Google Classroom they can use ASSISTments without any training.  

    Training is needed if the teacher has to create their own content or needs help finding what they need.  It may also be needed if a teacher does not see the value of using homework as a formative assessment strategy. That awareness is not always present in teachers so a motivational/informational session may be needed. 

    You can learn more at www.assistments.org and keep asking questions on this forum.  

     
    1
    Discussion is closed. Upvoting is no longer available

    Jennifer Stevens
  • Icon for: Hilary Kreisberg

    Hilary Kreisberg

    Lead Presenter
    May 15, 2019 | 11:22 a.m.

    Hi Jennifer,

    I'll let my co-presenters talk more about the training needed and their future aspirations of making ASSISTments easier to learn how to use without our facilitation. In terms of feedback, we have been strategic as to not provide many guidelines as we wanted to see what teachers do so we can compare and contrast how they give feedback. Additionally, because effective feedback is still very subjective we didn't want to impose what we like in feedback or think is necessary to change the way the teachers provided feedback. After collecting thousands of messages, we did ultimately ask teachers who responded to students with, "Perfect!" or "Excellent work!" to expand their feedback to specifically tell students what behaviors or actions they did well (e.g., "Excellent work including numbers in your response!") because we want the algorithms and AI model to be able to read student responses and offer more specific feedback. I hope this helps! 

     
    1
    Discussion is closed. Upvoting is no longer available

    Jennifer Stevens
  • Icon for: Kathryn Kozak

    Kathryn Kozak

    Higher Ed Faculty
    May 15, 2019 | 08:46 p.m.

    Very interesting project. Thank you for sharing.

  • Icon for: Cristina Heffernan

    Cristina Heffernan

    Co-Presenter
    May 15, 2019 | 09:42 p.m.

    It was fun to make the video and even more fun to do the project. Your StatPREP work looks great. We definitely need a community of people with a good understanding of statistics. We do a lot of work with large data sets! 

  • Icon for: Jared O'Leary

    Jared O'Leary

    Researcher
    May 16, 2019 | 02:25 p.m.

    This is an interesting tool that will help save time and help with further learning. I really appreciate that the feedback examples included questions rather than directives or statements alone and I look forward to hearing about the results! 

     
    1
    Discussion is closed. Upvoting is no longer available

    Abby Funabiki
  • Icon for: Cristina Heffernan

    Cristina Heffernan

    Co-Presenter
    May 18, 2019 | 09:07 a.m.

    Your point about what goes into the feedback messages is important.  We are interested in finding out what type of feedback works best for whom and when. Part of our research flow involves doing randomized controlled trials to see what type of feedback works best. 

     
    1
    Discussion is closed. Upvoting is no longer available

    Jared O'Leary
  • Icon for: Chuck Barlow Jr

    Chuck Barlow Jr

    Informal Educator
    May 17, 2019 | 10:48 a.m.

    Hi Hilary,

    Thanks for sharing your video.  Your team has done an awesome job! Having teachers from all parts of the country contribute to the database provides valuable input from different regions. What have been some of the challenges going into Phase 2, other than the capacity of your organization?

  • Icon for: Hilary Kreisberg

    Hilary Kreisberg

    Lead Presenter
    May 17, 2019 | 01:01 p.m.

    Hi Chuck,

    Thanks! I believe this has been answered in the thread above - check it out and let me know if any other questions still resonate with you!

  • Small default profile

    Daddy

    Parent
    May 19, 2019 | 11:47 a.m.

    Hilary,

    Great job.

    Obviously, I am prejudiced!

    Daddy

     

     

  • Icon for: Cristina Heffernan

    Cristina Heffernan

    Co-Presenter
    May 19, 2019 | 04:48 p.m.

    Hello Hilary's Dad!  Thanks for checking out our work.  Now I am motivated to get my dad to comment.  Lets see if I can!

  • Further posting is closed as the showcase has ended.