1. Nathan Holbert
  2. Formative Assessments for Computer Science in NYC
  3. http://play.beatsempire.org
  4. Teachers College, Columbia University
  1. Matthew Berland
  2. Formative Assessments for Computer Science in NYC
  3. http://play.beatsempire.org
  4. University of Wisconsin - Madison
  1. Betsy DiSalvo
  2. http://betsy.disalvo.com
  3. Associate Professor
  4. Formative Assessments for Computer Science in NYC
  5. http://play.beatsempire.org
  6. Georgia Institute of Technology
  1. Jeremy Roschelle
  2. http://digitalpromise.org/our-team/jeremy-roschelle/
  3. Executive Director, Learning Sciences Research
  4. Formative Assessments for Computer Science in NYC
  5. http://play.beatsempire.org
  6. Digital Promise Global
  1. Daisy Rutstein
  2. https://www.sri.com/about/people/daisy-wise-rutstein
  3. Principal Education Researcher
  4. Formative Assessments for Computer Science in NYC
  5. http://play.beatsempire.org
  6. SRI International
Public Discussion

Continue the discussion of this presentation on the Multiplex. Go to Multiplex

  • Icon for: Nathan Holbert

    Nathan Holbert

    Lead Presenter
    May 12, 2019 | 01:03 p.m.

    Thanks for checking out our video on Beats Empire and the Formative Assessments for Computer Science in NYC project. Play the game at play.beatsempire.org and let us know what you think!

  • Icon for: Gerad OShea

    Gerad OShea

    Facilitator
    Research Director
    May 13, 2019 | 12:24 p.m.

    Thanks for sharing your work!

    Can you share anything about how teachers are able to view students' work or track changes to the strategies students use to navigate the game?

  • Icon for: Satabdi Basu

    Satabdi Basu

    Researcher
    May 13, 2019 | 01:51 p.m.

    Hi Gerard, We are currently building a dashboard to help teachers view information about their students' progress, and finalizing what features/student characteristics to display in real time versus which to provide after a game play episode. We used a principled process using the Evidence Centered Design framework to come up with specific assessment targets for Data and Analysis - what we call FKSAs or Focal Knowledge, Skills and Abilities we would like to elicit evidence of through the game. We are attempting to capture information about as many of those FKSAs as we can from the log data and display them in a teacher-friendly manner using our dashboard.

     
    1
    Discussion is closed. Upvoting is no longer available

    Gerad OShea
  • Icon for: Jill Denner

    Jill Denner

    Researcher
    May 13, 2019 | 01:18 p.m.

    The game looks amazing. This is the most unique approach to teaching data science that I have seen. I look forward to hearing more about the research results as they come in.

  • Icon for: Jeremy Roschelle

    Jeremy Roschelle

    Co-Presenter
    Executive Director, Learning Sciences Research
    May 14, 2019 | 12:05 a.m.

    Jill, 

    Thanks for that comment. One place we have found the Standards challenging to work with is "Store data" -- in our music game context, we were able to give students to show their thinking about "collect" "visualize" and "model/make inferences" -- but what is appropriate for Store for middle school kids in the context. Would love to know what you think!

     

    jeremy

     

  • Icon for: Laurin Buchanan

    Laurin Buchanan

    Researcher
    May 15, 2019 | 12:34 p.m.

    I'm very interested in learning more about your project - musician who worked in the music biz for a while before a career change into cybersecurity...And as a practitioner turned researcher, I'm not sure what Standard you are referring to ---and this may be offroading a bit, but what occured me was approaching the "Store data" concept --particularly since this game models the entertainment business-- as a way for the students to think about data lifecycle (creation > use > transmission > storage) issues, and the requirements of storage: do you have sufficient file server space for data storage? Can copies be made and stored elsewhere, e.g. USB drive? how long should stored data be retained? And thinking about data protection requirements as well: who has access - is access recorded/monitored? Are changes to stored data allowed, and is stored data monitored for changes? (data integrity)? Is the stored data of high value (e.g., is it Big Artist's unreleased single you don't want released early / for free!) and should it be encrypted, monitored for access? Is it financial investment data about your record company that not everyone should have access to? These are the types of questions we will be addressing with cybersecurity curricula for middle school students in our project.

  • Icon for: Jeremy Roschelle

    Jeremy Roschelle

    Co-Presenter
    Executive Director, Learning Sciences Research
    May 15, 2019 | 04:29 p.m.

    Hi Laurin,

    Thanks for spending so much time thinking about this -- your ideas are generative and helpful to us. The standards are these CSK12 framework statements about data. In particular for grades 6-8

    pplications store data as a representation. Representations occur at multiple levels, from the arrangement of information into organized formats (such as tables in software) to the physical storage of bits. The software tools used to access information translate the low-level representation of bits into a form understandable by people.  Its true that lots of this pertains to music file formats, but it doesn't seem players of our game need to get into that level of thinking to play the game. We didn't find a way to make these sorts of things germane and interesting. 

      Description

    Computers can represent a variety of data using discrete values at many different levels, such as characters, numbers, and bits. Text is represented using character encoding standards like UNICODE, which represent text as numbers. All numbers and other types of data are encoded and stored as bits on a physical medium. Lossy and lossless data formats are used to store different levels of detail, but whenever digital data is used to represent analog measurements, such as temperature or sound, information is lost. Representations, or file formats, can contain metadata that is not always visible to the average user. There are privacy implications when files contain metadata, such as the location where a photograph was taken

  • Icon for: Satabdi Basu

    Satabdi Basu

    Researcher
    May 13, 2019 | 01:56 p.m.

    Thank-you Jill! We've run a few pilot studies at this point where we have already gathered some useful information about strategies students use and  pre-conceptions students have about data based decision making and the music industry. We have also been developing some follow-up activities and accompanying teacher materials to help teachers follow-up on the information they receive from the game dashboard. We look forward to piloting them in the next couple of months.

     
    1
    Discussion is closed. Upvoting is no longer available

    Laurin Buchanan
  • Icon for: Feng Liu

    Feng Liu

    Facilitator
    Researcher
    May 13, 2019 | 11:37 p.m.

    Thanks for sharing this great project. I know this game is designed to be a formative assessment tool to help students learn computer science specifically data science. I am wondering whether you also considered to look at it as a summative tool. Or another word, whether you considered measuring the impact of the game play on student outcomes such as student interest/attitude towards data science/computer science and possibly student academic achievement in those fields.

  • Icon for: Jeremy Roschelle

    Jeremy Roschelle

    Co-Presenter
    Executive Director, Learning Sciences Research
    May 14, 2019 | 12:10 a.m.

    Hi Feng Liu,

    I appreciate the push. Of course, we'd like to gain some confidence on formative before summative -- we're not there yet!

    More generally, I am frequently puzzled as to what would constitute a worthwhile measure of a change in "student interest/attitude toward..." Can you recommend a measure that you consider credible? How could we measure that sort of thing convincingly?

     

  • Icon for: Nathan Holbert

    Nathan Holbert

    Lead Presenter
    May 14, 2019 | 08:52 a.m.

    I second this interest in finding high quality interest/attitude metrics. 

    One interesting data point from our pilot studies that I think *does* speak to "interest/attitude towards..." (though maybe not CS/data science) is that students spend a lot of time tinkering with the combination of mood, topic, and song name as they record. In other words, though they may choose the "angry" mood for data-specific reasons, they often attempt to find a song topic and a song title that matches this angry mood or the specific artist that's recording. We see this as evidence that players are bringing their real world interest in music into the game and engaging with mechanics in a personal way.

    This doesn't point to an interest in CS or data science, but I think it does suggest deep engagement and interest in the game (perhaps more so than having students indicating on a likert scale whether or not they thought the game was fun).

     
    1
    Discussion is closed. Upvoting is no longer available

    Satabdi Basu
  • Icon for: Rachel Becker-Klein

    Rachel Becker-Klein

    Researcher
    May 14, 2019 | 08:25 a.m.

    This is such a creative way of getting students to do computer and data science in a fun and engaging way. I think there could be opportunities to embed assessment of skills into the game, and maybe you have already thought of that? For instance, if they are interpreting graphs to make predictions or decisions, can that be assessed in some way within the game? 

  • Icon for: Nathan Holbert

    Nathan Holbert

    Lead Presenter
    May 14, 2019 | 08:58 a.m.

    Thanks Rachel! Assessing students ability to make decisions about how to collect data, visualize data, and draw inferences from the data is the primary goal of the game. How students interact with data representations throughout the game, and what kind of predictions they make from these representations are logged. These logs are analyzed live and key information is presented to the teacher on a dashboard. While this dashboard system is still in development, our goal is to provide teachers with actionable and formative information about how players are encountering different kinds of data representations, how they are making decisions about those representations, where they are getting stuck etc.

  • Icon for: Rebecca Roberts

    Rebecca Roberts

    Higher Ed Faculty
    May 14, 2019 | 08:53 a.m.

    I am blown away. Do you ultimately plan on making this publicly available? I would love my own musically-inclined middle schooler to have access.

  • Icon for: Nathan Holbert

    Nathan Holbert

    Lead Presenter
    May 14, 2019 | 09:04 a.m.

    Thanks for the kind words Rebecca! The game is currently freely available at play.beatsempire.org. While the game is available for you and your students to play, it's worth mentioning that we are currently conducting small scale studies throughout NYC schools to evaluate and refine the game. The teacher dashboard is in development but not yet useable.

    If you do decide you'd like to try the game out with some students, please let us know so we can be in contact with you to get feedback!

     
    1
    Discussion is closed. Upvoting is no longer available

    Satabdi Basu
  • Icon for: Feng Liu

    Feng Liu

    Facilitator
    Researcher
    May 14, 2019 | 05:23 p.m.

    Hi Nathan and Jeremy,

    I agree that measuring student interest/attitude could you challenging especially for certain subject area and specific grades of students. One instrument I found is useful is Student Attitudes toward STEM Survey (S-STEM) (https://miso.ncsu.edu/articles/s-stem-survey?_sm_au_=iMVfPSD5Z2P6QPHH). It’s designed to measure the four constructs: Math Attitudes, Science Attitudes, Engineering and Technology Attitudes, and 21st Century Learning. It has two versions: one for 4-5th graders and one for 6-12th graders. You can find its reliability and validity evidence here: https://miso.fi.ncsu.edu/wp-content/uploads/2013/06/S-STEM_FridayInstitute_DevAndPsychometricProperties_FINAL.pdf. I also conducted a psychometric analysis of this instrument using data collected for one of the projects I have worked on and confirmed these evidence.

     
    2
    Discussion is closed. Upvoting is no longer available

    Laurin Buchanan
    Satabdi Basu
  • Icon for: Jeremy Roschelle

    Jeremy Roschelle

    Co-Presenter
    Executive Director, Learning Sciences Research
    May 14, 2019 | 05:39 p.m.

    Thank you Feng Liu! This is super helpful, as I am not expert in measuring attitudes.

    jeremy

     

  • Icon for: Joi Spencer

    Joi Spencer

    Higher Ed Administrator
    May 15, 2019 | 12:42 p.m.

    Thank you for presenting this very fascinating project. Just a few questions:

    - Have you thought of this work in terms of funds of knowledge and drawing in students who generally have little interest in traditional computer science?

    - Are you piloting this throughout the city? Since the television show Empire is focused on urban NYC, I wonder how you have connected with this population.

    - Do users have the ability to ask questions of the data? Can they run different analytics based on their own questions?

    Thank you again.  Looks like a wonderful project!

  • Icon for: Satabdi Basu

    Satabdi Basu

    Researcher
    May 15, 2019 | 10:01 p.m.

    Hi Joi,

    Thank-you for your kind words and your interest in our project!

    Yes, we have been thinking of students with limited funds of knowledge and interest in traditional CS. While Beats Empire has been designed as a formative assessment game and not as a learning game per say, we have developed a video where we have interviewed real life professionals and experts working in the music industry who have described similarities of the game with what they do in their day-to-day professional lives. Our hope is that the video will help students recognize various non-traditional career opportunities involving the application of computer science and data analytics.

    Students do not get to run analytics on data in the current version of the game. They can choose to view data visualizations of most popular types of songs in a  particular borough, or the trend for a particular genre or mood of music, and then make inferences from these visualizations to decide what song to record, what artist to hire, where to release a song, etc.

    As for the question about where we're piloting in NYC, I think our NY-based collaborators can answer that better than I can. I'll pass on your question to the rest of the team.

    Thanks!

     
    1
    Discussion is closed. Upvoting is no longer available

    Satabdi Basu
  • Icon for: Nathan Holbert

    Nathan Holbert

    Lead Presenter
    May 16, 2019 | 08:48 a.m.

    Joi,

    One of our goals during the design phase was absolutely to create a game that would tap into the interests and experiences of NYC Middle School students. We conducted multiple focus groups with students throughout NYC to talk to them about their interest in music, how they listen to it, where they find it, how they talk about it with their friends, etc. These conversations were incredibly insightful and directly informed the design of game mechanics and aesthetic.

    One way we try to tap into players' funds of knowledge is by using artist names that sound a lot like the names of real artists (for example, "Beyonde" and "Half Dollar") and music genres with which students will likely be familiar. We've found that some students spend a lot of time tinkering with the song topic, mood, and title to record songs that fit their experiences with music outside of the game. While this activity isn't always about optimizing decisions based on data, it is a meaningful choice for players and we wanted to be sure Beats Empire supported this play!

     
    1
    Discussion is closed. Upvoting is no longer available

    Satabdi Basu
  • Icon for: Nancy McGowan

    Nancy McGowan

    Facilitator
    Instructional Math Coach
    May 15, 2019 | 08:52 p.m.

    Very innovative approach.  What specific information will teachers be able to learn about students’ knowledge of data and data analysis concepts and practices?

  • Icon for: Satabdi Basu

    Satabdi Basu

    Researcher
    May 15, 2019 | 09:48 p.m.

    Hi Nancy,

    While we are still developing our teacher dashboard and finalizing what information we will provide to teachers via the dashboard, here are a few things that are likely candidates:

    - Whether a student attempts to access/view data before making a decision about what kind of song to record or which artist to hire

    - Whether a student makes correct inferences from provided data visualizations

    - Which kind or kinds of data visualizations is a student relying on in order to make decisions? (bar graph, line graph, a combination)

    - Is a student balancing the frequency of data collection with the storage requirements?

    We have also been developing some follow-up activities for the game, and teachers will get additional formative information through those activities. For example, can students identify a trend? Can students identify what data needs to be collected and in what representation to answer a question or generate a given visualization? Do students realize when it is better to process data manually versus program the computer to process data? How much data do students think is sufficient to collect for making a prediction about a broad population?

    Thank-you!

     
    1
    Discussion is closed. Upvoting is no longer available

    Satabdi Basu
  • Icon for: Nancy McGowan

    Nancy McGowan

    Facilitator
    Instructional Math Coach
    May 15, 2019 | 10:02 p.m.

    Thank you! This was very helpful.  Good luck with your program!

  • Icon for: James Laffey

    James Laffey

    Researcher
    May 17, 2019 | 02:05 p.m.

    Wow, very cool way to marry kid interest with ways of putting knowledge of data science into practice. I played a little bit, but not all the way to the ending. Couple of questions:

    1. Is the game connected with some specific curriculum or meant as a general assessment that could be applied across many contexts?

    2. Is there some form of game ending wherein the student sees his/her outcome and gets diagnostic feedback?

    3. One challenge we have in our efforts to embed assessment in our game play is characterizing motivation.....could kids be just exploring and not optimizing performance, could kids be looking for novel reactions from the game rather than trying to perform as well as possible, etc. Have you addressed these issues at all?

    Looks like a great team and great project!!

  • Icon for: Jeremy Roschelle

    Jeremy Roschelle

    Co-Presenter
    Executive Director, Learning Sciences Research
    May 17, 2019 | 03:46 p.m.

    Thanks, James.

    It is meant to be general -- in our pilot context in NYC, lots of different materials are in use.

    I don't think we have a game ending, but others on the team may know more.

    Our overall plan for addressing assessment subtleties (like the one you mention in your #3) is to give teachers guidance on how they should follow up to learn more about how their students' think with data. The game can serve as a context to elicit student thinking both in game play and in follow-up, unplugged discussions.

    best, jeremy

  • Icon for: Dennis Ramirez

    Dennis Ramirez

    Researcher
    May 17, 2019 | 05:59 p.m.

    Great work you all! Was there any player feedback that significantly impacted or changed the game as it was being developed, such as a cool idea that didn't stick, or an emergent feature that was implemented into the game?

  • Icon for: Nathan Holbert

    Nathan Holbert

    Lead Presenter
    May 20, 2019 | 09:38 a.m.

    Thanks Dennis! Not exactly, but one of the interesting observations of students playing the game has been how interested many of them are in aligning their game play with their real world experience with music. We've found that some students spend a lot of time tinkering with the song topic, mood, and title to record songs that fit their expectations about what kind of music their artists *should* record--such as having the artist "Beyonde" record some awesome angry love songs. It's been very important for us that the game support this kind of play! The trick now is how to capture/account for this meaningful, though maybe not data-based, play on the assessment side.

  • Further posting is closed as the showcase has ended.