In general, the content is created by the instructors using the tool and an instructor can generate whatever questions they wish if they want to use the system. 

We are, however, using the data gained from testing the tool in a relatively-large (~100 student) introductory physical geology course to craft a suite of data-validated assessment questions that will eventually be made available to any interested geology instructors who use the tool. In effect, if a geology instructor wanted to sign up for the system, we could immediately provide them with a suite of quizzes based on learning objectives they select. We have plans for similar endeavors in other STEM fields as well (e.g., chemistry, biology), but we started with the geosciences because that is our "home" discipline and are the courses we teach :)

Hi Stephen, thanks for your comment! I will address each point in sequence:

- The PD angle is a great suggestion and one that I definitely could see utility in pursuing! Thanks for that!

- Most students (from interviews I've conducted) buy into the system and enjoy using it because its their data measurements that we are providing, not just generic feedback. They all want to "get a good grade," and many seem to appreciate not only the feedback the system provides, but the opportunity to see a model of our assessment style prior to course exams (as we have generated the quizzes and use the same learning objectives course-wide).

- In developing/testing the tool on our own physical geology students, we have had a few different layouts of how it was integrated into the course. First it was completely optional and there was near ubiquitous buy-in (~90% of students taking quizzes), but they were taking them immediately prior to the exam (a strategy not supported by learning research). So now we ask them to distribute their attempts by having weekly quizzes that are for a grade (unlimited attempts, highest score counts).

- We have some general training on the data and measurements that CLASS provides at the beginning of the semester and we come back to their trends as the semester progresses.

- As far as what happens after the assessment, they use the quizzes to prepare for course exams and we consider their trends and results to communicate strategies for where to go for more help (i.e., book pages x, y, z). In addition, we can augment in-class instruction on low-performing or high bias (disparity between confidence and performance) concepts as a result of data gained from students' quiz results.

Hi Bess,

We have piloted the tool in introductory (undergraduate) physical geology courses here at NCSU and saw that (for example) students performed better and were more accurate in predicting their exam grades than in prior semesters, but we have not done any testing in high school classrooms as yet. We are, however, also testing the tool in similar courses at two 2-year colleges (one in NC and one in RI) to investigate comparative effects on students at different types of institutions. As the demographics and academic histories of the students at public research institutions and two-year colleges are generally very different, we are seeking to determine the best way to use the tool to support students of all backgrounds and levels of academic experience. 

Hi Patricia, thanks for your comment! I will address each point in sequence:

- For this question regarding participation, Stephen asked a similar question above so I will share my response to him below;

"In developing/testing the tool on our own physical geology students, we have had a few different layouts of how it was integrated into the course. First it was completely optional and there was near ubiquitous buy-in (~90% of students taking quizzes), but they were taking them immediately prior to the exam (a strategy not supported by learning research). So now we ask them to distribute their attempts by having weekly quizzes that are for a grade (unlimited attempts, highest score counts)."

- We have put a lot of time into generating a suite of quizzes and questions for introductory physical geology, but the tool itself can be used for anything that: a) someone wants to learn/teach and; b) can be assessed via questions, so it will definitely accommodate other content areas. The instructor would, however, have to have defined learning objectives to assess prior to using the tool effectively. Following the principles of backwards design (Wiggins & McTighe, 1998), these objectives should define the assessments, instruction, and skills the student should have as a result of completing the course activities. So to introduce the tool to new instructors, I would suggest a short primer on backwards design and on the nature of student confidence judgments and related accuracy (called "calibration) prior to generating quizzes and trying them out on their students. Towards this end, in the future, we will be seeking to partner with colleagues in other STEM disciplines (e.g., chemistry, biology) to generate curated suites of quizzes (like the ones we currently have for geology) that would be ready to use for instructors who may not want to have to generate their own quizzes from scratch.

- We ask for reflections on their CLASS feedback and related study habits as part of a weekly assignment in our course management system, so not in the system itself, but CLASS can provide specific feedback and data for them to track across the semester provided you prompt them to be mindful of its results. For example, we can ask students to define their highest bias learning objective at the time of the assignment (disparity between confidence and performance) and to come up with a plan to remedy that disparity. In addition, we can view that same metric for the course a a whole (from the instructor side) and suggest new reference material or augment instruction to help address the issue. 

- We have evidence that students do close the gap more as a result of using the tool (particularly in being able to predict their eventual exam score), but we have more analysis to complete on that in the near future in hopes to further elucidate the details of the relationship and how it may potentially vary across student variables (e.g., demographics, academic history, institution, etc.). 

Hi Daniel,

- For a brand new instructor in a brand new content area, it would work like any other quizzing system in that quizzes would need to be built via entering in questions one-at-a-time, albeit tying them to specific and defined learning objectives. Once this process is completed, however, instructors would then be able to generate new quizzes from existing objectives and share them with themselves or other instructors (e.g., a new quiz with objective A from quiz X and objective B from quiz Y), or make their quizzes public for others to take as students or use as instructors. In the future, we will be seeking to partner with colleagues in other STEM disciplines (e.g., chemistry, biology) to generate curated suites of quizzes (like the ones we currently have for geology) to make this "build out" as you call it more seamless in the future.

- We have quantitative data supporting increased exam scores and increased measures of accuracy related to their judgments (mounds of data/more analysis to come on that, so we will know more nuance to that in the near future). In addition, we have qualitative data (student interviews) that support the notion that they may be taking CLASS more seriously than other supplemental instruction endeavors because they see utility in the feedback we are providing ("After taking some quizzes, I saw I needed to really study concept A and not concept B") and that they are not seeing the quiz-taking process as "extra work" (anecdotally a common student view regarding supplemental instruction in general). A lot of students seek out practice quizzes on the internet as part of their study habits anyway, we are just providing them with ones that are not only generated by their instructors, but ones that are aligned to course objectives and that provide them information related to their learning beyond simply performance feedback. Finally, when we ask students to rank the components of our course that they felt helped them learn the most (e.g., the book, quizzes, lectures, homework assignments, videos, etc.), they have statistically significantly ranked the quizzes higher each semester they have been on CLASS (mode = #1) than when the same quizzes were administered via our course management system. We are, however, always trying to increase the student utility of the tool and hope to develop best practices for its use in college STEM classrooms.

Thanks for the detailed response. I'm looking forward to incorporating what you describe into our own new course sequence.

D

Hi Frank,

That is a great idea! Thanks for the suggestion! We are considering similar lines of inquiry via quantitative analysis (e.g., multiple regression models to investigate predictive relationships between student variables such as demographics and CLASS use and exam outcomes). The idea, however, of potentially developing a path model that may isolate some "turning points" of course content, if you will, that lead to variable achievement for the students (or some subset of students) would be a great path forward (*pun intended*).