Clickety Click Click: Awful Measures for Learning

Dec 19, 2016 by

I feel a little inspired by Sam Ford’s post The Year We Talk About Our Awful Metrics. Ford writes about the need for change in metrics of online media publications, but we could just as easily be discussing the metrics of learning management systems, ed-tech vendor platforms, and institutional analytics.

Ford argues that we need to “get serious” about better forms of measurement in 2017. As long as we are measuring metrics with little meaning, we aren’t really improving learning.

Let me give you a few examples to illustrate the similar problems in education.

Page Clicks

As in, how many pages of the ebook has the student accessed? Because the student must read every page they access, right? And they don’t just scroll through pages to see roughly what the book has in it? Realistically, I think we all acknowledge these inevitabilities, but that doesn’t stop us from creating blingy dashboards to display our metric wares.

Consider the following scenarios.

Scenario 1: Student A has clicked on 55 pages whereas student B has only clicked on 10 pages. This means:

a. Student A has read more than Student B. Student A is a more engaged student.

b. Student B was reading deeply and Student A was skimming.

c. Student A reads faster than student B.

d. Student A read more online. Student B borrowed a book from a friend and read more on paper.

e. None of the above. Who knows what it really means.

Scenario 2: Student A has clicked on 55 pages whereas student B has only clicked on 10 pages. Both students spent 2 hours in the eReader platform.

a. Student A has read more than Student B. Student A is a more engaged student.

b. Student B was reading deeply and Student A was skimming.

c. Student A reads faster than student B.

d. Student A read more online. Student B borrowed a book from a friend and read more on paper.

e. None of the above. Who knows what it really means.

In either case, how much do we really know about how much Students A and B have learned? Nothing. We know absolutely nothing. These metrics haven’t done a thing to see what either student is capable of recalling or retrieving from memory. There is nothing to help us to see whether the student can make sensical decisions related to the topics and nothing to show whether concepts can be transferred to new situations. Page clicks are a bad metric. All they tell me is that students log in more every Sunday night than on any other night (and that metric has been the same for a decade now).

But wait … there are more metrics …

Attendance

We can measure attendance – whether it be logging in to the LMS or physically showing up in the classroom. Surely this is a valuable measure of learning?

Again no, it’s not a measure of learning. It’s potentially a necessary condition of a necessary-but-not-sufficient metric for learning. Yes, we do need students to show up in some way to learn. In very active face-to-face classrooms that engage all students in learning activities, I might go so far as to say that showing up is a good measure of learning, but this is still the exception rather than the norm. And even if the classroom is active, learning is more effective with certain kinds of activities: those involving interaction, those involving varied practicethose where students have to learn to recognize and remedy their own errors.

Attendance, by itself, does not measure learning.

Time to Complete

At organizations where the learning is assessed directly (CBE and MOOCs, for example), there is often a metric around the “time to complete” a course.  This is a particularly dangerous metric because of the extreme variability. Again, let’s look at two scenarios.

Scenario 1: Course 1 is a 4-credit course that takes (on average) 45 days to complete. Course 2 is a 4-credit course that takes (on average) 30 days to complete.

a. Course 1 is poorly designed and Course 2 is well-designed.

b. Course 1 is harder than Course 2.

c. Course 1 and Course 2 seem about equal in terms of difficulty and design.

d. None of the above.

Scenario 2: Course 1 is a 4-credit course that takes (on average) 45 days to complete and requires students to turn in 4 papers. Course 2 is a 4-credit course that takes (on average) 30 days to complete and requires students to pass 2 exams.

a. Course 1 is poorly designed and Course 2 is well-designed.

b. Course 1 is harder than Course 2.

c. Course 1 and Course 2 seem about equal in terms of difficulty and design.

d. Students procrastinate more on writing papers than on taking exams.

e. None of the above.

In either case, what does the “time to complete” actually tell us about the quality of learning in the courses? If we were comparing two Calculus I courses, and they were taught with different platforms, equivalent assessment, and the same teacher, I might start to believe that time-to-complete was correlated with design, learning quality, or difficulty. But in most cases, comparing courses via this metric is like comparing apples to monkeys. It’s even worse if that data doesn’t have any kind of context around it.

Number of Clicks per Page

This is one of my favorites. I think you’ll see the problem as soon as you read the scenario.

Scenario 1: Page A got 400 clicks during the semester. Page B got only 29 clicks.

a. Page A has more valuable resources than Page B.

b. Students are accidentally wandering to Page A.

c. Page A is confusing so students visit it to reread it a lot.

d. Page B was only necessary for those students who did not understand a prerequisite concept.

e. Page A is more central in the structure of the course. Students click through it a lot on their way to somewhere else.

Scenario 1: Page A contains a video on finding the derivative using the Chain Rule and got 400 clicks during the semester. Page B contains a narrative on finding the derivative using the power rule and got only 29 clicks during the semester. 

a. Page A has more valuable resources than Page B.

b. Page A is a more difficult topic than Page B, so students revisit it a lot.

c. The video on Page A is confusing so students watch it on multiple occasions trying to figure it out.

d. Page B was only necessary for those students who did not understand a prerequisite concept.

e. Page A is more central in the structure of the course. Students click through it a lot on their way to somewhere else.

Number of clicks per page is meaningless unless there is a constructive relationship between pages. For example, if we are looking at 5 pages that each contain one resource for learning how to find the derivative using the chain rule, the comparison of data might be interesting. But even in this case, I would want to know the order the links appear to the students. And just because a student clicks on a page, it doesn’t mean they learned anything on the page. They might visit the page, decide they dislike the resource, and go find a better one.

Completion of Online Assignments

Surely we can use completion of assignments as a meaningful metric of learning? Surely?

Well, that depends. What do students access when they are working on assignments? Can they use any resource available online? Do they answer questions immediately after reading the corresponding section of the book? Are they really demonstrating learning? Or are they demonstrating the ability to find an answer? Maybe we are just measuring good finding abilities.

Many online homework platforms (no need to name names, it’s like all of them) pride themselves on delivering just-in-time help to students as they struggle (watch this video, look at this slide deck, try another problem just like this one). I think this is a questionable practice. It is important to target the moment of impasse, but too much help means the learning might not stick. Impasse is important because it produces struggle and a bit of frustration, both of which can improve learning outcomes. Perfect delivery of answers at just the right moment might not have strong learning impact because the struggle stops at that moment. I don’t think we know enough about this yet to say one way or another (correct me if you think I’m missing some important research).

Regardless, even completion of assignments is a questionable measure of learning. It’s just a measure of the student’s ability to meet a deadline and complete a task given an infinite number of resources.

Where do we go from here?

Ford hopes that the ramifications of 2016 will foster better journalism in 2017 in ways that people read, watch, or listen to more intentionally, maybe even (shock!) remembering a story and the publisher it came from the next day.

I hope that education can focus more on (shock!) finding meaningful ways to measure whether a student actually learned, not just whether they clicked or checked off tasks. Reflecting on my own online learning experiences in the last year, I am worried. I’m worried we have fallen so deep down the “data-driven decisions” rabbit hole that we are no longer paying attention to the qualitative data that orbits the metrics. Good instructors keep their finger on the pulse of the learners, ever adjusting for those qualitative factors. But as the data ports up to departments, institutions, and vendors, where does that qualitative picture go?

I will close with a few goals for institutions, instructors, and vendors for 2017:

  1. Demand better learning metrics from ed-tech vendors. What that measure is really depends on the platform. Begin asking for what you really want.
  2. Build more integrations that pass quality learning data from the ed-tech vendor to the institution. Sometimes the platform does have better metrics, but the institution can’t access them.
  3. Create metrics that measure learning mastery over time in your own courses. This means choosing a few crucial concepts and probing them repeatedly throughout the learning experience to ensure the concept is sticking.

These are all concepts I hope to continue exploring with more research and more detail over the next year. If you want to join on that journey, consider subscribing here.


Possibly Related Posts:


read more

Measuring Teaching and Learning in Mathematics

Nov 15, 2010 by

This weekend at AMATYC I presented this Prezi presentation on How can we measure teaching and learning in math? My husband was kind enough to act as the videographer for the presentation, and so I can also share the video presentation with you today.

I think the video should add quite a bit of context to the presentation, so I hope you’ll take the time to watch it.  What I propose (at the end) is a research solution that would help all of the math instructors in the country (who want to) participate in one massive data collection and data mining project to determine what actually works to improve learning outcomes.

If you have any suggestions for where to go from here, I’d be happy to hear them.

Possibly Related Posts:


read more

Student Conceptions of Mathematics

Jan 5, 2010 by

sotl_blog_button1

Do you ever get the feeling that you’re not making any real progress with your students?  Sure, they pass tests and progress through the courses (well, most of them), but have you ever just had the uneasy feeling that they really don’t get what math is all about?  Suppose you were to ask your students the following question:

Think about the math that you’ve done so far.  What do you think mathematics is?

What do you think they would tell you?

Well, in 1994, a research group from Australia did ask 300 first-year university students this question (Crawford, Gordon, Nicholas, and Prosser).  They identified patterns in the responses, and classified all 300 responses into categories of conceptions in order to explore the relationships between (a) conceptions of mathematics, (b) approaches to learning mathematics, and (c) achievement.

If you want all the gory details about the study, you’ll have to track down the journal article, but let me attempt to summarize their findings.

Students’ conceptions of mathematics fell into one of five categories (Crawford et al., 1994, p. 335):

  1. Math is numbers, rules, and formulas.
  2. Math is numbers, rules, and formulas which can be applied to solve problems.
  3. Math is a complex logical system; a way of thinking.
  4. Math is a complex logical system which can be used to solve complex problems.
  5. Math is a complex logical system which can be used to solve complex problems and provides new insights used for understanding the world.

The first two categories represent a student view of mathematics that is termed “fragmented” while the last three categories present a more “cohesive” view of mathematics.  Note that the terms fragmented and cohesive are well-used throughout the international body research.  The categories above, as you may have noticed, form a hierarchical list, with each one building on the one above it.

Here’s where this study gets (even more) interesting.  The researchers also asked students about how they studied (changing the language slightly here to “americanize it” a bit):

Think about some math you understood really well. How did you go about studying that?  (It may help you to compare how you studied this with something you feel you didn’t fully understand.) How do you usually go about learning math?

Again, the researchers went through a meticulous process of categorization and came up with five categories (Crawford et al., 1994, p. 337):

  1. Learning by rote memorization, with an intention to reproduce knowledge and procedures.
  2. Learning by doing lots of examples, with an intention to reproduce knowledge and procedures.
  3. Learning by doing lots of examples with an intention of gaining a relational understanding of the theory and concepts.
  4. Learning by doing difficult problems, with an intention of gaining a relational understanding of the entire theory, and seeing its relationship with existing knowledge.
  5. Learning with the intention of gaining a relational understanding of the theory and looking for situations where the theory will apply.

Again, these five categories were grouped, this time according to intention, into two general categories: reproduction and understanding.  In the first two approaches to learning math, students simply try to reproduce the math using rote memorization and by doing lots of examples.  In the last three categories, students do try to understand the math, by doing examples, by doing difficult problems, and by applying theory.  Other researchers in this community have seen similar results on both general surveys of student learning and on subject-specific surveys and have termed this to be surface approach and deep approach to learning (see Marton, 1988).

Still reading?  Good.  Remember my first question? Do you ever get the feeling that you’re not making any real progress with your students? Let’s answer that now.

Here’s how the conceptions of math and the approaches to learning math correlated in this study (Crawford et al., p. 341):

conceptions_of_math

Did you catch that?  Look at how strongly conception and approach correlates.  It’s probably what you’ve always suspected, deep down inside, … but there’s the cold, hard, proof.

Of course, all this is not so meaningful unless there’s a correlation with achievement.  At the end of their first year, the students’ final exam scores were compared to the conceptions and approaches to mathematics (again, for technical details, get the article). The researchers made two statistically significant findings:

  1. Students with a cohesive conception of math tended to achieve at a higher level (p < .05).
  2. Students with a deep approach to learning math tended to achieve at a higher level (p < .01).

Okay, so where does this leave us?  Well, we don’t have causation, only correlation (at least, that’s all we have from the 1994 study).  However, Crawford, Gordon, Nicholas, and Prosser were nice enough to use their research to develop a survey inventory that we can use to measure students’ conceptions of mathematics (1998).  The 19-item inventory (5-point Likert scales) has been thoroughly tested for validity and reliability, and can be found in their 1998 paper, University mathematics students’ conceptions of Mathematics (p. 91).

Suppose you want to try something innovative in your math class, but you don’t know how to tell if it works.  You could, at the very least, try to measure a positive change on your students’ conceptions of math (there are other ways to measure the approach to learning, but this is already a long blog post and you’ll have to either wait for another week, or view my presentation How can we measure teaching and learning in math?).

measure_tlm

To give the Conceptions of Mathematics Questionnaire (CMQ) would take approximately 10 minutes of class time (you should ask for permission from Michael Prosser before you launch into any potentially publishable research).  This would give a baseline of whether students’ conceptions are fragmented or cohesive.   If you were to give the survey again, at the end of the semester, you would be able to see if there is any significant gain in cohesive conceptions (or loss of fragmented conceptions).

So, I have permission (I met Michael Prosser this summer when he was at a conference in the U.S.), and I’m going to use this in all my classes starting next week.  I figure that when I start to see major differences on those pre and post-semester CMQ inventories, that I’m doing something right.  If I’m not seeing any gain in cohesive understanding of mathematics, then I’m going to keep changing my instructional practices until I do.

Papers that you will want to find (and read!):

Crawford, K., Gordon, S., Nicholas, J., & Prosser, M. (1994). Conceptions of Mathematics and how it is learned: The perspectives of students entering University. Learning and Instruction, 4, 331-345.

Crawford, K., Gordon, S., Nicholas, J., & Prosser, M. (1998, March). University mathematics students’ conceptions of Mathematics. Studies in Higher Education, 23, 87-94.

Possibly Related Posts:


read more

How can we measure Teaching and Learning in Math?

Oct 12, 2009 by

Last week I prepared a new presentation for the MichMATYC conference based partially on the literature review for my dissertation.  In my dissertation I am studying instructors, but in this talk I addressed both the instructor and the student side.  It was also the first presentation I’ve built using Prezi, and it was interesting to re-think presentation design using a new tool.  Of course, the presentation misses something without the accompanying verbal descriptions, but there is enough information on here that you can begin to understand the problem (we don’t actually know much) and the solution (common language, common measurement tools).

mtl_developingconceptions

There are also a few new cartoons/illustrations in this presentation.  I’ve started just paying for a couple of illustrations per presentation to help viewers to understand (and mostly to remember) difficult concepts.  Just to give you a rough idea in the time involved to create something like this, I spent about 18 hours on the Prezi build (which doesn’t even begin to account for the time spent doing the research).

How can we Measure Teaching and Learning in Math?
 

Possibly Related Posts:


read more