Learning at Scale Slides from ICTCM

Mar 11, 2017 by

Learning at Scale: Using Research To Improve Learning Practices and Technology for Teaching Math

In the last 5 years, there has been a rise in what we might call “large-scale digital learning experiments.”  These take the form of centralized courses, vendor-created courseware, online homework systems, MOOCs, and free-range learning platforms. If we mine the research, successes, and failures coming out of these experiments, what can we discover about designing better digital learning experiences and technology for the learning of mathematics?


Possibly Related Posts:

read more

Elaborations for Creative Thinking in STEM

Feb 25, 2017 by

As I watch the proliferation of digital learning platforms, particularly in STEM education (where there are lots of objective-type problems), I am excited by the increased focus on learning and adaptivity but also a little uneasy. For the most part the motivations to “go digital” are pure – increase access to courses that students need, provide help that is more tailored to each student, give immediate feedback, provide more practice if the student wants it, and let students move at their own pace.

My worry is that math and science students aren’t getting anything but highly-structured problems. Every problem that a computer delivers is one where there is a carefully constructed set of constraints on the problem and a highly uniform (and thus gradable) answer.

But the problems of the real world aren’t carefully constrained and the problem solutions aren’t highly uniform. If we only teach students to solve the “lives in an orderly box” kind of problems, what are we preparing them for? Are we creating experiences that lead to curiosity in STEM subjects? Do students even see subjects like math as highly connected networks of concepts or just as discrete concepts to be learned one-at-a-time inside a digital system?

What is a Problem Elaboration?

This semester I’ve fallen back on a very old assessment technique I developed about 15 years ago that I call a “Problem Elaboration.” Students have to turn in a two problems per topic/section where they complete the problem (the old-fashioned way on paper) and then do some kind of mathematical elaboration on the problem. In other words, they have to do something that wasn’t asked for. They have to think in the space around the problem and consider:

  • What else could I find?
  • What other mathematical things could I do with this?
  • How does this connect to other things we’ve learned?
  • How could I check this to make sure it’s right?

It’s fascinating to watch the initial struggle of many of the students with the sudden freedom of elaborations. Students fall into three categories in the beginning of the term, but every student that makes an honest effort moves up this ladder to increasingly creative and complex elaborations over time.

Trepidatious: How am I supposed to find something that I’m not asked to find? I don’t even know how to begin! This is crazy! Your job is to support these students with examples, suggestions, and a “you can do it” attitude. These students will eventually try, and when they do, encourage them!

Compliant: If there is some way to check the problem, I’ll do that. If I wasn’t asked to graph the problem I’ll do that. These are the not the most creative elaborations. But it is an exploration that contributes to mathematical maturity. This tends to end up being what students do when they are short on time.

Elaboration 1: Notice the actual problem is highlighted in a lighter color. The elaboration is to check the answer. It’s not particularly glamorous, but the student knows how to check the answer, she knows it’s right, and that’s pretty awesome.


Elaboration 2: This also has the standard “check your answer” type of elaboration, but it goes one step further. This student shows me that she knows how to GRAPH both sides of the equation and verify that she has the right solution!

Curious:  What happens to this graph if I change this number or this sign? I did “X” and was expecting “Y”. Why didn’t that work? How does this connect to what we did yesterday? How would I find this thing I don’t know how to find?

Elaboration 3: Now we see an elaboration that really begins to explore the “what happens if” space. What happens if this is a 5th root instead of a square root? What happens if there is a number multiplied by the square root in the original equation?


Elaboration 4: This students starts by checking the answer, noting where they made a mistake the first time, and redoing the check. Then the student explores whether they can square individual terms in the equation instead of isolating the square root first and get the same answer. See the question the student is left with? “Not sure if that means something or not.” That’s the opener to a conversation in pen pal form about the right way that squaring both sides actually works.

As I grade these assignments, I find that what I’m really doing is opening up a conversation with the student. The elaborations aren’t always right, and often the students are asking questions in the elaborations. This is our chance to explore the math together in a back-and-forth letter to each other, assignment after assignment. Over the course of a semester, I watch many students develop mathematical maturity, mathematical confidence, and ownership of the math they have learned.


I assign 2 problems to turn in with elaboration per topic, using a particular grading rubric for these problems, based on 5 points:

  • 1 point for rephrasing or rewriting the problem (students end up solving a slightly miscopied problem, or don’t catch all the parts of the problem)
  • 2 points for showing all appropriate work to solve the problem
  • 1 point for actually finding the correct answer
  • 1 point for the elaboration

I also award a bonus point for really great and thoughtful elaborations (at discretion of instructor). Typically students earn between 0 and 5 bonus points per unit depending on their effort and thoughtfulness.

If a student makes an honest and thoughtful attempt at an elaboration, they will get the point even if their elaboration ends up being wrong or using incorrect terminology. The point is that the student explores and tries. My role is to correct improper terminology and to help when their reasoning wanders into an area where it no longer holds.

At the beginning of the semester, I do provide some idea of what an elaboration might be. A single elaboration might be any one of the following, but this is not an all inclusive list of what could be done.

  • Show a different way to solve the problem.
  • Show how to check the answer.
  • Solve for something extra in the problem.
  • Relate the concepts in this problem to something else we’ve studied.
  • Relate the concept in the problem to a graph (if there wasn’t a graph required in the problem).
  • Hypothesize on how a change to the problem might change the answer, and then try it.
  • If it relates to this problem, investigate something we did in class that you found difficult to understand or remember.

Why the unease?

Online learning and learning through digital platforms is highly structured and scripted. Students learn exactly what they are scripted to learn. They only “explore” in the sense that we sometimes allow them to choose the next highly-scripted chunk of learning they engage with.

How do we build digital learning spaces where something like elaborations and this personal conversation between myself and the student about the mathematics can also be encouraged as a part of learning?

And if we don’t encourage our most engaged and curious students to go deeper, to question more, and to make more connections, what’s going to happen to our STEM pipelines?

Possibly Related Posts:

read more

Better to be Frustrated than Bored

Jan 23, 2017 by

For all of you who have taught students, you know that one of the rewards is seeing the “Aha moments” that students experience. One of the downsides (to instructors) of teaching online is that it is hard to “see” the reward of the “aha” in the same fulfilling way.

The reason we should care about the aha moments that students experience is that this kind of moment is tied closely to an emotional feeling. And memories with emotional attachments tend to be stronger (more memorable) than ones with no emotional attachment.

But have you ever stopped to think about what is going on right before the aha moment? Does the aha moment come after a well-organized lecture, a step-by-step example, a period of boredom, or a period of confusion or frustration?

An aha moment typically comes after a period of confusion or frustration. This means that you have to put students into that space where they are actually at the edge of what they know (the confusion/frustration space) to nudge them over to insights.

Anyways, I digress. I want to share findings from this paper: Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments (Baker, et al, 2010).

The researchers set out to focus on cognitive-affective states that were hypothesized to influence cognition and deep learning: boredom, confusion, delight, engaged concentration, frustration, and surprise. The researchers use Russell’s Core Affect framework (2003) to map these states in two dimensions: valence (pleasure to displeasure) and arousal (activation to deactivation).

In this study, the researchers examined:

  • the cognitive-affective states the students experienced during the learning process
  • how those states persist over time (e.g. do students move from boredom to frustration more often than frustration to boredom?)
  • how the state affects the students choices on how to interact with the system (e.g. what causes students to game the system?)

While I will leave you to read the whole paper if you want all the details (the methodology involves three different interactive learning systems and three different methodologies), I think the nuance of definitions between a few of these terms is important. As defined in the paper:

  • frustration is dissatisfaction or annoyance
  • confusion is a noticeable lack of understanding
  • engaged concentration is a state of engagement with a task such that concentration is intense, attention is focused, and involvement is complete

Now let’s jump ahead to (what I consider to be) some of the interesting results. Engaged concentration was the most common state during the observation periods (60%) followed by confusion (13%). While boredom was only observed about 4-6% of the time, it was also the most persistent state (once bored, the student stays bored) across all three learning systems.

Within two of the systems where “gaming the system” was observed, a more in-depth analysis was performed. Boredom was significantly more likely to lead to gaming the system. Guess what wasn’t likely to lead to gaming the system … confusion, frustration, and surprise. Better to be confused than bored, huh?

There is quite a bit of new research being performed on the role of confusion in learning, but my gut feeling here is that confusion leads to self-insight, and learning gained through self-insight (because this is the aha where emotions are attached) should be stickier than learning delivered through other states.

Challenge: Vigilantly watch for states of boredom in your classes, and when you find them, intervene. Do something different. Put students into a space where they are challenged and maybe even a little confused. Give the learners a chance to grapple with the concepts and have those moments of self-insight.


Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223-241.

Possibly Related Posts:

read more

Celebrate the Errors in Math Practice

Jan 11, 2017 by

Dear math students,

As you work through your mathematics practice, I’m going to challenge you to embrace making errors in an entirely new way. 

Many students believe that every problem in math homework should be perfectly constructed with no errors. It might look something like this:

A nicely ordered problem solution with no mistakes.


But when it’s time to study after the initial problem run-through, what does this perfectly constructed problem say? Does it coach you on remembering how you struggled? Does it remind you where you made an error? No.

When you make an error as you’re working a problem, please don’t erase it from the face of the earth. Certainly you should learn from the struggle and complete a correct solution, but record your deviations from the straightforward solution path in another color. Leave yourself notes (also in a different color) to remind you what you should have paid more attention to the first time around. Maybe that would look something like this:

Worked problem with highlights and notes to self



Sometimes you’re going to recognize but you don’t have the right answer but you’re not going to be sure what’s gone wrong. You should always try first to figure it out yourself first. This process of error analysis in a variety of different situations is key to developing problem solving skills in mathematics. Without exploration of the problem space (which happens with error analysis), your brain is just recording rote procedures without the ability to transfer those procedures to new kinds of problems. It will, essentially, stunt your mathematical growth.

Now, I don’t want you to get to the point of tears or breaking your keyboard out of anger. If you get near that stage please just ask a question (email, discussions, chat-a-friend, etc), leave a sticky note on the page as a reminder to go back, and move on to the next problem or section of problems. Just switching to a slightly different problem can not only get you unstuck, but sometimes give you insights into the “stuck” problem. 

When you figure out how to do the problem you were stuck on, make sure to go find that flagged problem (remember the sticky note?) and annotate your corrections.

Worked math problem with cross-outs and restarts and notes to self.


Now you might be thinking “why do all these error corrections and problem annotating in another color?” When it comes time to study for your major assessments, you will be able to see the places where you stumbled the first time you tried the problem. These “notes to self” are the places where you’re most likely to make the same mistake again. They benchmark places to remember to be careful and show you problem types to repeat practice before an exam.

Celebrate your errors.

Embrace the messiness.

Learn from your mistakes.

Study from your struggle points.

And be great at mathematics!

Possibly Related Posts:

read more

The Deliberate Practice Experiment

Jan 7, 2017 by

For some inspiration to try a more active approach to learning, I thought I would share a short video from Dr. Carl Wieman, who is a Nobel Laureate, a physicist, and more recently a researcher in learning science. Wieman has designed and run some very elegant experiments to demonstrate the effects of active learning techniques. Consider this one:

What happens if you run two classes for a week with the same learning objectives, the same time in class, and the same assessment. Prior to the experiment, you take care to ensure that the student makeup and performance is very similar up to this point. One class continues on, lecture-style, with a veteran instructor with lots of experience. The other class is assigned a freshly-minted PhD trained to lead the students in learning via deliberate practice (an active learning strategy). The courses meet under these conditions for the exact same time.

The technique of deliberate practice takes the form of a series of challenging questions and tasks. This forces students to spend their class time thinking scientifically, discussing concepts, critiquing each other’s predictions, and engaging in problem solving.

To see how this (and a few other elegant experiments) turn out, take the 15 minutes to watch the video: Finding New Ways to Learn Science.

Really … watch the video. Student pushback against these strategies is a real issue and usually worst at the beginning of the semester. We can all use a good foundational reminder about why we continue to strive to lead more active learning sessions in our classes.

It is also well worth the time to read the very well-designed study, Improved learning in a large-enrollment physics class, mentioned in the video.

Challenge: Consider the deliberate practice model described above. Where can you tackle a learning objective through a series of challenging questions and tasks instead of with lecture? Give yourself that extra nudge to try just a little more activity (or to build in just a little more activity).


Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science, 332(6031), 862-864.

Reminder: You can sign up to receive the Weekly Teaching Challenge in your email or share it with a friend.

Possibly Related Posts:

read more