Better to be Frustrated than Bored

Jan 23, 2017 by

For all of you who have taught students, you know that one of the rewards is seeing the “Aha moments” that students experience. One of the downsides (to instructors) of teaching online is that it is hard to “see” the reward of the “aha” in the same fulfilling way.

The reason we should care about the aha moments that students experience is that this kind of moment is tied closely to an emotional feeling. And memories with emotional attachments tend to be stronger (more memorable) than ones with no emotional attachment.

But have you ever stopped to think about what is going on right before the aha moment? Does the aha moment come after a well-organized lecture, a step-by-step example, a period of boredom, or a period of confusion or frustration?

An aha moment typically comes after a period of confusion or frustration. This means that you have to put students into that space where they are actually at the edge of what they know (the confusion/frustration space) to nudge them over to insights.

Anyways, I digress. I want to share findings from this paper: Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments (Baker, et al, 2010).

The researchers set out to focus on cognitive-affective states that were hypothesized to influence cognition and deep learning: boredom, confusion, delight, engaged concentration, frustration, and surprise. The researchers use Russell’s Core Affect framework (2003) to map these states in two dimensions: valence (pleasure to displeasure) and arousal (activation to deactivation).

In this study, the researchers examined:

  • the cognitive-affective states the students experienced during the learning process
  • how those states persist over time (e.g. do students move from boredom to frustration more often than frustration to boredom?)
  • how the state affects the students choices on how to interact with the system (e.g. what causes students to game the system?)

While I will leave you to read the whole paper if you want all the details (the methodology involves three different interactive learning systems and three different methodologies), I think the nuance of definitions between a few of these terms is important. As defined in the paper:

  • frustration is dissatisfaction or annoyance
  • confusion is a noticeable lack of understanding
  • engaged concentration is a state of engagement with a task such that concentration is intense, attention is focused, and involvement is complete

Now let’s jump ahead to (what I consider to be) some of the interesting results. Engaged concentration was the most common state during the observation periods (60%) followed by confusion (13%). While boredom was only observed about 4-6% of the time, it was also the most persistent state (once bored, the student stays bored) across all three learning systems.

Within two of the systems where “gaming the system” was observed, a more in depth analysis was performed. Boredom was significantly more likely to lead to gaming the system. Guess what wasn’t likely to lead to gaming the system … confusion, frustration, and surprise. Better to be confused than bored, huh?

There is quite a bit of new research being performed on the role of confusion in learning, but my gut feeling here is that confusion leads to self-insight, and learning gained through self-insight (because this is the aha where emotions are attached) should be stickier than learning delivered through other states.

Challenge: Vigilantly watch for states of boredom in your classes, and when you find them, intervene. Do something different. Put students into a space where they are challenged and maybe even a little confused. Give the learners a chance to grapple with the concepts and have those moments of self-insight.

Reference:

Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223-241.

Possibly Related Posts:


read more

Celebrate the Errors in Math Practice

Jan 11, 2017 by

Dear math students,

As you work through your mathematics practice, I’m going to challenge you to embrace making errors in an entirely new way. 

Many students believe that every problem in math homework should be perfectly constructed with no errors. It might look something like this:

A nicely ordered problem solution with no mistakes.

 

But when it’s time to study after the initial problem run-through, what does this perfectly constructed problem say? Does it coach you on remembering how you struggled? Does it remind you where you made an error? No.

When you make an error as you’re working a problem, please don’t erase it from the face of the earth. Certainly you should learn from the struggle and complete a correct solution, but record your deviations from the straightforward solution path in another color. Leave yourself notes (also in a different color) to remind you what you should have paid more attention to the first time around. Maybe that would look something like this:

Worked problem with highlights and notes to self

 

 

Sometimes you’re going to recognize but you don’t have the right answer but you’re not going to be sure what’s gone wrong. You should always try first to figure it out yourself first. This process of error analysis in a variety of different situations is key to developing problem solving skills in mathematics. Without exploration of the problem space (which happens with error analysis), your brain is just recording rote procedures without the ability to transfer those procedures to new kinds of problems. It will, essentially, stunt your mathematical growth.

Now, I don’t want you to get to the point of tears or breaking your keyboard out of anger. If you get near that stage please just ask a question (email, discussions, chat-a-friend, etc), leave a sticky note on the page as a reminder to go back, and move on to the next problem or section of problems. Just switching to a slightly different problem can not only get you unstuck, but sometimes give you insights into the “stuck” problem. 

When you figure out how to do the problem you were stuck on, make sure to go find that flagged problem (remember the sticky note?) and annotate your corrections.

Worked math problem with cross-outs and restarts and notes to self.

 

Now you might be thinking “why do all these error corrections and problem annotating in another color?” When it comes time to study for your major assessments, you will be able to see the places where you stumbled the first time you tried the problem. These “notes to self” are the places where you’re most likely to make the same mistake again. They benchmark places to remember to be careful and show you problem types to repeat practice before an exam.

Celebrate your errors.

Embrace the messiness.

Learn from your mistakes.

Study from your struggle points.

And be great at mathematics!

Possibly Related Posts:


read more

The Deliberate Practice Experiment

Jan 7, 2017 by

For some inspiration to try a more active approach to learning, I thought I would share a short video from Dr. Carl Wieman, who is a Nobel Laureate, a physicist, and more recently a researcher in learning science. Wieman has designed and run some very elegant experiments to demonstrate the effects of active learning techniques. Consider this one:

What happens if you run two classes for a week with the same learning objectives, the same time in class, and the same assessment. Prior to the experiment, you take care to ensure that the student makeup and performance is very similar up to this point. One class continues on, lecture-style, with a veteran instructor with lots of experience. The other class is assigned a freshly-minted PhD trained to lead the students in learning via deliberate practice (an active learning strategy). The courses meet under these conditions for the exact same time.

The technique of deliberate practice takes the form of a series of challenging questions and tasks. This forces students to spend their class time thinking scientifically, discussing concepts, critiquing each other’s predictions, and engaging in problem solving.

To see how this (and a few other elegant experiments) turn out, take the 15 minutes to watch the video: Finding New Ways to Learn Science.

Really … watch the video. Student pushback against these strategies is a real issue and usually worst at the beginning of the semester. We can all use a good foundational reminder about why we continue to strive to lead more active learning sessions in our classes.

It is also well worth the time to read the very well-designed study, Improved learning in a large-enrollment physics class, mentioned in the video.

Challenge: Consider the deliberate practice model described above. Where can you tackle a learning objective through a series of challenging questions and tasks instead of with lecture? Give yourself that extra nudge to try just a little more activity (or to build in just a little more activity).

Reference:

Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science, 332(6031), 862-864.

Reminder: You can sign up to receive the Weekly Teaching Challenge in your email or share it with a friend.

Possibly Related Posts:


read more

Write an Operating System for Your Brain

Jan 1, 2017 by

Let’s face it. We all have to interact with organizations and systems of relationships in which we might not “fit” perfectly. We can choose to stay and try to thrive, or leave and find another path. Both are valid options. If you are choosing to stay, you will likely have to make minor (or sometimes major) tweaks to how you interact and function within that system. It is unreasonable to expect a system with thousands of employees or a family with 4 people in it to all change and adapt to you when you are unwilling to make changes to adapt to it.

What if you began approaching your brain like a computer system? Suppose you could load in a set of operating instructions at the beginning of every day to govern how you operate in the world around you. As you find “bugs” (character flaws) in your system, suppose you could write new code to help you begin to correct those bugs. I think we can actually do this.

Computers boot up with an operating system that tells the hardware (printer, mouse, monitor, etc.) how to function and runs the software apps. The operating system is the first program the computer loads when you turn it on. In a sense, the operating system directs everything else in the computer system.

You can write a personal operating system (POS) that directs your physical and mental functioning. The POS can be a very powerful tool to help you change behavior that is holding you back in your professional and personal life.

Here’s the basic idea:

  1. Write a set of instructions (the POS) that you want to be your governing principles.
  2. Read the POS every morning out loud.
  3. Each evening before bed, celebrate the wins from the day as you reread the POS.

Aspirational Lines of Code

Since this is you talking to your own body and brain, it’s important that you are kind to yourself. Negative instructions tend to focus on past behavior. The personal operating system should be a set of instructions on what TO do in the present and future. The instructions you give yourself every day should be affirming and aspirational. Any instruction can be written in a negative or positive way.

Here’s a line of code for running your brain:

Negative code: Don’t brag.
Positive code: Be humble.

And another line of code for running your body:

Negative code: Don’t eat crap.
Positive code: Choose healthy food.

A personal operating system should consist of approximately 5-15 lines of code that set up instructions for your brain and body. Here are several lines of code from the personal operating system I used last year, when I worked with a very large team in a leadership position:

Schedule impulsive things.
Support the leaders around you.
Focus on the big deliverables.
Give everyone a chance to shine.
Communicate down as well as up.
Do something scary every day.

The goal for a personal operating system is to create a set of reminders that become so well-known to you that they pop up in your mind when you are confronted with situations that run counter to your operating system.

Writing the First Draft of your POS

How do you go about creating your own personal operating system?

There are several questions you can use for reflection that will give you a good first draft:

  1. If I were to ask people around you to describe you, what would you want them to say about you?
  2. What are the characteristics of the people you you admire and look to as role models?
  3. What are the character flaws that are keeping you from moving up or into new roles at work?
  4. What are the character flaws that are keeping you from having more meaningful relationships with your partner, children, or friends?

As you identify flaws, remember that you need to find ways to write affirming statements (not negative ones) for the operating system.

Getting Feedback on the POS

This may be too scary for your first POS, and if that’s the case, just skip this step. However, it can be a really valuable way to nail the instructions that will be the most powerful in transforming yourself.

To get feedback on your draft POS, you want to identify 2-5 trusted people (friends, family members, or former work colleagues) that will read the POS and tell you what you’re missing. Ask them questions like:

  1. What is holding me back at work?
  2. What is holding me back from deepening our relationship?
  3.  What should be on this list that isn’t?

Approach the feedback you get with this mindset: I will listen to what they say without argument.

Now revise your POS in a way that you think is appropriate. You can take it or leave it with respect to the feedback you got, but you did ask for it, so I’d consider taking it if you truly asked people you trusted.

Finalize the POS

As you get the POS ready to be used, consider how it should be written on paper. Do you want your POS to be loaded from shortest statement to longest statement?

Be humble.
Choose healthy foods.
Schedule impulsive things.
Focus on the big deliverables.
Do something scary every day.
Support the leaders around you.
Give everyone a chance to shine.
Communicate down as well as up.

Does it make more sense to you to group the POS so that personal and work goals are separated?

Be humble.
Choose healthy foods.
Do something scary every day.

Schedule impulsive things.
Focus on the big deliverables.
Support the leaders around you.
Give everyone a chance to shine.
Communicate down as well as up.

Or maybe it makes more sense to you to organize the list so that the hardest instructions always come first.

Be humble.
Choose healthy foods.
Focus on the big deliverables.
Schedule impulsive things.

Do something scary every day.
Support the leaders around you.
Give everyone a chance to shine.
Communicate down as well as up.

Finalize the order. Print the list (you may need a few copies depending on your morning and evening routines). You might want to laminate the list (packing tape works well), mount it on cardboard, decorate it, or do something else to make it feel more like a permanent structure in your life. It’s your POS, figure out what makes sense to you.

Loading the POS in the Morning

The POS will only work if you actually READ it every morning. So you have to find a place to put it where you will have the time to read the list out loud with minimal disruption to your normal routine. Try to think of a time when you’ll be likely to be able to read the list. For me, the ideal time is when I’m drying my hair, so my list is in the bathroom. For you it could be while you wait for your coffee to brew, when you are making your breakfast, or while you drive to work.

Find a home for your POS, and make it a part of your routine to read the list of instructions every morning.

Reread the POS in the Evening

Now find a time in the evening that you can skim your list and celebrate the wins and small corrections you made during the day. Again, the bathroom might be the right place (I reflect while I brush my teeth). You don’t want to beat yourself up if you didn’t manage to execute all your instructions. The goal is to find where an instruction did actually lead you to a new behavior during the day. You want your mind to begin to deliver the correct instruction at the moment you need to hear it. For example, at the moment that you are about to brag about an accomplishment at work, “Be humble.” floats into your mind and you congratulate the team on a job well done instead of bragging. That’s a win to be celebrated at the end of the day.

Of course, you might also identify moments in the day where you could have done better, and that’s okay. You simply need to acknowledge them, and then dismiss them with the thought that you’ve now recognized these types of moments and can improve another day.

Tuning Up the POS

As time passes and jobs and relationships change, your POS will need tuning up. Don’t operate on an old POS when your life has had significant changes. The example POS I’ve shared makes little sense now that I’m self-employed and no longer working with an enormous organization with a large team to manage. I’ve revised it. You should too as your situation changes.

I would love to compile a list of instructions that people can choose from to write a POS. What are some of the “lines of code” you’re writing into your own personal operating system?

Possibly Related Posts:


read more

To Focus on Learning, Use Words Not Section Numbers

Dec 30, 2016 by

Since many faculty are now approaching that time when syllabi get written and course shells get built, let’s focus on a very simple change that can increase student attention to learning the vocabulary and learning objectives/goals associated with the subject.

Take examine your syllabus, schedule, and course shell looking for places where you could have written the name of the course, the topics, or the learning objectives in words, but didn’t do it. When schedules and other course materials are passed out without any reference to the vocabulary of the subject, you are missing a great chance to put that vocabulary and learning objectives/goals front and center for your learners.

Consider this real course schedule I saw recently (altered a bit to protect the identity of the course and college).

This schedule contains no course name, no topics in words, and no learning goals. My course schedules used to look like this too (although they did at least have the name of the course at the top). But then I took a deep dive into how we learn vocabulary a couple years ago. Now I use every opportunity to engage students with the words that describe the topic and the learning goals.

This schedule was a golden opportunity for introducing (repeating) vocabulary and focusing on learning objective … and it was missed. Let’s look at a example of the same schedule with vocabulary and focus on learning objectives.

Now every day on the schedule has context to it, and the student is reminded of the language associated with the topic on every viewing of the schedule. If the student is making a judgement call about whether to skip class (face it, they do) or how long to procrastinate on doing their assigned homework, the context of what is in the schedule is very helpful to them. If they recognize no words in the topic, or recall that they have never figured it out in the past (dev math), then they have additional incentive to show up, leave more time to complete, etc.

The reality is that if we want students to actually learn the vocabulary of our subject area, they have to encounter the vocabulary as often as possible. They have to read the vocabulary like they will see it in context, and do that as much as possible. This repeated encounter helps them to begin to see the contrast between important differences. In the example above, there is a difference between solving equations and solving inequalities. The repeated exposure to those as two separate topics helps the student to begin to separate “equations” and “inequalities” as two distinct topics with rules associated with the overlap and contrast between them.

Working with the vocabulary and learning objectives is also helpful to faculty. Instructors need to see more than a section number as they plan out the learning for the semester. We’ve all encountered texts where one section is not equal to one hour of class. Writing out the learning topics / objectives gives the instructor a chance to reflect on where there needs to be supplementary learning activities and where the topic is actually too “lite” for a whole day of class. By planning for this natural fluctuation up front (assuming one has experience teaching the course on a previous occasion), it is more likely that the pacing of the course will match the real learning of the students.

Challenge: Take a fresh look at your syllabus, schedules and/or course shell. Find opportunities to place vocabulary and learning objectives/goals in more prominent locations.

Note: A weekly bite of learning design and a challenge goes out every week. If you’d like to have it delivered to your inbox, sign up at Weekly Teaching Challenge.

Possibly Related Posts:


read more

The Four Processors: A Neogeneralist Problem?

Dec 22, 2016 by

My husband has been joking for years that my brain has 4 processors. He says I can’t relax unless at least 3 processors are turned off, but that’s not an easy thing to do. When one processor can’t solve the problem it is stuck on, my brain brings up the next-most-interesting processor and begins working on that problem instead.

Often, I wake up at 5am to find that one processor has kicked out a solution to a problem and then it is impossible to go back to sleep. Sometimes I can’t sleep because the processors are occupied with such interesting problems they won’t turn off.

The interesting part is what happens if I do not have enough interesting topic matter to supply to the 4 processors. They don’t seem to redistribute existing problems and work in parallel. The other processors are uninterested if one processor is handling the problem. Rather the remaining processors become unsatisfied and depressed – they feel under-appreciated and unfulfilled.

Recently, I read The Neogeneralist, by Kenneth Mikkelsen and Richard Martin. Neogeneralist “are people who are able to find connections across fields, to continuously learn specialties and apply their learnings across disciplines.” As I try to figure out how I want to be spending my time (for neogeneralists, where you go is what you do), I’m starting to frame this problem as a 4-processor issue. Suppose I can (have to) focus on four major areas/problems/realms.

As Director of Learning Design at WGU, there were four major realms my brain worked on:

  1. Instructional Design
  2. Learning Technology Design
  3. Infrastructure/Organization Design
  4. Process Design

While these were all worthy problems, I felt increasingly locked into the job in a way that made me feel as though I was suffocating. None of the problems gave me the chance to interact directly with learners or faculty (which is odd, I know).  In this position, I was solving problems every day, but so far removed from “users” (students) that I couldn’t ever see whether the solutions would bear fruit, even at a small scale. I am used to having a “petri dish” to experiment in (a MOOC to teach, a course at a college). While my job originally encompassed “Improve Learning at WGU,” there became organizational and technological barriers to actually improving learning from this position as the organization grew and reorganized.

The other thing (but pretty normal at a growing organization) that happened at WGU was that we began to carve out some of these major realms of focus to other parts of the organization. Theoretically, this would be great because as an employee I’d have more time to focus, but here’s the reality of what happened:

  1. Processor 1 is unfulfilled and sad
  2. Learning Technology Design
  3. Processor 3 is unfulfilled and sad
  4. Processor 4 is unfulfilled and sad

Normally, this is where I would find some other project to work on in my free time to keep a processor or two occupied, but WGU had a very strong “no outside work” policy. There was a startup I wanted to spend time pursuing, but this would be “outside work.” I could teach as an adjunct somewhere, but this would be “outside work.” On top of that, Processor 3 became completely occupied/obsessed with a fundamental problem of the design of the organization and could not find any way out of the problem (and also lacked a position with any possible influence over this issue).

  1. Processor 1 is unfulfilled and sad
  2. Processor 2 is trying to figure out what job actually is
  3. Processor 3 is trying to solve organizational design and is stuck in an infinite loop
  4. Processor 4 is unfulfilled and sad, starts to catastrophize about T presidency

As organizations grow, they necessarily want employees to be more focused and less of generalists, but that is a problem if you happen to be a neogeneralist. Our unique abilities come from being experts in several areas and using cross-pollination of those areas to solve problems. If you’re going to shut down our ability to focus on multiple areas at work, don’t also shut down our ability to seek out new and interesting problems in our free time. It’s going to end in disaster.

Self-employment can also be rough, because if your processors aren’t fully occupied, then at least one of them will spend time in panic mode about not finding enough work. For the first month, the processors were occupied like this:

  1. Startup
  2. Trying to find clients to sustain income
  3. Trying to find long-term plans that would still work if 1 & 2 fail
  4. Unfulfilled and sad

Now onto the new problem. If I intentionally design my life, what do I really want the processors to be occupied with? Are they all fulfilled? Can I convince cofounders, investors, etc that I am simply not going to be happy with extreme focus? Probably most people keep at least one processor completely focused on their children, but I don’t have childeren. So I really do have 4 to allocate.

I also have a pretty strong history of allocating 4 processors. Here’s the processor allocation for my undergraduate work (I got 3 degrees while working close to 30 hours a week):

  1. Math
  2. Chemistry
  3. Biology
  4. Improving learning (TA, tutor, grading papers – payed the bills)

Here’s the processor allocation for my Masters graduate work (2 degrees, plus working):

  1. Math
  2. Business
  3. Improving learning (TA, adjunct for another college, doing learning research)
  4. Math curriculum work (writing texts, learning the assessment world, curriculum production)

Here’s the processor allocation for my PhD work (1 degree, full time faculty, consulting on the side). I still feel sorry for my PhD advisor for having to “manage” me. There were two years where “PhD” basically got bumped out when something in category 3 became a full-time processor job for a time.

  1. Teaching for MCC (5-5-2 load, typically 2-3 preps a term)
  2. Improving learning (experimenting, reading, writing, speaking)
  3. Math curriculum work (writing curriculum, building paper and digital math games)
  4. PhD Higher Ed Leadership

So what am I doing now?

  1. Startup (curriculum / learning design)
  2. Client work (product management / UX / LX, software) – also pays the bills 🙂
  3. Improving learning (teaching, reading, writing, speaking)
  4. Math curriculum work – also pays the bills and an area where I have a lot of time invested

As the startup grows, it will likely have to occupy more than one processor, and then the hard part – what goes? I don’t think I will be happy if I “improving learning” is not a category. It seems to be a consistent theme throughout the last two decades. My brain finds “Improving Learning” to be a fun distraction – what others might consider something akin to a “hobby” (though to most people, it is a job not a hobby).

I have strategies for managing conflicting priorities (good ones, in fact). I’ve always had strategies that keep my brain balanced.

If you’re still reading, wow. This is really an attempt to work through my own “post mortem” of why I felt so compelled to leave WGU a few months ago. There were a few other reasons (loss of authority, changes in leadership, nasty colleague) that probably just pushed me over the edge of the cliff, but not worth sharing the details – those things happen and there wasn’t anything I could do to change things. If I had been fundamentally happy, I would have stayed.

I’m trying to do a more intentional “Life Design” now, which I’ve come to realize is possibly different than what we traditionally consider “work life” balance. Yes, I should get exercise every day (I do). Yes, I should spend time with friends and family (I do). But those are necessary conditions, not sufficient ones. Sufficient, for me, means having a brain that is fully occupied solving problems that makes it happy.

Possibly Related Posts:


read more

Clickety Click Click: Awful Measures for Learning

Dec 19, 2016 by

I feel a little inspired by Sam Ford’s post The Year We Talk About Our Awful Metrics. Ford writes about the need for change in metrics of online media publications, but we could just as easily be discussing the metrics of learning management systems, ed-tech vendor platforms, and institutional analytics.

Ford argues that we need to “get serious” about better forms of measurement in 2017. As long as we are measuring metrics with little meaning, we aren’t really improving learning.

Let me give you a few examples to illustrate the similar problems in education.

Page Clicks

As in, how many pages of the ebook has the student accessed? Because the student must read every page they access, right? And they don’t just scroll through pages to see roughly what the book has in it? Realistically, I think we all acknowledge these inevitabilities, but that doesn’t stop us from creating blingy dashboards to display our metric wares.

Consider the following scenarios.

Scenario 1: Student A has clicked on 55 pages whereas student B has only clicked on 10 pages. This means:

a. Student A has read more than Student B. Student A is a more engaged student.

b. Student B was reading deeply and Student A was skimming.

c. Student A reads faster than student B.

d. Student A read more online. Student B borrowed a book from a friend and read more on paper.

e. None of the above. Who knows what it really means.

Scenario 2: Student A has clicked on 55 pages whereas student B has only clicked on 10 pages. Both students spent 2 hours in the eReader platform.

a. Student A has read more than Student B. Student A is a more engaged student.

b. Student B was reading deeply and Student A was skimming.

c. Student A reads faster than student B.

d. Student A read more online. Student B borrowed a book from a friend and read more on paper.

e. None of the above. Who knows what it really means.

In either case, how much do we really know about how much Students A and B have learned? Nothing. We know absolutely nothing. These metrics haven’t done a thing to see what either student is capable of recalling or retrieving from memory. There is nothing to help us to see whether the student can make sensical decisions related to the topics and nothing to show whether concepts can be transferred to new situations. Page clicks are a bad metric. All they tell me is that students log in more every Sunday night than on any other night (and that metric has been the same for a decade now).

But wait … there are more metrics …

Attendance

We can measure attendance – whether it be logging in to the LMS or physically showing up in the classroom. Surely this is a valuable measure of learning?

Again no, it’s not a measure of learning. It’s potentially a necessary condition of a necessary-but-not-sufficient metric for learning. Yes, we do need students to show up in some way to learn. In very active face-to-face classrooms that engage all students in learning activities, I might go so far as to say that showing up is a good measure of learning, but this is still the exception rather than the norm. And even if the classroom is active, learning is more effective with certain kinds of activities: those involving interaction, those involving varied practicethose where students have to learn to recognize and remedy their own errors.

Attendance, by itself, does not measure learning.

Time to Complete

At organizations where the learning is assessed directly (CBE and MOOCs, for example), there is often a metric around the “time to complete” a course.  This is a particularly dangerous metric because of the extreme variability. Again, let’s look at two scenarios.

Scenario 1: Course 1 is a 4-credit course that takes (on average) 45 days to complete. Course 2 is a 4-credit course that takes (on average) 30 days to complete.

a. Course 1 is poorly designed and Course 2 is well-designed.

b. Course 1 is harder than Course 2.

c. Course 1 and Course 2 seem about equal in terms of difficulty and design.

d. None of the above.

Scenario 2: Course 1 is a 4-credit course that takes (on average) 45 days to complete and requires students to turn in 4 papers. Course 2 is a 4-credit course that takes (on average) 30 days to complete and requires students to pass 2 exams.

a. Course 1 is poorly designed and Course 2 is well-designed.

b. Course 1 is harder than Course 2.

c. Course 1 and Course 2 seem about equal in terms of difficulty and design.

d. Students procrastinate more on writing papers than on taking exams.

e. None of the above.

In either case, what does the “time to complete” actually tell us about the quality of learning in the courses? If we were comparing two Calculus I courses, and they were taught with different platforms, equivalent assessment, and the same teacher, I might start to believe that time-to-complete was correlated with design, learning quality, or difficulty. But in most cases, comparing courses via this metric is like comparing apples to monkeys. It’s even worse if that data doesn’t have any kind of context around it.

Number of Clicks per Page

This is one of my favorites. I think you’ll see the problem as soon as you read the scenario.

Scenario 1: Page A got 400 clicks during the semester. Page B got only 29 clicks.

a. Page A has more valuable resources than Page B.

b. Students are accidentally wandering to Page A.

c. Page A is confusing so students visit it to reread it a lot.

d. Page B was only necessary for those students who did not understand a prerequisite concept.

e. Page A is more central in the structure of the course. Students click through it a lot on their way to somewhere else.

Scenario 1: Page A contains a video on finding the derivative using the Chain Rule and got 400 clicks during the semester. Page B contains a narrative on finding the derivative using the power rule and got only 29 clicks during the semester. 

a. Page A has more valuable resources than Page B.

b. Page A is a more difficult topic than Page B, so students revisit it a lot.

c. The video on Page A is confusing so students watch it on multiple occasions trying to figure it out.

d. Page B was only necessary for those students who did not understand a prerequisite concept.

e. Page A is more central in the structure of the course. Students click through it a lot on their way to somewhere else.

Number of clicks per page is meaningless unless there is a constructive relationship between pages. For example, if we are looking at 5 pages that each contain one resource for learning how to find the derivative using the chain rule, the comparison of data might be interesting. But even in this case, I would want to know the order the links appear to the students. And just because a student clicks on a page, it doesn’t mean they learned anything on the page. They might visit the page, decide they dislike the resource, and go find a better one.

Completion of Online Assignments

Surely we can use completion of assignments as a meaningful metric of learning? Surely?

Well, that depends. What do students access when they are working on assignments? Can they use any resource available online? Do they answer questions immediately after reading the corresponding section of the book? Are they really demonstrating learning? Or are they demonstrating the ability to find an answer? Maybe we are just measuring good finding abilities.

Many online homework platforms (no need to name names, it’s like all of them) pride themselves on delivering just-in-time help to students as they struggle (watch this video, look at this slide deck, try another problem just like this one). I think this is a questionable practice. It is important to target the moment of impasse, but too much help means the learning might not stick. Impasse is important because it produces struggle and a bit of frustration, both of which can improve learning outcomes. Perfect delivery of answers at just the right moment might not have strong learning impact because the struggle stops at that moment. I don’t think we know enough about this yet to say one way or another (correct me if you think I’m missing some important research).

Regardless, even completion of assignments is a questionable measure of learning. It’s just a measure of the student’s ability to meet a deadline and complete a task given an infinite number of resources.

Where do we go from here?

Ford hopes that the ramifications of 2016 will foster better journalism in 2017 in ways that people read, watch, or listen to more intentionally, maybe even (shock!) remembering a story and the publisher it came from the next day.

I hope that education can focus more on (shock!) finding meaningful ways to measure whether a student actually learned, not just whether they clicked or checked off tasks. Reflecting on my own online learning experiences in the last year, I am worried. I’m worried we have fallen so deep down the “data-driven decisions” rabbit hole that we are no longer paying attention to the qualitative data that orbits the metrics. Good instructors keep their finger on the pulse of the learners, ever adjusting for those qualitative factors. But as the data ports up to departments, institutions, and vendors, where does that qualitative picture go?

I will close with a few goals for institutions, instructors, and vendors for 2017:

  1. Demand better learning metrics from ed-tech vendors. What that measure is really depends on the platform. Begin asking for what you really want.
  2. Build more integrations that pass quality learning data from the ed-tech vendor to the institution. Sometimes the platform does have better metrics, but the institution can’t access them.
  3. Create metrics that measure learning mastery over time in your own courses. This means choosing a few crucial concepts and probing them repeatedly throughout the learning experience to ensure the concept is sticking.

These are all concepts I hope to continue exploring with more research and more detail over the next year. If you want to join on that journey, consider subscribing here.


Possibly Related Posts:


read more

The Importance of Findability for Learners

Dec 16, 2016 by

How do you feel when you go to find information on a website, and you just can’t find it? This happens to me all the time when I want to find out what some new ed-tech wonder product does, and I visit the website and can’t see any screenshots, any descriptions, or any videos of the product in action. I find it incredibly frustrating and this story generally ends by me giving up on even signing up for a trial. The same thing happens to students when they go to find information and it is buried in a non-sensical place.

As everyone finishes a semester, and prepares documents and course shells for the next, it seems a good time to share this article, The Impact of Findability on Student Motivation, Self-Efficacy, and Perceptions of Online Course QualityWhile the research targeted online courses, many face-to-face courses are now accompanied by a myriad of resources that live in an LMS course shell and I think there are also implications for findability in course packets and syllabi as well.

For this article, one of the researchers, Dr. David Robins, User Experience Design professor at Kent State University, has presented the study in a webinar format available on YouTube. Their research question: What happens when students have trouble finding components of a course?


 

The researchers took two courses that were well-designed and passed Quality Matters standards, and then “broke” them in terms of findability. The broken courses still technically passed QM standards, but the components were harder to find. Students were asked to perform scenario-based tasks in the online courses.

Sidenote: If you’ve never seen a standard software usability test, here’s a nice “findability fail reel” for a mobile website with questionable usability.

I don’t think anyone will be surprised to find that poor findability correlated with decreased self-efficacy and decreased motivation. However, there was an interesting set of actionable findings regarding navigation and visual design that came from researchers watching participants attempt to navigate the courses. Consider looking for these types of things in your course or syllabus and then improving them:

  • navigation items that are not grouped into logical categories
  • poor labeling (e.g. using the file name instead of a true description)
  • poor categorization (e.g. placing an exam review under “Course Documents” instead of in the section labeled “Prepare for the Exam”)
  • deeply buried content (e.g. syllabus is buried four levels deep)

This article also got me thinking about whether the most important items of a syllabus might be presented in a more 21st-century-friendly manner. There is a whole rabbit hole of syllabi created as infographics on the Interwebs.

Probably your university is still going to want an old-fashioned text version, but maybe students could use more visual infographics for what I would consider the top-5 syllabus items of interest to students:

  • How is this course graded?
  • What are tests like?
  • Are there any projects or papers?
  • Do I have to attend class?
  • Is there group work?

As well as the additional syllabus items that instructors want them to know:

  • What are you going to learn?
  • Why should you care about what you are going to learn in this class?
  • How strict is this instructor on deadlines?
  • What is considered good/bad behavior in this class?
  • What are the instructor’s pet peeves? (come on, that’s a real thing and whole chapters of your syllabi get devoted to these issues)

Challenge: Take a fresh look at your syllabus and/or course shell. Assume that you do have findability issues and look for them. If you don’t think you have them, had over the questions above to a friend or family member and see how long it takes them to find the key components. Revise and improve the findability of important components to lower student frustration for the next semester.

Note: A weekly bite of learning design and a challenge goes out every week. If you’d like to have it delivered to your inbox, sign up at Weekly Teaching Challenge.

Reference:

Simunich, B., Robins, D. B., & Kelly, V. (2015). The Impact of Findability on Student Motivation, Self-Efficacy, and Perceptions of Online Course Quality. American Journal of Distance Education, 29(3), 174-185.


Possibly Related Posts:


read more

Learners Need to Focus on Errors

Dec 9, 2016 by

Let’s move on to the excellent article It’s Not How Much; It’s How: Characteristics of Practice Behavior and Retention of Performance Skills, by Duke et al. (2009), which is another dive into analyzing what leads to good retention of learning in music education.  Just to be different, I’ll start with the conclusion, and then circle around to the study construction.

“The results showed that the strategies employed during practice were more determinative of performance quality at retention than was how much or how long the pianists practiced”

17 students (advanced piano performance students) were given a 3-measure passage from a difficult Concerto to learn. The students each followed the following protocol for practice:

  • 2-min warmup of their choice
  • As much time as they need with the passage, a metronome, a pencil, and a piano, to lean the passage well and play it confidently at the target tempo.
  • A 24-hour break away from the music, with a promise not to practice it at all

The next day, the students had the same 2-minute warmup period (asking that they do not warm up with the practiced passage). Then they were asked to play straight through the 3 measures at the target tempo 15 times without stopping (these are the retention trials).

After recording numerical data about the practice sessions and retention trials, the researchers watched the recorded practice sessions to make detailed observations about the techniques each participant used to practice. Then they ranked the retention trials of the 17 participants, examining tone, character, and expressiveness of musical performance.

Here are the practice variables that were NOT related significantly to the retention trial rankings:

  • total time practiced
  • total number of times through the passage during practice
  • total number of correct and near-correct times through the passage

Here are the practice variables that were significantly related to the retention trial rankings:

  • number of complete incorrect performance trials (positive correlation)
  • percentage of all complete trials that were correct (negative correlation)
  • percentage of complete trials that were correct and near-correct (negative correlation)

Three of the participants were ranked as clearly superior to the other 14 students. Examining the practice sessions from these three students yielded a set of 8 strategies (see p. 317 of the article for the full list), a few of which are summarized here :

  • early in practicing, the playing was hands-together
  • when errors appeared, they were addressed immediately
  • the precise location and source of each error was identified accurately, rehearsed, and corrected
  • the tempo was varied
  • target passages were repeated until stabilized (no more errors)

While the other 14 players would employ some of the 8 strategies, it was only the top 3 that used all of them. In other words, the top-performing practicers had very good metacognition skills. They could accurately identify their mistakes, sometimes anticipating them in advance, and knew what kinds of actions to take to work in the direction of correcting the mistakes and avoiding them in the future.

One of my key takeaways from this article was a quote from the literature review, which I think is much more widely applicable to many subjects (not just music):

“making practice assignments in terms of time practiced instead of goals accomplished remains one of the most curious and stubbornly persistent traditions in music pedagogy”

In mathematics, we assign lists of problems, and students are to do all the problems regardless of their mastery over time. I’m wondering if we shouldn’t give the students a suggested list to practice from, with goals about what they should learn, and tell them to practice until they are reasonably sure they have accomplished the goals. I might go so far as to ask them to do the assignment in pen, crossing out and correcting mistakes as they go, so they can see the progress towards mastery of the topic.

I’m afraid that our relentless focus on time or quantity practiced might be handicapping students’ metacognitive abilities. Do our students actually know when they have mastered a topic? Do they transparently remember the mistakes they make after erasing them and overwriting them? I’m reminded of the Trevor Ragan video about Blocked and Random Practice that says that practice is going to be ugly. I think practice that produces learning IS going to be ugly, and somehow we need to help students be okay with this.

Challenge: Think about how to structure student out-of-class practice so that the focus is not on time or quantity, but on the error analysis and thoughtful correction. Try to make the whole purpose of the assignment around accurately identifying mistakes and working towards mastery of the material rather than “checking off” the to-do items on a list.

Note: A weekly bite of learning and challenge goes out every week. If you’d like to have it delivered to your inbox, sign up at Weekly Teaching Challenge.

Reference:

Duke, R. A., Simmons, A. L., & Cash, C. D. (2009). It’s not how much; it’s how characteristics of practice behavior and retention of performance skills. Journal of Research in Music Education, 56(4), 310-321.

Possibly Related Posts:


read more