Learning at Scale Slides from ICTCM

Mar 11, 2017 by

Learning at Scale: Using Research To Improve Learning Practices and Technology for Teaching Math

In the last 5 years, there has been a rise in what we might call “large-scale digital learning experiments.”  These take the form of centralized courses, vendor-created courseware, online homework systems, MOOCs, and free-range learning platforms. If we mine the research, successes, and failures coming out of these experiments, what can we discover about designing better digital learning experiences and technology for the learning of mathematics?

 

Possibly Related Posts:


read more

Elaborations for Creative Thinking in STEM

Feb 25, 2017 by

As I watch the proliferation of digital learning platforms, particularly in STEM education (where there are lots of objective-type problems), I am excited by the increased focus on learning and adaptivity but also a little uneasy. For the most part the motivations to “go digital” are pure – increase access to courses that students need, provide help that is more tailored to each student, give immediate feedback, provide more practice if the student wants it, and let students move at their own pace.

My worry is that math and science students aren’t getting anything but highly-structured problems. Every problem that a computer delivers is one where there is a carefully constructed set of constraints on the problem and a highly uniform (and thus gradable) answer.

But the problems of the real world aren’t carefully constrained and the problem solutions aren’t highly uniform. If we only teach students to solve the “lives in an orderly box” kind of problems, what are we preparing them for? Are we creating experiences that lead to curiosity in STEM subjects? Do students even see subjects like math as highly connected networks of concepts or just as discrete concepts to be learned one-at-a-time inside a digital system?

What is a Problem Elaboration?

This semester I’ve fallen back on a very old assessment technique I developed about 15 years ago that I call a “Problem Elaboration.” Students have to turn in a two problems per topic/section where they complete the problem (the old-fashioned way on paper) and then do some kind of mathematical elaboration on the problem. In other words, they have to do something that wasn’t asked for. They have to think in the space around the problem and consider:

  • What else could I find?
  • What other mathematical things could I do with this?
  • How does this connect to other things we’ve learned?
  • How could I check this to make sure it’s right?

It’s fascinating to watch the initial struggle of many of the students with the sudden freedom of elaborations. Students fall into three categories in the beginning of the term, but every student that makes an honest effort moves up this ladder to increasingly creative and complex elaborations over time.

Trepidatious: How am I supposed to find something that I’m not asked to find? I don’t even know how to begin! This is crazy! Your job is to support these students with examples, suggestions, and a “you can do it” attitude. These students will eventually try, and when they do, encourage them!

Compliant: If there is some way to check the problem, I’ll do that. If I wasn’t asked to graph the problem I’ll do that. These are the not the most creative elaborations. But it is an exploration that contributes to mathematical maturity. This tends to end up being what students do when they are short on time.

Elaboration 1: Notice the actual problem is highlighted in a lighter color. The elaboration is to check the answer. It’s not particularly glamorous, but the student knows how to check the answer, she knows it’s right, and that’s pretty awesome.

 

Elaboration 2: This also has the standard “check your answer” type of elaboration, but it goes one step further. This student shows me that she knows how to GRAPH both sides of the equation and verify that she has the right solution!

Curious:  What happens to this graph if I change this number or this sign? I did “X” and was expecting “Y”. Why didn’t that work? How does this connect to what we did yesterday? How would I find this thing I don’t know how to find?

Elaboration 3: Now we see an elaboration that really begins to explore the “what happens if” space. What happens if this is a 5th root instead of a square root? What happens if there is a number multiplied by the square root in the original equation?

 

Elaboration 4: This students starts by checking the answer, noting where they made a mistake the first time, and redoing the check. Then the student explores whether they can square individual terms in the equation instead of isolating the square root first and get the same answer. See the question the student is left with? “Not sure if that means something or not.” That’s the opener to a conversation in pen pal form about the right way that squaring both sides actually works.

As I grade these assignments, I find that what I’m really doing is opening up a conversation with the student. The elaborations aren’t always right, and often the students are asking questions in the elaborations. This is our chance to explore the math together in a back-and-forth letter to each other, assignment after assignment. Over the course of a semester, I watch many students develop mathematical maturity, mathematical confidence, and ownership of the math they have learned.

Logistics

I assign 2 problems to turn in with elaboration per topic, using a particular grading rubric for these problems, based on 5 points:

  • 1 point for rephrasing or rewriting the problem (students end up solving a slightly miscopied problem, or don’t catch all the parts of the problem)
  • 2 points for showing all appropriate work to solve the problem
  • 1 point for actually finding the correct answer
  • 1 point for the elaboration

I also award a bonus point for really great and thoughtful elaborations (at discretion of instructor). Typically students earn between 0 and 5 bonus points per unit depending on their effort and thoughtfulness.

If a student makes an honest and thoughtful attempt at an elaboration, they will get the point even if their elaboration ends up being wrong or using incorrect terminology. The point is that the student explores and tries. My role is to correct improper terminology and to help when their reasoning wanders into an area where it no longer holds.

At the beginning of the semester, I do provide some idea of what an elaboration might be. A single elaboration might be any one of the following, but this is not an all inclusive list of what could be done.

  • Show a different way to solve the problem.
  • Show how to check the answer.
  • Solve for something extra in the problem.
  • Relate the concepts in this problem to something else we’ve studied.
  • Relate the concept in the problem to a graph (if there wasn’t a graph required in the problem).
  • Hypothesize on how a change to the problem might change the answer, and then try it.
  • If it relates to this problem, investigate something we did in class that you found difficult to understand or remember.

Why the unease?

Online learning and learning through digital platforms is highly structured and scripted. Students learn exactly what they are scripted to learn. They only “explore” in the sense that we sometimes allow them to choose the next highly-scripted chunk of learning they engage with.

How do we build digital learning spaces where something like elaborations and this personal conversation between myself and the student about the mathematics can also be encouraged as a part of learning?

And if we don’t encourage our most engaged and curious students to go deeper, to question more, and to make more connections, what’s going to happen to our STEM pipelines?

Possibly Related Posts:


read more

Better to be Frustrated than Bored

Jan 23, 2017 by

For all of you who have taught students, you know that one of the rewards is seeing the “Aha moments” that students experience. One of the downsides (to instructors) of teaching online is that it is hard to “see” the reward of the “aha” in the same fulfilling way.

The reason we should care about the aha moments that students experience is that this kind of moment is tied closely to an emotional feeling. And memories with emotional attachments tend to be stronger (more memorable) than ones with no emotional attachment.

But have you ever stopped to think about what is going on right before the aha moment? Does the aha moment come after a well-organized lecture, a step-by-step example, a period of boredom, or a period of confusion or frustration?

An aha moment typically comes after a period of confusion or frustration. This means that you have to put students into that space where they are actually at the edge of what they know (the confusion/frustration space) to nudge them over to insights.

Anyways, I digress. I want to share findings from this paper: Better to Be Frustrated than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments (Baker, et al, 2010).

The researchers set out to focus on cognitive-affective states that were hypothesized to influence cognition and deep learning: boredom, confusion, delight, engaged concentration, frustration, and surprise. The researchers use Russell’s Core Affect framework (2003) to map these states in two dimensions: valence (pleasure to displeasure) and arousal (activation to deactivation).

In this study, the researchers examined:

  • the cognitive-affective states the students experienced during the learning process
  • how those states persist over time (e.g. do students move from boredom to frustration more often than frustration to boredom?)
  • how the state affects the students choices on how to interact with the system (e.g. what causes students to game the system?)

While I will leave you to read the whole paper if you want all the details (the methodology involves three different interactive learning systems and three different methodologies), I think the nuance of definitions between a few of these terms is important. As defined in the paper:

  • frustration is dissatisfaction or annoyance
  • confusion is a noticeable lack of understanding
  • engaged concentration is a state of engagement with a task such that concentration is intense, attention is focused, and involvement is complete

Now let’s jump ahead to (what I consider to be) some of the interesting results. Engaged concentration was the most common state during the observation periods (60%) followed by confusion (13%). While boredom was only observed about 4-6% of the time, it was also the most persistent state (once bored, the student stays bored) across all three learning systems.

Within two of the systems where “gaming the system” was observed, a more in-depth analysis was performed. Boredom was significantly more likely to lead to gaming the system. Guess what wasn’t likely to lead to gaming the system … confusion, frustration, and surprise. Better to be confused than bored, huh?

There is quite a bit of new research being performed on the role of confusion in learning, but my gut feeling here is that confusion leads to self-insight, and learning gained through self-insight (because this is the aha where emotions are attached) should be stickier than learning delivered through other states.

Challenge: Vigilantly watch for states of boredom in your classes, and when you find them, intervene. Do something different. Put students into a space where they are challenged and maybe even a little confused. Give the learners a chance to grapple with the concepts and have those moments of self-insight.

Reference:

Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223-241.

Possibly Related Posts:


read more

Video of AMATYC Keynote

Jan 14, 2017 by

Here is the video of the AMATYC Keynote, Learning Math is Not a Spectator Sport.

 

Possibly Related Posts:


read more

Celebrate the Errors in Math Practice

Jan 11, 2017 by

Dear math students,

As you work through your mathematics practice, I’m going to challenge you to embrace making errors in an entirely new way. 

Many students believe that every problem in math homework should be perfectly constructed with no errors. It might look something like this:

A nicely ordered problem solution with no mistakes.

 

But when it’s time to study after the initial problem run-through, what does this perfectly constructed problem say? Does it coach you on remembering how you struggled? Does it remind you where you made an error? No.

When you make an error as you’re working a problem, please don’t erase it from the face of the earth. Certainly you should learn from the struggle and complete a correct solution, but record your deviations from the straightforward solution path in another color. Leave yourself notes (also in a different color) to remind you what you should have paid more attention to the first time around. Maybe that would look something like this:

Worked problem with highlights and notes to self

 

 

Sometimes you’re going to recognize but you don’t have the right answer but you’re not going to be sure what’s gone wrong. You should always try first to figure it out yourself first. This process of error analysis in a variety of different situations is key to developing problem solving skills in mathematics. Without exploration of the problem space (which happens with error analysis), your brain is just recording rote procedures without the ability to transfer those procedures to new kinds of problems. It will, essentially, stunt your mathematical growth.

Now, I don’t want you to get to the point of tears or breaking your keyboard out of anger. If you get near that stage please just ask a question (email, discussions, chat-a-friend, etc), leave a sticky note on the page as a reminder to go back, and move on to the next problem or section of problems. Just switching to a slightly different problem can not only get you unstuck, but sometimes give you insights into the “stuck” problem. 

When you figure out how to do the problem you were stuck on, make sure to go find that flagged problem (remember the sticky note?) and annotate your corrections.

Worked math problem with cross-outs and restarts and notes to self.

 

Now you might be thinking “why do all these error corrections and problem annotating in another color?” When it comes time to study for your major assessments, you will be able to see the places where you stumbled the first time you tried the problem. These “notes to self” are the places where you’re most likely to make the same mistake again. They benchmark places to remember to be careful and show you problem types to repeat practice before an exam.

Celebrate your errors.

Embrace the messiness.

Learn from your mistakes.

Study from your struggle points.

And be great at mathematics!

Possibly Related Posts:


read more

The Deliberate Practice Experiment

Jan 7, 2017 by

For some inspiration to try a more active approach to learning, I thought I would share a short video from Dr. Carl Wieman, who is a Nobel Laureate, a physicist, and more recently a researcher in learning science. Wieman has designed and run some very elegant experiments to demonstrate the effects of active learning techniques. Consider this one:

What happens if you run two classes for a week with the same learning objectives, the same time in class, and the same assessment. Prior to the experiment, you take care to ensure that the student makeup and performance is very similar up to this point. One class continues on, lecture-style, with a veteran instructor with lots of experience. The other class is assigned a freshly-minted PhD trained to lead the students in learning via deliberate practice (an active learning strategy). The courses meet under these conditions for the exact same time.

The technique of deliberate practice takes the form of a series of challenging questions and tasks. This forces students to spend their class time thinking scientifically, discussing concepts, critiquing each other’s predictions, and engaging in problem solving.

To see how this (and a few other elegant experiments) turn out, take the 15 minutes to watch the video: Finding New Ways to Learn Science.

Really … watch the video. Student pushback against these strategies is a real issue and usually worst at the beginning of the semester. We can all use a good foundational reminder about why we continue to strive to lead more active learning sessions in our classes.

It is also well worth the time to read the very well-designed study, Improved learning in a large-enrollment physics class, mentioned in the video.

Challenge: Consider the deliberate practice model described above. Where can you tackle a learning objective through a series of challenging questions and tasks instead of with lecture? Give yourself that extra nudge to try just a little more activity (or to build in just a little more activity).

Reference:

Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science, 332(6031), 862-864.

Reminder: You can sign up to receive the Weekly Teaching Challenge in your email or share it with a friend.

Possibly Related Posts:


read more

Write an Operating System for Your Brain

Jan 1, 2017 by

Let’s face it. We all have to interact with organizations and systems of relationships in which we might not “fit” perfectly. We can choose to stay and try to thrive, or leave and find another path. Both are valid options. If you are choosing to stay, you will likely have to make minor (or sometimes major) tweaks to how you interact and function within that system. It is unreasonable to expect a system with thousands of employees or a family with 4 people in it to all change and adapt to you when you are unwilling to make changes to adapt to it.

What if you began approaching your brain like a computer system? Suppose you could load in a set of operating instructions at the beginning of every day to govern how you operate in the world around you. As you find “bugs” (character flaws) in your system, suppose you could write new code to help you begin to correct those bugs. I think we can actually do this.

Computers boot up with an operating system that tells the hardware (printer, mouse, monitor, etc.) how to function and runs the software apps. The operating system is the first program the computer loads when you turn it on. In a sense, the operating system directs everything else in the computer system.

You can write a personal operating system (POS) that directs your physical and mental functioning. The POS can be a very powerful tool to help you change behavior that is holding you back in your professional and personal life.

Here’s the basic idea:

  1. Write a set of instructions (the POS) that you want to be your governing principles.
  2. Read the POS every morning out loud.
  3. Each evening before bed, celebrate the wins from the day as you reread the POS.

Aspirational Lines of Code

Since this is you talking to your own body and brain, it’s important that you are kind to yourself. Negative instructions tend to focus on past behavior. The personal operating system should be a set of instructions on what TO do in the present and future. The instructions you give yourself every day should be affirming and aspirational. Any instruction can be written in a negative or positive way.

Here’s a line of code for running your brain:

Negative code: Don’t brag.
Positive code: Be humble.

And another line of code for running your body:

Negative code: Don’t eat crap.
Positive code: Choose healthy food.

A personal operating system should consist of approximately 5-15 lines of code that set up instructions for your brain and body. Here are several lines of code from the personal operating system I used last year, when I worked with a very large team in a leadership position:

Schedule impulsive things.
Support the leaders around you.
Focus on the big deliverables.
Give everyone a chance to shine.
Communicate down as well as up.
Do something scary every day.

The goal for a personal operating system is to create a set of reminders that become so well-known to you that they pop up in your mind when you are confronted with situations that run counter to your operating system.

Writing the First Draft of your POS

How do you go about creating your own personal operating system?

There are several questions you can use for reflection that will give you a good first draft:

  1. If I were to ask people around you to describe you, what would you want them to say about you?
  2. What are the characteristics of the people you you admire and look to as role models?
  3. What are the character flaws that are keeping you from moving up or into new roles at work?
  4. What are the character flaws that are keeping you from having more meaningful relationships with your partner, children, or friends?

As you identify flaws, remember that you need to find ways to write affirming statements (not negative ones) for the operating system.

Getting Feedback on the POS

This may be too scary for your first POS, and if that’s the case, just skip this step. However, it can be a really valuable way to nail the instructions that will be the most powerful in transforming yourself.

To get feedback on your draft POS, you want to identify 2-5 trusted people (friends, family members, or former work colleagues) that will read the POS and tell you what you’re missing. Ask them questions like:

  1. What is holding me back at work?
  2. What is holding me back from deepening our relationship?
  3.  What should be on this list that isn’t?

Approach the feedback you get with this mindset: I will listen to what they say without argument.

Now revise your POS in a way that you think is appropriate. You can take it or leave it with respect to the feedback you got, but you did ask for it, so I’d consider taking it if you truly asked people you trusted.

Finalize the POS

As you get the POS ready to be used, consider how it should be written on paper. Do you want your POS to be loaded from shortest statement to longest statement?

Be humble.
Choose healthy foods.
Schedule impulsive things.
Focus on the big deliverables.
Do something scary every day.
Support the leaders around you.
Give everyone a chance to shine.
Communicate down as well as up.

Does it make more sense to you to group the POS so that personal and work goals are separated?

Be humble.
Choose healthy foods.
Do something scary every day.

Schedule impulsive things.
Focus on the big deliverables.
Support the leaders around you.
Give everyone a chance to shine.
Communicate down as well as up.

Or maybe it makes more sense to you to organize the list so that the hardest instructions always come first.

Be humble.
Choose healthy foods.
Focus on the big deliverables.
Schedule impulsive things.

Do something scary every day.
Support the leaders around you.
Give everyone a chance to shine.
Communicate down as well as up.

Finalize the order. Print the list (you may need a few copies depending on your morning and evening routines). You might want to laminate the list (packing tape works well), mount it on cardboard, decorate it, or do something else to make it feel more like a permanent structure in your life. It’s your POS, figure out what makes sense to you.

Loading the POS in the Morning

The POS will only work if you actually READ it every morning. So you have to find a place to put it where you will have the time to read the list out loud with minimal disruption to your normal routine. Try to think of a time when you’ll be likely to be able to read the list. For me, the ideal time is when I’m drying my hair, so my list is in the bathroom. For you it could be while you wait for your coffee to brew, when you are making your breakfast, or while you drive to work.

Find a home for your POS, and make it a part of your routine to read the list of instructions every morning.

Reread the POS in the Evening

Now find a time in the evening that you can skim your list and celebrate the wins and small corrections you made during the day. Again, the bathroom might be the right place (I reflect while I brush my teeth). You don’t want to beat yourself up if you didn’t manage to execute all your instructions. The goal is to find where an instruction did actually lead you to a new behavior during the day. You want your mind to begin to deliver the correct instruction at the moment you need to hear it. For example, at the moment that you are about to brag about an accomplishment at work, “Be humble.” floats into your mind and you congratulate the team on a job well done instead of bragging. That’s a win to be celebrated at the end of the day.

Of course, you might also identify moments in the day where you could have done better, and that’s okay. You simply need to acknowledge them, and then dismiss them with the thought that you’ve now recognized these types of moments and can improve another day.

Tuning Up the POS

As time passes and jobs and relationships change, your POS will need tuning up. Don’t operate on an old POS when your life has had significant changes. The example POS I’ve shared makes little sense now that I’m self-employed and no longer working with an enormous organization with a large team to manage. I’ve revised it. You should too as your situation changes.

I would love to compile a list of instructions that people can choose from to write a POS. What are some of the “lines of code” you’re writing into your own personal operating system?

Possibly Related Posts:


read more

To Focus on Learning, Use Words Not Section Numbers

Dec 30, 2016 by

Since many faculty are now approaching that time when syllabi get written and course shells get built, let’s focus on a very simple change that can increase student attention to learning the vocabulary and learning objectives/goals associated with the subject.

Take examine your syllabus, schedule, and course shell looking for places where you could have written the name of the course, the topics, or the learning objectives in words, but didn’t do it. When schedules and other course materials are passed out without any reference to the vocabulary of the subject, you are missing a great chance to put that vocabulary and learning objectives/goals front and center for your learners.

Consider this real course schedule I saw recently (altered a bit to protect the identity of the course and college).

This schedule contains no course name, no topics in words, and no learning goals. My course schedules used to look like this too (although they did at least have the name of the course at the top). But then I took a deep dive into how we learn vocabulary a couple years ago. Now I use every opportunity to engage students with the words that describe the topic and the learning goals.

This schedule was a golden opportunity for introducing (repeating) vocabulary and focusing on learning objective … and it was missed. Let’s look at a example of the same schedule with vocabulary and focus on learning objectives.

Now every day on the schedule has context to it, and the student is reminded of the language associated with the topic on every viewing of the schedule. If the student is making a judgement call about whether to skip class (face it, they do) or how long to procrastinate on doing their assigned homework, the context of what is in the schedule is very helpful to them. If they recognize no words in the topic, or recall that they have never figured it out in the past (dev math), then they have additional incentive to show up, leave more time to complete, etc.

The reality is that if we want students to actually learn the vocabulary of our subject area, they have to encounter the vocabulary as often as possible. They have to read the vocabulary like they will see it in context, and do that as much as possible. This repeated encounter helps them to begin to see the contrast between important differences. In the example above, there is a difference between solving equations and solving inequalities. The repeated exposure to those as two separate topics helps the student to begin to separate “equations” and “inequalities” as two distinct topics with rules associated with the overlap and contrast between them.

Working with the vocabulary and learning objectives is also helpful to faculty. Instructors need to see more than a section number as they plan out the learning for the semester. We’ve all encountered texts where one section is not equal to one hour of class. Writing out the learning topics / objectives gives the instructor a chance to reflect on where there needs to be supplementary learning activities and where the topic is actually too “lite” for a whole day of class. By planning for this natural fluctuation up front (assuming one has experience teaching the course on a previous occasion), it is more likely that the pacing of the course will match the real learning of the students.

Challenge: Take a fresh look at your syllabus, schedules and/or course shell. Find opportunities to place vocabulary and learning objectives/goals in more prominent locations.

Note: A weekly bite of learning design and a challenge goes out every week. If you’d like to have it delivered to your inbox, sign up at Weekly Teaching Challenge.

Possibly Related Posts:


read more

The Four Processors: A Neogeneralist Problem?

Dec 22, 2016 by

My husband has been joking for years that my brain has 4 processors. He says I can’t relax unless at least 3 processors are turned off, but that’s not an easy thing to do. When one processor can’t solve the problem it is stuck on, my brain brings up the next-most-interesting processor and begins working on that problem instead.

Often, I wake up at 5am to find that one processor has kicked out a solution to a problem and then it is impossible to go back to sleep. Sometimes I can’t sleep because the processors are occupied with such interesting problems they won’t turn off.

The interesting part is what happens if I do not have enough interesting topic matter to supply to the 4 processors. They don’t seem to redistribute existing problems and work in parallel. The other processors are uninterested if one processor is handling the problem. Rather the remaining processors become unsatisfied and depressed – they feel under-appreciated and unfulfilled.

Recently, I read The Neogeneralist, by Kenneth Mikkelsen and Richard Martin. Neogeneralist “are people who are able to find connections across fields, to continuously learn specialties and apply their learnings across disciplines.” As I try to figure out how I want to be spending my time (for neogeneralists, where you go is what you do), I’m starting to frame this problem as a 4-processor issue. Suppose I can (have to) focus on four major areas/problems/realms.

As Director of Learning Design at WGU, there were four major realms my brain worked on:

  1. Instructional Design
  2. Learning Technology Design
  3. Infrastructure/Organization Design
  4. Process Design

While these were all worthy problems, I felt increasingly locked into the job in a way that made me feel as though I was suffocating. None of the problems gave me the chance to interact directly with learners or faculty (which is odd, I know).  In this position, I was solving problems every day, but so far removed from “users” (students) that I couldn’t ever see whether the solutions would bear fruit, even at a small scale. I am used to having a “petri dish” to experiment in (a MOOC to teach, a course at a college). While my job originally encompassed “Improve Learning at WGU,” there became organizational and technological barriers to actually improving learning from this position as the organization grew and reorganized.

The other thing (but pretty normal at a growing organization) that happened at WGU was that we began to carve out some of these major realms of focus to other parts of the organization. Theoretically, this would be great because as an employee I’d have more time to focus, but here’s the reality of what happened:

  1. Processor 1 is unfulfilled and sad
  2. Learning Technology Design
  3. Processor 3 is unfulfilled and sad
  4. Processor 4 is unfulfilled and sad

Normally, this is where I would find some other project to work on in my free time to keep a processor or two occupied, but WGU had a very strong “no outside work” policy. There was a startup I wanted to spend time pursuing, but this would be “outside work.” I could teach as an adjunct somewhere, but this would be “outside work.” On top of that, Processor 3 became completely occupied/obsessed with a fundamental problem of the design of the organization and could not find any way out of the problem (and also lacked a position with any possible influence over this issue).

  1. Processor 1 is unfulfilled and sad
  2. Processor 2 is trying to figure out what job actually is
  3. Processor 3 is trying to solve organizational design and is stuck in an infinite loop
  4. Processor 4 is unfulfilled and sad, starts to catastrophize about T presidency

As organizations grow, they necessarily want employees to be more focused and less of generalists, but that is a problem if you happen to be a neogeneralist. Our unique abilities come from being experts in several areas and using cross-pollination of those areas to solve problems. If you’re going to shut down our ability to focus on multiple areas at work, don’t also shut down our ability to seek out new and interesting problems in our free time. It’s going to end in disaster.

Self-employment can also be rough, because if your processors aren’t fully occupied, then at least one of them will spend time in panic mode about not finding enough work. For the first month, the processors were occupied like this:

  1. Startup
  2. Trying to find clients to sustain income
  3. Trying to find long-term plans that would still work if 1 & 2 fail
  4. Unfulfilled and sad

Now onto the new problem. If I intentionally design my life, what do I really want the processors to be occupied with? Are they all fulfilled? Can I convince cofounders, investors, etc that I am simply not going to be happy with extreme focus? Probably most people keep at least one processor completely focused on their children, but I don’t have childeren. So I really do have 4 to allocate.

I also have a pretty strong history of allocating 4 processors. Here’s the processor allocation for my undergraduate work (I got 3 degrees while working close to 30 hours a week):

  1. Math
  2. Chemistry
  3. Biology
  4. Improving learning (TA, tutor, grading papers – payed the bills)

Here’s the processor allocation for my Masters graduate work (2 degrees, plus working):

  1. Math
  2. Business
  3. Improving learning (TA, adjunct for another college, doing learning research)
  4. Math curriculum work (writing texts, learning the assessment world, curriculum production)

Here’s the processor allocation for my PhD work (1 degree, full time faculty, consulting on the side). I still feel sorry for my PhD advisor for having to “manage” me. There were two years where “PhD” basically got bumped out when something in category 3 became a full-time processor job for a time.

  1. Teaching for MCC (5-5-2 load, typically 2-3 preps a term)
  2. Improving learning (experimenting, reading, writing, speaking)
  3. Math curriculum work (writing curriculum, building paper and digital math games)
  4. PhD Higher Ed Leadership

So what am I doing now?

  1. Startup (curriculum / learning design)
  2. Client work (product management / UX / LX, software) – also pays the bills 🙂
  3. Improving learning (teaching, reading, writing, speaking)
  4. Math curriculum work – also pays the bills and an area where I have a lot of time invested

As the startup grows, it will likely have to occupy more than one processor, and then the hard part – what goes? I don’t think I will be happy if I “improving learning” is not a category. It seems to be a consistent theme throughout the last two decades. My brain finds “Improving Learning” to be a fun distraction – what others might consider something akin to a “hobby” (though to most people, it is a job not a hobby).

I have strategies for managing conflicting priorities (good ones, in fact). I’ve always had strategies that keep my brain balanced.

If you’re still reading, wow. This is really an attempt to work through my own “post mortem” of why I felt so compelled to leave WGU a few months ago. There were a few other reasons (loss of authority, changes in leadership, nasty colleague) that probably just pushed me over the edge of the cliff, but not worth sharing the details – those things happen and there wasn’t anything I could do to change things. If I had been fundamentally happy, I would have stayed.

I’m trying to do a more intentional “Life Design” now, which I’ve come to realize is possibly different than what we traditionally consider “work life” balance. Yes, I should get exercise every day (I do). Yes, I should spend time with friends and family (I do). But those are necessary conditions, not sufficient ones. Sufficient, for me, means having a brain that is fully occupied solving problems that makes it happy.

Possibly Related Posts:


read more

Clickety Click Click: Awful Measures for Learning

Dec 19, 2016 by

I feel a little inspired by Sam Ford’s post The Year We Talk About Our Awful Metrics. Ford writes about the need for change in metrics of online media publications, but we could just as easily be discussing the metrics of learning management systems, ed-tech vendor platforms, and institutional analytics.

Ford argues that we need to “get serious” about better forms of measurement in 2017. As long as we are measuring metrics with little meaning, we aren’t really improving learning.

Let me give you a few examples to illustrate the similar problems in education.

Page Clicks

As in, how many pages of the ebook has the student accessed? Because the student must read every page they access, right? And they don’t just scroll through pages to see roughly what the book has in it? Realistically, I think we all acknowledge these inevitabilities, but that doesn’t stop us from creating blingy dashboards to display our metric wares.

Consider the following scenarios.

Scenario 1: Student A has clicked on 55 pages whereas student B has only clicked on 10 pages. This means:

a. Student A has read more than Student B. Student A is a more engaged student.

b. Student B was reading deeply and Student A was skimming.

c. Student A reads faster than student B.

d. Student A read more online. Student B borrowed a book from a friend and read more on paper.

e. None of the above. Who knows what it really means.

Scenario 2: Student A has clicked on 55 pages whereas student B has only clicked on 10 pages. Both students spent 2 hours in the eReader platform.

a. Student A has read more than Student B. Student A is a more engaged student.

b. Student B was reading deeply and Student A was skimming.

c. Student A reads faster than student B.

d. Student A read more online. Student B borrowed a book from a friend and read more on paper.

e. None of the above. Who knows what it really means.

In either case, how much do we really know about how much Students A and B have learned? Nothing. We know absolutely nothing. These metrics haven’t done a thing to see what either student is capable of recalling or retrieving from memory. There is nothing to help us to see whether the student can make sensical decisions related to the topics and nothing to show whether concepts can be transferred to new situations. Page clicks are a bad metric. All they tell me is that students log in more every Sunday night than on any other night (and that metric has been the same for a decade now).

But wait … there are more metrics …

Attendance

We can measure attendance – whether it be logging in to the LMS or physically showing up in the classroom. Surely this is a valuable measure of learning?

Again no, it’s not a measure of learning. It’s potentially a necessary condition of a necessary-but-not-sufficient metric for learning. Yes, we do need students to show up in some way to learn. In very active face-to-face classrooms that engage all students in learning activities, I might go so far as to say that showing up is a good measure of learning, but this is still the exception rather than the norm. And even if the classroom is active, learning is more effective with certain kinds of activities: those involving interaction, those involving varied practice, those where students have to learn to recognize and remedy their own errors.

Attendance, by itself, does not measure learning.

Time to Complete

At organizations where the learning is assessed directly (CBE and MOOCs, for example), there is often a metric around the “time to complete” a course.  This is a particularly dangerous metric because of the extreme variability. Again, let’s look at two scenarios.

Scenario 1: Course 1 is a 4-credit course that takes (on average) 45 days to complete. Course 2 is a 4-credit course that takes (on average) 30 days to complete.

a. Course 1 is poorly designed and Course 2 is well-designed.

b. Course 1 is harder than Course 2.

c. Course 1 and Course 2 seem about equal in terms of difficulty and design.

d. None of the above.

Scenario 2: Course 1 is a 4-credit course that takes (on average) 45 days to complete and requires students to turn in 4 papers. Course 2 is a 4-credit course that takes (on average) 30 days to complete and requires students to pass 2 exams.

a. Course 1 is poorly designed and Course 2 is well-designed.

b. Course 1 is harder than Course 2.

c. Course 1 and Course 2 seem about equal in terms of difficulty and design.

d. Students procrastinate more on writing papers than on taking exams.

e. None of the above.

In either case, what does the “time to complete” actually tell us about the quality of learning in the courses? If we were comparing two Calculus I courses, and they were taught with different platforms, equivalent assessment, and the same teacher, I might start to believe that time-to-complete was correlated with design, learning quality, or difficulty. But in most cases, comparing courses via this metric is like comparing apples to monkeys. It’s even worse if that data doesn’t have any kind of context around it.

Number of Clicks per Page

This is one of my favorites. I think you’ll see the problem as soon as you read the scenario.

Scenario 1: Page A got 400 clicks during the semester. Page B got only 29 clicks.

a. Page A has more valuable resources than Page B.

b. Students are accidentally wandering to Page A.

c. Page A is confusing so students visit it to reread it a lot.

d. Page B was only necessary for those students who did not understand a prerequisite concept.

e. Page A is more central in the structure of the course. Students click through it a lot on their way to somewhere else.

Scenario 1: Page A contains a video on finding the derivative using the Chain Rule and got 400 clicks during the semester. Page B contains a narrative on finding the derivative using the power rule and got only 29 clicks during the semester. 

a. Page A has more valuable resources than Page B.

b. Page A is a more difficult topic than Page B, so students revisit it a lot.

c. The video on Page A is confusing so students watch it on multiple occasions trying to figure it out.

d. Page B was only necessary for those students who did not understand a prerequisite concept.

e. Page A is more central in the structure of the course. Students click through it a lot on their way to somewhere else.

Number of clicks per page is meaningless unless there is a constructive relationship between pages. For example, if we are looking at 5 pages that each contain one resource for learning how to find the derivative using the chain rule, the comparison of data might be interesting. But even in this case, I would want to know the order the links appear to the students. And just because a student clicks on a page, it doesn’t mean they learned anything on the page. They might visit the page, decide they dislike the resource, and go find a better one.

Completion of Online Assignments

Surely we can use completion of assignments as a meaningful metric of learning? Surely?

Well, that depends. What do students access when they are working on assignments? Can they use any resource available online? Do they answer questions immediately after reading the corresponding section of the book? Are they really demonstrating learning? Or are they demonstrating the ability to find an answer? Maybe we are just measuring good finding abilities.

Many online homework platforms (no need to name names, it’s like all of them) pride themselves on delivering just-in-time help to students as they struggle (watch this video, look at this slide deck, try another problem just like this one). I think this is a questionable practice. It is important to target the moment of impasse, but too much help means the learning might not stick. Impasse is important because it produces struggle and a bit of frustration, both of which can improve learning outcomes. Perfect delivery of answers at just the right moment might not have strong learning impact because the struggle stops at that moment. I don’t think we know enough about this yet to say one way or another (correct me if you think I’m missing some important research).

Regardless, even completion of assignments is a questionable measure of learning. It’s just a measure of the student’s ability to meet a deadline and complete a task given an infinite number of resources.

Where do we go from here?

Ford hopes that the ramifications of 2016 will foster better journalism in 2017 in ways that people read, watch, or listen to more intentionally, maybe even (shock!) remembering a story and the publisher it came from the next day.

I hope that education can focus more on (shock!) finding meaningful ways to measure whether a student actually learned, not just whether they clicked or checked off tasks. Reflecting on my own online learning experiences in the last year, I am worried. I’m worried we have fallen so deep down the “data-driven decisions” rabbit hole that we are no longer paying attention to the qualitative data that orbits the metrics. Good instructors keep their finger on the pulse of the learners, ever adjusting for those qualitative factors. But as the data ports up to departments, institutions, and vendors, where does that qualitative picture go?

I will close with a few goals for institutions, instructors, and vendors for 2017:

  1. Demand better learning metrics from ed-tech vendors. What that measure is really depends on the platform. Begin asking for what you really want.
  2. Build more integrations that pass quality learning data from the ed-tech vendor to the institution. Sometimes the platform does have better metrics, but the institution can’t access them.
  3. Create metrics that measure learning mastery over time in your own courses. This means choosing a few crucial concepts and probing them repeatedly throughout the learning experience to ensure the concept is sticking.

These are all concepts I hope to continue exploring with more research and more detail over the next year. If you want to join on that journey, consider subscribing here.


Possibly Related Posts:


read more