How many answer choices is best for a multiple-choice question? Probably not what you think.

Last week I discussed a quiz I developed  to help people analyze what they know and don't know about developing valuable and valid multiple-choice questions (mcqs). The mcq quiz: https://forms.gle/FMcuPywmKFLJ69je8.

Last week I shared the answer to one of the questions: Is “describe how to” a good behavior/action to use in a workplace learning objective? The answer is no and I explained why.

This week I'm discussing another question on the quiz. Here's the question.

What is the optimal number of answer choices for a multiple-choice question, according to research? (select the best answer) 

• Three 

• Four

• Five 

The image below shows the pattern of replies from the 110 people who answered (so far). The green bar indicates the correct answer. The other two are incorrect.

Most people think MORE answers is better. That's because more answers choices appears to lower the chance of guessing correctly. 

In other words, four answer choices appears to have a 25% chance of guessing correctly. Five answers choices appears to have a 20% chance of guessing correctly. 

But the number of answer choices is not the only thing that determines the chance of guessing correctly. Quality of answer choices makes all of the difference.

Research shows that most people have a very hard time writing good multiple-choice questions. They have a hard time writing good answer choices. Mcqs with four or five answer choices often perform poorly because the answer choices are poorly written and make it easy to guess which are incorrect.

So research tells us to do the following when writing answer choices.

1.  Learn how to write good wrong and right answer choices!
2. Write two GOOD incorrect answer choices and one GOOD correct answer choice.

If you write mcqs, you cannot afford to have poorly written answer choices in your multiple-choice questions as they can damage assessments, people, and the organization.

Added:
Thanks to Anand Chandarana for reminding me to give others a link to an in-depth review of the research on the issue of number of answer choices.

Rodriguez, M. C. (2005). Three options are optimal for multiple‐choice items: A meta‐analysis of 80 years of research. Educational Measurement Issues and Practice, 24(2), 3 – 13.  http://www.highpoint.edu/citl/files/2017/06/Three_Options_Are_Optimal_for_MCQ_Rodriguez_2005.pdf. 

-----------

If you need to learn the secrets to designing valid mcq assessments, take my upcoming Write Learning Assessments course http://bit.ly/dlaw-wla. I'll teach you how to write GOOD correct and incorrect answer choices because these are essential to good mcqs and good mcq assessments. 

My take: If you design mcqs, you need to know how to do this difficult skill well. I'd love to teach you how.

"Describe how to..." is (usually) an inadequate learning objective

I developed a quiz (https://forms.gle/FMcuPywmKFLJ69je8) to help people analyze what they know and don't know about developing valuable and valid multiple-choice questions (mcqs). One of the questions:

Is “Describe how to” a good behavior/action to use in a workplace learning objective? (select the best answer)

• “Describe how to” is a measurable behavior/action so it is appropriate to use in a workplace learning objective.
• “Describe how to” is rarely part of tasks so this behavior/action is likely written at too low of a level.
• “Describe how to” is often part of tasks so this behavior/action is appropriate to use in a workplace learning objective.

The image below shows the pattern of replies from the first 54 people to answer. The green bar indicates the correct answer. The other two are incorrect.



Although "describe how to..." is a measurable behavior, it typically inadequate in that it is NOT what we want people to be able to do. So asking people to describe, especially in a workplace learning setting, is not usually what we want.

Let's say we're teaching people how to sum a column of numbers in a worksheet using MS Excel. They are not learning how to "describe" how to do it. They are learning how to sum a column of numbers, using the sum function.

In other words, asking people to describe is at a lower level than actual, needed performance. So while it's measurable, it's not really enough.

I teach people to write learning objectives (LOs) that describe actual performance including how achievement is measured. Actual performance-based LOs make writing meaningful mcqs MUCH easier.

---------------------------

If you need to learn the secrets to designing meaningful and valid mcq assessments, consider taking my upcoming Write Learning Assessments course http://bit.ly/dlaw-wla. I'll teach you how to write performance based LOs in the course as well because they are the foundation of good mcqs and the right course content. 

My take: If you design mcqs, you need to know how to do this difficult skill well. I'd love to teach you how.

Why multiple-choice questions are (too often) problematic

Research shows that too many multiple-choice questions are written poorly and therefore create bad assessments. A few of the common issues with multiple-choice questions are, according to research, is that they too often

• are unclear or otherwise poorly written.
• are too easy to guess.
• only test recall of content.
• don't measure what they intend to measure.
• become a test of something other than whether the test taker knows the content.

What's wrong with the following multiple-choice question?

Which of the following is not a good way to put out a grease fire in a pan on the stove? (Select the best answer.)

1. Smother the fire with a metal lid.
2. Smother the fire with water. 
3. Smother the fire with baking soda or salt.
4. Smother the fire with Class B dry chemical fire extinguisher contents.

The correct answer is 2. But research shows that many people who know the answer will get the question wrong anyway. That's because negatively-worded questions are harder to understand and easier to mess up. Answers 1, 3, and 4 are acceptable ways to put out a grease fire in a pan on the stove. And since they are correct, people are likely to select them and get the question wrong.

These and other problems (such as the problems in the list above) lead to invalid questions and assessments. Validity is the most important criteria for a good test. Validity refers to whether the test measures what it claims to measure. If it doesn't measure what it claims to measure, the test answers provide little (or inaccurate) information about what people know or can do. Those tests waste time and resources. If the test is used to make decisions (proceed to next course, prove competence, etc.), poorly written tests are a legal battle waiting to happen.

To make assessments more valid, there must be a very clear match between learning objectives and assessment items. Research shows that this is way too often not the case.

Instructional writing, as I discuss in my book, Write and Organize for Deeper Learning, is different than other kinds of writing. Writing multiple-choice questions is specialized instructional writing. Clarity and readability are critical. But here's something that multiple-choice questions also must do. They must be written so that participant's answers show who knows the content and who doesn't.

In the multiple-choice question at the beginning of this post, the negatively-worded question made it harder to understand. As a result, it was harder to answer correctly. Which makes the answer harder to interpret. If someone selects the wrong answer to the question at the beginning of this post, how sure are we that they didn't know the correct answer? We aren't sure.

Luckily, research also offers clear and actionable tactics for making questions clearer and a better match to learning objectives. I used to do multiple-choice questions writing workshops for companies and higher education staff development. I loved teaching them. But one day wasn't usually long enough to gain the needed skills and I feel a great need to help people gain real skills. 

So I decided to build my first hands-on skills course on assessments and writing multiple-choice questions. It's a critical skill and it's rarely taught. You can learn more (a LOT more) and register. Or ask me to deliver this course for your team. 

Can I ask you to do me a favor? Please tell others about my Write Learning Assessments course and send them the link (bit.ly/DLAW0001). I am building a set of instructional writing courses and this is the first. 

References

Chiavaroli, N. (2017). Negatively-worded multiple choice questions: An avoidable threat to validity, Practical Assessment, Research & Evaluation, 22(3), 1-14.

Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 37-50.

Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied Measurement in Education, 2(1), 51-78.

Hopkins, K.D. (1998). Educational and psychological measurement and evaluation. Needham Heights, MA: Allyn & Bacon. 

Marsh, E. J., Roediger, H. L., Bjork, R. A., & Bjork, E. L. (2007). The memorial consequences of multiple choice testing. Psychonomic Bulletin & Review, 14, 194-199.

Marsh, E. J. & Cantor, A. D. (2014). Chapter 02: Learning from the test: Dos and don’ts for using multiple-choice tests, in McDaniel, M. A., Frey, R. F., Fitzpatrick, S. M., & Roediger, H. L. (Eds.), Integrating Cognitive Science with Innovative Teaching in STEM Disciplines, Washington University, Saint Louis, Missouri.

Roediger, H. L., III, & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 1155-1159.

Schuwirth, L. W. T. & van der Vleuten, C. P. M. (2004). Different written assessment methods: what can be said about their strengths and weaknesses? Medical Education, 38, 974–979.

Shrock, S. A. & Coscarelli, W. C. C. (1989). Criterion-referenced test development. Reading, MA: Addison-Wesley.

Non-Conscious Aspects Of Learning And Performance

Being on autopilot has a lot of implications for learning and performance. Recently, Guy Wallace (@guywwallace on Twitter) posted about experts having difficulties figuring out what people must learn to perform a task. But experts often unintentionally leave things out. Their performance is highly automated so they no longer have conscious access to exactly what they are doing.

Automated and non-conscious prior knowledge is stored in long-term memory. An expert’s deep prior knowledge makes them far more capable of solving difficult problems in their area of expertise. But because it’s automated and non-conscious, they’re often unaware of exactly what they are doing.

Guy pointed me to Richard Clark’s article, The Impact of Non-Conscious Knowledge on Educational Technology Research and Design. And this article turned out to be a goldmine of important information. Experts, research finds, tend to be conscious of the physical actions they take, as well as the knowledge they use. But they are much more unaware of the mental activities used to perform tasks and solve problems.

Read the entire post on eLearning Industry.

Should We Use Background Music With Instruction? No.

The general rationale for not using background music is that it increases harmful cognitive load. Cognitive load relates to mental processes (like perception, thinking, and organizing) used for thinking, learning, and working. Working memory needs to process new information but it has considerable constraints (in capacity for new material and holding time). John Sweller, a well-known researcher and writer on memory and cognitive load and other aspects of learning, reminds us we must design with how our mental processes work. If we don’t, people can’t learn. And learning quickly is a mandate for current organizational conditions.

There are two types of cognitive load: helpful and harmful. We call the harmful type extraneous cognitive load and, when we don’t reduce this type of cognitive load, we make it harder to learn. Here are some examples of extraneous (harmful) cognitive load:

• Too much content
• Decorative and irrelevant graphics
• Unnecessary explanations
• Unnecessary media

Stop reading for a moment and think about why these items cause harmful cognitive load, given what I told you about working memory (Really! Try to answer the question before going ahead). Then look at my answer below.

Read the entire article on eLearning Industry.

Microlearning, Macrolearning. What Does Research Tell Us?

In the last year I have increasingly hear L&D practitioners talk about microlearning like it’s “the answer.” What is it the answer to, exactly? The response: Nearly everything. But knowing that we must create learning experiences that fit specific needs, I felt doubtful. Still, until I understand what the preponderance of research says, my opinion is just a guess based on what I already know. As a result, I set out to learn more and this article sums up what I learned.

What does research say about microlearning? In this article, I’ll offer some definitions of microlearning that offer clues about important aspects and explain what research and researchers have to say about microlearning. I’ll compare what people say are the benefits of microlearning against what we know from research. And I’ll discuss what micro and macro approaches offer workplace learning and how we might use each.

I can sum up much of this article with a specific insight from Professor Christian Glahn at the Hochschule für Technik und Wirtschaft, who studies learning and work:

Microlearning is not the solution to all workplace learning needs.

Read the entire article on eLearning Industry.

How Well Do We Learn From Experiential Or Inquiry Learning Approaches?

Direct instruction directly teaches the content. People are supplied with content and activities that help them build needed background knowledge. And we make sure that what they know is correct and usable. Indirect approaches use experiential or inquiry methods that prompt discovery of needed information and often simulate and test performance.

Training people to identify hazardous materials in the workplace, for example, would likely have lessons, labs, and tests in a direct approach. In an experiential approach, people would likely work through scenarios or case studies.

Paulo Freire, a learning theorist, disapproves of what he calls the “banking model of education,” where teachers (or trainers or instructors) deposit information into students’ heads. Learning sciences clearly shows that we cannot directly fill people up with knowledge (my new book, Manage Memory for Learning explains how we do learn). People do not “record” what they learn during instruction for playback during application.

Read the entire article on eLearning Industry.

Does Time Matter For Learning? It Does.

Stakeholders who request workplace training and other performance interventions often push for speed over quality. Workers are busy and time to learn is time where people could be accomplishing job tasks. We design primarily for speed as a result. Some of the most important learning tactics, such as adequate and varied practice and practice for remembering, are often left out.

For example, sales training for new mobile phones may include phone specifications, images, and diagrams. Designing training for speed too often doesn’t include practice needed for performance. For example, practice over time remembering key specifications helps people use the specifications on the job. Varied practice helping customers select from the newer models for their needs helps people use the specifications in helping people select the right phone. Research shows these types of practice are among key tactics for making training stick and useable.

Speed is a key part of efficiency. Efficiency is the time, effort, and other resources it takes to do something. Efficiency, however, isn’t an adequate outcome unless it also achieves the needed outcomes.

Read the entire article on eLearning Industry.

What Research Tells Us About Chunking Content

Research by usability experts Nielsen Norman Group tells us that people prefer content that is logically divided or chunked. They define chunking as breaking up content into smaller, distinct units of information (chunks), as shown in the right column of Figure 1. This is as opposed to presenting content in homogeneous blocks like in the left column of Figure 1.

Chunking doesn’t mean simply breaking up text into smaller pieces. It means breaking them up into related, logical, meaningful, and sequential segments.

Read the entire post at eLearning Industry.

Does Video Improve Engagement and Learning?

A Huffington Post article title tells us that Research Confirms Video Improves Learning Results. According to the article, the author asked 500 learning practitioners if they believed that video improves learning outcomes. He measured respondents’ opinions of video. No learning outcomes were measured.

A recent article about why we should use video explains that the brain prefers video because it retains visual content better than text content. We know this because people remember 95% of a video message. At least this has an element of (possible) truth based on the picture superiority effect, a researched phenomenon where pictures and images are more likely to be remembered than words (but this effect is debated). But percentages like these have been debunked by many, including my friend Will Thalheimerand the percentage (95%!) was likely made up or based on made up information found elsewhere.

My point: We need to be skeptical about wild claims about video. Or simulations. Or ice cream. The devil is in the details.

See the full post on eLearning Industry. 

Closer I Am to Fine

Are you dealing with a difficult personal decision? I relate as difficult decisions consumed my 2018. I had some important but really hard decisions to make and I was none-too-sure of myself. 

I found the strength to pack up my life and move mid year as it was the clear answer; one I had been struggling with for some time. I also ended a relationship with a family member because she took advantage of me far too often. Letting go of two important but unhealthy relationships was gut wrenching. I realized that accepting bad behavior from others is a choice. I had to make a different one.

Being in emotional pain comes from holding (often tightly) onto to things that cause us damage. We worry that releasing these things will leave a gaping hole in our lives. But the hole can only be refilled with something better after it has been emptied. There is no room before. Simple. Not easy.  

Gradually, a bit at a time, I found a new day-to-day life. New grocery stores, gas stations, restaurants, and movie theaters. Less mindless television to numb the pain. More commuting to see old friends, old local clients, and to the Denver airport for work trips. More deep talks. More walks in the (nearby) mountains. More joy. It's been a short time so my new normal is evolving.

The big ah-ha this year was that I simply had to trust the journey. I tried hard to listen to what the changes and struggle were telling me. Here are some of the most important messages: It's okay to be scared. Everything is figureoutable (thank you, Marie Forleo). One step at a time. And then another.

My friend Phil told me I'd feel disoriented but should realize that the most important parts of my life would remain the same. He was right, of course. The things I love--cooking, fitness classes, writing, and helping people get better results from instruction--didn't change. My work buddy, Karen Hyder (karenhyder.com), who encouraged me to write my research-driven learning books and now uses them in her own (amazing) practice, came to visit me in my new home. Laughing and walking around my new little town with an old friend felt perfect. My friend Steve and his daughter helped me KNOW that I was absolutely where I was supposed to be. 

Each year, I pick a song that conveys this year's main message. This year's is Indigo Girls' Closer I Am to Fine. I relate to these lines especially.

Well darkness has a hunger that's insatiable
And lightness has a call that's hard to hear
I wrap my fear around me like a blanket
I sailed my ship of safety 'til I sank it

In the last few years, darkness has called to me loudly. I had to make it stop before it consumed me. Deciding to and then making the move helped me again hear the soft voice of lightness, telling me I wasn't stuck and I would be fine. I began the difficult journey to becoming fine. If you are facing hard decisions, I wish you the peace, strength, and insight to make them. I get how hard it is.

“New beginnings are often disguised as painful endings."

If you read my articles, books, blog posts, or tweets, or sent me messages on email, LinkedIn, or Twitter you helped me as well. We think of writing as a one-direction activity. But it isn't. I've had many conversations with people who read and comment on my work. Some people have done a lot to help me but didn't have to. Special thanks to Mirjam Neelen (@MirjamN), Will Thalheimer (@WillWorkLearn), Bill Sawyer (@billsawyer94566), Guy Wallace (@guywwallace), and Christopher Pappas (@elearnindustry) for all of your help. 

I wish you a "fine" 2019. 

Pictures From Berlin (#oeb18)

I just returned from Online Educa Berlin 2018 (#oeb2018), a conference that brings together higher education and workplace learning practitioners from mostly Europe. I’d heard of the conference and knew it was considered one of the best adult learning conferences in the world. So, I was super excited when they asked me to present and knew it’d be a great experience.

The trip to Berlin for the conference began with me leaving my coat in my home airport. (Yep. Extra dumb. Walking around Berlin in the winter without a coat is a bad idea.) When going through security, the TSA took out my coat and some tools from one of my bins. Everything else came right through and I didn’t realize that the coat and tools were missing. I picked up my luggage and purse and walked off. I remembered about 10 minutes before boarding my flight that I didn’t have my coat. Too late to look for it.

The first morning after arriving, I met up with my friend Mirjam Neelen (twitter @MirjamN). We went coat shopping :-) and did some walking around the city. That night Will Thalheimer (twitter @WillWorkLearn) met up with us and we walked around looking for dinner.

The next day, Thursday, I was involved in the OEB Plenary Debate which was about whether all learning should be fun. There were two people debating each side. I debated on the “Against” side with Alex Beard, the Director at Teach for All, UK. The “For” side included Elliott Masie, the head of the Masie Center, USA, and Benjamin Doxtdator, an educator at the International School of Brussels, Belgium.

My answer in a nutshell: Learning can be fun, but deep learning (for skill and expertise) is often a lot of effort and often requires pushing through the not-fun. Here’s a picture from the debate. The room was large and each of us was on camera when we talked. I was nervous (massive understatement). You can watch a video of the debate here.

The following day I did a short workshop on Managing Memory. Here I am at that session with two workplace learning professionals who traveled to the conference from Poland. They asked to take a picture with me and I clearly needed to stand on a chair so I didn’t look child-sized. The picture belongs to Tomasz Jankowski (Twitter: jankowskit), who is on the left. Tomasz wrote a blog post about my session. It’s in Polish but Google Translate does a good job of making it readable in other languages. The person on the right of the picture is Bartlomiej Polakowski (twitter: @b_polakowski), a long-time twitter friend.

Before leaving, I got the chance to meet Donald Clark (Twitter @DonaldClark, blog https://donaldclarkplanb.blogspot.com) in person, who is one of the best minds in our industry.

A Moving Story

Gateway to the historic part of Monument, Colorado
http://www.coloradospringsweb.com/photo/Monument-CO/historic-monument-gateway-panoramic 

For the past few years, I have struggled with some major difficulties. We all have them. I've come to realize that pretty much everyone is struggling with something difficult. Sometimes really difficult things. Sometimes many difficult things.

I've had very bad migraines the past 15+ years. (Found a large piece of what reduces them to manageable just recently so if you're struggling with daily migraines and no meds seem to help, feel free to reach out.) Recently I added getting divorced. (Not so great for migraines, by the way.) People inevitably change over 30 years and sometimes where they each end up is not compatible with where their spouse ends up. We're friends.

I recently moved from the Denver metro area (Colorado, USA) to Monument (Colorado, USA). I picked it because it's small, has locally owned shops and restaurants, and there are free Wednesday night concerts in Limbach Park. (New band discovered: Dreamfeed... they're amazing).

Limbach Park

And I can do something regularly that was more difficult in suburban Denver: get to trails within ten minutes. Below is a view from trails on Mt. Herman looking eastward...

View from trails on Mt Herman

This experience, especially the move, helped me remember that each time there is a change in our circumstances, we are forced to learn. Many times we don't want to learn... we want things to stay the same. It's easier to coast. But either we learn or we stay stuck.

Learning helps us get back to our path (or change it, intentionally). Staying stuck does not. To survive under change (job change or loss, relationship change or loss), we simply have to allow for learning and the discomfort of instability. Feel the pain and learn and change anyway.

Mt Herman trail view

If you've gone through a difficult change and are willing to share what you learned and how you opened yourself to it, I'd love to read your comments.

Next Button: Yay or Nay? A Research Perspective on a "Pointed" Question

Research offers important clues to difficult and complex questions. (The reason I say clues and not answers is that science is a moving target and some research is not done well or is not applicable to the specific question.) I find that research doesn't always tell me what I expect, however.

When research gives me unexpected or surprising clues, I sometimes have to overcome my biases, including the natural desire to look for and apply solutions that agree with what I personally think is true (confirmation bias).

Can I let you in on a secret? I held some views about learning that research showed me were off base. (Shocking!) It happened regularly when reading and analyzing the research to write Write and Organize for Deeper Learning and Practice and Feedback for Deeper Learning. For example, I learned that instructional feedback has a different purpose than I realized. Although my previous view wasn't wrong, it lacked the big picture. When reading the research on writing for instruction, for example, I learned a lot about readability research and how applying these insights improves comprehension.

The research that I read recently on the next button offered insights that I hadn't considered earlier as well.

Next Button: Force It Down People's Throats. Or Put it In Jail.

I started with a search to see what L&Ders are saying about the next button. And this heading jumped out at me (slightly changed to protect the writer):

Disabling the Next Button Until Learners Finish the Slide.

Whaaaat? Sadly, I thought I knew why (which is why the heading jumped off the page) as I've built many eLearning courses. Stakeholders or clients wanted to disable the next button so people are FORCED to stay on the page. But is this what this article was discussing?

It was exactly what they were discussing (slightly changed):  

You can prevent users from moving forward to make sure people perform specific actions (for example, listening to the entire video or clicking on certain interactions) by...

So one point of view is that we should use the next button to force-feed content. (No!) Another point of view is that the next button is the just awful and we shouldn't use it. Here's a statement made on a L&D blog post (slightly changed):

You shouldn't use the next button for navigation. Have learners choose their own path. 

Except... research does not support this statement (discussed in detail in my eLearning Industry post).

Okay, that's enough for opinions. What does research support?

Next Button: Useful

The next button (and the player controls) are simple interaction tools that help people control pace.

Mayer and Chandler's When Learning is Just a Click Away: Does Simple Interaction Foster Deeper Understanding of Multimedia Messages? researched the learning benefits from simple user interactions, specifically in multimedia explanations. First, some definitions:

• Multimedia explanation means words (for example, text or narration) and pictures (for example diagrams, images, video, or animation) that show how something works. 
• Simple user interactions give users control over the words and pictures they see.

These are both common in eLearning.

Simple user interactions, according to the authors, includes the next button (go to next segment). But they might also include player controls (stop, play, go forward, rewind) as they have a similar purpose. Their conclusion is that these simple types of user interaction have two positive and important impacts on learning. They:

• Reduce the load on working memory 
• Allow people to build understanding a piece at a time.

When multimedia explanations are force-fed at the designer's pace, the flow of words and pictures can move too fast for needed processing. Processing is needed to learn. Working memory processing capacity is limited so force-feeding "instructional stuff" means likely means less processing and confusion. (So the force-feeding view of the next button can actually damage learning). It's harder to make sense of information and mentally integrate it with what we already know (prior knowledge) when multimedia explanations are too long and the user cannot control them.

What we are talking about here is control of the pace. This is one of the learner control issues I discuss in the eLearning Industry article I mentioned earlier in this post. That article offers more information about when learners should have more control and when they shouldn't.

The bottom line is whenever possible, we should let learners chose the pace. Breaking up segments with a next button (or player controls) allows people to continue to the next segment when ready.

When I discuss this with other L&D practitioners, they sometimes tell me that people learn on their own all the time, so why be worried about this? Yes, people do learn on their own, and it's easy to get frustrated when doing so (ask me about learning Tableau, a data visualization tool). My point here is not that we cannot learn on our own. We can. We do. It's this: If we are going to design instruction, it should be done using what we know helps people learn.

So maybe we should stop treating the next button like Clippy and give it some respect. Especially when we use it to control pace.

Comments? I love them.


References:

Chandler, P., & Sweller, J. (1991). Cognitive load theory and the format of

instruction. Cognition and Instruction, 8, 293-332.

Chung, J. & Davies, I. K. (1995). An instructional theory for learner control: Revisited, Proceedings of the 1995 National Convention of the Association for Educational Communications and Technology, 72-86.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge Univer­sity Press. 

Mayer, R. E. & Chandler, P. A. (2001). When learning is just a click away. Does simple user interaction foster deeper understanding of multimedia messages?. Journal of Educational Psychology, 93 (2), 390-397.

Shank, P. (February 19, 2018). Microlearning, macrolearning. What does research tell us? eLearning Industry. https://elearningindustry.com/microlearning-macrolearning-research-tell-us 

Shank, P. (April 16, 20) https://elearningindustry.com/learning-participants-control-research-says 18). https://elearningindustry.com/learning-participants-control-research-says

The MOST Important Thing

A few days ago, someone asked me a most interesting question: What is the MOST important thing L&D practitioners can do to improve the instructional content they build?

There are MANY truly critical things (I describe truly critical strategies in Write and Organize for Deeper Learning and Practice and Feedback for Deeper Learning). But there's one thing that makes those other things possible:  Deeply understanding the jobs our content supports. If we don't understand the tasks and outcomes of these jobs, how can we...

• Deliver what people really need?
• Design for relevancy?
• Offer the right content and activities?
• Provide what's needed at the right time?
• Write at the correct level?
• Provide needed support?

Simple. We can't. Building for anyone is the same as building for no one, especially for workplace instruction.

People don't have time or motivation to slog through things that are not relevant to them, especially in today's chaotic and fast-paced workplace. And if it's "compulsory," they will learn only enough to pass whatever barrier we provide (assessments, etc.). This is shallow learning but deep learning (and here) is what's needed for people to be able to use what they learn.

Training research shows relevance is a critical factor for transfer of trained skills to the workplace and to motivation to learn and persevere. Without these, our efforts become busywork for people who already have too much on their plate.

Far too many L&D practitioners think their job is to take content from a content expert and put it into instructional authoring tools and output it to [whatever]. Nonsense. We may do that but that's not our chief responsibility. Our chief responsibility is to use knowledge of jobs and the organization to help people grow skills and performance so that organizations can remain viable. We simply cannot do this without a deep understanding of the work and organizations we support.

Guy Wallace (Twitter@guywwallace), website https://www.eppic.biz) has some great articles about how to understand the job and why a single content expert cannot give you all the information you need for valuable instruction.

For now, I'll leave you with a few ideas to help you better understand what people do and how what WE do connects with their job.

• Understand what your organization does and how it does it. In detail. Find out its major pain points and what gets in the way.
• Spend time in different departments to better understand what they do and how it impacts organizational success.
• Watch how key tasks are done and document the process. Ask good performers to evaluate your documentation for holes and flaws. Ask questions.
• Review tangible work products and see how they are used. 
• Watch workers work. Ask questions about their processes, their work, what doesn't work, and how what you want to teach connects to what they do.
• Ask tons of questions when someone asks for training: What specific results are needed? How do you know there's a problem?  What gets in the way of producing needed results? If they cannot answer, get the answers from people who can answer them.
• Figure out what foundational knowledge and skills are needed to perform the tasks. Do people have it? What happens if they don't?

Understanding your organization, the business they are in, and how jobs support organizational survival isn't nice-to-know. You simply cannot be effective without this understanding. Content development is not enough. Not even nearly.

Reflections on the Learning Technologies Conference (#LT18UK)

First thing: Plan to come to the Learning Technologies conference. Learning from L&D folks from the UK, Europe, and elsewhere gives you a wider view of workplace learning around the world. And having international L&D connections are priceless. How else would I have had the opportunity to meet Lesley Price's cats? And work with the amazing Mirjam Neelen?

Although US learning conferences often have some international presence, most of what we discuss relates to what we do in the US and Canada. Globalization has changed learning and work. It's not enough to know how workplace learning supports organizations where you live. Globalization means understanding work globally and how to support it.

The conference is a great place to meet, discuss, and learn from others. Donald Taylor knows how to put together a terrific conference and personally understands global workplace learning. The Expo is 10+ times larger than any you've been to in the USA. The conference is medium sized and you don't become a small dot in a sea of dots. It's easy to walk up to just about anyone and ask about their work (and it's a great way to learn).

I spent most of my time either talking to people, preparing for our presentation (Mirjam Neelen and I discussed how technology and globalization are changing the workplace and how that changes L&D's role), or answering questions about my work. Here are some impressions from the sessions I was able to attend.

• A LOT of people are investing in microlearning like it's THE ANSWER. I'm writing an evidence-based (research-driven) response and it'll be out soon.
• There is a wide disparity among organizations in the use of learning technologies. Some are just getting started and others are looking at ways to use virtual reality and augmented reality. We need to be very careful to meet organizations where they are.
• There was a terrific session on globalization and women in our field. I saw that globalization has impacted the careers of other women just as has affected and is continuing to impact mine. 
• Connie Malamud, Will Thalheimer, Mirjam Neelan, and I discussed research-driven approaches! Thank you, Don Taylor, for allowing us to put forth our research-driven L&D practices coalition (I just made this term up... it's not a thing. Yet.)

To help you adopt a more global understanding of our field, here are just a few of the L&D peeps I got to meet at Learning Technologies to follow on Twitter:

Jo Cook @LightbulbJo

Kate Graham @kategraham23

Mirjam Neelen @mirjamn

Lesley Price @lesleywprice

Julie Drybrough @fuchsia_blue

Barbara Thompson @CaribThompson

Rob Hubbard @RobHubbard

As usual, the best part of conferences is learning from each other. Learning Technologies is an especially good conference for this!

Poll results: Which books come next?

Last week I asked people on Twitter to help me decide which book to write next by answering a simple poll question. (And many of you retweeted... thank you!) I posted a question listing six books. Respondents could select any or all of the books. Here are the results:

Survey question and results

I received 39 completed responses. And that leads to my dilemma: Should I write the learnable eLearning book next? The results suggest that this is a good idea. But in survey analysis terms, these results are not necessarily significant. When doing surveys for decision making, you want to know that your data is "representative" and not chance. Representative means that it accurately represents the answers of the entire group. My "entire group" is everyone who reads or will read my books.

There are many reasons why this information may NOT be representative (biased).

• There aren't a lot of responses and data from fewer people is less likely to be representative than responses from a lot of people
• I only asked people on Twitter and my extended audience is unlikely to all be on Twitter
• People who didn't answer may have different opinions from people who did answer

So what should I do? The truth is, I'm not sure. I wrote the first two books, Write and Organize for Deeper Learning and Practice and Feedback for Deeper Learning based on what I thought were needed. And they are doing well!

Am leaning towards starting on the learnable eLearning book next because I know this information will be super helpful. For example, what does research specifically tell us to do differently when writing for mobile? I spent time finding the applicable research on this topic during the recent holiday break. And the research is fascinating and actionable.

Writing a book, especially one based solely on research, is a heavy undertaking. That's because it takes a lot of time to search for widely applicable (generalizable, in research terms) research. This is made even more difficult when searching for training research rather than the more widely available learning research ( which is typically K-12 and higher education).

If you have advice, please let me know. I hope to have the next book ready by May.  If you are one of my readers, THANK YOU. I absolutely love doing this.

The delusion of attention

Two Things at Once - AT&T TV Commercial 2013

In the AT&T commercial above, a man asks a group of children, "What's better, doing two things at once or just one?" All the children yell, "Two!" The commercial ends with "It's not complicated. Doing two things at once is better."

Except AT&T is wrong. It might seem better. But our cognitive abilities are not built to allow us to do two things at once. Hell, we often have a hard time doing one thing. (Or is this just me?)

I just finished reading A Deadly Wandering, written by Pulitzer Prize-winning journalist Matt Richtel. I wanted to know more about how technology impacts attention because attention is critical for learning. Since we regularly use technology for workplace learning, does technology positively or negatively impact our ability to attend? Most people in our field seem to assume that technology is good for learning. Here's my take: Assume nothing. Follow the evidence.

A Deadly Wandering, by Matt Richtel

In the book, we follow Reggie's story along with how neuroscience experts see his story. Reggie Shaw makes a tragic mistake and accidentally kills two men while texting and driving. The story follows the accident, Reggie's prosecution, and what we can learn as a result of his horrific mistake.

It also follows what science knows and is learning about attention and technology. Richtel offers insights throughout the book from Adam Gazzaley, M.D., Ph.D., Professor of Neurology, Physiology, and Psychiatry at the University of California, San Francisco. And from David Strayer, PhD., University of Utah.

Earlier attention scientists found that neural conduction (how the brain and nervous system transmit signals and commands) is much slower than expected. Mental processing is not instantaneous. As we increase demands on mental processing (too much to look at, process, think through, make sense of), we "lose the thread" and make mistakes. Add in technology and demands on mental processing can increase a lot, sometimes exponentially.

For example, when we take our attention away from driving for a second or two (for example, to see where something dropped on the floor or to talk to someone in the car), it takes many seconds to regain full focus and understanding of what we are seeing and doing. Texting and driving don't only impact processing of the road while looking at the phone (which is terrible enough). It affects the ability to process what is happening around us for many seconds after looking at the phone. It's like no one is driving for those seconds because thinking is elsewhere engaged.

My insight: Anything that diverts attention while learning (music, interruptions, bells and whistles, going elsewhere to get a question answered) makes it MUCH harder to return to full focus.

People simply cannot learn or perform well under these conditions. Reggie's accident was a prime example. Anything else we try to do while driving, diminishes our ability to perform. Anything else we have to deal with while learning diminishes our ability to learn.

The bottom line here is that attention can be a delusion because our processing capabilities are quite limited. We easily fool ourselves into thinking we can do more than one thing at a time. But focus and attention are fragile and we are easily distracted. And while distracted, our processing capabilities are elsewhere.

Richter explains that the distractibility of the technologies in our lives and work have increased a great deal and continue to increase. To keep up we feel we must multi-task. But this is not how our minds work. We cannot change how we process and the speed of processing so we must change what we do to accommodate what is so.

In this section of the book, I want answers to the following questions.

• How can we keep technology from capturing attention away from learning?
• How do we learn while facing an onslaught of distraction information and demands?

Feel free to offer your insights and answers. I'll be back soon with more insights from A Deadly Wandering.  

Thank you, 2017

This has been a good year. Not all good, of course. Everyone's life has enough pain to keep it real and mine is no different. My most favorite kitty, Cleo, left this world and many months later, I still look for her in the bay window in my office. I used to squeeze three cat beds into the window and now there are two. :-( 

Redd (Minnie's brother) has taken Cleo's place on my lap. But he refuses to let me read on the couch. I can watch television there, as long as I have a certain blanket on my lap that he likes to lay on. But no reading. Books evidently distract me too much from my petting duties. The cover of the book on attention and technology that I'm currently reading has teeth marks in it.

Cleo

Redd and Minnie

There have been other problems, but I've learned (lately) that sometimes you just need to sit with them and let them be. I'm a doer and that's truly hard. But often, problems are opportunities to see how much you have and what you don't have that you don't want.

I've been thinking a lot about where I was last year at this time. Many people know I have chronic migraines and they're not easy to live with. I didn't realize until recently that migraines can actually do damage to your brain. Nothing I'd done in the last thirty years had really improved the situation, and the pain and other effects became more frequent. The drugs had pretty horrible side effects as well.

This year, in desperation, I started Angela Stanton's migraine protocol (her book is on Amazon). Nothing else worked well so my expectations were low. But only two to three months later, my migraines have lessened by by 90% (and I believe they will be gone, eventually). I am slowly reducing my meds and feel better than I have in years.

Two years ago, I started writing two evidence-based training books. My first efforts were not loved (by any means) by the people who agreed to review them. I was depressed at my results. But after a few months, as my brain subconsciously worked on solutions to the problems the reviewers described, I thought of how to not only start again, but do it better. The result has been two books (so far) that people have valued. As a result, I'm going to start on the next two in 2018.

I have deep gratitude to the early reviewers, who planted the seeds of more valuable efforts. It's not always fun to learn you're on the wrong track. But if you push through, you can learn a great deal from failure. Especially if you're open to what you don't (yet) see.

Most years, there is a song that fits the year (for me). This year it's Happy People by the country-pop group, Little Big Town. Here's the youtube video. (Hope it makes you smile.) My two favorite lines:
Happy people don't fail
Happy people just learn

Thank you for reading what I write, telling me where I need to do a better job, and giving me the opportunity to do just that. I hope your 2017 was also a chance to grow and learn and see what is absolutely terrific. I think that's the key to sanity.

L&D's fake news

L&D has its own version of fake news: fads, made-up theories (and how to implement) in conflict with what research or science says, and people selling ideas using devious means. I just had the latter happen. Just had an email sent to me that said the following (copied directly from the email):

Hi Patti

Just wanted to let you know we featured you in our recent post: 
Why Employers Should Consider Multiple Learning Styles in Their Training Processes
https://www.bobtheba.com/blog/20171128/why-employers-should-consider-multiple-learning-styles-in-their-training-processes.

Someone used me to push learning styles for training? Noooooooooo.  As you likely know, the research is clear that learning styles aren't "a thing." (See Learning Styles: Concepts and Evidence by Harold Pashler, Mark McDaniel, Doug Rohrer, and Robert Bjork to learn more.) Like so many ideas in training, it's based on people trying to make money off of the less knowledgeable or made-up insights that have nothing to do with what the research says. In fact, research shows that quite often, what is best for people for learning is the opposite of what they prefer. And this makes sense as isn't what's best for you nutritionally the opposite of what you love to snack on?

I get why they used my name. People are likely to tweet or tell others about articles that include their name. See, I just did! I hope I don't see my name associated with an article about neuroscience and learning.

No, we AREN'T learning a lot from neuroscience that we can use in learning. It's cognitive science, not neuroscience that most of these people are talking about. But calling it Neuroscience or Brain Science gives it the woo-woo factor. Except it's a lie. John Medina, the author of Brain Rules tells us we simply don't know much yet from neuroscience we can use. The research that might help us understand how neurons impact learning (fMRI studies) have been recently have been called into question for having software that creates errors.

What sciences can we count on to tell us what works? Here's a short list to start (there are many others).
Attention science
Cognitive science
Reading comprehension
Education psychology
Behavioral psychology
Information design
Usability

I'm annoyed that L&D practitioners buy the crap that others (including our own practitioners) put out there to part you from your money and gain recognition for their efforts. You must learn to tell truth from lies in order to do what's needed for good training results. But that's a good thing. Thinking for yourself is always a good thing.

I asked the company that sent that email to take me out of the article. Will let you know if that happens.