As a Faculty Member, What I Need To Do About Quality in Online Learning?

Printer-friendly versionPrinter-friendly version

The most common concern of learners and faculty about online programs and courses are issues about “quality”, by which they mean:

  • Is this program or course the same in terms of content, skills and knowledge to a program or course I would teach or study in a classroom?
  • Will there be mindful and meaningful interactions between instructors and learners, the learner and other learners, and between learners and the course materials?
  • Will there be full and valuable feedback on assignments and projects, just as there would be in a classroom environment?
  • Will learners get the support they need from instructors, technology support staff, library services and others when they need it?
  • Will adequate time be given for quality interaction and instruction?
  • Will the learning outcomes be those required for the course?

Quality assurance frameworks have been established, which seek to ensure the answers to these and related questions are positive, at least as far as the institution is concerned (for example, see here and here).

But what does a faculty member need to do so as to ensure that their answer for their courses to these questions is all positive and that their students agree?

Faculty need to engage in a process of instructional design for engaged learning. This document will describe this process in a way intended to help a faculty member meet the quality expectations of the quality standards for online learning.

As you look at these materials, understand that the idea is that quality is “designed in” to all aspects of the course not “inspected in” at the end, and that quality is as much about the way in which the design of the course enables the learner to engage in learning as it is about the outcomes of that learning.

Instructional Design

Instructional design began in the 1960s as a way of thinking about purposeful and planned instruction, especially for technical skills and trades. It quickly developed into a discipline appropriate for all forms of teaching and instruction, with a variety of models and frameworks all based on theories of adult learning and motivation. In the late 1960s, when open and distance education was developing rapidly, significant use was made of instructional design both to give shape and structure to course development, but also to ensure quality.

The dominant instructional design framework for course development that has stood the test of time is ADDIE – an acronym for analysis, design, development, implementation and evaluation. Originally developed at Florida State University for military in-service training, it now is in widespread use globally as a framework for designing effective quality learning at any level (Morrison, 2010). While there are other instructional design frameworks (see here), ADDIE is the one used most extensively and is the backbone of many other approaches, which then add components to the basic ADDIE model.

What follows is a description of ADDIE, presented as a set of questions it will be helpful to explore to ensure that quality is designed into a course or a revision of a course. Quality is based on how you use the analysis to shape design and development.

Analysis

The task of designing a course goes through a number of phases, some of which are undertaken in parallel. In the analysis phase, the faculty member, working together with instructional designers and technology advisors, asks and seeks to understand the responses to these questions:

  • What kinds of learners will register and seek to complete this course – what are their strengths, weaknesses and typical learning challenges for this kind of content?
  • What are the desired outcomes for knowledge, understanding, skills and competencies and what do the students bring to their learning that will enable or hinder these desired outcomes?
  • What type of delivery options do I have – videoconference, audioconference, learning management system activities and online dialogue, simulations, games, project-based group assignments? What has been done before online for this content that has been shown to work well?
  • What open education resources (free-to-use learning materials) can I easily incorporate into my course to add proven components?
  • Are there any other teaching considerations – sensitivities, known areas of difficulty, common mistakes that need to be factored into the design?
  • How much time is there to complete the development of the program or course, have peer review and make sure the technological requirements function before students start to study this course?
  • What supports are available to me as a faculty member to support my development of this course – library services (especially for intellectual property and permissions), instructional design, support for assessment activities and design, technology supports?

These questions may be answered quickly or may take some time, but provide the basis on which the next phase of work can build. In considering these questions, faculty talks with each other, with colleagues in technology support and with instructional designers.

By exploring these questions, some design issues may emerge. For example, if the students who will be taking an online course have challenges with literacy, does this mean that it would be better to make more use of video and audio segments, supported by text rather than offering the learners a substantial text-based course? Issues concerning intellectual property and access to some materials may also constrain what can be presented to learners online, which is why librarians and intellectual property managers should be involved early in the analysis, design and development phases.

Design

The second phase is the design phase where a blueprint for the course is established. This is not the same as actually developing the course – the focus here is on structure, sequence and the teaching and learning strategy rather than with the specifics of what the students will study. The typical faculty member questions at this stage are:

  • Given the length of the course in time, how can I divide the work of the course in terms of learning outcomes for each unit of time? For example, week by week, what outcomes do I want leaners to master and complete for each week?
  • At the end of a body of work, what kind of assessment will help the learner and the instructor determine what progress has been made in mastering the knowledge, understanding, skills and competencies expected of them?
  • Given the areas which we know learners find difficult, do I need to structure these components of the course differently from other aspects of the course?
  • Where should assignments fit into the schedule and what kind of assignments should they be? How can I ensure that learners receive timely feedback from their assignments so as to shape their learning?
  • If I want to use video, simulations, games or other resources, where do they fit in the schedule, how can I make best use of these resources and what constraints do they impose on how, where and when learners can learn?

In this phase, the faculty member is not developing the course materials or assessment – instead, they are defining the overall shape and “architecture” of the course. Many use a basic “storyboard” to capture the key components of the design – read more about this idea here.

In face-to-face instruction, some look at this phase and say “ah, a course outline!” As you will see from the questions being asked, this is rather more comprehensive and thorough. It is “a course outline on steroids”, as one faculty member described it. Such a comprehensive storyboard makes development much easier. It also enables peers, who should be given an opportunity to review and provide feedback, a chance to have an influence on the design of a course before too much energy has been devoted to developing the course in detail.

You can look at best practice guidelines for course design here, and explore what this looks like in practice by taking a look at the key elements in the design for a course available here.

Development

Given the design storyboard just developed, the work now begins in earnest. The faculty member, working with instructional designers and others, seeks to assemble all of the materials, activities, assessment tools and resources needed for the learner. Sometimes this involves writing new materials, developing commentaries or analysis tools for textbook chapters or readings, new small group activities or using third party open education resources (OERs)[1].

For example, an instructor teaching statistics might want to make use of freely available science simulations from the University of Colorado at Boulder (here) or the environmental games from Ecogamer (here) or the Open University materials in social science, arts and humanities (here).  They will need to make sure that they correctly acknowledge the source and respect the terms of the Commons License under which it can be used (see here for details).

They may also want to ensure that the OERs they do use meet the guidelines for quality, such as those suggested by the Commonwealth of Learning (available here, see especially section 2.3).

Development takes time. For each week of learning activity, the generally accepted assumption is that some two to three weeks of development will be needed (Khan, 2005) – this can be greatly accelerated by the systematic use of OER.

From a quality assurance perspective, the full course (design + developed materials) should undergo peer review by individuals fully cognizant with the subject matter covered by the course and familiar with online learning. Such peer review provides an opportunity to validate design assumptions, test the veracity of the learning process and review assessment and evaluation rubrics. Such a peer review also enables exchange of ideas and an opportunity for faculty development.

One particular focus for design should be around student participation and interactivity. We can think of four levels of interactivity:

Level

Description

1: Passive

The learner is receiving and exploring information on their own – reading, watching videos, listening to a podcast, exploring the web. They are not interacting with other students or with faculty. This is their self-study time.

2: Limited Interaction

The learner makes simple responses to questions, quizzes and cues. For example, they respond to a quiz which then provides feedback to them automatically about their knowledge, skill and understanding. Or they are engaged in a simple dialogue with the faculty member.

3: Complex

The learner may be engaged in a simulation as part of a team, engaged in a dialogue online where they are asked to share their ideas and materials or engaged in a project group.

4: Real-Time Participation

The learner is presenting, either on their own or as part of team, a set of ideas or skills in a real-time video or audioconference or working with others in a real-time simulation.

 

As you develop the course, you need to make decisions about what time of engagement and interactivity you are seeking to encourage for each time period (e.g. each week) and for each key skill or learning outcome.

Both at the program level and the course level, faculty need to give consideration to the overall “architecture” of assessment. For example, over the course of a program of studies in a specific discipline, what kinds of assessment will the learner be asked to complete and are there a sufficient variety of assessment processes to assess learning outcomes appropriately? If every course in a program uses the same form of assessment (e.g. multiple-choice or the long-form essay) are we seeing a sufficient range of abilities from the learner?

There are no “standard” answers to this question, but it is one which faculty needs to consider – quality assurance reviews at the program level will look at this question.  

Some suggestions for faculty working on a course within a program are:

  • Within a course and across courses within a program, there should be a variety of assessment approaches.
  • Assessment should enable the assessment of both what students have learned and how they can apply that learning.
  • Assessment should not just focus on learning outcomes, but also on the learning process – learners need to demonstrate that they know how to learn in relation to the discipline they are studying so that they can maintain their learning as new knowledge and understanding emerges in that discipline.
  • Assessment rubrics and processes should be transparent and fair.
  • Assessment is of two kinds – formative (helping the learner see where they are in their learning through feedback) and summative (a grade which counts towards the final mark for the course). In online learning environments, both are important and learners expect feedback from a formative assessment before they face a summative assessment.
  • Assessment can look both at what an individual is able to demonstrate in terms of knowledge, skill and understanding or what a team can demonstrate – assessing team performance is also possible using simple assessment tools (see here for an example).

Faculty members should review the assessments used in related courses to the ones they are developing so as to ensure that learners experience both variety of forms of assessment and consistency in terms of the rigour of assessment. Quality assurance reviews of assessment will look at the rubrics, forms of assessment and rigour by sampling learner work and comparing that work with the rubrics established for the course.

There are generally five categories of assessment activities in widespread use in online learning (see here for a research study looking at the extent of their use). These are:

  1. Written assignment: This large category encompasses assignments like research papers, case-study responses, and short essays.
  2. Online discussion: Assessments in this category include any asynchronous discussion activity undertaken on a discussion board, blog, or wiki.
  3. Fieldwork: This is a special type of written assignment requiring students to collect field data and write up some kind of report.
  4. Test/quiz/exam: In this group are traditional assessments composed of multiple-choice or short-answer questions.
  5. Presentation: This category includes student presentations; given the absence of face-to-face communication, the presentation delivery format had to be adapted for the online environment.

The design issue is which to use when and what weight can be attached to each one.

Many faculty members take a pragmatic approach to the design phase for a course, summarized in these steps:

  1. Step 1: Identify the course to be developed – what the learning outcomes have to be, how it fits into programs offered by your institution and what the successful learner will be expected to know.
  2. Step 2: Collect and review all of the materials (books, videos, open education resources, readings, podcasts, etc.), which you think may be relevant to the course as offered online. Make sure you have permission to use these materials (check with the appropriate administrator with respect to copyrights and intellectual property).
  3. Step 3: For each week of the course, identify the learning objectives, activities, required interactions, the skills and knowledge mastery required and assessment activities which will be needed to ensure that students are mastering the required knowledge and skills.
  4. Step 4: For each assignment associated with the course, define the requirements and the associated rubric for assessment/marking.
  5. Step 5: For each student of the course, identify the assumptions you are making about their prior learning and skills (pre-requisite knowledge and skills) – what do they need to know before they start the course to be a successful learner?
  6. Step 6:  For the student who is doing well in the course, identify additional reading or learning activities that may assist them in deepening their knowledge and skills for each week of the course.
  7. Step 7: For the student who is struggling with the course, identify activities which may help them improve their knowledge and skills for each week of the course or an alternative method of remediation for each week.
  8. Step 8: Identify activities or parts of the course which require synchronous interaction between students and instructors. Specify the purpose of this interaction.
  9. Step 9: Identify activities or parts of the course which would benefit from peer-to-peer interaction (not in real-time or in real-time) or interaction between students and the instructor (not in real-time or in real-time).
  10. Step 10: For each activity or component of the course, identify its expected shelf life – “when do we need to revise or review this material to ensure that it is current?”

This seemingly simple approach to the work of design requires the analysis work to have been done before you start.

Design is the process by which quality is “built into courses” – each step above requires considerable thought and preparation so that the learning outcomes and experience of the course can be of sufficient quality to enable learning to be effective.

Once developed, courses need to be reviewed by peers who are knowledgeable about the subject matter and have an understanding of how such a course can work online. This peer review process can be informal (asking colleagues to look over the material) or formal – using independent peer reviewers and a formal rubric (see here  and here for examples) to assess whether the design and development represents a quality course.

Implementation (sometimes known as Deployment)

The course has been designed, developed and reviewed, and now it is time to offer the course to learners. In large scale dedicated online learning organizations, it is sometimes the case that new courses are piloted (sometimes for credit) before being offered to a wider audience.

For example, when the Open University (UK) or the Indira Gandhi University (India) plans to launch a new course, it sometimes pilots with an initial small cohort before launching to a mass audience – some courses have more than two thousand learners. When this occurs, learners are often given a discount on their tuition in exchange for more frequent evaluation, focus group and other forms of feedback. They also can secure appropriate credit for the successful completion of assignments.

A lot can happen at this stage – course materials may not be available as expected, technology may not function as expected for all learners (especially if rich multimedia resources are being used by learners who “bring their own devices”), more learners struggle with key concepts or activities or assignments than anticipated. From a quality point of view, faculty and support staff needs to be efficient and responsive to these issues, making adjustments as needed and ensuring that learners receive the best possible learning experience from the course.

Faculty should keep a “log book” of things to improve, redesign, remove or supplement. They may also want to maintain a dialogue with learners about how they are experiencing the course so that the log can be enriched with learner insights about what works and doesn’t work.

Increasingly, so-called “expert learners” are being used to test and experiment with courses to debug them before they are offered to the general student body. Such learners are also valuable sources of information as the course progresses – as they experience the course, what ideas do they have for improvement both in terms of the mastery of knowledge, skill and understanding, and also in terms of learner engagement?

The faculty member, who designs, develops and teaches a course, is busy – good course design should lead to a lot of interaction and engagement. Faculty need to make clear to learners what the expectations are for their involvement as faculty in the course. Some faculty take the view that they must respond to every posting and interaction by learners – this can be very demanding if there are 30-40 learners registered and active in a specific course. The key concern is that the learner understands how the faculty member is available to them, for what, when.

Here is an example of clear guidance to learners about what to expect from a faculty member during the delivery of the course itself:

What You Can Expect from Me as Your Instructor:

  • I will respond within one working day to questions and queries related to the course (on many occasions it will be the same day).
  • I will review all postings in our weekly discussions and make comments on some and ensure that all of you are making timely and appropriate contributions.
  • I will be available each week on Mondays and Wednesdays between 9:00 a.m. and 10:30 a.m. for a telephone conversation by appointment – use the “I would like to talk to you” button on the course page to make an appointment.
  • When you submit either your team or individual assignment, I will review, provide feedback and a grade within four working days of receiving it.
  • I will post updates on the Your Professors Thoughts page relating to recent events and developments relevant to this course, so you can connect what is going on in the world to the materials in this course, including relevant and recent research.
  • I will not take calls at home, except in a genuine health emergency.
  • I will not guarantee availability to respond to telephone calls made to my academic office outside of the hours given above (Mondays and Wednesdays, 9:00 a.m. – 10:30 a.m.).

Evaluation

There are a variety of forms of evaluation which need to be considered when the course is being reviewed by the faculty member. These include, but are not limited to:

  • End of course evaluations completed by the learners, including survey items focused on learner engagement.
  • The performance of learners on the course assessment activities and examinations – are the results similar to those expected in other learning environments (e.g. face-to-face)? Are the exam results consistent with assignments completed during the course?
  • Did more learners requiring remedial help and assistance than expected?
  • Did learners engage in the activities designed to encourage and enable engagement?
  • How did the learners use the range of materials available to them? Are changes in what was available for learners suggested by the patterns of use?
  • When you look at how students responded to formative or summative assessments, did they use the right approaches to problem-solving or the right methodologies? If not, is there something that could be done to support better problem solving or more appropriate methodologies?
  • As a faculty member, what surprises occurred in terms of demands on your time, response of learners to the learning materials and assessment activities?

The purpose of such an evaluation is to take us back to the design and development of the course to see what needs to be changed or improved – if anything.

Four questions are useful for faculty members to explore during this phase:

  1. What should stop being a part of this course so as to improve the quality of the learning experience and the standard of learning outcomes?
  2. What should be improved? It works, but could be better.
  3. What should be redesigned? The intention is still valid, but the way in which the learners experienced this component suggests that there should be a fundamental re-think.
  4. What should be included that is not currently in the course?

There are a great many examples of end-of-course evaluations in use today available online. One that looks both at the learner experience and at the effectiveness of the course design is available here. This is worth reviewing since some of the questions asked are intended to support the continuous improvement of courses and the design and delivery of courses by this organization. Other examples are available here and here.  There is also a comparison of such evaluation models here.

Making Quality Work

Faculty members want to improve their teaching and the learning outcomes for their learners. The systematic approach to quality in online learning using ADDIE is a proven approach to quality which many faculty use instinctively.

What is suggested is that the use of ADDIE becomes much more rigorous and systematic as a tool for translating the concerns about quality into a process which faculty members can follow.

Such an approach takes time and involves several people – individuals with experience and knowledge of instructional design, technology, intellectual property management, library services and expert learners. If quality is a core component of the strategic intent behind online learning, and it needs to be to secure widespread acceptance and strong learning outcomes, then it requires such teamwork.

References

Khan, B. (2005) Managing e-Learning Strategies – Design, Delivery, Implementation and Evaluation. Hershey, PA: Information Science Publishing.

Morrison, G. R. (2010) Designing Effective Instruction (6th Edition). New York: John Wiley & Son.

[1] A comprehensive guide to open education resources is available at http://www.col.org/PublicationDocuments/Basic-Guide-To-OER.pdf