I recently made this comment about teaching:
When you teach classes on highly technical subjects with hands-on labs, you get evaluated every class. That can be two times a week. Evaluations are from people you have just met. Evaluations are specific to the class and content being taught, and are heavily influenced by the perspective of the student.
I ended that comment with, “More on that in a separate blog post.”
Well, this is that post.
And I am going to say something that may annoy instructors, training organizations, students, HR departments, and probably a few people who design evaluation forms:
I think course evaluations are useful. I also think they are deeply flawed.
Sometimes they are almost absurd.
Imagine teaching a highly technical class to 15 people.
Fourteen students rate the course a 5 out of 5.
One student gives it a 3.
No explanation.
No comment.
Nothing identifying what could have been better.
What exactly am I supposed to learn from that?
Was the material too difficult?
Too easy?
Was I moving too quickly?
Too slowly?
Was the lab environment frustrating?
Did the student disagree with something I said?
Did the student simply have a bad week?
Did the hotel coffee stink?
I have no idea.
Yet mathematically, that unexplained rating is treated just as seriously as every thoughtful evaluation submitted by everyone else.
That bothers me.
I Am Not Against Criticism
Before someone interprets this as an instructor who only wants praise, let me make something very clear.
Good instructors need criticism.
Over the years, student feedback has caused me to change explanations, rearrange course material, rewrite labs, add examples, remove examples, slow down certain sections and completely rethink how I teach particular subjects.
That is exactly what evaluations should accomplish.
If several students tell me:
“The subnetting section moved too quickly.”
That is useful.
If someone writes:
“The Wireshark lab assumed knowledge we hadn’t covered yet.”
Very useful.
If a student says:
“There was too much lecture before we actually touched the equipment.”
I may not agree, but I can evaluate the criticism and decide whether something should change.
Even:
“Andrew, you completely lost me during the explanation of TCP window scaling.”
Great.
Tell me that.
I can do something with it.
But clicking 3 out of 5 and disappearing into the night?
That isn’t feedback.
It is a number.
Technical Training Makes Evaluations Even Stranger
There is another problem that becomes especially apparent when teaching networking, telecommunications, packet analysis, Wi-Fi, IPv6, routing, or other technical subjects.
Students do not all arrive at the same starting point.
Far from it.
A class might contain:
- someone who has been a network engineer for 20 years,
- someone who moved into networking six months ago,
- someone who manages technicians but rarely configures equipment,
- someone who knows Cisco extremely well but has barely used Wireshark,
- and someone who was told by their manager to attend the class.
They are all sitting in the same room.
They are all taking the same course.
And at the end, they are all asked:
How good was the course?
That sounds like a simple question.
It isn’t.
One student may think:
“Excellent. I finally understand what I have been seeing in packet captures for years.”
Another may think:
“Too basic. I already knew half of this.”
A third may think:
“Way too advanced. I could barely keep up.”
All three evaluations may be completely sincere.
And all three may be describing exactly the same class.
Sometimes Difficulty Gets Evaluated as Quality
This may be the most uncomfortable part of the discussion.
Students occasionally evaluate how comfortable the learning experience was rather than how valuable it was.
Those are not necessarily the same thing.
Real technical learning can be uncomfortable.
You struggle with a lab.
Your configuration does not work.
Your packet capture does not contain what you expected.
You discover that something you thought you understood for ten years isn’t quite how the protocol actually works.
That frustration can be part of learning.
A good technical course should absolutely be organized, understandable and achievable.
But I do not believe every student should finish every lab without struggling.
If they do, I occasionally wonder whether the lab was difficult enough.
Training is not entertainment.
The instructor’s job is not simply to make everyone feel successful every minute of the day.
The job is to help people become more capable than they were when they walked into the room.
Those are different objectives.
The Amazon-Star-Rating Problem Has Invaded Education
We now rate almost everything.
- Restaurants.
- Hotels.
- Uber drivers.
- Products.
- Podcasts.
- Apps.
- Online courses.
- And instructors.
The problem is that we have started treating education like every other consumer product.
- Five stars: fantastic.
- Four stars: pretty good.
- Three stars: something is wrong.
- Two stars: bad.
- One star: terrible.
But education is not a cheeseburger.
Sometimes the thing being evaluated requires considerable effort from the person doing the evaluating.
That fundamentally changes the relationship.
The student is not merely consuming a product.
The student is participating in the outcome.
An instructor can explain a concept clearly, build a good lab, provide documentation and spend 20 minutes helping someone troubleshoot it.
The student still has to learn.
That makes course evaluation much more complicated than:
Did you enjoy your stay?
Online Course Ratings Can Be Even Worse
Udemy and similar platforms introduce another version of this problem. Someone can consume hours of material, click three stars and leave no meaningful explanation.
From the instructor’s perspective, that rating is maddening.
Not because someone disliked the course.
They are entitled to dislike it.
But because there is absolutely nothing actionable about the feedback.
- What was missing?
- What was inaccurate?
- What should have been explained differently?
- Was the audio bad?
- Was the course too slow?
- Was it too fast?
- Was the material outdated?
- Was it simply not the course the student expected?
Tell me.
A low rating accompanied by a thoughtful explanation is far more valuable to me than a five-star rating that simply says:
“Great course!”
At least criticism gives me something I can improve.
An unexplained low score doesn’t.
And yes, occasionally an unexplained one-star or two-star review feels less like constructive feedback and more like somebody throwing a rock through your window and walking away.
Perhaps that was not the student’s intention.
But that is how anonymous scoring systems can feel from the other side.
There Is Another Side to This
Instructors also have to be careful. It is very easy to look at fourteen glowing evaluations and dismiss the fifteenth person as a crank.
That would be a mistake.
- The outlier may be the person who noticed something everyone else missed.
- Maybe the instructor really did gloss over an important concept.
- Maybe a lab contained an error.
- Maybe an explanation assumed too much prior knowledge.
- Maybe that student was reluctant to write a detailed criticism.
So I do read the outliers.
I think every serious instructor should.
But an unexplained outlier should be treated as a signal to investigate, not automatically as proof that something is wrong.
One unexplained rating is a data point.
A pattern is evidence.
There is an important difference.
What I Actually Look for in Evaluations
After decades of teaching, I care much less about individual numbers than I do about patterns.
- If one student says something, I notice it.
- If three students independently say the same thing, I pay attention.
- If the same comment appears across several deliveries of a course, I change something.
That is useful evaluation data.
The best evaluations answer questions like:
- What helped you learn?
- Where did you struggle?
- What should receive more time?
- What should receive less time?
- Which lab helped the most?
- Which explanation did not work?
- What prerequisite knowledge would have helped you?
- What should the instructor do differently next time?
Those questions actually help improve a course. These types of questions are in my course evaluation forms.
“Rate the instructor from 1 to 5” tells us considerably less than we pretend it does.
Perhaps We Are Measuring the Wrong Thing
This is ultimately my biggest objection to traditional course evaluations. They frequently attempt to reduce an enormously complicated educational experience to a number.
4.7 out of 5.
That looks wonderfully scientific.
But what exactly does 4.7 measure?
- Instructor knowledge?
- Communication ability?
- Course design?
- Difficulty?
- Lab quality?
- Student preparedness?
- Expectations?
- Personality?
- Whether the instructor told good jokes?
Probably some combination of all of them.
And that makes the precision of the number somewhat ridiculous.
A decimal point does not automatically turn subjective opinion into objective measurement.
Here Is the Evaluation I Really Want
At the end of a technical class, I would rather ask one question than receive ten numerical ratings:
What could I change that would help the next group of students learn this material better?
Now we are getting somewhere.
- Tell me where you struggled.
- Tell me what worked.
- Tell me what didn’t.
- Challenge something I said.
- Point out an error.
- Tell me a lab was terrible.
- Tell me you wanted more examples.
- Tell me I spent too much time on something.
You won’t hurt my feelings.
Well…
Maybe for five minutes. But then I can actually do something with the information. That is the entire purpose of an evaluation.
At least, it should be.
So Are Course Evaluations Worthless?
Absolutely not.
Good student feedback is extraordinarily valuable. Some of the best improvements I have made to courses came directly from students.
But we should stop pretending that every rating has equal informational value.
Fourteen 5s and one unexplained 3 does not necessarily mean the instructor has discovered a mysterious 20% defect in the course.
It means fourteen people gave it a 5 and one person gave it a 3.
The interesting question is:
Why?
Without that answer, we have measurement.
We do not necessarily have insight.
And if the objective of course evaluations is genuinely to improve education, insight should be what we are after.