All my marbles in one place

A blog about course design, data display, C++, and Python. Yes, these are related, at least as I see them.

© 2021. All rights reserved.

Assignments with coding only teach coding

26 Oct 2018 Tags: Course design, Big data

Fourth in a series on designing a course on data engineering. The previous post listed the minimal operating system and networking knowlege.

When designing an upper-division or graduate-level class, we often set our learning outcomes to include higher-level concepts characterizing systems architecture. For example, I am designing a course in data engineering for a professional master’s program. This course features such concepts as

I begin from Biggs’s rule: Learning results from what the learner does. The activities I ask learners to perform should align with the way I expect them to use the relevant knowledge after the class; my assessment of learners against those outcomes should align with the expected use as well.

Set aside the issue of assessment for a moment. The immediate question is how shall I design the exercises to best align with the desired outcomes?

Defining the outcomes in detail

Abstract concepts in a graduate course ought to be expressed via one of the higher-level categories of the cognitive process dimension of the revised Bloom Taxonomy, using verbs such as “analyze”, “evaluate”, or “create”. As is typical with applications of this taxonomy, the level at which to categorize a given outcome is sensitive to how we frame it. For example, consider the learning outcome of “student will be able to profile the distribution of latencies for a microservice”. At first, this might seem like “analyze”, with its implication of attribution. But if the students are asked to contrast this profile with a specified Service Level Objective (SLO), the process is arguably “evaluate”. On further consideration, latency profiling requires designing a test plan, specifying input distributions, adding metrics, and creating a final report—processes more directly fitting “create”.

The learning outcome could be set as any of the three verbs, “Analyze”, “Evaluate”, or “Create”, depending upon the emphasis of the instructor.

From learning what they do …

Having specified the detailed outcome, we can now create an activity whose performance leads the student to that outcome.

A typical activity will ask the students to build something and as part of that assignment, analyze the performance of what they build.

But consider this from the Biggs perspective, that what the student does is what the student learns. What are students spending most of their time doing? Programming, of course. Designing (well, not really, but in a perfect course, they would design), coding, testing, debugging, coding some more, integrating, and arguing about what to do next and which team member needs to contribute more.

How much time have they spent profiling the latency? Well, they could only do that after the system is built and almost reliable—a tiny window at the end of a multi-week programming frenzy. Human memory records most clearly the activities upon which the learner spent the most time and attention. Given how much more time they spent building the system than instrumenting and tuning it, their strongest memory will be programming.

This is reflected in how the students respond to job interviews. When asked to summarize what they did in class (itself a misdirecting question, focusing the students on their activities and not on the principles nominally guiding those activities), they will reply, “We built a distributed system in [language X]” rather than “We measured latency and tuned the architecture to achieve our Service Objectives”.

… to doing what we want them to learn

To increase the mindshare occupied by the intended learning outcome, we have to reduce the time spent on less relevant activities and increase the time spent on activities directly relevant to the outcome. In our example case, we want to reduce the time spent programming and increase the time spent reasoning about the performance implications of a program’s design. In fact, the key metric is not really total time but proportion of time. In an ideal exercise, students would not work particularly long but all their time would be on activities directly achieving the learning outcome.

Brevity is a desirable property for an exercise. Long activities, especially those the length of a major assignment, engender substantial fatigue and stress, which reduce the amount and quality of learning. Pushing the real outcome to the end of such an assignment, as occurs with an instruction such as, “Measure your system’s performance under a sample load and write up your analysis”, means that students engage with the most important learning outcomes precisely when their fatigue, stress, and deadlines most work against effective learning.

Creating environments for learning skills of analysis

If “program an X and then analyze it” is ineffective for learning the analysis skills (and probably isn’t particularly effective at teaching them programming skills, either), what sorts of activity are effective? Here’s some that I’ve come up with:

Analyze a paper model
Provide a paper design, some sample output metrics, and ask them to evaluate the system. This sort of exercise focuses them directly on the theory, considering how the pieces fit together, applying formulas, and so forth. It might be a good prologue to working with an actual system. It has the merit that with some computational support, the instructor can give the class several distinct cases, reducing the risk of cheating. It is of course the only sort of question you can pose on an exam.
Analyze an interactive model
The instructor can build an interactive model, which the students can analyze. An interactive model can have hidden complexities that the student has to tease out, an impossible challenge to create in a paper exercise. Every group can be given different parameters, reducing the risk of cheating. And the system might even do some portion of the assessment automatically. The primary disadvantage is the effort to create the simulation.
Analyze a system they have already built
In this exercise, the students instrument, analyze, and perhaps tune a system they built in another course or earlier in the same course. This approach has some inherent interest for students, in that they are improving a system in which they have some investment. It also reduces system learning for those team members who developed the original project. It has the further merit of reducing cheating, as every group has their own project. The biggest risk is projects that are too buggy or poorly-designed to instrument and get usable results.
Analyze a system provided by the instructor
In contrast to the approach of students instrumenting their own system, in this style the instructor provides a system for them to instrument and analyze. This has the advantage that the instructor is assured that the system is correct and will yield interesting analyses at a level attainable by this class. It also helps the instructor grade, as the results will follow one of a few known patterns. Its two disadvantages are the risk of cheating when every group has the same project, as well as the effort to create such a system.
Analyze a system selected from a public repository
Students could be asked to analyze a system selected from repositories such as GitHub or BitBucket. This has mostly the same advantages and disadvantages of using a system the students had previously developed (though the students start with no prior experience with the code). It has the extra disadvantage of requiring some gate-keeping by the instructor to ensure the students pick a system that is likely to yield an interesting analysis.

So what will I do?

Programming assignments are not particularly effective or efficient ways for students to achieve non-programming learning outcomes. But they are one of the most efficient ways for the instructor to set up an exercise. All the alternatives that I’ve listed above, with the possible exception of paper exercises, require significantly more pre-class system development by the instructor. I suspect the instructor’s ease of developing programming assignments contributes to their persistent use in many courses.

I haven’t decided how to structure assignments for my upcoming data engineering course. This post has started my thinking about what I might do. My time is limited before the course begins in January 2019, so I must pick realistic goals.