What genres of readings should be assigned to Prof. MSc. students?

10 Jan 2020 Tags: Course design

What sort of materials should students read in a professional MSc. course in computing science? Journal articles? Research conference papers? Systems-oriented conference papers? Articles from practitioner-oriented magazines? Blog posts? Wikipedia entries? Ultimately, what is the purpose of the readings?

I use a broad, eclectic range of sources in such courses. In this post, I articulate my rationale for my choices. The outcome driving them is preparing the students for the activities and audiences that they will perform after graduation.

Activities graduates are preparing to perform

I think the essential distinction between the goals of an CS undergraduate degree and those of a CS professional MSc. is what each strives to develop in their students. Whereas the undergraduate degree emphasizes development of professional technique, the graduate degree emphasizes development of professional judgement. Although the majority of such a professional’s time may be spent exercising techniques, building and testing systems, the majority of their contribution—the reason they were hired—will result from decisions informed by expertise gained through graduate study.

Activities exercising professional judgement

The activities in which graduates of a CS professional MSc. program might exercise judgement include:

• Analyzing prospective designs
• Assessing the suitability of products (which might well include open-source products) for current needs
• Diagnosing malfunctions in current systems
• Defining requirements and designing systems to address those requirements
• Documenting the rationale for an existing design

The key skills are analyzing and justifying.

Audiences for presentations of professional judgement

The results of the above activities will be presented to one or more of the following audiences:

• Other members of their technical team
• Technical specialists outside the team, potentially including outside the organization altogether at fora such as conferences
• Generalists and specialists in other areas, often including higher management at the speaker or writer’s own organization

The key presentation skills include providing the right level of detail for the audience and addressing the audience’s concerns.

In addition to the above outward-facing activities, professionals must also perform the inward-facing activity of ongoing professional development, staying abreast of recent developments in computing.

What readings will prepare students for these activities and audiences?

The activities and audiences listed above suggest that the most appropriate readings will be those focused on the analysis and design of actual systems, systems in production rather than the smaller research systems used in research to validate a theoretical construct. The distinction hinges on the paper scope and intent, not the forum in which it was published. Many conferences will feature a mix of academic research that elucidates general principles and papers describing the design of production systems. Both types can reasonably be called “research papers” but only one seems appropriate as readings for a professional master’s course.

Basic research is ill-matched

Basic research in the more systems-oriented areas of computing science, typically done in universities, most often takes the form of proposing a design principle that will improve software quality by some metric (latency, reliability, security, and so forth), constructing a proof of concept system, and evaluating that system in a controlled environment. This work is difficult and requires a team of research assistants, often graduate students, to develop the software, both the demonstration system and its evaluation environment. Such projects can often consume one or more programmer-years.

Unfortunately, the research papers produced by such projects are not well-matched to preparing students for professional work. The mismatch arises on multiple levels:

• The audience for these papers is fellow researchers in the same specialist community. Such communities have specific questions of interest and specialized practices for answering those questions.
• The contribution of these papers is advancing the scientific and engineering principles for system design.
• The goal of these papers is to persuade the audience of researchers that the contribution is a substantial and validated advancement of the community’s understanding.

• The structure of the papers requires a comprehensive survey of prior results, both to demonstrate the authors’ membership in the community and to justify the novelty of the paper’s contribution.

  The structure also requires a formal evaluation under controlled or at least well-measured and documented conditions.

These are in stark contrast to the activities that the graduates of a professional master’s program will perform:

• Their professional community is primarily concerned with delivering software within constraints of time, budget, and team expertise.
• Their professional contribution is meeting the design criteria within acceptable tolerances.
• Their success critera are almost exclusively budget and schedule.
• The genre of their proposals and reports is focused on presenting decision-makers with a select group of the most viable options, often only two or three options. The author has winnowed out the obscure, overly risky, or fantastic options. There is no requirement of comphrehensiveness. Indeed, the overarching goal is the minimization or management of risk.

Some academic research papers break the mold, pursuing a more observational approach, such as the University of Toronto team that categorized the sources of failures in long-running systems. Such efforts aim to characterize the strengths and weaknesses of production systems, knowledge that can directly inform future practice without any period of adaptation from a proof-of-concept study.

System design reports are a better match

Conferences such as the Usenix series or VLDB include many contributions from teams building and running production systems. These papers typically include substantial contributions from teams at a large vendor. Some examples of papers that I have assigned to the benefit of professional graduate students include:

Paxos Made Live

The complexities of implementing the Paxos algorithm in a high-volume production environment (Google).

Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases

Adapting a database design for a managed, high-throughput cloud environment (Amazon).

File Systems Unfit as Distributed Storage Backends: Lessons From 10 Years of Ceph Evolution

The limitations of operating system file stores as backends for a distributed file system and the advantages of backends purpose-built to support distributed files (Red Hat).

FoundationDB Record Layer: A Multi-Tenant Structured Database

A record layer for a database with a huge number of cotenants (Apple).

The particular advantage of this genre over reports of principle-driven basic research is that the authors are reflecting after implementing a full system, with all the complexities of supporting high throughput, tolerating faults in both software and hardware, and production deployment cycles. The authors are reporting the most salient lessons from the entire process.

Such processes of development, deployment, measurement, and hardening against failure are the processes for which students in a professional master’s program are being trained and reading “after-action” reports about actual systems focuses and sharpens the students’ judgement. These papers also exemplify the pinnacle of the writing genre the students will perform in their career.

Reading papers such as these is good preparation for the audiences and activities with which the students will professionally engage.

Blog posts and long-form discussion replies are also valuable

A third possible reading source is blog posts, which may be published on corporate sites such as the AWS Architecture Blog, personal sites such as this one, GitHub Pages, or Medium. A closely-related format is long-form answers on discussion fora such as Quora.

As a genre, blog posts are less complete, less authoritative, more tentative, and not necessarily as well written as published systems papers published in reviwed fora. They are written in a personal voice (though this may be tempered when the author writes on behalf of an organization) and viewed by both authors and audience not as authoritative statements but as a first draft of ideas. The audience is typically from the same community as the author and posts have a “within-the-family” feel, which can include the sort of nasty arguments that arise in families.

Though these qualities impose a higher interpretative burden on the reader in the form of cross-checking claims and accounting for the open bias of frankly opinionated writing, posts such as these provide information unavailable in other formats. They are timely, specific to context, yet paradoxically in the hands of informed authors can also provide background and overview lacking in more formal genres that emphasize novelty and rigour of contribution over informed recommendation.

This genre is also likely forms in which students will write during their careers. Blog posts often serve as an initial form that builds an author’s confidence and leads to publishing in more formal contexts.

Articles in the trade press and vendor white papers orient students

Articles in the trade press and white papers by product vendors represent another genre, characterized by a tone of hyperbole and the expectation of great things to come. Their content can be limited, their charged language tiring, but they can be useful for orienting students to current industrial concerns, defining terms, and mapping the contemporary technical landscape. They can also counterbalance the longer-term focus of research articles by revealing the more mundane and dated concerns of production systems outside the charmed circles of well-funded industrial leaders.

Discussion threads alert students to “hot” questions and topics

Finally, threads in Twitter and Reddit can be limited use. These fora may more often promote confusion than clarity but they do have merit for pointing students to recent longer articles of interest and familiarizing them with current debates.

Conclusion

There are many sources of readings that can prepare students for careers as professionals exercising informed judgement. Of these sources, the genre of a classic research paper incrementally advancing long-term scientific knowledge, though a staple of academic writing and reading, is the most poorly-adapted. Refereed reports of system designs and expert opinions in longer unrefereed fora such as blogs are a closer match to the reasoning, audience, and concerns to which graduates of these programs will devote their time. These genres are also the most likely forms in which the graduates will ultimately contribute back to their profession. I think such forms should form the bulk of the assigned readings for students in professional MSc. programs.