What does a technical reviewer do?

Before technical documentation gets published, it’s common (and important) for a draft to go through a technical review process. A technical review, like all other reviews, is a chance for a document to be read from a new perspective. I’ve served as both a paid and volunteer technical reviewer on countless articles and books, and so I thought it might be helpful to share my experiences in that role so that other people who have been asked to do a technical review might learn from my successes and failures. Specifically, I’m hoping to answer the broad question of what, exactly, a technical reviewer does.

A technical reviewer is anyone asked to review a document with the goal of noticing and reporting technical inaccuracies. I strongly believe that a technical reviewer usually doesn’t need to be specially qualified, and in fact possessing special knowledge of a topic can sometimes give you an “unfair” advantage. When you lack special qualifications, it’s fair to say that if you can’t understand a document then most people reading it after you won’t be able to understand it either. This doesn’t always hold true, because not every document is written with the assumption that its audience has no knowledge of its subject matter, but generally being one or two steps behind a document shouldn’t dissuade you from reviewing it. I’ve learned a lot from documents I’ve reviewed. Sometimes I’ve just learned a new method of doing a familiar task, but in other cases I’ve learned a whole new skill.

Before the review starts

Before I review a document, I negotiate expectations with the person requesting a review. Common questions I ask:

• Do you want me to fix errors, or only comment on errors?
• How do I make comments or changes? Depending on the client, I’ve made literal comments in Markdown or HTML or XML source code, I’ve used comment functions in a word processor and specialised documentation platforms, and I’ve created bug reports and merge requests.
• Should I correct (or comment on) spelling and grammatical errors when I see them, or only focus on technical issues?
• Are there related files I need for the document to make sense? You’d be amazed at how often I’ve reviewed documents that “secretly” refer to sample code or files that nobody’s told me about, much less given me access to.
• What’s my deadline?

Defining an error

When I review a document, I consider anything that confuses me to be a “bug”. It might not be an error, strictly speaking. It could be factually correct or technically accurate, but as a reviewer it’s my duty to look for every opportunity to improve the document. That often means questioning a detail that nobody thought twice about. Sometimes, it turns out that it doesn’t matter. Other times, it reveals a huge gap in documentation and results in a whole new chapter being written.

Doing the review

When I review a document, I read it from start to finish. My goal is to quantify and maximise 3 things: Clarity, accuracy, and reproducibility.

Clarity

From the document title onward, my immediate goal is to establish the document’s self-stated purpose. Ideally, a document’s goal is stated in its introductory paragraph. For example, you might recall that the introductory paragraph to the document you’re reading right now was:

Specifically, I’m hoping to answer the broad question of what, exactly, a technical reviewer does.

If a document fails to tell me what it hopes to achieve up front, then I make note of that for the author and then proceed in hopes that the point of the document reveals itself as I read.

Documentation is fraught with assumptions, and it’s up to the reviewer to question all of them. Assumptions are necessary in any document, because it defines where the “story” starts. An article about computer programming necessarily assumes that the reader is familiar with the concept of a computer. It may or may not, however, assume that the reader has access to a compiler or interpreter, and that’s the kind of assumption that usually needs to be declared.

Clarity is cumulative. It starts with the title. A title like “A beginner’s guide to…” creates a certain expectation, but so does a title containing advanced and specialised terminology, like “Optimising render times for EXR to VP8”. One is obviously intended for “beginners”, and one implies that it’s for readers familiar with specific technologies. Those example titles aren’t mutually exclusive, and a document could be titled “A beginner’s guide to optimising video render times for EXR and VP8”, which implies yet a different scope.

Whatever the case, I review documents to establish who its audience is, what objective a reader can hope to achieve by reading the document, and whether the document makes both of those clear up front.

Accuracy

As I read, I’m also looking for any claim the author makes about technology. We’re all guilty of making assumptions, so it’s not uncommon for an author to repeat a misconception about how a popular technology works or does not work. Whenever I come across a statement about how some technology can or cannot perform some task, or communicate over some protocol, I stop and do research.

Because I do technical reviews for open source technology, I’m spoilt for reference material. Most open source protocols and standards are defined by an RFC. Thanks to the RFC process, you can read the entire history of development of open source technology.

When I need to verify the capability of a technology that’s not defined by an RFC, I just look at the source code instead. This is sometimes more circuitous than an RFC (although to be honest some RFCs are more verbose than maybe they need to be), but with a little grepping it’s usually pretty easy to find code relating to most broad statements.

Finally, because I’m usually asked to technically review documents that are within my area of expertise, I often install the technology on a test system of my own and try to create a proof of concept. When a claim is made, I try to prove or disprove it.

Accuracy is mostly about taking the extra time to verify assertions. The author is justifiably focused on achieving the central thesis of the document, so it falls to the technical reviewer to monitor for unverified assumptions.

Sometimes an assumption is superfluous and can be removed. It’s common for authors to “defend” the stance of the document. An author might write something like this:

I wrote this article because OpenWidget v2 cannot be automated, but the industry needs widget verification in CI/CD pipelines.

This may or may not be true, on several counts. Maybe it doesn’t matter. The author’s point in this sentence isn’t about the shortcomings of a tool, it’s about what the document provides to the reader: A method to automate widget verification for CI/CD. Focus on that instead:

When you have widget verification in your CI/CD pipeline, you can deploy widget factories with confidence.

This nuance is important because as a technical reviewer you want to produce a document that is technically accurate. The more claims about technology you throw around in a document, the more opportunity there is for a document to prove itself inaccurate. It might seem silly, but if someone is willing to suffer inefficiency then an author’s claim that something is “impossible”, when it’s actually just inconvenient, is technically inaccurate.

I don’t need to find a definitive answer to whether a claim about technology is true or false. I only need to find reasonable doubt. A technical review is the start of a conversation with the author. If I have reason to believe that a claim is incorrect, all I need to do is cite my source and propose that the author might be mistaken, or might be speaking in broad terms where specificity is needed. Sometimes all it takes is a small adjustment to a sentence. For example, this might be true for the author:

OpenWidget v2 cannot, despite its name, create widgets.

If the RFC and the source code and online forums suggest that OpenWidget can create widgets, then I might propose this rewrite:

I was unable to create a widget using OpenWidget v2 on my system, so I used OpenWidget v1.

It’s an easy fix, and it’s technically accurate.

Reproducibility

A lot of technical documentation contains steps for getting specific result with technology. As a technical reviewer, it’s my job to verify that those steps reliably produce what the author is attempting to demonstrate. I do this by performing the steps on a test system. Ideally, the author provides a repository with the source files required to complete an exercise in the book, but sometimes those files aren’t ready for distribution as I review, or else the point of the exercise is to create those files. Whatever the case, I run all the code, or follow steps in a GUI interface to verify that a document’s examples work as expected.

It might seem redundant to run examples that the author has surely already executed while devising them in the first place. However, writing technical documentation is rarely a linear process. It’s all too easy, and common, to write out the steps required to accomplish a task, and then to consolidate or re-order the steps for clarity. Usually that works, but sometimes when you consolidate steps you fail to realise that a file created by one step is required for something else, and by re-grouping tasks you’ve created a circular dependency, or you’ve prevented a dependency from being created in the first place. A fresh run from start to finish is the only way to verify an example, so that’s what I do.

Just as with verifying technical claims, it’s not my place to force the steps to work as expected. I’m only doing quality assurance. When something doesn’t work, I assume one of two things is true:

• I’m not technically proficient enough to make the example work, in which case the documentation has failed to train me for the example.
• The documentation is wrong, and it needs to be corrected before publication.

In either case, I’ve given the author an opportunity to improve the document.

Technical reviews are a conversation

I find very little subjectivity in technical review. Certainly, sometimes a lesson isn’t expressed the way I would have expressed it, but then again I didn’t write the document. The technical reviewer’s mandate is to optimise the document being reviewed. Different people learn in different ways, so presentation is flexible. Nobody buys a book for incorrect information, though.

When I review a document, I read with no presumption and no expectation. I take each document on its own terms, and objectively communicate inaccuracies to the author. Often, as a contractor, I never find out what an author does with my findings, because most publishers (in my experience) only want to pay for a single review. A single thorough review is what I aim for, and that means reading a document “differently” than I would read the same document for leisure. The ability to read objectively, and resist getting lost in the story (such as it is) of a document might be the only actual skill required for a good technical review. Everything else is polite pedantry, research, and the willingness to follow instructions.