I wrote a book using ChatGPT

Generative AI like OpenAI's ChatGPT (and more recently, Google's Bard) has transformed how we create content. Available since November 2022, users everywhere have experimented with how ChatGPT can create text across a variety of domains, styles, and topics.

I first explored ChatGPT in January and February this year, and was immediately impressed with what ChatGPT could do. I provided prompts about programming, history, and fiction, and I was surprised to see ChatGPT generate content that was readable and generally correct.

I should note that ChatGPT isn't correct everywhere. I found that ChatGPT's responses were usually correct, but sometimes included errors. Some errors were obvious, others were subtle.

The start of a project

I realized even then, in early 2023, that ChatGPT was a turning point in technical communication. I wanted to explore what ChatGPT could do if you really pushed it. Rather than be satisfied with a short response, I gave ChatGPT successive prompts that required it to expand on its answers.

My personal interest is in technology, including open source software, operating systems, and programming. I thought it would be funny to ask ChatGPT, a new-age generative AI, to describe an antiquated programming language in 2023. So my prompts quickly focused on FORTRAN 77, one of the oldest higher level programming languages, yet a version of the language that was made obsolete with Fortran90 in 1990, and later editions of the language.

And that's when I had an idea: Could I use ChatGPT to write a book?

Creating an outline

Writing a book is a large project, and I wanted to structure the book with a clear outline. At the same time, I was doing the book as an experiment, so I didn't want to go overboard. I focused the book to "A gentle introduction to FORTRAN 77" so I could keep things at a high level, yet provide enough material to exercise ChatGPT as an authoring tool. My high level outline looked like this:

1. Introduction
2. Editor's note (the only section I would write entirely as me)
3. Why FORTRAN 77
4. Introduction to FORTRAN 77 (character set, fixed columns, comments, and line labels)
5. Data and variables (variable types, implicit variables, constants, arrays, input and output)
6. Doing the work (program flow, comparisons, loops, and jump execution)
7. Functions and subroutines
8. Let's practice (a collection of exercises with solutions)
9. Appendix (reference material)

I later added a colophon to provide information about the tools and fonts used in the book, an "About the cover" section with notes about an AI-generated image I selected for the book's cover, and a "Praise" section with ChatGPT-generated quotes from made-up people about the book; the "praise" quotes were also included on the book's back cover, and I wanted to be transparent that these quotes were generated by ChatGPT.

Engineering the prompts

With my book planned out as an outline, all that remained was prompting ChatGPT to generate the book's content. My first few prompts were awkward and required a few attempts to get a useful response. But I soon figured out how to ask ChatGPT to generate a book section.

ChatGPT has a length limit for its responses. Most of the time, ChatGPT will generate a response that fits within that limit - but for very long responses, ChatGPT will just give up after a certain amount of generated text. I was able to work within that limit because of my book's outline; because each chapter included multiple sections, I really only needed ChatGPT to generate text for a section at a time, not a full chapter.

Prompts for my book included:

• Introduction: "Write an introduction by an editor in first person about how this book was written by an AI chat bot and not by a human."
• Why FORTRAN 77: "Write a position paper about why programmers should learn FORTRAN 77 in 2023."
• Introduction to FORTRAN 77: "Write a book chapter that introduces FORTRAN 77 and what kinds of problems it can help solve."
• Character set: "Write a book chapter about the character set for FORTRAN 77."
• Fixed columns: "Write a book chapter about what the fixed columns mean in FORTRAN 77, including comments and continuation lines. Explain how these derive from punched cards."
• Comments: "Write a book chapter about comments in FORTRAN 77. Discuss comment lines and in-line comments."
• Line labels: "Write a book chapter about line labels in FORTRAN 77."

In crafting my prompts, I realized that if you ask ChatGPT a simple question like "Describe comments in FORTRAN 77," it will respond with a passage that briefly describes programming comments. Instead, asking ChatGPT to "Write a book chapter" provided a longer response, but missed some details. By asking detailed questions that included requirements for the generated text, ChatGPT was more easily able to write text that fit my needs.

Watch for errors

ChatGPT is trained on a vast data set that encompasses a variety of sources, styles, and topics. In other experiments with ChatGPT, I found that ChatGPT could reliably and correctly describe well-documented topics, such as "Describe how to use the puts function in the C programming language." Many technical writers have documented how to use puts in programming, and ChatGPT is able to leverage that history to generate its own version.

I assumed the same for FORTRAN 77. FORTRAN is a very old programming language, with much written about it. So I thought it was safe to assume ChatGPT could describe FORTRAN 77 based on an ocean of other material written about FORTRAN 77.

But I found that ChatGPT didn't always describe FORTRAN 77. The FORTRAN 77 version of the language was superseded by Fortran 90 (with a new spelling) in the early 1990s, and later versions since then. Much of ChatGPT's generated responses did not describe the FORTRAN 77 programming language, but instead some more recent variant - yet ChatGPT blindly claimed that the code was FORTRAN 77.

This was my first flag that you cannot trust what ChatGPT gives you. Anyone who uses ChatGPT (and especially technical writers who leverage ChatGPT to generate content for them) must verify and fact-check everything that ChatGPT produces. Do not assume its responses are correct. Sometimes ChatGPT provides glaring errors that are immediately obvious, but much of the time it will generate subtle errors.

For example, FORTRAN 77 provides implicit rules for INTEGER and REAL variables. Any variable name that starts with the letters I through N is assumed to be of INTEGER type. All other variable names are assumed to be REAL variables. ChatGPT described these rules well enough, but got it backwards when it provided an example:

It is important to note that the use of implicit typing can lead to potential errors if a variable name is accidentally chosen that starts with a letter that implies a different type than the one intended. For example, if a variable named "age" is intended to be a real variable, but it starts with the letter "a", the compiler will assume it is an integer variable.

Note that a variable named AGE starts with the letter A, so the implicit rules would make this a REAL variable, not INTEGER.

Margin notes and chapter notes

But my exercise was using ChatGPT to write a book; I wanted to see how far I could push the tool to create content. I didn't want to edit the generated text. Instead, I decided the book should be a demonstration of what ChatGPT can do, while highlighting the current limitations.

I decided on a compromise: I would include ChatGPT's responses verbatim, but I would make corrections as margin notes, and provide my own commentary as end-of-chapter notes. To make clear what text was generated by ChatGPT and what was written by me, a human, I used formatting: ChatGPT's responses were included verbatim as standard text, and my notes were added in italics.

For example, to note the error about implicit variable types, I added this margin note on page 23:

This is backwards, “age” is an implicit REAL variable name

At the end of each chapter, I wrote a summary of the major issues I found in ChatGPT's responses. I noted where ChatGPT responded correctly, but focused on where ChatGPT failed.

Most of ChatGPT's errors were because it described some later version of Fortran, and not the FORTRAN 77 version. For example, my summary at the end of the "Doing the work" chapter included this note:

ChatGPT has a tendency to write Fortran 90 code samples throughout this chapter, despite claiming they are FORTRAN 77. For example, FORTRAN 77 uses a slightly different DO loop syntax that requires a line label:

      DO label variable = start, end, step

In the FORTRAN 77 version of the DO loop, the loop continues until the statement with the indicated line label. The statement could be a program instruction or the CONTINUE statement. For example, a FORTRAN 77 program to print the numbers from 1 to 10 would use a DO loop like this:

      PROGRAM LOOP
      INTEGER I
      DO 99 I = 1, 10
 99   PRINT *, I
      END

Fortran 90 added the END DO syntax.

Other obvious errors include ChatGPT describing labels that include letters such as START: and modern Fortran functions such as EQ. FORTRAN 77 does not provide an EQ function, instead only providing comparisons including .EQ. (equal), .NE. (not equal), .LT. (less than), and .LE. (less than or equal).

ChatGPT as a co-author tool

This was an interesting experiment that pushed ChatGPT to its limits as a co-author tool. The exercise helped me engineer better prompts that yielded more complete responses. Looking back on the project, I identify three major lessons:

1. Verify and fact-check ChatGPT's responses. Most responses are correct, but a generative AI doesn't understand the material. The generative algorithm is essentially predictive text. As a result, errors are likely. If you use ChatGPT as a co-author tool, you should verify the results before including them in a larger work.
2. Break up the work into smaller units that ChatGPT can process more easily. In my case, I found ChatGPT was much easier to use as a co-author tool if I first segmented the content to create. My outline provided a natural structure.
3. Prompt engineering will be a new skill for co-author tools. If you ask ChatGPT a simple question, you will get a simple answer. Over time, I learned to craft more complete questions that allowed ChatGPT to provide a more complete response.

Interested in seeing the complete work? Getting Started with FORTRAN 77 is available for purchase via Lulu.com