Open Sourcing My Tools For Generating Tutorials From Source Code
I have been working on my game engine, Pixel Vision 8, for the better part of 6 years now. One of the challenges of working on any sizeable open source project is writing all the documentation and tutorials. I've always been fascinated with designing automated build systems, and it occurred to me that I could create a tool to help me streamline this entire process. The idea was straightforward, could I analyze a code file and break it down into individual steps?
I started using Google Apps Script to automate converting Google Docs to generate markdown for PV8's Github wiki. I was having so much success with this workflow that I created a course for LinkedIn Learning called Google Apps Script for JavaScript Developers. Then I took the JS code out and put it into Atom as a plugin to do the tutorial generation in real-time. But last weekend, I packaged up the core logic and uploaded it to NPM. I named this project Tutorial Writer, and now it works with any NodeJS build system I would want to use.
Tutorial Writer is still more of a POC than a fully-fledged tool. While I continue to clean it up and add more features, I thought I'd walk through some of the logic that powers it. Let's start by looking at the following Lua script:
-- This is a local variable
local total = 0
-- Here is a function
function Init()
-- Here is a generic block of code
table.insert(tileIDs, index)
end
Using Tutorial Writer is straightforward. Once you install it from NPM like so:
> npm i tutorial-writer
You just need to reference the package and pass it the contents of a Lua file:
const tutorialWriter = require('../index');
const fs = require('fs');
let filePath = "./examples/code.lua"
let text = fs.readFileSync(filePath, 'utf8');
let markdown = tutorialWriter.toMarkdown("code.lua", text, tutorialWriter.luaTemplate);
console.log("# Tutorial Writer Markdown\n", markdown);
Tutorial Writer is going to take the script and convert it into a step by step tutorial in markdown like this:
Step 1
Create a new file called code.lua in your project folder.
Step 2
Create a new local variable called total inside the script:
01 local total = 0
This is a local variable
Step 3
Create a new function called Init():
02 function Init()
03
04 end
Here is a function
Step 4
Add the following code to the Init() function:
03 table.insert(tileIDs, index)
Here is a generic block of code
Final Code
When you are done, you should have the following code in the code.lua file:
01 local total = 0
02 function Init()
03 table.insert(tileIDs, index)
04 end
Pretty cool, right? Now let's walk through how Tutorial Writer actually works. If you want to see the full potential of Tutorial Writer, be sure to check out my Pixel Vision 8 HashNode account, where I'm working on posting 50+ tutorials created from the API examples.
If you look at the sample Lua code above, you may notice it's formatted in a unique way. Each piece of code is on its own line, and above it is a comment. Let's look at the original code example and step through it, code block by code block.
So we have three blocks of code here: a variable, a function, and some generic code that triggers something to happen. To keep the parser simple, I only look for a few common types of code blocks:
- Variables
- Functions
- Comments
- Conditions
- Loops
- Generic Code
I'm currently in the process of making Tutorial Writer more modular. Ideally, it should support different rules to parse something like C#, which my game engine also supports. For now, Lua is easier to discuss, so let's talk about how Tutorial Writer breaks down the code.
The first step is to split up all of the code based on the empty lines in the file. Each code group is converted into a code block object, which I can process later. Here are the 4 code blocks Tutorial Writer sees:
Block 1
-- This is a local variable
local total = 0
Block 2
-- Here is a function
function Init()
Block 3
-- Here is a generic block of code
table.insert(tileIDs, index)
Block 4
end
Once we have each code block, the parser can loop through them and convert them into a step in the final tutorial. When I ask a code block to return markdown, it loops through each line and determines what kind of code it contains. Here's how block 1 is parsed.
There are two lines in this code block:
Line 1
-- This is a local variable
Line 2
local total = 0
I have a set of regex patterns I use to determine what's in each line. Here are some of the patterns I search for:
- Variable:
/(local)+\s+(\w+)/ - Function:
/(function|\s)+\s+(\w+) *\([^\)]*\)/ - Condition:
/if/ - Loop:
/for/ - Else:
/else/ - BlockEnd:
/end/
Determining if a line is a comment is easy because I just need to test the first two characters to see if they start with --. If there is a comment in the code block, I simply pull that line out and save it for later. Then, based on the regex test, I assign a type to the entire code block and move on to the next one.
If a block of code has a comment, that becomes the instructions at the bottom of the step. You can have any number of comments above a block of code as long as there are no empty lines between them. If the parser encounters a comment not attached to a code block, it's converted into a blockquote in markdown by adding > in front of each line.
Now that the block of code has been assigned a type of variable, we need to look up the step template to convert it into markdown. I have another object that contains templates for each code type. Here are a few that I use for Lua:
- Code:
Add the following code to the {0}: - Condition:
Add the following condition to the {0}: - Loop:
Create the following Loop: - Function:
Create a new {0} called {1}(): - Variable:
Create a new {0} variable called {1}{2}:
Now that I have the step template, I analyze the variable's line to try and determine its scope. In Lua, I simply search for local since global is a bit more challenging to determine. Here is what the final markdown will look like since step one is always reserved for creating the code file itself:
Step 2
Create a new local variable called total inside the script:
01 local total = 0
This is a local variable
You'll notice I also assign the code a line number. The old programming books I read in the 80s heavily inspired this feature in Tutorial Writer. In these books, you'd have pages of code to type out with line numbers, so you didn't lose your place. Later on, this number plays an important role when I combine all of the steps back into the final step that presents all the code at once.
Each code block is responsible for determining what line it belongs on, and I have to do some unique things behind the scenes to make sure the numbers are correct and increment at the right place, especially when nested in other blocks of code.
Now we can look at block 2, which is a bit different because it's a function that has an opening and a close. In Lua, closing a statement requires an end which you can see in block 4, but the parser isn't aware of it yet. When a block of code that requires an ending is encountered, the parser automatically adds 2 lines to the code block, an empty line, and the close statement like so:
Step 3
Create a new function called Init():
02 function Init()
03
04 end
Here is a function
The parser also sets a flag that it is now inside of a function, so when it encounters the following code block, it will reference the name of the function the code is being added to like so:
Step 4
Add the following code to the Init() function:
03 table.insert(tileIDs, index)
Here is a generic block of code
Notice how the line was changed to 3 even though the previous code block ended at 4. This is because the parser knows it's inside of a function and steps back a line to make sure you add the code inside of it correctly.
The last thing the parser needs to handle is the remaining end statement. Since this has been accounted for already in the function code block, it can just be ignored.
At this point, the parser is done and needs to generate the final code, which looks something like this:
Final Code
When you are done, you should have the following code in the code.lua file:
01 local total = 0
02 function Init()
03 table.insert(tileIDs, index)
04 end
And there you have it, a completely automated way of converting code into a step-by-step tutorial. While I originally wrote this to help me create scripts for my LinkedIn learning courses, it's evolved into something powerful enough to do complete tutorials that most developers wouldn't even know were generated by a script. Even better, it forces me to write clean, well-documented code as a byproduct of formatting things in a way that the parser can read.
While this is still a simple POC, I have plans to continue to build upon it and see how far I can take it.
If you like this project, please leave a 👍 and ⭐️ on Github. I'd love to help developers just getting started with technical writing and looking to share their work with others by simply automating the entire process!