18 Jul 2023

Using Unix tools with expandfile

This note describes how to use standard free Unix file tools to maintain web sites built with expandfile.

Unix tools flow

My rule of thumb is, "if it's worth doing once, it's worth automating it."

Detailed information on the Multics website build process and its Makefile is contained in Multics Web Site Build Process.

Unix commands

Unix provides many commands. Personal computers using the Macintosh, Linux, and Windows operating systems have many Unix-like commands built in, or available for installation. The commands are almost all free. You can invoke these commands by shell scripts or BAT files. You can also use these commands by calling them with expandfile's *shell builtin function to extend expandfile's abilities in a simple and portable way.


On a Mac, you type commands in a Terminal window.
On a computer running Unix or Linux, you type commands into a "shell window."
On a Windows computer, you use Command Prompt, PowerShell, MINGW, MSYS2, or Cygwin.

Source Files and Object files

expandfile reads one or more source files and writes corresponding object files. The object files are HTML files that make up an image of the website being created. The source files are "extended HTML (HTMX)" files that expandfile expands to create the object files; source files contain

I keep all the source files for a website in one directory, and all the object files in another. The object files are a locally-viewable copy of a production website. (Some object files are graphics, PDFs, and Javascript files that have no corresponding source file.)

Local Image of the Site

A website viewable from the Internet is made up of files in the file system of (one or more) server computers. If that set of files becomes inconsistent or is lost, the website's content cannot be viewed, so it's a good idea to have a complete spare copy of the files that you can restore to the server quickly. (In general, you should store copies of any file you care about in more than one place.) I keep a complete, consistent set of object files for each site I create on my own computer, and make the server's copies read-only. When I change my sites, I change my local copy and check that it looks right, and then publish the files to the servers.

Using rsync

The Unix command rsync securely copies files from a directory on a local computer to a directory on a server computer. rsync copies only the files that need to be changed, and when it does copy, it uses compressed I/O so the copy is fast. rsync has many features: I only use a few.

The rsync command line looks complicated, but I never have to type it: the Makefile issues the command; see below. 

  rsync -avzu --blocking-io -e ssh --exclude source-pages/ object_dir/ myuserid@servername.com:server_directory

copies all the files in /object_dir on my computer to the directory /server_directory on the server servername.com using the server user ID myuserid. Files in the subdirectory /source-pages on my computer are not copied. This command assumes that I have properly set up SSH authentication keys (look it up).

The control argument -azvu means:

rsync sends files between computers using the ssh secure shell protocol. It manages permissions sensibly: if I try to update a file that's read-only on the server, rsync updates the file and then restores the read-only setting.

Local MySQL Database

Some of my web pages contain lists of similar things. Instead of maintaining big HTML pages, I store the raw data as table rows in SQL databases. expandfile's *sqlloop builtin function iterates a macro expansion over every row returned by a query. This is especially useful for table data displayed in more than one way.

To do this, I install the MySQL database application on my computer, and load the data into database tables. (MySQL is free and is available for Windows, Macintosh, and Unix.)

Automated Installation

I use standard Unix tools to invoke expandfile when necessary, and to publish any files that changed. The Unix utility make regenerates any static page whose source is newer, and uses rsync to publish files that have changed. rsync uses secure compressed transmission. Using make means that all pages are regenerated if I change the boilerplate (wrapper), and that I won't forget to generate an object page if I make a little change to some source file. Database-backed pages are supported by loading the database table from SQL input and then translating a template to generate HTMX. make invokes other tools, such as HTML Tidy, to check each page for valid HTML and warn me about errors. Using rsync to synchronize my whole file tree means that I don't forget to upload updated graphics and other auxiliary files.

The make command uses a file called Makefile to tell it what to do, but instead of executing a script with a fixed set of steps, make figures out which steps need to be done, and in what order, and then performs them.

The Makefiles in my source directories cause make to execute expandfile to compile HTMX to HTML whenever an HTMX file is newer than the corresponding HTML file, so all my HTML files are up to date with whatever edits I made. My Makefiles declare that some files depend on others, so if I edit a file that is included by many HTMX files, they will all get expanded. When I am using a local database, make loads the .sql files for each database before using the data.

I also set things up so that typing make install will first do a make to ensure all object files are up to date, and then invoke rsync to publish files from my local computer to the server if the local file is newer, so all my website files are up to date with whatever local changes I made.

One Line Change

To make a one-line change to a web page, I

  1. Edit the HTMX file on my computer
  2. type make
  3. Use one or more web browsers on my computer to make sure it looks right
  4. type make install

Changing a Few Files

Suppose I create a new web page and want to link to if from two existing pages. I do this:

  1. Create an HTMX file for the new page on my computer. Often, I copy the contents of an existing file and replace old information with new. If I'm adding a graphic, I use a macro like getimgdiv to avoid mistakes.
  2. Edit the HTMX files for the existing pages on my computer, adding HREF links to the new page. (Sometimes I generate the links using macros.)
  3. type make
  4. Check the new page in web browsers on my computer to make sure it looks right, and check that the links from the existing pages bring up the new page.
  5. type make install

Changing Many Files

One example of why this setup is useful is the case where I need to re-expand many files.

A few years ago, the HTML language was changed to support "mobile friendly" features. I got mail from Google saying that my site would not work well on tiny screens. One thing I had to do was to insert 

  <meta name="viewport" content="width=device-width, initial-scale=1">

into the HEAD section of every file on my websites, probably about 600 files. Because I was using expandfile with wrappers, and make with rsync, all I had to do was

  1. Edit the wrapper HTMX file on my computer to add the new line
  2. type make
  3. Check on my computer to make sure the files looked right
  4. type make install
  5. Check the newly installed files to make sure the files looked OK on mobile devices
Copyright (c) 2018-2023, Tom Van Vleck