These are my notes on the site building machinery, including usage, internal details, and possible future improvements.
Rules for maintenance and content of multicians.org are in Web Site Design.
Detailed instructions for maintenance tasks for multicians.org are listed in Maintenance tasks for the Multicians Website.
Overview
The multicians.org website is built on the site editor's computer
from HTMX source using Unix tools and the open source program expandfile, and installed on one or more web host computers.
Expandfile is available from 
github.
- The web site is served as static HTML pages.
- HTML pages are created at compile time from HTMX source.
- Other online static objects (PDFs, JPGs, etc.) are not built from HTMX. (biggest part of storage)
- Local SQL databases are used to build some HTMX source into HTML at compile time offline.
- The site is served by an Apache web server configured for security.
- The site provides CGI response to mail sending and Multician sign-up forms, backed by an online SQL database.
- multicians.org mail addresses are forwarded using procmail.
Parts (5), (6), and (7) utilize features provided by the ISP, pair.net. Hosting at Pair costs me money every month. I think it is worth it: they have good uptime, multiple backbone connections, good bandwidth, and keep FreeBSD updated with security fixes.
Building (1) from (2) is done using expandfile. This program is open source on GitHub. expandfile is also used to build some files in (2) from (4), e.g. the bibliography.
A note on (4). I do not leave data in SQL overnight. (Old prejudice from the 80s.) There are about 25 tables, stored in text files, which I load into MySQL if they have changed. About 71 SQL queries are used to build the site, mostly in the bibliography files.
Functions (6) and (7) could be hosted on a different server computer from the web site. When I set up Multicians mail forwarding, I promised Multicians I would not let their addresses be visible to spammers scraping web pages. I keep the protected mail addresses in an online SQL database and use the data in (6) and (7). The result is that the SQL source for the mail address table must not be public. There is also spam filtering inbound and outbound in (6) and (7), with blacklists at Pair and unpublished spam detection algorithms.
1. Build Process Overview
1.1 General Principles
The build process attempts to
- Separate content from presentation.
- Store each fact once, to simplify updating and avoid inconsistency.
- Avoid HTML syntax errors and missing files.
- Produce standards-compliant HTML in order to work with a wide range of visitor devices.
1.2 ISP Host Copy and Editor's copy
The domain multicians.org maps to a web host computer, operated by Information Services Provider (ISP) pair.com that serves the Multicians website. Web host computers run a web server process (Apache) that responds to visitor HTML requests with the content of the requested site resources.
The site has been mirrored at additional other ISP hosts in the past, but as of 2020 there are no mirrors. (The mirroring at stratus.com was done using an FTP server, unable to support server-side execution, and this constrained the site implementation. Turned out to be a good limitation.)
The site editor's computer has a complete copy of the multicians.org website content. To make a change, the editor prepares updated or added content on the local computer, checks it locally, and then publishes modified pages to the multicians account on the ISP using rsync. When the site was mirrored, updates were pushed to the mirrors every few months.
1.3 Source and Object
Rather than editing HTML language files, the site editor works on local source files that are translated to HTML. Source files are written in the "HTMX" (HTML eXtended) language. .htmx files are translated to .html files by a program called expandfile.
Source and object are kept in separate directories on the editor's computer. (A copy of the source directory is also stored at the ISP, for backup, not available on the Web.)
A few HTML files, supplied by others, do not have HTMX source. PDF and graphic files do not have a corresponding source. Copies of these files are kept on the site editor's local computer and sync'd to the web host computer if they change.
1.4 Steps in updating an HTML file
The basic steps in generating an update to multicians.org are:
- Edit one or more local source files, and possibly add graphics or other files to the object directories.
- Execute the make command to rebuild local object files by expanding SQL and HTMX files to HTML with expandfile.
- View and check the local site, including object files generated by make and links to and from them.
- Iterate 1-3 until satisfied.
- Execute the make install command to push the local image to the web host directory.
This process depends on software and configuration files installed on the site editor's local computer.
1.5 Use of Make Utility
The website's files on the web host computer(s) are kept up to date by executing the make command in the source directory on the editor's computer. (The make command is available on Mac, Linux, Unix, and Windows systems. The original make command was created for Unix by Multician Stuart Feldman.)
The make command is used for two major tasks:
- Create updated local .html files from local .htmx and .sql source if they have changed.
- Publish all new or modified local files to the web host computer(s) using rsync.
Why do it this way? This setup
- .. remembers all the details of what goes where and how to publish files. The editor just types one command.
- .. keeps all the site formatting boilerplate in separate files, so all pages have a consistent format.
- .. warns if any files have HTML errors before they are published.
Editors don't need to know how the machinery works to use it for routine changes.
There are many features of make that I haven't learned yet. The Makefile could be better. Suggestions are welcome.
1.6 Use of Rsync utility
Why rsync?
- .. files already up to date on the host are not sent unnecessarily.
- .. every modified file is copied, so no part of a change is forgotten.
- .. if a file vanishes from the Unix side for whatever reason, rsync puts it back.
- .. files are sent protected by a crypto key pair.
- .. rsync sends deltas rather than whole files.
- .. files are compressed before sending, which makes sending them at least twice as fast.
2. Installing and Configuring the Build Machinery
A number of files have be be in the right places on the editor's computer for the build process to work.
2.1 Object files
2.1.1 HTML Object files
On the editor's computer, there is a complete local image of all files needed to display the web site, such that the web hosting computer can be updated using the Unix rsync command. These files are kept in the object directory. These files include .html files, .pdf files, JavaScript files, CSS files, and graphics.
After making changes and recompiling the site, the site editor should view the local copy of the website with a few web browsers to check the site's appearance, before updating it to the web host. A "smoke test" listing a few breakable features is defined in Maintenance tasks for the Multicians Website.
A few files are excluded from the rsync command that updates the ISP object directory from the local copy: such as configuration files that define local directory structure, and artifacts of the Mac file system.
2.1.2 Subdirectories
The object directory has some subdirectories for groups of files and graphics, such as the /mtbs directory and /reunion-04. These subdirectories contain their own index.html. Subdirectories symlink to CSS and graphic files and the /js directory in the parent directory.
2.1.3 Graphics
Graphics files are stored in the subdirectory /mulimg, with thumbnails in /thumbnails150. Many graphics have a standard version and a -2x version for High DPI displays. Graphics files are listed in multhumbs.sql so that the graphic index multics-images.html can be generated.
2.1.4 PDF Files
PDF files are like graphics: they are stored in the object directory but do not have a corresponding source. If PDFs are generated from other software, the site editor keeps the source on the local computer and does the generation manually.
PDF files scanned from original documents should have OCR text so that web indexers can find them. I have used Adobe Acrobat Pro, which works OK on many documents. Old documents with colored backgrounds, poorly printed text, strange fonts, multi-column documents, and figures with text labels often lead to nonsense text. Retyping a whole document and re-drawing the figures may be preferable.
2.2 Source files
On the editor's computer, there is a source directory containing source files that are used to generate the object files. For most html files in the object directory, there is one .htmx file in the source directory in the object directory. (Sometimes it's more complicated. See below.)
2.2.1 Subdirectories
Wherever there is a subdirectory in the object hierarchy, there is a corresponding subdirectory in the source tree. These subdirectories contain their own index.htmx and have sub-Makefiles pointed to by the main Makefile. In such subdirectories, there are symlinks to .htmi files and graphic and thumbnail directories in the main directory.
There is a symlink to menubar.htmi in the main source directory. This file, generated by gen-menubar.htmt, prefixes each filename with %[hrprefix]% which is set by configx.htmi to be ../. Every invocation of expandfile in the Makefile expands configx.htmi before expanding the source file. This ensures that all links in the menus point correctly to files in the parent directory.
2.3 Expandfile
The Makefile uses the expandfile utility program to create HTML object files from HTMX source. expandfile and the software it depends on must be installed on the editor's computer. This program is written in Perl and is open source provided by the editor. See expandfile Installation Instructions.
2.3.1 Expandfile wrappers
Each HTMX file for a web page defines its content inside several BLOCKs. One is named body and lays out the page content. The HTMX file ends with an *include of a wrapper file that defines the template look and feel of multicians.org, and expands the body block. There are several different wrappers, depending on the type of page. Most wrappers produce a header, footer, and drop-down and alternate menus. Most page wrappers contain HTML that adapts the page to mobile devices by setting a "viewport" and providing alternate CSS rules by testing screen size.
| Wrapper file | Notes |
|---|---|
| pagewrapper.htmi | Standard page wrapper |
| .. class2head.htmi | Lays out the DIV for the headbar including date and title, inserts menubar.htmi |
| headpagewrapper.htmi | Special wrapper for index.html |
| codewrapper.htmi | Wrapper for a program listing, no viewport set |
| glospagewrapper.htmi | Wrapper for glossary pages |
| mspmwrapper.htmi | Wrapper for an MSPM page |
| mtbwrapper.htmi | Wrapper for an MTB, no viewport set |
| noncenteredpagewrapper.htmi | Wrapper for pages with long lines |
| paperwrapper.htmi | Wrapper for conference papers, no viewport set |
| widepagewrapper.htmi | Wrapper for site-timeline.htmx, cannot be reflowed, no viewport set |
2.3.1.1 Auxiliary include files
Page wrappers and source files include additional files. Here are the most popular.
| File | Notes |
|---|---|
| class2head.htmi | Lays out the DIV for the headbar including date and title, inserts menubar.htmi and altmenubar |
| menubar.htmi | Generated by gen-menubar.htmt from menubar.sql. Defines DIVs redmenubar and altmenubar. |
| mxstdfmt.htmi | included by pagewrapper.htmi. Standard page layout, CSS that hides/shows Drop-down and and mobile menus, loads jquery,includes ddm.htmi |
| ddm.htmi | JavaScript function for dropdown menus. |
2.3.2 Expandfile Macros
Expandfile provides useful HTML-generating macros. The Multics source file mxlib.htmi contains Multics-specific macros such as bibliolink and includes the library htmxlib.htmi, which provides macros for generating image references, such as getimgdiv. Generating multicians.org uses macro calls to generate IMG tags for photos. HTMX source files begin by including mxlib.htmi.
2.3.3 Tools invoked by Expandfile Scripts
Some expandfile templates invoke the *shell builtin function to execute helper tools that return character strings to be used in the expansion. These tools are either standard Unix utilities such as date, or tools provided by the site editor (many written in Perl). Tools specific to the website are stored in the subdirectory /tools of the source directory. The most important tool in this directory is expandfile and its components. (The /tools directory also contains a few other files for completeness, such as copies of .js files used in the object directory.)
2.4 SQL database
Loops Over Data Tables
About 45 expandfile template files use the *sqlloop builtin function one or more times to access my local MySQL database. The database tables are loaded from text .sql files which DROP, CREATE, and reload the tables each time the .sql file changes.
The Expandfile *sqlloop builtin expands templates for each selected row returned by an SQL query. Some MySQL tables are used to generate several HTMX templates: for example sites.html and site-timeline.html are created from loadsites.sql.
Formatting Lookup Macros
Special syntax in the source is expanded by expandfile into data retrieved from SQL queries. For example {{term anchor text}} expands to a hyperlink to the glossary entry for "term," with a TITLE attribute that previews the link target. Parameter values set in config2.htmi control which SQL tables are accessed for these definitions.
Database Connection Parameters
The *sqlloop builtin function and the special syntax references need the variables _xf_hostname, _xf_username, _xf_password, and _xf_database to be set to appropriate values for the local SQL database. Assignments to these values are kept in the local configuration file config2.htmi and most file expansions in the Makefile are performed by executing the little shell script expandfile_config, which invokes expandfile config2.htmi (rest of args)
The Makefile and shell scripts that execute mysql commands require that the file $HOME/.my.cnf is also set up with values for host, user, password, and database.
2.4.1 SQL tables
| Text file | Tables | Usage | Notes | Rows | Ref |
|---|---|---|---|---|---|
| g1.sql | glossary | One row per paragraph in the Glossary | printgloss3.pl writes 27 HTMX files | 1189 | {{ }} |
| home-slider.sql | homeslider | One row per photo on multics.html | Page build generates CSS sprites. | 15 | |
| loadbib.sql | bibtypes, biblio, authors, aka | One row per document, plus author info | expand 7 templates to produce 7 HTMX files | 5028 | |
| loadc.sql | changes | One row per change | Creates topchanges.htmi included by multics.htmx, changes.htmx, changes-old.htmx | 2212 | |
| loadext.sql | extref | One row per external page | Used in many pages. | 799 | {! !} |
| loadm.sql | m | List of Multicians | produces multicians.htmx | 2053 | {[ ]} |
| loadpages.sql | pages | Pages of the website | sitemap.htmx, Google sitemap | 459 | {@ @} |
| loadsites.sql | s | List of sites | extra rows for sites with complex history; used by sites.htmx and site-timeline.htmx | 102 | |
| chrono.sql | chdate, chevent, chflag | Chronological list of events. | 301 | ||
| chrono2.sql | ch2year, ch2thumb | Pictures in thumbnails150/ to decorate chrono.html. | 28 | ||
| menubar.sql | menubar | Layout of jQuery dropdown menus and altmenubar | two levels | 80 | |
| msource.sql | msource | One row per source file | source-index.htmx | 5877 | |
| multhumbs.sql | multhumbs | One row per thumbnail | produces multics-images.htmx | 660 | |
| mxawards.sql | mxawards | Awards earned by Multicians | used with loadm.sql | 252 | |
| publist.sql | publist | Documents produced by Multicians | produced by addwp.htmt, used with loadm.sql | 720 | |
| hiddenmail.sql | mmail | Email forwarding addresses (see 3.7). | 754 |
2.4.2 CSV files
| Text file | Usage | Rows |
|---|---|---|
| grpsio.csv | Members of groups.io/multicians | 167 |
This list is created manually for now, by a group admin going to the "Admin" section of the mailing list page and clicking "Members" and "download" and saving the result in grpsio.csv.
2.5 Executable files
The editor's computer has a standard OS installation with programming tools and a command shell and standard Unix tools installed. The Unix or Mac operating system can be used. Tools such as Perl and MySQL are installed by a package manager in /usr/local/bin (I use Homebrew on the Mac). My tool expandfile comes with some helper executables written in Perl and is installed in $HOME/bin. Additional tools are in the /tools subdirectory of the source directory.
Windows users can use a Unix-like environment such as Cygwin. For Windows 10, there is "Windows Subsystem for Linux". I have never used it; supposedly it provides a Unix kernel and Bash-like shell, and supports Perl and MySQL.
The editor's account needs a bin directory below the home directory, and a search path set up to find commands in these directories.
expandfile Install instructions describes the setup of Perl, CPAN, MySQL and expandfile.
2.6 Environment
The editor's computer has the environment variable PERL5LIB set to point to the correct version of Perl. .ssh keys are set up so that files can be sent from the editor's computer to the web host, and should be automatically loaded using ssh-add when the editor logs in.
2.7 Symbolic links
The source directory needs a few links to items in the object directory. In particular, in order for the image reference macros such as getimgdiv to work correctly, they need to access the pixel dimensions of target images. Symbolic links to the object subdirectories /mulimg and /thumbnails150 are placed in the source directory.
3. How the Build Process Works
This section describes the inner workings of the build machinery, so that it can be changed when new features are required.
3.1 Make
The Unix make utility generates the object directory from the source directory. It is driven by a file in the source directory called Makefile.
The Multics website's Makefile has recipes that expand any .htmx file that is older than the corresponding .html. In addition, each SQL table is loaded from a .sql text file, and the Makefile ensures that if the .sql file changes, it is loaded into SQL and then relevant .htmx files are generated. make takes care of ordering the generation steps.
3.1.1 Makefile Configuration
The Makefile refers to some external-world facts by symbolic configuration names. The actual values of these symbols is set in the file configmake.mk which is included by the Makefile. (This makes it possible to test new versions of the site or Makefile, and may someday support multiple editors.)
# Configuration constants for Makefile for Multics website # place this in the multicians source directory # this is where site-dependent values like path names are kept PRIMARYWEBACCT = account@server PRIMARYWEBPATH = directory path on server PRIMARYWEBPATHSRC = directory path on server
3.1.2 Makefile Rule for Expanding Files
If you want to know how it works, you have to learn to read the Makefile. Makefiles define values for some variables, and then provide a set of rules. The Makefile for multicians.org rarely changes. Here is an important rule that invokes expandfile:
# Rule that makes the HTML files from HTMX files and includes. $(OBJ_DIR)/%.html: %.htmx $(CONFIG) $(INCLUDES) @chmod +w $@ $(EXPAND) $< > $@ @checknonempty $@ @-/usr/local/bin/tidy --markup no --quiet yes --show-warnings no $@ @chmod 444 $@
This rule means "Use expandfile to build the object file if the corresponding source, or any of the include files, or the config file, have changed."
Makefile rules have three parts: the "target" file(s) being made, a colon, a list of the "dependencies" -- files that the target depends on, and then an indented list of commands to execute to create the target from the dependencies.
The rule above says "do the commands if ANY of the dependency files is newer than the target."
(Note that if there is an .htmx file that does not have a corresponding .html file, make will do nothing. So when adding a new file to the site, the first step is to create an empty .html file that will be older than the new .htmx.)
make examines all the rules in Makefile and executes the rules in the "right" order, so that dependencies are built before the files that depend on them.
The percent sign in the target is a wildcard that matches any name. (Used in a dependency, on the other side of the colon, it has the same value.) This example will cause fred.html to be remade if fred.htmx is newer.
In the Makefile rule for making HTML files:
- Lines beginning with # are comments.
- The make variable OBJ_DIR is a pathname prefix for files in the object directory. Its value is usually ../multics.
- The variable CONFIG is the name of the configuration file, config2.htmi. It is included to cause all object files to be recompiled if the configuration is edited.
- The variable INCLUDES is a list of wrappers and common files. Changes to any of these files cause all object files to be recompiled.
- The variable EXPAND is the command to invoke expandfile.
- The first line of a rule has a target to be built, a colon, and the prerequisite files.
- Subsequent lines of a rule begin with TAB and are commands that are executed to build the target.
- The expression $@ refers to the pathname of the target, e.g. ../multics/fred.html.
- The expression $< refers to the pathname of the source file, e.g. fred.htmx.
- Beginning a command with @ does not echo the command when make runs.
- Beginning a command with - does not abort make if the command gets an error.
- The chmod commands make the target writeable only while it is being regenerated, as a reminder not to edit the HTML file.
- The checknonempty command prints a message and terminates with an error code to stop the build if an expansion generates a zero length target HTML file. (This can happen if expandfile encounters an error such as a missing *include file.)
- The tidy command prints messages if there are HTML errors in the target. (Install it from https://w3c.github.com/tidy-html5/.) The build is not stopped. Usually this error occurs because there is an HTML syntax error in the source file. I re-run tidy without the --quiet yes to see what errors are in the file, looking at the new .html file if necessary, fix the problems, and rebuild.
3.1.3 Makefile Variables
| Variable | Contents | Meaning |
|---|---|---|
| CONFIG | config2.htmi | Constants for multicians.org used by expandfile, inclding the MySQL database connection parameters. |
| OBJ_DIR | ../multics | relative pathname to the object dir from the source dir. config2.htmi has a var named obj_dir that must match this value. |
| INCLUDES | menubar.touch class2head.htmi linktags.htmi textnav.htmi addrtag.htmi leftpanelstyle.htmi mspmstyle.htmi mxstdfmt.htmi pagewrapper.htmi headpagewrapper.htmi glospagewrapper.htmi noncenteredpagewrapper.htmi paperwrapper.htmi codewrapper.htmi htmxlib.htmi mxlib.htmi | a list of files that all expansions depend on |
| DATABASE | thvv_userlist | local MySQL database on the editor's computer |
| HOMEREL | ../ | prefix used by subdirectories to find the main directory |
| BINDIR | $(HOME)/bin | library directory on the editor's computer |
| EXPAND | $(BINDIR)/expandfile_config | Shell script that invokes expandfile with first argument config2.htmi |
| SUBTGTS | r2012d r2004d rvand mtbsd mabsd debugd blwd pgd | list of sub-make-targets that should be built as part of 'all' |
| LOGACTION | $(BINDIR)/wtlog.sh | command executed to log updates |
Some variables must be different on different build environments. These variables are set in the included configuration file configmake.mk. For example, OBJ_DIR might be set to ../multics_object, and EXPAND is set to $HOME/bin/expandfile.
| Shell Variable | Meaning |
|---|---|
| PRIMARYWEBACCT | username@DNS name of ISP web hosting computer |
| PRIMARYWEBPATH | pathname of web directory on web host |
| PRIMARYWEBPATHSRC | pathname of source directory on web host |
(It may occur to you that the Makefile rule rebuilds some files that do not need rebuilding. For example, if any wrapper include file in the INCLUDES list changes, every file will be re-expanded, not just the ones that include the changed file. We could fix this, at the cost of increased complexity in the Makefile. Since it takes less than 2 minutes to re-expand all files, the savings are not worth the development and future maintenance effort.)
Besides rules for creating HTML files, there are also rules that specify additional dependencies for some files, in order to cause some files to be built after other files.
3.1.4 Other Unix Tools invoked by Makefile
As mentioned above, when .html files are generated, the Makefile executes the Unix HTML Tidy command to check if the files have HTML syntax errors.
The "install" recipe in the Makefile uses the rsync command to copy modified files from the object directory to the web host computer(s). The recipe causes a "make all" before executing, so to make a one-line change, the editor just edits the file and types "make install". (Check the version of rsync on the editor's computer. MacOS shipped an old version, 2.6.9. I needed to install version 3.2.7 from Homebrew, Pair has 3.2.5.)
3.2 Generation of HTMX from SQL
Certain HTMX files are generated by make from the SQL database if their .sql text file source changes. (These HTMX files are marked read-only so that an editor doesn't forget and edit them directly.) The Makefile notices that the .sql file has changed by noticing that the date modified for the .sql file is newer than the corresponding zero-length .touch file, loads the .sql file into the database, expands a template file for the modified table which generates one or more .htmx files, and executes touch filename.touch so the reload will be done only once. Updating the date modified on filename.touch also causes make rules that depend on this file to be executed.
Additional dependency rules for compiling .htmx files into .html are also present in the Makefile. The .touch file is listed as a dependency in a Makefile rule to cause HTML files to be rebuilt if they may depend on the table contents. There are about 10 such rules in Makefile. For example, the Makefile rule
# .. rebuild about-multics for count of pages, graphics, or changes $(OBJ_DIR)/about-multics.html: changes.touch pages.touch multhumbs.touch
declares that about-multics.htmx should be re-expanded into about-multics.html if any of the "changes", "pages", or "multhumbs" database tables change, in order to ensure that the counts of changes, pages, or images are re-computed. This rule adds reasons to regenerate its target to those declared in the standard rule for .html files, and ensures that about-multics.html is regenerated after its prerequisites are generated. (If we discover that we have to make twice to cause an output file to be correct, this is a sign that a dependency rule is missing.)
3.2.1 HTMX files generated from SQL
| Filename | Table | SQL file | Template |
|---|---|---|---|
| sites.htmx | sites | loadsites.sql | glostpt.htmt |
| site-timeline.htmx | sites | loadsites.sql | glostpt.htmt |
| mgloss.htmx | mgloss | g1.sql | glostpt.htmt |
| mga.htmx | mgloss | g1.sql | glostpt.htmt |
| mgb.htmx | mgloss | g1.sql | glostpt.htmt |
| mgc.htmx | mgloss | g1.sql | glostpt.htmt |
| mgd.htmx | mgloss | g1.sql | glostpt.htmt |
| mge.htmx | mgloss | g1.sql | glostpt.htmt |
| mgf.htmx | mgloss | g1.sql | glostpt.htmt |
| mgg.htmx | mgloss | g1.sql | glostpt.htmt |
| mgh.htmx | mgloss | g1.sql | glostpt.htmt |
| mgi.htmx | mgloss | g1.sql | glostpt.htmt |
| mgj.htmx | mgloss | g1.sql | glostpt.htmt |
| mgk.htmx | mgloss | g1.sql | glostpt.htmt |
| mgl.htmx | mgloss | g1.sql | glostpt.htmt |
| mgm.htmx | mgloss | g1.sql | glostpt.htmt |
| mgn.htmx | mgloss | g1.sql | glostpt.htmt |
| mgo.htmx | mgloss | g1.sql | glostpt.htmt |
| mgp.htmx | mgloss | g1.sql | glostpt.htmt |
| mgq.htmx | mgloss | g1.sql | glostpt.htmt |
| mgr.htmx | mgloss | g1.sql | glostpt.htmt |
| mgs.htmx | mgloss | g1.sql | glostpt.htmt |
| mgt.htmx | mgloss | g1.sql | glostpt.htmt |
| mgu.htmx | mgloss | g1.sql | glostpt.htmt |
| mgv.htmx | mgloss | g1.sql | glostpt.htmt |
| mgw.htmx | mgloss | g1.sql | glostpt.htmt |
| mgx.htmx | mgloss | g1.sql | glostpt.htmt |
| mgy.htmx | mgloss | g1.sql | glostpt.htmt |
| mgz.htmx | mgloss | g1.sql | glostpt.htmt |
| biblio.htmx | biblio,bibtypes,authors,aka | loadbib.sql | glostpt.htmt |
| mspmtoc.htmx | biblio | loadbib.sql | mspmtoc.htmt |
| multicians.htmx | m,mxawards,awards,publist | loadm.sql,mxawards.sql,publist.sql | multicians.htmt |
| changes.htmx | changes | loadc.sql | changes.htmt |
| sitemap.htmx | pages | loadpages.sql | sitemap.htmt |
3.2.2 HTMX files that fetch values from SQL
| Filename | Table | SQL file | Values |
|---|---|---|---|
| multics.htmx | homeslider | home-slider.sql | sliding picture file |
| about-multics.html | changes | loadc.sql | count of changes |
| articles.html | pages | loadpages.sql | count of articles |
| changes-old.html | changes | loadc.sql | counts of changes by year |
| mac50-videos.html | multhumbs | multhumbs.sql | list of videos |
| mult-links.html | groupsio | groupsio.sql | count of mailing list members |
| multics-images.html | multhumbs | multhumbs.sql | thumbnail image paths |
| multics-stories.html | pages | loadpages.sql | tables of stories |
| people-pictures.html | multhumbs | multhumbs.sql | image paths |
| site-timeline.html | s | loadsites.sql | site names and dates, data for pie and bar charts |
| sites.html | s | loadsites.sql | site information |
| source-index.html | msource | msource.sql | source file names and info |
| chrono.htmx | chrono | chrono.sql, chrono2.sql | dates and events, thumbnails |
The following Multics specific expressions appear in many HTMX files and are replaced by the results of an SQL query.
| Expression | Table | SQL file |
|---|---|---|
| {! ... !} | extref | loadext.sql |
| {@ ... @} | pages | loadpages.sql |
| {[ ... }} | multicians | loadm.sql |
| {{ ... }} | mgloss | g1.sql |
| bibliolink macro calls | biblio | loadbib.sql |
| mtblink macro calls | biblio | loadbib.sql |
3.3 Makefile Rule for Sending files to the Web Host Computer(s)
The Makefile invokes the rsync command to publish files from the personal computer to the web host computer(s). rsync doesn't copy files that it doesn't need to. Files are sent securely so they can't be intercepted or tampered with, and the file transmission is compressed so it will go quickly.
The rsync command is issued by make instead of typed directly. The Makefile gets the rsync local and remote path names from Makefile variables. rsync uses SSH keys in $HOME/.ssh to be able to send files to the web host computer(s).
Specific Makefile rules can be executed by specifying their name as an argument to make. The command make install invokes rsync:
# make install pushes modified files to the primary website and also backs up the source install: all incrementversion vnfile expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv $(LOGACTION) updated multicians rsync -avzu --blocking-io -e ssh --exclude .DS_Store $(OBJ_DIR)/ $(WEBACCT):$(WEBPATH) rsync -avzu --blocking-io -e ssh --exclude .DS_Store --exclude $(CONFIG) . $(WEBACCT):$(WEBPATHSRC) date
This rule means "Use rsync to send any modified object files to the web host computer, and also update a copy of the source files on the web host." It also updates a file dashboard.csv which visitors can look at to see counts of files on the site and the time of last update.
See Using Unix tools with expandfile for an explanation of rsync args.
3.4 Other Makefile Rules
The Makefile has other interesting rules.
3.4.1 Version Number File
The rule that pushes files to the ISP calls the shell script incrementversion to update vnfile. The contents of the file vnfile look like this:
6226 2023-03-13 10:54
The sequential number is incremented every time a make install is done, so this is the number of installs done since some time in 2006, when I first set up this file. It is followed by the date and time of the push. The shell script incrementversion updates these values. vnfile is included into an HTML comment in multics.html when that file is built. (The sequential number is not the same as the number of file changes, since a single install may install many files, or I may do repeated installs to get a single change just right.)
3.4.2 Dashboard File
The rule that pushes files to the ISP executes expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv to update dashboard.csv. The file dashboard.csv is created by expanding the template dashboard.csvt. It contains two lines, a list of column headers and a list of corresponding column values:
changect,asof,multicians,multiciansmail,multiciansurl,multiciansdec,glossent,npages,\ npdf,alldoc,alldoc-online,images,mspm,mspm-online,nsites 5388,"2020-07-02 13:35",2039,749,169,90,839,466,601,4994,2669,639,838,54,84
This file repeats the version number, and then has counts of the number of various items on multicians.org, such as number of multicians. It is intended to assist in the construction of other sites that refer to our site's contents. A file with this format can be read using the Expandfile builtin *bindcsv.
3.4.3 Subdirectories
Each subdirectory has a sub-Makefile which is invoked via a "subdirectory target" in the Makefile. The $SUBTGTS variable names these targets, and this variable is listed as a dependency for the "all" target. The result of this setup is that just typing make will also invoke the sub-Makefiles and make whatever is out of date. Most subdirectories have symbolic links to standard .htmi files in the parent directory, as well as the image and thumbnail directories. Subdirectories have a configx.htmi file that may override some path definitions in config2.htmi so that subdirectory files refer correctly to other files on the site. If you are setting up a subdirectory, it is best to clone the Makefile from one that already works, and make the same symbolic links.
3.4.4 Special build instructions
Several rules in the Makefile create special files which are published to the host ISP, using Unix utility commands in addition to expandfile. For example, make_mx_tar is a shell script that crates a gzipped tarfile listing all files that should be shipped to mirror sites. Generating it automatically saves me from having to remember to edit another file whenever I add a page to the site.
| File | Function |
|---|---|
| sitemap.xml.gz | Gzipped XML sitemap used by Google crawler |
| multicians.procmail | Multicians email forwarding file (not visible on web) |
| make_mx_tar | shell script that creates a tarfile of the site, for mirrors (not visible on web) |
3.4.5 Error checking performed by the build process
File building rules check for errors and alert the editor of any problems whenever make rebuilds a file. For ordinary HTML file creation, the auxiliary program checknonempty is invoked to check for the case where a target file is nonexistent or empty after expandfile is invoked. (This can happen when expandfile aborts due to a missing include file.) If this happens, make exits with an error. After you fix the original error, make sure that the HTML file was not deleted.
Newly created HTML files are also checked by the HTML Tidy utility, which prints a line if there are any warnings or errors. When this happens, the editor should investigate the failing file and correct the problem.
For a few cases where SQL files are reloaded, the make recipe runs custom shell scripts to check that foreign key relationships are satisfied. In particular, the script checkbib.sh ensures that the tables loaded by loadbib.sql have authors entries for each document, that aka rows are present for each author, and that m rows for each author exist. If any errors occur, the editor should investigate loadbib.sql and correct the problem.
The rule that pushes files to the ISP executes check_missing_extref.sh, which checks output files for the sequence ###, which is produced when a {@path anchor@} sequence is executed and does not find path. This warns that we have generated a dead intra-website link. (Some of these are OK, and represent unfinished improvements.)
The rule that pushes files to the ISP also executes check_quotes.sh, which checks HTMX files for characters such as curly quotes that display poorly in browsers. (This can happen if text is copied and pasted from mail messages or Word documents.) These characters should be replaced by HTML entities.
3.5 Source file suffixes
These suffixes are my convention for indicating the use of the content in its file name. expandfile doesn't interpret files differently depending on their names. Macros defined in mxlib.htmi have some suffix dependencies. The Makefile generates .html from .htmx files.
| Suffix | Meaning |
|---|---|
| htmx | HTML eXtended, generates HTML |
| htmt | HTMX Template, generates HTMX |
| htmi | HTMX Include file |
| html | HTML |
| tpt | template, generates non-HTML files such as site map |
| css | style sheet |
| csv | Comma Separated Values |
| csvt | template that generates CSV file |
| e | text file (emacs) |
| faq | Usenet source, posted to alt.os.multics |
| gif | bitmap graphic |
| htmq | test htmx file |
| jpg | JPEG graphic |
| js | JavaSciipt |
| mk | Makefile include |
| pl | Perl language |
| png | portable network graphic |
| procmail | program for procmail |
| py | Python |
| sh | shell script for /bin/sh |
| sql | MySQL source |
| txt | text file |
3.6 Standard Template Expansion with Wrappers
Most pages' HTMX files conform to a standard pattern. Each file contains
- Introductory comments in %[** ... ]%
- *set statements for title, description, keywords, etc
- An optional *set statement for headingdate (If this value is not supplied, the wdate from the pages table is used. If that is not supplied either, the date modified of the source file is used.)
- %[*include,=mxlib.htmi]% which contains *block definitions for macros.
- Comments including copyright date and revision history.
- Optional *sqlloop statements that do SQL queries and expand template blocks, appending the expansions to a variable which is used in the body.
- Optional %[*block,&extrastyle,^END]% followed by items for the HEAD, ending with a line with just END
- Other optional %[*block,&...,^END]% constructs containing iterator blocks expanded for each row by *sqlloop, ending with END
- %[*block,&body,^ENDBODY]% followed by body HTML, ending with a line with just ENDBODY. (The body may contain references to variables and invocations of builtin functions.)
- The last line is *%[include,=wrapper.htmi]% -- everything else just set things up: this statement actually generates output.
Expansion of normal HTMX files proceeds as follows. expandfile
- Reads config2.htmi to set configuration constants, MySQL configuration, etc.
- Ignores comments in the input HTMX file.
- Sets variables and blocks. (Does not expand variable references when blocks are defined.)
- Optionally executes *sqlloop builtins to execute SQL queries that create variables containing the expansion of an iterator template block for each query row.
-
Includes and expands the wrapper,
which outputs standard HTML structure and expands variable references, including %[body]%.
If an expanded variable like body contains variable references, these are expanded also.
Page wrappers begin by querying the pages table:
- Executes SELECT wdate FROM pages WHERE sourcefile='%[mainfilename]%'
- Executes SELECT crumbs FROM pages WHERE sourcefile='%[mainfilename]%'
- Executes SELECT l2navfile FROM pages WHERE sourcefile='%[mainfilename]%'
- Outputs the HEAD section
- Expands title, description, and keywords obtained from the *set statements, and minwidth, maxwidth, and contentwidth obtained from config2.htmi
- Executes *include,=mxstdfmt.htmi, which includes standard CSS boilerplate for the site
- Expands extrastyle if it was defined in the source file. This block may include JavaScript files and define STYLE settings local to the file.
- Outputs the BODY section:
- Includes class2head.htmi, which defines the page's standard top bar (which will use the wdate and crumbs from above) and drop-down menus.
- Expands the body block, set by the HTMX file, and variable references contained in it. This block may include result variables set by *sqlloop.
- Expands the extratail if the file supplied it.
- Formats and outputs tail navigation, inclding l2navfile if it was obtained from pages.
- Closes the BODY and HTML.
Alternate wrappers support variations for special page formats, such as MSPM sections or Glossary pages.
3.6.1 Why do this?
Expanding items that expand other items may seem indirect and convoluted. However, this scheme spares the writer of simple pages from the details of the site implementation, reduces the chances of a simple change breaking a page, and enables global changes when site design or HTML standards change. In practice, a one-line change can be made and published in about a minute; adding a new page to the site, cross-linked to other pages, visible to Google, and findable by standard navigation, can be done in only a few more minutes.
3.7 Additional Functions of Website Host
HTML pages, whether hosted on the primary site or mirror sites, invoke CGI functions on the primary host site.
The primary host provides additional functions for the support of multicians.org:
- Inbound email filtering and forwarding for Multicians, invoked by procmail
- Web server CGI support for form-based outbound mail to Multicians and the editor
- Web server CGI support for form-based signup and registration editing requests by Multicians, emailed to the editor
Protected mail addresses are in a file called hiddenmail.sql which is not published on the web. This SQL file is loaded into a MySQL database on the primary host site. The database is not accessible from the Web; CGIs on the primary host site access it and return values to web pages, after checking that the request is valid. Mail forwarding addresses are not visible to web crawlers (unless the ISP account gets hacked).
3.8 Tools
Tools needed to maintain Multics site provided by others
These are installed in /usr/bin by the OS distribution or /usr/local/bin by package managers.
- Shell (might work with Cygwin, haven't tried it for years)
- Perl language, including CPAN modules DBI and DBD::mysql
- Remote copy command rsync
- Remote access commands ssh (or PuTTY)
- Unix commands make, touch, chmod, diff, cp, cat, ls, echo, mv, sh, grep, sed, ed, gzip, git
- SQL (MySQL), including command line interface and Perl DBI (currently no version 4/5 features are used)
- HTML-aware text editor, such as emacs
- Photoshop, Affinity Photo, or Gimp
- ImageMagick tool convert
- JQuery 3.4.1. - https://jquery.com
- EasySlider 1.7. - Alen Grakalic - https://github.com/atierant/easySlider (2011)
- fancybox jquery plugin: see http://fancyapps.com/fancybox/3/
- Optional: HTML Tidy (HTML5 version) - https://github.com/htacg/tidy-html5
- Optional: Integrity link checker (Mac)
- Optional: free geolocation database from https://maxmind.com/ (Needs license account.)
3.8.1 Tools needed to maintain Multics site provided by editor
See expandfile web pages. These are mostly installed in the /tools subdirectory of the source directory.
- HTMX language (Perl): expandfile, expandfile.pm, readbindsql.pm, readbindxml.pm
- HTMX Site macros and templates, especially htmxlib.htmi and mxlib.htmi
- MySQL schemas
- Configuration file config2.htmi
- HTMX helper programs (Perl) called by *shell: filemodshort, dbcounts, filesizek, nargs, filedaysold, filedmodiso, checknonempty, gifsize, gifsize2
- Special page generators (Perl/SQL) generating HTMX: printgloss3.pl, printsites2.pl, gen_site_timeline.pl
- Special page generators (Perl/SQL) generating SQL: addwp.htmt
- Generate USENET postings: auto1a.sh, post.pl (and a USENET account somewhere, I use a free one at eternal-september.org)
3.8.2 The /mxs/tools directory
-- /mxs/tools --- $HOME/bin .. should be hard-linked to tools dir ----- wtlog.sh, calls $HOME/wtx/wtlog (Perl) ----- expandfile_config (script, calls expandfile with config file) ----- check_missing_extref.sh ----- printsites2.pl ----- gen_site_timeline.pl ----- auto1a.sh, calls post.pl
3.8.3 Web Host Programs and data files in cgi-bin directory provided by editor
- Configuration file mxconfig.htmi
- mxmail.cgi, mxmailform.cgi, ckem.cgi, thvvutil.pm, challenge.pm
- mxregister.cgi, mxregister4.cgi
- Template files for mxmail and mxregister
- special.pmrc for spam checking messages sent via the web form, using procmail
Some of these tools invoke the subroutine version of expandfile and link to expandfile.pm in $HOME/bin
3.8.4 Web Host Programs and data provided by web host and site admin
- Apache web server, including .htaccess support, NCSA combined web log, CGI Perl execution, DEFLATE, Content Security Policy, Strict Transport Security.
- cgiwrap support (Pair written)
- procmail support for incoming mail and forwarding, invoked by qmail (Pair installed)
- Shell access via ssh (freeBSD, Pair configured)
- scp/rsync access (freeBSD, Pair configured)
- SQL (MySQL), including command line interface and Perl DBI and CGI execution. (Pair installed and configured)
- CPAN and library modules that support GeoIP. (Pair installed)
3.8.5 Web Host Runtime Functions Provided by Editor
These functions are installed in the js subdirectory of the object directory.
- Pie Chart: piecanvas.js used to display statistics in site-timeline.html
- Timeline: timelinecanvas.js used to display graphical time lines of events in history.html and b2.html
3.9 Performance
Expandfile is written in an interpretive language, so it is not blazingly fast, but it is fast enough. Re-expanding all 467 pages on the site takes about 2 minutes. Pages that have a lot of *shell calls are slower: for example, a page like people-pictures.htmx that has to involve a lot of *shell calls to Perl helper program gifsize2 has to launch a shell and a Perl interpreter for each of over 100 pictures. If I built gifsize2 into expandfile this could be a lot faster. Not worth the effort at this time.
3.10 Git
The source directory contains a local Git repository that contains all the source .htmx and {.sql:} files. Doing a make install does a git status. After making and installing a set of changes, the editor should do a git commit -a with a brief message describing the changes.
--- what files are in git, where the repo is --- email address file not included --- ignored files --- future: upstream repo
The Makefile rule that pushes files to the ISP ends with a git status which reminds the editor to add or commit changes to the git repository.
3.11 Files and their purpose
(by THVV unless indicated)
3.11 1 Tools invoked by Makefile
auto1a.sh (post FAQ to USENET) checknonempty config2.htmi expandfile2 gen_site_timeline.pl incrementversion nextpostingexpires post.pl printgloss3.pl printsites2.pl
3.11.2 Tools invoked by *shell in an HTMX file
dbcounts (queries SQL database for totals) get-facebook.sh (calls get-facebook-fan-count.pl) filemodshort (comes with expandfile) filemodyear (comes with expandfile) filesizek (comes with expandfile) firstletter (comes with expandfile) gifsize2 (comes with expandfile) nargs (comes with expandfile)
3.11 3 Files used to support ISP mail and signup on multicians.org host /cgi directory
errtpt.txt mform-thanks.txt mformtpt.txt mxaddrtag.htmi mxclass2head.htmi mxh1style.htmi mxmailerr.txt mxolwrapper.htmi mxregerr.txt mxregister.cgi mxregister4.cgi mxstdfmt.htmi mxtextnav.htmi reloadm special.pmrc thvvutil.pm
3.12 Displaying graphics files on the site
There are five main methods the site uses to display graphics on a web page, depending on the desired appearance and visitor experience. These methods are supported by Expandfile macros. The macros generate HTML IMG tags to display the graphics: they make it easy for page authors by looking up the the image size at compilation time, and using additional versions of graphics to support High DPI displays. Using the macros makes page creation easier and generates sharp looking graphics.
Graphic images in jpg, png, and gif format are stored in the directory /mulimg, and are used at compile time and at page display time. Every thumbnail is stored in /thumbnails150 as a 150x150 thumbnail to be displayed in a 75x75 space.
There are five methods used to display graphics on a page, supported by different macros. Examine the source of existing pages to see how these macros are used. (For example, picnics.htmx uses both the second and third methods to display pictures.)
-
Display a graphic in a DIV, possibly floated to the left or right. Use the getimgdiv macro to do this. Given a graphic name such as xxx.jpg, this macro looks for both xxx.jpg and xxx-2x.jpg. If both are found, an IMG tag that uses the SRCSET attribute is generated.
-
Display a graphic in a DIV, possibly floated to the left or right, and if the graphic is clicked, display a larger version of the graphic in a popup with a Multics banner and a "close" control. In the extrastyle block of a multicians.org source page, include the file use-dhtml-lightbox.htmi. At the top of the body block of a multicians.org source page, include the file dhtml-lightbox.htmi. Use the getimgpopdiv macro to insert a graphic in a DIV block, with an optional CSS tag. Use the getimgpopdiv75 macro if the base graphic is a 150x150 thumbnail displayed in a 75x75 space. These macros look for the given file name and a -2x version of the file, and generate the appropriate IMG tags and popups.
-
Some pages display galleries of thumbnails. (cisl-wake.html, cno-pictures.html, lcs25.html, lcs35.html, people-pictures.html, picnics.html, etc.) Clicking on any thumbnail opens a popup, with arrow keys that allow the visitor to go to the next item in the gallery. In the extrastyle block of a multicians.org source page, include the file usefancybox.htmi to include the JQuery JavaScript file fancybox.js. Create a DIV container for the gallery with a unique ID tag; inside the DIV, place a UL container. Then insert a call to the getfancybox_li macro for each thumbnail, specifying the thumbnail/image name and the caption, to generate an LI element for the thumbnail. This method and the "popdiv" method can be used on the same page. The thumbnails are displayed using CSS sprites, to speed up page loading. lightbox_li is used with an older image popup display used in several pages that display image galleries. Other pages use use-dhtml-lightbox.htmi and dhtml-lightbox.htmi and generate IMG tags with the macro getimgpopdiv. (For example, Picnics uses both.)
-
On the home page multics.html, a 300x244 window with a sequence of sliding images is displayed by easySlider.js. The pictures and their attributes are specified in the homeslider table, queried when the home page is compiled. The query generates a packed graphic image with CSS sprite definitions for each individual picture. Each picture is 600x488, so that they will look sharp on a High DPI display. References to the pictures are in individual LI items in a UL element: EasySlider handles the formatting and picture changing. When the home page is compiled, a caption specified by the homeslider table is overlaid on each graphic: this caption's definition may invoke command line programs with *shell to get counts from the database, to produce legends like "84 Multics Stories."
-
The Image Gallery page multics-images.html displays all thumbnails on the site, grouped by general type. It uses CSS sprites, so that instead of loading 462 small graphics, it loads five large ones, which is much faster. Each thumbnail links to a page that contains the graphic. All thumbnail links are generated at compile time by five SQL queries of the multhumbs table.
See How I use Affinity Photo for a cookbook on processing graphics, sharpening, etc.
3.12.1 Generating 2x Graphics from Adobe Illustrator files Using Affinity Designer and Affinity Photo
Here is my recipe for generating .png graphics from .ai files. (So I don't forget.) The goal is to take a diagram in Illustrator and make 2x and 1x versions of the graphic as PNGs.
- Read the xxx.ai file into Affinity Designer
- In the dialog box, accept the font substitution, set bold, set the DPI to 144, leave colorspace at 'estimate'
- Select all, export PNG, set Area to "selection with background", leave size alone
- Save with name to mulimg/xxx-2x.png
- Read in xxx-2x.png into Affinity Photo
- Resize to half size, sharpen, and save as mulimg/xxx.png (remember to make it PNG, not JPG)
- Edit the source file to invoke a macro such as getimgdiv referencing mulimg/xxx.png
- Recompile. On a machine with a HiDPI display, browsers will load xxx-2x.png and display it in 1/2 the CSS dimensions, because the SRCSET selects the 2x file; the browser knows the screen resolution and uses all the pixels to make a sharp image. On a machine without a HiDPI display, browsers will load xxx.png and display it, because the SRCSET selects the 1x.
- Generate a thumbnail for the image using gth2x xxx-2x.png and move /mulimg/thumb2-xxx-2x.png to /thumbnails150/xxx.png.
- Update multhumbs.sql to use the new thumbnail.
- Verify that everything displays correctly.
- If there are /mulimg/xxx.gif or /thumbnails150/xxx.gif files, remove them.
4. Expandfile Documentation
- Expandfile
- Expandfile Tutorial
- Expandfile Multics Arguments
- Expandfile Wrappers
- Expandfile Macros
- Using Unix Tools with expandfile