Multics > About
29 Jul 2020

Web Site Build Process

History | People | Library | Sites | About Search

Tom Van Vleck

The multicians.org website is built on the site editor's computer from HTMX source using Unix tools and the open source program expandfile, and installed on the web host computer.

These are my rough notes on the site building machinery, including usage, internal details, and possible future improvements.

Rules for maintenance and content of multicians.org are in Web Site Design.
Detailed instructions for maintenance tasks for multicians.org are listed in Maintenance tasks for the Multicians Website.

Unix tools flow

1. Build Process Overview

1.1 General Principles

The site's build process attempts to

1.2 ISP Host Copy and Editor's copy

The domain multicians.org maps to a computer operated by an Information Services Provider (ISP) that serves the Multicians website. That computer runs a web server that responds to HTML requests with an URL pointing to multicians.org with the content of the requested site resources. Other mirror sites can also provide the same content at other URLs.

The site editor has a complete copy of the multicians.org website content. To make a change, the editor prepares an updated or added page on the local computer, checks it locally, and then publishes modified pages to the multicians account on the ISP.

1.3 Source and Object

Rather than editing HTML language files, the editor works on local source files that are translated to HTML. Source files are written in the "HTMX" (HTML eXtended) language. .htmx files are translated to .html files by a program called expandfile.

Source and object are kept in separate directories on the editor's computer. (A copy of the source directory is also stored at the ISP, for backup, not available on the Web.)

1.4 Steps in updating an HTML file

The basic steps in generating an update to multicians.org are:

  1. Edit one or more local source files, and possibly add graphics or other files to the object directories.
  2. Execute the make command to build a new local image of the site by expanding HTMX files to HTML with expandfile.
  3. View and check the local site object files generated by make.
  4. Iterate 1-3 until satisfied.
  5. Execute the make install command to push the local image to the web host directory.

This process requires that software and configuration files are installed correctly.

1.5 Use of Make Utility

The website's files on the web server are kept up to date by executing the make command in the source directory on the editor's computer. (The make command is available on Mac, Linux, Unix, and Windows systems. The original make command was created for Unix by Multician Stuart Feldman.)

The make command is used for two major tasks:

Why do it this way? This setup

Editors don't need to know how it works to use it for routine changes.

There are many features of make that I haven't learned yet. The Makefile could be a lot better. Suggestions are welcome.

2. Installing and Configuring the Build Machinery

A number of files have be be in the right places on the editor's computer for the build process to work.

2.1 HTML Object files

On the editor's computer, there is a complete local image of all files needed to display the web site, such that the web hosting computer can be updated using the Unix rsync command. These files are kept in the object directory. These files include .html files, .pdf files, and graphics. The object directory has some subdirectories for groups of files.

An editor can view the local image with a web browser to see how the site will display before updating.

2.2 Source files

On the editor's computer, there is a source directory containing source file that are used to generate the object files. For most html files in the object directory, there is one .htmx file in the source directory in the object directory. (Sometimes it's more complicated. See below.)

Expandfile

The Makefile uses the expandfile utility program to create HTML object files from HTMX source. expandfile and the software it depends on must be installed on the editor's computer. See expandfile Installation Instructions.

Expandfile wrappers

Each HTMX file for a web page ends with a *include of a wrapper file that contains the template look and feel of multicians.org. There are several different wrappers, depending on the type of page. Most wrappers produce a header, footer, and drop-down menus.

Wrapper fileNotes
pagewrapper.htmiStandard page wrapper
headpagewrapper.htmiSpecial wrapper for index.html
codewrapper.htmiWrapper for a program listing, no viewport set
glospagewrapper.htmiWrapper for glossary pages
mspmwrapper.htmiWrapper for an MSPM page
mtbwrapper.htmiWrapper for an MTB, no viewport set
noncenteredpagewrapper.htmiWrapper for pages with long lines
paperwrapper.htmiWrapper for conference papers, no viewport set
widepagewrapper.htmiWrapper for site-timeline.htmx, cannot be reflowed, no viewport set

Expandfile Macros

Expandfile provides useful HTML-generating macros. The Multics source file mxlib.htmi contains site-specific macros such as bibliolink and includes the library htmxlib.htmi, which provides macros for generating image references, such as getimgdiv. Generating multicians.org uses macro calls to generate IMG tags for photos. Almost every multicians.org HTMX source file begins by including mxlib.htmi.

2.3 SQL database

Some expandfile template files use the *sqlloop builtin function to access a local MySQL database. The database tables are loaded from text .sql files which DROP, CREATE, and reload the tables each time the .sql file changes.

The *sqlloop builtin expands templates for each selected row returned by an SQL query. Some MySQL tables are used to generate several HTMX files.

Special references in the source are expanded by expandfile into data retrieved from SQL queries. For example {{term anchor text}} expands to the glossary entry for "term." Parameter values set in config.htmi control which tables are accessed for these definitions.

The *sqlloop builtin function and the special references need the variables host, user, password, and database to be set to appropriate values. Assignments to these values are kept in the local configuration file config.htmi and most file expansions are performed by executing the little shell script expandfile2, which invokes expandfile config.htmi (rest of args)

The Makefile and shell scripts that execute mysql commands require that the file $HOME/.my.cnf is also set up with values for host, user, password, and database.

SQL tables

Text fileTablesUsageNotesRowsRef
g1.sqlglossaryOne row per paragraph in the Glossaryprintgloss3.pl writes 27 HTMX files1182{{ }}
home-slider.sqlhomesliderOne row per photo on the main pageused by loadm.sql15
loadbib.sqlbibtypes,biblio, authors, akaDocuments about Multicsexpand 7 templates to produce 7 HTMX files5000
loadc.sqlchangesOne row per changetopchanges.htmi, changes.htmx, changes-old.htmx2138
loadext.sqlextrefhyperlinks to external pages735{! !}
loadm.sqlmList of Multiciansproduces multicians.htmx2048{[ ]}
loadpages.sqlpagesPages of the websitesitemap.htmx, Google sitemap447{@ @}
loadsites.sqlsList of sitesextra rows for sites with complex history102
menubar.sqlmenubarLayout of jQuery dropdown menustwo levels78
msource.sqlmsourceOne row per source filesource-index.htmx5877
multhumbs.sqlmulthumbsOne row per thumbnailproduces multics-images.htmx640
mxawards.sqlmxawardsAwards earned by Multiciansused with loadm.sql238
publist.sqlpublistDocuments produced by Multiciansproduced by addwp.htmt, used with loadm.sql700
groupsio.sqlgroupsioMembers of groups.io/multicians157
hiddenmail.sqlmmailEmail forwarding addresses (see 3.7).754

2.4 Executable files

The editor's computer has a standard OS installation with programming tools and a command shell and standard Unix tools installed. The computer can run the Unix or Mac operating system, or use Windows with a Unix-like environment such as Cygwin. Some tools such as Perl and MySQL are installed in /usr/local/bin. expandfile comes with some helper executables written in Perl. Additional tools are part of the multicians.org source.

The local account needs a bin directory in the home directory, and a search path set up to find commands in these directories.

expandfile Install instructions describes the setup of Perl, CPAN, MySQL and expandfile.

2.5 Environment

The editor's computer has the environment variable PERL5LIB set to point to the correct version of Perl. .ssh keys are set up so that files can be sent from the editor's computer to the web host, and should be automatically loaded using ssh-add when the editor logs in.

2.6 Symbolic links

The source directory needs a few links to items in the object directory. In particular, in order for the image reference macros such as getimgdiv to work correctly, they need to access the pixel dimensions of images. Links to the object subdirectories mulimg/ and thumbnails150/ are placed in the source directory.

3. How the Build Process Works

This section describes the inner workings of the build machinery, so that it can be changed when new features are required.

3.1 Make

The Unix make utility generates the object directory from the source directory. It is driven by a file in the source directory called Makefile.

The Multics website's Makefile has recipes that expand any .htmx file that is older than the corresponding .html. In addition, each SQL table is loaded from a .sql text file, and the Makefile ensures that if the .sql file changes, it is loaded into SQL and then relevant .htmx files are generated. make takes care of ordering the generation steps.

Makefile Configuration

The Makefile refers to some external-world facts by symbolic configuration names. The actual values of these symbols is set in the file configmake.mk which is included by the Makefile. (This makes it possible to test new versions of the site or Makefile, and may someday support multiple editors.)

# Configuration constants for Makefile for Multics website
# place this in the multicians source directory
# this is where site-dependent values like path names are kept
PRIMARYWEBACCT = account@server
PRIMARYWEBPATH = directory path on server
PRIMARYWEBPATHSRC = directory path on server

Makefile Rule for Expanding Files

If you want to know how it works, you have to learn to read the Makefile. Makefiles define values for some variables, and then provide a set of rules. The Makefile for multicians.org rarely changes. Here is an important rule that invokes expandfile:

# Rule that makes the HTML files from HTMX files and includes. $(OBJ_DIR)/%.html: %.htmx $(CONFIG) $(INCLUDES) @chmod +w $@ $(EXPAND) $< > $@ @checknonempty $@ @-/usr/local/bin/tidy --markup no --quiet yes --show-warnings no $@ @chmod 444 $@

This rule means "Use expandfile to build the object file if the corresponding source, or any of the include files, or the config file, have changed."

Makefile rules have three parts: the "target" file(s) being made, a colon, a list of the "dependencies" -- files that the target depends on, and then an indented list of commands to execute to create the target from the dependencies.

In the rule above, this says "do the commands if ANY of the dependency files is newer than the target."

make examines all the rules in Makefile and executes the rules in the "right" order, so that dependencies are built before the files that depend on them.

The percent sign in the target is a wildcard that matches any name. (Used in a dependency, on the other side of the colon, it has the same value.) This example will cause fred.html to be remade if fred.htmx is newer.

In the Makefile rule for making HTML files:

Makefile Variables

VariableContentsMeaning
CONFIGconfig.htmiConstants for multicians.org used by expandfile
OBJ_DIR../multicsrelative pathname to the object dir from the source dir. config.htmi has a var named obj_dir that must match this value.
INCLUDES menubar.touch class2head.htmi linktags.htmi textnav.htmi addrtag.htmi leftpanelstyle.htmi mspmstyle.htmi mxstdfmt.htmi pagewrapper.htmi headpagewrapper.htmi glospagewrapper.htmi noncenteredpagewrapper.htmi paperwrapper.htmi codewrapper.htmi htmxlib.htmi mxlib.htmia list of files that all expansions depend on
DATABASEthvv_userlistlocal MySQL database on the editor's computer
HOMEREL../prefix used by subdirectories to find the main directory
BINDIR$(HOME)/binlibrary directory on the editor's computer
EXPAND$(BINDIR)/expandfile2Macro name that invokes expandfile with first argument config.htmi
SUBTGTSr2012d r2004d rvand mtbsd mabsd debugd blwd pgdlist of sub-make-targets that should be built as part of 'all'
LOGACTION$(BINDIR)/wtlog.shcommand executed to log updates

Some variables must be different on different build environments. These variables are set in the included configuration file configmake.mk. For example, OBJ_DIR might be set to ../multics_object, and EXPAND is set to $HOME/bin/expandfile.

Shell VariableMeaning
PRIMARYWEBACCTusername@DNS name of ISP web hosting computer
PRIMARYWEBPATHpathname of web directory on web host
PRIMARYWEBPATHSRCpathname of source directory on web host

(It may occur to you that the Makefile rule rebuilds some files that do not need rebuilding. For example, if any wrapper include file in the INCLUDES list changes, every file will be re-expanded, not just the ones that include the changed file. We could fix this, at the cost of increased complexity in the Makefile. Since it takes less than 2 minutes to re-expand all files, the savings are not worth the development and future maintenance effort.)

Besides rules for creating HTML files, there are also rules that specify additional dependencies for some files, in order to cause some files to be built after other files.

Other Executable Tools invoked by Makefile

When .html files are generated, the Makefile executes the HTML Tidy command to check if the files have HTML syntax errors. (We obtain the HTML5 version of Tidy from SourceForge.)

The "install" recipe in the Makefile uses the rsync command to copy modified files from the object directory to the web server. It causes a "make all" before executing, so to make a one-line change, the editor just edits the file and types "make install". (Check the version of rsync on the local computer. MacOS ships version 2.6.9. I needed to install version 3.1.3 from MacPorts, same as on Pair.)

3.2 Generation of HTMX from SQL

Certain HTMX files are generated from the SQL database. (These files are marked read-only so that an editor doesn't forget and edit them directly. Instead, the editor changes rows in a .sql text file.) The Makefile notices that the .sql file has changed by noticing that the date modified for the .sql file is newer than the corresponding .touch file, loads the .sql file into the database, expands a template file for the modified table which generates one or more .htmx files, and executes touch filename.touch so the reload will be done only once. Updating the date modified on filename.touch also causes make rules that depend on this file to be executed. An additional dependency rule for compiling the .htmx file into .html is also present in the Makefile. The .touch file is listed as a dependency in a Makefile rule to cause HTML files to be rebuilt if they may depend on the table contents. There are about 10 such rules in Makefile. For example, the Makefile rule

  # .. rebuild about-multics for count of pages, graphics, or changes
  $(OBJ_DIR)/about-multics.html:  changes.touch pages.touch multhumbs.touch

declares that about-multics.htmx should be re-expanded into about-multics.html if any of the "changes", "pages", or "multhumbs" tables change, in order to ensure that the counts of changes, pages, or images are re-computed. This rule adds reasons to regenerate its target to those declared in the standard rule for .html files, and ensures that about-multics.html is regenerated after its prerequisites are generated. (If we discover that we have to make twice to cause an output file to be correct, this is a sign that a dependency rule is missing.)

HTMX files generated from SQL

FilenameTableSQL fileTemplate
sites.htmxsitesloadsites.sqlglostpt.htmt
site-timeline.htmxsitesloadsites.sqlglostpt.htmt
mgloss.htmxmglossg1.sqlglostpt.htmt
mga.htmxmglossg1.sqlglostpt.htmt
mgb.htmxmglossg1.sqlglostpt.htmt
mgc.htmxmglossg1.sqlglostpt.htmt
mgd.htmxmglossg1.sqlglostpt.htmt
mge.htmxmglossg1.sqlglostpt.htmt
mgf.htmxmglossg1.sqlglostpt.htmt
mgg.htmxmglossg1.sqlglostpt.htmt
mgh.htmxmglossg1.sqlglostpt.htmt
mgi.htmxmglossg1.sqlglostpt.htmt
mgj.htmxmglossg1.sqlglostpt.htmt
mgk.htmxmglossg1.sqlglostpt.htmt
mgl.htmxmglossg1.sqlglostpt.htmt
mgm.htmxmglossg1.sqlglostpt.htmt
mgn.htmxmglossg1.sqlglostpt.htmt
mgo.htmxmglossg1.sqlglostpt.htmt
mgp.htmxmglossg1.sqlglostpt.htmt
mgq.htmxmglossg1.sqlglostpt.htmt
mgr.htmxmglossg1.sqlglostpt.htmt
mgs.htmxmglossg1.sqlglostpt.htmt
mgt.htmxmglossg1.sqlglostpt.htmt
mgu.htmxmglossg1.sqlglostpt.htmt
mgv.htmxmglossg1.sqlglostpt.htmt
mgw.htmxmglossg1.sqlglostpt.htmt
mgx.htmxmglossg1.sqlglostpt.htmt
mgy.htmxmglossg1.sqlglostpt.htmt
mgz.htmxmglossg1.sqlglostpt.htmt
biblio.htmxbiblio,bibtypes,authors,akaloadbib.sqlglostpt.htmt
mspmtoc.htmxbiblioloadbib.sqlmspmtoc.htmt
multicians.htmxm,mxawards,awards,publistloadm.sql,mxawards.sql,publist.sqlmulticians.htmt
changes.htmxchangesloadc.sqlchanges.htmt
sitemap.htmxpagesloadpages.sqlsitemap.htmt

HTMX files that fetch values from SQL

FilenameTableSQL fileValues
multics.htmxhomesliderhome-slider.sqlsliding picture file
about-multics.htmlchangesloadc.sqlcount of changes
articles.htmlpagesloadpages.sqlcount of articles
changes-old.htmlchangesloadc.sqlcounts of changes by year
mac50-videos.htmlmulthumbsmulthumbs.sqllist of videos
mult-links.htmlgroupsiogroupsio.sqlcount of mailing list members
multics-images.htmlmulthumbsmulthumbs.sqlthumbnail image paths
multics-stories.htmlpagesloadpages.sqltables of stories
people-pictures.htmlmulthumbsmulthumbs.sqlimage paths
site-timeline.htmlsloadsites.sqlsite names and dates
sites.htmlsloadsites.sqlsite information
source-index.htmlmsourcemsource.sqlsource file names and info

The following Multics specific expressions appear in many HTMX files and are replaced by the results of an SQL query.

ExpressionTableSQL file
{! ... !}extrefloadext.sql
{@ ... @}pagesloadext.sql
{[ ... }}multiciansloadm.sql
{{ ... }}mglossg1.sql
bibliolink macro callsbiblioloadbib.sql
mtblink macro callsbiblioloadbib.sql

3.3 Makefile Rule for Sending files to the Web Server

The Makefile uses the rsync command to publish files from the personal computer to the web server. rsync doesn't copy files that it doesn't need to. Files are sent securely so they can't be intercepted or tampered with, and the file transmission is compressed so it will go quickly.

rsync is configured to get the local and remote path names from the Makefile. It uses SSH keys to be able to send files to the webserver.

The rsync command is issued by make instead of typed directly.

Specific Makefile rules can be executed by specifying their name as an argument to make. The command make install invokes rsync:

# make install pushes modified files to the primary website and also backs up the source install: all incrementversion vnfile expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv $(LOGACTION) updated multicians rsync -avzu --blocking-io -e ssh --exclude .DS_Store $(OBJ_DIR)/ $(WEBACCT):$(WEBPATH) rsync -avzu --blocking-io -e ssh --exclude .DS_Store --exclude $(CONFIG) . $(WEBACCT):$(WEBPATHSRC) date

This rule means "Use rsync to send any modified object files to the web server, and also update a copy of the source files on the web server." It also updates a file dashboard.csv which visitors can look at to see counts of files on the site and the time of last update.

See Using Unix tools with expandfile for an explanation of rsync args.

3.4 Other Makefile Rules

The Makefile has other interesting rules.

Version Number File

The rule that pushes files to the ISP calls the shell script incrementversion to update vnfile. The contents of the file vnfile look like this:

  5388 2020-07-02 13:35

The sequential number is incremented every time a make install is done, so this is the number of installs done since some time in 2006, when I first set up this file. It is followed by the date and time of the push. The shell script incrementversion updates these values. vnfile is included into an HTML comment in multics.html when that file is built. (The sequential number is not the same as the number of file changes, since a single install may install many files, or I may do repeated installs to get a single change just right.)

Dashboard File

The rule that pushes files to the ISP executes expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv to update dashboard.csv. The file dashboard.csv is created by expanding the template dashboard.csvt. It contains two lines, a list of column headers and a list of corresponding column values:

  changect,asof,multicians,multiciansmail,multiciansurl,multiciansdec,glossent,npages,\
   npdf,alldoc,alldoc-online,images,mspm,mspm-online,nsites
  5388,"2020-07-02 13:35",2039,749,169,90,839,466,601,4994,2669,639,838,54,84

This file repeats the version number, and then has counts of the number of various items on multicians.org, such as number of multicians. It is intended to assist in the construction of other sites that refer to our site's contents. A file with this format can be read using the Expandfile builtin *bindcsv.

Subdirectories

Each subdirectory has a sub-Makefile which is invoked via a "subdirectory target" in the Makefile. The $SUBTGTS variable names these targets, and this variable is listed as a dependency for the "all" target. The result of this setup is that just typing make will also invoke the sub-Makefiles and make whatever is out of date. Most subdirectories have symbolic links to standard .htmi files in the parent directory, as well as the image and thumbnail directories. Subdirectories have a configx.htmi file that may override some path definitions in config.htmi so that subdirectory files refer correctly to other files on the site. If you are setting up a subdirectory, it is best to clone the Makefile from one that already works, and make the same symbolic links.

Special build instructions

Several rules in the Makefile create special files which are published to the host ISP, using Unix utility commands in addition to expandfile. For example, make_mx_tar is a shell script listing all files that should be shipped to mirror sites. Generating it automatically saves me from having to remember to edit another file whenever I add a page to the site.

FileFunction
sitemap.xml.gzGzipped XML sitemap used by Google crawler
multicians.procmailMulticians email forwarding file (not visible on web)
make_mx_tarshell script that creates a tarfile of the site, for mirrors (not visible on web)

Error checking performed by the build process

File building rules check for errors and alert the editor of any problems whenever make rebuilds a file. For ordinary HTML file creation, the auxiliary program checknonempty is invoked to check for the case where a target file is nonexistent or empty after expandfile is invoked. If this happens, make exits with an error.

Newly created HTML files are also checked by the HTML tidy utility, which prints a line if there are any warnings or errors. When this happens, the editor should investigate the failing file and correct the problem.

For a few cases where SQL files are reloaded, the make recipe runs custom shell scripts to check that foreign key relationships are satisfied. In particular, the script checkbib.sh ensures that the tables loaded by loadbib.sql have authors entries for each document, that aka rows are present for each author, and that m rows for each author exist. If any errors occur, the editor should investigate loadbib.sql and correct the problem.

The rule that pushes files to the ISP executes check_missing_extref.sh, which checks output files for the sequence "###", which is produced when a {@path anchor@} sequence is executed and does not find path. This warns that we have generated a dead intra-website link. (Some of these are OK, and represent unfinished improvements.)

3.5 Source file suffixes

SuffixMeaning
htmxHTML eXtended, generates HTML
htmtHTMX Template, generates HTMX
htmiHTMX Include file
tpttemplate, generates non-HTML files such as site map
csvComma Separated Values
csvttemplate that generates CSV file
etext file
htmqtest htmx file
mkMakefile include
plPerl language
pngportable network graphic
procmailprogram for procmail
pyPython
shshell script
sqlMySQL source
txttext file

3.6 Standard Template Expansion with Wrappers

Most pages' HTMX files conform to a standard pattern. Each file contains

Expansion of normal HTMX files proceeds as follows. expandfile

  1. Ignores comments
  2. Sets variables and blocks. (Does not expand variable references when blocks are defined.)
  3. Reads in and expands the wrapper, which outputs standard HTML structure and expands variable references, including %[body]%. If body contains variable references, these are expanded also. The standard page wrappers begin by querying the pages table.
    1. Executes SELECT wdate FROM pages WHERE sourcefile='%[mainfilename]%'
    2. Executes SELECT crumbs FROM pages WHERE sourcefile='%[mainfilename]%'
    3. Executes SELECT l2navfile FROM pages WHERE sourcefile='%[mainfilename]%'
    4. Outputs the HEAD section
      1. Expands title, description, and keywords obtained from the *set statements, and minwidth, maxwidth, and contenwidth obtained from config.htmi
      2. Executes *include,=mxstdfmt.htmi, which includes standard CSS boilerplate for the site
      3. Expands extrastyle if it was defined in the source file. This block may include JavaScript files and define STYLE settings local to the file.
    5. Outputs the BODY section:
      1. Includes class2head.htmi, which defines the page's standard top bar (which will use the date and crumbs from above) and drop-down menus.
      2. Expands the body block, set by the HTMX file, and variable references contained in it.
      3. Expands the extratail if the file supplied it.
      4. Formats and outputs tail navigation, inclding l2navfile if it was obtained from pages.
    6. Closes the BODY and HTML.

Alternate wrappers support variations for special page formats, such as MSPM sections or Glossary pages.

Why do this?

Expanding items that expand other items may seem indirect and convoluted. However, this scheme spares the writer of simple pages from the details of the site implementation, reduces the chances of a simple change breaking a page, and enables global changes when site design or HTML standards change. In practice, a one-line change can be made and published in about a minute; adding a new page to the site, cross-linked to other pages, visible to Google, and findable by standard navigation, can be done in only a few more minutes.

3.7 Additional Functions of Website Host

The website host provides additional functions for the support of multicians.org that are not further described here. These include:

Site mirrors do not provide these functions: they refer to the primary site.

When I set up multicians mail forwarding, I promised people I would not let their addresses be visible to spammers scraping web pages. I keep the protected mail addresses in a file called hiddenmail.sql which is not available on the web. The forwarding data is not visible to random web crawlers (unless the ISP gets cracked).

3.8 Tools

Tools needed to maintain Multics site provided by others

Tools needed to maintain Multics site provided by editor

See expandfile web pages.

Programs and data files in cgi-bin directory provided by editor

Programs and data provided by web host and site admin

The /mxs/tools directory

-- /mxs/tools
--- $HOME/bin .. should be hard-linked to tools dir
----- wtlog.sh, calls $HOME/wtx/wtlog (Perl)
----- expandfile2
----- check_missing_extref.sh
----- printsites2.pl
----- gen_site_timeline.pl
----- auto1a.sh, calls post.pl

3.9 Performance

Expandfile is written in an interpretive language, so it is not blazingly fast, but it is fast enough. Re-expanding all 467 pages on the site takes less than 2 minutes. Pages that have a lot of *shell calls are slower: for example, a page like people-pictures.htmx that has to involve a lot of *shell calls to Perl helper program gifsize2 has to launch a shell and a Perl interpreter for each of over 100 pictures. If I built gifsize2 into expandfile this could be a lot faster. Not worth the effort at this time.

3.10 Git

The source directory contains a Git repository that contains all the source .htmx and {.sql:} files. Doing a make install does a git status. After making and installing a set of changes, the editor should do a git commit -a with a brief message describing the changes.

--- what files are in git, where the repo is
--- email address file not included
--- ignored files
--- future: upstream repo

The Makefile rule that pushes files to the ISP ends with a git status which reminds the editor to add or commit changes to the git repository.

3.11 Files and their purpose

(by THVV unless indicated)

Tools invoked by Makefile

auto1a.sh (post FAQ to USENET)
checknonempty
config.htmi
config.txt
expandfile
expandfile2
gen_site_timeline.pl
incrementversion
nextpostingexpires
post.pl
printgloss3.pl
printsites2.pl

Tools invoked by *shell in an HTMX file

dbcounts
get-facebook.sh (calls get-facebook-fan-count.pl)
filemodshort (comes with expandfile)
filemodyear (comes with expandfile)
filesizek (comes with expandfile)
firstletter (comes with expandfile)
gifsize2 (comes with expandfile)
nargs (comes with expandfile)

Files used to support ISP mail and signup on multicians.org host /cgi directory

errtpt.txt
mform-thanks.txt
mformtpt.txt
mxaddrtag.htmi
mxclass2head.htmi
mxh1style.htmi
mxmailerr.txt
mxolwrapper.htmi
mxregerr.txt
mxregister.cgi
mxregister4.cgi
mxstdfmt.htmi
mxtextnav.htmi
reloadm
special.pmrc
thvvutil.pm

4. Expandfile Documentation