Multics > About
24 May 2020

Maintenance tasks

History | People | Library | Sites | About Search

Tom Van Vleck

This file describes maintenance tasks for multicians.org. General design principles are listed in Design Principles for the Multicians Website.

Roles

These are the current roles.

Hosting for multicians.org

Hosting for this site is provided by the hosting site admin, who pays the domain name registration fees and web hosting fees as a contribution to the online community. The web site has been hosted at Pair Networks in Pittsburgh PA since 2001. Pair is referred to as the Information Service Provider (ISP).

We are limited in both storage space and bandwidth as a result of Pair's account parameters. The editor will take whatever steps are necessary to avoid exceeding the (fairly high) limits imposed by the ISP, since this could cost the editor a lot more money than he can spare.

The ISP has a fairly reasonable policy on "slashdotting" but more than one in a month could be a problem.

We are also bound by Pair's Acceptable Use Policies. So far this has not been a problem.

The advantages of hosting at Pair are very high bandwidth and redundant connectivity, very good server response and uptime, reasonable prices, outstanding policies and facilities, and 24 hour technical support. The ISP has been in business since 1996 and says it does not plan to change management, merge, or be sold.

The ISP supports virtual domain hosting, CGI and CGIWRAP usage, custom mail forwarding with procmail, server log extracts, SQL database access, cron, SSH and SCP for updating, shadowed backup, free HTTPS support including Let's Encrypt, and other features that not all ISPs provide.

Site Maintenance by Site Editor

Files on the site include graphics and PDFs, HTML provided by others, and HTML compiled from HTMX.

The expandfile tool translates HTMX to HTML. (Overview of HTMX. Details of HTMX Language.)

Translation is invoked using the Unix make command for any file whose source is newer than its object.

After translation, the Makefile invokes HTML Tidy to warn if there are errors. Cure these and re-install.

Images

Images are stored in directory /mulimg in the object directory. (This directory is linked to from the source directory so that image macros can get the size of the image.) They can be .jpg, .gif, or .png files.

Links in text

Use the bitsaversmultics HTMX macro to generate references to scanned Multics manuals at bitsavers.org.

Use the mitsourcefile and mitsourcearc macros to generate references to Multics source at MIT.

Generate external links using the HTMX construct {!tag text!} which looks up "tag" in external links table and inserts globe icon and TITLE tag.

Generate multicians links using {[tag text]} which looks up "tag" in multicians table and inserts TITLE tag.

Generate glossary links using {{tag text}} which looks up "tag" in glossary table and inserts TITLE tag.

Generate internal links using {@tag text@} which looks up "tag" in pages table and inserts TITLE tag.

Use {:text:} for "cmd", {=text=} for "pathname", {+text+} for "code".

Update main page

Edit multics.htmx in the source directory.

Execute make install

Main page is automatically updated if many other pages change. See Makefile.

(explain how to add a photo to the sliding gallery)

Add or update a Multician

A site visitor may use the Multician add/update form at mform.html

If the visitor has JavaScript enabled, changing the person name in mform.htmx sends an XHR (Ajax) request to mxregister4.cgi at www.multicians.org to look for existing database entries in the m table and fill in the rest of the form. If multiple database entries match the partial person name, the form code presents a pick list.

When the visitor submits the completed form, this action invokes mxregister.cgi at www.multicians.org, which sends mail to the editor with SQL rows for database input. (It doesn't update the database.) The mail looks like this:

  loadm.sql
  ('fooch,melvint', 'Fooch, Melvin T', 'User (MIT): stuff', '', '', 'mail 2020-04-01 11:54', '', 'Y', '', null),

  hiddenmail.sql
  ('fooch,melvint', 'fooch@supertool.com', '', '2020-04-01 11:54'),

Alternatively, a Multician may send a mail message to the editor instead of using the form, or the editor may discover a Multician in some other publication, web page, or forum.

The editor verifies the form info, including whether the submitter is actually a Multics contributor. The editor may know the sender personally or know someone who does. They editor may send mail to the submitter and ask for details. The editor has a standard form letter that says, "Sorry, liking Multics doesn't make you a Multician."

The editor checks the URL if supplied, and may correspond with the submitter if it has a problem.

The editor adjusts the submitted "did" field to match terminology in other entries.. e.g. OU versus Oakland University. Formatting tags such as <cite> and <em> in "did" fields canot be used: the user's browser will remove them when parsing the Ajax XML message. It is OK to use HTMX constructs such as {@page title@} to link to other site pages.

The editor checks the mail info add adds or updates the corresponding row in hiddenmail.sql in the source directory. Note that if a new multicians.org email address is added, the site editor must also log into the web account hosting control panel and add a mail forwarding recipe. See "mail reflector maintenance" below.

If a Multician has won an award or medal, the editor should add or update a row in the mxawards table in mxawards.sql in the source directory. For an award type not seen before, the editor must also add a row describing the award in the awards table.

When the editor executes make after updating loadm.sql or hiddenmail.sql, the Makefile will rsync updated copies of the m, mmail, and mgemail tables to the web hosting computer and then execute the remote command reloadm on the hosting computer via SSH. The reloadm command loads the SQL database on the hosting site with new copies of the m and mmail tables. Its output looks like this:

  total 892
  -rw-r--r--  1 thvv  users    7897 Apr 25 16:28 grpmail.sql
  -rw-r--r--  1 thvv  users   52710 Mar 31 19:44 hiddenmail.sql
  -rw-r--r--  1 thvv  users  258790 Apr 25 16:39 loadm.sql
  reloaded m on timayo
  reloaded mmail on timayo

The editor sends a welcome/thanks email to each new or updated Multician.

Add or update an offsite link

Offsite links are specified in loadext.sql, loadbib.sql, or loadm.sql.

For a link that is used in article text, edit loadext.sql to add or modify a row in the external links table and reload it. This row specifies an internal tag and the lin target URL.

Add a reference to the place where the link goes, using the {!tag anchor text!} syntax. The globe icon, etc will be generated by expandfile.

Multicians home page URLs are specified in a row in the m table defined in loadm.sql.

External URLs of items in the bibliography are specified in a row in the biblio table defined in loadbib.sql.

After editing the SQL files, execute make install to recompile source files that reference them. In some cases it may be necessary to touch a source file to cause it to be re-expanded.

Check every few months for dead links. I use the Mac tool Integrity.

Add or update site information

Verify info and add row or update in loadsites.sql in the source directory.

Update the site history if there is one. Write a new site history if you get enough information to make it interesting.

If new dates are added, fix site timeline datafile site-timeline.txt by hand.

Execute make install

Add or update bibliography

Verify info and add or update row in table "biblio" loadbib.sql in the source directory. This row will have a document identifier like "Smith100" with an author name and unique sequence number.

For each author of the document, add a row to table "authors" giving the document identifier, the author name as it appears in the document, and a sequence number.

For each author name, ensure that there is a row in table "aka" that links the document-style name with the canonical "name" field in loadm.sql.

Executing make will load the SQL tables and regenerate many HTMX and HTML files. The Makefile will execute the shell script checkbib.sh to check that there are no missing rows in the bibliography tables and that they match the Multicians table. Fix any problems reported.

make install

Add or update glossary entries

Edit or add paragraphs to g1.sql in the source directory. Use special tags to generate links to glossary entries and Multicians list entries, and to mark commands, pathnames, and code. One oddity: to insert a backslash in the output, you have to put EIGHT backslashes in the database file, because it gets expanded three times.

make install

Add a new story

If necessary, propose changes to the submitter so that the context of the story is clear to a broad audience, and so that story fits into the rest of the site.

If a Multician sends information in plain text, convert it to simple HTMX. Use <p> and </p> around paragraphs and similar markup ... HTML Tidy should find no errors or warnings in the converted file.

If a Multician provides a story in HTML, turn it into a HTMX file by removing the HEAD and BODY tags and wrapping the user's text in a {:*block,&body,^END:} ... END construct, followed by an *include of the standard wrapper (look at an existing story for the pattern). Use the standard site CSS definitions; if the story uses specialized CSS constructs, create a {:*block,&extrastyle,^END:} (css) ... END block. Use <p> and </p> around paragraphs and similar simple markup ... HTML Tidy should find no errors or warnings in the converted file.

If possible, include pictures. See "Images" above. Getting useful pictures may require a mail dialogue with the submitter.

Create a new file storyname.htmx in the source directory.

Do the other "new page" operations below. Adding the page to loadpages.sql with a nonblank "site" entry and a kind of "people" triggers its inclusion in the stories index. Set the nextstory field of an existing page to point to this one, and the nextstory field of the new entry to the old value, so "Next Story" link works. (Touch the source of the existing page so its navigation links will be regenerated also.)

make and then use W3 validator to check the new file. Fix any errors or warnings.

make install

Add or update other class 2 and class 3 pages

Edit or add filename.htmx in the source directory. Don't edit the HTML files directly.

Use the wrapper macros: set title, description, keywords, and headingdate, then define a block called body that contains the HTML of the content. At the bottom, include pagewrapper.htmi to cause the content to be formatted with standard headings and footers. If necessary, use optional blocks extrastyle for extra lines in the HEAD section, and extratail to add content to the footer.

Use the getimgtag, getimgdiv or getimgpopdiv macro for references to images. For new images, make thumbnail and update multhumbs.sql.

When a new HTML page is added,

make and then use W3 validator to check the new file.

make install

Add or update class 4 pages (formatted by others)

Either convert the file to a class 3 page, or

Place the file directly in the object directory. Edit it to insert navigation to the rest of the site at the bottom, so that if a visitor finds the file from a search engine, they can find the rest of the site.

Add the file's name to loadpages.sql and makemxtarextrafiles in the source directory.

Optionally add mxstyle.css for uniform heading styles if this doesn't mess up the appearance of the document; discuss this with the file author.

Change mail addresses to links via the Multicians database, so that the submitter's mail address cannot be scraped by spammers.

make install

Add or update a PDF

Ensure that the PDF has OCR text so that indexers can see its content.

Place the file directly in the object directory.

Add its name to loadpages.sql. For the "source" use something like "../multics/foo.pdf" -- that is, the file is both source and object. This entry ensures that it is in sitemap.html, and that its date modified is that of the file.

Add its name to makemxtarextrafiles in the source directory. This ensures that the PDF file will be included in the TAR of the whole site.

make install

For All Changes

Run W3 Validator on each new class 1-3 file. It checks for more errors than HTML Tidy. (Both Tidy and W3 Validator fuss about some things that we include anyway: for example, the Facebook link code and the Google search code use non-standard HTML, and we include a few deprecated constructs needed for correct display in old browsers like MSIE.)

Update loadc.sql with a one-line description of each change, and execute make to generate changes.html describing what's new. Changing this file will regenerate the HTML site map, RSS feed, and Google sitemap.xml.gz files. (Some minor edits, typo corrections, etc. do not need a line in 'changes.' For intermal machinery changes, enter a line in loadc.sql indicating that we did make a change but that changes.html need not be updated.)

The htmx, sql and other source files are stored in a local git repository (graphics, generated files, and binary files are not included). The make install command will do a git status to remind you to do a git commit -a with a summary of changes. Currently this repository is local to the site editor's machine.

To make a tarfile of the whole site, first touch makemxtarextrafiles and then do a make install. This updates the shell script make_mx_tar.

To manage the Google Custom Search box, log into https://cse.google.com/cse/all. Add and delete sites that are searched.

Monitoring Implementation Features

Editors and admins must keep aware of and respond to the following issues:

Smoke Test by Site Editor

Before updating the hosting site, check the site's appearance locally. Check major updates in Firefox/Mac, Safari/Mac, Chrome/Mac. (I probably ought to check Chrome and Edge on Windows also, but I do not have a Windows 10 machine.) Also check when new browser versions of Chrome, Firefox, and Edge are released. Run the W3C HTML5 Validator on a sample of pages, in case the validator changes its criteria or the HTML language changes (which happens sometimes).

Check for:

If all looks OK, execute make install.

Occasionally check that the actual online site loads correctly and looks right, to make sure that nothing is down or misconfigured at the ISP. Especially check this after making any kind of change to the web server configuration or file access controls.

Details to Check on Hosting Site When Deploying

Hosting site admins should ensure the following.

Hosting site admins can execute shell script make_mx_tar to generate a tarfile for transmission to mirror sites when needed.

Site Maintenance by Hosting Site Admin

The main hosting site is currently at the ISP pair.net. The account has a control panel that manages incoming email, domain names, billing, and other details. The domain name multicians.org is registered at pairdomains.com and this site has its own password and control panel.

multicians.org mail reflector maintenance

The procmail script multicians.procmail in the home directory of the account hosting multicians.org redirects selected mail messages.

See multicians-org.html for details.

We are trying to phase this facility out since the ISP may not support it forever.

The redirector script is automatically generated by expandfile with procmail.tpt from the multicians table by the Makefile when loadm.sql is updated.

When a new multicians.org address is added, it must also be added to the list of valid entries in the ISP's mail recipe control panel.

Mail Forwarding Repairs

Very occasionally, Pair screws up the mail forwarding. This happened in 2004, 2007, 2014, and 2019 (usually while I am traveling). Usually this happens when some other problem, such as disk errors or a spam attack, causes web server load to spike. Inexperienced sysadmins then panic, notice multiple procmail processes running (checking incoming mail for spam), decide that procmail is the problem, and either break .procmailrc or set access on /usr/local/bin/procmail to null. If they choose the first option, then all incoming mail including spam gets dumped into a single file with no filtering. If they choose the second option, then all incoming mail is bounced with a permanent fatal error from MAILER-DAEMON. The site editor must then call Pair support on the phone and teach them their jobs, which takes about an hour on hold. The right solution is usually to fix any I/O problems, and then wait 10 or 15 minutes for procmail to clean out the mail queue. Recovery from such a mess depends on how they panicked: in the first case, this means taking the huge mailbox and piping it through procmail; in the second case, the mails sent are lost forever, but Pair support can grep the qmail log to find out the senders of messages that were discarded.

alt.os.multics FAQ Maintenance

At some point we will stop doing this, when USENET is no longer interesting.

In order to post to USENET you need an account at some USENET service. Currently we have a free account at etermal-september.org.

Post the FAQ to alt.os.multics on (or after) the first of each month. Short FAQ 12 times a year, one file.

To post the short FAQ, run make auto2. This operation invokes a program that updates the FAQ with totals from the website.

Groups.io Multicians group administration

If applicants meet the group criteria (Multics contributor, including user) accept them. (Otherwise send a form letter.) Use the 'admin' control panel of groups.io to approve new applications. Add the user's subscription email to grpmail.sql (nothing uses this table yet).

LinkedIn Multicians group administration

If applicants meet the group criteria (Multics contributor, including user) accept them.

Domain name management

The domain name multicians.org is registered at pairdomains.com, which has a somewhat cryptic website. Some domain name operations need to be done partly with the domain registrar site and partly with the Pair control panel.

Pair website management supports https access and automatic renewal, for free, in the Pair control panel. So far, it has worked with no problems. It is valuable to enable this feature, since Google search rankings penalize non-https sites.

Search Engine Optimization

Most search engine optimization activity is not relevant to multicians.org. Keyword stuffing and fake links to our site from link farms are a bad idea. Make sure that META DESCRIPTION tags are accurate and incorporate the words that visitors might use in a query.

Google Webmaster Tools

We have a Google Webmaster Tools account. They sometimes send warnings that our pages have problems they detected when indexing our site. Some of these are issues that should be fixed.. others are not useful.

We don't use Google Analytics, because it would inject JavaScript into our pages, and because users may not wish to have Google tracking their web browsing. For the same reason, we removed the code that creates a Facebook "like".

Tools

Tools needed to maintain Multics site provided by others

Tools needed to maintain Multics site provided by editor

See expandfile web pages.

Programs and data files in cgi-bin directory provided by editor

Programs and data provided by web host and site admin