expandfile Tutorial and examples

This page contains an expandfile tutorial consisting of a sequence of examples.

Each example shows input file contents, the command line invocation of expandfile, and the resulting output. You can read the input files in light green, and the command invocation in pink, and see if you understand why the output in blue came out that way.

Refer to the expandfile reference document for an explanation of the language and the builtin functions.

More examples will be supplied in the future. Suggestions are welcome.

Basic expansion

expandfile reads an input file and writes an output file.

Example 1: copy input to output

Input characters not inside %[...]% are just copied to the output.

Example 2: backslash

The backslash character \ makes the next character "unspecial" and is removed. To output a single backslash use \\.

Example 3: variables

expandfile keeps a list of variables and their values. Variables have names and values that are strings of characters. (Upper and lower case letters in variable names are different.)

expandfile replaces a variable name enclosed in %[...]% by its value.

The builtin variable "date" returns the date. (There are currently 16 builtin variables.)

A construct like %[*name...]% invokes a builtin function provided by expandfile. Builtin functions have names that begin with *. The invocation is replaced by the result of the function (some builtin functions return no value).

Variable names are case sensitive. Upper and lower case letters in variable names are different.

Expanding Variables

Example 4: setting a variable

Expanding the builtin function *set,&varname,=constant sets the value of variable varname to constant. Literal constant values in an argument list begin with =. Builtin function arguments are separated by commas; to specify a constant value that contains a comma, use double quotes, like ="dogs, cats".

If you accidentally left out the =, you would be trying to access the content of the variable named "3". expandfile warns missing = before numeric argument '3'. You could use double quotes and write ="3" instead of =3 above, to make it clear that the 3 is a constant.

Example 5: Quoting, Constants and Escaping

(In builtin argument lists, percent-bracket characters inside double-quoted strings are not expanded.)

Within the expansion of %[...]%, backslash (\) removes the special meaning of the following character. The example .htmx file below creates the constants '<a href="' and '">' by using a backslash in front of the double quote.

The third *set builtin above is invoked with six arguments:

1. The variable to be set: link_ref
2. A constant string, <a href="
3. A variable, url, whose value is https://multicians.org/index.html
4. A constant string, ">
5. A variable, anchor, whose value is Multics
6. A constant string, </a>

The *set builtin concatenates the values of the last five arguments and stores the result in link_ref.

Example 6: variables that were never set

Expanding a variable that was never set expands to an empty value.

Notice that the single and double quote characters are not special.. they are just copied. (Inside an argument list, double quoted strings are special: they are replaced by their content. Single quotes in an argument list are not special.)

Example 7: each run of expandfile has its own list of variable values

Each invocation of expandfile has its own list of variables and values.

Example 8: setting variables with command line arguments to expandfile

You can set variables set in the command line invocation of expandfile with arguments of the form varname=value.

Example 9: getting variable values from the command environment

You can set variables in the shell command environment with the Unix export command. If expandfile does not find a value for a variable in its list of values, it will search the command processor's environment variables.

Example 10: setting an expandfile value does not affect the command environment

Setting a variable in expandfile's list of values does not affect the command processor's environment variables.

Example 11: setting variables in one input file and using them in another file

expandfile will expand each input file named in its arguments in order. Values set by one file can be accessed in the expansion of subsequent files in the same invocation of expandfile. (For web site building, I often create a file config.htmi with settings for the whole site, and specify the file as the first argument to every expandfile invocation.)

Example 12: multiple value arguments to *set, *shell, *fwrite, *fappend, and *htmlescape

*set, *shell, *fwrite, *fappend, and *htmlescape concatenate all their value arguments (with no separator).

Example 13: comments

Comments are enclosed inside %[**....]%. Comments expand to nothing.

In the example output, why wasn't there a blank line? The comment expands to nothing, and is followed by a newline character. There is a special rule for lines with only expansions: if a %[...]% construct is the only thing on a line, the ending newline is not output. This rule applies to any expansion, not just comments.

Example 14: quote marks

Single and double quote marks not inside %[...]% are just regular characters.

Example 15: quote marks inside expansions

Double quote marks inside %[...]% cause the quoted characters to be un-special. Single quotes don't.

Example 16: backslashes inside expansions

Inside %[...]%, a backslash makes a quote mark un-special. (Why would you need to do this? One common case is building up HTML strings that need to contain quote marks, like href="fred.html".)

Example 17: *include reads in a file

%[*include,=filename]% reads in the entire contents of filename and expands its contents, replacing variable references with their values, and executing built-in functions.

If you want to *include a file but not expand its contents, use the *includeraw builtin.

Example 18: *block saves text for later expansion

%[*block,&blockname,regexp]% starts saving text into a variable without expanding it, until a line matches regular expression "regexp." No quote or backslash processing or %[...]% expansion is done while saving the text. The contents of the variable can be expanded later, and variable references in the text will be evaluated at expansion time. When a block is expanded, it can produce output and set variables' values.

As we shall see, blocks are used as iterators with the *..loop builtins, and as macros invoked by *callv. The other major use of blocks is with expandfile wrappers .

Builtin Functions

Above we have seen the *set builtin function. It is one of 37 builtin functions. There is a summary table in the expandfile documentation. Here are examples and discussion of some of the builtins.

Set

Example 19: Using variable names in *set

%[*set,&varname,value]% sets the value of variable "varname" to a value, either a literal or a variable's value. If you forget the & you get a warning, but the program still executes.

Expand and Expandv

Example 20: *expand and *expandv

%[*expand,varname]% takes the text in varname, expands it, and outputs the result. Variables and builtin functions in varname are expanded. If builtin functions in varname set variables' values, the changes will persist after expansion. *expandv is like *expand, but stores its result in a variable instead of writing it to the output.

Conditional Execution

Example 21: *if

%[*if,relop,value1,value2,restofline]% expands restofline if value1 relop value2.
For instance, when relational operator relop is gt then it checks if value1 > value2.
restofline can be any builtin function invocation, including another *if.
The 14 relational operators are listed in the expandfile documentation for *if.

External Commands

Example 22: *shell

%[*shell,&result,commandline...]% sends a command line to the OS shell and captures its output in result. All the arguments to *shell after the &result parameter are concatenated with no separator to make the command line. The command line is executed in the shell environment: if it outputs characters to STDOUT, they are stored in result. Newline characters in result are changed to spaces (actually to the current value of _xf_ssvsep, see below).

A number of useful helper commands are provided with expandfile, such as fmtnum and nargs. See the list of expandfile utility functions.. In addition, standard Unix commands such as sed and ls can be invoked with parameters supplied by expandfile.

SSVs (Space Separated Values)

The command interpreter on Tandem systems used lists of Space Separated Values extensively. They were called SSVs. expandfile generalizes SSVs to allow HTMX programs to use some other separator character, but I still call them SSVs.

expandfile can handle values that are lists of items; For example, "a b c d" is a four-element SSV. If I change "_xf_ssvsep" to "|" then "e|f|g|h" is also a four-element SSV.

Example 23: Constructing SSVs and *popssv

Create an SSV by simple assignment. Add elements to the right end using ordinary concatenation. Pop an element off the left end using *popssv, and rewrite the SSV.

Example 24: Looping over an SSV with *ssvloop

An interesting use of SSVs is to expand a template variable for each element in an SSV. *ssvloop uses a copy of the SSV, so the SSV is unchanged by the loop. Each time the template is expanded, the left-most element of the SSV is popped off and _xf_ssvitem is set to it, and *ssvloop expands the iterator template and appends the expansion to the output. The iterator block can in turn expand variables and builtin functions. If the SSV is empty, nothing happens. (This example shows how to change the "space" to another SSV delimiter, and how to set up the iterator variable using a *block. It also shows the good practice of saving and restoring _xf_ssvsep.)

Example 25: Looping over a MySQL query with *sqlloop

If you have data in a MySQL database, you can query it and generate formatted output with *sqlloop. As each row is returned, values from the row will be bound to names like "table.name". For values that have no table name, like "COUNT(*) AS xyz", the name will be ".xyz". *sqlloop also binds _xf_nrows to the count of rows returned, and _xf_colnames to an SSV listing the column names in the query.

The variables hostname, database, username, and password must be set up to point to the database server. I set them in a file called config.htmi that I expand before I expand the template.

The multicians.org website has multiple pages generated by *sqlloop from SQL tables. This is especially valuable when the same information is displayed in multiple formats: for example, the HTML sitemap and the Google XML sitemap are generated from the same table; the changes table produces an update history, an RSS feed, and a recent updates panel for the main page; the bibliography database table generates seven HTML files; and the list of contributors is annotated with counts (from the bibliography table) of the documents they produced.

I also built a complex application, swt, that generates daily Web usage reports in HTML from web server logs. It makes extensive use of *sqlloop. Web server log input data is transformed into MySQL input, and then swt produces over 30 report sections by processing the data with *sqlloop, according to configuration and options also stored in SQL tables. Each report section is created by expanding a template file with expandfile. Each section obtains the queries to perform from the configuration: these queries often use multiple inner, outer, and self joins. Most report sections generate bar graphs in HTML illuminating some view of the data. swt has about 300 different queries in its basic configuration; users may tailor and extend the reports with their own queries and templates.

Example 26: Looping over a CSV file with *csvloop

Spreadsheet programs and some Web interfaces export data in a Comma Separated Values file (CSV). (RFC-4180 defines the format of CSV files.) expandfile can process CSV files. The input file may be gzipped. The first row of the CSV file should be a list of column names. *csvloop reads a CSV file and expands an iterator block for each row. As each row is returned, values from the row will be bound to column names from the first row. *csvloop also binds _xf_nrows to the count of rows returned, and _xf_colnames to an SSV listing the column names in the query.

I have used *csvloop to convert CSV data exported from an online Web store application to a different format CSV file for a cash register application. This required translating fields from one encoding to another, and combining multiple records from the online store format into a single record with several variant fields for the cash register.

Example 27: Looping over an XML file with *xmlloop

expandfile can also process XML files that contain a sequence of similar items. (The XML file may be gzipped.) *xmlloop reads such a file and expands an iterator block for each item matching an XPATH. (The default XPATH is "/*/*".) For each item found by the XPATH, the loop binds the values of sub-items "./*" and binds the values of attributes "./@*". *xmlloop also binds _xf_nrows to the count of rows returned, and _xf_colnames to an SSV listing the item and attribute names found.

*xmlloop actually reads the file twice, once to build up the list of variables to bind, and again to set them. Fields that are not present in a particular item are set to the empty string, so they aren't copied from the previous item.

XML files can have very complex structure: *xmlloop doesn't handle every possible XML schema.

Some Web application interfaces export their data in an XML file containing an array of similar items. *xmlloop can loop over the array, extracting and processing characteristics of the items. I have used *xmlloop to generate reports from APIs for trouble tickets and mobile device management.

(Currently there is no "*JSONloop" builtin. Some Web APIs return information in JSON format. For simple enough JSON, one can map it into XML. I have successfully used a simple Perl program called json2xml on JSON data I fetched from a web API for a training management system: I translated the data to XML, processed it with *xmlloop, and generated HTML reports.)

Example 28: Looping over a file system directory with *dirloop

*dirloop looks at a file directory and lists its contents. For each file, stat() is called and variables are set with the file's attributes, and the iterator is expanded. *dirloop also binds _xf_nrows to the count of files found.

Macros

Example 29: Defining macros, and invoking them with with *callv

You can define HTMX macros using a *block, and invoke them with *callv. Macros can be called with any number of arguments. Macro invocations can either write output, set variables, or both.

Example 30: The title macro in this document

Each example section in this document begins with a macro that increments the example number, and outputs an H3 block with specified text.

See Macros in Expandfile for more examples and discussion about macros.

MORE EXAMPLES TO ADD

• Examples of *subst, interaction with escaping, newlines etc
• examples of writing and reading values using *fwrite and *fread
• Cool things you can do, various idioms
• Example of *format
• Use of *onchange
• Examples of *decrement, *product, *quotient, *quotientrounded, *scale
• Example of *ncopies
• Examples of *bindcsv, *urlfetch
• Examples of *dump, *exit, *warn
• Generating graphics: getimgdiv, pie charts (point to pie.js), bar charts, timelines, etc
• Two stage expansions, HTMT => HTMX => HTML