XPath 3: Conversations and Bridges

XPath, XSLT, and XQuery have always been declarative languages: you say what you want and the computer has to figure out how to get there. A strength of XML with these languages has been that they help people to do advanced text processing, analysis, and application building, without necessarily thinking of themselves as programmers. But just under the surface is a whole tropical paradise: the XML stack bridges the gaps between experienced professional programmers, digital humanists, script writers, and many other people, and enables a whole ecosystem in which people of all backgrounds are equally welcome. This is the hidden centre of XML: that it forms a bridge, not only between programs and documents, not only between paper and digital, but between people and roles.

As the languages have evolved and grown over the years, the change that’s probably the most visible is the move towards supporting people who think in terms of programming. Syntax can be more compact, and new ways of expressing ideas become possible. But this can also help more people to be willing to try to use XSLT to work with documents: the bridge is crossed in both directions.

One of the things I teach in the courses I run is that you have to think about the rhetorical nature of what you write: who will be reading, understanding, editing what you have written, and in what context? Often it’s you, months or years later, when you’ve forgotten what you were doing.

With this background in mind, let’s look at some specific changes in XPath 3 that can help to make that bridge wider—or if misused, can put up a barrier to the bridge by making the XPath expressions inscrutable to everyone!

We’ll look at three new operators in XPath, and then think about how they affect the rhetoric of the XSLT, XPath or even XQuery we write, whether they make obstacles to people or whether they widen the bridge between them.

The arrow operator, =>, takes a value (or an expression producing a value) and supplies it as the first argument to a function. It’s simpler than it sounds:

This is the same as writing:

Similarly,

is the same as,

and, if you have three staff members, might produce "Smith, Bowen, Walker" as output (in document order).

This new arrow operator might seem like a really minor convenience; in some ways it is, but when used carefully it can make your XPath expressions much easier to read. XPath gives the following example:

Remember also that since XPath 2 you can have comments in expressions, even when they’re inside XSLT select attributes or whatever, so we can write this more clearly:

If you’re not used to commented code, notice that the comments are right next to the things they describe, and that they explain the reason, not simply what is done. In this way they are useful and also likely to be kept up to date when the expression is changed.

Now, looking at this example, remember that with the arrow operator, the entire sequence of input is passed as the first argument to the function, not just one item at a time. To help you remember this, let’s look next at an operator that passes each item in turn, and then compare the two of them.

This is defined in XPath 3.1 (section 3.15).

The simple map operator is an exclamation mark (!) and works a bit like / in a Path expression. Since / looks a little like !, the similarity helps us to remember what it means and how you use it on a sequence:

Some Sequence ! Expression

This evaluates the expression once for each item in the sequence and then builds a new sequence with all of the results. It is like,

except (and this is important) the context item is set to the current item each time round, just like in a predicate:

This gives us as output:

The map operator does not filter the sequence: you always get the same number of items out as you put in. It also doesn’t sort into document order or eliminate duplicates as XPath’s / does.

If you have only one item in your sequence on the left, then ! and => are pretty similar. For example,

both give 5 as a result, although note the . in the second example. But when there’s more than one item in the sequence the difference starts to matter more:

gives a single string as a result: "socks, paint, sealing-wax" but

gives a sequence of three separate strings as a result, ("socks", "paint", "sealing-wax"). This is because ! takes each item in the input sequence in turn, evaluates string-join(.) with the item as . and then reassembles the pieces to make a result sequence with the three separate strings in it. And of course string-join() doesn't make any change if it only has one string as input: it puts the separator you give it in the second argument between strings when there’s more than one.

So in this way ! is like / in a path, and you might wonder why you can’t simply write this instead:

The answer is that—no, hold that thought. Why can’t you? If i try that in BaseX for example, i get an error,

There’s our clue (although the message could perhaps be clearer): the / path operator expects a sequence of nodes on the left. So i can write something like this XQuery fragment:

and get the sequence (5 5 11) as a result. In fact i could use ! instead of / for this example, but if these elements had been in the document, / also sorts them into document order and eliminates duplicates, where ! does not. It turns out that we most often want path results in document order and without duplicates, so we really do want separate operators. Use ! when the left-hand-side is a sequence of things other than document nodes, and use / when you want sorting and duplicate-weeding.

Finally, let’s look at one more new operator in XPath before going back to bridges.

The || operator joins two strings to make a single longer one. Sometimes i think it’s called concatenation because a programmer was given a big dictionary for Christmas one year and wanted to show off one of their new words, but who knows? At any rate “concatenation” has been the word used in computing for joining things together for many decades now, long before XPath. It’s why the Unix and Linux program that joins files together is called cat too, short for concatenate. I suppose con sounded a bit fishy. At any rate, in case it’s hard to see, || is made up of two vertical bars (the Unix pipe operator) one after the other.

I said it joins two strings, but you can write something like this, with the two expressions being essentially the same:

Either way, this makes a single string, sockspaintsealing-wax.

Oh dear, it’s not very pretty. It’d be much clearer here to put spaced between, but it starts getting ugly:

and you start wondering whether the pairs of vertical bars are inside or outside the quotes! This particular example is clearer as,

except that doesn’t work, because the strings are joined together before string-join sees them. It also doesn’t work for another reason:

gives the result, sockspaint11. That’s because the => happens first and then the string joining.

Moral: always use parentheses when combining different sorts of operators, because even if you just looked up the precedence you will have forgotten tomorrow. We need, of course,

The || operator has another surprise for us. In C-like languages, including JavaScript, Perl, bash, awk, Java and more, || is a short-circuiting logical or operator. In English, that means a || b is true if either a or b is true, and it has the value of the first true one in the sequence. It’s similar to this:

in XPath, except that the XPath engine might evaluate all four items and then choose the first if it wants, possibly in separate threads. This is a very common XPath idiom, often written as, e.g.,

In those other languages you’d write,

but in XPath 3 that just takes the string value of each of those and joins them. Well, not a problem except for people who use other scripting or programming language, which turns out to be most of us. As a reminder, i always make sure to have a constant string as one of the arguments, so,

Of course, it’s easier in this example to use:

and the person maintaining it doesn’t need to think about the || operator. Sometimes it’s better to use plainer code simply to help whoever will have to change it later—and almost all code does have to be changed, even if it was once perfect, because requirements and needs changes.

XPath 3 brings a lot of new ways to write things to XPath, and hence to XSLT and XQuery and XForms and Schematron and anywhere else XPath 3 could be used. We’ve only scratched the surface here: i run a three-day course on XSL 3 for XSLT 2 people, and experience suggests it could easily be a two-week course. So my goal is to get people comfortably reading the specifications and asking questions in the right places so they can continue learning on their own. And, of course, to focus on thinking about how we express ourselves: the rhetoric of XPath and XSLT. That we think about who might read what we write, and what they might know and be comfortable with. Are they programmers? Or digital humanists? Or scripters? Or perhaps they have crossed the bridge and become competent in more than one area without ever noticing it.

Because that’s the power of XML and XPath: they help us to find new ways to think about interesting challenges and problems, and they help us get things done that we need to do.

Thank you for reading—and you can read more about the course at https://www.delightfulcomputing.com/