I am just learning about command line processing, and hoping someone can confirm how I am interpreting the following statement. In the book I am reading the first step in command line processing is:

  1. Splits the command into tokens that are separated by the fixed set of metacharacters: SPACE, TAB, NEWLINE, ;, (, ), <, >, |, and &. Types of tokens include words, keywords, I/O redirectors, and semicolons

Am I right in thinking that for the command:

ls | more

ls and more are the tokens, and the pipe character is the meta character separating the two tokens?

I got bit confused as it goes on to say that < and > are meta characters, but then says that tokens can be I/O directors.

  • 2
    In practice, an isolated metacharacter (like |) or some sequences of them(like || or >&) will also constitute tokens; that's why the sentence goes on to say that redirectors (>, <, >&, and so on) are tokens. But the sentence in your book isn't formulated in a way to acknowledge this. I agree with you that as it's written, the metacharacters aren't themselves tokens. The sentence as written is incorrect. – dubiousjim Oct 8 '15 at 13:22
  • it depends a great deal on the shell's current parse level. basically, the words aren't tokens until they're tokenized - the shell just stacks them a character at a time as it reads until it encounters one of your metacharacters - each of which is a recognizable token unto itself at that point - and the shell can call all that its stacked so far a token. – mikeserv Oct 8 '15 at 16:41

This is not a very good explanation. A token is a sequence of characters that forms a word or punctuation sign. Characters like < and | are part of tokens too. You may call them metacharacters but this is not useful terminology. The basic rules are:

  • Whitespace is not part of a token and separates tokens.
  • A token is made up of ordinary characters, or of operator characters ()<>&|;, but not both. For example, foo<@a&>b consists of the tokens foo (ordinary), < (operator), @a (ordinary), &> (operator) and b.

Then there are additional rules about quoting: special characters lose their meaning if they're quotes, with different rules depending on the type of quote. For example, foo'&&'bar\|qux is a single token with the character sequence foo&&bar|qux.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.