bash tab completion with spaces

Question

I'm having a problem with bash-completion when the possible options may contain spaces.

Let's say I want a function which echoes the first argument:

function test1() {
        echo $1
}

I generate a list of possible completion options (some have spaces, some not), but I don't manage to handle spaces correctly.

function pink() {
    # my real-world example generates a similar string using awk and other commands
    echo "nick\ mason syd-barrett david_gilmour roger\ waters richard\ wright"
}

function _test() {
    cur=${COMP_WORDS[COMP_CWORD]}
    use=`pink`
    COMPREPLY=( $( compgen -W "$use" -- $cur ) )
}
complete -o filenames -F _test test

When I try this, I get:

$ test <tab><tab>
david_gilmour  nick           roger          waters
mason          richard        syd-barrett    wright
$ test r<tab><tab>
richard  roger    waters   wright

which is obviously not what I meant.

If I don't assign an array to COMPREPLY, i.e. only $( compgen -W "$use" -- $cur ), I get it working if only one option remains:

$ test n<tab>
$ test nick\ mason <cursor>

But if several options remain, they are all printed within single quotes:

$ test r<tab><tab>
$ test 'roger waters
richard wright' <cursor>

There must be something wrong with my COMPREPLY variable, but I can't figure out what...

(running bash on solaris, in case that makes a difference...)

Answer 1

The problem with OP's code is just the expansion of the output of compgen. It uses shell's sloppy expansion COMPREPLY=(...) which breaks on every space.

Replace that line with

mapfile -t COMPREPLY < <( compgen -W "$use" -- $cur )

and the code works fine.

Note that Shellcheck would have complained about that COMPREPLY=(...) expansion, and would have suggested using mapfile instead (see here).

Also, note that you don't see the escape character in the suggested completions, but when you press TAB the shell will add it to the command. If you start your argument with quotes (either single or double), the shell will use quotes (the one you chose) instead of backslashes to protect the whitespaces. As long as COMPREPLY is built properly, this all comes for free, so there's no need to mangle the completion patterns.

---

As others pointed out, instead of as an escaped string, the command providing the expansion patterns could return them one per line (like e.g. awk or ls). As you now surely know, you can read the output into an array using mapfile:

mapfile -t my_array < <(the_command)

The problem is that compgen wants a string and not an array, so you need to expand it again, but you need to escape the whitespaces inside the single elements. This is probably the only tricky part of the whole process.

Fortunately, you can do array expansion and substring substitution at the same time:

"${my_array[*]// /\\ }"

and you can feed the result directly to compgen:

function _test() {
    local use cur
    cur=${COMP_WORDS[COMP_CWORD]}
    mapfile -t use < <(pink)
    mapfile -t COMPREPLY < <(compgen -W "${use[*]// /\\ }" -- "$cur")
}

Disclaimer: I apologize for the many concepts here that are already present in the previous answers. I just wanted to stress that properly splitting the completions into the COMPREPLY array is the only requirement for tab completion. The shell takes care of everything after that, so it's not that difficult after all. Moreover, as a teaching habit, I prefer to work out the problems in the OP's solution before providing a reference solution.

Answer 2

I don't have mapfile available. The following seems a bit simpler than the other answers and worked fine for me:

_completion() {
  local CUR=${COMP_WORDS[COMP_CWORD]}
  local OPT
  local -a OPTS

  while read -r OPT; do
    local OPT_ESC
    OPT_ESC="$(printf '%q' "$OPT")"
    [[ -z "$CUR" ]] || [[ "$OPT_ESC" == "$CUR"* ]] && \
      COMPREPLY+=("$OPT_ESC")
  done < "${TOKEN_FILE}"
}

• Each line in TOKEN_FILE is a completion option.
• Replace "${TOKEN_FILE}" with <(token_generator) where token_genorator is some command than generates completion tokens, on line at a time.
• The printf '%q' gives us a bash-escaped string suitable for consumption on the command line as a single token.

So, if TOKEN_FILE is:

option a
another option
an option with many spaces

Then tab-completing would print:

an\ option\ with\ many\ spaces     another\ option
option\ a

Answer 3

Custom tab-completing words which might include whitespace is annoyingly difficult. And as far as I know there is no elegant solution. Perhaps some future version of compgen will be kind enough to produce an array rather than outputting possibilities one line at a time, and even accept the wordlist argument from an array. But until then, the following approach may help.

It's important to understand the problem, which is that ( $(compgen ... ) ) is an array produced by splitting the output of the compgen command at the characters in $IFS, which by default is any whitespace character. So if compgen returns:

roger waters
richard wright

then COMPREPLY will effectively be set to the array (roger waters richard wright), for a total of four possible completions. If you instead use ( "$(compgen ...)"), then COMPREPLY will be set to the array ($'roger waters\nrichard wright'), which has only one possible completion (with a newline inside the completion). Neither of those are what you want.

If none of the possible completions has a newline character, then you could arrange for the compgen return to be split at the newline character by temporarily resetting IFS and then restoring it. But I think a more elegant solution is to just use mapfile:

_test () { 
    cur=${COMP_WORDS[COMP_CWORD]};
    use=`pink`;
    ## See note at end of answer w.r.t. "$cur" ##
    mapfile -t COMPREPLY < <( compgen -W "$use" -- "$cur" )
}

The mapfile command places the lines sent by compgen to stdout into the array COMPREPLY. (The -t option causes the trailing newline to be removed from each line, which is almost always what you want when you use mapfile. See help mapfile for more options.)

This doesn't deal with the other annoying part of the problem, which is mangling the wordlist into a form acceptable by compgen. Since compgen does not allow multiple -W options, and nor does it accept an array, the only option is to format a string in a such a way that bash word-splitting (with quotes and all) would generate the desired list. In effect, that means manually adding escapes, as you did in your function pink:

pink() {
    echo "nick\ mason syd-barrett david_gilmour roger\ waters richard\ wright"
}

But that's accident-prone and annoying. A nicer solution would allow the specification of the alternatives directly, particularly if the alternatives are being generated in some fashion. A good way of generating alternatives which might include whitespace is to put them into an array. Given an array, you can make good use of printf's %q format to produce a properly-quoted input string for compgen -W:

# This is a proxy for a database query or some such which produces the alternatives
cat >/tmp/pink <<EOP
nick mason
syd-barrett
david_gilmour
roger waters
richard wright
EOP

# Generate an array with the alternatives
mapfile -t pink </tmp/pink

# Use printf to turn the array into a quoted string:
_test () { 
    mapfile -t COMPREPLY < <( compgen -W "$(printf '%q ' "${pink[@]}")" -- "$2" )
}

As written, that completion function does not output completions in a form which will be accepted by bash as single words. In other words, the completion roger waters is generated as roger waters instead of roger\ waters. In the (likely) case that the goal is to produce correctly quoted completions, it is necessary to add escapes a second time, after compgen filters the completion list:

_test () {
    declare -a completions
    mapfile -t completions < <( compgen -W "$(printf '%q ' "${pink[@]}")" -- "$2" )
    local comp
    COMPREPLY=()
    for comp in "${completions[@]}"; do
        COMPREPLY+=("$(printf "%q" "$comp")")
    done
}

Note: I replaced the computation of $cur with $2, since the function invoked through complete -F is passed the command as $1 and the word being completed as $2. (It's also passed the previous word as $3.) Also, it's important to quote it, so that it doesn't get word-split on its way into compgen.

Answer 4

Okay, this crazy contraption draws heavily on rici’s solution, and not only fully works, but also quotes any completions that need it, and only those.

pink() {
    # simulating actual awk output
    echo "nick mason"
    echo "syd-barrett"
    echo "david_gilmour"
    echo "roger waters"
    echo "richard wright"
}

_test() {
  cur=${COMP_WORDS[COMP_CWORD]}
  mapfile -t patterns < <( pink )
  mapfile -t COMPREPLY < <( compgen -W "$( printf '%q ' "${patterns[@]}" )" -- "$cur" | awk '/ / { print "\""$0"\"" } /^[^ ]+$/ { print $0 }' )
}

complete -F _test test

So as far as I could test it, it fully implements ls-like behavior, minus the path-specific parts.

Verbose example

Here’s a more verbose version of the _test function, so it becomes a bit more understandable:

_test() {
  local cur escapedPatterns
  cur=${COMP_WORDS[COMP_CWORD]}
  mapfile -t patterns < <( pink )
  escapedPatterns="$( printf '%q ' "${patterns[@]}" )"
  mapfile -t COMPREPLY < <( compgen -W "$escapedPatterns" -- "$cur" | quoteIfNeeded )
}

quoteIfNeeded() {
  # Only if it contains spaces. Otherwise return as-is.
  awk '/ / { print "\""$0"\"" } /^[^ ]+$/ { print $0 }'
}

None of this is even remotely optimized for efficiency. Then again, this is only tab completion, and it’s not causing a noticeable delay for any reasonably large list of completions.

It works by:

1. Pulling the awk output into an array, using mapfile.
2. Escaping the array and putting it into a string.
3. Having a single space behind the %q as a separation marker.
4. Quoting $cur, Very important!
5. Quoting the output of compgen. And only if it contains spaces.
6. Feeding that output into COMPREPLY, using another mapfile call.
7. Not using -o filenames.

And it only works with all those tricks. It fails if even a single one is missing. Trust me; I’ve tried. ;)

Answer 5

If you need to process the data from the string you can use Bash's built-in string replacement operator.

function _test() {
    local iter use cur
    cur=${COMP_WORDS[COMP_CWORD]}
    use="nick\ mason syd-barrett david_gilmour roger\ waters richard\ wright"
    # swap out escaped spaces temporarily
    use="${use//\\ /___}"
    # split on all spaces
    for iter in $use; do
        # only reply with completions
        if [[ $iter =~ ^$cur ]]; then
            # swap back our escaped spaces
            COMPREPLY+=( "${iter//___/ }" )
        fi
    done
}