CODEWITHSUNDEEP
linuxbash
adjs1157
adjs1157

Reputation: 103

Why does Bash evaluate my variable expansion I stored in a prior variable?

I'm trying to figure out why Bash is seemingly evaluating a variable expansion for a variable in which I use a previously defined variable that is using variable expansion. Perhaps there is something else going on here, but what it is, I can't quite figure out.

I define $LATEST_VERSION to take the everything past the last '/' in $LOCATION, the URL I am being redirected to from $LATEST_URL.

At the moment $LATEST_VERSION gets evaluated to the following:

v0.11.17

This is the code that I am executing:

#!/bin/bash
OS=${1:-"linux"}
ARCH=${2:-"arm"}
LATEST_URL="https://github.com/syncthing/syncthing/releases/latest"
LOCATION=`curl -I $LATEST_URL | perl -n -e '/^Location: (.*)$/ &&  print "$1\n"'`
LATEST_VERSION=${LOCATION##*/}
VERSION=${3:-$LATEST_VERSION}
DOWNLOAD_URL="https://github.com/syncthing/syncthing/release/download/${VERSION}/syncthing-${OS}-${ARCH}-${VERSION}.tar.gz"
echo "DOWNLOAD_URL: ${DOWNLOAD_URL}"

Everything is fine and dandy until I echo "DOWNLOAD_URL: ${DOWNLOAD_URL}":

.tar.gzing-linux-arm-v0.11.17com/syncthing/syncthing/release/download/v0.11.17

If I take out $VERSION from $DOWNLOAD_URL, it looks as expected ($OS and $ARCH are substituted):

DOWNLOAD_URL: https://github.com/syncthing/syncthing/release/download//syncthing-linux-arm-.tar.gz

I tried everything from prepending the strings with echo -e, i.e.

LATEST_VERSION=`echo -e ${LOCATION##*/}`
VERSION=`echo -e ${3:-$LATEST_VERSION}`

to 

LATEST_VERSION=`basename $LOCATION`

and they all give me unexpected output. Is there something that I'm missing here?

Thank you.

Upvotes: 1

Views: 60

Answers (2)

Jonathan Leffler
Jonathan Leffler

Reputation: 754570

You've probably got CRLF line endings in the data coming off the internet and the carriage returns (CR) are screwing things up. They move the printing position to the start of the line without forcing out a newline (LF, or line feed). That's why you're seeing .tar.gz at the start of the line.

If you echo the value of LATEST_VERSION through a program like od -c, you'll see a carriage return \r at the end.

Your download URL is created from:

DOWNLOAD_URL="https://github.com/syncthing/syncthing/release/download/${VERSION}/syncthing-${OS}-${ARCH}-${VERSION}.tar.gz"
echo "DOWNLOAD_URL: ${DOWNLOAD_URL}"

Showing the segments, you'd see the following three segments overlaid:

DOWNLOAD_URL: https://github.com/syncthing/syncthing/release/download/v0.11.17\r
/syncthing-linux-arm-v0.11.17\r
.tar.gz

leading to the observed output:

.tar.gzing-linux-arm-v0.11.17com/syncthing/syncthing/release/download/v0.11.17

To fix, you can edit out the carriage return (see parameter expansion in the Bash manual):

LATEST_VERSION=${LATEST_VERSION/$'\r'/}

Upvotes: 2

melpomene
melpomene

Reputation: 85837

There's a special character called carriage return (ASCII 13, also known as CR or \r) that moves the cursor to the beginning of the line when printed.

HTTP headers are terminated by a carriage return / line feed combo (CR LF).

Your Perl regex (/^Location: (.*)$/) strips off the LF (. doesn't match newline by default) but keeps the CR.

LATEST_VERSION ends up containing v0.11.17\r (where \r represents the invisible carriage return).

DOWNLOAD_URL ends up being https://github.com/syncthing/syncthing/release/download/v0.11.17\r/syncthing-linux-arm-v0.11.17\r.tar.gz (again using \r to represent carriage return), which when printed looks garbled because \r moves the cursor back:

https://github.com/syncthing/syncthing/release/download/v0.11.17
/syncthing-linux-arm-v0.11.17
.tar.gz
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.tar.gzing-linux-arm-v0.11.17syncthing/release/download/v0.11.17

One way to fix this is to change the regex to /^Location: (.*)\r\n/, which moves the \r out of $1.

Upvotes: 4

Related Questions