CODEWITHSUNDEEP
bash
Thomas Eizinger
Thomas Eizinger

Reputation: 1462

Unexpected bash quoting behaviour

I have a pipeline of bash commands that looks like this:

cat report.json | \
./jq '.outdated.dependencies[] |  {group, name, "version": .available.milestone} | .group + ":" + .name + ":" + .version' | \
xargs -0 -d'\n' -I DEPENDENCY echo ./gradlew clean test -PdependencyOverrides=DEPENDENCY

The output of this command is:

./gradlew clean test -PdependencyOverrides="org.flywaydb:flyway-core:4.0.3"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-c3p0:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-core:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-entitymanager:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-java8:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-validator:5.3.3.Final"
./gradlew clean test -PdependencyOverrides="com.fasterxml.jackson.core:jackson-databind:2.8.5"
./gradlew clean test -PdependencyOverrides="com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.8.5"
./gradlew clean test -PdependencyOverrides="javax.el:javax.el-api:3.0.1-b04"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-api:2.7"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-core:2.7"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-slf4j-impl:2.7"
./gradlew clean test -PdependencyOverrides="org.mockito:mockito-core:2.2.28"
./gradlew clean test -PdependencyOverrides="org.postgresql:postgresql:9.4.1212"
./gradlew clean test -PdependencyOverrides="com.getsentry.raven:raven-log4j2:7.8.1"
./gradlew clean test -PdependencyOverrides="com.sparkjava:spark-core:2.5.4"
./gradlew clean test -PdependencyOverrides="org.springframework:spring-jdbc:4.3.4.RELEASE"
./gradlew clean test -PdependencyOverrides="org.springframework.restdocs:spring-restdocs-restassured:1.1.2.RELEASE"

This is exactly what I am expecting. If I copy and paste one of these commands, they do exactly what I want them to do. Now I don't want to print these commands but directly execute them. However, as soon as I remove the echo from the xargs command, it doesn't work. "It doesn't work" means that the property is not correctly passed to gradle.

In order to debug this issue, I tried to append | bash -x to my pipeline which revealed the issue but I don't know how to solve it.
Appending this yields the following output as the first line:

+ ./gradlew clean test $'-PdependencyOverrides=org.flywaydb:flyway-core:4.0.3\r'

I assume, the problem is that bash for some reason I don't know puts the arguments in quotes and prepends them with $. If I execute the command exactly as it is printed here, the same error occurs as if I let xargs directly run the command.

What do I have to change so the commands are executed exactly the way they are printed when I add the echo to the xargs command?

Here is the report.json file if anyone wants to reproduce this locally and jq can be downloaded here.

Gradle is not needed to reproduce this problem, as executing the command in a folder without a gradlew file just yields:

+ ./gradlew clean test $'-PdependencyOverrides=org.flywaydb:flyway-core:4.0.3\r'
bash: line 1: ./gradlew: No such file or directory

which still demonstrates the problem.

I am not sure what the exact problem is, so if there is a better title feel free to change it.

Upvotes: 1

Views: 48

Answers (1)

l0b0
l0b0

Reputation: 58958

$'' is a valid way of quoting strings in Bash. It's being used because xtrace escapes strings if necessary to print the exact command which was run. You get the same kind of result if you try to printf '%q\n' "$something", where something contains special characters like newline, tab, backspace, etc.

\r won't be visible when you look at output in some editor/terminal which does not escape special characters. For example, if you open a file with \r at the end of lines it will simply show [dos] at the bottom of the window to indicate that the line separator is \r\n (DOS/Windows standard) rather than \n (*nix standard).

Upvotes: 3

Related Questions