find <> -exec curl ... for ftp file upload having problem with url encoding, for filenames with random characters

Question

Use case: I am trying to copy data/everything from my android mobile to laptop on same wifi, maintaining directory structure. I know there are tons of methods, but I love to do it this way (below) and need help. And then do whatever on the laptop with those files.

File samples/directory structure to upload from mobile to laptop ftp server -

dir1
  |--1.mp3
  |--dir2
       |--download  1.jpg
       |--download (2).jpg

How to reproduce: In mobile I have terminal similar to Ubuntu. In laptop I am running ftp server using python. From mobile I am running below command and it works fine, just that directory structure is not maintained and is dumping every file to the root directory of ftp (and I know why bcz there is no path in url).

find dir1 -type f -exec curl -u ftp:ftp --ftp-create-dirs -T {} ftp://192.168.x.x/ \;

Point is, there is no filename/path in url of curl so there is no issue of url encoding. All files as it is uploaded to root directory.

So to maintain sub directory structure I have to do below url call.

find dir1 -type f -exec curl -u ftp:ftp --ftp-create-dirs -T {} ftp://192.168.x.x/{} \;

Now it will throw error for filenames having spaces, (, ) or anything like that due to url encoding. For example for "download 1.jpg" and "download (2).jpg"

curl: (3) URL using bad/illegal format or missing URL

Stuffs I have tried so far (echo I have used below to show the variables values) -

find dir1 -type f -exec bash -c 'y=$(echo "$1"); fname=$(echo $y | sed "s/ /%20/g"); echo $y;echo $fname;echo ""; curl -u ftp:ftp --ftp-create-dirs -T $y ftp://192.168.x.x/$fname' bash {} \;

Output -

dir1/1.mp3
dir1/1.mp3

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

dir1/dir2/download 1.jpg
dir1/dir2/download%201.jpg

curl: Can't open 'dir1/dir2/download'
curl: try 'curl --help' for more information
curl: (26) Failed to open/read local data from file/application

dir1/dir2/download (2).jpg
dir1/dir2/download%20(2).jpg

curl: (3) URL using bad/illegal format or missing URL

---

So you can see the problem I am having is with url encoding. Even I have done encoding for space it is not working. And there will be much more random characters.

I thought to use --data-urlencoding of curl but that won't allow -G with -T, or -T with --data. Plus there will be a "?" string added this way if it was going to work which is not my use case.

Answer 1

Ok I did it. Since it was mobile I wanted to keep it free from any other package/tool and to basics. sed helped me in url encoding. Also this-page of github was helpful in gathering those mappers in one place.

I had to test them and few were causing problem, bcz of my code how it is written. e.g. "s/\'/%27/g" "s/\\/%5c/g" I did not bother to fix them as my job was done without these.

This was causing to put $ at end of every uploaded file - "s/\$/%24/g"

There was one mistake I was doing in curl, I was not putting double quotes here and few errors were bcz of this and I was thinking curl is bad -T "$y"

This is hot it worked.

find dir1 -type f -exec bash -c 'y=$(echo "$1"); fname=$(echo $y | sed -e "s/%/%25/g" -e "s/ /%20/g" -e "s/!/%21/g" -e "s/#/%23/g" -e "s/(/%28/g" -e "s/)/%29/g" -e "s/+/%2b/g" -e "s/,/%2c/g" -e "s/-/%2d/g" -e "s/:/%3a/g" -e "s/;/%3b/g" -e "s/?/%3f/g" -e "s/@/%40/g" -e "s/\&/%26/g" -e "s/\*/%2a/g" -e "s/\./%2e/g" -e "s/\//%2f/g" -e "s/\[/%5b/g" -e "s/\]/%5d/g" -e "s/\^/%5e/g" -e "s/_/%5f/g" -e "s/{/%7b/g" -e "s/|/%7c/g" -e "s/}/%7d/g" -e "s/~/%7e/g" -e "s/\"/%22/g" -e "s/\`/%60/g"); echo $y;echo $fname;echo ""; curl -u ftp:ftp --ftp-create-dirs -T "$y" ftp://192.168.x.x/$fname ; echo ""' bash {} \;

Basically my mobile command is like this - full mobile files backup maintaining directory structure. Suppose your phone screen is broken, it is not booting properly bcz of storage full, it is stuck at boot bcz of multiple reasons, etc. etc. ADB helps.

cd /

find sdcard -type f -exec bash -c 'y=$(echo "$1"); fname=$(echo $y | sed -e "s/%/%25/g" -e "s/ /%20/g" -e "s/!/%21/g" -e "s/#/%23/g" -e "s/(/%28/g" -e "s/)/%29/g" -e "s/+/%2b/g" -e "s/,/%2c/g" -e "s/-/%2d/g" -e "s/:/%3a/g" -e "s/;/%3b/g" -e "s/?/%3f/g" -e "s/@/%40/g" -e "s/\&/%26/g" -e "s/\*/%2a/g" -e "s/\./%2e/g" -e "s/\//%2f/g" -e "s/\[/%5b/g" -e "s/\]/%5d/g" -e "s/\^/%5e/g" -e "s/_/%5f/g" -e "s/{/%7b/g" -e "s/|/%7c/g" -e "s/}/%7d/g" -e "s/~/%7e/g" -e "s/\"/%22/g" -e "s/\`/%60/g"); echo $y;echo $fname;echo ""; curl -u ftp:ftp --ftp-create-dirs -T "$y" ftp://192.168.x.x/$fname ; echo ""' bash {} \;