CODEWITHSUNDEEP
curlnexusnexus3
Neoneuron
Neoneuron

Reputation: 193

Download entire repository from Nexus 3.37.1

I was wondering as to how I can download an entire folder or a repository from Nexus 3.37. I tried the following command

curl -X GET -u userid:password "https://nexus.com/abc/def/ghi/" -O

I was able to download a single file in a directory using the above mentioned command, Does anyone have ideas as to how I can download the entire folder?

Thanks!

Upvotes: 2

Views: 14234

Answers (4)

khurram
khurram

Reputation: 1

thanks for sharing, it works for me, but i wanted to check, after running this scripts it generate few files e.g .log and .txt and tmp files, do we need to use .txt file to upload onto new nexus machine?

Dimos Koul

Upvotes: 0

Dimos Koul
Dimos Koul

Reputation: 71

You can use curl to retrieve a list of all your artifacts download URLs by executing:

curFolder=`pwd`
filters=("\\.jar$" "\\.pom$" "\\.zip$")
sourceServer=https://myserver.com:1234
sourceRepo=myrepo
sourceUser=username
sourcePassword=password
logfile=$sourceRepo-backup.log
outputFile=$sourceRepo-artifacts.txt
# ======== GET DOWNLOAD URLs =========
url=$sourceServer"/service/rest/v1/assets?repository="$sourceRepo
contToken="initial"
while [ ! -z "$contToken" ]; do
    if [ "$contToken" != "initial" ]; then
        url=$sourceServer"/service/rest/v1/assets?continuationToken="$contToken"&repository="$sourceRepo
    fi
    echo Processing repository token: $contToken | tee -a $logfile
    response=`curl -ksSL -u "$sourceUser:$sourcePassword" -X GET --header 'Accept: application/json' "$url"`
    artifacts=( $(echo $response | sed -n 's|.*"downloadUrl" : "\([^"]*\)".*|\1|p') )
    printf "%s\n" "${artifacts[@]}" > artifacts.temp
    for filter in "${filters[@]}"; do
        cat artifacts.temp | grep "$filter" >> $outputFile
    done
    contToken=( $(echo $response | sed -n 's|.*"continuationToken" : "\([^"]*\)".*|\1|p') )
done

At this point you have all the artifacts filtered out from sha1 and md5 files, metadata etc. so you can loop through the lines of the output file and download them using curl.

# ======== DOWNLOAD EVERYTHING =========
    echo Downloading artifacts...
    urls=($(cat $outputFile)) > /dev/null 2>&1
    for url in "${urls[@]}"; do
        path=${url#http://*:*/*/*/}
        dir=$sourceRepo"/"${path%/*}
        mkdir -p $dir
        cd $dir
        curl -vks -u "$sourceUser:$sourcePassword" -D response.header -X GET "$url" -O  >> /dev/null 2>&1
        responseCode=`cat response.header | sed -n '1p' | cut -d' ' -f2`
        if [ "$responseCode" == "200" ]; then
            echo Successfully downloaded artifact: $url  | tee -a $logfile 2>&1
        else
            echo ERROR: Failed to download artifact: $url  with error code: $responseCode  | tee -a $logfile 2>&1
        fi
        rm response.header > /dev/null 2>&1
        cd $curFolder
    done

This is downloading everything and preserving the folder structure so later you can upload everything using Maven (assuming Maven2 repo). It will also report if it was successful at downloading it or not. Try to check first the credentials, otherwise you will end up doing thousands of wrong attempts and you will end up locking your account.

Upvotes: 1

xJom
xJom

Reputation: 31

I am a little bit intrigued by the fact that there is no "download all files in a folder" in the REST API

So here are my changes:

  • Taking back a simple filter (for e g a sub-folder)
  • Preserving spaces in the URLs and converting to %20 for download
  • Setting the curFolder to the result of pwd
  • Removing the output file if it exists when you do consecutive runs
  • Changed from http to https
sourceServer=
sourceRepo=
sourceFolder=
sourceUser=
sourcePassword=
logfile=$sourceRepo-backup.log
outputFile=$sourceRepo-artifacts.txt
[ -e $outputFile ] && rm $outputFile

# ======== GET DOWNLOAD URLs =========
url=$sourceServer"/service/rest/v1/assets?repository="$sourceRepo
contToken="initial"
while [ ! -z "$contToken" ]; do
    if [ "$contToken" != "initial" ]; then
        url=$sourceServer"/service/rest/v1/assets?continuationToken="$contToken"&repository="$sourceRepo
    fi
    echo Processing repository token: $contToken | tee -a $logfile
    response=`curl -ksSL -u "$sourceUser:$sourcePassword" -X GET --header 'Accept: application/json' "$url"`
    readarray -t artifacts < <( jq  '[.items[].downloadUrl]' <<< "$response" )
    printf "%s\n" "${artifacts[@]}" > artifacts.temp
    sed 's/\"//g' artifacts.temp > artifacts1.temp
    sed 's/,//g' artifacts1.temp > artifacts2.temp
    sed 's/[][]//g' artifacts2.temp > artifacts3.temp
    cat artifacts3.temp | grep "$sourceFolder" >> $outputFile
    contToken=( $(echo $response | sed -n 's|.*"continuationToken" : "\([^"]*\)".*|\1|p') )
done


# ======== DOWNLOAD EVERYTHING =========
    echo Downloading artifacts...
    IFS=$'\n' read -d '' -r -a urls < $outputFile
    for url in "${urls[@]}"; do
        url="$(echo -e "${url}" | sed -e 's/^[[:space:]]*//')"
        path=${url#https://*/*/*/}
        dir="\""$sourceRepo"/"${path%/*}"\""
        curFolder=$(pwd)
        mkdir -p $dir
        cd $dir
        url="$(echo -e "${url}" | sed -e 's/\s/%20/g')"
        curl -vks -u "$sourceUser:$sourcePassword" -D response.header -X GET "$url" -O  >> /dev/null 2>&1
        responseCode=`cat response.header | sed -n '1p' | cut -d' ' -f2`
        if [ "$responseCode" == "200" ]; then
            echo Successfully downloaded artifact: $url
        else
            echo ERROR: Failed to download artifact: $url  with error code: $responseCode
        fi
        rm response.header > /dev/null 2>&1
        cd $curFolder
    done

Upvotes: 3

lildazeez
lildazeez

Reputation: 59

The script above really helped me with this issue. I slightly modified it and posted it below. This script is by no means perfect and you may need to edit it. I know one issue is that the very first download this script initiates is put in the wrong directory.

sourceServer=
sourceRepo=
sourceUser=
sourcePassword=
logfile=$sourceRepo-backup.log
outputFile=$sourceRepo-artifacts.txt

# ======== GET DOWNLOAD URLs =========
url=$sourceServer"/service/rest/v1/assets?repository="$sourceRepo
contToken="initial"
while [ ! -z "$contToken" ]; do
    if [ "$contToken" != "initial" ]; then
        url=$sourceServer"/service/rest/v1/assets?continuationToken="$contToken"&repository="$sourceRepo
    fi
    echo Processing repository token: $contToken | tee -a $logfile
    response=`curl -ksSL -u "$sourceUser:$sourcePassword" -X GET --header 'Accept: application/json' "$url"`
    readarray -t artifacts < <( jq  '[.items[].downloadUrl]' <<< "$response" )
    printf "%s\n" "${artifacts[@]}" > artifacts.temp
    sed 's/\"//g' artifacts.temp > artifacts1.temp
    sed 's/,//g' artifacts1.temp > artifacts.temp
    sed 's/[][]//g' artifacts.temp > artifacts1.temp
    cat artifacts1.temp >> $outputFile
#for filter in "${filters[@]}"; do
     #   cat artifacts.temp | grep "$filter" >> $outputFile
    #done
    #cat maven-public-artifacts.txt
    contToken=( $(echo $response | sed -n 's|.*"continuationToken" : "\([^"]*\)".*|\1|p') )
done


# ======== DOWNLOAD EVERYTHING =========
    echo Downloading artifacts...
    urls=($(cat $outputFile)) > /dev/null 2>&1
    for url in "${urls[@]}"; do
        path=${url#http://*:*/*/*/}
        dir=$sourceRepo"/"${path%/*}
        mkdir -p  $dir
        cd $dir
        pwd
        curl -vks -u "$sourceUser:$sourcePassword" -D response.header -X GET "$url" -O  >> /dev/null 2>&1
        responseCode=`cat response.header | sed -n '1p' | cut -d' ' -f2`
        if [ "$responseCode" == "200" ]; then
            echo Successfully downloaded artifact: $url
        else
            echo ERROR: Failed to download artifact: $url  with error code: $responseCode
        fi
        rm response.header > /dev/null 2>&1
        cd $curFolder
    done

Upvotes: 2

Related Questions