CODEWITHSUNDEEP
shellartifactorycontinuous-deployment
user1338413
user1338413

Reputation: 2551

How to download the latest artifact from Artifactory repository?

I need the latest artifact (for example, a snapshot) from a repository in Artifactory. This artifact needs to be copied to a server (Linux) via a script.

What are my options? Something like Wget / SCP? And how do I get the path of the artifact?

I found some solutions which require Artifactory Pro. But I just have Artifactory, not Artifactory Pro.

Is it possible at all to download from Artifactory without the UI and not having the Pro-Version? What is the experience?

I'm on OpenSUSE 12.1 (x86_64) if that matters.

Upvotes: 85

Views: 249176

Answers (17)

Bruce Edge
Bruce Edge

Reputation: 2189

jFrog now has a "gavc" search: https://jfrog.com/help/r/jfrog-rest-apis/gavc-search

Here's a few line wrapper you can use to search using the "jf" CLI, assuming you have that setup locally that uses the maven format directly:

function jfrog_gavs() {

# parse $1 into 3 parameters separated by ":"
    local input_string=${1:?}
    IFS=':' read -r group artifact type version <<< "$input_string"
    if [ "${DEBUG_JF}" ] ; then
        echo "group: $group"
        echo "artifact: $artifact"
        echo "type: $type"
        echo "version: $version"
    fi
    jf rt curl --silent -XGET  "/api/search/gavc?g=${group}&a=${artifact}&v=$version"
}

eg:

❯ jfrog_gavs org.jasypt:jasypt-spring31:jar:1.9.2
{
  "results" : [ {
    "uri" : "https://myinstance.jfrog.io/artifactory/api/storage/maven-central-mirror-cache/org/jasypt/jasypt-spring31/1.9.2/jasypt-spring31-1.9.2.pom"
  }, {
    "uri" : "https://myinstance.jfrog.io/artifactory/api/storage/maven-central-mirror-cache/org/jasypt/jasypt-spring31/1.9.2/jasypt-spring31-1.9.2.jar"
  } ]
}

Upvotes: 0

bZx
bZx

Reputation: 131

I was in a similar situation but I wanted to give a download link to my users. So I made a php script. I share it here because it may be useful to others and this stackoverflow page is well referenced by search engines.

<?php

/* Redirects to the latest artifact url from a maven repository
 * Usages:
 * http://localhost/download.php                                        : get the latest release
 * http://localhost/download.php?artifact=sources                       : get the latest release sources.jar
 * http://localhost/download.php?artifact=javadoc                       : get the latest release javadoc.jar
 * http://localhost/download.php?includeSnapshot=true                   : get the latest release or snapshot
 * http://localhost/download.php?includeSnapshot=true&artifact=sources  : get the latest release or snapshot sources.jar
 * http://localhost/download.php?includeSnapshot=true&artifact=javadoc  : get the latest release or snapshot javadoc.jar
*/

/**** constants (modify to fit your needs) ****/

// the url of the path to the artifact in the maven repository
$baseUrl = "https://repository.skytale.fr/artifactory";
$repository = "public";
$group = "eu/lasersenigma";
$artifactName = "LasersEnigma";
$artifactDirUrl = $baseUrl . "/" . $repository . "/" . $group . "/" . $artifactName . "/";

// Download a file and follow redirect
function downloadAndParseXml($url) {    
    $output = file_get_contents($url);
    // create curl resource
    //$ch = curl_init();

    // set url
    //curl_setopt($ch, CURLOPT_URL, $url);

    //return the transfer as a string
    //curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    //curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
    //curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    

    // $output contains the output string
    //$output = curl_exec($ch);
    //echo($output);

    // close curl resource to free up system resources
    //curl_close($ch);
    return simplexml_load_string($output);
}

// Download and parse artifact maven-metadata.xml from its url and find latest artifact (including snapshot or not according to second parameter)
function getLatestVersionFromMetadata($xmlUrl, $includeSnapshot = false) {
    $metadata = downloadAndParseXml($xmlUrl);
    $latestVersion="";
    if ($includeSnapshot) {
        $latestVersion = (string) $metadata->versioning->latest;
    } else {
        $latestVersion = (string) $metadata->versioning->release;
    }
    return $latestVersion;
}

// Download and parse artifact snapshot version maven-metadata.xml from its url and find latest snapshot timestamp and buildNumber
function getLatestSnapshotFromMetadata($xmlUrl) {
    $metadata = downloadAndParseXml($xmlUrl);
    $timestamp = (string) $metadata->versioning->snapshot->timestamp;
    $buildNumber = (string) $metadata->versioning->snapshot->buildNumber;
    return $timestamp . "-" . $buildNumber;
}

$mavenMetadataFileName = "maven-metadata.xml";

// Process parameters
$includeSnapshot = isset($_GET["includeSnapshot"]) && $_GET["includeSnapshot"] == "true";
$file = "";
if (isset($_GET["artifact"])) {
    if ($_GET["artifact"] == "javadoc") {
        $file = "-javadoc";
    } else if ($_GET["artifact"] == "sources") {
        $file = "-sources";
    }
}

// Parsing artifact directory maven-metadata.xml and get latest version directory name
$latestVersion = getLatestVersionFromMetadata($artifactDirUrl . $mavenMetadataFileName, $includeSnapshot);

if ($latestVersion == null) {
    echo("
        <h3 style=\"text-align: center\">Could not redirect to latest artifact automatically</h3>
        <p style=\"text-align:center\">
            Sorry for the inconvenience. You can find the file yourself here:<br/>
            <a href=\"" .$artifactDirUrl . "\">" . $artifactDirUrl . "</a>
        </p>
        ");
    exit();
}

$artifactoryLatestVersionDirPath = $artifactDirUrl . $latestVersion . "/";
$finalUrl = $artifactoryLatestVersionDirPath . $artifactName . "-";

if (str_contains($latestVersion, "SNAPSHOT")) {
    $timeStampAndRelease = getLatestSnapshotFromMetadata($artifactoryLatestVersionDirPath . $mavenMetadataFileName);
    $finalUrl .=  str_replace("-SNAPSHOT", "", $latestVersion) . "-" . $timeStampAndRelease;
} else {
    $finalUrl .= $latestVersion;
}

// add javadoc/sources and extension
$finalUrl .= $file . ".jar";

// Redirect
header("Location: " . $finalUrl);

exit();

Upvotes: 0

Steve Chambers
Steve Chambers

Reputation: 39404

If using the JFrog CLI, this can be done in a single line as follows:

jf rt dl "path/to/artifacts/-/" --sort-by=created --sort-order=desc --limit=1

(The above is for v2 but think the v1 command should be the same but substituting jf with jfrog.)

Upvotes: 1

lojza
lojza

Reputation: 1891

No one mention xmllint aka proper xml parser, install it with:

sudo apt-get update -qq
sudo apt-get install -y libxml2-utils

use it:

ART_URL="https://artifactory.internal.babycorp.com/artifactory/api-snapshot/com/babycorp/baby-app/CD-684-my-branch-SNAPSHOT"
ART_VERSION=`curl -s $ART_URL/maven-metadata.xml | xmllint --xpath '//snapshotVersion[1]/value/text()' -`

and finally:

curl -s -o baby-app.jar ${ART_URL}/baby-app-${ART_VERSION}.jar

or

wget ${ART_URL}/baby-app-${ART_VERSION}.jar

to keep the filename

Upvotes: 0

Daniel
Daniel

Reputation: 329

In case you need to download an artifact in a Dockerfile, instead of using wget or curl or the likes you can simply use the 'ADD' directive:

ADD ${ARTIFACT_URL} /opt/app/app.jar

Of course, the tricky part is determining the ARTIFACT_URL, but there's enough about that in all the other answers.

However, Docker best practises strongly discourage using ADD for this purpose and recommend using wget or curl.

Upvotes: 0

Ebrahim Salehi
Ebrahim Salehi

Reputation: 71

With awk:

     curl  -sS http://the_repo/com/stackoverflow/the_artifact/maven-metadata.xml | grep latest | awk -F'<latest>' '{print $2}' | awk -F'</latest>' '{print $1}'

With sed:

    curl  -sS http://the_repo/com/stackoverflow/the_artifact/maven-metadata.xml | grep latest | sed 's:<latest>::' | sed 's:</latest>::'

Upvotes: 1

Rafik
Rafik

Reputation: 455

If you want to download the latest jar between 2 repositores, you can use this solution. I actually use it within my Jenkins pipeline, it works perfectly. Let's say you have a plugins-release-local and plugins-snapshot-local and you want to download the latest jar between these. Your shell script should look like this

NOTE : I use jfrog cli and it's configured with my Artifactory server.

Use case : Shell script

# your repo, you can change it then or pass an argument to the script
# repo = $1 this get the first arg passed to the script
repo=plugins-snapshot-local
# change this by your artifact path, or pass an argument $2
artifact=kshuttle/ifrs16
path=$repo/$artifact
echo $path
~/jfrog rt download --flat $path/maven-metadata.xml version/
version=$(cat version/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/")
echo "VERSION $version"
~/jfrog rt download --flat $path/$version/maven-metadata.xml build/
build=$(cat  build/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\([^<]*\)<\/value>.*/\1/")
echo "BUILD $build"
# change this by your app name, or pass an argument $3
name=ifrs16
jar=$name-$build.jar
url=$path/$version/$jar

# Download
echo $url
~/jfrog rt download --flat $url

Use case : Jenkins Pipeline

def getLatestArtifact(repo, pkg, appName, configDir){
    sh """
        ~/jfrog rt download --flat $repo/$pkg/maven-metadata.xml $configDir/version/
        version=\$(cat $configDir/version/maven-metadata.xml | grep latest | sed "s/.*<latest>\\([^<]*\\)<\\/latest>.*/\\1/")
        echo "VERSION \$version"
        ~/jfrog rt download --flat $repo/$pkg/\$version/maven-metadata.xml $configDir/build/
        build=\$(cat  $configDir/build/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\\([^<]*\\)<\\/value>.*/\\1/")
        echo "BUILD \$build"
        jar=$appName-\$build.jar
        url=$repo/$pkg/\$version/\$jar

        # Download
        echo \$url
        ~/jfrog rt download --flat \$url
    """
}

def clearDir(dir){
    sh """
        rm -rf $dir/*
    """

}

node('mynode'){
    stage('mysstage'){
        def repos =  ["plugins-snapshot-local","plugins-release-local"]

        for (String repo in repos) {
            getLatestArtifact(repo,"kshuttle/ifrs16","ifrs16","myConfigDir/")
        }
        //optional
        clearDir("myConfigDir/")
    }
}

This helps alot when you want to get the latest package between 1 or more repos. Hope it helps u too! For more Jenkins scripted pipelines info, visit Jenkins docs.

Upvotes: 1

Onko
Onko

Reputation: 21

For me the easiest way was to read the last versions of the project with a combination of curl, grep, sort and tail.

My format: service-(version: 1.9.23)-(buildnumber)156.tar.gz

versionToDownload=$(curl -u$user:$password 'https://$artifactory/artifactory/$project/' | grep -o 'service-[^"]*.tar.gz' | sort | tail -1)

Upvotes: 2

Kris
Kris

Reputation: 4823

You could also use Artifactory Query Language to get the latest artifact.

The following shell script is just an example. It uses 'items.find()' (which is available in the non-Pro version), e.g. items.find({ "repo": {"$eq":"my-repo"}, "name": {"$match" : "my-file*"}}) that searches for files that have a repository name equal to "my-repo" and match all files that start with "my-file". Then it uses the shell JSON parser ./jq to extract the latest file by sorting by the date field 'updated'. Finally it uses wget to download the artifact.

#!/bin/bash

# Artifactory settings
host="127.0.0.1"
username="downloader"
password="my-artifactory-token"

# Use Artifactory Query Language to get the latest scraper script (https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language)
resultAsJson=$(curl -u$username:"$password" -X POST  http://$host/artifactory/api/search/aql -H "content-type: text/plain" -d 'items.find({ "repo": {"$eq":"my-repo"}, "name": {"$match" : "my-file*"}})')

# Use ./jq to pars JSON
latestFile=$(echo $resultAsJson | jq -r '.results | sort_by(.updated) [-1].name')

# Download the latest scraper script
wget -N -P ./libs/ --user $username --password $password http://$host/artifactory/my-repo/$latestFile

Upvotes: 10

btiernay
btiernay

Reputation: 8129

Something like the following bash script will retrieve the lastest com.company:artifact snapshot from the snapshot repo:

# Artifactory location
server=http://artifactory.company.com/artifactory
repo=snapshot

# Maven artifact location
name=artifact
artifact=com/company/$name
path=$server/$repo/$artifact
version=$(curl -s $path/maven-metadata.xml | grep latest | sed "s/.*<latest>\([^<]*\)<\/latest>.*/\1/")
build=$(curl -s $path/$version/maven-metadata.xml | grep '<value>' | head -1 | sed "s/.*<value>\([^<]*\)<\/value>.*/\1/")
jar=$name-$build.jar
url=$path/$version/$jar

# Download
echo $url
wget -q -N $url

It feels a bit dirty, yes, but it gets the job done.

Upvotes: 80

Jason
Jason

Reputation: 3179

This may be new:

https://artifactory.example.com/artifactory/repo/com/example/foo/1.0.[RELEASE]/foo-1.0.[RELEASE].tgz

For loading module foo from example.com . Keep the [RELEASE] parts verbatim. This is mentioned in the docs but it's not made abundantly clear that you can actually put [RELEASE] into the URL (as opposed to a substitution pattern for the developer).

Upvotes: 2

Sateesh Potturu
Sateesh Potturu

Reputation: 329

Using shell/unix tools

  1. curl 'http://$artiserver/artifactory/api/storage/$repokey/$path/$version/?lastModified'

The above command responds with a JSON with two elements - "uri" and "lastModified"

  1. Fetching the link in the uri returns another JSON which has the "downloadUri" of the artifact.

  2. Fetch the link in the "downloadUri" and you have the latest artefact.

Using Jenkins Artifactory plugin

(Requires Pro) to resolve and download latest artifact, if Jenkins Artifactory plugin was used to publish to artifactory in another job:

  1. Select Generic Artifactory Integration
  2. Use Resolved Artifacts as ${repokey}:**/${component}*.jar;status=${STATUS}@${PUBLISH_BUILDJOB}#LATEST=>${targetDir}

Upvotes: 22

noamt
noamt

Reputation: 7815

Artifactory has a good extensive REST-API and almost anything that can be done in the UI (perhaps even more) can also be done using simple HTTP requests.

The feature that you mention - retrieving the latest artifact, does indeed require the Pro edition; but it can also be achieved with a bit of work on your side and a few basic scripts.

Option 1 - Search:

Perform a GAVC search on a set of group ID and artifact ID coordinates to retrieve all existing versions of that set; then you can use any version string comparison algorithm to determine the latest version.

Option 2 - the Maven way:

Artifactory generates a standard XML metadata that is to be consumed by Maven, because Maven is faced with the same problem - determining the latest version; The metadata lists all available versions of an artifact and is generated for every artifact level folder; with a simple GET request and some XML parsing, you can discover the latest version.

Upvotes: 37

djhaskin987
djhaskin987

Reputation: 10057

You can use the REST-API's "Item last modified". From the docs, it retuns something like this:

GET /api/storage/libs-release-local/org/acme?lastModified
{
"uri": "http://localhost:8081/artifactory/api/storage/libs-release-local/org/acme/foo/1.0-SNAPSHOT/foo-1.0-SNAPSHOT.pom",
"lastModified": ISO8601
}

Example:

# Figure out the URL of the last item modified in a given folder/repo combination
url=$(curl \
    -H 'X-JFrog-Art-Api: XXXXXXXXXXXXXXXXXXXX' \
    'http://<artifactory-base-url>/api/storage/<repo>/<folder>?lastModified'  | jq -r '.uri')
# Figure out the name of the downloaded file
downloaded_filename=$(echo "${url}" | sed -e 's|[^/]*/||g')
# Download the file
curl -L -O "${url}"

Upvotes: 6

spuder
spuder

Reputation: 18407

With recent versions of artifactory, you can query this through the api.

https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-RetrieveLatestArtifact

If you have a maven artifact with 2 snapshots

name => 'com.acme.derp'
version => 0.1.0
repo name => 'foo'
snapshot 1 => derp-0.1.0-20161121.183847-3.jar
snapshot 2 => derp-0.1.0-20161122.00000-0.jar

Then the full paths would be

https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-20161121.183847-3.jar

and

https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-20161122.00000-0.jar

You would fetch the latest like so:

curl https://artifactory.example.com/artifactory/foo/com/acme/derp/0.1.0-SNAPSHOT/derp-0.1.0-SNAPSHOT.jar

Upvotes: 5

Robert Longson
Robert Longson

Reputation: 123995

The role of Artifactory is to provide files for Maven (as well as other build tools such as Ivy, Gradle or sbt). You can just use Maven together with the maven-dependency-plugin to copy the artifacts out. Here's a pom outline to start you off...

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>A group id</groupId>
    <artifactId>An artifact id</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <id>copy</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>The group id of your artifact</groupId>
                                    <artifactId>The artifact id</artifactId>
                                    <version>The snapshot version</version>
                                    <type>Whatever the type is, for example, JAR</type>
                                    <outputDirectory>Where you want the file to go</outputDirectory>
                                </artifactItem>
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Just run mvn install to do the copy.

Upvotes: 3

ruhsuzbaykus
ruhsuzbaykus

Reputation: 13380

You can use the wget --user=USER --password=PASSWORD .. command, but before you can do that, you must allow artifactory to force authentication, which can be done by unchecking the "Hide Existence of Unauthorized Resources" box at Security/General tab in artifactory admin panel. Otherwise artifactory sends a 404 page and wget can not authenticate to artifactory.

Upvotes: 2

Related Questions