easy way to deploy 3rd party dependencies to a maven repository

Question

I work for a very tightly controlled organization, which absolutely will not allow access to outside repositories.

I would like to use Spring & Hibernate framework for our applications and use maven to resolve dependencies and build a project.

I've set up a nexus "3rd party" repository on a local server, now comes the part of actually adding libraries. Hibernate 3.6.7.Final comes with the following jars:

./hibernate-testing.jar
./hibernate3.jar
./lib/required/commons-collections-3.1.jar
./lib/required/dom4j-1.6.1.jar
./lib/required/jta-1.1.jar
./lib/required/slf4j-api-1.6.1.jar
./lib/required/antlr-2.7.6.jar
./lib/required/javassist-3.12.0.GA.jar
./lib/jpa/hibernate-jpa-2.0-api-1.0.1.Final.jar
./lib/optional/c3p0/c3p0-0.9.1.jar
./lib/optional/swarmcache/swarmcache-1.0RC2.jar
./lib/optional/proxool/proxool-0.8.3.jar
./lib/optional/jbosscache/jbosscache-core-3.2.1.GA.jar
./lib/optional/oscache/oscache-2.1.jar
./lib/optional/infinispan/infinispan-core-4.2.1.CR1.jar
./lib/bytecode/javassist/javassist-3.12.0.GA.jar
./lib/bytecode/cglib/cglib-2.2.jar

Spring 3.0.6.RELEASE comes with the following:

org.springframework.aop-3.0.6.RELEASE.jar                
org.springframework.jdbc-3.0.6.RELEASE.jar
org.springframework.asm-3.0.6.RELEASE.jar                
org.springframework.jms-3.0.6.RELEASE.jar
org.springframework.aspects-3.0.6.RELEASE.jar            
org.springframework.orm-3.0.6.RELEASE.jar
org.springframework.beans-3.0.6.RELEASE.jar              
org.springframework.oxm-3.0.6.RELEASE.jar
org.springframework.context-3.0.6.RELEASE.jar            
org.springframework.test-3.0.6.RELEASE.jar
org.springframework.context.support-3.0.6.RELEASE.jar    
org.springframework.transaction-3.0.6.RELEASE.jar
org.springframework.core-3.0.6.RELEASE.jar               
org.springframework.web-3.0.6.RELEASE.jar
org.springframework.expression-3.0.6.RELEASE.jar         
org.springframework.web.portlet-3.0.6.RELEASE.jar
org.springframework.instrument-3.0.6.RELEASE.jar         
org.springframework.web.servlet-3.0.6.RELEASE.jar
org.springframework.instrument.tomcat-3.0.6.RELEASE.jar  
org.springframework.web.struts-3.0.6.RELEASE.jar

I understand that the 2 preferred methods for deploying dependencies are

1) set up a pom; use command "mvn deploy"

2) use command-line like so, for individual jars:

 mvn deploy:deploy-file -Durl=file://C:\m2-repo \

                   -DrepositoryId=some.id \

                   -Dfile=your-artifact-1.0.jar \

                   [-DpomFile=your-pom.xml] \

                   [-DgroupId=org.some.group] \

                   [-DartifactId=your-artifact] \

                   [-Dversion=1.0] \

                   [-Dpackaging=jar] \

                   [-Dclassifier=test] \

                   [-DgeneratePom=true] \

                   [-DgeneratePom.description="My Project Description"] \

                   [-DrepositoryLayout=legacy] \

                   [-DuniqueVersion=false]

For 1) I understand I can either create a single pom for the entire hibernate project, or create multiple poms, I guess 1 for each external library. For example, cglib-2.2.jar can be its own pom, because I know spring has a similar dependency, so for the sake of not having (2)x cglib(s), I'd have 1 cglib and then include it as a dependency in my org.spring and org.hibernate poms, respectively.

For 2) I guess I could write a shell script to read the file list and parse out GAV info from maybe another list somewhere and dump them to repo 1 by 1, then each jar will be listed as dependency in my project's POM.

This last part is what I find confusing... If I deploy entire Spring release as 1 group - org.Spring and Hibernate in another org.Hibernate, could there be collisions between dependencies (ex. cglib)?

So I guess my question is:

What is the easiest, fastest, least effort required way to deploy downloaded Spring and Hibernate releases to my local nexus repository, so that I can use them in my project quickly? (Without having to list too many dependencies in too many poms.)

---

Edit:

A little clarification on what I want. I have 1 Nexus manager running on a local server (with access to the Internet). I want to be able to download from the Internet repositories all dependencies that I need to run Spring & Hibernate. I want to do this ONLY ONCE, then have them sit in my INTERNAL REPOSITORY (Nexus), which I want to NEVER EVER talk to central, jboss, spring or any other PUBLIC REPOSITORY until I SPECIFICALLY allow it. I want to be able to turn this on and off like a switch.

ON mode = "you can grab dependencies from public repos"

OFF mode = "you must not connect to anything external"

It seems to me like this is accomplish-able by doing the following:

in Nexus

1) Set up a group

2) Set up proxie for each external repository you need

3) Put all of your proxies in 1 group

on the dev workstation (this is part I'm not sure about)

1) edit settings.xml to override central location and point repository to local nexus group as described here:

maven repository mirrors

and here

Disable Maven central repository

and here

http://www.sonatype.com/books/nexus-book/reference/maven-sect-single-group.html

2) run mvn install or compile or whatever, so that it grabs the dependencies.

After that I'm not really sure how it's going to work. Somehow tell maven to mirror the external group in one of my local Nexus repositories? I guess I will try that and come back and post my solution if successful. In the meantime, feel free to comment or advise.

Answer 1

Nexus has a procurement feature to tightly control what comes in your repo from the outside while still using the original poms from the upstream. Without them you are loosing all transitive dependency information and will most likely break you build or runtime deployment in the future.

JFrog Artifactory has a similar feature (can define includes/exclude patterns) for free.

Answer 2

Figured it out.

Here's how it works. I'll start with Nexus set up.

We have 2 "groups" with the following repositories in them following :

1) Our Repo (nexus installation on our server)

-3rd party

-Releases

-Snapshots

2) External Proxies

-Maven central

-Spring

-Hibernate

-etc.

On the dev workstation I have the following in conf/settings.xml (main settings file for mvn): 2 groups, 2 mirrors, 2 profiles. Looks like so:

<mirrors>
<mirror>
  <!--This sends everything else to nexus repo -->
  <id>nexus</id>
  <mirrorOf>*</mirrorOf>
  <url>http://<ip>:8081/nexus/content/groups/nexus</url>
</mirror>
 <mirror>
  <!--This sends everything else to /public -->
  <id>external</id>
  <mirrorOf>*</mirrorOf>
  <url>http://<ip>:8081/nexus/content/groups/external</url>
</mirror>
</mirrors>

And:

 <profiles>
  <profile>
  <id>external</id>
  <!--Enable snapshots for the built in central repo to direct -->
  <!--all requests to nexus via the mirror -->
  <repositories>
    <repository>
      <id>central</id>
      <url>http://central</url>
      <releases><enabled>true</enabled></releases>
      <snapshots><enabled>true</enabled></snapshots>
    </repository>
  </repositories>
 <pluginRepositories>
    <pluginRepository>
      <id>central</id>
      <url>http://central</url>
      <releases><enabled>true</enabled></releases>
      <snapshots><enabled>true</enabled></snapshots>
    </pluginRepository>
  </pluginRepositories>
</profile>

<profile>
  <id>nexus</id>
  <!--Enable snapshots for the built in central repo to direct -->
  <!--all requests to nexus via the mirror -->
  <repositories>
    <repository>
      <id>central</id>
      <url>http://central</url>
      <releases><enabled>true</enabled></releases>
      <snapshots><enabled>true</enabled></snapshots>
    </repository>
  </repositories>
 <pluginRepositories>
    <pluginRepository>
      <id>central</id>
      <url>http://central</url>
      <releases><enabled>true</enabled></releases>
      <snapshots><enabled>true</enabled></snapshots>
    </pluginRepository>
  </pluginRepositories>
</profile>
</profiles>

And:

<activeProfiles>
<activeProfile>external</activeProfile>   
<activeProfile>nexus</activeProfile>
</activeProfiles>

The workflow goes like this.

Maven uses mirrors on a first-come-first-serve basis. That is, if you're mirroring *, whatever you list first (first mirror) is the only thing that it will check. So what I do is, I list nexus (local server) as first repo by default to mirror *. That way it will be forced to use only packages in local repositories. What happens if a package is missing? I tell maven to use "external" mirror. I edit

conf/settings.xml

and put external as first mirror. What this will do is tell maven to get packages from external repositories and cache them in storage on nexus machine. Once I have the packages and I see that the build process works, I do the following.

1) Take nexus offline.

2) in nexus installation, find this folder:

sonatype-work/nexus/storage/

This is where nexus caches packages for repositories. Note, it doesn't cache them per group, but per individual repository. This will be important in a second. Now that I've got packages I need from external proxies, I need to go through them, make sure it's all valid and move them to my "3rd parties" branch on nexus. I do the following (you can write a shell script to do this):

1) Bring nexus offline

2) for each external proxie

cp * from sonatype-work/nexus/storage/ to from sonatype-work/nexus/storage/thirdparty

3) Bring nexus online

4) In the GUI rightclick thirdparty repo and do:

a)Rebuild metadata

b)Repair Index

Done.

You now have external dependencies available locally and can thus control access to external artifacts. Probably good idea to delegate role of repository upkeep to a team lead or someone responsible. Because first time a project might require something from central and not have it, someone will have to go through this process.

---

Edit:

One thing to note on topic of plugins, I changed my the conf/settings.xml file to allow maven to grab plugins from central. So in both profiles do:

<pluginRepositories>
<pluginRepository>
  <id>central</id>
  <url>http://repo1.maven.org/maven2/</url>
  <releases><enabled>true</enabled></releases>
  <snapshots><enabled>true</enabled></snapshots>
</pluginRepository>
</pluginRepositories>

This is not really 100% optimal, as I don't have full control over what maven is doing here, but it seems like 90% of maven does depends on managing complicated plugin dependency relationships, so I figure I would let it manage that aspect of things on its own.

If anyone has an advice on how to better streamline this process, I'm all ears.

Answer 3

Not sure if this helps but we've been using the maven-proxy from codehaus for a while now. I'd suggest running the proxy, setting up your settings.xml file (see below), and doing your mvn eclipse:evlipse to download all of the dependencies through the cache. You could also fill the cache by hand. You can then tune the cache to specifically serve the packages that you want and nothing else.

If you get packages that you don't want then you'll need to add exclusions for them in your pom.xml files:

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring</artifactId>
    <version>2.0.6</version>
    <exclusions>
        <exclusion>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
        </exclusion>
        <exclusion>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
        </exclusion>
    </exclusions>
</dependency>

The proxy is a java application that is running on one of our dev boxes. In our settings.xml file we have something like the following:

<settings>
 <mirrors>
    <mirror>
      <id>maven-proxy</id>
      <name>Maven-Proxy Mirror</name>
      <url>http://mvnproxy.some-host-name.com:9999/repository</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
    <mirror>
      <id>maven-proxy</id>
      <name>Maven-Proxy Mirror</name>
      <url>http://mvnproxy.some-host-name.com:9999/repository</url>
      <mirrorOf>plugins</mirrorOf>
    </mirror>
  </mirrors>
  ...

Here's our start script which I don't think comes with the release:

#!/bin/sh
umask 022
cd /var/www/maven-proxy
java -jar maven-proxy-standalone-0.2-app.jar maven-proxy.prop >| maven-proxy.log 2>&1 &
echo $! > mp.pid

Answer 4

I'd reuse existing pom's for these projects in order to create effectively a mirror of the public repositories.