Utilizing Library Dependencies in sbt plugin

Question

I'm working with Lift 3 and trying to get FlywayDb to work, in order to manage database migrations (via Slick). There is a Flyway sbt plugin for Slick, described here. Following those instructions, I have the following in my project/plugins.sbt file:

resolvers += "Flyway" at "https://flywaydb.org/repo"

addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "2.1.0")
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")
addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.6.1")
addSbtPlugin("org.flywaydb" % "flyway-sbt" % "4.0.3")

I've also added a flyway.sbt to my project root, but since I am not using TypeSafe's app configuration, but instead using Lift's Props, it looks like this:

import sbt._
import net.liftweb.util.Props

lazy val flywayDBName = "adb"

//Set up local setting key to load configuration into.
lazy val flywayDbConf = settingKey[(String, String, String)]("Lift config file with Slick settings")

flywayDbConf := {
  (Props.get("db.mysql.url"), Props.get("db.mysql.user"), Props.get("db.mysql.password"))
}

flywayUrl := flywayDbConf.value._1

flywayUser := flywayDbConf.value._2

flywayPassword := flywayDbConf.value._3

This, of course, doesn't work because net.liftweb.util.Props can't be imported - while my build.sbt includes the net.liftweb library dependencies that include net.liftweb.util.* (I know this because I'm using it to access properties in Boot.scala as well as other places), this file does not have access to those dependencies. When executing sbt from the command line:

path/to/project/flyway.sbt:2: error: not found: object net
import net.liftweb.util.Props
       ^
sbt.compiler.EvalException: Type error in expression

(Incidentally, my IDE also reports the library is not available.)

Clearly, I'm going about this the wrong way. It makes sense that library dependencies aren't available at build time, but the above resources indicate that there should be some way to achieve the same effect. What am I missing? How do I close the gap between the properties file access and the build script?

Note that I am attempting to do this so I can manage database changes in the project with FlywayDb, knowing that the underlying db may be either a development (MySQL), production (MySQL or maybe PostGres) or testing (h2) database. I want the code to be agnostic to these details, but all on the same page.

Answer 1

This is a partial solution: I haven't figured out yet how to get a complete solution, for reasons that only touch on this question.

There are a few important concepts that are not all well-spelled out in the documentation:

• In the root of your project you should have your build.sbt file. Also in this root is a project directory. Putting a build.sbt file in that directory is what is called the build definition for the meta-build. (As opposed to the proper build.) This is described here.
• If the proper build's build definition has a library dependency (which is what I was trying to get at without having the words for it), then the meta build needs to import the library.
• If the meta-build has as a libraryDependency some library, it is available to the proper build to use. Likewise, if you go a layer deeper, the meta-meta-build definition can import libraries for the meta-build definition to use.
• If the scalaVersion is set in the meta-build, then the proper-build definition will not function correctly. I'm not entirely clear why, but at the least it gets confused as to what libraries it should be downloading. Some of this is related to the fact that SBT 0.13.x is pinned to Scala 2.10, so any other version of Scala defined in scalaVersion will cause issues.
• Likewise, if you download libraries built against a version of Scala that is not 2.10, there will be issues.

I've only had limited time to explore this and determine complete solutions to my total problem, so my understanding is still vague. However, this is how you import a library.

In project/build.sbt:

// Example library
libraryDependencies += "net.liftweb" % "lift-webkit_2.10" % "2.6"

In build.sbt:

import net.liftweb.util.Props

val urlProperty = Props.get("db.mysql.url")

I'll update this when I get a more complete solution together.

Answer 2

Others may reply to the part about writing an sbt plugin, etc, but what we have been doing on production and has worked well for at least 3 years is to have the db migration code all inside the actual app, instead of being delegated to an sbt task that you manually run.

Granted I didn't write it (All credit to Tim Nelson) , but iirc part of it includes what Matt Farmer did on this project https://github.com/farmdawgnation/anchortab/blob/master/src/main/scala/bootstrap/liftweb/Boot.scala#L34

(which is great that he open sourced)

The way it works for us is:

1. Boot.scala we call a method on our Migration object, doMigrations()
2. doMigrations() gets the latest version of the running database (we have a table (we use Mongo so technically it's a collection, but it's the same for MySQl, let's call it a table from here on)) in this table, every time you run a migration, we add one row, with the description of the migration and then a numeric id, it's sequential, starting at 1.

3. Then we compare that to the latest migration number hard coded in our app, we have this on another object called MigrationVersion, a val called "latest" so then we do something like:

   (versionStoredOnDatabase+1 to latestVersionFromMigrationVersion).foreach { ver =>
     ver is the version so here we use reflection to find the migration 
     classes, run the,. save the migration id and description to the
     database so we don't run the migration twice, etc. I think this 
     is where you will want to call flydb
   }
   

so this will run every migration we have so that the database migration version matches the last one we added to our app.

And once the migration(s) are done, then Boot continues with the normal initialization and then our app is running again. No sbt needed, and it runs just fine locally in dev mode, on our staging, pilot and production servers.

In our case we have several web servers, so we run the migration on a dedicated server and then the rest of the app on the web servers continue to run, but I think you get the idea, if you need any more info on this way of handling migration, just let us know here.

Thanks

Diego