How to do a sbt-(local)multi-project without the same project-root?

Question

I want to work on multiple sbt-projects at the same time in intellij. The projects are "one way dependent", meaning that one is the (reusable) core and the other is an actual application built upon that core. Both are currently in development.

I want the core to reside in another base/root-directory than the application, so:

- /core
-- build.sbt

- /application
-- build.sbt

I want to be able to

1. modify both projects in the same IntelliJ-window
2. leave both projects in their respective folder (no wrapper-folder around them!). Core will be used in other applications as well, applications that are no siblings to the current "application", so I do not want them to reside under the same root-folder!

What I've tried and which problems I've found so far:

setups like

lazy val core = project.in(file("../core"))
lazy val application = project.in(file(".")).dependsOn(core)

are not working, because an sbt asserts that the directory of each project in a multi-project-setup are contained in the same build root:

sbt java.lang.AssertionError: assertion failed: Directory /core is not contained in build root /application

setups like

lazy val core = RootProject(file("../core"))
lazy val application = project.in(file(".")).dependsOn(core)

are not a solution because:

1. I can not have both projects in one IntelliJ-window then
2. Strangely the classes of core are not found in application, although the imports worked right away

Now I am a sbt-newbie and I guess (and hope) that there must be a solution for this problem. I can't be the only one wanting to separate my projects without a wrapper-layer and still be able to modify them in the IDE of my choice.

edit:

@OlegRudenko´s solution is semi-working for me. As core has some dependencies as well, I cannot compile or work with it in application.

core pulls in some dependencies like e.g. a Logger and when I'm in application and try to use a component of core the compiler screams at me, because it can't find the dependencies in core (e.g. the logger).

Also, core pulls in e.g. lwjgl and I want to use some components of it in application, no chance, because it can't find the packages for that dependency of core.

For now what I do is a hacky non-solution. I just develop both core and application in the same intellij-project and keep the git-repo private.

This is not a solution at all, as I want to open source core while application is closed source for now and I still want to work on both at the same time, refine the core etc.

Answer 1

You could use ProjectRef:

lazy val my_application = (project in file ("."))
    .aggregate(my_core)
    .dependsOn(my_core)

lazy val my_core = ProjectRef(file("../my_core"), "my_core")

Then, regarding the directory structure, my_application and my_core are siblings.

Answer 2

If some framework does not allow me something, I prefer to not fight but to find a compromise.

SBT does not allow to have a sub-project outside of project root. So my solution would be to create a symbolic link inside my project to the sub-project, which is located outside:

common/
common/core/  - I want this as a sub-project inside of my project
application/
application/ui/  - This is my project

application/ui/core/ -> ../../common/core/  - This is my goal

It has to

• work automatically
• work on different operating systems
• be clear and understandable
• use standard tools

Solution is to add following lines into the SBT file:

val __createLinkToCore = {
  import java.nio.file.{FileSystemException, Files, Paths}
  import scala.util.Try

  val corePath = Paths.get("..", "..", "common", "core")
  if(!Files.exists(corePath)) throw new IllegalStateException("Target path does not exist")

  val linkPath = Paths.get("core")
  if(Files.exists(linkPath)) Files.delete(linkPath)

  Try {Files.createSymbolicLink(linkPath, corePath)}.recover {
    case cause: FileSystemException if System.getProperty("os.name").toLowerCase.contains("windows") =>
      val junctionCommand = Array("cmd", "/c", "mklink", "/J", linkPath.toAbsolutePath.normalize().toString, corePath.toAbsolutePath.normalize().toString)
      val junctionResult = new java.lang.ProcessBuilder(junctionCommand: _*).inheritIO().start().waitFor()
      if(junctionResult != 0) throw new Exception("mklink failed", cause)
      linkPath
  }.get
}

It is executed every time when SBT is started but before any action on your code - before compilation and so on. It does NOT recreate the link on every compilation - only once per SBT start.

The code does following:

• get path to desired folder (and fail if it doesn't exist)
• get path to link (and delete if it exists)
• try to create a new link using Java 7+ API
• if it runs on Windows 7+ and without admin right it will fail. In this case I create a junction using standard but windows-specific tool mklink /J

Limits of this solution:

• it will not work on FAT file systems
• have you seen other limits? let me know!

Answer 3

I don't think this is possible in SBT (but I'd love to hear otherwise).

Here's what I'm doing in IntelliJ to work around this issue:

• Open "application" in IntelliJ; refresh the SBT definition
• Turn off SBT auto-update (file -> settings -> build -> SBT)
• Do file -> new -> module from existing sources, choose the "core" checkout dir, import it from SBT
  • Tick “download sources”, but not “auto-update”
• You should now have both “application” and “core” in the same IntelliJ window (but "application" still depends on a "core" JAR not your live sources)
• Right click on the "application" project in IntelliJ and choose “open module settings”
  • Open the “dependencies” tab
  • Find the "core" SBT dependency(-ies) in the libraries and delete them
  • Add a "module dependency" on the "core" project
• You should manually rebuild the project at this point, as IntelliJ has been known to get confused about classpaths when adding in modules.
• You can now develop with the two projects as a single codebase in IntelliJ
  • Note that SBT command line tools like “sbt test” in "application" will continue to use the "core" JAR and not the locally modified sources
  • (The SBT Build.scala file in "application" still references the "core" JAR by version number)
• When you are ready to commit, you will need to:
  • Commit your changes to "core", update the version number
  • Run "sbt publishLocal" in "core", make a note of the version numbers generated
  • Update the Build.scala file in "application" to reference the new "core" version
  • Run "sbt test" or similar to check that everything works in SBT mode
  • Commit the "application" changes, push both