How to use the Heroku DATABASE_URL environment variable in Scala?

Question

I am trying to deploy my app to Heroku. Heroku provides a DATABASE_URL environment variable. I am wondering what is the best way to use this.

When in dev the DATABASE_URL environment variable will be jdbc:postgresql://localhost:5400/bookswapdb

When in prd the DATABASE_URL environment variable will be something like this: postgres://<USER>:<PASSWORD>@<HOST>:<PORT>/<DATABASE>

1. Is checking the DATABASE_URL really the best way to discover whether the app is in prod or dev? (I would have thought they would also pass in a variable like ENV=prd or PRD=true)

2. Given the code below, what is the smartest way to achieve checking the DATABASE_URL and passing it into initFlyway function.

3. What's the cleanest way to chop up this string to get the details i need postgres://<USER>:<PASSWORD>@<HOST>:<PORT>/<DATABASE>

4. The DATABASE_URL might be in this format: jdbc:postgresql://<host>:<port>/<dbname>?user=<username>&password=<password> if so how is the best way to divide that?

The below doesn't seem to work when deployed to heroku but it does work on a local db.

package com.fullstackryan.appone.server

import cats.effect.{ConcurrentEffect, ContextShift, Sync, Timer}
import cats.implicits._
import com.fullstackryan.appone.config.{Config, DbConfig, LoadConfig, ServerConfig}
import com.fullstackryan.appone.database.Database
import com.fullstackryan.appone.repo.{BookSwap, HelloWorld, Jokes}
import com.fullstackryan.appone.routing.ApponeRoutes
import fs2.Stream
import org.flywaydb.core.Flyway
import org.http4s.client.blaze.BlazeClientBuilder
import org.http4s.implicits._
import org.http4s.server.blaze.BlazeServerBuilder
import org.http4s.server.middleware.Logger
import pureconfig.generic.auto._

import java.net.URI
import scala.concurrent.ExecutionContext.global


object ApponeServer {

  def initFlyway[F[_] : Sync](url: String, username: String, password: String): F[Int] = Sync[F].delay {
    val flyway = Flyway.configure().dataSource(url, username, password).baselineOnMigrate(true).load()
    println("inside flyway")
    flyway.migrate()
  }

  def prodConfig(): Config = {
        val dbUri = new URI(System.getenv("DATABASE_URL"))
        val username = dbUri.getUserInfo.split(":")(0)
        val password = dbUri.getUserInfo.split(":")(1)
        val dbUrl = "jdbc:postgresql://" + dbUri.getHost + dbUri.getPath

        Config(ServerConfig(5432, dbUri.getHost), DbConfig(dbUrl, username, password, 10))
  }

  def stream[F[_] : ConcurrentEffect : ContextShift : Timer]: Stream[F, Nothing] = {
    for {
      client <- BlazeClientBuilder[F](global).stream
      // below line loads config from application.conf
      config <- Stream.eval(LoadConfig[F, Config].load)
      // This is meant to check if DATABASE_URL is dev or prd
      isProdConfig = if (config.dbConfig.url.contains("localhost")) config else prodConfig()
      // Below line hopefully passes correct prd or dev config into initFlyway to get a connnection 
      _ <- Stream.eval(initFlyway(isProdConfig.dbConfig.url, isProdConfig.dbConfig.username, isProdConfig.dbConfig.password))
      xa <- Stream.resource(Database.transactor(isProdConfig.dbConfig))
      helloWorldAlg = HelloWorld.impl[F]
      jokeAlg = Jokes.impl[F](client)
      bookAlg = BookSwap.buildInstance[F](xa)
      httpApp = (
        ApponeRoutes.helloWorldRoutes[F](helloWorldAlg) <+>
          ApponeRoutes.bookRoutes[F](bookAlg) <+>
          ApponeRoutes.jokeRoutes[F](jokeAlg)
        ).orNotFound

      finalHttpApp = Logger.httpApp(true, true)(httpApp)

      exitCode <- BlazeServerBuilder[F](global)
        .bindHttp(8080, "0.0.0.0")
        .withHttpApp(finalHttpApp)
        .serve
    } yield exitCode
  }.drain
}

ERROR

2021-01-17T14:05:29.437876+00:00 heroku[web.1]: State changed from crashed to starting
2021-01-17T14:05:35.808791+00:00 heroku[web.1]: Starting process with command `target/universal/stage/bin/appone -Dhttp.port=${PORT}`
2021-01-17T14:05:39.711887+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2021-01-17T14:05:40.047036+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8
2021-01-17T14:05:48.838694+00:00 app[web.1]: [ioapp-compute-0] INFO o.h.c.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
2021-01-17T14:05:48.972995+00:00 app[web.1]: pureconfig.error.ConfigReaderException: Cannot convert configuration to a scala.runtime.Nothing$. Failures are:
2021-01-17T14:05:48.973019+00:00 app[web.1]: at 'appone.db-config':
2021-01-17T14:05:48.973021+00:00 app[web.1]: - (application.conf @ jar:file:/app/target/universal/stage/lib/com.fullstackryan.appone-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'username'.
2021-01-17T14:05:48.973022+00:00 app[web.1]: - (application.conf @ jar:file:/app/target/universal/stage/lib/com.fullstackryan.appone-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'password'.
2021-01-17T14:05:48.977706+00:00 app[web.1]:
2021-01-17T14:05:48.977988+00:00 app[web.1]: at com.fullstackryan.appone.config.LoadConfig$$anon$1.$anonfun$load$1(LoadConfig.scala:25)
2021-01-17T14:05:48.978182+00:00 app[web.1]: at cats.syntax.EitherOps$.leftMap$extension(either.scala:172)
2021-01-17T14:05:48.992062+00:00 app[web.1]: at com.fullstackryan.appone.config.LoadConfig$$anon$1.load(LoadConfig.scala:25)
2021-01-17T14:05:48.992224+00:00 app[web.1]: at com.fullstackryan.appone.server.ApponeServer$.$anonfun$stream$1(ApponeServer.scala:50)
2021-01-17T14:05:48.992352+00:00 app[web.1]: at com.fullstackryan.appone.server.ApponeServer$.$anonfun$stream$1$adapted(ApponeServer.scala:48)
2021-01-17T14:05:48.992496+00:00 app[web.1]: at fs2.Stream$.$anonfun$flatMap$1(Stream.scala:1188)
2021-01-17T14:05:48.992649+00:00 app[web.1]: at fs2.internal.FreeC$.go$2(Algebra.scala:609)
2021-01-17T14:05:48.992861+00:00 app[web.1]: at fs2.internal.FreeC$.$anonfun$flatMapOutput$1(Algebra.scala:616)
2021-01-17T14:05:48.993129+00:00 app[web.1]: at fs2.internal.FreeC$$anon$1.cont(Algebra.scala:53)
2021-01-17T14:05:48.996922+00:00 app[web.1]: at fs2.internal.FreeC$ViewL$$anon$9$$anon$10.cont(Algebra.scala:242)
2021-01-17T14:05:48.997120+00:00 app[web.1]: at fs2.internal.FreeC$ViewL$.mk(Algebra.scala:231)
2021-01-17T14:05:48.997247+00:00 app[web.1]: at fs2.internal.FreeC$ViewL$.apply(Algebra.scala:220)
2021-01-17T14:05:48.997395+00:00 app[web.1]: at fs2.internal.FreeC.viewL(Algebra.scala:106)
2021-01-17T14:05:48.997537+00:00 app[web.1]: at fs2.internal.FreeC$.go$1(Algebra.scala:414)
2021-01-17T14:05:48.997707+00:00 app[web.1]: at fs2.internal.FreeC$.$anonfun$compile$8(Algebra.scala:464)
2021-01-17T14:05:48.998481+00:00 app[web.1]: at fs2.internal.FreeC$.$anonfun$compile$1(Algebra.scala:430)
2021-01-17T14:05:48.998648+00:00 app[web.1]: at map @ fs2.internal.CompileScope.interruptibleEval(CompileScope.scala:393)
2021-01-17T14:05:48.998778+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:490)
2021-01-17T14:05:48.998907+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.$anonfun$compile$5(Algebra.scala:450)
2021-01-17T14:05:48.999061+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:447)
2021-01-17T14:05:48.999202+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:05:48.999348+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:05:49.000021+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
2021-01-17T14:05:49.000182+00:00 app[web.1]: at flatten @ fs2.internal.ScopedResource$$anon$1.acquired(ScopedResource.scala:139)
2021-01-17T14:05:49.000285+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$1(CompileScope.scala:183)
2021-01-17T14:05:49.000409+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.acquireResource(CompileScope.scala:180)
2021-01-17T14:05:49.000547+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.$anonfun$compile$10(Algebra.scala:498)
2021-01-17T14:05:49.000665+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:05:49.000782+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:05:49.000911+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:05:49.001062+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
2021-01-17T14:05:49.001165+00:00 app[web.1]: at flatten @ fs2.internal.ScopedResource$$anon$1.acquired(ScopedResource.scala:139)
2021-01-17T14:05:49.171354+00:00 heroku[web.1]: Process exited with status 1
2021-01-17T14:05:49.208726+00:00 heroku[web.1]: State changed from starting to crashed
2021-01-17T14:05:49.211138+00:00 heroku[web.1]: State changed from crashed to starting
2021-01-17T14:05:55.214070+00:00 heroku[web.1]: Starting process with command `target/universal/stage/bin/appone -Dhttp.port=${PORT}`
2021-01-17T14:05:58.596071+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2021-01-17T14:05:58.994577+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8
2021-01-17T14:05:59.000000+00:00 app[api]: Build succeeded
2021-01-17T14:06:04.235862+00:00 app[web.1]: [ioapp-compute-0] INFO o.h.c.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
2021-01-17T14:06:04.319171+00:00 app[web.1]: pureconfig.error.ConfigReaderException: Cannot convert configuration to a scala.runtime.Nothing$. Failures are:
2021-01-17T14:06:04.319174+00:00 app[web.1]: at 'appone.db-config':
2021-01-17T14:06:04.319195+00:00 app[web.1]: - (application.conf @ jar:file:/app/target/universal/stage/lib/com.fullstackryan.appone-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'username'.
2021-01-17T14:06:04.319196+00:00 app[web.1]: - (application.conf @ jar:file:/app/target/universal/stage/lib/com.fullstackryan.appone-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'password'.
2021-01-17T14:06:04.319206+00:00 app[web.1]:
2021-01-17T14:06:04.319341+00:00 app[web.1]: at com.fullstackryan.appone.config.LoadConfig$$anon$1.$anonfun$load$1(LoadConfig.scala:25)
2021-01-17T14:06:04.319403+00:00 app[web.1]: at cats.syntax.EitherOps$.leftMap$extension(either.scala:172)
2021-01-17T14:06:04.319496+00:00 app[web.1]: at com.fullstackryan.appone.config.LoadConfig$$anon$1.load(LoadConfig.scala:25)
2021-01-17T14:06:04.319683+00:00 app[web.1]: at com.fullstackryan.appone.server.ApponeServer$.$anonfun$stream$1(ApponeServer.scala:50)
2021-01-17T14:06:04.319688+00:00 app[web.1]: at com.fullstackryan.appone.server.ApponeServer$.$anonfun$stream$1$adapted(ApponeServer.scala:48)
2021-01-17T14:06:04.319785+00:00 app[web.1]: at fs2.Stream$.$anonfun$flatMap$1(Stream.scala:1188)
2021-01-17T14:06:04.319847+00:00 app[web.1]: at fs2.internal.FreeC$.go$2(Algebra.scala:609)
2021-01-17T14:06:04.319951+00:00 app[web.1]: at fs2.internal.FreeC$.$anonfun$flatMapOutput$1(Algebra.scala:616)
2021-01-17T14:06:04.320042+00:00 app[web.1]: at fs2.internal.FreeC$$anon$1.cont(Algebra.scala:53)
2021-01-17T14:06:04.320188+00:00 app[web.1]: at fs2.internal.FreeC$ViewL$$anon$9$$anon$10.cont(Algebra.scala:242)
2021-01-17T14:06:04.320257+00:00 app[web.1]: at fs2.internal.FreeC$ViewL$.mk(Algebra.scala:231)
2021-01-17T14:06:04.320336+00:00 app[web.1]: at fs2.internal.FreeC$ViewL$.apply(Algebra.scala:220)
2021-01-17T14:06:04.320405+00:00 app[web.1]: at fs2.internal.FreeC.viewL(Algebra.scala:106)
2021-01-17T14:06:04.320481+00:00 app[web.1]: at fs2.internal.FreeC$.go$1(Algebra.scala:414)
2021-01-17T14:06:04.320561+00:00 app[web.1]: at fs2.internal.FreeC$.$anonfun$compile$8(Algebra.scala:464)
2021-01-17T14:06:04.320641+00:00 app[web.1]: at fs2.internal.FreeC$.$anonfun$compile$1(Algebra.scala:430)
2021-01-17T14:06:04.320719+00:00 app[web.1]: at map @ fs2.internal.CompileScope.interruptibleEval(CompileScope.scala:393)
2021-01-17T14:06:04.320785+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:490)
2021-01-17T14:06:04.320862+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.$anonfun$compile$5(Algebra.scala:450)
2021-01-17T14:06:04.320929+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.go$1(Algebra.scala:447)
2021-01-17T14:06:04.320995+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:06:04.321076+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:06:04.321138+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
2021-01-17T14:06:04.321230+00:00 app[web.1]: at flatten @ fs2.internal.ScopedResource$$anon$1.acquired(ScopedResource.scala:139)
2021-01-17T14:06:04.321289+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$1(CompileScope.scala:183)
2021-01-17T14:06:04.321385+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.acquireResource(CompileScope.scala:180)
2021-01-17T14:06:04.321473+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.$anonfun$compile$10(Algebra.scala:498)
2021-01-17T14:06:04.321531+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:06:04.321607+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:06:04.321668+00:00 app[web.1]: at flatMap @ fs2.internal.FreeC$.interruptGuard$1(Algebra.scala:429)
2021-01-17T14:06:04.321779+00:00 app[web.1]: at flatMap @ fs2.internal.CompileScope.$anonfun$acquireResource$4(CompileScope.scala:185)
2021-01-17T14:06:04.321811+00:00 app[web.1]: at flatten @ fs2.internal.ScopedResource$$anon$1.acquired(ScopedResource.scala:139)
2021-01-17T14:06:04.453054+00:00 heroku[web.1]: Process exited with status 1
2021-01-17T14:06:04.497636+00:00 heroku[web.1]: State changed from starting to crashed
2021-01-17T14:28:36.868097+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=appone2021.herokuapp.com request_id=a2e9c2cb-9e29-4e90-a528-2b93822a5b22 fwd="2.221.116.154" dyno= connect= service= status=503 bytes= protocol=https
2021-01-17T14:28:37.225963+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=appone2021.herokuapp.com request_id=d1e00110-d2b1-4512-9391-2ab54ac11947 fwd="2.221.116.154" dyno= connect= service= status=503 bytes= protocol=https
2021-01-17T14:42:04.309046+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=appone2021.herokuapp.com request_id=b6446286-f99c-4490-9ed4-ed8071f2d5c1 fwd="2.221.116.154" dyno= connect= service= status=503 bytes= protocol=https
2021-01-17T14:42:04.481028+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=appone2021.herokuapp.com request_id=c6bff236-696b-491d-9831-930f1d114fd8 fwd="2.221.116.154" dyno= connect= service= status=503 bytes= protocol=https
2021-01-17T14:45:18.000000+00:00 app[api]: Build started by user fullstackryan@gmail.com
2021-01-17T14:46:57.165052+00:00 app[api]: Release v25 created by user fullstackryan@gmail.com
2021-01-17T14:46:57.165052+00:00 app[api]: Deploy 5ab4cf89 by user fullstackryan@gmail.com
2021-01-17T14:46:58.334991+00:00 heroku[web.1]: State changed from crashed to starting
2021-01-17T14:47:02.441681+00:00 heroku[web.1]: Starting process with command `target/universal/stage/bin/appone -Dhttp.port=${PORT}`
2021-01-17T14:47:04.351208+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2021-01-17T14:47:04.458310+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8

Answer 1

I recommend using JDBC_DATABASE_URL instead. It will be automatically set for your app: https://devcenter.heroku.com/articles/connecting-to-relational-databases-on-heroku-with-java#using-the-database_url-in-plain-jdbc

Answer 2

The line below

   config <- Stream.eval(LoadConfig[F, Config].load)

was reading my application.conf file which in dev mode is fine as I source the variables mentioned within the application.conf by running source meta/dev.env. meta/dev.env being the location where I store the environment variables mentioned in the application.conf.

However in production (specifically Heroku environment) I could not source meta/prd.env because I could not store variables statically in that location/file as Heroku updates its variables every so often, meaning those variables would become out of date.

With the above in mind, I had to code in such as way that every time the application is run, my apps gets the DATABASE_URL variable injected by Heroku. I created a function for this:

  def prodConfig(): Config = {
    val dbUri = new URI(System.getenv("DATABASE_URL"))
    val username = dbUri.getUserInfo.split(":")(0)
    val password = dbUri.getUserInfo.split(":")(1)
    val dbUrl = "jdbc:postgresql://" + dbUri.getHost + dbUri.getPath

    Config(ServerConfig(5432, dbUri.getHost), DbConfig(dbUrl, username, password, 10))
  }

The above code gets DATABASE_URL by using System.getenv("DATABASE_URL"). Then the function splits it into various parts such as username, password etc which I can then pass database config etc.

When in dev I uncomment the first line below and comment out the second line below.

//    config <- Stream.eval(LoadConfig[F, Config].load)
      conifg = prodConfig()

In dev it reads from application.conf and in prod it will do the System.getenv("DATABASE_URL") which rely on heroku to make sure its provided.

It's not the nicest solution as it relies on commenting and uncommenting when in dev mode but for now it's the best I've got. Would love to hear recommendations on how to make this better.

Hope this helps others who are stuck.

Answer 3

You seem to have most of the pieces, but I don't think you're putting them together correctly.

Heroku Postgres sets the DATABASE_URL environment variable for you:

As part of the provisioning process, a DATABASE_URL config var is added to your app’s configuration. This contains the URL your app uses to access the database.

You appear to already be using this environment variable in your application.conf:

db-config {
    driver = "org.postgresql.Driver"
    url = ${?DATABASE_URL}
    username = ${?DATABASE_USERNAME}
    password = ${?DATABASE_PASSWORD}
    connection-threads = 4
    pool-size = 10
}

The problem is that you are also depending on environment variables called DATABASE_USERNAME and DATABASE_PASSWORD, which Heroku does not provide. That's what is causing your application to fail:

pureconfig.error.ConfigReaderException: Cannot convert configuration to a scala.runtime.Nothing$. Failures are:
at 'appone.db-config':
- (application.conf @ jar:file:/app/target/universal/stage/lib/com.fullstackryan.appone-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'username'.
- (application.conf @ jar:file:/app/target/universal/stage/lib/com.fullstackryan.appone-0.0.1-SNAPSHOT.jar!/application.conf: 10) Key not found: 'password'.

You could try setting them with heroku config:set, but that's not a good idea because

The value of your app’s DATABASE_URL config var might change at any time. You should not rely on this value either inside or outside your Heroku app.

I suggest only setting the url in your application.conf. Then, in your application code, you can parse the URL and connect to your database as you are already trying to do. Since it is a URL, your current approach of instantiating a URI is a good fit.

Side note: Your application code is currently getting DATABASE_URL directly from the environment again. I suspect it would be more idiomatic to retrieve it from whatever configuration object you get from your application.conf, where you have set db-config.url. But I'm not experienced enough in Scala to show the correct approach.