Recommended way to declare Datomic schema in Clojure application

Question

I'm starting to develop a Datomic-backed Clojure app, and I'm wondering what's the best way to declare the schema, in order to address the following concerns:

1. Having a concise, readable representation for the schema
2. Ensuring the schema is installed and up-to-date prior to running a new version of my app.

Intuitively, my approach would be the following:

1. Declaring some helper functions to make schema declarations less verbose than with the raw maps
2. Automatically installing the schema as part of the initialization of the app (I'm not yet knowledgeable enough to know if that always works).

Is this the best way to go? How do people usually do it?

Answer 1

I would suggest using Tupelo Datomic to get started. I wrote this library to simplify Datomic schema creation and ease understanding, much like you allude in your question.

As an example, suppose we’re trying to keep track of information for the world’s premiere spy agency. Let’s create a few attributes that will apply to our heroes & villains (see the executable code in the unit test).

  (:require [tupelo.datomic   :as td]
            [tupelo.schema    :as ts])

  ; Create some new attributes. Required args are the attribute name (an optionally namespaced
  ; keyword) and the attribute type (full listing at http://docs.datomic.com/schema.html). We wrap
  ; the new attribute definitions in a transaction and immediately commit them into the DB.
  (td/transact *conn* ;   required              required              zero-or-more
                      ;  <attr name>         <attr value type>       <optional specs ...>
    (td/new-attribute   :person/name         :db.type/string         :db.unique/value)      ; each name      is unique
    (td/new-attribute   :person/secret-id    :db.type/long           :db.unique/value)      ; each secret-id is unique
    (td/new-attribute   :weapon/type         :db.type/ref            :db.cardinality/many)  ; one may have many weapons
    (td/new-attribute   :location            :db.type/string)     ; all default values
    (td/new-attribute   :favorite-weapon     :db.type/keyword ))  ; all default values

For the :weapon/type attribute, we want to use an enumerated type since there are only a limited number of choices available to our antagonists:

  ; Create some "enum" values. These are degenerate entities that serve the same purpose as an
  ; enumerated value in Java (these entities will never have any attributes). Again, we
  ; wrap our new enum values in a transaction and commit them into the DB.
  (td/transact *conn*
    (td/new-enum :weapon/gun)
    (td/new-enum :weapon/knife)
    (td/new-enum :weapon/guile)
    (td/new-enum :weapon/wit))

Let’s create a few antagonists and load them into the DB. Note that we are just using plain Clojure values and literals here, and we don’t have to worry about any Datomic specific conversions.

  ; Create some antagonists and load them into the db.  We can specify some of the attribute-value
  ; pairs at the time of creation, and add others later. Note that whenever we are adding multiple
  ; values for an attribute in a single step (e.g. :weapon/type), we must wrap all of the values
  ; in a set. Note that the set implies there can never be duplicate weapons for any one person.
  ; As before, we immediately commit the new entities into the DB.
  (td/transact *conn*
    (td/new-entity { :person/name "James Bond" :location "London"     :weapon/type #{ :weapon/gun :weapon/wit   } } )
    (td/new-entity { :person/name "M"          :location "London"     :weapon/type #{ :weapon/gun :weapon/guile } } )
    (td/new-entity { :person/name "Dr No"      :location "Caribbean"  :weapon/type    :weapon/gun                 } ))

Enjoy! Alan

Answer 2

Proposal: using transaction functions to make declaring schema attributes less verbose in EDN, this preserving the benefits of declaring your schema in EDN as demonstrated by @Guillermo Winkler's answer.

Example:

;; defining helper function
[{:db/id #db/id[:db.part/user]
  :db/doc "Helper function for defining entity fields schema attributes in a concise way."
  :db/ident :utils/field
  :db/fn #db/fn {:lang :clojure
                 :require [datomic.api :as d]
                 :params [_ ident type doc opts]
                 :code [(cond-> {:db/cardinality :db.cardinality/one
                                 :db/fulltext true
                                 :db/index true
                                 :db.install/_attribute :db.part/db

                                 :db/id (d/tempid :db.part/db)
                                 :db/ident ident
                                 :db/valueType (condp get type
                                                 #{:db.type/string :string} :db.type/string
                                                 #{:db.type/boolean :boolean} :db.type/boolean
                                                 #{:db.type/long :long} :db.type/long
                                                 #{:db.type/bigint :bigint} :db.type/bigint
                                                 #{:db.type/float :float} :db.type/float
                                                 #{:db.type/double :double} :db.type/double
                                                 #{:db.type/bigdec :bigdec} :db.type/bigdec
                                                 #{:db.type/ref :ref} :db.type/ref
                                                 #{:db.type/instant :instant} :db.type/instant
                                                 #{:db.type/uuid :uuid} :db.type/uuid
                                                 #{:db.type/uri :uri} :db.type/uri
                                                 #{:db.type/bytes :bytes} :db.type/bytes
                                                 type)}
                                doc (assoc :db/doc doc)
                                opts (merge opts))]}}]

;; ... then (in a later transaction) using it to define application model attributes
[[:utils/field :person/name :string "A person's name" {:db/index true}]
 [:utils/field :person/age :long "A person's name" nil]]

Answer 3

1. Raw maps are verbose, but have some great advantages over using some high level api:

   • Schema is defined in transaction form, what you specify is transactable (assuming the word exists)
   • Your schema is not tied to a particular library or spec version, it will always work.
   • Your schema is serializable (edn) without calling a spec API.
   • So you can store and deploy your schema more easily in a distributed environment since it's in data-form and not in code-form.

For those reasons I use raw maps.

2. Automatically installing schema.

This I don't do either.

Usually when you make a change to your schema many things may be happening:

• Add new attribute
• Change existing attribute type
• Create full-text for an attribute
• Create new attribute from other values
• Others

Which may need for you to change your existing data in some non obvious and not generic way, in a process which may take some time.

I do use some automatization for applying a list of schemas and schema changes, but always in a controlled "deployment" stage when more things regarding data updating may occur.

Assuming you have users.schema.edn and roles.schema.edn files:

(require '[datomic-manage.core :as manager])
(manager/create uri)
(manager/migrate uri [:users.schema
                      :roles.schema])

Answer 4

My preference (and I'm biased, as the author of the library) lies with datomic-schema - It focusses on only doing the transformation to normal datomic schema - from there, you transact the schema as you would normally.

I am looking to use the same data to calculate schema migration between the live datomic instance and the definitions - so that the enums, types and cardinality gets changed to conform to your definition.

The important part (for me) of datomic-schema is that the exit path is very clean - If you find it doesn't support something (that I can't implement for whatever reason) down the line, you can dump your schema as plain edn, save it off and remove the dependency.

Conformity will be useful beyond that if you want to do some kind of data migration, or more specific migrations (cleaning up the data, or renaming to something else first).

Answer 5

For #1, datomic-schema might be of help. I haven't used it, but the example looks promising.

Answer 6

I Use Conformity for this see Conformity repository. There is also a very useful blogpost from Yeller Here which will guide you how to use Conformity.