How to populate a composite attribute based tuple after adding it to an existing Datomic database with existing entities

Question

I have an existing Datomic database with existing entities with attributes of the form :user/first and :user/last. Now I wish to add the tuple with a unique constraint combining those attributes and call it :user/first+last. The problem I'm having is that the existing entities don't populate the tuple and Datomic permits adding new entities with the same :user/first and :user/last (since the tuple is not yet populated).

My question is: is there a way to have Datomic populate this tuple for me? Or do I have to go through the existing entities and populate it myself?

Here's some code to clearly illustrate my question.

(ns test-tuple
  (:require [processdb.ion.db :as db]
            [datomic.client.api :as d]))

(def schema
  [{:db/ident :user/first
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one}
   {:db/ident :user/last
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one}])

(def user-data
  [{:user/first "Joe"
    :user/last "Smith"}
   {:user/first "Jane"
    :user/last "Doe"}])

(def tuple-schema
  [{:db/ident :user/first+last
    :db/valueType :db.type/tuple
    :db/tupleAttrs [:user/first :user/last]
    :db/cardinality :db.cardinality/one}])

(defn load-schema []
  (d/transact (db/get-connection) {:tx-data schema}))

(defn load-tuple-schema []
  (d/transact (db/get-connection) {:tx-data tuple-schema}))

(defn load-data []
  (d/transact (db/get-connection) {:tx-data user-data}))

(defn query-names []
  (d/q '[:find (pull ?e [:user/first :user/last :user/first+last]) :in $ :where [?e :user/first]]
       (db/get-db)))

If I execute the following on the REPL, I'll have multiple entities with the names "Joe Smith" and "Jane Doe".

> (require '[test-tuple :as tt] :reload)
> (tt/load-schema)
> (tt/load-data)
> (tt/load-tuple-schema)
> (tt/load-data)

The query-names fn will then show the duplicates with one entity with the :user/first+last attribute and one without.

> (tt/query-names)
[[#:user{:first "Joe", :last "Smith"}] [#:user{:first "Jane", :last "Doe"}] [#:user{:first "Joe", :last "Smith", :first+last ["Joe" "Smith"]}] [#:user{:first "Jane", :last "Doe", :first+last ["Jane" "Doe"]}]]

Note how the tuple is automatically populated for newly created entities, which I appreciate.

If I have to populate the tuple manually, is there a straight forward query/transaction to do so?

Answer 1

If I understand correctly, you'd like to be able to have Datomic populate at this point:

> (require '[test-tuple :as tt] :reload)
> (tt/load-schema)
> (tt/load-data)
> (tt/load-tuple-schema) ;; populate after executing this

The documentation references how to do that here: https://docs.datomic.com/pro/schema/schema.html#adding-composite-existing

See below, but note that I use datomic.api with an in-memory store rather than datomic.cloud.api

(ns test-tuple
  (:require  [datomic.api :as d]))

(def db-uri "datomic:mem://hello")
(d/create-database db-uri)

(def conn (d/connect db-uri))

(def db (d/db conn))

(def schema
  [{:db/ident :user/first
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one}
   {:db/ident :user/last
    :db/valueType :db.type/string
    :db/cardinality :db.cardinality/one}])

(def user-data
  [{:user/first "Joe"
    :user/last "Smith"}
   {:user/first "Jane"
    :user/last "Doe"}])

(def tuple-schema
  [{:db/ident :user/first+last
    :db/valueType :db.type/tuple
    :db/tupleAttrs [:user/first :user/last]
    :db/cardinality :db.cardinality/one}])

(defn load-schema []
  (d/transact conn schema))

(defn load-tuple-schema []
  (d/transact conn tuple-schema))

(defn load-data []
  (d/transact conn  user-data))

(defn query-names []
  (d/q '[:find (pull ?e [:user/first :user/last :user/first+last]) :in $ :where [?e :user/first]]
       db))

(load-schema)
(load-data)
(load-tuple-schema)
(query-names) ;; [[#:user{:first "Joe", :last "Smith"}] [#:user{:first "Jane", :last "Doe"}]]

(defn establish-composite
  "Reasserts all values of attr, in batches of batch-size, with
  pacing-sec pause between transactions. This will establish values
  for any composite attributes built from attr."
  [conn {:keys [attr batch-size pacing-sec]}]
  (let [db (d/db conn)
        es (d/datoms db :aevt attr)]
    (doseq [batch (partition-all batch-size es)]
      (let [es (into #{} (map :e batch))
            result @(d/transact conn (map (fn [{:keys [e v]}]
                                            [:db/add e attr v])
                                          batch))
            added (transduce
                   (comp (map :e) (filter es))
                   (completing (fn [x ids] (inc x)))
                   0
                   (:tx-data result))]
        (println {:batch-size batch-size :first-e (:e (first batch)) :added added})
        (Thread/sleep (* 1000 pacing-sec))))))



(def composite-attributes {:attr :user/first
                           :batch-size 1000
                           :pacing-sec 1})

(establish-composite conn composite-attributes) ;; {:batch-size 1000, :first-e 17592186045418, :added 2}

(query-names) ;; [[#:user{:first "Joe", :last "Smith", :first+last ["Joe" "Smith"]}] [#:user{:
              ;; first "Jane", :last "Doe", :first+last ["Jane" "Doe"]}]]

If you were in the situation where you had duplicates, like here:

> (require '[test-tuple :as tt] :reload)
> (tt/load-schema)
> (tt/load-data)
> (tt/load-tuple-schema)
> (tt/load-data)

you could first back out all constituents of the composite, which would back out the composite, then use the solution the above.