CODEWITHSUNDEEP
sqljsonpostgresqllispcommon-lisp
deadghost
deadghost

Reputation: 5227

insert json data into postgresql db with postmodern

I want to insert lots of JSON data into my db.

[{
    "term": "wine",
    "name": "Bubba Wine & Spirits",
    "address": "1234 N San Fake Rd,"
}, {
    "term": "wine",
    "name": "Wine Shop",
    "address": "123 N Not Real Blvd,"
}]

I use cl-json to convert to lisp objects.

(defvar *data*
  (decode-json (open "my-json-file.json")))

Results look like:

(((:TERM . "wine") (:NAME . "Bubba Wine & Spirits")
  (:ADDRESS . "1234 N San Fake Rd,"))
 ((:TERM . "wine") (:NAME . "Wine Shop")
  (:ADDRESS . "123 N Not Real Blvd,")))

Postmodern lists one way to insert multiple rows with insert-rows-into here: https://sites.google.com/site/sabraonthehill/postmodern-examples/postmodern-insert#multiple-row-inserts

(:insert-rows-into 'table :columns 'a 'b :values '((10 20) (30 40)))

It's not quite the default JSON format.

It looks like I have two options:

  1. Massage the data to fit
  2. Find a function that takes it as is.

I suspect :insert-rows-into does what I want but I'm not quite sure how to cram it in there.

Upvotes: 0

Views: 466

Answers (2)

user797257
user797257

Reputation:

You can modify the parser's behaviour to produce the output you need, something like this should do:

(defun json->insert ()
  (labels ((%string->symbol (s) (intern (string-upcase s))))
    (let (keys values row)
      (json:bind-custom-vars
          (:object-key
           (lambda (key)
             (unless (member key keys :test #'equal)
               (push key keys)))
           :object-value
           (lambda (value) (push value row))
           :end-of-object
           (lambda () (setf values (cons row values) row nil)))
        (json:decode-json-from-string
         "[{
    \"term\": \"wine\",
    \"name\": \"Bubba Wine & Spirits\",
    \"address\": \"1234 N San Fake Rd,\"
    }, {
    \"term\": \"wine\",
    \"name\": \"Wine Shop\",
    \"address\": \"123 N Not Real Blvd,\"
    }]"))
      (list (mapcar #'%string->symbol keys) values))))

;; (json->insert)
;; ((ADDRESS NAME TERM)
;;  (("123 N Not Real Blvd," "Wine Shop" "wine")
;;   ("1234 N San Fake Rd," "Bubba Wine & Spirits" "wine")))

However, in my opinion cl-json is over-engineered, you might do better using something like cl-yacc to build ad hoc parser and be done with it. I know it's a weird thing to suggest to reinvent something, especially when there's already a tool for it, but that's what I ended up doing at some point (I needed a pull parser). But I encourage you to apply your own judgement.

Upvotes: 1

Lars Brinkhoff
Lars Brinkhoff

Reputation: 14325

Does this help?

(defun compose (&rest fns)
  (lambda (x)
    (reduce #'funcall fns :initial-value x :from-end t)))

(defun quotify (x)
  `',x)

(defun guess-columns (data *package*)
  (mapcar (compose #'quotify #'intern #'symbol-name #'first) (first data)))

(defun guess-values (data)
  (loop for x in data collect (mapcar #'cdr x)))

(defun insert-rows (data package)
  `(:insert-rows-into 'table :columns ,@(guess-columns data package)
                  :values ',(guess-values data)))

Calling insert-rows with your *data* gives the result

(:INSERT-ROWS-INTO 'TABLE :COLUMNS 'TERM 'NAME 'ADDRESS :VALUES
 '(("wine" "Bubba Wine & Spirits" "1234 N San Fake Rd,")
   ("wine" "Wine Shop" "123 N Not Real Blvd,")))

Upvotes: 4

Related Questions