CODEWITHSUNDEEP
asynchronousclojurecore.async
C.J.
C.J.

Reputation: 23

how can I insure all data is written to a file from clojure's core.async channels?

I'm quite new to core.async, and I have been attempting to understand how best to make use of core.async with file IO. I have put together a test that fails to write the last file though printing to the console works. Any idea what I am missing?

First, some functions...

(defn thread-write-to-files [channel]
  (let [writer (atom nil)]
    (thread
      (loop []
        (when-some [value (<!! channel)]
          (if (and (map? value) (= :FILE (:type value)))
            (do (when @writer (.close ^Writer @writer))
                (reset! writer (io/writer (File. ^String (:name value))))
                (recur))
            (do (when @writer (.write @writer value)
                              (println value))
                (recur)))))
      (when @writer
        (do (.flush @writer)
            (.close ^Writer @writer))))))

(defn add-line-number [channel-in channel-out]
  (go-loop [line-number 1]
    (when-some [value (<! channel-in)]
      (if (and (map? value) (= :FILE (:type value)))
        (do (>! channel-out value)
            (recur 1))
        (do (>! channel-out (str line-number ". " value))
            (recur (inc line-number)))))))

Now a test that makes use of them...

(deftest test-thread-write-to-file
  (let [input-coll ["This gets skipped"
                   {:type :FILE :name "foo.txt"}
                   "This is the first line of foo!\n"
                   "This is the second line of foo.\n"
                   {:type :FILE :name "bar.txt"}
                   "Bar me 1.\n"
                   "Bar me 2.\n"
                   "Bar me 3.\n"
                   {:type :FILE :name "baz.txt"}
                   "BBBBBBBBBBB\n"
                   "AAAAAAAAAAA\n"
                   "ZZZZZZZZZZZ\n"]
        input-channel (async/to-chan input-coll)
        output-channel (chan)
        foo (File. "foo.txt")
        bar (File. "bar.txt")
        baz (File. "baz.txt")]
    (when (.exists foo) (.delete foo))
    (when (.exists bar) (.delete bar))
    (when (.exists baz) (.delete baz))
    (add-line-number input-channel output-channel)
    (thread-write-to-files output-channel)
    (Thread/sleep 1000)
    (is (.exists foo))
    (is (.exists bar))
    (is (.exists baz))
    (is (> (.length foo) 0))
    (is (> (.length bar) 0))
    (is (> (.length baz) 0))))

The last condition tested fails. File baz.txt is created, but empty. My REPL prints out each line from the input, so I am confused as to why the file is still empty.

Upvotes: 2

Views: 202

Answers (1)

Magos
Magos

Reputation: 3014

In thread-write-to-files you perform the flush and close of the final file when the input channel is closed (when (when-some [value (<!! channel)] ...) gets a nil and exits the loop).
Your test never closes the channel, so this doesn't happen. Try using close! on output-channel, or maybe use onto-chan instead of to-chan to put the test data collection into the system.

Upvotes: 1

Related Questions