CODEWITHSUNDEEP
file-ioclosurescommon-lispgeneratorstate
Gwang-Jin Kim
Gwang-Jin Kim

Reputation: 9940

reading from file line by line (low-memory) in Common Lisp

I am searching for a way to read from several files 1 s-expression (data list) at a time.

The thing is that the files are huge - hundreds of megabytes or gigabytes. And I need the RAM for calculations.

For output files,

(defun add-to-file (process-result file-path)
  (with-open-file (os file-path :direction :output
                                :if-exists :append
                                :if-does-not-exist :create)
    (print process-result os)))

does the job well to append line by line the result-string or s-expression. (I don't know - maybe it is not the most efficient way?).

Some time ago, I asked for a macro which opens as many files as I want with with-open-file and where I can access from the body all the files whose stream-variables I could create and give. However, since the number of the open input files and output files are variable, maybe it is much easier for the design to call each file with such callers - open them - get to the right position - write or read - and then close it again, I thought.

For the output, the given function does the job. However, for the input, I would love to have a function which everytime I invoke it, reads the next lisp-expression (s-expression) and has kind of a memory where it read the last time in the file and everytime I invoke it - opens the file anew and knows where to read - and returns the value - and the next time reads and returns the next value etc. Similar to Python generators over iterators - which yield the next value in the sequence.

I want to process - read-in - the file expression by expression - to have minimal memory usage.

How would you attack such task? Or do you have a good strategy in mind?

Upvotes: 3

Views: 508

Answers (1)

Rainer Joswig
Rainer Joswig

Reputation: 139331

Sketch:

make a structure or class, which stores the last position read.

(defstruct myfile
  path
  (last-position 0))

(defmethod next-expression ((mf myfile))
  (with-open-file (s (myfile-path mf) :direction :input)
    (file-position s (myfile-last-position mf))
    (prog1
        (read s)
      (setf (myfile-last-position mf) (file-position s)))))

Usage example:

(defparameter *mf1* (make-myfile :path (pathname "/foo/bar.sexp")))

(print (next-expression *mf1*)) ;; get first s-expr from file
;; do sth else
(myfile-last-position *mf1*)  ;; check current position
;; do sth else
(print (next-expression *mf1*)) ;; gives next s-expr from file

Then write a method to check if a new s-expression is available. Etc.

Upvotes: 6

Related Questions