CODEWITHSUNDEEP
haskellbinarybytestringconduitcereal
jorgen
jorgen

Reputation: 3593

Binary file writer adds extra byte

I'm building a Conduit that writes a binary file consisting of a header followed by a Double matrix as a row-ordered list. Here's the code:

import Conduit ((.|), ConduitM, mapC, sinkFileBS, yield) 
import Control.Monad.Trans.Except (ExceptT) 
import Control.Monad.Trans.Resource (ResourceT) 
import Data.ByteString (ByteString) 
import Data.ByteString.Conversion (toByteString') 
import Data.Serialize.IEEE754 (putFloat64be) 
import Data.Serialize.Put (putListOf, runPut) 
import Data.Void (Void) 
import Numeric.LinearAlgebra.Data ((><), Matrix, toLists) 
import System.FilePath (FilePath) 

type FileWriter = ResourceT (ExceptT String IO) 

matrixSink :: FilePath -> ConduitM (Matrix Double) Void FileWriter ()
matrixSink path = byteBuilder .| sinkFileBS path where
  byteBuilder = do
    yield $ toByteString' "header" 
    mapC fromDoubleMatrix

fromDoubleMatrix :: Matrix Double -> ByteString
fromDoubleMatrix matrix = runPut $
   putListOf putFloat64be (concat toLists matrix)

This almost works. If I test it using

runExceptT . runConduitRes $ yield matrix .| matrixSink "test.dat"
  where matrix = (2 >< 2) [1, 2, 3, 4]

I get the expected file but with an extra byte between the header and the list of doubles. When displayed using show the extra byte looks like this:

"\NUL\NUL\NUL\NUL\NUL\NUL\NUL\t"

Any idea how not to print this byte? Or if it's a canonical separator or something (so that I can ignore it in the reader)?

EDIT: The problem seems to occur in the putListOf construction in fromDoubleMatrix.

Upvotes: 1

Views: 98

Answers (1)

NovaDenizen
NovaDenizen

Reputation: 5325

putListOf :: Putter a -> Putter [a]
putListOf pa = \l -> do
  putWord64be (fromIntegral (length l))
  mapM_ pa l

putListOf encodes the length of the list before encoding the individual list elements. I think maybe you are dealing with fixed 2x2 matrices so you don't need that length, and you just want:

fromDoubleMatrix :: Matrix Double -> ByteString
fromDoubleMatrix matrix = runPut $
   mapM_ putFloat64be (concat toLists matrix)

Upvotes: 1

Related Questions