CODEWITHSUNDEEP
haskellmemory-leakshaskell-snap-framework
Simon Bourne
Simon Bourne

Reputation: 455

Haskell Snap memory usage for large HTTP response bodies

I'm trying to deliver a large HTTP response using the Haskell Snap framework, but memory usage grows in proportion to the size of the response. Here's a couple of cut down test cases that use a large lazy ByteString:

import Snap.Core (Snap, writeLBS, readRequestBody)
import Snap.Http.Server (quickHttpServe)
import Control.Monad.IO.Class (MonadIO(liftIO))
import qualified Data.ByteString.Lazy.Char8 as LBS (ByteString, length, replicate)

main :: IO ()
main = quickHttpServe $ site test1 where
    test1, test2 :: LBS.ByteString -> Snap ()

    -- Send ss to client
    test1 = writeLBS

    -- Print ss to stdout upon receiving request
    test2 = liftIO . print

    site write = do
        body <- readRequestBody 1000
        -- Making ss dependant on the request stops GHC from keeping a
        -- reference to ss as pointed out by Reid Barton.
        let bodyLength = fromIntegral $ LBS.length body
        write $ ss bodyLength

    ss c = LBS.replicate (1000000000000 * (c + 1)) 'S'

The responses are delivered using chunked encoding, so I thought Snap should also be able to deliver the response in constant memory. Is there any way to achieve this? It's also worth noting that the response starts being delivered immediately. I've tried using transformRequestBody, but ran into the same problem.

Memory was measured using "top" on linux and was observed to steadily grow to 15GB resident at which point it was just starting to swap. The request did complete, so the memory usage is a couple of orders of magnitude less than the size of the ByteString. Once the request had completed, it sat at 15Gb resident. When I fired another request at it, it still remained steady at 15Gb and completed the request as before. The virtual size stayed within 5% of the resident size.

Firing 2 concurrent requests at it resulted at first in a drop in virtual and resident memory to about 5Gb, followed by an increase to about 17Gb at which point the machine was getting unusable so I killed the process.

GHC version 7.8.3

Snap version 0.14.0.5

Upvotes: 4

Views: 153

Answers (0)

Related Questions