CODEWITHSUNDEEP
javascriptnode.jsmemcachedbrotli
owengall
owengall

Reputation: 463

Nodejs Brotli compression returning zeros

I’m trying to store large database query results in a memcached server for quicker retrieval, and am compressing the data prior to entry in memcache with brotli.

I’ve been trying to test with an example object like so:

//cacheserver already defined and connected
let brotli_config = {
    mode: 1, //0=regular, 1=text
    quality: 8 //1=fast .. 11=small
}

let test_key = 'paths_100'
let test_value = [
    {
        text: 'this is text',
        outlines: 'this is an outline',
        inlines: 'this is an inline'
    }
]

let compressed_in = require('brotli/compress')(JSON.stringify(test_value), brotli_config)
console.log(compressed_in) //LOG 1
cacheserver.set(test_key, compressed_in, {
        expires: 3600
    },
    function(err) {
        cacheserver.get(test_key, function(err, compressed_out) {
            console.log(compressed_out) //LOG 2

            let string_out = require('brotli/decompress')(compressed_out)
            console.log(string_out) //LOG 3
        })
    }
)

And this is the output:

//LOG 1
Uint8Array(10) [
  27, 86,   0,  0, 36,
   0,  2, 177, 64, 32
]

//LOG 2
<Buffer 1b 56 00 00 24 00 02 b1 40 20>

//LOG 3
Uint8Array(87) [
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0
]

I’m turning the object into a string (probably not a good idea for performance, but using brotli mode 0 and not stringifying wasn’t working; returning null...), compressing it, storing the 10-entry uint8array in the cache server, retrieving a 10-byte buffer, and decompressing it. However, the decompressed result is just an array of zeros. What’s happening here? I’m using this compression library and the memcached client (memjs) for the first time, so any information on how to properly use them is appreciated.

Upvotes: 2

Views: 992

Answers (1)

owengall
owengall

Reputation: 463

I figured out a solution (let me know if it could be sped up). Many thanks to this blog post for clarifying the relationship between nodejs objects, strings, and buffers.

I was missing a conversion from a string to a buffer before passing the object data to brotli. The full procedure looks like this now:

let data = {/* stuff */}
let test_key = 'paths_100'
let compressed_in = require('brotli/compress')(Buffer.from(JSON.stringify(test_value),'utf-8'), brotli_config)
cacheserver.set(test_key, compressed_in, {
        expires: 3600
    },
    function(err) {
        cacheserver.get(test_key, function(err, compressed_out) {
            let data_out = JSON.parse(Buffer.from(require('brotli/decompress')(compressed_out)).toString('utf-8'))
        })
    }
)

Upvotes: 2

Related Questions