CODEWITHSUNDEEP
javascriptnode.jsgolz4
Anand Samuel
Anand Samuel

Reputation: 71

Invalid magic number when decompressing using javascript. Headers not added when compressing using lz4 in golang

I have an gin api in which i compress a string using lz4 frame compression and send the result in the response to a frontend. In the javascript I fetch that api and decompress the data using lz4 package by pierrec, at that time I am getting Invalid magic number error. Can anyone help me with this?

Backend

package main

import (
    "bytes"
    "fmt"
    "io"

    "github.com/pierrec/lz4"

    "github.com/gin-gonic/gin"
)

var fileContent = `CompressBlock compresses the source buffer starting at soffet into the destination one. This is the fast version of LZ4 compression and also the default one. The size of the compressed data is returned. If it is 0 and no error, then the data is incompressible. An error is returned if the destination buffer is too small.`

func main() {
    router := gin.Default()

    // Define the API endpoint
    router.POST("/api/endpoint", func(c *gin.Context) {
        toCompress := []byte(fileContent)
        // compressed := make([]byte, len(toCompress))

        result, err := compress(toCompress)
        if err != nil {
            fmt.Println(err)
        }

        c.JSON(200, gin.H{"result": result})
    })

    // Start the server
    router.Run(":8000")
}

func compress(in []byte) ([]byte, error) {
    r := bytes.NewReader(in)
    w := &bytes.Buffer{}
    zw := lz4.NewWriter(w)
    _, err := io.Copy(zw, r)
    if err != nil {
        return nil, err
    }
    // Closing is very important
    if err := zw.Close(); err != nil {
        return nil, err
    }
    return w.Bytes(), nil
}

Frontend

import fetch from 'node-fetch';
import LZ4 from 'lz4';

const apiUrl = 'http://localhost:8000/api/endpoint';

const postData = async () => {
  try {
    const response = await fetch(apiUrl, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({}),
    });

    if (!response.ok) {
      throw new Error('Request failed with status ' + response.status);
    }
    const result = await response.json();
    // const decoded = base64ToArrayBuffer(result.result);
    const buffer = Buffer.from(result.result);
    const decompressed = LZ4.decode(buffer);
    const decodedBuffer = Buffer.from(decompressed);
    console.log(decodedBuffer.toString());
  } catch (error) {
    console.error(error);
  }
};

postData();

Error

Error: Invalid magic number: 4D466941 @0
    at Decoder.emit_Error (/home/anand/go/src/github.com/anand-dotworld/test/client/node_modules/lz4/lib/decoder_stream.js:64:22)
    at Decoder.read_MagicNumber (/home/anand/go/src/github.com/anand-dotworld/test/client/node_modules/lz4/lib/decoder_stream.js:93:8)
    at Decoder._main (/home/anand/go/src/github.com/anand-dotworld/test/client/node_modules/lz4/lib/decoder_stream.js:289:25)
    at Decoder._transform (/home/anand/go/src/github.com/anand-dotworld/test/client/node_modules/lz4/lib/decoder_stream.js:60:7)
    at Decoder.Transform._write (node:internal/streams/transform:184:23)
    at writeOrBuffer (node:internal/streams/writable:389:12)
    at _write (node:internal/streams/writable:330:10)
    at Decoder.Writable.end (node:internal/streams/writable:609:17)
    at Object.LZ4_uncompress [as decode] (/home/anand/go/src/github.com/anand-dotworld/test/client/node_modules/lz4/lib/decoder.js:14:10)
    at postData (file:///home/anand/go/src/github.com/anand-dotworld/test/client/index.js:24:30)

Thank you

Upvotes: 3

Views: 674

Answers (1)

Zeke Lu
Zeke Lu

Reputation: 7485

There is an error in the JavaScript code. You should replace

const buffer = Buffer.from(result.result);

with

const buffer = Buffer.from(result.result, 'base64');

Here is the doc for json.Marshal:

Array and slice values encode as JSON arrays, except that []byte encodes as a base64-encoded string, and a nil slice encodes as the null JSON value.

You see that the backend code encodes the []byte as base64-encoded string.

But according to the doc for Buffer.from(string[, encoding]), the default value of encoding is utf8. So in order to decode the base64-encoded string, you should specify the encoding as base64 explicilty.

I have tested with this configuration:

go.mod:

module m

go 1.20

require (
    github.com/gin-gonic/gin v1.9.1
    github.com/pierrec/lz4 v2.6.1+incompatible
)

require (
    github.com/bytedance/sonic v1.9.1 // indirect
    github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
    github.com/frankban/quicktest v1.14.5 // indirect
    github.com/gabriel-vasile/mimetype v1.4.2 // indirect
    github.com/gin-contrib/sse v0.1.0 // indirect
    github.com/go-playground/locales v0.14.1 // indirect
    github.com/go-playground/universal-translator v0.18.1 // indirect
    github.com/go-playground/validator/v10 v10.14.0 // indirect
    github.com/goccy/go-json v0.10.2 // indirect
    github.com/google/go-cmp v0.5.9 // indirect
    github.com/json-iterator/go v1.1.12 // indirect
    github.com/klauspost/cpuid/v2 v2.2.4 // indirect
    github.com/leodido/go-urn v1.2.4 // indirect
    github.com/mattn/go-isatty v0.0.19 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/pelletier/go-toml/v2 v2.0.8 // indirect
    github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
    github.com/ugorji/go/codec v1.2.11 // indirect
    golang.org/x/arch v0.3.0 // indirect
    golang.org/x/crypto v0.9.0 // indirect
    golang.org/x/net v0.10.0 // indirect
    golang.org/x/sys v0.8.0 // indirect
    golang.org/x/text v0.9.0 // indirect
    google.golang.org/protobuf v1.30.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)

package.json:

{
  "name": "demo",
  "version": "1.0.0",
  "main": "index.js",
  "type": "module",
  "dependencies": {
    "lz4": "^0.6.5",
    "node-fetch": "^3.3.1"
  }
}

Run the js code with this command:

node index.js

Upvotes: 0

Related Questions