damniatx
damniatx

Reputation: 234

Decode chunked response on nodeJS Request

How do you guys read chunked response on nodejs ? Raw Data :

\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0003���W�H�\u0000�s8��\u001f:�������G�\u001d�\u0005B&�$�\u0000��2\\N�j�\n���\u0003�\u0002�����Բe\u001b��;s���,��Q]]]]]U�]z�s;��5\u000b#7�m�a�\r��^����6����n\u001b;��7/�\u001e���~|@�1�?�����>1��گ����۳����g�?\u0011\u0000@�B�Gn\f\u0000�W�\u001d\u001c\u001a�\u0018���U����X7�V\u0010\u000ejg'�[������\u001ak5-'v����\u001b�bϣQd\u001b~P�\u001a\u0011?�\u0005=\u001a3��1\u001bE�`��S<\u00193[5\u0017��lDys�a�$�� \u001eŎ1� �q?�\u000bPk�~�Z`$\u000b��AZ.\u0018��\u0011���\u000f\u0006����/�Ӭ���z�\u0006�U��\u000f\n�u��Μ:[5�+�\u0010�o�y�\u001b��>:�\u0002����-�ƴz\u001d�h7�h8�\u0015Ӣ�n\u000en��X7\b��^0B�۵~WR��\u0012��\t�Ng�ś\u0011�)A\u0000U�g�BÿU��V��ј�n׃��\u0005~���6>\u001c��\u00190�7\f�\u0011�\u001b\u0006�\u0015B�\u0017\u0015�g0�Z����5\u001c�6�\ri\u0018��N�>p'�y\u0013�Bw\u001c\u00131���WzME��)߸�\u0013�X�'\u0007_>�ߗ��}�P�X^@\u001d\u0016^B+}w`�ݎ]�e|��Ï_��~��fw�˻���϶�О\u0007ż����\u0010�\u000f��O�\u001e2p96��W��;\t�����/ٹQ���!\tl|����8\b��u��Ў!��~a����\u0003Kf�)X��򂍋sv�\u000eY���Bp�*\u000ff`�\u0015+\u000f2\u0013[R�\u000f��� \u001a��t\u000b�D�GEV���� ,������oy�\u001f��v��V�ޝ\u0007\u0017�\u0014���]�u��\u0018q�.\u000e'w\u0011��\u001f\u0005\u001e[]�\u000f0&\u0003�}�Ѹ7��><�6M׎�\u0006cFŤ��ج�\u0011���P\u0002�\u0003\u0006�N� �\u0003f\rX�\u0001dA����=:���\u0015�=\u0017 �\u0000 m��\n\u001a\u001a���\u0014?�eݬ6^�v`\u0001\u001f��# �î�\n/\u0004OP�2S\u0002��\n��[QLu�\u0007��ɫ��\u0005t2\\`�\u0010�_��a\u0010\u001aٸ\u0003��hV\u0014��U\u0005����j�},��\u0016��Q��Q\u000e���\t������|8$o\u000f~9�tt�\u0019_?\u001f�=\u0000�@\tN�\u00161�h9ʁ\u0005�b�㇊�5p�u�\u0018\u0015��y�pa6\u0017\u000f:���

toString('utf8')

���iw�8��9>���=ג��x�#��xK'��v�N/����DHbL�j.�5�����B�%9���~�M��"� �B����܍��'F�V�u}�V;8? ��;���rR?rcH�Z��� �0�ǭZ���ֺ]��pP;?��!�V���X�i9�ctVV��{�"���ꗈ�/�јč�(�cƟ�ɘ٪��7d#ʛ; '���(v������]�Z����#Y��r�`l�X͏~0�� �~!�f�^��5ȯ T~Poz4t��٪�\ф(~�+ܨ�������,�l94�՛�G��G� ���wsp��ǺApm��B߮��� ��p|Lu:+/ތXL	��?���y����4v�O/�c�Ƕ���f΀��a���0H�¾�P=��Ԫ��.�ᘴIoHÈ�v��;̛��㘈��%��*R�N������::=���������J����� Z����n�:-��������ۏ���������~����<(�Օ^��~��~����˱�L�r��IX�7�_~�.���� I`��}���AG����v)�K�G=�X2�L�2U�l\^��v��$���Vy43�Xy��ؒJ|4Ѱm�[�'z?*�����aY4^oo|�c� ���� ��"����<����Ĉ+�q8�����(����|�1@�c�ƽ!v����i�v\63*&���fŌ��m��0�w!0k��� �Ǿ��AdT�h���i�T��X]-���/�f��Ҷ��� vcTx!x����|V�D1չtN&�����p��C�~U�Ahd���YQL{�`:ަ�����^Z�Fe�F9,G�d��ãs����r�����~:>8�@ N�1�h9ʁ�b��NJ�%p�u���y�xi6:�����wv���h�0������n�g�G��0>�뱱�R�-ա�+*Ff+��P��?wG�\1_�/uR˩P0����E�coR��nd�2 ��NT��^6�.�C����E�v��`���$�u�!�[�:�\߈��|Q����GA���ӏ6	��3�Ϛ� RrŅ��.�|��eH}�Kg��5�z�$���m)���k&�ˆ��<�Y}F�l,� g@����vȧa��� N��*I�U� ΄���������b@����G71�@zu5.��Yw����vq����D�^��#Y���,�'Д�f7�/�&X�ӛw��N ��}G�����ꬮ:�:6X�\4�3��p��V���J�l�\�h^�E��e��9�crf�{@��4+vG�f‡~�,ڛ����O��fq[��+dB�c4�����P|�[cU�0�m����AZ3��"*Y3��n�!q2yڸl�G�(ڽ�o�f��[TKX^-����h ����d��l� ���Q�q+m>�%޲>2/bE�[b	�$^]���7/-˸�ʞ;E=]��0@V㴰Y�wlͲ��A��X^`�@�QX�F��]9_e �=[eQF���x�,-�x�$4B�ϡ��Q(��8�ƚk��?��A��<��p�ƪh*q=_ h��� �~Ԩ�m/7�du�U�̔9�#��g�Dx������'�.�ʔ�p�w�Ot�����k�DFfW�@h��	�5H�=�����qu��z^Z �2U}@\����Lo4b�+X#Zs*{�f�8�o t },� ��A�|��9���z�0U)��!��$B���05!�(7l��c>Mf'wt���6���@�4���5uY��90C��>��ЫVRU��VqiƸ�i��m.NAx2�}`Wf�; ��V����,�0~s��?1w�@�7�%��2�����(�$�,�'2|TgI\��1�r��hr]�����l	�WG2eR@����{�o�NA���OǨ�ό�8z��� �f-���TfJ��oܒ�gXnBk��Xr9���v�Q�(S;��ڳ���1g���@Cb� ��%�d����t�5a~��Y��O��茔%��G��U�W���9��S{�=k����Y0f��sȃb�9��%����Y�+�E�G�@�z ��CA���0��R �l��a"+*m$��+�F �X�kޣ�����������~��/%��*��8l{��@˩�S�Z�,�81G��.p���6 

Translating this with toString('utf8') make it more unreadable.

Here is the header,

Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Vary: Accept-Encoding, User-Agent

Is there any option i can read the chunked data like this. To make it readable for me. I was scraping a website, and the body is displayed like that.

Any help is much appreciated.

Thanks

Upvotes: 2

Views: 4871

Answers (2)

Nurai
Nurai

Reputation: 11

This is the answer you are looking for. Works for http2, I bet it does for http as well:

const http2 = require('http2');
const zlib = require('zlib');

let zip = zlib.createGunzip();

... client ...
const req = client.request()
req.pipe(zip);

let buffer = [];

zip.on('data', (chunk) => {
  buffer.push(chunk.toString());
});

zip.on('end', () => {
  console.log(buffer.join(''));

  client.close();
});
req.end();

Upvotes: 0

LearnDriver
LearnDriver

Reputation: 571

It is likely that the response is gzipped; if you add gzip: true, the response will likely be human readable again.

From the request docs https://github.com/request/request:

For backwards-compatibility, response compression is not supported by default. To accept gzip-compressed responses, set the gzip option to true. Note that the body data passed through request is automatically decompressed while the response object is unmodified and will contain compressed data if the server sent a compressed response.

var request = require('request')
  request(
    { method: 'GET'
    , uri: 'http://www.google.com'
    , gzip: true
    }
  , function (error, response, body) {
      // body is the decompressed response body
      console.log('server encoded the data as: ' + (response.headers['content-encoding'] || 'identity'))
      console.log('the decoded data is: ' + body)
    }
  ).on('data', function(data) {
    // decompressed data as it is received
    console.log('decoded chunk: ' + data)
  })
  .on('response', function(response) {
    // unmodified http.IncomingMessage object
    response.on('data', function(data) {
      // compressed data as it is received
      console.log('received ' + data.length + ' bytes of compressed data')
    })
  })

Upvotes: 3

Related Questions