decode websocket received data

Question

I am working on project of my own website where I need to scrape data from target site using websocket. Data is live feed or tick for price movement of currency and stocks. I am getting output in following format.

try on http://websocket.org/echo.html:

Location: wss://streamer.finance.yahoo.com/

Click on Connect.

Message: {"subscribe":["AMZN"]}

OUTPUT:

CONNECTED

SENT: {"subscribe":["AMZN"]}

RECEIVED: CgRBTVpOFaQY3EQY4Kn0/99bKgNOTVMwCDgBRYjKzDxIyvN9ZQBQ4T7YAQQ=

RECEIVED: CgRBTVpOFaQY3EQY4Kn0/99bKgNOTVMwCDgBRYnKzDxIzPV9ZQBQ4T7YAQQ=

RECEIVED: CgRBTVpOFT0a3EQYsLn0/99bKgNOTVMwCDgBRYMG5DxIkP99ZQDg+j7YAQQ=

RECEIVED: CgRBTVpOFQAY3EQYwIf1/99bKgNOTVMwCDgBRYd5wzxIxod+ZQAQ1z7YAQQ=

RECEIVED: CgRBTVpOFQAY3EQYwIf1/99bKgNOTVMwCDgBRYd5wzxIroh+ZQAQ1z7YAQQ=

RECEIVED: CgRBTVpOFRQS3EQY8PT1/99bKgNOTVMwCDgBRYC1WjxIhI5+ZQCgcD7YAQQ=

RECEIVED: CgRBTVpOFRQS3EQY8PT1/99bKgNOTVMwCDgBRYG1WjxImo5+ZQCgcD7YAQQ=

RECEIVED: CgRBTVpOFUgN3EQY4KP2/99bKgNOTVMwCDgBRSBhnjtIvpJ+ZQBArj3YAQQ=

RECEIVED: CgRBTVpOFUgN3EQY4KP2/99bKgNOTVMwCDgBRSBhnjtI9J1+ZQBArj3YAQQ=

RECEIVED: CgRBTVpOFUgN3EQY4KP2/99bKgNOTVMwCDgBRSBhnjtIsqR+ZQBArj3YAQQ=

RECEIVED: CgRBTVpOFUgN3EQY4KP2/99bKgNOTVMwCDgBRSBhnjtInq5+ZQBArj3YAQQ=

I don't know how to decode or what type of encryption it is. Can Anybody tell me how to decode it of what is encode/decode type is it? I will use PHP for decode(if decoding posible).

Answer 1

Thanks to @Maxim, I was able to get better understanding how this works and made NodeJS version of it.

Here is code for basic example and I will spend some time to build dynamic subscription and final app for it based on Electron. The idea is to get decoded data and push to local SocketIO server where it can be used with VueJS in different apps.

const WebSocket = require('ws')
var ProtoBuf = require("protobufjs");

"use strict";
let Message = ProtoBuf
.loadProtoFile('./PricingData.proto', (err, builder)=>{
    Message = builder.build('PricingData')
    loadMessage()
})



let loadMessage = ()=> {
    const url = 'wss://streamer.finance.yahoo.com'
    const connection = new WebSocket(url)
    connection.onopen = () => {
    connection.send('{"subscribe":["TSLA","AXSM","UBER","MIRM","GRKZF","BTCUSD=X","ETHUSD=X","AUDUSD=X","^DJI","^IXIC","^RUT","^TNX","^VIX","^CMC200","^FTSE","^N225"]}') 
    }

    connection.onerror = (error) => {
    console.log(`WebSocket error: ${error}`)
    }

    connection.onmessage = (e) => {
    let msg = Message.decode(e.data)
    console.log('Decoded message', msg)
    }
}

Quick update: Here is full example on my repo https://github.com/markosole/yahoo-node-streamer

Answer 2

You can use or refer to my repository (thanks to Maxim for the proto file!).

It is an easy to use Python package.

1. Install the package

    pip install yliveticker

2. Create livemarket.py file with the following code

    import yliveticker


    # this function is called on each ticker update
    def on_new_msg(msg):
        print(msg)


    # insert your symbols here
    yliveticker.YLiveTicker(on_ticker=on_new_msg, ticker_names=[
    "BTC=X", "^GSPC", "^DJI", "^IXIC", "^RUT", "CL=F", "GC=F", "SI=F", "EURUSD=X", "^TNX", "^VIX", "GBPUSD=X", "JPY=X", "BTC-USD", "^CMC200", "^FTSE", "^N225"])

3. Run code

    python livemarket.py

4. Watch live market data appearing in the console output.

If you don't see any results, make sure you are within trading hours of your stock exchange

Answer 3

I looked into that. I have no clue what to do for understand work flow and debug the function. JS is not my strong suite.

You are not stuck to any particular language when you are interfacing external system which uses protobuf. Protobuf is open technology which allows to create marshalling code for multiple languages automatically when meta description of message is known. So it is not required to re-use available code but to extract Protobuf structure out of it. Protobuf compiler will do all dirty work for you.

You can easily reconstruct proto file by just looking into __finStreamer-proto.js file

PricingData.proto

syntax = "proto3";

message PricingData {

enum QuoteType {
    NONE = 0;
    ALTSYMBOL = 5;
    HEARTBEAT = 7;
    EQUITY = 8;
    INDEX = 9;
    MUTUALFUND = 11;
    MONEYMARKET = 12;
    OPTION = 13;
    CURRENCY = 14;
    WARRANT = 15;
    BOND = 17;
    FUTURE = 18;
    ETF = 20;
    COMMODITY = 23;
    ECNQUOTE = 28;
    CRYPTOCURRENCY = 41;
    INDICATOR = 42;
    INDUSTRY = 1000;
};

enum OptionType {
    CALL = 0;
    PUT = 1;
};

enum MarketHoursType {
    PRE_MARKET = 0;
    REGULAR_MARKET = 1;
    POST_MARKET = 2;
    EXTENDED_HOURS_MARKET = 3;
};

    string id = 1;
    float price = 2;
    sint64 time = 3;
    string currency = 4;
    string exchange = 5;


    QuoteType quoteType = 6;
    MarketHoursType marketHours = 7;
    float changePercent = 8;
    sint64 dayVolume = 9;
    float dayHigh = 10;
    float dayLow = 11;
    float change = 12;
    string shortName = 13;
    sint64 expireDate = 14;
    float openPrice = 15;
    float previousClose = 16;
    float strikePrice = 17;
    string underlyingSymbol = 18;
    sint64 openInterest = 19;
    OptionType optionsType = 20;
    sint64 miniOption = 21;
    sint64 lastSize = 22;
    float bid = 23;
    sint64 bidSize = 24;
    float ask = 25;
    sint64 askSize = 26;
    sint64 priceHint = 27;
    sint64 vol_24hr = 28;
    sint64 volAllCurrencies = 29;
    string fromcurrency = 30;
    string lastMarket = 31;
    double circulatingSupply = 32;
    double marketcap = 33;
};

then you can use protobuf compiler to build php files out of it:

mkdir yahoo
protoc --php-out=yahoo PricingData.proto

also here is our composer.json

{
    "require": {
        "google/protobuf": "^3.11",
        "ratchet/pawl": "^0.3.4"
    },
    "autoload": {
        "classmap": [
            "yahoo"
        ]
    }
}

and php file to suck the data:

#!/usr/bin/php
<?php

require __DIR__ . '/vendor/autoload.php';

    \Ratchet\Client\connect('wss://streamer.finance.yahoo.com:443')->then(function($conn) {
        $conn->on('message', function($msg) use ($conn) {
            echo "Received: {$msg}\n";
        $packed = base64_decode($msg);
        $msg = new PricingData();
        $msg->mergeFromString($packed);
        var_dump($msg->serializeToJsonString());
        });

    $conn->send('{"subscribe":["BTC-USD","ETH-USD","XRP-USD","USDT-USD","BCH-USD","BA","TSLA","AXSM","UBER","MIRM","GRKZF","SCGPY","BDVSF","WPX","BIPSX","ENPIX","ENPSX","BPTUX","BPTIX","CL=F","GC=F","SI=F","EURUSD=X","GBPUSD=X","JPY=X","EZA","IXC","IYE","FILL","EWT","CGIX1191220P00005000","TORC191220P00002500","RIOT191213C00001000","TPCO191220C00002500","DHR","AMRN","AMD","PCG","VIX191218P00012500","VIX191218P00014000","EEM191220P00039000","EEM200117C00045000","BTCUSD=X","ETHUSD=X","AUDUSD=X","NZDUSD=X","EURJPY=X","GBPJPY=X","EURGBP=X","EURCAD=X","EURSEK=X","EURCHF=X","EURHUF=X","CNY=X","HKD=X","SGD=X","INR=X","MXN=X","PHP=X","IDR=X","THB=X","MYR=X","ZAR=X","RUB=X","ZG=F","ZI=F","PL=F","HG=F","PA=F","HO=F","NG=F","RB=F","BZ=F","B0=F","C=F","O=F","KW=F","RR=F","SM=F","BO=F","S=F","FC=F","LH=F","LC=F","CC=F","KC=F","CT=F","LB=F","OJ=F","SB=F","IFF","CRS","RLLCF","BGNE","^GSPC","^DJI","^IXIC","^RUT","^TNX","^VIX","^CMC200","^FTSE","^N225"]}');

    }, function ($e) {
        echo "Could not connect: {$e->getMessage()}\n";
    });

and.... here we go:

Received: CgNQQ0cVH4UvQRiQvr/a4lsqA05ZUTAIOAFFKBlXQUj61YQeZWhmpj/YAQQ=
string(202) "{"id":"PCG","price":10.97,"time":"1576616325000","exchange":"NYQ","quoteType":"EQUITY","marketHours":"REGULAR_MARKET","changePercent":13.443642,"dayVolume":"31495549","change":1.3000002,"priceHint":"2"}"
Received: CghFVVJHQlA9WBW6a1k/GODNv9riWyoDQ0NZMA44AUV+6bc/ZYAZRTzYAQg=
string(193) "{"id":"EURGBP=X","price":0.84930003,"time":"1576616326000","exchange":"CCY","quoteType":"CURRENCY","marketHours":"REGULAR_MARKET","changePercent":1.4368131,"change":0.012030005,"priceHint":"4"}"
Received: CghHQlBKUFk9WBVxvQ9DGODNv9riWyoDQ0NZMA44AUUQl7S/ZcClA8DYAQg=
string(192) "{"id":"GBPJPY=X","price":143.74001,"time":"1576616326000","exchange":"CCY","quoteType":"CURRENCY","marketHours":"REGULAR_MARKET","changePercent":-1.4108601,"change":-2.0569916,"priceHint":"4"}"
Received: CgVNWE49WBWqgpdBGJC+v9riWyoDQ0NZMA44AUXYFZ89ZQDYcDzYAQg=
string(191) "{"id":"MXN=X","price":18.938801,"time":"1576616325000","exchange":"CCY","quoteType":"CURRENCY","marketHours":"REGULAR_MARKET","changePercent":0.077678382,"change":0.014699936,"priceHint":"4"}"
Received: CgVTR0Q9WBWCi60/GJC+v9riWyoDQ0NZMA44AUXWfb49ZQAkpTq9AX6MrT/NAXGPrT/YAQg=
string(219) "{"id":"SGD=X","price":1.3558199,"time":"1576616325000","exchange":"CCY","quoteType":"CURRENCY","marketHours":"REGULAR_MARKET","changePercent":0.093013451,"change":0.001259923,"bid":1.35585,"ask":1.35594,"priceHint":"4"}"
Received: CgVKUFk9WBVQDdtCGODNv9riWyoDQ0NZMA44AUW/HUa9ZQAYWb3YAQg=
string(191) "{"id":"JPY=X","price":109.526,"time":"1576616326000","exchange":"CCY","quoteType":"CURRENCY","marketHours":"REGULAR_MARKET","changePercent":-0.048368212,"change":-0.053001404,"priceHint":"4"}"
Received: CgRVQkVSFZqZ7UEY4M2/2uJbKgNOWVEwCDgBRY8Vlb9IxqSRHGUAM7O+2AEE
string(210) "{"id":"UBER","price":29.700001,"time":"1576616326000","exchange":"NYQ","quoteType":"EQUITY","marketHours":"REGULAR_MARKET","changePercent":-1.1647204,"dayVolume":"29501731","change":-0.34999847,"priceHint":"2"}"
Received: CgRUU0xBFR9lvUMYwK6/2uJbKgNOTVMwCDgBRZvZNb9IoL/SB2WAcC3A2AEE
string(209) "{"id":"TSLA","price":378.79001,"time":"1576616324000","exchange":"NMS","quoteType":"EQUITY","marketHours":"REGULAR_MARKET","changePercent":-0.71035165,"dayVolume":"8015824","change":-2.7099915,"priceHint":"2"}"
Received: CghFVVJHQlA9WBXLZ1k/GLDdv9riWyoDQ0NZMA44AUWW/rY/ZcAdRDzYAQg=
string(193) "{"id":"EURGBP=X","price":0.84924001,"time":"1576616327000","exchange":"CCY","quoteType":"CURRENCY","marketHours":"REGULAR_MARKET","changePercent":1.4296443,"change":0.011969984,"priceHint":"4"}

As you can see protobuf is awesome thing which is language agnostic so you do not have to cope with some unfamiliar language