how do I listen to a webhook (https address end point) from python client?

Question

I have a nodejs end point which will be called by a job with a POST HTTP call containing the details of the job data in JSON format?how do I listen to a webhook (https address) from python client to get the the job data?

https://company/api/bats_push

app.post("/api/bats_push",(req, res) => {
        //console.log("Calling bats_push...")
        const d = {
            method: req.method,
            headers: req.headers,
            query: req.query,
            body: ''
        }

        req.on('data', (c) => {
            d.body = d.body + c
        });


        req.on('end', () => {
            DATA.push(d);
            res.end('Saved');
        });
});

python client:

data = {'name': 'joe', 'age': 20}
webhook_url = 'http://localhost:3000/api/bats_push'
json_data = json.dumps(data)
print json_data
try:
    r = requests.post(webhook_url,data=json_data.encode("utf8"),verify=False,headers={"Content-Type": "application/json"},timeout=10.0)
    print "posted"
    print(r.status_code, r.reason)
    print r.url
except Exception as e:
    print (e)

Error:-

HTTPConnectionPool(host='localhost', port=3000): Read timed out. (read timeout=10.0)

Answer 1

You save the posted data and return it to an interested consumer. It means you create a webhook bridge.

Example Node script:

const http = require('http');
const URL = require('url');

const DATA = [];

const routes = {
    '/api/bats_push': (req, res) => {
        const d = {
            method: req.method,
            headers: req.headers,
            query: req.query,
            body: ''
        }

        req.on('data', (c) => {
            d.body = d.body + c
        });

        req.on('end', () => {
            DATA.push(d);
            res.end('Saved');
        });
    },

    '/api/bats_pull': (req, res) => {
        const json = JSON.stringify(DATA, true); 
        DATA.length = 0;
        res.statusCode = 200;
        res.setHeader('content-type','application/json');
        res.end(json);
    }
};

const server = http.createServer((req, res) => {
    const reqUrl = URL.parse(req.url);
    const route = routes[reqUrl.pathname];
    if (route) { 
        req.query = reqUrl.query;
        route(req, res);
    } else {
        res.statusCode = 404;
        res.end('Not found');
    }
})

server.listen(8080,() => console.info('Server started'));

And, a python client script to test it:

import http.client
import json

print('\nPosting data...')

conn1 = http.client.HTTPConnection('localhost', 8080)
headers1 = {'content-type': 'application/json'}
body1 = {'name': 'joe', 'age': 20}

conn1.request('POST', '/api/bats_push?q=100', json.dumps(body1), headers1)

resp1 = conn1.getresponse()
print(resp1.status,resp1.reason, resp1.read())
conn1.close()

print('\nRetrieving data...')

conn2 = http.client.HTTPConnection('localhost', 8080)
headers2 = {'accept': '*'}

conn2.request('POST', '/api/bats_pull', None, headers1)

resp2 = conn2.getresponse()
resp2Json = json.loads(resp2.read())
print(resp2.status,resp2.reason, resp2Json)

output:

Posting data...
200 OK b'Saved'

Retrieving data...
200 OK [{'method': 'POST', 'headers': {'host': 'localhost:8080', 'accept-encoding': 'identity', 'content-length': '26', 'content-type': 'application/json'}, 'query': 'q=100', 'body': '{"name": "joe", "age": 20}'}]

Also, The HTTP request is saved "as is" . Contents of header, query and body are not processed. The clients should parse the data accordingly after pulling.

Answer 2

Working with payment related systems in my career; in general webhooks are a callback that you get for a process initiated but it is separated from the main flow so that that memory is not held up without a reason.

Going by what you said

[Webhook] => Python => Node.js

and I wasn't able to understand what exactly is making a call for the process which adds the webhook flow in the whole cycle. Regardless it shouldn't change the answer.

So basically what you need is an API endpoint which will be called from this third party webhook. How you decide to facilitate that on your side using python is entirely your call. An API system up and running wouldn't be that complicated using frameworks like Django, performance etc. related questions is something I wouldn't dwell into.

Let me know if it clear to you.

Answer 3

Normally, a "webhook" is a feature of an online service that sends a request to your app at a public url where you have a HTTP server deployed, in response to some event within the service. The url needs to be configured in whatever is sending the request, it's not explicitly set in the receiving code.

The thing that listens to the webhook is an HTTP server with a suitable endpoint defined. To create a server you can use many different packages, but something simple like Flask is a good place to start.

The minimal view code would look something like:

@app.route('/api/bats_hook')
def bats_hook():
    data = request.get_json()
    print(data)
    return 'OK'

If whatever is sending the POST (your nodejs service?) needs to be able to send requests to your server, so you'll either need to deploy the server so it's publicly accessible, or set the service to send the requests to your public IP. If it's on the same machine you can use a private IP, or localhost.

On the other hand, if a node process is currently receiving the POST requests at https://company/api/bats_hook/, and you want to notify a python process whenever a job comes in, that's a different problem. You could either send that information on to a separate python server via axios.post or similar (this might be the easiest option), or store the data somewhere that the python process can access it. Redis would be a good option, because you can easily set up a publish/subscribe system, and the node and python processes can be on the same machine or on different machines. If they're on the same machine, you could also just dump the data to a log file and read that into python. Sending data from one process to another is called inter-process communication, and it gets complicated quickly.

If you were to use redis, with node_redis and redis-py, the implementation might look like:

// javascript
var redis = require("redis"),
    redisClient = redis.createClient();

app.post("/api/bats_hook", (req, res, next) => {
  console.log(req.query.params)
  console.log(req.body)
  console.log(req.body.name)
  // This publishes a message to the "bats hook channel"
  redisClient.publish("bats hook channel", JSON.stringify({
    params: req.query.params,
    body: req.body,
  }));
  res.status(200).json({
    message: "BATS object",
    posts: req.body
  }); 
});

# python
import redis

r = redis.Redis()
p = r.pubsub()
p.subscribe('bats hook channel')

# This blocks and reads any messages from the "bats hook channel" as they come in
for message in p.listen():
    print(message)

A lot depends on your intended usage. If the python part is going to be just used for debugging locally, you'd make different choices than if it needs to be deployed.

Answer 4

The setup you describe looks a bit weird, at least to me.

To clarify (mostly for myself):

• You do have a node.js server serving the /api/bats_hook endpoint using POST on HTTP.
• There are jobs running that access that enpoint now and then to obtain information.
• You ask for a way to obtain that response (= job data) using Python.

The conditions can only hold if the job is the Python part that sends the POST request to your node.js server and receives the JSON on HTTP 200. Otherwise you'd try to wiretap to get foreign requests and their corresponding responses from the node.js server without being the actual communication partner.

If the job code is Python and you want to properly POST the node.js service, use the requests Python package. Sending a request and obtaining the JSON response is two lines of code then.

If the job code is not under your control or not the location where you want the Python "hook" to be active, that either smells like bad architecture or misusing a setup a way it's not intended. Please clarify.