CODEWITHSUNDEEP
pythonpython-3.x
Ashok Developer
Ashok Developer

Reputation: 381

paramiko module is not catching the output

I'm trying to get Kafka lags using paramiko module to catch sum of the kafka lag but the output always giving zero using python paramiko ssh module, while manually same command giving the correct result.

    # check list of consumers
        stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --list")
        list_consumers = stdout.readlines()
        print(list_consumers)
# check lag for each consumer
        for consumer in list_consumers:
            print(consumer)
            # | awk '{sum += $5} END {print sum}'
            stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {{0}} | awk '{{sum+=$5}} END {{print sum}}' ".format(consumer))
            consumer_lag_out = stdout.readlines()
            print(consumer_lag_out)

sample output enter image description here

manual shell command :

enter image description here

Upvotes: 0

Views: 65

Answers (1)

micromoses
micromoses

Reputation: 7507

You have double curly-brackets in the second ssh command. Change from --group {{0}} | awk to --group {0} | awk
The full line should be (EDIT: see followup):

stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {0} | awk '{{sum+=$5}} END {{print sum}}' ".format(consumer))

This way, {0} is replaced by format with consumer (which you redacted in the image, but looks like it should start with adapter.). Otherwise, consumer is ignored, and the actual command sent is:

/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {0} | awk '{sum+=$5} END {print sum}'

When format encounters double-curly-brackets, it simply strips the outer ones. It will not error in this case. For example:

>>> consumer = "adapter"
>>> print("value = {{0}}, some brackets = {{inside}}".format(consumer))
value = {0}, some brackets = {inside}
>>> print("value = {0}, some brackets = {{inside}}".format(consumer))
value = adapter, some brackets = {inside}

Following up with further comments, newlines should be stripped from original output. While the sample image is redacted so much so it hard to read, a simple POC can illustrate the situation:

>>> lines = ssh.exec_command("ls / | grep etc")[1].readlines()
>>> lines
['etc\n']
>>> consumer = lines[0]
>>> _, stdout, stderr = ssh.exec_command("ls /{0} | grep init.d".format(consumer))
>>> len(stdout.readlines())
106
>>> # It is unlikely we have 106 files with 'init.d' under /etc/ .. Let's check for errors:
>>> print(stderr.readlines())
["bash: -c: line 2: syntax error near unexpected token `|'\n", "bash: -c: line 2: ` | grep init.d'\n"]
>>> # Now let's try without newlines:
>>> _, stdout, stderr = ssh.exec_command("ls /{0} | grep init.d".format(consumer.strip()))
>>> print(stdout.readlines())
['init.d\n']
>>> # That's more like it.

Therefore, the full line should be:

stdin, stdout, stderr = ssh.exec_command("/home/prduser/kafka_2.12-2.1.0/bin/kafka-consumer-groups.sh --bootstrap-server localhost:19092,localhost:29092,localhost:39092 --describe --group {0} | awk '{{sum+=$5}} END {{print sum}}' ".format(consumer.strip()))

Upvotes: 1

Related Questions