Reputation: 1320

Encoding issues (i think) when reading serial interface

I am trying to read out my smart meter through a P1 interface. Some python scripts are posted on the web to do so, and I tried them, but seem to be getting some partial garbage out of my serial interface (not completely - so my assumption is that the serial settings are correct though. Also since I am copying this from other scripts that seem to be working).

When trying to readout the interface using cu command it does show the right output, so my hardware seems to work.

I am running this on: Linux version 3.10.25+ (dc4@dc4-arm-01) (gcc version 4.7.2 20120731 (prerelease) (crosstool-NG linaro-1.13.1+bzr2458 - Linaro GCC 2012.08) ) #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014

Here is what the output should look like - retrieved through:

command:

cu -l /dev/ttyUSB0 -s 9600 --parity=none

output

/INTENTIONALLY ALTERED
0-0:96.1.1(39373936353532302020202020202020)
1-0:1.8.1(05090.742*kWh)
1-0:1.8.2(06618.743*kWh)
1-0:2.8.1(00000.000*kWh)
1-0:2.8.2(00000.000*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(0000.71*kW)
1-0:2.7.0(0000.00*kW)
0-0:17.0.0(999*A)
0-0:96.3.10(1)
0-0:96.13.1()
0-0:96.13.0()
0-1:96.1.0(3238303131303038323033333632313132)
0-1:24.1.0(03)
0-1:24.3.0(140806220000)(2C)(60)(1)(0-1:24.2.0)(m3)
(03447.404)
0-1:24.4.0(1)
!

When I use the following Python code:

# DSMR P1 uitlezen
# (c) 10-2012 - GJ - gratis te kopieren en te plakken
versie = "1.0"
import sys
import serial


##############################################################################
#Main program
##############################################################################
print ("DSMR P1 uitlezen",  versie)
print ("Control-C om te stoppen")
print ("Pas eventueel de waarde ser.port aan in het python script")

#Set COM port config
ser = serial.Serial()
ser.baudrate = 9600
ser.bytesize=serial.SEVENBITS
ser.parity=serial.PARITY_EVEN
ser.stopbits=serial.STOPBITS_ONE
ser.xonxoff=0
ser.rtscts=0
ser.timeout=20
ser.port="/dev/ttyUSB0"

#Open COM port
try:
    ser.open()
except:
    sys.exit ("Fout bij het openen van %s. Aaaaarch."  % ser.name)


#Initialize
#p1_teller is mijn tellertje voor van 0 tot 20 te tellen
p1_teller=0

while p1_teller < 20:
    p1_line=''
#Read 1 line van de seriele poort
    try:
        p1_raw = ser.readline()
        print str(p1_teller),':', p1_raw
    except:
        sys.exit ("Seriele poort %s kan niet gelezen worden. Aaaaaaaaarch." % ser.name )

    #p1_str=str(p1_raw)
    #p1_line=p1_str.strip()
# als je alles wil zien moet je de volgende line uncommenten
    #print (p1_line.encode('ascii','ignore'))
    p1_teller = p1_teller +1

#Close port and show status
try:
    ser.close()
except:
    sys.exit ("Oops %s. Programma afgebroken. Kon de seriele poort niet sluiten." % ser.name )

The output turns into this:

('DSMR P1 uitlezen', '1.0')
Control-C om te stoppen
Pas eventueel de waarde ser.port aan in het python script
0 : INTENTIONALLY ALTERED BUT ALSO WITH THE ? SYMBOLS IN THE ORIGINAL OUTPUT

1 : �

2 : 0-0:96.�.�(393�393635353�30�0�0�0�0�0�0�0�0��

3 : �-0:�.�.�(05090.�9����詍

4 : �-0:�.�.�(066��.��3���詍

5 : �-0:�.�.�(00000.000���詍

6 : �-0:�.�.�(00000.000���詍

7 : 0-0:96.��.0(000���

8 : �-0:�.�.0(0000.�0��ש�

9 : �-0:�.�.0(0000.00��ש�

10 : 0-0:��.0.0(999�A��

11 : 0-0:96.3.�0(���

12 : 0-0:96.�3.�(��

13 : 0-0:96.�3.0(��

14 : 0-�:96.�.0(3�3�303�3�30303�3�303333363�3�3�3���

15 : 0-�:��.�.0(03��

16 : 0-�:��.3.0(��0�06��0000�(�é(60�(��(0-�:��.�.0�(�3��

17 : (03���.�0���

18 : 0-�:��.�.0(���

19 : !�

so there are a lot of � characters in my python output, that I expect to be some encoding issue.. but not sure and not sure how to fix this... so any help is appreciated.

Upvotes: 0

Answers (5)

increddibelly

Reputation: 1229

Thanks to Geoffrey I finally figured out how to setup my C# SerialPort.

private static SerialPort CreateP1SerialPort(string name)
{
    return new SerialPort
    {
        BaudRate = 115200,
        DataBits = 8,
        Handshake = Handshake.XOnXOff,
        Parity = Parity.None,
        PortName = name,
        RtsEnable = false,
        StopBits = StopBits.One,
        ReadTimeout = 10 * 1000,
        WriteTimeout = 1000
    };
}

simple example:

public static void Main()
{
   // this will help you find your com port name
   var portNames = SerialPort.GetPortNames().OrderBy(x => x).ToArray();

   // Create a new SerialPort object with default settings.
   var serialPort = CreateP1SerialPort("COM7");
   serialPort.Open(); 

   while (true)
   {
       var line = serialPort.ReadExisting();
       if (string.IsNullOrEmpty(line))
       {
           Thread.Sleep(10000); // one message every 10 seconds
       } else 
           Console.WriteLine(line);
    }
}

Upvotes: 0

LdeV

Reputation: 21

I guess it has to do with some decoding of data. Using textIOWrapper the decoding works fine for me. Hopefully you can do anything with my view on the case:

To make the readout of the p1 protocol as easy as possible I'd suggest to use TextIOWrapper as I mentioned before, this way you can still read the serial port with the readline method. The "!" always ends the P1 telegram, so that can be used to detect the end of the message instead of using a counter. When a full telegram has been received, the telegram can be processed. Example:

import io
import serial

serialport = serial.Serial(  # Configure Serial communication port
                            port     = "/dev/ttyUSB0"
                            baudrate = 9600,
                            timeout  = 11,
                            bytesize = serial.SEVENBITS,
                            parity   = serial.PARITY_EVEN,
                            stopbits = serial.STOPBITS_ONE )

p1port = io.TextIOWrapper(io.BufferedReader(serialport, buffer_size=1), newline='\n', encoding='ascii')

P1Message = []

while True:
    try:
         rawline = p1port.readline()
    except UnicodeDecodeError:
         print "Encode error on readline"

    if '!' in rawline:
        # Process your P1Message here

        P1Message = [] # Clear message, wait for new one
    else:
        P1Message.append(rawline)

Good luck!

Upvotes: 1

Geoffrey Stoel

Reputation: 1320

This is how I in the end fixed it - Will try Ldevries method too.... looks pretty clean:

#
# MBSolget P1 Telegram Catch
#

version = "v1.00"
import sys
import os
import stat
import serial
import datetime
import locale

###############################################################################################################
# Main program
###############################################################################################################
#Initialize
p1_telegram  = False
p1_timestamp = ""
p1_teller    = 0
p1_log       = True

#Set COM port config
ser          = serial.Serial()
ser.baudrate = 9600
ser.bytesize = serial.EIGHTBITS
ser.parity   = serial.PARITY_NONE
ser.stopbits = serial.STOPBITS_ONE
ser.xonxoff  = 1
ser.rtscts   = 0
ser.timeout  = 30
ser.port     = "/dev/ttyUSB0"

#Show startup arguments
print ("MBSolget P1 Telegram Catch %s" % version)
print ("Control-C om af te breken")
print ("Poort: (%s)" % (ser.name) )

#Open COM port
try:
    ser.open()
except:
    sys.exit ("Fout bij het openen van poort %s. "  % ser.name)

while p1_log:
    p1_line = ''
    try:
        p1_raw = ser.readline()
    except:
        sys.exit ("Fout bij het lezen van poort %s. " % ser.name )
        ser.close()


    p1_raw = ''.join(chr(ch & 0x7f) for ch in p1_raw)
#    for ch in p1_raw:
       # print(chr(ch & 0x7f))
#        print( chr(ord(ch) & 0x7f))
#    print(p1_raw)
    p1_str  = p1_raw #str(p1_raw ,"utf-8",errors="ignore")
    p1_line = p1_str.strip()
    print (p1_line)

    if p1_line[0:1] == "/":
        p1_telegram = True
        p1_teller   = p1_teller + 1
        f=open("/home/geoffrey/p1_temp.log", "w")
    elif p1_line[0:1] == "!":
        if p1_telegram:
            p1_teller   = 0
            p1_telegram = False
            p1_log      = False
            f.write (p1_line)
            f.write ('\r\n')
            f.close()
            os.chmod("/home/geoffrey/p1_temp.log", stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)
    if p1_telegram:
        f.write (p1_line)
        f.write ('\r\n')

#Close port and show status
try:
    ser.close()
except:
    sys.exit ("Fout bij het sluiten van %s. Programma afgebroken." % ser.name )

Upvotes: 1

Mark Ransom

Reputation: 308121

It looks like the bytes you read may still contain the parity bit, making them invalid characters. Try this to remove the 8th bit:

p1_raw = ''.join(chr(ord(ch) & 0x7f) for ch in p1_raw)

Upvotes: 1

user3911174

Reputation: 1

In the code, change the byte size to EIGHTBITS and parity to PARITY_NONE. Those match what cu is using (and are also the defaults for PySerial).

Upvotes: 0

Encoding issues (i think) when reading serial interface

Answers (5)

Related Questions