Reading more than 2048 bytes from QLocalSocket

Question

I have a problem reading more than 2048 bytes from a QLocalSocket. This is my server-side code:

clientConnection->flush();  // <-- clientConnection is a QLocalSocket

QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);

out.setVersion(QDataStream::Qt_5_0);
out << (quint16)message.size() << message;  // <--- message is a QString

qint64 c = clientConnection->write(block);
clientConnection->waitForBytesWritten();
if(c == -1)
    qDebug() << "ERROR:" << clientConnection->errorString();

clientConnection->flush();

And this is how I read the data in my client:

QDataStream in(sock);  // <--- sock is a QLocalSocket
in.setVersion(QDataStream::Qt_5_0);

while(sock->bytesAvailable() < (int)sizeof(quint16)){
    sock->waitForReadyRead();
}
in >> bytes_to_read; // <--- quint16

while(sock->bytesAvailable() < (int)bytes_to_read){
    sock->waitForReadyRead();
}

in >> received_message;

The client code is connected to the readyRead signal and it's disconnected after the first call to the slot.

Why I'm able to read only 2048 bytes?

==EDIT==

After peppe's reply I updated my code. Here is how it looks now:

server side code:

clientConnection->flush();

QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);

out.setVersion(QDataStream::Qt_5_0);

out << (quint16)0;
out << message;
out.device()->seek(0);
out << (quint16)(block.size() - sizeof(quint16));
qDebug() << "Bytes client should read" << (quint16)(block.size() - sizeof(quint16));

qint64 c = clientConnection->write(block);
clientConnection->waitForBytesWritten();

client side code:

QDataStream in(sock);
in.setVersion(QDataStream::Qt_5_0);

while(sock->bytesAvailable() < sizeof(quint16)){
    sock->waitForReadyRead();
}

quint16 btr;
in >> btr;

qDebug() << "Need to read" << btr << "and we have" << sock->bytesAvailable() << "in sock";
while(sock->bytesAvailable() < btr){
    sock->waitForReadyRead();
}

in >> received_message;

qDebug() << received_message;

I'm still not able to read more data.

Answer 1

out.setVersion(QDataStream::Qt_5_0);
out << (quint16)message.size() << message;  // <--- message is a QString

This is wrong. The serialized length of "message" will be message.size() * 2 + 4 bytes, as QString prepends its own length as a quint32, and each QString character is actually a UTF-16 code unit, so it requires 2 bytes. Expecting only message.size() bytes to read in the reader will cause QDataStream to short read, which is undefined behaviour.

Please do check the size of "block" after those lines -- it'll be 2 + 4 + 2 * message.size() bytes. So you need to fix the math. You can safely assume it won't change, as the format of serialization of Qt datatypes is known and documented.

I do recognize the "design pattern" of prepending the length, though. It probably comes from the fortune network example shipped with Qt. The notable difference there is that the example doesn't prepend the length of the string in UTF-16 code units (which is pointless, as it's not how it's going to be serialized) -- it prepends the length of the serialized QString. Look at what it does:

    out << (quint16)0;
    out << fortunes.at(qrand() % fortunes.size());
    out.device()->seek(0);
    out << (quint16)(block.size() - sizeof(quint16));

First it reserves some space in the output, by writing a 0. Then it serializes a QString. Then it backtracks and overwrites the 0 with the length of the serialized QString -- which at this point, is exactly block.size() minus the prepended integer stating the lenght (and we know that the serialized length of a quint16 is sizeof(quint16))

To repeat myself, there actually two reasons about why that example was coded that way, and they're somehow related:

1. QDataStream has no means to recover from short reads: all the data it needs to successfully decode an object must be available when you use the operator>> to deserialize the object. Therefore, you cannot use it before being sure that all data was received. Which brings us to:
2. TCP has no built in mechanism for separating data in "records". You can't just send some bytes followed by a "record marker" which tells the receiver that he has received all the data pertinent to a record. What TCP provides is a raw stream of bytes. Eventually, you can (half-)close the connection to signal the other peer that the transmission is over.

1+2 imply that you must use some other mechanism to know (on the receiver side) if you already have all the data you need or you must wait for some more. For instance, you can introduce in-band markers like \r\n (like IRC or - up to a certain degree - HTTP do).

The solution in the fortune example is prepending to the "actual" data (the serialized QString with the fortune message) the length, in bytes, of that data; then it sends the length (as a 16 bit integer) followed by the data itself.

The receiver first reads the length; then it reads up that many bytes, then it knows it can decode the fortune. If there's not enough data available (both for the length - i.e. you received less than 2 bytes - and the payload itself) the client simply does nothing and waits for more.

Note that:

• the design ain't new: it's what all most protocols do. In the "standard" TCP/IP stack, TCP, IP, Ethernet and so on all have a field in their "headers" which specify the lenght of the payload (or of the whole "record");
• the transmission of the "length" uses a 16bit unsigned integer sent in a specific byte order: it's not memcpy()d into the buffer, but QDataStream is used on it to both store it and read it back. Although it may seem trivial, this actually completes the definition of the protocol you're using.
• if QDataStream had been able to recover from short reads (f.i. by throwing an exception and leaving the data in the device), you would not have needed to send the length of the payload, since QDataStream already sends the length of the string (as a 32 bit unsigned bigendian integer) followed by the UTF-16 chars.