Retrieve correct data with two consecutive calls to boost::asio::read

Question

I am currently implementing a network protocol with Boost Asio. The domain classes already exist and I am able to

• write packets to a std::istream
• and read packets from a std::ostream.

A Network Packet contains a Network Packet Header. The header starts with the Packet Length field, which has a size of two bytes (std::uint16_t).

I am using TCP/IPv4 as the transport layer, therefore I try to implement the following:

1. Read the length of the packet to know its total length. This means reading exactly two bytes from the socket.
2. Read the rest of the packet. This means reading exactly kActualPacketLength - sizeof(PacketLengthFieldType) bytes from the socket.
3. Concat both read binary data.

Therefore I need at least two calls to boost::asio::read (I am starting synchronously!).

I am able to read a packet with one call to boost::asio::read if I hard-code the expected length:

Packet const ReadPacketFromSocket() {
    boost::asio::streambuf stream_buffer;    
    boost::asio::streambuf::mutable_buffers_type buffer{
        stream_buffer.prepare(Packet::KRecommendedMaximumSize)};
    std::size_t const kBytesTransferred{boost::asio::read(
        this->socket_,
        buffer,
        // TODO: Remove hard-coded value.
        boost::asio::transfer_exactly(21))};
    stream_buffer.commit(kBytesTransferred);
    std::istream input_stream(&stream_buffer);
    PacketReader const kPacketReader{MessageReader::GetInstance()};

    return kPacketReader.Read(input_stream);
  }

This reads the complete packet data at once and returns a Packet instance. This works, so the concept is working.

So far so good. Now my problem:

If I make two consecutive calls to boost::asio::read with the same boost::asio::streambuf I can't get it to work.

Here is the code:

Packet const ReadPacketFromSocket() {
  std::uint16_t constexpr kPacketLengthFieldSize{2};

  boost::asio::streambuf stream_buffer;    
  boost::asio::streambuf::mutable_buffers_type buffer{
      stream_buffer.prepare(Packet::KRecommendedMaximumSize)};

  std::size_t const kBytesTransferred{boost::asio::read(
      // The stream from which the data is to be read.
      this->socket_,
      // One or more buffers into which the data will be read.
      buffer,
      // The function object to be called to determine whether the read
      // operation is complete.
      boost::asio::transfer_exactly(kPacketLengthFieldSize))};

  // The received data is "committed" (moved) from the output sequence to the
  // input sequence.
  stream_buffer.commit(kBytesTransferred);
  BOOST_LOG_TRIVIAL(debug) << "bytes transferred: " << kBytesTransferred;
  BOOST_LOG_TRIVIAL(debug) << "size of stream_buffer: " << stream_buffer.size();

  std::uint16_t packet_size;
  // This does seem to modify the streambuf!
  std::istream istream(&stream_buffer);
  istream.read(reinterpret_cast<char *>(&packet_size), sizeof(packet_size));
  BOOST_LOG_TRIVIAL(debug) << "size of stream_buffer: " << stream_buffer.size();
  BOOST_LOG_TRIVIAL(debug) << "data of stream_buffer: " << std::to_string(packet_size);

  std::size_t const kBytesTransferred2{
      boost::asio::read(
          this->socket_,
          buffer,
          boost::asio::transfer_exactly(packet_size - kPacketLengthFieldSize))};
  stream_buffer.commit(kBytesTransferred2);

  BOOST_LOG_TRIVIAL(debug) << "bytes transferred: " << kBytesTransferred2;
  BOOST_LOG_TRIVIAL(debug) << "size of stream_buffer: " << stream_buffer.size();

  // Create an input stream with the data from the stream buffer.
  std::istream input_stream(&stream_buffer);

  PacketReader const kPacketReader{MessageReader::GetInstance()};

  return kPacketReader.Read(input_stream);
}

I have the following problems:

1. Reading the packet length from the boost::asio::streambuf after the first socket read seems to remove the data from the boost::asio::streambuf.
2. If I use two distinct boost::asio::streambuf instances I do not know how to "concat" / "append" them.

At the end of the day I need a std::istream with the correct data obtained from the socket.

Can someone please guide me into the correct direction? I've tried to make this work for several hours now...

Maybe this approach isn't the best, so I am open to suggestions to improve my design.

Thanks!

Answer 1

Before I forget, I want to summarize my current solution, which doesn't use a boost::asio::streambuf, since it seems to be impossible to read from it without modifying it. Instead I use a std::vector<std::uint8_t> (ByteVector) as the data holder for the buffers.

The following source code contains my current solution:

Packet const ReadPacketFromSocket() {
  ByteVector const kPacketLengthData{this->ReadPacketLengthFromSocket()};
  PacketHeader::PacketLengthType kPacketLength{
      static_cast<PacketHeader::PacketLengthType>(
          (kPacketLengthData[1] << 8) | kPacketLengthData[0])};

  ByteVector rest_packet_data(Packet::KRecommendedMaximumSize);
  boost::asio::read(
      this->socket_,
      boost::asio::buffer(rest_packet_data),
      boost::asio::transfer_exactly(
          kPacketLength - sizeof(PacketHeader::PacketLengthType)));

  ByteVector data{
      VectorUtils::GetInstance().Concatenate(
          kPacketLengthData,
          rest_packet_data)};

  // Create an input stream from the vector.
  std::stringstream input_stream;
  input_stream.rdbuf()->pubsetbuf(
      reinterpret_cast<char *>(&data[0]), data.size());

  PacketReader const kPacketReader{MessageReader::GetInstance()};

  return kPacketReader.Read(input_stream);
}

ByteVector ReadPacketLengthFromSocket() {
  ByteVector data_holder(sizeof(PacketHeader::PacketLengthType));

  boost::asio::read(
      this->socket_,
      boost::asio::buffer(data_holder),
      boost::asio::transfer_exactly(sizeof(PacketHeader::PacketLengthType)));

  return data_holder;
}

This works like a charm, I have successfully exchanged packets with messages from my domain model between two processes using this approach.

But: This solution feels wrong, since I have to do lots of conversions. Maybe someone else can provide me with a cleaner approach? What do you think about my solution?

Answer 2

1. I believe the behaviour is by design.

2. To concatenate the buffers, you can use BUfferSequences (using make_buffers) and use buffer iterators, or you can stream the second into the first:

   boost::asio::streambuf a, b;
   std::ostream as(&a);
   
   as << &b;
   

   Now you can throw away b as it's pending data have been appended to a

See it Live on Coliru