CODEWITHSUNDEEP
gogrpc
Aakash Goyal
Aakash Goyal

Reputation: 1100

Send huge data from client to server

I am working with Go and Grpc. I have one XML file to be sent from client to server. When server calls a method on client, client responds back with an XML document which could range from few kB to 100s MB or may be more. Please help me with how should I do it? Should I

  1. assign the value to a string variable and send normally -> rpc GetCMData (CmRequest) returns (CmReply) {}
  2. assign the value to a string variable and send as a stream -> rpc GetCMData (CmRequest) returns (stream CmReply) {}
  3. Is there any other better way to do so?

This is how CmReply looks like:

message CmReply {
  string name = 1;
}

Upvotes: 4

Views: 5091

Answers (2)

Vitaly Isaev
Vitaly Isaev

Reputation: 5815

You have two options:

  1. Implement client-side streaming protocol in the way you have described above (split your file into chunks and send them one-by-one to the server): https://github.com/grpc/grpc-go/issues/414#issuecomment-284515856
  2. Extend the default 4 MB limits for a message size for your GRPC client and server (then you'll be able to send and receive big messages without streaming): https://godoc.org/google.golang.org/grpc#MaxMsgSize https://godoc.org/google.golang.org/grpc#WithMaxMsgSize

Upvotes: 1

Citrullin
Citrullin

Reputation: 2321

The second one is more the way you want to go. But another point: Are you sure you want to have these hundreds of megabytes in your RAM? I'm not the Go-Geek, but I generally would prefer to work on a File Stream. And probably implement something like a back-pressure-system.

If you work with streams: I would also implement something to make sure that you can handle a break of the stream. Let's say you have the half of your stream and the stream breaks. You don't want to start again and get all previous stream elements, which you already have on the client side. You risk again a connection break within the stream. Maybe at the same point. So, something like pagination. Let's say 

message CMRequest{
  int startAt = 0;
}

message CmReply {
  string id = 0;
  string name = 1;
}

If the stream breaks, you send a new request with startAt= LastCmReply.id+1

Here's a video of some good practices in grpc: https://www.youtube.com/watch?v=Z_yD7YPL2oE&index=17&list=WL

Upvotes: 2

Related Questions