CODEWITHSUNDEEP
phpapiarchitecturecurlmessage-queue
Serge
Serge

Reputation: 1601

How to send a big set of data from one server to another?

There are two LAMP servers, A and B, in different countries. Once a day A has an updated set of data, that is to be processed on B. What would be the best way to set a communication between them?

Estimated size of an update is 20Mb compressed.

Options that comes to my mind are:

  1. curl POST request, uploading compressed data in JSON;
  2. SQL dump of a tmp table, compressed, then rsync'ed to B. B watching specific folder for update. Then importing the table and processing;
  3. Message Queuing with RabbitMQ or Amazon Simple Queue;
  4. ..other option?

Should you have any experience with a similar tasks, please share your thoughts.

Upvotes: 1

Views: 185

Answers (1)

moxy
moxy

Reputation: 1624

option 2, either if they are on a network or even using ftp, you can use cronjobs to run tasks at scheduled hours (probably during the night).

Upvotes: 1

Related Questions