CODEWITHSUNDEEP
phpmicroservicesfile-handlingapi-design
SeaFuzz
SeaFuzz

Reputation: 1237

How can I handle File Uploads in a Microservice Environment?

I am in the process of trying to make a decision on how, when and where to handle uploaded files from users. We are in a MicroService environment (PHP + Linux) for a new system to be deployed in the coming months. One key component is incoming files.

Currently as I see it There are 3 options (Maybe more that I am not yet aware of). They are as follows:

(1)

[CLIENT:file] -> 
    [GATEWAY API 
        FILE STORAGE HANDLER ->
        [a: MICROSERVICE-News]
        [b: MICROSERVICE-Authors]
        [c: MICROSERVICE-Logger]
    ] -> {response}`

In this scenario, the Gateway API is designed to handle talking directly with a storage service (S3, GCS), sets a filename, validates, etc. When a storage confirmation is received it then passes that filename and other data to other MicroServices as needed. I see this as being overall beneficial as the file is handled as soon as it's received and can fail without impacting anything else further down the line. It does however add complexity to the Gateway and can potentially slow things down quickly at peak times.

(2)

[CLIENT:file] ->
    [GATEWAY API
        [a: MICROSERVICE-Files]
        [b: MICROSERVICE-News]
        [c: MICROSERVICE-Authors]
        [d: MICROSERVICE-Logger]
    ] -> {response}

In this scenario, the file is received by the Gateway API and then has to pass it along to a files MicroService. This can be beneficial because it takes visibility away from the gateway and offers the flexibility of easily making changes inside of a service without impacting the gateway for instance. The major downside of this is that now a single file is being handled twice and will require computing additional resources.

(3)

[CLIENT:file] -> 
    [FILE API] -> {response} -> 
[CLIENT] -> 
    [GATEWAY API
        [a: MICROSERVICE-News]
        [b: MICROSERVICE-Authors]
        [c: MICROSERVICE-Logger]
    ] -> {response}

In this scenario, the Client is accountable for sending files to a separate service and using the response to send to the Gateway API. From a resource perspective this takes a huge load off of the Gateway API and allows it to only be concerned with data, never files. The major drawback of this is a Client can send faulty or malicious information to the Gateway API and would require additional validation to ensure the file is valid and exists. It also creates potential congruence problems in the future between services and clients.

I may be missing other options and would love to know if there are. Does anyone have experience with this and how did you solve or approach handling files in a MicroService architecture?

Upvotes: 21

Views: 9807

Answers (3)

Georg Schwarz
Georg Schwarz

Reputation: 164

I think to those architectural questions there is no one-fits-all solution, it always depends on your context and quality goals.

If you prefer the encapsulation over performance, then go with solution (2). You might want to consider using a client-based service discovery mechanism for the file service instead of a full-blown API-Gateway to reduce the load on the gateway.

If you prefer performance and the client is under your control, then you can go with solution (3).

I'd avoid solution (1), though. Microservices have a principle "keep the smarts in the endpoints", meaning to avoid putting logic into infrastructure components like API-Gateways.

Upvotes: 1

Fatih Erol
Fatih Erol

Reputation: 699

After a long research, I finally got a solution.

Scenario 3 is appropriate for you and you can skip the Gateway api as it says in @Tree

[CLIENT:file] -> 
    [FILE API] -> {response} -> 
[CLIENT] -> 
    [GATEWAY API
        [a: MICROSERVICE-News]
        [b: MICROSERVICE-Authors]
        [c: MICROSERVICE-Logger]
    ] -> {response}

Example Scenario

  • user requests file upload

Sample Request

POST https://api.site.com/media/upload

Sample Response

{
    "location": "JPG-IMG-54f022ae8b3f4d979e925b4dp68e87"
}
  • The user sends the request to share the news

Sample Request

POST https://api.site.com/v2/news

{
    "Text": "Foo and Bar",
    "location": "JPG-IMG-54f022ae8b3f4d979e925b4dp68e87"
}
  • Uploaded files are controlled by api gateway

Sample Request

Get https://api.site.com/v2/JPG-IMG-54f022ae8b3f4d979e925b4dp68e87
  • After verification, each microservice performs its own task

Upvotes: 1

Tree
Tree

Reputation: 31361

[CLIENT:file] -> 
    [FILE API]
     -> {response}
     -> 
        [a: MICROSERVICE-News]
        [b: MICROSERVICE-Authors]
        [c: MICROSERVICE-Logger]
    ] -> {publish}

You can skip Gateway api in this example.

You File api verifies the file upload, and the data content. After it is succesfull, you make a response to a client that the upload was succesfull.

Later you process the upload in your microservices, and you use a sort of publisher to notify the client if for this data adds new actions in your other services.

In case your client depends on the processing of data, you have then 2 steps. First step is file upload response, that tells you that file was succesfully uploaded and verified.

Here you can react to this information by changing the gui, or still wait for the publisher information. Publisher will send the imlpementation of the file information with either success or error, so you can finish your steps with your client

Upvotes: 0

Related Questions