CODEWITHSUNDEEP
apiservicefile-storage
Kasbolat Kumakhov
Kasbolat Kumakhov

Reputation: 721

Suggestion about creating custom File Storage Service

In my project one of the services is a File Storage service with just 2 tasks: upload file and get an ID, download file by ID. Nothing else. No fancy stuff, like UI, encryption (HTTPS will suffice), 100% availability, deduplication, etc. Just 2 simple tasks.

Using ASP .Net Core this service takes half an hour to create and start using, since it will be used only in organization's local network.

But there are a few things i would like to consider before start:

  1. Which Db engine?
  2. How to store files?

While investigating, i came to 2 variants:

  1. SQLite + storing files in subfolders like year/month/day/guid.bin. The database will contain full path to the file with an appropriate GUID.
  2. LiteDB and store everything in it.

Pros of (1):

Cons of (1):

Pros of (2):

Cons of (2):

Also, picking an embedded file-based database is forcing to use some manual backuping (third party service) and relying only on that. It makes things simple to develop and deploy (for example, files will always be in sync with database when backuping), but hard on the other end.

What would be the best choice? Or perhaps there are more better solutions? Perhaps a simple third-party service without a lot of bloating of functionality?

Upvotes: 0

Views: 440

Answers (1)

hedberggames
hedberggames

Reputation: 116

Assuming multiple users will use the same database, I recommend using a database server (Sql Server / MongoDB) instead of a file-based database (SqlLite/LiteDb). Even if it's not a separate physical server, using technology designed to be accessed by multiple simultaneous clients will give you much better scalability. File-based databases are typically designed for single-user scenarios and will quickly become a bottleneck since only a single thread can access the file at a time.

Regarding whether to use Sql or NoSql - I recommend designing the code you write to minimize the cost of changing your mind. Your current list of requirements don't seem to benefit much from picking one over the other, and, in my personal experience, this situation typically leads to one of two outcomes:

  1. the project is used in a limited capacity and the choice of technology didn't matter
  2. the project is adopted widely, being used in ways we couldn't have anticipated, and now we're rushing to catch up

In either case, time spent up-front trying to decide doesn't do much good. That being said, it is still worthwhile to research up-front (as you have) since prior knowledge is an excellent tool for making good decisions while under time pressure.

Lastly, you didn't mention anything about security - if your organization has security experts I highly recommend working closely with them as you design the system. Arbitrary file uploads can be used as an attack vector, so validating the contents and destination of an upload is very important.

Upvotes: 1

Related Questions