Reputation:

Implementing a virtual file system in .NET

A while back a found a great-looking framework that allowed .net developers to implement a virtual file system. I thought I had bookmarked it, but it seems I haven't.

Does anyone know any frameworks for doing this?

EDIT: Here's a hint... It had a catchy, short name and it's own domain. Sorry, that's all I can remember :p

Upvotes: 12

Answers (4)

IT Hit WebDAV

Reputation: 5894

Here I will try to summarize my knowledge of building Virtual File Systems for Windows, Mac, and iOS in .NET. This summary is mainly for file systems with slow remote storage, such as cloud storage.

First of all, let me point out that both applications and OS make a large number of reads and writes to the local file system and expect a very fast response from it. As a result, it is virtually impossible to map every file system call into the remote storage call. Even with a very high-speed connection (or local storage), your virtual file system may become very slow and unusable with such a design. I guess that the new OS API, provided in recent OS updates (2018-2021), as well as end-user tools, such as OneDrive, created on top of it, are guided by these considerations.

The new OS API for creating Virtual File Systems mainly relies on synchronization, rather than on every operation mapping, providing additional hooks for implementing on-demand folders listing, on-demand file download, delete and move operations.

Windows

Cloud Provider API

The Windows Cloud Provider API (also known as Cloud Files, Cloud Filter API, and Cloud Sync Engine) runs on top of a new Windows driver, installed as part of Windows 10 updates, released in the year 2018 (Windows Creator Update).

This API provides callbacks for on-demand folders listing (so you can load only a small portion of your server file system, requested by the client app), on-demand file content download as well as callbacks for delete and move/rename operations. It also provides functions to determine file status, that you will use to detect that the file was modified on the client and needs to e sent to the server. Another useful feature is that it provides integration with Windows File Manager, displaying file status column, file transfer progress, and posts file download events to Windows Actions Center panel. The API does not require admin privileges and allows all operations under a regular user, including the initial file system mounting.

Fortunately, Windows Cloud Provider API provides file Open and Close events, which may be vital for some applications, such as file locking/unlocking or check-out/check-in. Unlike on Mac OS, where Open/Close events are limited to certain application types, see in the Mac section below.

Example on GitHub in C++
Example on GitHub in C#.

Projected File System API

The Projected File System (ProjFS) is designed to represent (project) hierarchical data as a file system. It is designed to publish data from high-speed storage, such as a registry.

Its major feature is that unlike the Cloud Provider API it hides the fact that your storage is remote. It does not provide any indication of the file status, progress, or anything that will tell the user that this is not a local file system.

Example on GitHub in C#

Kernel-Mode File System Drivers and Filters

This is a traditional approach that existed for many years. It works on all Windows versions, even released 20 years ago. Building a stable driver requires a lot of special kernel-mode development and debugging experience, fighting system crashes, and a long development cycle. In return, it gives you 100% access to all file system functions at a low level. Probably the most appropriate usage of the kernel-mode drivers is for creating a Virtual File System for local hardware.

My concern is that creation of file system drivers/filters may be limited in future Windows versions. See below what Apple has done with Kext drivers on Mac.

Shell Namespace Extensions

Windows Shell Namespace Extensions allows you to customize the Windows File Manager. In particular, you can build nodes that will be displayed inside Windows File Manager and visually look and behave like a file system. You can also extend the Windows File Manager context menu using Shell Extensions. Shell Extension is not a real file system, applications will not be able to read/write data into your files (unless you put a real or some virtual file system under your shell extension). In many cases, you will create a Shell Extension on top of your Virtual File System to extend its functionality and add context menus.

MacOS

File Provider API for Mac and System Extensions.

Since macOS 11 Big Sur, Apple has provided a new API for syncing with cloud storage - File Provider API. Its main features are similar to the Cloud Provider API on Windows: on-demand folder listing, on-demand file content loading, file status (in the cloud/downloading/in the local file system, in-sync/modified), Mac OS Finder integration. The application that is using this API does NOT require admin privileges for installation and usage, can be installed in the per-user and per-machine mode as well as can be published in the Apple App Store.

Looks like the major issue with this API is that it does not provide file Open and file Close events. Without these events, it may be difficult or impossible to create some features, for example, to automatically lock/unlock or check-out/check-in file on open/close. For some reason, Mac provides Open/Close events for Endpoint Security system extensions, which requires special entitlement (for example antivirus scanner).

Example on GitHub in Xamarin/C#

Kext File System Drivers (deprecated).

Since macOS 11, Apple allows Kext drivers installation on ARM machines in Recovery mode only. They are also not allowed in the App store. Kexts are replaced with Dexts and System Extensions.

iOS

File Provider API for iOS

iOS 11+ provides File Provider API which is similar to File Provider API on Mac. Apple is trying to unify the API and make programs run on both Mac and iOS with minimum changes. The iOS and macOS API still has some differences that limit creating a single file provider for both OSs.

Example on GitHub in .NET/C#

Upvotes: 12

user47322

Reputation:

Found it just now (happy!)

Dokan

Upvotes: 1