I need to implement some atomic writes to secondary storage. How can I make this fool proof? If I open a C# file handle using File.Open I will receive a handle. I can write some data to it. Flush it and close it. But I still have some questions. I guess the below statements are true? Data might not be written to disk but rather exist in the Windows Disk cache Data might not be written to disk but rather exist in the HDD cache And this will lead to the following issues: A power outage will make the edits in the file I made reverted (On a transactional FS like NTFS) A kernel-panic will make the edits in the file I made reverted (On a transactional FS like NTFS) Am I correct in my assumptions? If so, how can I make a fool proof write to the disk? I have looked a little bit into NoSQL and have been thinking there might be a nosql server that could talk to the system closer to the hardware and not return to me software until it can guarantee that the bytes are written to disk. All ideas and thoughts are welcome Jens [edit] Maybe there is a specific amount of time I can wait before being sure that all changes are written to physical disk?

www.jensolsson.se

Reputation: 3083

fs.write(); fs.flush(); When is it really written to disk? What if kernel-panic or power outage?

I need to implement some atomic writes to secondary storage. How can I make this fool proof?

If I open a C# file handle using File.Open I will receive a handle. I can write some data to it. Flush it and close it. But I still have some questions. I guess the below statements are true?

Data might not be written to disk but rather exist in the Windows Disk cache
Data might not be written to disk but rather exist in the HDD cache

And this will lead to the following issues:

A power outage will make the edits in the file I made reverted (On a transactional FS like NTFS)
A kernel-panic will make the edits in the file I made reverted (On a transactional FS like NTFS)

Am I correct in my assumptions? If so, how can I make a fool proof write to the disk? I have looked a little bit into NoSQL and have been thinking there might be a nosql server that could talk to the system closer to the hardware and not return to me software until it can guarantee that the bytes are written to disk.

All ideas and thoughts are welcome

Jens

[edit] Maybe there is a specific amount of time I can wait before being sure that all changes are written to physical disk?

Upvotes: 0

Answers (2)

Gregory A Beamer

Reputation: 17010

The only way to make an operation fully "fool proof" is queue, run the operation, and confirm. Things stay in queue and can be run again, until confirmed, or if the confirmation is negative "rolled back".

The window of time you are talking about, assuming you are not involving a network (everything is local), is very small. Still, if you want to ensure things, you queue them. MSMQ is one option. If the data comes from SQL Server, you can consider it's queueing mechanism Service Broker (not recommending this direction, but it is one way).

Ultimately, the idea here is a lot like a handshake, as used in most server to server communication. Everyone agrees things are done before both sides get rid of their piece of the work.

Upvotes: 1

NateTheGreat

Reputation: 2305

I am not an expert in Windows internals, but I believe you are correct. I didn't test it in great detail, but was able to use MSMQ as a pretty reliable place to store data, with another process that monitored the queue for final processing.

Upvotes: 0

fs.write(); fs.flush(); When is it really written to disk? What if kernel-panic or power outage?

Answers (2)

Related Questions