CODEWITHSUNDEEP
file
Ilya
Ilya

Reputation: 61

File writes per second

I want to log visits to my website with a high visits rate to file. How much writes to log file can I perform per second?

Upvotes: 4

Views: 1300

Answers (8)

mhaller
mhaller

Reputation: 14222

If your hard disk drive can write 40 MB/s and your log file lines are approx. 300 bytes in length, I'd assume that you can write 140000 HTTP requests per second to your logfile if you keep it open.

Anyway, you should not do that on your own, since most web servers already write to logfiles and they know very good how to do that, how to roll the files if a maximum limit is reached and how to format the log lines according to some well-known patterns.

Upvotes: 2

Rob Wells
Rob Wells

Reputation: 37149

If using Apache, I'd recommend using the rotatelogs utility supplied as a part of the standard kit.

We use this to allow rotating the server logs out on a daily basis without having to stop and start the server. N.B. Use the new "||" syntax when declaring the log directive.

The site I'm involved with is one of the largest on the Internet with hit rates peaking in the millions per second for extended periods of time.

Edit: I forgot to say that the site uses standard Apache logging directives and we have not needed to customise the Apache logging code at all.

Edit: BTW Unless you really need it, don't log bytes served as this causes all sorts of issues around the midnight boundary.

Upvotes: 1

aardbol
aardbol

Reputation: 2295

Or just analyze the Apache access log files. For example with AWStats.

Upvotes: 4

Paul Nathan
Paul Nathan

Reputation: 40319

Let Apache do it; do the analysis work on the back-end.

Upvotes: 0

Antoine P.
Antoine P.

Reputation: 4315

File writes are not expensive until you actually flush the data to disk. Usually your operating system will cache things aggressively so you can have very good write performance if you don't try to fsync() your data manually (but of course you might lose the latest log entries if there's a crash).

Another problem however is that file I/O is not necessarily thread-safe, and writing to the same file from multiple threads or processes (which will probably happen if we're talking about a Web app) might produce the wrong results: missing or duplicate or intermingled log lines, for example.

Upvotes: 2

Sampson
Sampson

Reputation: 268424

Don't do that, use Google Analytics instead. You'd end up running into many problems trying to open files, write to them, close them, so on and so forth. Problems would arise when you overwrite data that hasn't yet been committed, etc.

If you need your own local solution (within a private network, etc) you can look into an option like AWStats which operates off of crawling through your log files.

Upvotes: 4

Colin O'Dell
Colin O'Dell

Reputation: 8647

File access is very expensive, especially when doing writes. I would recommend saving them to RAM (using whatever cache method suits you best) and periodically writing the results to disk.

You could also use a database for this. Something like:

UPDATE stats SET hits = hits + 1

Try out a couple different solutions, benchmark the performance, and implement whichever works fast enough with minimal resource usage.

Upvotes: 1

Adam Crossland
Adam Crossland

Reputation: 14213

If you can't use Analytics, why wouldn't you use your webserver's existing logging system? If you are using a real webserver, it almost certainly as a logging mechanism that is already optimized for maximum throughput.

Your question is impossible to answer in all other respects. The number of possible writes is governed by hardware, operating system and contention from other running software.

Upvotes: 8

Related Questions