Is reading a file on UNIX faster than writing a file?
A rather obtuse performance bottleneck has boiled down to this very small query. I have done some empirical analysis and I think I might be getting victimized by disk caching strategies. Fundamentally (if disk caching were disabled) would writing a file be as fast, slower, or faster than reading a file? I would assume that the answer would depend on the fragmentation (and file size) but the operation to write a file would have to do an additional look up of where the next free block is rather than just following the pointer to it.
files filesystems hard-disk
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
asked Aug 22 '14 at 18:11
ojblass
1589
add a comment |
A rather obtuse performance bottleneck has boiled down to this very small query. I have done some empirical analysis and I think I might be getting victimized by disk caching strategies. Fundamentally (if disk caching were disabled) would writing a file be as fast, slower, or faster than reading a file? I would assume that the answer would depend on the fragmentation (and file size) but the operation to write a file would have to do an additional look up of where the next free block is rather than just following the pointer to it.
files filesystems hard-disk
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
asked Aug 22 '14 at 18:11
ojblass
1589
take a look at here
– Networker
Aug 22 '14 at 18:17
@ojblass, please check the edit and change it if you feel it is incorrect.
– Ramesh
Aug 22 '14 at 19:53
add a comment |
A rather obtuse performance bottleneck has boiled down to this very small query. I have done some empirical analysis and I think I might be getting victimized by disk caching strategies. Fundamentally (if disk caching were disabled) would writing a file be as fast, slower, or faster than reading a file? I would assume that the answer would depend on the fragmentation (and file size) but the operation to write a file would have to do an additional look up of where the next free block is rather than just following the pointer to it.
files filesystems hard-disk
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
asked Aug 22 '14 at 18:11
ojblass
1589
A rather obtuse performance bottleneck has boiled down to this very small query. I have done some empirical analysis and I think I might be getting victimized by disk caching strategies. Fundamentally (if disk caching were disabled) would writing a file be as fast, slower, or faster than reading a file? I would assume that the answer would depend on the fragmentation (and file size) but the operation to write a file would have to do an additional look up of where the next free block is rather than just following the pointer to it.
files filesystems hard-disk
files filesystems hard-disk
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
asked Aug 22 '14 at 18:11
ojblass
1589
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
asked Aug 22 '14 at 18:11
ojblass
1589
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
edited Dec 15 at 21:58
Rui F Ribeiro
38.9k1479129
38.9k1479129
asked Aug 22 '14 at 18:11
ojblass
1589
asked Aug 22 '14 at 18:11
ojblass
1589
asked Aug 22 '14 at 18:11
ojblass
1589
1589
take a look at here
– Networker
Aug 22 '14 at 18:17
@ojblass, please check the edit and change it if you feel it is incorrect.
– Ramesh
Aug 22 '14 at 19:53
add a comment |
take a look at here
– Networker
Aug 22 '14 at 18:17
@ojblass, please check the edit and change it if you feel it is incorrect.
– Ramesh
Aug 22 '14 at 19:53
take a look at here
– Networker
Aug 22 '14 at 18:17
take a look at here
– Networker
Aug 22 '14 at 18:17
@ojblass, please check the edit and change it if you feel it is incorrect.
– Ramesh
Aug 22 '14 at 19:53
@ojblass, please check the edit and change it if you feel it is incorrect.
– Ramesh
Aug 22 '14 at 19:53
add a comment |
2 Answers
2
active
oldest
votes
It depends. There is no general answer to this question.
In the absence of caching, writing a disk file is usually measurably slower than reading. This has little to do with the operating system and everything to do with the hardware: both hard disks and solid state media read faster than they write. A secondary factor is related to filesystem structure: reading only needs to traverse the directory tree and block list down to the data, then read the data, whereas writing needs to perform the same traversal, then write the data, then update some metadata.
When caching comes into play, things change. Reading data that's in cache is very fast, but reading data that isn't in cache has to go and fetch it from the disk. Operating systems might try to anticipate reads, but that only works in very specific cases (mainly sequential reads from a file). Writing, on the other hand, can be near-instantaneous as long as the amount of data isn't too large, as the data is only written to a memory buffer. The buffer has to be written to disk eventually, but by that time your application has already moved on to do more stuff.
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
add a comment |
Writing should be faster.
The free block list is kept in memory, so finding the next free block will be very fast. Unless you're writing in synchronous mode, when you try to write something it will simply copy the data into a kernel buffer and queue the write; it doesn't have to wait for the I/O to complete.
On the other hand, a read has to wait for the I/O to complete, since the caller can't do anything until the data arrives.
answered Aug 22 '14 at 19:33
Barmar
6,9081223
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f151665%2fis-reading-a-file-on-unix-faster-than-writing-a-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
It depends. There is no general answer to this question.
In the absence of caching, writing a disk file is usually measurably slower than reading. This has little to do with the operating system and everything to do with the hardware: both hard disks and solid state media read faster than they write. A secondary factor is related to filesystem structure: reading only needs to traverse the directory tree and block list down to the data, then read the data, whereas writing needs to perform the same traversal, then write the data, then update some metadata.
When caching comes into play, things change. Reading data that's in cache is very fast, but reading data that isn't in cache has to go and fetch it from the disk. Operating systems might try to anticipate reads, but that only works in very specific cases (mainly sequential reads from a file). Writing, on the other hand, can be near-instantaneous as long as the amount of data isn't too large, as the data is only written to a memory buffer. The buffer has to be written to disk eventually, but by that time your application has already moved on to do more stuff.
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
add a comment |
It depends. There is no general answer to this question.
In the absence of caching, writing a disk file is usually measurably slower than reading. This has little to do with the operating system and everything to do with the hardware: both hard disks and solid state media read faster than they write. A secondary factor is related to filesystem structure: reading only needs to traverse the directory tree and block list down to the data, then read the data, whereas writing needs to perform the same traversal, then write the data, then update some metadata.
When caching comes into play, things change. Reading data that's in cache is very fast, but reading data that isn't in cache has to go and fetch it from the disk. Operating systems might try to anticipate reads, but that only works in very specific cases (mainly sequential reads from a file). Writing, on the other hand, can be near-instantaneous as long as the amount of data isn't too large, as the data is only written to a memory buffer. The buffer has to be written to disk eventually, but by that time your application has already moved on to do more stuff.
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
add a comment |
It depends. There is no general answer to this question.
In the absence of caching, writing a disk file is usually measurably slower than reading. This has little to do with the operating system and everything to do with the hardware: both hard disks and solid state media read faster than they write. A secondary factor is related to filesystem structure: reading only needs to traverse the directory tree and block list down to the data, then read the data, whereas writing needs to perform the same traversal, then write the data, then update some metadata.
When caching comes into play, things change. Reading data that's in cache is very fast, but reading data that isn't in cache has to go and fetch it from the disk. Operating systems might try to anticipate reads, but that only works in very specific cases (mainly sequential reads from a file). Writing, on the other hand, can be near-instantaneous as long as the amount of data isn't too large, as the data is only written to a memory buffer. The buffer has to be written to disk eventually, but by that time your application has already moved on to do more stuff.
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
It depends. There is no general answer to this question.
In the absence of caching, writing a disk file is usually measurably slower than reading. This has little to do with the operating system and everything to do with the hardware: both hard disks and solid state media read faster than they write. A secondary factor is related to filesystem structure: reading only needs to traverse the directory tree and block list down to the data, then read the data, whereas writing needs to perform the same traversal, then write the data, then update some metadata.
When caching comes into play, things change. Reading data that's in cache is very fast, but reading data that isn't in cache has to go and fetch it from the disk. Operating systems might try to anticipate reads, but that only works in very specific cases (mainly sequential reads from a file). Writing, on the other hand, can be near-instantaneous as long as the amount of data isn't too large, as the data is only written to a memory buffer. The buffer has to be written to disk eventually, but by that time your application has already moved on to do more stuff.
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
answered Aug 22 '14 at 23:04
Gilles
527k12810561581
527k12810561581
add a comment |
add a comment |
Writing should be faster.
The free block list is kept in memory, so finding the next free block will be very fast. Unless you're writing in synchronous mode, when you try to write something it will simply copy the data into a kernel buffer and queue the write; it doesn't have to wait for the I/O to complete.
On the other hand, a read has to wait for the I/O to complete, since the caller can't do anything until the data arrives.
answered Aug 22 '14 at 19:33
Barmar
6,9081223
add a comment |
Writing should be faster.
The free block list is kept in memory, so finding the next free block will be very fast. Unless you're writing in synchronous mode, when you try to write something it will simply copy the data into a kernel buffer and queue the write; it doesn't have to wait for the I/O to complete.
On the other hand, a read has to wait for the I/O to complete, since the caller can't do anything until the data arrives.
answered Aug 22 '14 at 19:33
Barmar
6,9081223
add a comment |
Writing should be faster.
The free block list is kept in memory, so finding the next free block will be very fast. Unless you're writing in synchronous mode, when you try to write something it will simply copy the data into a kernel buffer and queue the write; it doesn't have to wait for the I/O to complete.
On the other hand, a read has to wait for the I/O to complete, since the caller can't do anything until the data arrives.
answered Aug 22 '14 at 19:33
Barmar
6,9081223
Writing should be faster.
The free block list is kept in memory, so finding the next free block will be very fast. Unless you're writing in synchronous mode, when you try to write something it will simply copy the data into a kernel buffer and queue the write; it doesn't have to wait for the I/O to complete.
On the other hand, a read has to wait for the I/O to complete, since the caller can't do anything until the data arrives.
answered Aug 22 '14 at 19:33
Barmar
6,9081223
answered Aug 22 '14 at 19:33
Barmar
6,9081223
answered Aug 22 '14 at 19:33
Barmar
6,9081223
answered Aug 22 '14 at 19:33
Barmar
6,9081223
6,9081223
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f151665%2fis-reading-a-file-on-unix-faster-than-writing-a-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
take a look at here
– Networker
Aug 22 '14 at 18:17
@ojblass, please check the edit and change it if you feel it is incorrect.
– Ramesh
Aug 22 '14 at 19:53