Why is rsync taking a long time on large files that already exist?
I made a backup of one of my computer's internal drives to an external. The next time I used rsync to sync the two drives, I noticed that large (40 + GB) files that weren't modified were still taking a long time to "copy". I thought rsync looked at mod-times and file size first? Why would it take so long; as though it were using checksum?
I had originally copied the files using rsync -rv --delete /src/path/ /dest/path/
rsync
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
add a comment |
I made a backup of one of my computer's internal drives to an external. The next time I used rsync to sync the two drives, I noticed that large (40 + GB) files that weren't modified were still taking a long time to "copy". I thought rsync looked at mod-times and file size first? Why would it take so long; as though it were using checksum?
I had originally copied the files using rsync -rv --delete /src/path/ /dest/path/
rsync
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
add a comment |
I made a backup of one of my computer's internal drives to an external. The next time I used rsync to sync the two drives, I noticed that large (40 + GB) files that weren't modified were still taking a long time to "copy". I thought rsync looked at mod-times and file size first? Why would it take so long; as though it were using checksum?
I had originally copied the files using rsync -rv --delete /src/path/ /dest/path/
rsync
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
I made a backup of one of my computer's internal drives to an external. The next time I used rsync to sync the two drives, I noticed that large (40 + GB) files that weren't modified were still taking a long time to "copy". I thought rsync looked at mod-times and file size first? Why would it take so long; as though it were using checksum?
I had originally copied the files using rsync -rv --delete /src/path/ /dest/path/
rsync
rsync
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
edited Nov 5 '18 at 16:16
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
asked Nov 5 '18 at 16:04
DeepDeadpool
275211
275211
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Since you are not copying the metadata (which you would do if you used --archive or -a instead of just -r), the metadata (timestamps, ownerships etc.) will be different between the copy and the original. When you run rsync again, since the timestamps are different, the file is copied again.
So, you would instead want to use
rsync -ai --delete /src/path/ /dest/path
I'm using -i (--itemize-changes) since it also tells me why a file was copied.
Also note that when you do a local copy with rsync, it will not use its delta algorithm, but will instead behave as if --whole-file (or -W) was specified. This is because the delta algorithm is assumed to only be faster than a whole file transfer when transferring over a network. When using the delta algorithm, the whole file would need to be read and checksummed on both the source and target systems. Doing this locally seems a bit wasteful, so the file is just copied in full instead.
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with--no-whole-fileor--no-Wif you wished to use the delta algorithm for a local transfer.
– Kusalananda
Nov 5 '18 at 16:33
1
@DeepDeadpool Note also that if you're copying over a network with-rinstead of-a, then, when you invokersyncagain, it would be forced to use the delta algorithm, whereas if you had used-ait could just have skipped that transfer completely.
– Kusalananda
Nov 5 '18 at 16:36
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479936%2fwhy-is-rsync-taking-a-long-time-on-large-files-that-already-exist%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Since you are not copying the metadata (which you would do if you used --archive or -a instead of just -r), the metadata (timestamps, ownerships etc.) will be different between the copy and the original. When you run rsync again, since the timestamps are different, the file is copied again.
So, you would instead want to use
rsync -ai --delete /src/path/ /dest/path
I'm using -i (--itemize-changes) since it also tells me why a file was copied.
Also note that when you do a local copy with rsync, it will not use its delta algorithm, but will instead behave as if --whole-file (or -W) was specified. This is because the delta algorithm is assumed to only be faster than a whole file transfer when transferring over a network. When using the delta algorithm, the whole file would need to be read and checksummed on both the source and target systems. Doing this locally seems a bit wasteful, so the file is just copied in full instead.
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with--no-whole-fileor--no-Wif you wished to use the delta algorithm for a local transfer.
– Kusalananda
Nov 5 '18 at 16:33
1
@DeepDeadpool Note also that if you're copying over a network with-rinstead of-a, then, when you invokersyncagain, it would be forced to use the delta algorithm, whereas if you had used-ait could just have skipped that transfer completely.
– Kusalananda
Nov 5 '18 at 16:36
add a comment |
Since you are not copying the metadata (which you would do if you used --archive or -a instead of just -r), the metadata (timestamps, ownerships etc.) will be different between the copy and the original. When you run rsync again, since the timestamps are different, the file is copied again.
So, you would instead want to use
rsync -ai --delete /src/path/ /dest/path
I'm using -i (--itemize-changes) since it also tells me why a file was copied.
Also note that when you do a local copy with rsync, it will not use its delta algorithm, but will instead behave as if --whole-file (or -W) was specified. This is because the delta algorithm is assumed to only be faster than a whole file transfer when transferring over a network. When using the delta algorithm, the whole file would need to be read and checksummed on both the source and target systems. Doing this locally seems a bit wasteful, so the file is just copied in full instead.
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with--no-whole-fileor--no-Wif you wished to use the delta algorithm for a local transfer.
– Kusalananda
Nov 5 '18 at 16:33
1
@DeepDeadpool Note also that if you're copying over a network with-rinstead of-a, then, when you invokersyncagain, it would be forced to use the delta algorithm, whereas if you had used-ait could just have skipped that transfer completely.
– Kusalananda
Nov 5 '18 at 16:36
add a comment |
Since you are not copying the metadata (which you would do if you used --archive or -a instead of just -r), the metadata (timestamps, ownerships etc.) will be different between the copy and the original. When you run rsync again, since the timestamps are different, the file is copied again.
So, you would instead want to use
rsync -ai --delete /src/path/ /dest/path
I'm using -i (--itemize-changes) since it also tells me why a file was copied.
Also note that when you do a local copy with rsync, it will not use its delta algorithm, but will instead behave as if --whole-file (or -W) was specified. This is because the delta algorithm is assumed to only be faster than a whole file transfer when transferring over a network. When using the delta algorithm, the whole file would need to be read and checksummed on both the source and target systems. Doing this locally seems a bit wasteful, so the file is just copied in full instead.
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
Since you are not copying the metadata (which you would do if you used --archive or -a instead of just -r), the metadata (timestamps, ownerships etc.) will be different between the copy and the original. When you run rsync again, since the timestamps are different, the file is copied again.
So, you would instead want to use
rsync -ai --delete /src/path/ /dest/path
I'm using -i (--itemize-changes) since it also tells me why a file was copied.
Also note that when you do a local copy with rsync, it will not use its delta algorithm, but will instead behave as if --whole-file (or -W) was specified. This is because the delta algorithm is assumed to only be faster than a whole file transfer when transferring over a network. When using the delta algorithm, the whole file would need to be read and checksummed on both the source and target systems. Doing this locally seems a bit wasteful, so the file is just copied in full instead.
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
edited Dec 20 '18 at 15:05
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
answered Nov 5 '18 at 16:11
Kusalananda
122k16229374
122k16229374
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with--no-whole-fileor--no-Wif you wished to use the delta algorithm for a local transfer.
– Kusalananda
Nov 5 '18 at 16:33
1
@DeepDeadpool Note also that if you're copying over a network with-rinstead of-a, then, when you invokersyncagain, it would be forced to use the delta algorithm, whereas if you had used-ait could just have skipped that transfer completely.
– Kusalananda
Nov 5 '18 at 16:36
add a comment |
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with--no-whole-fileor--no-Wif you wished to use the delta algorithm for a local transfer.
– Kusalananda
Nov 5 '18 at 16:33
1
@DeepDeadpool Note also that if you're copying over a network with-rinstead of-a, then, when you invokersyncagain, it would be forced to use the delta algorithm, whereas if you had used-ait could just have skipped that transfer completely.
– Kusalananda
Nov 5 '18 at 16:36
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer
– DeepDeadpool
Nov 5 '18 at 16:30
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with
--no-whole-file or --no-W if you wished to use the delta algorithm for a local transfer.– Kusalananda
Nov 5 '18 at 16:33
@DeepDeadpool Because of performance reasons, as I mentioned. You could run it with
--no-whole-file or --no-W if you wished to use the delta algorithm for a local transfer.– Kusalananda
Nov 5 '18 at 16:33
1
1
@DeepDeadpool Note also that if you're copying over a network with
-r instead of -a, then, when you invoke rsync again, it would be forced to use the delta algorithm, whereas if you had used -a it could just have skipped that transfer completely.– Kusalananda
Nov 5 '18 at 16:36
@DeepDeadpool Note also that if you're copying over a network with
-r instead of -a, then, when you invoke rsync again, it would be forced to use the delta algorithm, whereas if you had used -a it could just have skipped that transfer completely.– Kusalananda
Nov 5 '18 at 16:36
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f479936%2fwhy-is-rsync-taking-a-long-time-on-large-files-that-already-exist%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
