Move the oldest files into a directory, source directory has more than 10,000 files constantly incoming
I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find"
result into "ls"
or just use the "ls
" .. please suggest.
ls -ltr `/bin/find ${in}/* -prune -name "*.txt" -type f` | head -10 |
while read -r infile ; do
-move the file
done
or should i just use ls.
the reason i am using find
is: i read some online content that ls
should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls
.
i am worried if find
result is too big would it might cause any problem.
shell-script ls ksh mv hp-ux
add a comment |
I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find"
result into "ls"
or just use the "ls
" .. please suggest.
ls -ltr `/bin/find ${in}/* -prune -name "*.txt" -type f` | head -10 |
while read -r infile ; do
-move the file
done
or should i just use ls.
the reason i am using find
is: i read some online content that ls
should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls
.
i am worried if find
result is too big would it might cause any problem.
shell-script ls ksh mv hp-ux
I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35
No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08
unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22
If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago
add a comment |
I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find"
result into "ls"
or just use the "ls
" .. please suggest.
ls -ltr `/bin/find ${in}/* -prune -name "*.txt" -type f` | head -10 |
while read -r infile ; do
-move the file
done
or should i just use ls.
the reason i am using find
is: i read some online content that ls
should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls
.
i am worried if find
result is too big would it might cause any problem.
shell-script ls ksh mv hp-ux
I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find"
result into "ls"
or just use the "ls
" .. please suggest.
ls -ltr `/bin/find ${in}/* -prune -name "*.txt" -type f` | head -10 |
while read -r infile ; do
-move the file
done
or should i just use ls.
the reason i am using find
is: i read some online content that ls
should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls
.
i am worried if find
result is too big would it might cause any problem.
shell-script ls ksh mv hp-ux
shell-script ls ksh mv hp-ux
edited Dec 28 '18 at 19:52
Rui F Ribeiro
39.3k1479131
39.3k1479131
asked Dec 28 '18 at 18:55
user3342678user3342678
1
1
I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35
No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08
unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22
If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago
add a comment |
I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35
No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08
unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22
If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago
I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35
I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35
No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08
No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08
unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32
unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22
If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago
If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago
add a comment |
2 Answers
2
active
oldest
votes
While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.
The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *
. The inner loop then compares every file's timestamp to see if it is older than (-ot
) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.
for((count=0; count < 3; count++))
do
set -- *
oldest=$1
for f in ./*
do
if [ "$f" -ot "$oldest" ]
then
oldest=$f
fi
done
echo mv -- "$oldest" ./.archive
mv -- "$oldest" ./.archive
done
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to justls command
, i could just use ls -ltr | head -x then for each file will do the move operation.
– user3342678
Dec 28 '18 at 21:52
add a comment |
Python heapq.nsmallest
bash:
find -printf '%T@ %pn' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files
Here find
command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs
command takes filenames from stdin one by one and applies mv xxx directory_for_old_files
command substituting xxx with them.
noldest.py implementation could be:
import sys
from heapq import nsmallest
from datetime import datetime
n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy
def parse_date(date_and_filename):
seconds_from_1970 = float(date_and_filename[0])
return datetime.fromtimestamp(int(seconds_from_1970))
for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines
The perfomance depends on implementation of nsmallest
algorithm from python standard library.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491356%2fmove-the-oldest-files-into-a-directory-source-directory-has-more-than-10-000-fi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.
The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *
. The inner loop then compares every file's timestamp to see if it is older than (-ot
) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.
for((count=0; count < 3; count++))
do
set -- *
oldest=$1
for f in ./*
do
if [ "$f" -ot "$oldest" ]
then
oldest=$f
fi
done
echo mv -- "$oldest" ./.archive
mv -- "$oldest" ./.archive
done
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to justls command
, i could just use ls -ltr | head -x then for each file will do the move operation.
– user3342678
Dec 28 '18 at 21:52
add a comment |
While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.
The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *
. The inner loop then compares every file's timestamp to see if it is older than (-ot
) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.
for((count=0; count < 3; count++))
do
set -- *
oldest=$1
for f in ./*
do
if [ "$f" -ot "$oldest" ]
then
oldest=$f
fi
done
echo mv -- "$oldest" ./.archive
mv -- "$oldest" ./.archive
done
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to justls command
, i could just use ls -ltr | head -x then for each file will do the move operation.
– user3342678
Dec 28 '18 at 21:52
add a comment |
While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.
The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *
. The inner loop then compares every file's timestamp to see if it is older than (-ot
) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.
for((count=0; count < 3; count++))
do
set -- *
oldest=$1
for f in ./*
do
if [ "$f" -ot "$oldest" ]
then
oldest=$f
fi
done
echo mv -- "$oldest" ./.archive
mv -- "$oldest" ./.archive
done
While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.
The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *
. The inner loop then compares every file's timestamp to see if it is older than (-ot
) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.
for((count=0; count < 3; count++))
do
set -- *
oldest=$1
for f in ./*
do
if [ "$f" -ot "$oldest" ]
then
oldest=$f
fi
done
echo mv -- "$oldest" ./.archive
mv -- "$oldest" ./.archive
done
answered Dec 28 '18 at 20:45
Jeff SchallerJeff Schaller
39k1054125
39k1054125
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to justls command
, i could just use ls -ltr | head -x then for each file will do the move operation.
– user3342678
Dec 28 '18 at 21:52
add a comment |
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to justls command
, i could just use ls -ltr | head -x then for each file will do the move operation.
– user3342678
Dec 28 '18 at 21:52
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just
ls command
, i could just use ls -ltr | head -x then for each file will do the move operation.– user3342678
Dec 28 '18 at 21:52
Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just
ls command
, i could just use ls -ltr | head -x then for each file will do the move operation.– user3342678
Dec 28 '18 at 21:52
add a comment |
Python heapq.nsmallest
bash:
find -printf '%T@ %pn' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files
Here find
command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs
command takes filenames from stdin one by one and applies mv xxx directory_for_old_files
command substituting xxx with them.
noldest.py implementation could be:
import sys
from heapq import nsmallest
from datetime import datetime
n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy
def parse_date(date_and_filename):
seconds_from_1970 = float(date_and_filename[0])
return datetime.fromtimestamp(int(seconds_from_1970))
for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines
The perfomance depends on implementation of nsmallest
algorithm from python standard library.
add a comment |
Python heapq.nsmallest
bash:
find -printf '%T@ %pn' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files
Here find
command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs
command takes filenames from stdin one by one and applies mv xxx directory_for_old_files
command substituting xxx with them.
noldest.py implementation could be:
import sys
from heapq import nsmallest
from datetime import datetime
n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy
def parse_date(date_and_filename):
seconds_from_1970 = float(date_and_filename[0])
return datetime.fromtimestamp(int(seconds_from_1970))
for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines
The perfomance depends on implementation of nsmallest
algorithm from python standard library.
add a comment |
Python heapq.nsmallest
bash:
find -printf '%T@ %pn' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files
Here find
command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs
command takes filenames from stdin one by one and applies mv xxx directory_for_old_files
command substituting xxx with them.
noldest.py implementation could be:
import sys
from heapq import nsmallest
from datetime import datetime
n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy
def parse_date(date_and_filename):
seconds_from_1970 = float(date_and_filename[0])
return datetime.fromtimestamp(int(seconds_from_1970))
for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines
The perfomance depends on implementation of nsmallest
algorithm from python standard library.
Python heapq.nsmallest
bash:
find -printf '%T@ %pn' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files
Here find
command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs
command takes filenames from stdin one by one and applies mv xxx directory_for_old_files
command substituting xxx with them.
noldest.py implementation could be:
import sys
from heapq import nsmallest
from datetime import datetime
n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin) # generators are lazy
def parse_date(date_and_filename):
seconds_from_1970 = float(date_and_filename[0])
return datetime.fromtimestamp(int(seconds_from_1970))
for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
print(filename.rstrip('n')) # lines in sys.stdin have trailing newlines
The perfomance depends on implementation of nsmallest
algorithm from python standard library.
answered Dec 29 '18 at 1:07
belkkabelkka
1307
1307
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491356%2fmove-the-oldest-files-into-a-directory-source-directory-has-more-than-10-000-fi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I noticed you tagged ksh and hp-ux; is zsh an option?
– Jeff Schaller
Dec 28 '18 at 19:35
No, our OS is hp-ux and we have ksh scripts in that i had to implement this part.
– user3342678
Dec 28 '18 at 20:08
unix.stackexchange.com/a/449771/117549 may be useful
– Jeff Schaller
Dec 28 '18 at 20:32
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files.
– pbhj
Dec 29 '18 at 1:22
If any of the answers solved your problem, please accept it by clicking the checkmark next to it. Thank you!
– Jeff Schaller
2 days ago