How to split a large folder into smaller folders of equal size [closed]
1
I have one big folder with many file types in it (e.g .txt, .sh). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?
files directory split size
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
asked Dec 20 '18 at 13:25
user2689877
61
closed as unclear what you're asking by Jeff Schaller, Mr Shunz, RalfFriedl, roaima, GAD3R Dec 21 '18 at 12:20
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
add a comment |
1
I have one big folder with many file types in it (e.g .txt, .sh). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?
files directory split size
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
asked Dec 20 '18 at 13:25
user2689877
61
closed as unclear what you're asking by Jeff Schaller, Mr Shunz, RalfFriedl, roaima, GAD3R Dec 21 '18 at 12:20
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
7
Any try so far?
– Kevin Lemaire
Dec 20 '18 at 13:33
I don't suppose you have any individual files larger than 10 Gb?
– Jeff Schaller
Dec 20 '18 at 13:37
take a look at split command maybe ?
– Pierre-Alain TORET
Dec 20 '18 at 13:51
Do you need to access the files, or is it for archiving? One could use multi volume archive format.
– Kusalananda
Dec 20 '18 at 14:11
add a comment |
1
1
1
I have one big folder with many file types in it (e.g .txt, .sh). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?
files directory split size
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
asked Dec 20 '18 at 13:25
user2689877
61
I have one big folder with many file types in it (e.g .txt, .sh). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?
files directory split size
files directory split size
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
asked Dec 20 '18 at 13:25
user2689877
61
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
asked Dec 20 '18 at 13:25
user2689877
61
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
edited Dec 20 '18 at 14:15
Jeff Schaller
38.8k1053125
38.8k1053125
asked Dec 20 '18 at 13:25
user2689877
61
asked Dec 20 '18 at 13:25
user2689877
61
asked Dec 20 '18 at 13:25
user2689877
61
61
closed as unclear what you're asking by Jeff Schaller, Mr Shunz, RalfFriedl, roaima, GAD3R Dec 21 '18 at 12:20
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
closed as unclear what you're asking by Jeff Schaller, Mr Shunz, RalfFriedl, roaima, GAD3R Dec 21 '18 at 12:20
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
7
Any try so far?
– Kevin Lemaire
Dec 20 '18 at 13:33
I don't suppose you have any individual files larger than 10 Gb?
– Jeff Schaller
Dec 20 '18 at 13:37
take a look at split command maybe ?
– Pierre-Alain TORET
Dec 20 '18 at 13:51
Do you need to access the files, or is it for archiving? One could use multi volume archive format.
– Kusalananda
Dec 20 '18 at 14:11
add a comment |
7
Any try so far?
– Kevin Lemaire
Dec 20 '18 at 13:33
I don't suppose you have any individual files larger than 10 Gb?
– Jeff Schaller
Dec 20 '18 at 13:37
take a look at split command maybe ?
– Pierre-Alain TORET
Dec 20 '18 at 13:51
Do you need to access the files, or is it for archiving? One could use multi volume archive format.
– Kusalananda
Dec 20 '18 at 14:11
7
7
Any try so far?
– Kevin Lemaire
Dec 20 '18 at 13:33
Any try so far?
– Kevin Lemaire
Dec 20 '18 at 13:33
I don't suppose you have any individual files larger than 10 Gb?
– Jeff Schaller
Dec 20 '18 at 13:37
I don't suppose you have any individual files larger than 10 Gb?
– Jeff Schaller
Dec 20 '18 at 13:37
take a look at split command maybe ?
– Pierre-Alain TORET
Dec 20 '18 at 13:51
take a look at split command maybe ?
– Pierre-Alain TORET
Dec 20 '18 at 13:51
Do you need to access the files, or is it for archiving? One could use multi volume archive format.
– Kusalananda
Dec 20 '18 at 14:11
Do you need to access the files, or is it for archiving? One could use multi volume archive format.
– Kusalananda
Dec 20 '18 at 14:11
add a comment |
2 Answers
2
active
oldest
votes
1
Without trying to solve the bin packing problem, you could use a script like this:
#!/bin/bash
directory=${1:-testdir}
sizelimit=${2:-1000} # in MB
sizesofar=0
dircount=1
du -s --block-size=1M "$directory"/* | while read -r size file
do
if ((sizesofar + size > sizelimit))
then
(( dircount++ ))
sizesofar=0
fi
(( sizesofar += size ))
mkdir -p -- "$directory/sub_$dircount"
mv -- $file "$directory/sub_$dircount"
done
answered Dec 20 '18 at 14:23
user000001
902713
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
add a comment |
0
If it is a single 10GB file, you can try
split -b 4000000000 filename
edited Dec 20 '18 at 14:19
sebasth
8,16832046
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
3
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
3
... I think....
– Kusalananda
Dec 20 '18 at 14:11
2
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
Without trying to solve the bin packing problem, you could use a script like this:
#!/bin/bash
directory=${1:-testdir}
sizelimit=${2:-1000} # in MB
sizesofar=0
dircount=1
du -s --block-size=1M "$directory"/* | while read -r size file
do
if ((sizesofar + size > sizelimit))
then
(( dircount++ ))
sizesofar=0
fi
(( sizesofar += size ))
mkdir -p -- "$directory/sub_$dircount"
mv -- $file "$directory/sub_$dircount"
done
answered Dec 20 '18 at 14:23
user000001
902713
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
add a comment |
1
Without trying to solve the bin packing problem, you could use a script like this:
#!/bin/bash
directory=${1:-testdir}
sizelimit=${2:-1000} # in MB
sizesofar=0
dircount=1
du -s --block-size=1M "$directory"/* | while read -r size file
do
if ((sizesofar + size > sizelimit))
then
(( dircount++ ))
sizesofar=0
fi
(( sizesofar += size ))
mkdir -p -- "$directory/sub_$dircount"
mv -- $file "$directory/sub_$dircount"
done
answered Dec 20 '18 at 14:23
user000001
902713
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
add a comment |
1
1
1
Without trying to solve the bin packing problem, you could use a script like this:
#!/bin/bash
directory=${1:-testdir}
sizelimit=${2:-1000} # in MB
sizesofar=0
dircount=1
du -s --block-size=1M "$directory"/* | while read -r size file
do
if ((sizesofar + size > sizelimit))
then
(( dircount++ ))
sizesofar=0
fi
(( sizesofar += size ))
mkdir -p -- "$directory/sub_$dircount"
mv -- $file "$directory/sub_$dircount"
done
answered Dec 20 '18 at 14:23
user000001
902713
Without trying to solve the bin packing problem, you could use a script like this:
#!/bin/bash
directory=${1:-testdir}
sizelimit=${2:-1000} # in MB
sizesofar=0
dircount=1
du -s --block-size=1M "$directory"/* | while read -r size file
do
if ((sizesofar + size > sizelimit))
then
(( dircount++ ))
sizesofar=0
fi
(( sizesofar += size ))
mkdir -p -- "$directory/sub_$dircount"
mv -- $file "$directory/sub_$dircount"
done
answered Dec 20 '18 at 14:23
user000001
902713
answered Dec 20 '18 at 14:23
user000001
902713
answered Dec 20 '18 at 14:23
user000001
902713
answered Dec 20 '18 at 14:23
user000001
902713
902713
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
add a comment |
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
– eckes
Dec 20 '18 at 21:42
add a comment |
0
If it is a single 10GB file, you can try
split -b 4000000000 filename
edited Dec 20 '18 at 14:19
sebasth
8,16832046
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
3
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
3
... I think....
– Kusalananda
Dec 20 '18 at 14:11
2
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
add a comment |
0
If it is a single 10GB file, you can try
split -b 4000000000 filename
edited Dec 20 '18 at 14:19
sebasth
8,16832046
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
3
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
3
... I think....
– Kusalananda
Dec 20 '18 at 14:11
2
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
add a comment |
0
0
0
If it is a single 10GB file, you can try
split -b 4000000000 filename
edited Dec 20 '18 at 14:19
sebasth
8,16832046
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
If it is a single 10GB file, you can try
split -b 4000000000 filename
edited Dec 20 '18 at 14:19
sebasth
8,16832046
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
edited Dec 20 '18 at 14:19
sebasth
8,16832046
edited Dec 20 '18 at 14:19
sebasth
8,16832046
edited Dec 20 '18 at 14:19
sebasth
8,16832046
8,16832046
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
answered Dec 20 '18 at 13:58
Vineeth Thomas
264
264
3
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
3
... I think....
– Kusalananda
Dec 20 '18 at 14:11
2
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
add a comment |
3
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
3
... I think....
– Kusalananda
Dec 20 '18 at 14:11
2
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
3
3
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
– Kusalananda
Dec 20 '18 at 14:03
3
3
... I think....
– Kusalananda
Dec 20 '18 at 14:11
... I think....
– Kusalananda
Dec 20 '18 at 14:11
2
2
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
– Jeff Schaller
Dec 20 '18 at 14:21
add a comment |
7
Any try so far?
– Kevin Lemaire
Dec 20 '18 at 13:33
I don't suppose you have any individual files larger than 10 Gb?
– Jeff Schaller
Dec 20 '18 at 13:37
take a look at split command maybe ?
– Pierre-Alain TORET
Dec 20 '18 at 13:51
Do you need to access the files, or is it for archiving? One could use multi volume archive format.
– Kusalananda
Dec 20 '18 at 14:11