'No space left on device' error thrown when generating large amount of files, but there is no clear limit...

up vote
1
down vote

favorite

Running a python script to generate millions of empty text files just for fun on my Mint machine (18.1 Serena) before I wipe it.

Recently the script failed with this error:

OSError: [Errno 28] No space left on device: '1013162169.txt'

There is clearly memory and inode table space left available, and restarting the script running processes for a bit before throwing the error again.

$ df -i

Filesystem       Inodes    IUsed    IFree IUse% Mounted on

udev             194936      488   194448    1% /dev

tmpfs            205500      716   204784    1% /run

/dev/sda1      60563456 18707860 41855596   31% /

tmpfs            205500        1   205499    1% /dev/shm

tmpfs            205500        6   205494    1% /run/lock

tmpfs            205500       18   205482    1% /sys/fs/cgroup

cgmfs            205500       14   205486    1% /run/cgmanager/fs

tmpfs            205500        6   205494    1% /run/user/1000



$ df

Filesystem     1K-blocks     Used Available Use% Mounted on

udev             3831012        0   3831012   0% /dev

tmpfs             770428    17620    752808   3% /run

/dev/sda1      953630580 14641452 890524440   2% /

tmpfs            3852140        0   3852140   0% /dev/shm

tmpfs               5120        4      5116   1% /run/lock

tmpfs            3852140        0   3852140   0% /sys/fs/cgroup

cgmfs                100        0       100   0% /run/cgmanager/fs

tmpfs             770428        4    770424   1% /run/user/1000

Is there some other space constraint being run into here? The specific case is a meaningless script but I'm very curious about underlying cause.

Files are being created in /home/jon/test which is on sda1.

edited Nov 21 at 16:10

avidenat

asked Nov 21 at 15:17

Jon Reinhold

846614

2

Check your filesystem for constrains about the number of the files and/or directories in one directory
– Romeo Ninov
Nov 21 at 15:20

Do you have just the one filesystem? Where are the files being generated?
– Jeff Schaller
Nov 21 at 15:21

Added the full df outputs. They're being created in the directory the script runs in which is on sda1. Most of those 18 million inodes are files from this process.
– Jon Reinhold
Nov 21 at 15:29

1

@RomeoNinov ext4, which doesn't seem to have a limit defined per directory, and several billion for the whole system.
– Jon Reinhold
Nov 21 at 15:34

3

Looking at the the related questions (right-hand sidebar), it suggests you hit birthday paradox on the directory index, which uses a 32-bit hash table. unix.stackexchange.com/questions/222221/…
– sourcejedi
Nov 21 at 16:18

|
show 3 more comments

up vote
1
down vote

favorite

Running a python script to generate millions of empty text files just for fun on my Mint machine (18.1 Serena) before I wipe it.

Recently the script failed with this error:

OSError: [Errno 28] No space left on device: '1013162169.txt'

There is clearly memory and inode table space left available, and restarting the script running processes for a bit before throwing the error again.

$ df -i

Filesystem       Inodes    IUsed    IFree IUse% Mounted on

udev             194936      488   194448    1% /dev

tmpfs            205500      716   204784    1% /run

/dev/sda1      60563456 18707860 41855596   31% /

tmpfs            205500        1   205499    1% /dev/shm

tmpfs            205500        6   205494    1% /run/lock

tmpfs            205500       18   205482    1% /sys/fs/cgroup

cgmfs            205500       14   205486    1% /run/cgmanager/fs

tmpfs            205500        6   205494    1% /run/user/1000



$ df

Filesystem     1K-blocks     Used Available Use% Mounted on

udev             3831012        0   3831012   0% /dev

tmpfs             770428    17620    752808   3% /run

/dev/sda1      953630580 14641452 890524440   2% /

tmpfs            3852140        0   3852140   0% /dev/shm

tmpfs               5120        4      5116   1% /run/lock

tmpfs            3852140        0   3852140   0% /sys/fs/cgroup

cgmfs                100        0       100   0% /run/cgmanager/fs

tmpfs             770428        4    770424   1% /run/user/1000

Is there some other space constraint being run into here? The specific case is a meaningless script but I'm very curious about underlying cause.

Files are being created in /home/jon/test which is on sda1.

edited Nov 21 at 16:10

avidenat

asked Nov 21 at 15:17

Jon Reinhold

846614

2

Check your filesystem for constrains about the number of the files and/or directories in one directory
– Romeo Ninov
Nov 21 at 15:20

Do you have just the one filesystem? Where are the files being generated?
– Jeff Schaller
Nov 21 at 15:21

Added the full df outputs. They're being created in the directory the script runs in which is on sda1. Most of those 18 million inodes are files from this process.
– Jon Reinhold
Nov 21 at 15:29

1

@RomeoNinov ext4, which doesn't seem to have a limit defined per directory, and several billion for the whole system.
– Jon Reinhold
Nov 21 at 15:34

3

Looking at the the related questions (right-hand sidebar), it suggests you hit birthday paradox on the directory index, which uses a 32-bit hash table. unix.stackexchange.com/questions/222221/…
– sourcejedi
Nov 21 at 16:18

|
show 3 more comments

up vote
1
down vote

favorite

Running a python script to generate millions of empty text files just for fun on my Mint machine (18.1 Serena) before I wipe it.

Recently the script failed with this error:

OSError: [Errno 28] No space left on device: '1013162169.txt'

There is clearly memory and inode table space left available, and restarting the script running processes for a bit before throwing the error again.

$ df -i

Filesystem       Inodes    IUsed    IFree IUse% Mounted on

udev             194936      488   194448    1% /dev

tmpfs            205500      716   204784    1% /run

/dev/sda1      60563456 18707860 41855596   31% /

tmpfs            205500        1   205499    1% /dev/shm

tmpfs            205500        6   205494    1% /run/lock

tmpfs            205500       18   205482    1% /sys/fs/cgroup

cgmfs            205500       14   205486    1% /run/cgmanager/fs

tmpfs            205500        6   205494    1% /run/user/1000



$ df

Filesystem     1K-blocks     Used Available Use% Mounted on

udev             3831012        0   3831012   0% /dev

tmpfs             770428    17620    752808   3% /run

/dev/sda1      953630580 14641452 890524440   2% /

tmpfs            3852140        0   3852140   0% /dev/shm

tmpfs               5120        4      5116   1% /run/lock

tmpfs            3852140        0   3852140   0% /sys/fs/cgroup

cgmfs                100        0       100   0% /run/cgmanager/fs

tmpfs             770428        4    770424   1% /run/user/1000

Is there some other space constraint being run into here? The specific case is a meaningless script but I'm very curious about underlying cause.

Files are being created in /home/jon/test which is on sda1.

edited Nov 21 at 16:10

avidenat

asked Nov 21 at 15:17

Jon Reinhold

846614

Running a python script to generate millions of empty text files just for fun on my Mint machine (18.1 Serena) before I wipe it.

Recently the script failed with this error:

OSError: [Errno 28] No space left on device: '1013162169.txt'

There is clearly memory and inode table space left available, and restarting the script running processes for a bit before throwing the error again.

$ df -i

Filesystem       Inodes    IUsed    IFree IUse% Mounted on

udev             194936      488   194448    1% /dev

tmpfs            205500      716   204784    1% /run

/dev/sda1      60563456 18707860 41855596   31% /

tmpfs            205500        1   205499    1% /dev/shm

tmpfs            205500        6   205494    1% /run/lock

tmpfs            205500       18   205482    1% /sys/fs/cgroup

cgmfs            205500       14   205486    1% /run/cgmanager/fs

tmpfs            205500        6   205494    1% /run/user/1000



$ df

Filesystem     1K-blocks     Used Available Use% Mounted on

udev             3831012        0   3831012   0% /dev

tmpfs             770428    17620    752808   3% /run

/dev/sda1      953630580 14641452 890524440   2% /

tmpfs            3852140        0   3852140   0% /dev/shm

tmpfs               5120        4      5116   1% /run/lock

tmpfs            3852140        0   3852140   0% /sys/fs/cgroup

cgmfs                100        0       100   0% /run/cgmanager/fs

tmpfs             770428        4    770424   1% /run/user/1000

Is there some other space constraint being run into here? The specific case is a meaningless script but I'm very curious about underlying cause.

Files are being created in /home/jon/test which is on sda1.

filesystems inode

edited Nov 21 at 16:10

avidenat

asked Nov 21 at 15:17

Jon Reinhold

846614

edited Nov 21 at 16:10

avidenat

asked Nov 21 at 15:17

Jon Reinhold

846614

edited Nov 21 at 16:10

avidenat

edited Nov 21 at 16:10

avidenat

edited Nov 21 at 16:10

avidenat

asked Nov 21 at 15:17

Jon Reinhold

846614

asked Nov 21 at 15:17

Jon Reinhold

846614

asked Nov 21 at 15:17

Jon Reinhold

846614

2

Check your filesystem for constrains about the number of the files and/or directories in one directory
– Romeo Ninov
Nov 21 at 15:20

Do you have just the one filesystem? Where are the files being generated?
– Jeff Schaller
Nov 21 at 15:21

Added the full df outputs. They're being created in the directory the script runs in which is on sda1. Most of those 18 million inodes are files from this process.
– Jon Reinhold
Nov 21 at 15:29

1

@RomeoNinov ext4, which doesn't seem to have a limit defined per directory, and several billion for the whole system.
– Jon Reinhold
Nov 21 at 15:34

3

Looking at the the related questions (right-hand sidebar), it suggests you hit birthday paradox on the directory index, which uses a 32-bit hash table. unix.stackexchange.com/questions/222221/…
– sourcejedi
Nov 21 at 16:18

|
show 3 more comments

2

Check your filesystem for constrains about the number of the files and/or directories in one directory
– Romeo Ninov
Nov 21 at 15:20

Do you have just the one filesystem? Where are the files being generated?
– Jeff Schaller
Nov 21 at 15:21

Added the full df outputs. They're being created in the directory the script runs in which is on sda1. Most of those 18 million inodes are files from this process.
– Jon Reinhold
Nov 21 at 15:29

1

@RomeoNinov ext4, which doesn't seem to have a limit defined per directory, and several billion for the whole system.
– Jon Reinhold
Nov 21 at 15:34

3

Looking at the the related questions (right-hand sidebar), it suggests you hit birthday paradox on the directory index, which uses a 32-bit hash table. unix.stackexchange.com/questions/222221/…
– sourcejedi
Nov 21 at 16:18

Check your filesystem for constrains about the number of the files and/or directories in one directory
– Romeo Ninov
Nov 21 at 15:20

Do you have just the one filesystem? Where are the files being generated?
– Jeff Schaller
Nov 21 at 15:21

Added the full df outputs. They're being created in the directory the script runs in which is on sda1. Most of those 18 million inodes are files from this process.
– Jon Reinhold
Nov 21 at 15:29

@RomeoNinov ext4, which doesn't seem to have a limit defined per directory, and several billion for the whole system.
– Jon Reinhold
Nov 21 at 15:34

Looking at the the related questions (right-hand sidebar), it suggests you hit birthday paradox on the directory index, which uses a 32-bit hash table. unix.stackexchange.com/questions/222221/…
– sourcejedi
Nov 21 at 16:18

|
show 3 more comments

active

oldest

votes

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f483235%2fno-space-left-on-device-error-thrown-when-generating-large-amount-of-files-bu%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

active

oldest

votes

draft saved

draft discarded

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Cfrtjryk