Cfrtjryk

There's some problems with ulimit. Here's a useful read on the topic: Limiting time and memory consumption of a program in Linux, which lead to the timeout tool, which lets you cage a process (and its forks) by time or memory consumption.

The timeout tool requires Perl 5+ and the /proc filesystem mounted. After that you copy the tool to e.g. /usr/local/bin like so:

curl https://raw.githubusercontent.com/pshved/timeout/master/timeout | 

  sudo tee /usr/local/bin/timeout && sudo chmod 755 /usr/local/bin/timeout

After that, you can 'cage' your process by memory consumption as in your question like so:

timeout -m 500 pdftoppm Sample.pdf

Alternatively you could use -t <seconds> and -x <hertz> to respectively limit the process by time or CPU constraints.

The way this tool works is by checking multiple times per second if the spawned process has not oversubscribed its set boundaries. This means there actually is a small window where a process could potentially be oversubscribing before timeout notices and kills the process.

A more correct approach would hence likely involve cgroups, but that is much more involved to set up, even if you'd use Docker or runC, which among things, offer a more user-friendly abstraction around cgroups.

Another way to limit this is to use Linux's control groups. This is especially useful if you want to limit a process's (or group of processes') allocation of physical memory distinctly from virtual memory. For example:

cgcreate -g memory:/myGroup

echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.limit_in_bytes

echo $(( 5000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.memsw.limit_in_bytes

will create a control group named myGroup, cap the set of processes run under myGroup up to 500 MB of physical memory and up to 5000 MB of swap. To run a process under the control group:

cgexec -g memory:myGroup pdftoppm

Note that on a modern Ubuntu distribution this example requires installing the cgroup-bin package and editing /etc/default/grub to change GRUB_CMDLINE_LINUX_DEFAULT to:

GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1"

and then running sudo update-grub and rebooting to boot with the new kernel boot parameters.

If your process doesn't spawn more children that consume the most memory, you may use setrlimit function. More common user interface for that is using ulimit command of the shell:

$ ulimit -Sv 500000     # Set ~500 mb limit

$ pdftoppm ...

This will only limit "virtual" memory of your process, taking into account—and limiting—the memory the process being invoked shares with other processes, and the memory mapped but not reserved (for instance, Java's large heap). Still, virtual memory is the closest approximation for processes that grow really large, making the said errors insignificant.

If your program spawns children, and it's them which allocate memory, it becomes more complex, and you should write auxiliary scripts to run processes under your control. I wrote in my blog, why and how.

In addition to the tools from daemontools, suggested by Mark Johnson, you can also consider chpst which is found in runit. Runit itself is bundled in busybox, so you might already have it installed.

The man page of chpst shows the option:

-m bytes
limit memory. Limit the data segment, stack segment, locked physical pages, and total of all segment per process to bytes bytes
each.

I'm using the below script, which works great. It uses cgroups through cgmanager. Update: it now uses the commands from cgroup-tools. Name this script limitmem and put it in your $PATH and you can use it like limitmem 100M bash. This will limit both memory and swap usage. To limit just memory remove the line with memory.memsw.limit_in_bytes.

Disclaimer: I wouldn't be surprised if cgroup-tools also breaks in the future. The correct solution would be to use the systemd api's for cgroup management but there are no command line tools for that a.t.m.

#!/bin/sh



# This script uses commands from the cgroup-tools package. The cgroup-tools commands access the cgroup filesystem directly which is against the (new-ish) kernel's requirement that cgroups are managed by a single entity (which usually will be systemd). Additionally there is a v2 cgroup api in development which will probably replace the existing api at some point. So expect this script to break in the future. The correct way forward would be to use systemd's apis to create the cgroups, but afaik systemd currently (feb 2018) only exposes dbus apis for which there are no command line tools yet, and I didn't feel like writing those.



# strict mode: error if commands fail or if unset variables are used

set -eu



if [ "$#" -lt 2 ]

then

    echo Usage: `basename $0` "<limit> <command>..."

    echo or: `basename $0` "<memlimit> -s <swaplimit> <command>..."

    exit 1

fi



cgname="limitmem_$$"



# parse command line args and find limits



limit="$1"

swaplimit="$limit"

shift



if [ "$1" = "-s" ]

then

    shift

    swaplimit="$1"

    shift

fi



if [ "$1" = -- ]

then

    shift

fi



if [ "$limit" = "$swaplimit" ]

then

    memsw=0

    echo "limiting memory to $limit (cgroup $cgname) for command $@" >&2

else

    memsw=1

    echo "limiting memory to $limit and total virtual memory to $swaplimit (cgroup $cgname) for command $@" >&2

fi



# create cgroup

sudo cgcreate -g "memory:$cgname"

sudo cgset -r memory.limit_in_bytes="$limit" "$cgname"

bytes_limit=`cgget -g "memory:$cgname" | grep memory.limit_in_bytes | cut -d  -f2`



# try also limiting swap usage, but this fails if the system has no swap

if sudo cgset -r memory.memsw.limit_in_bytes="$swaplimit" "$cgname"

then

    bytes_swap_limit=`cgget -g "memory:$cgname" | grep memory.memsw.limit_in_bytes | cut -d  -f2`

else

    echo "failed to limit swap"

    memsw=0

fi



# create a waiting sudo'd process that will delete the cgroup once we're done. This prevents the user needing to enter their password to sudo again after the main command exists, which may take longer than sudo's timeout.

tmpdir=${XDG_RUNTIME_DIR:-$TMPDIR}

tmpdir=${tmpdir:-/tmp}

fifo="$tmpdir/limitmem_$$_cgroup_closer"

mkfifo --mode=u=rw,go= "$fifo"

sudo -b sh -c "head -c1 '$fifo' >/dev/null ; cgdelete -g 'memory:$cgname'"



# spawn subshell to run in the cgroup. If the command fails we still want to remove the cgroup so unset '-e'.

set +e

(

set -e

# move subshell into cgroup

sudo cgclassify -g "memory:$cgname" --sticky `sh -c 'echo $PPID'`  # $$ returns the main shell's pid, not this subshell's.

exec "$@"

)



# grab exit code 

exitcode=$?



set -e



# show memory usage summary



peak_mem=`cgget -g "memory:$cgname" | grep memory.max_usage_in_bytes | cut -d  -f2`

failcount=`cgget -g "memory:$cgname" | grep memory.failcnt | cut -d  -f2`

percent=`expr "$peak_mem" / ( "$bytes_limit" / 100 )`



echo "peak memory used: $peak_mem ($percent%); exceeded limit $failcount times" >&2



if [ "$memsw" = 1 ]

then

    peak_swap=`cgget -g "memory:$cgname" | grep memory.memsw.max_usage_in_bytes | cut -d  -f2`

    swap_failcount=`cgget -g "memory:$cgname" |grep memory.memsw.failcnt | cut -d  -f2`

    swap_percent=`expr "$peak_swap" / ( "$bytes_swap_limit" / 100 )`



    echo "peak virtual memory used: $peak_swap ($swap_percent%); exceeded limit $swap_failcount times" >&2

fi



# remove cgroup by sending a byte through the pipe

echo 1 > "$fifo"

rm "$fifo"



exit $exitcode