SEGFAULT after the inclusion of FFI::shutdown() to release memory consumption

[craig410]

Running libvips 8.13.3 on Debian bullseye with PHP 8.1 under Apache using jcupitt/vips:2.10

Calling FFI::shutdown() at the end of a script after an Image::Thumbnail() operation results in the process SEGFAULT-ing.

The call to shutdown() is required as without it the memory consumption climbs and is never released back to the system (at least from our experiments).

There are more details and a full reproduction case, including Dockerfile to match the environment: https://github.com/ingenerator/libvips-segfault-repro

Thanks to @acoulton: Failing tests proving either case here: https://github.com/ingenerator/libvips-segfault-repro/actions/runs/3444654670/jobs/5747485403

And a further attempt to debug the source of the SEGFAULT here: ingenerator/libvips-segfault-repro#3

We have followed as much of the documentation as we can find around usingphp-vips appropriately however, it may be that it is not suited to running in this environment, in this way (ie mpm-prefork / multiuser). While happy to spend a little more time debugging, if you have any suggestions of where to go next with it? It would be good to know if it is at least intended to work this way?

[jcupitt]

Hi @craig410,

I've just made a PR that (I think!) works around what is (again, I think) a bug in php-ffi. Would you be able to test it?

#171

You shouldn't need to call shutdown, and hopefully it won't leak.

[craig410]

Hi @jcupitt,

Thanks for your speedy response. We're still seeing huge memory accumulation without calling shutdown. Roughly 250% increase over 10 requests. https://github.com/ingenerator/libvips-segfault-repro/actions/runs/3446792614/jobs/5752119000

[jcupitt]

Could that be the operation cache? libvips will try to cache recent operations, and has a 100mb cache limit by default.

You can change the default cache sizing here:

https://libvips.github.io/php-vips/classes/Jcupitt-Vips-Config.html#method_cacheSetMaxMem

You can also limit the cache by number of operations and number of open files.

I'd also beware of memory fragmentation. It'll take quite a few iterations (up to several 100?) before memory use stabilises.

[jcupitt]

Oh sigh, this still leaks:

#!/usr/bin/env php
<?php

require __DIR__ . '/vendor/autoload.php';
use Jcupitt\Vips;

#Vips\Config::setLogger(new Vips\DebugLogger());

Vips\Config::cacheSetMax(0);

if (count($argv) != 4) {
    echo("usage: ./watermark-text.php input output \"some text\"\n");
    exit(1);
}

$input_filename = $argv[1];
$output_filename = $argv[2];
$message = $argv[3];

function watermark($image, $message)
{
    $text = Vips\Image::text($message, [
      'width' => $image->width,
      'dpi' => 600,
      'rgba' => true
    ]);

    // scale the alpha down to make it semi-transparent, and rotate by 45
    // degrees
    $text = $text
        ->multiply([1, 1, 1, 0.3])
        ->rotate(45);

    // replicate and crop to match the size of the image
    $text = $text
        ->replicate(1 + $image->width / $text->width,
            1 + $image->height / $text->height)
        ->crop(0, 0, $image->width, $image->height);

    // and overlay on the image
    return $image->composite($text, "over");
}

echo "iteration, growth (bytes)\n";
$prev = 0;
for ($i = 0; $i < 2000; $i++) {
    $image = Vips\Image::newFromFile($input_filename, [
      'access' => 'sequential',
    ]);

    // no leak
    //$image = $image->invert()

    // leaks 192 bytes per loop
    $image = $image->add(1);

    // leaks 288 bytes per loop
    //$image = watermark($image, $message);

    $image->writeToFile($output_filename);

    if ($i % 10 == 0) {
        gc_collect_cycles();
        $now = memory_get_usage();
        $use = $now - $prev;
        $prev = $now;
        echo "$i, $use\n";
    }
}

Looks like a problem with constant handling. I'll have a further look.

[jcupitt]

... let's close this issue and discuss and test in the PR.

[craig410]

Could that be the operation cache? libvips will try to cache recent operations, and has a 100mb cache limit by default.

You can change the default cache sizing here:

https://libvips.github.io/php-vips/classes/Jcupitt-Vips-Config.html#method_cacheSetMaxMem

You can also limit the cache by number of operations and number of open files.

We've already tried using a tiny cache as well as disabling it completely and noticed no appreciable difference whatsoever. I kept it out of the reproduction to keep it simple.

I'd also beware of memory fragmentation. It'll take quite a few iterations (up to several 100?) before memory use stabilises.

I've run locally 400 iterations of the same operation with the default options and memory peaked at 200MiB and settled at 198MiB

With the options

Config::cacheSetMax(0);
Config::cacheSetMaxMem(1_000_000);
Config::cacheSetMaxFiles(0);

memory peaked at 156MiB after ~92 iterations and remained fairly constant.

We can pick up on PR #171 assuming we're fighting the same issue

[jcupitt]

Actually, I think this is something different. That PR is about small leaks in php-ffi (just a few 100 bytes per iteration), but it sounds like you're seeing something more serious.

[jcupitt]

I made this test program based on your example:

#!/usr/bin/env php
<?php

require __DIR__ . '/vendor/autoload.php';
use Jcupitt\Vips;

if (count($argv) != 2) {
    echo("usage: ./thumbnail-loop.php input\n");
    exit(1);
}

function thumbnail($filename)
{   
    $image = Vips\Image::thumbnail($filename, 600, [
        'height' => 10_000_000, 
        'export-profile' => 'srgb'
    ]);

    $buf = $image->writeToBuffer('.jpg', [
        'Q' => 75, 
        'strip' => TRUE, 
        'optimize_coding' => TRUE, 
        'profile'=>'srgb'
    ]); 

    return $buf;
}

echo "iteration, growth (bytes)\n";
$prev = 0;
for ($i = 0; $i < 1000; $i++) {
    $buf = thumbnail($argv[1]);

    if ($i % 10 == 0) {
        gc_collect_cycles();
        $now = memory_get_usage();
        $use = $now - $prev;
        $prev = $now;
        echo "$i, $use\n";
    }
}

Running against the php-vips in that PR I see:

$ ./thumbnail-loop.php ~/pics/k2.jpg
iteration, growth (bytes)
0, 1138920
10, 0
20, 0
...
980, 0
990, 0

So no leaks in 1000 iterations, according to memory_get_usage().

I'll try a version that watches RSS in ps instead.

[jcupitt]

This seems to work OK:

echo "iteration, now (kb), growth (kb)\n";
$prev = 0;
for ($i = 0; $i < 10000; $i++) {
    $buf = thumbnail($argv[1]);

    if ($i % 10 == 0) {
        gc_collect_cycles();
        $pid = getmypid();
        $now = intval(`ps --pid $pid --no-headers -orss`);
        $use = $now - $prev;
        $prev = $now;
        echo "$i, $now, $use\n";
    }
}

I tried graphing it:

So 160mb after 10,000 iterations. I think the slight upward trend is probably fragmentation. I'll try running it for 100,000 iterations.

[jcupitt]

Here's 55,000 iterations:

So I think it's fragmentation and there's no leak.

[acoulton]

Hi @jcupitt, I work with @craig410. Thank you for continuing to look at this & your comments over the weekend.

Unfortunately I think your test scripts don't accurately represent our runtime environment, so I wanted to give a bit more background to what we're seeing.

It looks like you are testing multiple iterations within the lifetime of a single start-to-finish PHP script (e.g. run as a shell command)?

We're running PHP within apache / mod_php to respond to incoming HTTP requests. There are multiple apache worker threads, each of which contains an embedded PHP runtime (there are no separately identifiable PHP processes). Those threads handle a succession of HTTP requests. The process does not exit between requests, it just resets state - clearing all variables and releasing all memory (apart from anything explicitly placed in shared memory e.g. caches).

Based on the normal behaviour of other PHP extensions, we would expect php-vips only to consume noticeable memory during requests that actually process images.

In production, image requests are a very small minority of our overall traffic, just occasionally thumbnailing relatively small files.

Historically we were using imagemagick through ext-imagick. The memory consumption of an apache worker process would peak a bit during an image request, then drop once the response had been served. Therefore the overall average memory usage across all apache processes was fairly low & stable.

As soon as we deployed our switch to libvips, we observed that apache memory usage would start to climb linearly with each new image processing request, and once memory was taken it would never be released. Memory growth seems to flatten over time, but it never actually drops until/unless the apache processes are terminated either during a deployment or due to the container hitting memory limits.

You can see that quite clearly in our production monitoring (mean of non-evictable memory per application container over time) - we switched to libvips on the 10th October. Metrics before the 22nd September have been archived, but the flat line at around 160 - 180 MiB at the left of the chart is pretty typical over the long term.

It was this that prompted @craig410 to start looking at options to control the libvips consumption:

• Disabling all the caching - which didn't seem to have a noticeable effect
• Then starting to explore FFI::shutdown() - which got memory consumption back to normal but triggered segfaults on subsequent requests

I know apache / mod_php could be considered a slightly dated setup, so I have tried today to run our reproduction example in php-fpm. It looks to me like the behaviour there is the same, with memory growing over time and never returning to the baseline.

I'm not particularly familiar with low-level memory allocation / fragmentation, but I wouldn't expect fragmentation to cause long-term issues in a mod_php / php_fpm runtime. That implies libvips is holding empty / fragmented runtime memory blocks between requests? My understanding is that normally a PHP extension would release these when the userland references are garbage collected / at the end of each request. FFI::shutdown() does reduce the measured memory, so even if the problem is fragmentation that implies it is within blocks that libvips is managing?

To summarise:

• In a long-lived process (mod_php / php_fpm), the process hosting libvips appears to hold a large amount of memory indefinitely - long after any userland variables have been garbage collected, and even if subsequent requests are not performing image operations at all. This is not behaviour we've seen with other php extensions in the past.
• We agree that running php-vips from a command line script appears to work as expected (regardless of number of iterations within the script), memory is released once the process exits back to the shell.
• Calling FFI::shutdown() after an image operation and before sending the response solves the memory consumption issue, but causes a segfault - apparently within glib - on subsequent requests.

Part of the motivation for migrating to libvips was the advertised memory consumption performance, but so far what we're seeing is that it's better than imagemagick while processing a specific image operation, but significantly worse in terms of requirements for long-term resource allocation.

As @craig410 said originally, we're partly trying to understand whether libvips / php-vips should actually work in a web hosting environment with parallel / long-running processes. It may be the architecture of the library & php bindings is only really designed for use in command-line scripting. If so, we can look at shelling out to a new process to perform an image operation and allow the operating system to clean that up.

Any further advice would be very welcome.

[jcupitt]

Hello @acoulton,

Ooof, I'm a bit vague on the details, but my understanding is that php does two types of memory allocation: per request (via emalloc) and persistent (via the system malloc).

Per request memory allocated via emalloc is automatically released at the end of a request when the request thread exits, and because it's allocated and freed in a huge chunk, can't get fragmented. Persistent objects on the other hand exist for the lifetime of the php process and are shared between threads. You can get fragmentation here, since the memory isn't pooled.

The php side of php-vips will mostly use emalloc, so that's all fine, but the native libvips binary will of course be using the system malloc, so it'll operate like php's persistent memory. If you call shutDown it'll free all the libvips memory for the whole process, so yes, it'll trigger segvs.

Looking at your graph, I think it's operating as expected. You're seeing an extra 100mb, typically, cased by having a native library doing significant amounts of malloc() in your webserver process. I don't think you're seeing a leak (ie. libvips is not losing memory).

As to what you should do I suppose the choices are:

1. Leave it like this. Is the extra 100mb an issue? If not, I'd be inclined to leave it. libvips ought to be significantly faster than ext-imagick (maybe a factor of 6 on this benchmark) so you could possibly use a lower powered server and see a saving that way.
2. Move image processing to a separate process. If there's significant image processing, it's probably not a good idea to do it in your main server process. I've put image handling into a separate worker process which takes jobs from a jobs table in the DB and works asynchronously. Keeping heavy CPU load tasks off your main server can be a good idea.
3. You could fork vipsthumbnail. This works fine for resizing and will give pretty good performance.

@kleisauke could I ping you on this (sorry)? You know a lot more about this subject than me!

[jcupitt]

Oh, as another option, libvips is fast enough that you could also resize images on demand rather than resizing on upload.

You might see a useful drop in storage costs, and users would have images sized exactly for their device.