Generate and Cache Thumbnails on Demand with lighttpd

Generating every possible image size during upload wastes storage and processing time when most variants are never requested. Generating every variant on every request is worse: latency becomes unpredictable and an image decoder sits directly on a public hot path. A useful middle ground is to generate a derivative on its first request, publish it into a filesystem cache, and let the web server handle every later request as an ordinary static file.

The important boundary is the cache miss. Current lighttpd mod_magnet documentation warns that blocking work inside a Lua magnet script blocks lighttpd's event loop. ImageMagick therefore belongs in a bounded FastCGI worker, not in mod_magnet through os.execute(). A cache hit still returns a static file without entering PHP; only a missing derivative reaches the worker.

Use a finite, content-addressed URL space

Allow named presets rather than accepting arbitrary width, height, quality, or format parameters. The examples below expose two variants:

/media/cache/square-120/9d7f...e21a.jpg
/media/cache/fit-480/9d7f...e21a.jpg

The final component is the lowercase SHA-256 hash assigned to the original during ingestion. In the real request it contains exactly 64 hexadecimal characters. Store originals outside the public document root as /srv/media/original/<hash>.jpg or .png, provision writable preset directories under /srv/media/cache, and create a private lock directory at /run/thumbnail-locks. The FastCGI account needs read access to originals and write access only to the derivative cache and lock directory.

A content-addressed identifier prevents an existing URL from silently acquiring new source pixels. Preset names bound the work and stop clients from filling the cache with arbitrary dimensions. When an original changes, its new content hash creates a new URL.

Send cache misses to FastCGI

Enable mod_rewrite, mod_fastcgi, and mod_setenv. The exact PHP-FPM socket varies by operating system, but the request routing can remain small:

server.modules += ( "mod_rewrite", "mod_fastcgi", "mod_setenv" )

fastcgi.server = ( ".php" =>
  ( "php" =>
    ( "socket" => "/run/php/php-fpm.sock",
            "broken-scriptfilename" => "enable",
            "x-sendfile" => "enable",
            "x-sendfile-docroot" => ( "/srv/media/cache/" )
    )
  )
)

$HTTP["url"] =~ "^/media/cache/" {
  url.rewrite-if-not-file = (
    "^/media/cache/(square-120|fit-480)/[0-9a-f]{64}\.jpg$" => "/thumbnail.php"
  )

  setenv.add-response-header = (
    "Cache-Control" => "public, max-age=31536000, immutable"
  )
}

url.rewrite-if-not-file is the fast-path switch. Existing derivatives continue through lighttpd's static-file handler. A miss is internally rewritten to thumbnail.php, while the original request URI remains available to PHP for strict parsing. After generation, X-Sendfile hands the completed file back to lighttpd instead of copying its bytes through PHP. The mandatory x-sendfile-docroot boundary prevents this trusted backend from asking lighttpd to expose files outside the derivative cache.

The lighttpd condition supplies the cache policy to both response paths, so the generator does not emit a second Cache-Control field. Lighttpd obtains Content-Length from the file and handles byte ranges itself. Restrict this virtual host to GET and HEAD if no other methods are required.

Generate one complete derivative

The worker below accepts only the two routes declared above. It selects a source beneath a fixed root, rejects unknown formats and oversized input, constructs ImageMagick as an argument array, and never interpolates request data into a shell command. The preset arguments are server-owned constants.

<?php
declare(strict_types=1);

const ORIGINAL_ROOT = '/srv/media/original';
const CACHE_ROOT = '/srv/media/cache';
const LOCK_ROOT = '/run/thumbnail-locks';
const MAX_SOURCE_BYTES = 50_000_000;
const MAX_SOURCE_PIXELS = 40_000_000;

function fail(int $status, string $message): never
{
    http_response_code($status);
    header('Content-Type: text/plain; charset=utf-8');
    exit($message . "\n");
}

function handOffThumbnail(string $path): never
{
    header('Content-Type: image/jpeg');
    header('Last-Modified: ' . gmdate('D, d M Y H:i:s', (int) filemtime($path)) . ' GMT');
    header('X-Sendfile: ' . str_replace('%2F', '/', rawurlencode($path)));
    exit;
}

if (!in_array($_SERVER['REQUEST_METHOD'], ['GET', 'HEAD'], true)) {
    header('Allow: GET, HEAD');
    fail(405, 'Method not allowed');
}

$requestPath = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);
if (!is_string($requestPath) || !preg_match(
    '#\A/media/cache/(square-120|fit-480)/([0-9a-f]{64})\.jpg\z#D',
    $requestPath,
    $matches,
)) {
    fail(404, 'Not found');
}

$preset = $matches[1];
$sourceId = $matches[2];
$presets = [
    'square-120' => ['-thumbnail', '120x120^', '-gravity', 'center', '-extent', '120x120'],
    'fit-480' => ['-thumbnail', '480x480>'],
];

$targetDirectory = CACHE_ROOT . '/' . $preset;
$target = $targetDirectory . '/' . $sourceId . '.jpg';
if (is_file($target)) {
    handOffThumbnail($target);
}

$source = null;
foreach (['jpg', 'png'] as $extension) {
    $candidate = ORIGINAL_ROOT . '/' . $sourceId . '.' . $extension;
    if (is_file($candidate)) {
        $source = $candidate;
        break;
    }
}
if ($source === null) {
    fail(404, 'Source not found');
}

$sourceBytes = filesize($source);
$sourceInfo = getimagesize($source);
$allowedMimeTypes = ['image/jpeg', 'image/png'];
if (
    $sourceBytes === false
    || $sourceBytes > MAX_SOURCE_BYTES
    || $sourceInfo === false
    || !in_array($sourceInfo['mime'] ?? '', $allowedMimeTypes, true)
    || $sourceInfo[0] * $sourceInfo[1] > MAX_SOURCE_PIXELS
) {
    fail(415, 'Unsupported source image');
}

$decoder = $sourceInfo['mime'] === 'image/jpeg' ? 'jpeg:' : 'png:';
$lockPath = LOCK_ROOT . '/' . $preset . '-' . $sourceId . '.lock';
$lock = fopen($lockPath, 'c');
if ($lock === false || !flock($lock, LOCK_EX)) {
    fail(503, 'Unable to lock derivative');
}

$generationError = null;
$temporary = null;

try {
    if (is_file($target)) {
        handOffThumbnail($target);
    }

    $temporary = tempnam($targetDirectory, '.thumbnail-');
    if ($temporary === false) {
        throw new RuntimeException('Unable to allocate a temporary file');
    }

    $command = array_merge(
        [
            '/usr/bin/magick',
            '-limit', 'thread', '2',
            '-limit', 'memory', '256MiB',
            '-limit', 'map', '512MiB',
            '-limit', 'disk', '1GiB',
            '-limit', 'time', '20',
            $decoder . $source . '[0]',
            '-auto-orient',
            '-strip',
        ],
        $presets[$preset],
        [
            '-background', '#fff',
            '-alpha', 'remove',
            '-alpha', 'off',
            '-quality', '85',
            'jpeg:' . $temporary,
        ],
    );

    $descriptorSpec = [
        0 => ['file', '/dev/null', 'r'],
        1 => ['file', '/dev/null', 'a'],
        2 => ['pipe', 'w'],
    ];
    $process = proc_open(
        $command,
        $descriptorSpec,
        $pipes,
        null,
        [],
        ['bypass_shell' => true],
    );
    if (!is_resource($process)) {
        throw new RuntimeException('Unable to start ImageMagick');
    }

    $errorOutput = stream_get_contents($pipes[2]);
    fclose($pipes[2]);
    $exitCode = proc_close($process);

    $outputInfo = is_file($temporary) ? getimagesize($temporary) : false;
    if ($exitCode !== 0 || $outputInfo === false || ($outputInfo['mime'] ?? '') !== 'image/jpeg') {
        throw new RuntimeException('ImageMagick failed: ' . trim((string) $errorOutput));
    }

    if (!rename($temporary, $target)) {
        throw new RuntimeException('Unable to publish derivative');
    }
    $temporary = null;
} catch (Throwable $error) {
    $generationError = $error;
} finally {
    if (is_string($temporary) && is_file($temporary)) {
        unlink($temporary);
    }
    flock($lock, LOCK_UN);
    fclose($lock);
}

if ($generationError !== null) {
    error_log($generationError->getMessage());
    fail(500, 'Thumbnail generation failed');
}

handOffThumbnail($target);

The first request for a derivative takes an exclusive per-target lock. Requests arriving during the same miss wait, recheck is_file($target), and reuse the result instead of starting another conversion. tempnam() creates the temporary output in the target directory; consequently rename($temporary, $target) publishes the complete file atomically on the same filesystem. A failed decoder cannot leave a partial JPEG at the public cache path. rawurlencode() protects the response header, while the replacement preserves path separators required for the absolute X-Sendfile path.

PHP is convenient for the validation and ImageMagick orchestration above, but the final FastCGI response does not require the PHP engine. A native worker can use the same handoff once its complete validation, locking, conversion, and atomic-publication routine has returned a trusted path. The deliberately unresolved validate_and_generate_thumbnail() declaration below is that application-specific routine; it must never be replaced by directly concatenating REQUEST_URI onto the cache root.

#include <fcgi_stdio.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>

#define CACHE_ROOT "/srv/media/cache/"

int validate_and_generate_thumbnail(
    const char *request_uri,
    char *path,
    size_t path_size
);

static int is_cache_path(const char *path)
{
    const size_t root_length = sizeof(CACHE_ROOT) - 1;

    if (strncmp(path, CACHE_ROOT, root_length) != 0) {
        return 0;
    }

    const char *relative = path + root_length;
    return *relative != '\0'
        && strstr(relative, "..") == NULL
        && strspn(relative, "abcdefghijklmnopqrstuvwxyz0123456789-./")
            == strlen(relative);
}

int main(void)
{
    while (FCGI_Accept() >= 0) {
        const char *request_uri = getenv("REQUEST_URI");
        char path[4096] = {0};

        if (request_uri == NULL
            || validate_and_generate_thumbnail(request_uri, path, sizeof(path)) != 1) {
            printf("Status: 404 Not Found\r\n");
            printf("Content-Type: text/plain; charset=utf-8\r\n\r\n");
            printf("Not found\n");
        } else if (!is_cache_path(path)) {
            printf("Status: 500 Internal Server Error\r\n\r\n");
        } else {
            printf("Status: 200 OK\r\n");
            printf("Content-Type: image/jpeg\r\n");
            printf("X-Sendfile: %s\r\n\r\n", path);
        }
    }

    return 0;
}

The PHP process is still a security boundary. Run a small dedicated PHP-FPM pool with a finite worker count, request timeout, read-only access to originals, and no network access where the operating system can enforce it. The source-ingestion path must compute the hash, reject files over its own limits, and ensure that the web account cannot replace originals with symlinks.

Constrain ImageMagick outside the script

Command-line -limit options provide per-invocation bounds, but an ImageMagick Security Policy is the non-bypassable backstop. Start from the web-safe policy shipped for the installed ImageMagick release, then restrict the service to the coders it actually needs. A minimal policy fragment for this JPEG-and-PNG pipeline is:

<policy domain="resource" name="memory" value="256MiB" />
<policy domain="resource" name="map" value="512MiB" />
<policy domain="resource" name="disk" value="1GiB" />
<policy domain="resource" name="time" value="20" />
<policy domain="coder" rights="none" pattern="*" />
<policy domain="coder" rights="read" pattern="{JPEG,PNG}" />
<policy domain="coder" rights="write" pattern="JPEG" />
<policy domain="path" rights="none" pattern="@*" />

Confirm the effective policy with magick identify -list policy. Policy locations and defaults are installation-specific. Keep ImageMagick patched, disable unused delegates and coders, and test the exact production build with malformed and unusually large inputs.

Cache validation and invalidation

In Cache-Control, max-age is a duration in seconds, not an absolute Unix timestamp. One year is therefore max-age=31536000. The immutable directive is valid here because changing source bytes creates a new content hash and therefore a new URL.

A preset also describes processing behavior. If dimensions, crop rules, quality, color handling, or encoder settings change, deploy a new name such as square-120-v2. New HTML can reference that name immediately while old URLs remain valid in browser and CDN caches. After their retention period, purge the old preset directory as a batch.

Do not use access time to decide which derivatives are popular. Many filesystems disable or defer atime updates, and a CDN hit never touches the origin file. Use cache-directory size and creation or modification time for a simple age-based policy, or collect request metrics at lighttpd or the CDN when popularity matters. Remove only known preset directories, never paths assembled from unvalidated request input.

Operational checks

• Request one new URL concurrently and verify that only one ImageMagick process runs.
• Confirm that the generated file is a JPEG with the exact preset dimensions.
• Request it again and verify that PHP-FPM receives no second request.
• Send unknown presets, malformed hashes, traversal strings, oversized files, animated images, and unsupported formats; all must fail without creating a public cache file.
• Monitor generation count, duration, failures, lock wait time, PHP-FPM queue depth, cache size, and free disk space.

At larger scale, pre-generate the handful of variants visible above the fold and retain lazy generation for the long tail. A bounded worker pool protects the server during a miss burst; a CDN in front of the content-addressed URLs keeps most requests away from the origin entirely.

References

• [lighttpd-magnet]lighttpd mod_magnet documentation, current request API, hooks, and warning about blocking operations.
• [lighttpd-rewrite]lighttpd mod_rewrite documentation, conditional rewriting for missing files.
• [lighttpd-fastcgi]lighttpd mod_fastcgi documentation, FastCGI backend options and constrained X-Sendfile support.
• [ImageMagick]ImageMagick Security Policy, resource limits and coder restrictions.
• [PHP-proc-open]PHP Manual: proc_open, array commands and process descriptors.
• [RFC9111]RFC 9111: HTTP Caching, freshness lifetimes and cache validation.