Faster PHP behind FastCGI

Historical context: This is an experiment from 2011, built around lighttpd 1.4.28 and a custom PHP 5.3.6 fork. It is preserved because it documents an interesting attempt to reduce response copying across the FastCGI boundary. The required hooks were never part of stock PHP, and the accompanying lighttpd patch was tied to the server internals of that release.

The ordinary response path placed the generated body in PHP's output buffer, copied it through the FastCGI transport, and then let the web server send it to the client. For a large buffered response, the experiment instead wrote the body to a file on a memory-backed filesystem and returned only headers through FastCGI. lighttpd then opened that file and delivered it through its X-Sendfile mechanism.

The 2011 Setup

The setup combined three separate modifications:

1. A small patch for lighttpd 1.4.28 added the connection file descriptor as CFD and the server's cached timestamp as RAW_TIME to the FastCGI environment.
2. The custom PHP 5.3.6 fork provided ob_fwrite(), which copied PHP's active output buffer directly into an open stream. That function is not available in stock PHP.
3. /pipe was mounted as tmpfs. The generated response lived there until lighttpd served it with X-Sendfile or it was moved into a persistent cache.

The downloadable patch only exports CFD and RAW_TIME. It does not create the tmpfs mount, implement ob_fwrite(), configure X-Sendfile, or arrange cleanup. Those were independent parts of the original server and PHP setup.

The resulting response path was:

PHP output buffer -> tmpfs file -> lighttpd -> client

On a cache hit, PHP only selected an existing file and returned its path:

PHP cache lookup -> X-Sendfile header -> lighttpd -> client

A Corrected Version of the Original Example

The following listing keeps the PHP 5.3-era structure and the custom ob_fwrite() call, but fixes several defects in the published snippet. cachePathForRequest() stands for the application's own cache-key policy; cached output must not mix users, permissions, cookies, or other response variants.

<?php

function cachePathForRequest()
{
    $uri = isset($_SERVER['REQUEST_URI']) ? $_SERVER['REQUEST_URI'] : '/';
    return '/cache/' . hash('sha256', $uri) . '.html';
}

function acceptsGzip($header)
{
    foreach (explode(',', strtolower($header)) as $coding) {
        $parts = array_map('trim', explode(';', $coding));
        if ($parts[0] !== 'gzip' && $parts[0] !== '*') {
            continue;
        }

        foreach (array_slice($parts, 1) as $parameter) {
            if (preg_match('/^q\s*=\s*0(?:\.0*)?$/', $parameter)) {
                continue 2;
            }
        }
        return true;
    }
    return false;
}

function writeAll($fd, $buffer)
{
    $offset = 0;
    $length = strlen($buffer);

    while ($offset < $length) {
        $written = fwrite($fd, substr($buffer, $offset));
        if ($written === false || $written === 0) {
            return false;
        }
        $offset += $written;
    }
    return true;
}

$rawTime = isset($_SERVER['RAW_TIME']) ? (int) $_SERVER['RAW_TIME'] : time();
$clientFd = isset($_SERVER['CFD']) ? (int) $_SERVER['CFD'] : -1;

if ($clientFd < 0) {
    throw new RuntimeException('The patched FastCGI environment is missing CFD');
}

$pipePath = sprintf('/pipe/%u-%u.html', $rawTime, $clientFd);
$fd = @fopen($pipePath, 'xb');

// A descriptor can be reused within the same second. Never overwrite its file.
if ($fd === false) {
    $pipePath = tempnam('/pipe', 'fcgi-');
    $fd = $pipePath === false ? false : @fopen($pipePath, 'wb');
}

if ($fd === false) {
    throw new RuntimeException('Unable to create the response file');
}

ob_start(null, 0x20000);
echo 'Write the generated response into the output buffer';

if (BUFFER_CONTENTS) {
    $buffer = ob_get_clean();
    $writeOk = writeAll($fd, $buffer);
} else {
    $writeOk = ob_fwrite($fd) !== false;
    ob_end_clean();
}

if (!fclose($fd) || !$writeOk) {
    @unlink($pipePath);
    throw new RuntimeException('Unable to write the complete response');
}

$sendPath = $pipePath;

if (CACHE_CONTENTS) {
    $cachePath = cachePathForRequest();
    if (!rename($pipePath, $cachePath)) {
        @unlink($pipePath);
        throw new RuntimeException('Unable to publish the cached response');
    }
    $sendPath = $cachePath;

    $acceptEncoding = isset($_SERVER['HTTP_ACCEPT_ENCODING'])
        ? $_SERVER['HTTP_ACCEPT_ENCODING']
        : '';

    if (acceptsGzip($acceptEncoding)) {
        $gzipPath = $cachePath . '.gz';
        $command = 'gzip -c -- ' . escapeshellarg($cachePath)
            . ' > ' . escapeshellarg($gzipPath);
        exec($command, $output, $exitCode);

        if ($exitCode === 0) {
            header('Content-Encoding: gzip');
            header('Vary: Accept-Encoding');
            $sendPath = $gzipPath;
        } else {
            @unlink($gzipPath);
        }
    }
}

$httpStatus = 200;
header('X-Sendfile: ' . $sendPath, true, $httpStatus);

What Was Fixed

• fopen() now has a write mode. The original one-argument call cannot open a PHP file stream.
• The BUFFER_CONTENTS branch now writes the value returned by ob_get_clean(). Previously it closed an empty file and discarded the generated body.
• $sendPath, $cachePath, and $gzipPath are initialized in the branches where they are used. The original non-cache gzip branch referenced an undefined $path.
• gzip reads the absolute cache path, writes to a separate .gz file, checks its exit status, and escapes both shell arguments.
• An Accept-Encoding entry with q=0 no longer enables gzip, and compressed responses include Vary: Accept-Encoding.
• The timestamp plus descriptor name is opened with exclusive creation. On file-descriptor reuse within the same second, tempnam() supplies a collision-free fallback instead of overwriting another response.

Limits of the Technique

X-Sendfile does not delete a file after sending it. Persistent cache files need eviction, while transient files need server-side deletion support or a separate cleanup job. PHP cannot safely unlink a transient file immediately after setting the header because lighttpd opens it only after the FastCGI response has finished.

The tmpfs path must be visible to both processes, so this design assumes PHP and lighttpd run on the same host or share an equivalent filesystem. A remote FastCGI backend cannot hand the front-end server a local pathname. The X-Sendfile configuration must also restrict allowed roots; accepting arbitrary application-controlled paths would expose files outside the response cache.

The optimization only applies to a complete, buffered response. It delays the time to first byte because the web server cannot start delivery until PHP has finished generating and closing the file. That runs contrary to early flushing, and it may delay discovery of stylesheets and scripts in the returned HTML.

Writing to tmpfs does not eliminate every copy: PHP still copies bytes into its output buffer and then into the memory-backed file, while lighttpd reads that file for transmission. The avoided work is specifically the transfer of the full body through the FastCGI stream. The extra file operations can outweigh that saving for small responses; large responses are the plausible target, and the result has to be measured on the actual kernel, FastCGI setup, and workload.

Historical Takeaway

The durable idea was not the patched descriptor variable itself. It was the separation between generating a response and serving a finished file. X-Sendfile remains useful for authorized downloads and prebuilt cache entries, where the web server can handle the transfer after PHP has made the access decision. The direct tmpfs response path described here should be read as a record of a specific 2011 optimization experiment, not as a portable replacement for normal FastCGI output.

References

• [lighttpd]lighttpd project documentation. X-Sendfile and X-LIGHTTPD-send-file.
• [PHPStreams]PHP Documentation Group. PHP Manual: fopen and filesystem streams.
• [PHPShell]PHP Documentation Group. PHP Manual: escapeshellarg.