Zstandard: Next-Gen Real-Time Compression

Facebook's high-speed compression algorithm with superior compression ratios

What is Zstandard (zstd)?

Zstandard is a real-time compression algorithm developed by Facebook that provides high compression ratios with very fast compression and decompression speeds. It's designed to scale from fast real-time scenarios to high-ratio archival use cases.

Accept-Encoding: zstd
Content-Encoding: zstd

Key Features of Zstandard

3-5x faster than gzip at similar compression ratios
Better compression than gzip (typically 10-15% smaller)
Compression levels from 1-22 (including negative levels)
Dictionary compression for small data
Streaming compression with low memory usage
Fast enough for real-time compression

Zstd Performance Comparison

vs Gzip

Speed: 3-5x faster

Ratio: 10-15% better

Zstd outperforms gzip in every metric

vs Brotli

Speed: Much faster

Ratio: Similar at high levels

Zstd better for real-time, Brotli for static

vs LZ4

Speed: Slightly slower

Ratio: Much better

Zstd offers better balance

Memory Usage

Compression: Configurable

Decompression: Very low

Scales from 1MB to 1GB+

Benchmark Results

# Compression Speed (MB/s) on Silesia Corpus
Algorithm  | Level | Ratio | Compress | Decompress
-----------|-------|-------|----------|------------
zstd       |   1   | 2.88  |   470    |    1380
zstd       |   3   | 3.17  |   300    |    1200  
gzip       |   1   | 2.74  |    90    |     370
gzip       |   6   | 3.09  |    34    |     380
brotli     |   1   | 2.87  |   340    |     350
brotli     |   4   | 3.29  |    48    |     400

Browser Support for Zstandard

Zstandard is newer than gzip and Brotli, with growing browser support:

Chrome/Edge

✅ Chrome 123+ (2024)

Behind flag in earlier versions

Firefox

✅ Firefox 126+ (2024)

Enabled by default

Safari

⏳ In development

Expected in future releases

Server Support

✅ Growing rapidly

CDNs adding support

# Modern browsers send:
Accept-Encoding: gzip, deflate, br, zstd

Implementing Zstandard Compression

Nginx Configuration (with module)

# Install nginx-module-zstd first
load_module modules/ngx_http_zstd_filter_module.so;
load_module modules/ngx_http_zstd_static_module.so;

http {
    # Enable Zstd compression
    zstd on;
    zstd_comp_level 3;  # 1-22, default 3
    zstd_min_length 256;
    zstd_types text/plain text/css text/javascript 
        application/javascript application/json 
        application/xml+rss;
    
    # Enable pre-compressed .zst files
    zstd_static on;
    
    # Also enable gzip/brotli for compatibility
    gzip on;
    brotli on;
}

Apache Configuration (experimental)

# Requires mod_zstd (not standard)
<IfModule mod_zstd.c>
    SetOutputFilter ZSTD
    ZstdCompressionLevel 3
    
    # Configure MIME types
    AddOutputFilterByType ZSTD text/html text/plain
    AddOutputFilterByType ZSTD text/css text/javascript
    AddOutputFilterByType ZSTD application/javascript
    AddOutputFilterByType ZSTD application/json
</IfModule>

Node.js Implementation

const express = require('express');
const zstd = require('@mongodb-js/zstd');

const app = express();

// Zstd compression middleware
function zstdMiddleware(options = {}) {
    const level = options.level || 3;
    
    return (req, res, next) => {
        const acceptEncoding = req.headers['accept-encoding'] || '';
        
        if (!acceptEncoding.includes('zstd')) {
            return next();
        }
        
        res.setHeader('Content-Encoding', 'zstd');
        res.setHeader('Vary', 'Accept-Encoding');
        
        const originalWrite = res.write;
        const originalEnd = res.end;
        const chunks = [];
        
        res.write = function(chunk) {
            chunks.push(chunk);
            return true;
        };
        
        res.end = function(chunk) {
            if (chunk) chunks.push(chunk);
            
            const buffer = Buffer.concat(chunks);
            zstd.compress(buffer, level).then(compressed => {
                res.setHeader('Content-Length', compressed.length);
                originalWrite.call(res, compressed);
                originalEnd.call(res);
            });
        };
        
        next();
    };
}

app.use(zstdMiddleware({ level: 3 }));

Zstandard Compression Levels

Zstd offers an unprecedented range of compression levels:

Negative Levels (-7 to -1)

Ultra-fast compression, lower ratios than gzip-1. Perfect for real-time streaming.

Low Levels (1-3)

Fast compression, better than gzip. Default level 3 recommended for web content.

Medium Levels (4-9)

Balanced compression. Level 6-7 matches Brotli quality 4.

High Levels (10-22)

Maximum compression for archival. Level 19+ requires significant memory.

Level Selection Guide

# Recommended levels for different use cases:
Real-time API:      -1 to 1   (>500 MB/s)
Dynamic content:     2 to 3   (300-400 MB/s)
Static assets:       3 to 6   (100-300 MB/s)
Pre-compression:     9 to 19  (<50 MB/s)
Archival:          19 to 22  (very slow)

Dictionary Compression with Zstd

Zstd's unique dictionary feature dramatically improves compression for small, similar files:

Creating a Dictionary

# Train a dictionary from sample files
zstd --train *.json -o dictionary.zstd

# Compress with dictionary
zstd -D dictionary.zstd input.json -o output.json.zst

# Results: 90%+ compression for small JSON files!

Use Cases for Dictionaries

API Responses

Similar JSON structures compress extremely well with shared dictionary.

Log Files

Repetitive log formats benefit greatly from dictionary compression.

Configuration Files

Small config files that would normally compress poorly.

Database Records

Similar structured data with common fields.

Pre-compressing with Zstandard

Command Line Usage

# Install zstd
apt-get install zstd      # Debian/Ubuntu
brew install zstd         # macOS
yum install zstd          # RHEL/CentOS

# Compress files
zstd -3 style.css -o style.css.zst
zstd -6 app.js -o app.js.zst

# Compress with multiple threads
zstd -T0 -6 large-file.json  # T0 = use all CPU cores

# Compress directory
tar cf - ./dist | zstd -3 -o dist.tar.zst

Build Tool Integration

// webpack.config.js
const CompressionPlugin = require('compression-webpack-plugin');
const zstd = require('@mongodb-js/zstd');

module.exports = {
    plugins: [
        new CompressionPlugin({
            filename: '[path][base].zst',
            algorithm: (input, compressionOptions, callback) => {
                zstd.compress(input, compressionOptions.level || 3)
                    .then(result => callback(null, result))
                    .catch(callback);
            },
            test: /\.(js|css|html|svg|json)$/,
            threshold: 1024,
            minRatio: 0.8,
            compressionOptions: { level: 6 },
        }),
    ],
};

Testing Zstandard Compression

Check Browser Support

// JavaScript detection
function supportsZstd() {
    const testUrl = '/test.txt';
    return fetch(testUrl, {
        headers: { 'Accept-Encoding': 'zstd' }
    }).then(response => {
        return response.headers.get('Content-Encoding') === 'zstd';
    }).catch(() => false);
}

Performance Testing

# Compare compression algorithms
echo "Test data" > test.txt
time gzip -c test.txt > test.gz
time brotli -c test.txt > test.br
time zstd -c test.txt > test.zst

# Check sizes
ls -la test.*

# Test decompression speed
time gzip -d < test.gz > /dev/null
time brotli -d < test.br > /dev/null
time zstd -d < test.zst > /dev/null

Zstandard Adoption Strategy

Implement zstd alongside existing compression for future-proof performance:

# Nginx: Enable all compression algorithms
zstd on;
zstd_comp_level 3;
brotli on;
brotli_comp_level 4;
gzip on;
gzip_comp_level 6;

# Server selects best supported algorithm automatically

As browser support grows, zstd will become the preferred compression method for its superior speed and compression ratio.