Compression Module

📦 Network compression utilities for ascii-chat More...

Files
file	compression.c
	🗜️ Fast zstd compression/decompression utilities for network payload optimization

Detailed Description

📦 Network compression utilities for ascii-chat

Compression README

Overview

Welcome to the Compression module! This is where we shrink your data down to save bandwidth. Compression is all about balance—we want to save bandwidth without wasting CPU, so we compress intelligently only when it actually helps.

What does the compression module do?

The compression module provides fast zstd compression/decompression utilities for optimizing network payload size in ascii-chat. Compression reduces bandwidth usage for large packets like video frames and audio batches, which improves performance over low-bandwidth connections.

Here's what it gives you:

• zstd compression algorithm (level 1 for speed/ratio balance)—we use zstd because it's fast and gives good compression ratios
• Automatic compression ratio checking (only compresses if beneficial)—we don't waste CPU compressing data that doesn't benefit
• Minimum size threshold to avoid overhead on small packets—we don't bother compressing tiny packets where compression overhead isn't worth it
• Memory-efficient compression/decompression**—we don't use a lot of memory for compression
• **Pure utility functions (no state management)—simple functions that you can call from anywhere

Implementation: lib/compression.c, lib/compression.h

Compression Strategy

Compression isn't always worth it—sometimes the compressed data is actually larger than the original, or the CPU cost isn't worth the bandwidth savings. We've designed the compression system to be smart about when to compress.

When do we compress?

Compression is automatically applied when:

• Packet size exceeds minimum threshold (1KB)—we don't compress tiny packets because the overhead isn't worth it
• Compression achieves at least 20% size reduction (<80% of original size)— if compression doesn't save at least 20%, we skip it

What problems does this prevent?

This strategy prevents several common problems:

• Overhead of compressing already-compressed data: If something is already compressed (like a JPEG image), trying to compress it again usually makes it bigger—we detect this and skip compression
• CPU waste on small packets: Compressing tiny packets often takes more CPU time than the bandwidth savings are worth—we don't bother with small packets
• Compression expansion: Sometimes compressed data is actually larger than the original (this is called "expansion")—we check for this and use the original data if compression expanded it

Algorithm

We use zstd (Zstandard) compression, which is a modern compression algorithm developed by Facebook/Meta. Here's why we chose it:

Why zstd?

• Uses zstd level 1 for optimal speed/ratio balance—level 1 is fast enough for real-time streaming but still gives good compression
• **Compatible with standard zstd decompression**—any zstd-compatible tool can decompress our data
• **Provides good compression ratio for text/ASCII data**—our ASCII frames compress really well (often 70-90% compression!)
• **Reasonable CPU overhead for real-time streaming**—we can compress at 200-500 MB/s, which is fast enough for real-time video

zstd level 1 is perfect for our use case—it's fast enough that it doesn't slow things down, but still gives us significant bandwidth savings.

API Usage

Basic Compression

void *compressed_data;
size_t compressed_size;
 
// Compress data (allocates output buffer)
if (compress_data(input_data, input_size, &compressed_data, &compressed_size) == 0) {
  // Check if compression is worthwhile (optional)
  if (should_compress(input_size, compressed_size)) {
    // Send compressed data
    send_packet(compressed_data, compressed_size);
  } else {
    // Send original data (compression didn't help)
    send_packet(input_data, input_size);
  }
  SAFE_FREE(compressed_data);
}

Decompression

void *decompressed_data;
size_t decompressed_size;
 
// Allocate output buffer (must know original size)
decompressed_data = SAFE_MALLOC(original_size, void *);
 
// Decompress data
if (decompress_data(compressed_data, compressed_size,
                    decompressed_data, original_size) == 0) {
  // Use decompressed data
  process_data(decompressed_data, original_size);
}
 
SAFE_FREE(decompressed_data);

Network Integration

Compression is automatically integrated into the packet protocol:

// In packet send path
if (data_size >= COMPRESSION_MIN_SIZE) {
  void *compressed;
  size_t compressed_size;
  if (compress_data(data, data_size, &compressed, &compressed_size) == 0) {
    if (should_compress(data_size, compressed_size)) {
      send_packet_header(type, compressed_size, compressed_flags);
      send_packet_data(compressed, compressed_size);
      free(compressed);
      return;
    }
    free(compressed);
  }
}
// Fall back to uncompressed
send_packet_header(type, data_size, flags);
send_packet_data(data, data_size);

Performance Characteristics

Compression performance is all about balancing speed and compression ratio. We've tuned the system for real-time video streaming, so speed is critical, but we still want good compression. Let's look at what you can expect:

How fast is compression?

Compression Speed:

• zstd level 1: ~200-500 MB/s compression (CPU-dependent)—this is fast enough that compression doesn't become a bottleneck
• Suitable for real-time video streaming (60fps)—we can compress video frames at 60 FPS without slowing things down
• **Low CPU overhead compared to higher compression levels**—level 1 is much faster than higher levels (level 3+), but still gives good compression

The speed varies depending on your CPU, but even on a modest CPU, you can compress at 200 MB/s, which is more than enough for real-time video.

How much bandwidth do we save?

Compression Ratio:

• ASCII text: 70-90% compression (significant bandwidth savings!)—this is the big win for ASCII frames
• Binary data: 60-80% compression—binary data compresses less well, but we still save bandwidth
• Already-compressed: Often expands (detected and avoided)—if something is already compressed (like a JPEG), trying to compress it again usually makes it bigger, so we detect this and skip compression

ASCII frames compress really well because they're text—all those repeated spaces and characters compress down nicely. You can often get 70-90% compression, which means a 6 MB frame becomes 600 KB-1.8 MB. That's a huge bandwidth savings!

How much memory does it use?

Memory Usage:

• Output buffer allocated by compress_data() (must be freed by caller)— we allocate the compressed output buffer, but you're responsible for freeing it
• Input buffer must be provided by caller for decompress_data()**—you provide the decompressed output buffer
• **No persistent state (pure functions)—no global state, no initialization needed, just call the functions whenever you need them

Memory usage is pretty reasonable—we allocate the compressed output buffer (which is usually smaller than the input), but there's no persistent state, so memory usage is minimal.

Configuration

Compression settings control when and how we compress. We've tuned these values for real-time video streaming, but let's understand what they mean:

What thresholds do we use?

Compression Thresholds:

• **COMPRESSION_MIN_SIZE** (1024 bytes): Minimum packet size to attempt compression—we don't compress packets smaller than 1KB because the overhead isn't worth it
• **COMPRESSION_RATIO_THRESHOLD** (0.8): Maximum acceptable compression ratio (80% of original)—if compression doesn't reduce size by at least 20%, we skip it

These thresholds ensure we only compress when it's actually beneficial. Small packets and packets that don't compress well are skipped, saving CPU without wasting bandwidth.

What compression level do we use?

Compression Level:

• Fixed at zstd level 1 for optimal speed/ratio balance—level 1 is the sweet spot for real-time streaming
• **Higher levels provide better compression but slower performance**—you could get better compression with level 3 or 5, but it would be much slower
• **Level 1 is ideal for real-time streaming applications**—fast enough for 60 FPS, but still gives good compression

We could use higher compression levels (which would save more bandwidth), but the CPU cost would be too high for real-time streaming. Level 1 is the perfect balance.

Error Handling

Compression Errors:

• Returns negative error code on failure
• Sets asciichat_errno for detailed error context
• Output buffer not allocated on failure (safe to ignore)

Decompression Errors:

• Returns negative error code on failure
• Sets asciichat_errno for detailed error context
• Output buffer unchanged on failure

Limitations

• zstd dependency: Requires zstd library at compile time
• Memory allocation: Compression allocates output buffer (must be freed)
• Size requirement: Must know original size for decompression
• Compression expansion: May expand small or already-compressed data

See also

lib/compression.h

lib/compression.c

topic_network

topic_av