Compression Module

📦 Network compression utilities for ascii-chat More...

Files
file	compression.c
	🗜️ Fast zstd compression/decompression utilities for network payload optimization
file	compression.h
	📦 Network Packet Compression Utilities

Functions
asciichat_error_t	compress_data (const void *input, size_t input_size, void **output, size_t *output_size, int compression_level)
	Compress data using zstd with configurable compression level.
asciichat_error_t	decompress_data (const void *input, size_t input_size, void *output, size_t output_size)
	Decompress data using zstd.
bool	should_compress (size_t original_size, size_t compressed_size)
	Determine if compression should be used for given data sizes.

Compression Settings
#define	COMPRESSION_RATIO_THRESHOLD   0.8f
	Compression ratio threshold - only use if <80% original size.
#define	COMPRESSION_MIN_SIZE   1024
	Minimum packet size to attempt compression (1KB)

Detailed Description

📦 Network compression utilities for ascii-chat

This header provides compression and decompression utilities for network packets in ascii-chat. The system uses zstd compression to reduce bandwidth usage for large packets like video frames.

CORE FEATURES:

• zstd compression algorithm
• Automatic compression ratio checking (only use if beneficial)
• Minimum size threshold to avoid compressing small packets
• Memory-efficient compression/decompression
• Pure utility functions (no state management)

COMPRESSION STRATEGY:

Compression is only applied when:

• Packet size exceeds minimum threshold (1KB)
• Compression achieves at least 20% size reduction (<80% of original)

This prevents:

• Overhead of compressing already-compressed data
• CPU waste on small packets that don't benefit
• Compression expansion (when compressed data is larger)

ALGORITHM:

• Uses Facebook's zstd deflate for compression
• Provides good compression ratio for text/ASCII data
• Reasonable CPU overhead for real-time streaming

Note

Compression is optional and only used when beneficial.

Frame sending functions have been moved to network.h/network.c.

The compression ratio threshold ensures compression is only used when it provides meaningful bandwidth savings.

Author

Zachary Fogg me@zf.nosp@m.o.gg

Date

August 2025

Compression README

Overview

Welcome to the Compression module! This is where we shrink your data down to save bandwidth. Compression is all about balance—we want to save bandwidth without wasting CPU, so we compress intelligently only when it actually helps.

What does the compression module do?

The compression module provides fast zstd compression/decompression utilities for optimizing network payload size in ascii-chat. Compression reduces bandwidth usage for large packets like video frames and audio batches, which improves performance over low-bandwidth connections.

Here's what it gives you:

• zstd compression algorithm (level 1 for speed/ratio balance)—we use zstd because it's fast and gives good compression ratios
• Automatic compression ratio checking (only compresses if beneficial)—we don't waste CPU compressing data that doesn't benefit
• Minimum size threshold to avoid overhead on small packets—we don't bother compressing tiny packets where compression overhead isn't worth it
• Memory-efficient compression/decompression**—we don't use a lot of memory for compression
• **Pure utility functions (no state management)—simple functions that you can call from anywhere

Implementation: lib/compression.c, lib/compression.h

Compression Strategy

Compression isn't always worth it—sometimes the compressed data is actually larger than the original, or the CPU cost isn't worth the bandwidth savings. We've designed the compression system to be smart about when to compress.

When do we compress?

Compression is automatically applied when:

• Packet size exceeds minimum threshold (1KB)—we don't compress tiny packets because the overhead isn't worth it
• Compression achieves at least 20% size reduction (<80% of original size)— if compression doesn't save at least 20%, we skip it

What problems does this prevent?

This strategy prevents several common problems:

• Overhead of compressing already-compressed data: If something is already compressed (like a JPEG image), trying to compress it again usually makes it bigger—we detect this and skip compression
• CPU waste on small packets: Compressing tiny packets often takes more CPU time than the bandwidth savings are worth—we don't bother with small packets
• Compression expansion: Sometimes compressed data is actually larger than the original (this is called "expansion")—we check for this and use the original data if compression expanded it

Algorithm

We use zstd (Zstandard) compression, which is a modern compression algorithm developed by Facebook/Meta. Here's why we chose it:

Why zstd?

• Uses zstd level 1 for optimal speed/ratio balance—level 1 is fast enough for real-time streaming but still gives good compression
• **Compatible with standard zstd decompression**—any zstd-compatible tool can decompress our data
• **Provides good compression ratio for text/ASCII data**—our ASCII frames compress really well (often 70-90% compression!)
• **Reasonable CPU overhead for real-time streaming**—we can compress at 200-500 MB/s, which is fast enough for real-time video

zstd level 1 is perfect for our use case—it's fast enough that it doesn't slow things down, but still gives us significant bandwidth savings.

API Usage

Basic Compression

void *compressed_data;
size_t compressed_size;
 
// Compress data (allocates output buffer)
if (compress_data(input_data, input_size, &compressed_data, &compressed_size) == 0) {
  // Check if compression is worthwhile (optional)
  if (should_compress(input_size, compressed_size)) {
    // Send compressed data
    send_packet(compressed_data, compressed_size);
  } else {
    // Send original data (compression didn't help)
    send_packet(input_data, input_size);
  }
  SAFE_FREE(compressed_data);
}

Decompression

void *decompressed_data;
size_t decompressed_size;
 
// Allocate output buffer (must know original size)
decompressed_data = SAFE_MALLOC(original_size, void *);
 
// Decompress data
if (decompress_data(compressed_data, compressed_size,
                    decompressed_data, original_size) == 0) {
  // Use decompressed data
  process_data(decompressed_data, original_size);
}
 
SAFE_FREE(decompressed_data);

Network Integration

Compression is automatically integrated into the packet protocol:

// In packet send path
if (data_size >= COMPRESSION_MIN_SIZE) {
  void *compressed;
  size_t compressed_size;
  if (compress_data(data, data_size, &compressed, &compressed_size) == 0) {
    if (should_compress(data_size, compressed_size)) {
      send_packet_header(type, compressed_size, compressed_flags);
      send_packet_data(compressed, compressed_size);
      free(compressed);
      return;
    }
    free(compressed);
  }
}
// Fall back to uncompressed
send_packet_header(type, data_size, flags);
send_packet_data(data, data_size);

Performance Characteristics

Compression performance is all about balancing speed and compression ratio. We've tuned the system for real-time video streaming, so speed is critical, but we still want good compression. Let's look at what you can expect:

How fast is compression?

Compression Speed:

• zstd level 1: ~200-500 MB/s compression (CPU-dependent)—this is fast enough that compression doesn't become a bottleneck
• Suitable for real-time video streaming (60fps)—we can compress video frames at 60 FPS without slowing things down
• **Low CPU overhead compared to higher compression levels**—level 1 is much faster than higher levels (level 3+), but still gives good compression

The speed varies depending on your CPU, but even on a modest CPU, you can compress at 200 MB/s, which is more than enough for real-time video.

How much bandwidth do we save?

Compression Ratio:

• ASCII text: 70-90% compression (significant bandwidth savings!)—this is the big win for ASCII frames
• Binary data: 60-80% compression—binary data compresses less well, but we still save bandwidth
• Already-compressed: Often expands (detected and avoided)—if something is already compressed (like a JPEG), trying to compress it again usually makes it bigger, so we detect this and skip compression

ASCII frames compress really well because they're text—all those repeated spaces and characters compress down nicely. You can often get 70-90% compression, which means a 6 MB frame becomes 600 KB-1.8 MB. That's a huge bandwidth savings!

How much memory does it use?

Memory Usage:

• Output buffer allocated by compress_data() (must be freed by caller)— we allocate the compressed output buffer, but you're responsible for freeing it
• Input buffer must be provided by caller for decompress_data()**—you provide the decompressed output buffer
• **No persistent state (pure functions)—no global state, no initialization needed, just call the functions whenever you need them

Memory usage is pretty reasonable—we allocate the compressed output buffer (which is usually smaller than the input), but there's no persistent state, so memory usage is minimal.

Configuration

Compression settings control when and how we compress. We've tuned these values for real-time video streaming, but let's understand what they mean:

What thresholds do we use?

Compression Thresholds:

• **COMPRESSION_MIN_SIZE** (1024 bytes): Minimum packet size to attempt compression—we don't compress packets smaller than 1KB because the overhead isn't worth it
• **COMPRESSION_RATIO_THRESHOLD** (0.8): Maximum acceptable compression ratio (80% of original)—if compression doesn't reduce size by at least 20%, we skip it

These thresholds ensure we only compress when it's actually beneficial. Small packets and packets that don't compress well are skipped, saving CPU without wasting bandwidth.

What compression level do we use?

Compression Level:

• Fixed at zstd level 1 for optimal speed/ratio balance—level 1 is the sweet spot for real-time streaming
• **Higher levels provide better compression but slower performance**—you could get better compression with level 3 or 5, but it would be much slower
• **Level 1 is ideal for real-time streaming applications**—fast enough for 60 FPS, but still gives good compression

We could use higher compression levels (which would save more bandwidth), but the CPU cost would be too high for real-time streaming. Level 1 is the perfect balance.

Error Handling

Compression Errors:

• Returns negative error code on failure
• Sets asciichat_errno for detailed error context
• Output buffer not allocated on failure (safe to ignore)

Decompression Errors:

• Returns negative error code on failure
• Sets asciichat_errno for detailed error context
• Output buffer unchanged on failure

Limitations

• zstd dependency: Requires zstd library at compile time
• Memory allocation: Compression allocates output buffer (must be freed)
• Size requirement: Must know original size for decompression
• Compression expansion: May expand small or already-compressed data

See also

lib/compression.h

lib/compression.c

topic_network

topic_av

Macro Definition Documentation

COMPRESSION_MIN_SIZE

#define COMPRESSION_MIN_SIZE   1024

#include <compression.h>

Minimum packet size to attempt compression (1KB)

Definition at line 61 of file compression.h.

COMPRESSION_RATIO_THRESHOLD

#define COMPRESSION_RATIO_THRESHOLD   0.8f

#include <compression.h>

Compression ratio threshold - only use if <80% original size.

Definition at line 58 of file compression.h.

Function Documentation

compress_data()

asciichat_error_t compress_data	(	const void *	input,
		size_t	input_size,
		void **	output,
		size_t *	output_size,
		int	compression_level
	)

#include <compression.h>

Compress data using zstd with configurable compression level.

Parameters

input	Input data to compress (must not be NULL)
input_size	Size of input data in bytes
output	Output buffer pointer (pointer-to-pointer; function allocates and stores address here)
output_size	Size of compressed data in bytes (output parameter, must not be NULL)
compression_level	zstd compression level (1-9) Level 1: Fastest compression, lowest ratio (best for real-time streaming) Level 3: Balanced speed/ratio Level 9: Slower compression, best ratio (for limited bandwidth)

Returns

ASCIICHAT_OK on success, error code on failure

Compresses input data using zstd's compression algorithm with the specified compression level. The output buffer is automatically allocated by the function and must be freed by the caller using SAFE_FREE() or the appropriate memory management function.

Note

The output buffer is allocated using SAFE_MALLOC(). Caller must free it with SAFE_FREE() when done using the compressed data.

Compression may fail if input data is already compressed or if compression would expand the data significantly.

For best performance, use should_compress() first to determine if compression is beneficial before calling this function.

Compression level must be between 1 and 9. Levels above 9 are not recommended for real-time streaming due to CPU overhead.

Warning

Caller must SAFE_FREE() the output buffer to avoid memory leaks.

Definition at line 14 of file compression.c.

                                                       {
  if (!input || input_size == 0 || !output || !output_size) {
    return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid parameters: input=%p, input_size=%zu, output=%p, output_size=%p",
                     input, input_size, output, output_size);
  }
 
  // Validate compression level (1-9 for real-time streaming)
  if (compression_level < 1 || compression_level > 9) {
    return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid compression level %d: must be between 1 and 9", compression_level);
  }
 
  // Calculate maximum compressed size
  size_t compressed_size = ZSTD_compressBound(input_size);
 
  // Sanity check ZSTD_compressBound result
  // ZSTD_compressBound returns 0 for errors or very large values for huge inputs
  if (compressed_size == 0 || compressed_size > 256 * 1024 * 1024) { // Max 256MB compressed buffer
    return SET_ERRNO(ERROR_INVALID_PARAM, "ZSTD_compressBound returned unreasonable size: %zu for input %zu",
                     compressed_size, input_size);
  }
 
  unsigned char *compressed_data = NULL;
  compressed_data = SAFE_MALLOC(compressed_size, unsigned char *);
 
  if (!compressed_data) {
    return SET_ERRNO(ERROR_MEMORY, "Failed to allocate compressed data buffer");
  }
 
  // Compress using zstd with user-specified compression level
  size_t ret = ZSTD_compress(compressed_data, compressed_size, input, input_size, compression_level);
 
  if (ZSTD_isError(ret)) {
    SAFE_FREE(compressed_data);
    return SET_ERRNO(ERROR_GENERAL, "zstd compression failed: %s", ZSTD_getErrorName(ret));
  }
 
  *output = compressed_data;
  *output_size = ret;
 
  return ASCIICHAT_OK;
}

References ASCIICHAT_OK, ERROR_GENERAL, ERROR_INVALID_PARAM, ERROR_MEMORY, SAFE_FREE, SAFE_MALLOC, and SET_ERRNO.

Referenced by send_packet_secure().

decompress_data()

asciichat_error_t decompress_data	(	const void *	input,
		size_t	input_size,
		void *	output,
		size_t	output_size
	)

#include <compression.h>

Decompress data using zstd.

Parameters

input	Compressed input data (must not be NULL)
input_size	Size of compressed data in bytes
output	Pre-allocated output buffer (must not be NULL)
output_size	Size of output buffer in bytes (must be >= decompressed size)

Returns

ASCIICHAT_OK on success, error code on failure

Decompresses zstd-compressed data into a pre-allocated output buffer. The output buffer must be large enough to hold the decompressed data.

Note

The output buffer size must be known in advance (typically from packet header or protocol specification). This function does not dynamically allocate the output buffer.

This function uses zstd inflate algorithm, compatible with standard zstd compression.

Warning

Output buffer must be large enough for decompressed data or buffer overflow will occur. Ensure output_size is correct before calling.

Definition at line 58 of file compression.c.

                                                                                                          {
  if (!input || input_size == 0 || !output || output_size == 0) {
    return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid parameters: input=%p, input_size=%zu, output=%p, output_size=%zu",
                     input, input_size, output, output_size);
  }
 
  size_t ret = ZSTD_decompress(output, output_size, input, input_size);
 
  if (ZSTD_isError(ret)) {
    return SET_ERRNO(ERROR_GENERAL, "zstd decompression failed: %s", ZSTD_getErrorName(ret));
  }
 
  return ASCIICHAT_OK;
}

References ASCIICHAT_OK, ERROR_GENERAL, ERROR_INVALID_PARAM, and SET_ERRNO.

Referenced by handle_image_frame_packet(), packet_decode_frame_data_buffer(), and packet_decode_frame_data_malloc().

should_compress()

bool should_compress	(	size_t	original_size,
		size_t	compressed_size
	)

#include <compression.h>

Determine if compression should be used for given data sizes.

Parameters

original_size	Original (uncompressed) data size in bytes
compressed_size	Compressed data size in bytes

Returns

true if compression should be used, false otherwise

Determines whether compression is beneficial based on size thresholds and compression ratio. Returns true if:

• Original size >= COMPRESSION_MIN_SIZE (1KB minimum)
• Compressed size < COMPRESSION_RATIO_THRESHOLD * original_size (<80% of original)

This prevents:

• Compressing small packets that don't benefit
• Using compression when it doesn't reduce size meaningfully
• Wasting CPU on compression that expands data

Note

Call this function before compress_data() to determine if compression is worth the CPU overhead.

The function compares sizes only. Actual compression must be performed to get compressed_size for comparison.

Definition at line 74 of file compression.c.

                                                                   {
  if (original_size == 0)
    return false;
 
  float ratio = (float)compressed_size / (float)original_size;
  return ratio < COMPRESSION_RATIO_THRESHOLD;
}

References COMPRESSION_RATIO_THRESHOLD.

Referenced by send_packet_secure().