ascii-chat
Binary v0.16.50-198-g78f51665f
Library lib/v0.3.2-2240-g78f51665f
Video chat in your terminal
Loading...
Searching...
No Matches
Compression Module

📦 Network compression utilities for ascii-chat More...

Files

file  compression.c
 đŸ—œď¸ Fast zstd compression/decompression utilities for network payload optimization
file  compression.h
 đŸ“Ś Network Packet Compression Utilities

Functions

asciichat_error_t compress_data (const void *input, size_t input_size, void **output, size_t *output_size, int compression_level)
 Compress data using zstd with configurable compression level.
asciichat_error_t decompress_data (const void *input, size_t input_size, void *output, size_t output_size)
 Decompress data using zstd.
asciichat_error_t decompress_data_exact (const void *input, size_t input_size, void *output, size_t expected_size)
bool should_compress (size_t original_size, size_t compressed_size)
 Determine if compression should be used for given data sizes.

Compression Settings

#define COMPRESSION_RATIO_THRESHOLD   0.8f
 Compression ratio threshold - only use if <80% original size.
#define COMPRESSION_MIN_SIZE   1024
 Minimum packet size to attempt compression (1KB).

Detailed Description

📦 Network compression utilities for ascii-chat

This header provides compression and decompression utilities for network packets in ascii-chat. The system uses zstd compression to reduce bandwidth usage for large packets like video frames.

CORE FEATURES:

  • zstd compression algorithm
  • Automatic compression ratio checking (only use if beneficial)
  • Minimum size threshold to avoid compressing small packets
  • Memory-efficient compression/decompression
  • Pure utility functions (no state management)

COMPRESSION STRATEGY:

Compression is only applied when:

  • Packet size exceeds minimum threshold (1KB)
  • Compression achieves at least 20% size reduction (<80% of original)

This prevents:

  • Overhead of compressing already-compressed data
  • CPU waste on small packets that don't benefit
  • Compression expansion (when compressed data is larger)

ALGORITHM:

  • Uses Facebook's zstd deflate for compression
  • Provides good compression ratio for text/ASCII data
  • Reasonable CPU overhead for real-time streaming
Note
Compression is optional and only used when beneficial.
Frame sending functions have been moved to network.h/network.c.
The compression ratio threshold ensures compression is only used when it provides meaningful bandwidth savings.
Author
Zachary Fogg me@zf.nosp@m.o.gg
Date
August 2025

Compression README

Overview

Welcome to the Compression module! This is where we shrink your data down to save bandwidth. Compression is all about balance—we want to save bandwidth without wasting CPU, so we compress intelligently only when it actually helps.

What does the compression module do?

The compression module provides fast zstd compression/decompression utilities for optimizing network payload size in ascii-chat. Compression reduces bandwidth usage for large packets like video frames and audio batches, which improves performance over low-bandwidth connections.

Here's what it gives you:

  • zstd compression algorithm (level 1 for speed/ratio balance)—we use zstd because it's fast and gives good compression ratios
  • Automatic compression ratio checking (only compresses if beneficial)—we don't waste CPU compressing data that doesn't benefit
  • Minimum size threshold to avoid overhead on small packets—we don't bother compressing tiny packets where compression overhead isn't worth it
  • Memory-efficient compression/decompression**—we don't use a lot of memory for compression
  • **Pure utility functions (no state management)—simple functions that you can call from anywhere

Implementation: lib/compression.c, lib/compression.h

Compression Strategy

Compression isn't always worth it—sometimes the compressed data is actually larger than the original, or the CPU cost isn't worth the bandwidth savings. We've designed the compression system to be smart about when to compress.

When do we compress?

Compression is automatically applied when:

  • Packet size exceeds minimum threshold (1KB)—we don't compress tiny packets because the overhead isn't worth it
  • Compression achieves at least 20% size reduction (<80% of original size)— if compression doesn't save at least 20%, we skip it

What problems does this prevent?

This strategy prevents several common problems:

  • Overhead of compressing already-compressed data: If something is already compressed (like a JPEG image), trying to compress it again usually makes it bigger—we detect this and skip compression
  • CPU waste on small packets: Compressing tiny packets often takes more CPU time than the bandwidth savings are worth—we don't bother with small packets
  • Compression expansion: Sometimes compressed data is actually larger than the original (this is called "expansion")—we check for this and use the original data if compression expanded it

Algorithm

We use zstd (Zstandard) compression, which is a modern compression algorithm developed by Facebook/Meta. Here's why we chose it:

Why zstd?

  • Uses zstd level 1 for optimal speed/ratio balance—level 1 is fast enough for real-time streaming but still gives good compression
  • **Compatible with standard zstd decompression**—any zstd-compatible tool can decompress our data
  • **Provides good compression ratio for text/ASCII data**—our ASCII frames compress really well (often 70-90% compression!)
  • **Reasonable CPU overhead for real-time streaming**—we can compress at 200-500 MB/s, which is fast enough for real-time video

zstd level 1 is perfect for our use case—it's fast enough that it doesn't slow things down, but still gives us significant bandwidth savings.

API Usage

Basic Compression

void *compressed_data;
size_t compressed_size;
// Compress data (allocates output buffer)
if (compress_data(input_data, input_size, &compressed_data, &compressed_size) == 0) {
// Check if compression is worthwhile (optional)
if (should_compress(input_size, compressed_size)) {
// Send compressed data
send_packet(compressed_data, compressed_size);
} else {
// Send original data (compression didn't help)
send_packet(input_data, input_size);
}
SAFE_FREE(compressed_data);
}
#define SAFE_FREE(ptr)
bool should_compress(size_t original_size, size_t compressed_size)
Determine if compression should be used for given data sizes.
asciichat_error_t compress_data(const void *input, size_t input_size, void **output, size_t *output_size, int compression_level)
Compress data using zstd with configurable compression level.
Definition compression.c:29
int send_packet(socket_t sockfd, packet_type_t type, const void *data, size_t len)
Send a basic packet without encryption.
Definition packet.c:581

Decompression

void *decompressed_data;
size_t decompressed_size;
// Allocate output buffer (must know original size)
decompressed_data = SAFE_MALLOC(original_size, void *);
// Decompress data
if (decompress_data(compressed_data, compressed_size,
decompressed_data, original_size) == 0) {
// Use decompressed data
process_data(decompressed_data, original_size);
}
SAFE_FREE(decompressed_data);
#define SAFE_MALLOC(size, cast)
asciichat_error_t decompress_data(const void *input, size_t input_size, void *output, size_t output_size)
Decompress data using zstd.
Definition compression.c:79

Network Integration

Compression is automatically integrated into the packet protocol:

// In packet send path
if (data_size >= COMPRESSION_MIN_SIZE) {
void *compressed;
size_t compressed_size;
if (compress_data(data, data_size, &compressed, &compressed_size) == 0) {
if (should_compress(data_size, compressed_size)) {
send_packet_header(type, compressed_size, compressed_flags);
send_packet_data(compressed, compressed_size);
free(compressed);
return;
}
free(compressed);
}
}
// Fall back to uncompressed
send_packet_header(type, data_size, flags);
send_packet_data(data, data_size);
#define COMPRESSION_MIN_SIZE
Minimum packet size to attempt compression (1KB).
Definition compression.h:66

Performance Characteristics

Compression performance is all about balancing speed and compression ratio. We've tuned the system for real-time video streaming, so speed is critical, but we still want good compression. Let's look at what you can expect:

How fast is compression?

Compression Speed:

  • zstd level 1: ~200-500 MB/s compression (CPU-dependent)—this is fast enough that compression doesn't become a bottleneck
  • Suitable for real-time video streaming (60fps)—we can compress video frames at 60 FPS without slowing things down
  • **Low CPU overhead compared to higher compression levels**—level 1 is much faster than higher levels (level 3+), but still gives good compression

The speed varies depending on your CPU, but even on a modest CPU, you can compress at 200 MB/s, which is more than enough for real-time video.

How much bandwidth do we save?

Compression Ratio:

  • ASCII text: 70-90% compression (significant bandwidth savings!)—this is the big win for ASCII frames
  • Binary data: 60-80% compression—binary data compresses less well, but we still save bandwidth
  • Already-compressed: Often expands (detected and avoided)—if something is already compressed (like a JPEG), trying to compress it again usually makes it bigger, so we detect this and skip compression

ASCII frames compress really well because they're text—all those repeated spaces and characters compress down nicely. You can often get 70-90% compression, which means a 6 MB frame becomes 600 KB-1.8 MB. That's a huge bandwidth savings!

How much memory does it use?

Memory Usage:

  • Output buffer allocated by compress_data() (must be freed by caller)— we allocate the compressed output buffer, but you're responsible for freeing it
  • Input buffer must be provided by caller for decompress_data()**—you provide the decompressed output buffer
  • **No persistent state (pure functions)—no global state, no initialization needed, just call the functions whenever you need them

Memory usage is pretty reasonable—we allocate the compressed output buffer (which is usually smaller than the input), but there's no persistent state, so memory usage is minimal.

Configuration

Compression settings control when and how we compress. We've tuned these values for real-time video streaming, but let's understand what they mean:

What thresholds do we use?

Compression Thresholds:

  • COMPRESSION_MIN_SIZE (1024 bytes): Minimum packet size to attempt compression—we don't compress packets smaller than 1KB because the overhead isn't worth it
  • COMPRESSION_RATIO_THRESHOLD (0.8): Maximum acceptable compression ratio (80% of original)—if compression doesn't reduce size by at least 20%, we skip it

These thresholds ensure we only compress when it's actually beneficial. Small packets and packets that don't compress well are skipped, saving CPU without wasting bandwidth.

What compression level do we use?

Compression Level:

  • Fixed at zstd level 1 for optimal speed/ratio balance—level 1 is the sweet spot for real-time streaming
  • **Higher levels provide better compression but slower performance**—you could get better compression with level 3 or 5, but it would be much slower
  • **Level 1 is ideal for real-time streaming applications**—fast enough for 60 FPS, but still gives good compression

We could use higher compression levels (which would save more bandwidth), but the CPU cost would be too high for real-time streaming. Level 1 is the perfect balance.

Error Handling

Compression Errors:

  • Returns negative error code on failure
  • Sets asciichat_errno for detailed error context
  • Output buffer not allocated on failure (safe to ignore)

Decompression Errors:

  • Returns negative error code on failure
  • Sets asciichat_errno for detailed error context
  • Output buffer unchanged on failure

Limitations

  • zstd dependency: Requires zstd library at compile time
  • Memory allocation: Compression allocates output buffer (must be freed)
  • Size requirement: Must know original size for decompression
  • Compression expansion: May expand small or already-compressed data
See also
lib/compression.h
lib/compression.c
topic_network
topic_av

Macro Definition Documentation

◆ COMPRESSION_MIN_SIZE

#define COMPRESSION_MIN_SIZE   1024

#include <include/ascii-chat/network/compression.h>

Minimum packet size to attempt compression (1KB).

Definition at line 66 of file compression.h.

Referenced by acip_send_ascii_frame_contract(), and send_packet_secure().

◆ COMPRESSION_RATIO_THRESHOLD

#define COMPRESSION_RATIO_THRESHOLD   0.8f

#include <include/ascii-chat/network/compression.h>

Compression ratio threshold - only use if <80% original size.

Definition at line 63 of file compression.h.

Referenced by send_packet_secure(), and should_compress().

Function Documentation

◆ compress_data()

asciichat_error_t compress_data ( const void * input,
size_t input_size,
void ** output,
size_t * output_size,
int compression_level )

#include <include/ascii-chat/network/compression.h>

Compress data using zstd with configurable compression level.

Parameters
inputInput data to compress (must not be NULL)
input_sizeSize of input data in bytes
outputOutput buffer pointer (pointer-to-pointer; function allocates and stores address here)
output_sizeSize of compressed data in bytes (output parameter, must not be NULL)
compression_levelzstd compression level (1-9)
  • Level 1: Fastest compression, lowest ratio (best for real-time streaming)
  • Level 3: Balanced speed/ratio
  • Level 9: Slower compression, best ratio (for limited bandwidth)
Returns
ASCIICHAT_OK on success, error code on failure

Compresses input data using zstd's compression algorithm with the specified compression level. The output buffer is automatically allocated by the function and must be freed by the caller using buffer_release().

Note
The output buffer is allocated using buffer_alloc(). Caller must release it with buffer_release() when done using the compressed data.
Compression may fail if input data is already compressed or if compression would expand the data significantly.
For best performance, use should_compress() first to determine if compression is beneficial before calling this function.
Compression level must be between 1 and 9. Levels above 9 are not recommended for real-time streaming due to CPU overhead.
Warning
Caller must buffer_release() the output buffer to avoid memory leaks.

Definition at line 29 of file compression.c.

29 {
30 ASCII_OTEL_SCOPE(ZSTD_COMPRESS);
31 ASCII_OTEL_ATTR_STRING("compression.algorithm", "zstd");
32 ASCII_OTEL_ATTR_STRING("compression.library.version", ZSTD_versionString());
33 ASCII_OTEL_ATTR_INT("compression.level", compression_level);
34 ASCII_OTEL_ATTR_INT("compression.input.size", (int64_t)input_size);
35 if (!input || input_size == 0 || !output || !output_size) {
36 return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid parameters: input=%p, input_size=%zu, output=%p, output_size=%p", input, input_size, output, output_size);
37 }
38
39 // Validate compression level (1-9 for real-time streaming)
40 if (compression_level < 1 || compression_level > 9) {
41 return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid compression level %d: must be between 1 and 9", compression_level);
42 }
43
44 // Calculate maximum compressed size
45 size_t compressed_size = ZSTD_compressBound(input_size);
46
47 // Sanity check ZSTD_compressBound result
48 // ZSTD_compressBound returns 0 for errors or very large values for huge inputs
49 if (compressed_size == 0 || compressed_size > 256 * 1024 * 1024) { // Max 256MB compressed buffer
50 return SET_ERRNO(ERROR_INVALID_PARAM, "ZSTD_compressBound returned unreasonable size: %zu for input %zu", compressed_size, input_size);
51 }
52
53 unsigned char *compressed_data = NULL;
54 compressed_data = buffer_alloc(NULL, compressed_size);
55
56 if (!compressed_data) {
57 return SET_ERRNO(ERROR_MEMORY, "Failed to allocate compressed data buffer");
58 }
59 defer_named(release_compressed, buffer_release(compressed_data));
60
61 // Compress using zstd with user-specified compression level
62 size_t ret = ZSTD_compress(compressed_data, compressed_size, input, input_size, compression_level);
63
64 if (ZSTD_isError(ret)) {
65 return SET_ERRNO(ERROR_GENERAL, "zstd compression failed: %s", ZSTD_getErrorName(ret));
66 }
67
68 *output = compressed_data;
69 defer_cancel(release_compressed);
70 *output_size = ret;
71 ASCII_OTEL_ATTR_INT("compression.output.size", (int64_t)ret);
72 ASCII_OTEL_ATTR_INT("compression.ratio.permille", (int64_t)(1000.0 * (double)ret / (double)input_size));
73 compression_measure(input_size, ret, true);
74
75 return ASCIICHAT_OK;
76}
#define SET_ERRNO(code,...)
void * buffer_alloc(buffer_domain_t *domain, size_t size)
Definition buffer.c:272
#define buffer_release(data)
Definition buffer.h:99
@ ERROR_MEMORY
Definition error_codes.h:55
@ ASCIICHAT_OK
Definition error_codes.h:50
@ ERROR_GENERAL
Definition error_codes.h:51
@ ERROR_INVALID_PARAM
#define ASCII_OTEL_SCOPE(id)
Definition otel.h:77
#define ASCII_OTEL_ATTR_INT(key, value)
Definition otel.h:72
#define ASCII_OTEL_ATTR_STRING(key, value)
Definition otel.h:71

References ASCII_OTEL_ATTR_INT, ASCII_OTEL_ATTR_STRING, ASCII_OTEL_SCOPE, ASCIICHAT_OK, buffer_alloc(), buffer_release, ERROR_GENERAL, ERROR_INVALID_PARAM, ERROR_MEMORY, and SET_ERRNO.

Referenced by send_packet_secure().

◆ decompress_data()

asciichat_error_t decompress_data ( const void * input,
size_t input_size,
void * output,
size_t output_size )

#include <include/ascii-chat/network/compression.h>

Decompress data using zstd.

Parameters
inputCompressed input data (must not be NULL)
input_sizeSize of compressed data in bytes
outputPre-allocated output buffer (must not be NULL)
output_sizeSize of output buffer in bytes (must be >= decompressed size)
Returns
ASCIICHAT_OK on success, error code on failure

Decompresses zstd-compressed data into a pre-allocated output buffer. The output buffer must be large enough to hold the decompressed data.

Note
The output buffer size must be known in advance (typically from packet header or protocol specification). This function does not dynamically allocate the output buffer.
This function uses zstd inflate algorithm, compatible with standard zstd compression.
Warning
Output buffer must be large enough for decompressed data or buffer overflow will occur. Ensure output_size is correct before calling.

Definition at line 79 of file compression.c.

79 {
80 ASCII_OTEL_SCOPE(ZSTD_DECOMPRESS);
81 ASCII_OTEL_ATTR_STRING("compression.algorithm", "zstd");
82 ASCII_OTEL_ATTR_STRING("compression.library.version", ZSTD_versionString());
83 ASCII_OTEL_ATTR_INT("compression.input.size", (int64_t)input_size);
84 ASCII_OTEL_ATTR_INT("compression.output.capacity", (int64_t)output_size);
85 if (!input || input_size == 0 || !output || output_size == 0) {
86 return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid parameters: input=%p, input_size=%zu, output=%p, output_size=%zu", input, input_size, output, output_size);
87 }
88
89 size_t ret = ZSTD_decompress(output, output_size, input, input_size);
90
91 if (ZSTD_isError(ret)) {
92 return SET_ERRNO(ERROR_GENERAL, "zstd decompression failed: %s", ZSTD_getErrorName(ret));
93 }
94
95 ASCII_OTEL_ATTR_INT("compression.output.size", (int64_t)ret);
96 ASCII_OTEL_ATTR_INT("compression.ratio.permille", (int64_t)(1000.0 * (double)ret / (double)input_size));
97 compression_measure(input_size, ret, false);
98 return ASCIICHAT_OK;
99}

References ASCII_OTEL_ATTR_INT, ASCII_OTEL_ATTR_STRING, ASCII_OTEL_SCOPE, ASCIICHAT_OK, ERROR_GENERAL, ERROR_INVALID_PARAM, and SET_ERRNO.

◆ decompress_data_exact()

asciichat_error_t decompress_data_exact ( const void * input,
size_t input_size,
void * output,
size_t expected_size )

#include <include/ascii-chat/network/compression.h>

Decode an exact-size payload, rejecting short output as well as overflow.

Definition at line 103 of file compression.c.

103 {
104 ASCII_OTEL_SCOPE(ZSTD_DECOMPRESS_EXACT);
105 ASCII_OTEL_ATTR_STRING("compression.algorithm", "zstd");
106 ASCII_OTEL_ATTR_STRING("compression.library.version", ZSTD_versionString());
107 ASCII_OTEL_ATTR_INT("compression.input.size", (int64_t)input_size);
108 ASCII_OTEL_ATTR_INT("compression.expected_output.size", (int64_t)expected_size);
109 if (!input || !input_size || !output || !expected_size)
110 return SET_ERRNO(ERROR_INVALID_PARAM, "Invalid exact decompression bounds");
111 size_t decoded = ZSTD_decompress(output, expected_size, input, input_size);
112 if (ZSTD_isError(decoded) || decoded != expected_size)
113 return SET_ERRNO(ERROR_COMPRESSION, "Decompressed payload does not match expected size %zu", expected_size);
114 ASCII_OTEL_ATTR_INT("compression.output.size", (int64_t)decoded);
115 compression_measure(input_size, decoded, false);
116 return ASCIICHAT_OK;
117}
@ ERROR_COMPRESSION

References ASCII_OTEL_ATTR_INT, ASCII_OTEL_ATTR_STRING, ASCII_OTEL_SCOPE, ASCIICHAT_OK, ERROR_COMPRESSION, ERROR_INVALID_PARAM, and SET_ERRNO.

Referenced by browser_client_decode_frame_data(), client_decode_frame_data(), and packet_decode_frame_data_buffer().

◆ should_compress()

bool should_compress ( size_t original_size,
size_t compressed_size )

#include <include/ascii-chat/network/compression.h>

Determine if compression should be used for given data sizes.

Parameters
original_sizeOriginal (uncompressed) data size in bytes
compressed_sizeCompressed data size in bytes
Returns
true if compression should be used, false otherwise

Determines whether compression is beneficial based on size thresholds and compression ratio. Returns true if:

  • Original size >= COMPRESSION_MIN_SIZE (1KB minimum)
  • Compressed size < COMPRESSION_RATIO_THRESHOLD * original_size (<80% of original)

This prevents:

  • Compressing small packets that don't benefit
  • Using compression when it doesn't reduce size meaningfully
  • Wasting CPU on compression that expands data
Note
Call this function before compress_data() to determine if compression is worth the CPU overhead.
The function compares sizes only. Actual compression must be performed to get compressed_size for comparison.

Definition at line 120 of file compression.c.

120 {
121 if (original_size == 0)
122 return false;
123
124 float ratio = (float)compressed_size / (float)original_size;
125 return ratio < COMPRESSION_RATIO_THRESHOLD;
126}
#define COMPRESSION_RATIO_THRESHOLD
Compression ratio threshold - only use if <80% original size.
Definition compression.h:63

References COMPRESSION_RATIO_THRESHOLD.

Referenced by acip_send_ascii_frame_contract(), and send_packet_secure().