Protocol Handler

📡 Network packet processing and protocol implementation More...

Files
file	protocol.c
	📡 Server packet processor: client communication handling, protocol state management, and packet dispatch
file	protocol.h
	Server packet processing and protocol implementation.

Functions
void	handle_audio_opus_batch_packet (client_info_t *client, const void *data, size_t len)
	Process AUDIO_OPUS_BATCH packet - efficient Opus-encoded audio batch from client.
void	handle_audio_opus_packet (client_info_t *client, const void *data, size_t len)
	Process AUDIO_OPUS packet - decode single Opus frame from client.

Detailed Description

📡 Network packet processing and protocol implementation

Server Protocol README

Overview

The protocol handler module processes all incoming packets from clients, validates packet structure, updates client state, and stores media data for render threads. It serves as the communication bridge between clients and the server, translating network packets into actionable state changes and media data storage.

Implementation: src/server/protocol.c, src/server/protocol.h

Key Responsibilities:

• Parse and validate incoming packets from clients
• Update client state based on received packet data
• Store media data (video frames, audio samples) in client buffers
• Coordinate with other modules for media processing
• Generate appropriate server responses to client requests
• Maintain protocol compliance and packet format standards

Packet Processing Architecture

Processing Flow

Packet processing follows a three-stage pipeline:

Stage 1: Packet Reception (in client.c receive thread):

// Receive raw packet data from socket
int packet_len = recv_with_timeout(client->socket, buffer, sizeof(buffer), timeout);
 
// Validate packet header (magic number, CRC32, size)
if (validate_packet_header(buffer, packet_len) < 0) {
  log_warn("Invalid packet header from client %u", client_id);
  return -1;
}
 
// Decrypt packet if encryption enabled
if (client->crypto_initialized) {
  decrypt_packet(buffer, packet_len, &client->crypto_ctx);
}
 
// Dispatch to appropriate handler based on packet type
dispatch_packet_to_handler(client_id, buffer, packet_len);

Stage 2: Handler Function (this module):

// Validate packet payload structure
if (packet_len < sizeof(packet_type_t)) {
  log_warn("Packet too small from client %u", client_id);
  return -1;
}
 
// Parse packet type
packet_type_t type = get_packet_type(buffer);
 
// Route to handler function based on type
switch (type) {
  case PACKET_TYPE_CLIENT_JOIN:
    handle_client_join_packet(client_id, buffer, packet_len);
    break;
  case PACKET_TYPE_IMAGE_FRAME:
    handle_image_frame_packet(client_id, buffer, packet_len);
    break;
  // ... other packet types ...
}

Stage 3: Response Generation (via packet queues):

// Queue response packet for delivery
packet_queue_enqueue(client->video_packet_queue, response_packet, response_len);
 
// Send thread will deliver packet asynchronously
// (no blocking I/O in handler functions)

Supported Packet Types

Client Lifecycle Packets

PACKET_TYPE_CLIENT_JOIN:

• Initial client capabilities and identity
• Contains display name, terminal capabilities, client version
• Triggers client state initialization
• Generates server welcome response

PACKET_TYPE_CLIENT_LEAVE:

• Clean disconnect notification
• Triggers graceful client cleanup
• No response required

PACKET_TYPE_CLIENT_CAPABILITIES:

• Terminal capabilities and preferences update
• Contains color depth, palette preferences, UTF-8 support
• Updates client state for frame generation
• Triggers frame regeneration with new capabilities

Media Streaming Packets

PACKET_TYPE_STREAM_START:

• Signals client is ready to send audio/video
• Contains stream type flags (audio, video, both)
• Enables render threads for this client
• Triggers frame generation activation

PACKET_TYPE_STREAM_STOP:

• Signals client is stopping audio/video transmission
• Disables render threads for this client
• Clears media buffers

PACKET_TYPE_IMAGE_FRAME:

• Raw RGB video frame data
• Contains frame dimensions, timestamp, compressed data
• Stored in client's incoming video buffer
• Processed by video render thread

PACKET_TYPE_AUDIO_BATCH:

• Batched audio samples (efficient format)
• Contains sample count, sample rate, compressed audio data
• Stored in client's incoming audio ring buffer
• Processed by audio render thread

PACKET_TYPE_AUDIO (legacy):

• Single audio sample packet (deprecated)
• Replaced by PACKET_TYPE_AUDIO_BATCH
• Still supported for backwards compatibility

Control Protocol Packets

PACKET_TYPE_PING:

• Client keepalive request
• Generates PACKET_TYPE_PONG response
• Used for connection health monitoring

PACKET_TYPE_PONG:

• Server keepalive response
• Acknowledges client ping
• Confirms bidirectional connectivity

State Management

Client State Updates

All client state modifications use the snapshot pattern:

// Handler function updates client state
void handle_client_capabilities_packet(uint32_t client_id,
                                       const void *data, size_t len)
{
  // 1. Get client (with read lock)
  rwlock_rdlock(&g_client_manager_rwlock);
  client_info_t *client = NULL;
  HASH_FIND_INT(g_client_manager.clients_by_id,
                                            client_id);
 
  // 2. Acquire per-client mutex
  mutex_lock(&client->client_state_mutex);
 
  // 3. Update client state fields
  client->width = packet->width;
  client->height = packet->height;
  client->capabilities = packet->capabilities;
  client->palette = packet->palette;
 
  // 4. Release mutex immediately
  mutex_unlock(&client->client_state_mutex);
 
  // 5. Release read lock
  rwlock_rdunlock(&g_client_manager_rwlock);
 
  // 6. Process using local copies (no locks held)
  log_info("Client %u capabilities updated: %dx%d, palette=%s", client_id,
           packet->width, packet->height, palette_name(packet->palette));
}

State Fields:

• Terminal dimensions (width, height)
• Terminal capabilities (color depth, UTF-8 support, palette)
• Stream status (audio enabled, video enabled)
• Client identity (display name, client version)
• Connection metadata (connect time, last packet time)

Media Buffer Coordination

Video Frames:

• Stored in client->incoming_video_buffer (double-buffer system)
• Thread-safe buffer access via mutex
• Processed by video render thread at 60fps
• Frame dropping under load (keep only latest frame)

Audio Samples:

• Stored in client->incoming_audio_buffer (lock-free ring buffer)
• Thread-safe lock-free operations
• Processed by audio render thread at 172fps
• Automatic overflow handling (dropped samples logged)

Packet Validation Strategy

All handlers validate packets before processing:

void handle_image_frame_packet(uint32_t client_id,
                                const void *data, size_t len)
{
  // 1. Validate packet size
  if (len < sizeof(image_frame_packet_t)) {
    log_warn("Image frame packet too small from client %u: %zu < %zu",
             client_id, len, sizeof(image_frame_packet_t));
    return;
  }
 
  // 2. Validate packet structure
  const image_frame_packet_t *packet = (const image_frame_packet_t *)data;
  if (packet->width == 0 || packet->height == 0) {
    log_warn("Invalid frame dimensions from client %u: %ux%u",
             client_id, packet->width, packet->height);
    return;
  }
 
  // 3. Validate frame data size
  size_t expected_size = packet->width * packet->height * 3; // RGB
  if (packet->data_size > expected_size) {
    log_warn("Frame data size mismatch from client %u: %zu > %zu",
             client_id, packet->data_size, expected_size);
    return;
  }
 
  // 4. Validate client capabilities
  if (!client->video_enabled) {
    log_warn("Client %u sent video but video not enabled", client_id);
    return;
  }
 
  // 5. Process packet (all validations passed)
  store_video_frame(client_id, packet);
}

Validation Checks:

• Packet size matches expected structure size
• Packet type matches handler function
• Payload fields are within valid ranges
• Client capabilities permit the operation
• Buffer pointers are valid before access
• Network byte order conversion where needed

Error Handling Philosophy

Invalid Packets:

• Logged but don't disconnect clients
• Malformed packets are silently dropped
• Detailed error logging for troubleshooting
• Protocol compliance checking prevents crashes

Buffer Allocation Failures:

• Handled gracefully with error return
• Client continues receiving other packets
• Render threads handle missing frames gracefully
• No server crash from allocation failures

Network Errors During Responses:

• Don't affect client state
• Logged for debugging
• Send thread handles connection loss detection
• No blocking I/O in handler functions

Shutdown Conditions:

• Handlers check g_server_should_exit flag
• Early return from handlers during shutdown
• Avoids error spam during cleanup
• Clean thread exit without errors

Integration with Other Modules

Integration with client.c

Called By:

• Receive threads call protocol handler functions
• Receive thread receives packets and dispatches to handlers

Provides To:

• Client state update functions
• Media buffer storage functions
• Packet validation utilities

Integration with render.c

Consumed By:

• Render threads consume media data stored by handlers
• Video render thread reads from incoming_video_buffer
• Audio render thread reads from incoming_audio_buffer

Provides To:

• Video frame data for frame generation
• Audio sample data for audio mixing
• Client state for frame customization

Integration with stream.c

Used By:

• Stream generation uses client capabilities set by handlers
• Frame generation adapts to terminal dimensions
• Palette selection based on client preferences

Provides To:

• Client terminal capabilities
• Client rendering preferences
• Frame generation parameters

Performance Characteristics

Processing Speed:

• Packet validation is O(1) per packet
• State updates use minimal locking
• Media storage uses efficient buffer systems
• No blocking I/O in handler functions

Memory Usage:

• Packet buffers: Temporary allocations (freed after processing)
• Media buffers: Per-client fixed allocations
• State structures: Minimal overhead per client

Concurrency:

• Per-client handler isolation (no shared state)
• Thread-safe state updates (mutex protection)
• Lock-free media buffers where possible
• Minimal lock contention (snapshot pattern)

Best Practices

DO:

• Always validate packet size before accessing fields
• Use snapshot pattern for client state access
• Check client capabilities before operations
• Handle errors gracefully without disconnecting clients
• Use atomic operations for thread control flags

DON'T:

• Don't access packet fields without size validation
• Don't hold locks during media processing
• Don't perform blocking I/O in handler functions
• Don't skip error checking on state updates
• Don't ignore shutdown flags during processing

See also

src/server/protocol.c

src/server/protocol.h

Server Overview

Client Management

Render Threads

topic_packet_types

Function Documentation

handle_audio_opus_batch_packet()

void handle_audio_opus_batch_packet	(	client_info_t *	client,
		const void *	data,
		size_t	len
	)

#include <protocol.c>

Process AUDIO_OPUS_BATCH packet - efficient Opus-encoded audio batch from client.

Handles batched Opus-encoded audio frames sent by the client. This provides ~98% bandwidth reduction compared to raw PCM audio while maintaining excellent audio quality.

PACKET STRUCTURE EXPECTED:

• opus_batch_header_t (16 bytes):
  • sample_rate (4 bytes)
  • frame_duration (4 bytes)
  • frame_count (4 bytes)
  • reserved (4 bytes)
• Opus encoded data (variable size, typically ~60 bytes/frame @ 24kbps)

PROCESSING FLOW:

1. Parse batch header
2. Decode each Opus frame back to PCM samples (960 samples/frame @ 48kHz)
3. Write decoded samples to client's incoming audio buffer

PERFORMANCE:

• Bandwidth: ~60 bytes/frame vs ~3840 bytes/frame for raw PCM (64:1 compression)
• Quality: Excellent at 24 kbps VOIP mode
• Latency: 20ms frames for real-time audio

Parameters

client	Client that sent the audio packet
data	Packet payload data
len	Packet payload length

See also

av_receive_audio_opus_batch() For Packet Types parsing

opus_codec_decode() For Opus decoding

handle_audio_batch_packet() For raw PCM Audio batch handling

Definition at line 1156 of file server/protocol.c.

                                                                                         {
  log_debug_every(LOG_RATE_SLOW, "Received Opus audio batch from client %u (len=%zu)", atomic_load(&client->client_id),
                  len);
 
  VALIDATE_NOTNULL_DATA(client, data, "AUDIO_OPUS_BATCH");
  VALIDATE_AUDIO_STREAM_ENABLED(client, "AUDIO_OPUS_BATCH");
  VALIDATE_RESOURCE_INITIALIZED(client, client->opus_decoder, "Opus decoder");
 
  // Parse Opus batch packet
  const uint8_t *opus_data = NULL;
  size_t opus_size = 0;
  const uint16_t *frame_sizes = NULL;
  int sample_rate = 0;
  int frame_duration = 0;
  int frame_count = 0;
 
  asciichat_error_t result = packet_parse_opus_batch(data, len, &opus_data, &opus_size, &frame_sizes, &sample_rate,
                                                     &frame_duration, &frame_count);
 
  if (result != ASCIICHAT_OK) {
    disconnect_client_for_bad_data(client, "Failed to parse AUDIO_OPUS_BATCH packet");
    return;
  }
 
  VALIDATE_NONZERO(client, frame_count, "frame_count", "AUDIO_OPUS_BATCH");
  VALIDATE_NONZERO(client, opus_size, "opus_size", "AUDIO_OPUS_BATCH");
 
  // Calculate samples per frame (20ms @ 48kHz = 960 samples)
  int samples_per_frame = (sample_rate * frame_duration) / 1000;
  VALIDATE_RANGE(client, samples_per_frame, 1, 4096, "samples_per_frame", "AUDIO_OPUS_BATCH");
 
  // Use static buffer for common case to avoid malloc in hot path
  // Typical batches: 1-32 frames of 960 samples = up to 30,720 samples
  // Static buffer holds 32 frames @ 48kHz 20ms = 30,720 samples (120KB)
#define OPUS_DECODE_STATIC_MAX_SAMPLES (32 * 960)
  static float static_decode_buffer[OPUS_DECODE_STATIC_MAX_SAMPLES];
 
  size_t total_samples = (size_t)samples_per_frame * (size_t)frame_count;
  float *decoded_samples;
  bool used_malloc = false;
 
  if (total_samples <= OPUS_DECODE_STATIC_MAX_SAMPLES) {
    decoded_samples = static_decode_buffer;
  } else {
    // Unusual large batch - fall back to malloc
    log_warn("Client %u: Large audio batch requires malloc (%zu samples)", atomic_load(&client->client_id),
             total_samples);
    decoded_samples = SAFE_MALLOC(total_samples * sizeof(float), float *);
    if (!decoded_samples) {
      SET_ERRNO(ERROR_MEMORY, "Failed to allocate buffer for Opus decoded samples");
      return;
    }
    used_malloc = true;
  }
 
  // Decode each Opus frame using frame_sizes array
  int total_decoded = 0;
  size_t opus_offset = 0;
 
  for (int i = 0; i < frame_count; i++) {
    // Get exact frame size from frame_sizes array (convert from network byte order)
    size_t frame_size = (size_t)NET_TO_HOST_U16(frame_sizes[i]);
 
    // DEBUG: Log the actual bytes of each Opus frame
    if (frame_size > 0) {
      log_debug_every(LOG_RATE_DEFAULT, "Client %u: Opus frame %d: size=%zu, first_bytes=[0x%02x,0x%02x,0x%02x,0x%02x]",
                      atomic_load(&client->client_id), i, frame_size, opus_data[opus_offset] & 0xFF,
                      frame_size > 1 ? (opus_data[opus_offset + 1] & 0xFF) : 0,
                      frame_size > 2 ? (opus_data[opus_offset + 2] & 0xFF) : 0,
                      frame_size > 3 ? (opus_data[opus_offset + 3] & 0xFF) : 0);
    }
 
    if (opus_offset + frame_size > opus_size) {
      log_error("Client %u: Frame %d size overflow (offset=%zu, frame_size=%zu, total=%zu)",
                atomic_load(&client->client_id), i + 1, opus_offset, frame_size, opus_size);
      if (used_malloc) {
        SAFE_FREE(decoded_samples);
      }
      return;
    }
 
    // SECURITY: Bounds check before writing decoded samples to prevent buffer overflow
    // An attacker could send malicious Opus frames that decode to more samples than expected
    if ((size_t)total_decoded + (size_t)samples_per_frame > total_samples) {
      log_error("Client %u: Opus decode would overflow buffer (decoded=%d, frame_samples=%d, max=%zu)",
                atomic_load(&client->client_id), total_decoded, samples_per_frame, total_samples);
      if (used_malloc) {
        SAFE_FREE(decoded_samples);
      }
      return;
    }
 
    int decoded_count = opus_codec_decode((opus_codec_t *)client->opus_decoder, &opus_data[opus_offset], frame_size,
                                          &decoded_samples[total_decoded], samples_per_frame);
 
    if (decoded_count < 0) {
      log_error("Client %u: Opus decoding failed for frame %d/%d (size=%zu)", atomic_load(&client->client_id), i + 1,
                frame_count, frame_size);
      if (used_malloc) {
        SAFE_FREE(decoded_samples);
      }
      return;
    }
 
    total_decoded += decoded_count;
    opus_offset += frame_size;
  }
 
  log_debug_every(LOG_RATE_DEFAULT, "Client %u: Decoded %d Opus frames -> %d samples", atomic_load(&client->client_id),
                  frame_count, total_decoded);
 
  // DEBUG: Log sample values to detect all-zero issue
  static int server_decode_count = 0;
  server_decode_count++;
  if (total_decoded > 0 && (server_decode_count <= 10 || server_decode_count % 100 == 0)) {
    float peak = 0.0f, rms = 0.0f;
    for (int i = 0; i < total_decoded && i < 100; i++) {
      float abs_val = fabsf(decoded_samples[i]);
      if (abs_val > peak)
        peak = abs_val;
      rms += decoded_samples[i] * decoded_samples[i];
    }
    rms = sqrtf(rms / (total_decoded > 100 ? 100 : total_decoded));
    // Log first 4 bytes of Opus data to compare with client encode
    log_info("SERVER OPUS DECODE #%d from client %u: decoded_rms=%.6f, opus_first4=[0x%02x,0x%02x,0x%02x,0x%02x]",
             server_decode_count, atomic_load(&client->client_id), rms, opus_size > 0 ? opus_data[0] : 0,
             opus_size > 1 ? opus_data[1] : 0, opus_size > 2 ? opus_data[2] : 0, opus_size > 3 ? opus_data[3] : 0);
  }
 
  // Write decoded samples to client's incoming audio buffer
  // Note: audio_ring_buffer_write returns error code, not sample count
  // Buffer overflow warnings are logged inside audio_ring_buffer_write if buffer is full
  if (client->incoming_audio_buffer && total_decoded > 0) {
    asciichat_error_t result = audio_ring_buffer_write(client->incoming_audio_buffer, decoded_samples, total_decoded);
    if (result != ASCIICHAT_OK) {
      log_error("Client %u: Failed to write decoded audio to buffer: %d", atomic_load(&client->client_id), result);
    }
  }
 
  if (used_malloc) {
    SAFE_FREE(decoded_samples);
  }
}

References ASCIICHAT_OK, audio_ring_buffer_write(), client_info::client_id, disconnect_client_for_bad_data(), ERROR_MEMORY, client_info::incoming_audio_buffer, log_debug_every, log_error, log_info, LOG_RATE_DEFAULT, LOG_RATE_SLOW, log_warn, NET_TO_HOST_U16, opus_codec_decode(), OPUS_DECODE_STATIC_MAX_SAMPLES, client_info::opus_decoder, packet_parse_opus_batch(), SAFE_FREE, SAFE_MALLOC, SET_ERRNO, VALIDATE_AUDIO_STREAM_ENABLED, VALIDATE_NONZERO, VALIDATE_NOTNULL_DATA, VALIDATE_RANGE, and VALIDATE_RESOURCE_INITIALIZED.

Referenced by process_decrypted_packet().

handle_audio_opus_packet()

void handle_audio_opus_packet	(	client_info_t *	client,
		const void *	data,
		size_t	len
	)

#include <protocol.c>

Process AUDIO_OPUS packet - decode single Opus frame from client.

Handles single Opus-encoded audio frames sent by clients. The packet format includes a 16-byte header followed by the Opus-encoded data.

PACKET STRUCTURE:

• Offset 0: sample_rate (uint32_t, native byte order - bug in client)
• Offset 4: frame_duration (uint32_t, native byte order - bug in client)
• Offset 8: reserved (8 bytes)
• Offset 16: Opus-encoded audio data

Parameters

client	Client info structure
data	Packet payload
len	Packet length

Definition at line 1318 of file server/protocol.c.

                                                                                   {
  log_debug_every(LOG_RATE_DEFAULT, "Received Opus audio from client %u (len=%zu)", atomic_load(&client->client_id),
                  len);
 
  if (VALIDATE_PACKET_NOT_NULL(client, data, "AUDIO_OPUS")) {
    return;
  }
 
  // Minimum size: 16-byte header + at least 1 byte of Opus data
  if (len < 17) {
    disconnect_client_for_bad_data(client, "AUDIO_OPUS packet too small: %zu bytes", len);
    return;
  }
 
  if (!atomic_load(&client->is_sending_audio)) {
    disconnect_client_for_bad_data(client, "AUDIO_OPUS received before audio stream enabled");
    return;
  }
 
  if (!client->opus_decoder) {
    disconnect_client_for_bad_data(client, "Opus decoder not initialized");
    return;
  }
 
  // Parse header (16 bytes) - convert from network byte order
  const uint8_t *buf = (const uint8_t *)data;
  uint32_t sample_rate_net, frame_duration_net;
  memcpy(&sample_rate_net, buf, 4);
  memcpy(&frame_duration_net, buf + 4, 4);
  uint32_t sample_rate = NET_TO_HOST_U32(sample_rate_net);
  uint32_t frame_duration = NET_TO_HOST_U32(frame_duration_net);
 
  // Extract Opus data (after 16-byte header)
  const uint8_t *opus_data = buf + 16;
  size_t opus_size = len - 16;
 
  // Validate parameters
  if (sample_rate == 0 || sample_rate > 192000) {
    disconnect_client_for_bad_data(client, "AUDIO_OPUS invalid sample_rate: %u", sample_rate);
    return;
  }
 
  if (frame_duration == 0 || frame_duration > 120) {
    disconnect_client_for_bad_data(client, "AUDIO_OPUS invalid frame_duration: %u ms", frame_duration);
    return;
  }
 
  // Calculate expected samples per frame
  int samples_per_frame = (int)((sample_rate * frame_duration) / 1000);
  if (samples_per_frame <= 0 || samples_per_frame > 5760) { // Max 120ms @ 48kHz
    disconnect_client_for_bad_data(client, "AUDIO_OPUS invalid samples_per_frame: %d", samples_per_frame);
    return;
  }
 
  // Decode Opus frame
  float decoded_samples[5760]; // Max Opus frame size (120ms @ 48kHz)
  int decoded_count =
      opus_codec_decode((opus_codec_t *)client->opus_decoder, opus_data, opus_size, decoded_samples, samples_per_frame);
 
  if (decoded_count < 0) {
    log_error("Client %u: Opus decoding failed (size=%zu)", atomic_load(&client->client_id), opus_size);
    return;
  }
 
  log_debug_every(LOG_RATE_VERY_FAST, "Client %u: Decoded Opus frame -> %d samples", atomic_load(&client->client_id),
                  decoded_count);
 
  // Write decoded samples to client's incoming audio buffer
  if (client->incoming_audio_buffer && decoded_count > 0) {
    asciichat_error_t write_result =
        audio_ring_buffer_write(client->incoming_audio_buffer, decoded_samples, decoded_count);
    if (write_result != ASCIICHAT_OK) {
      log_error("Failed to write decoded Opus samples to buffer: %s", asciichat_error_string(write_result));
    }
  }
}

References ASCIICHAT_OK, audio_ring_buffer_write(), client_info::client_id, disconnect_client_for_bad_data(), client_info::incoming_audio_buffer, client_info::is_sending_audio, log_debug_every, log_error, LOG_RATE_DEFAULT, LOG_RATE_VERY_FAST, NET_TO_HOST_U32, opus_codec_decode(), client_info::opus_decoder, and VALIDATE_PACKET_NOT_NULL.