stream.c File Reference

🎬 Multi-client video mixer: frame generation, ASCII conversion, and per-client personalized rendering More...

Go to the source code of this file.

Data Structures
struct	image_source_t
	Image source structure for multi-client video mixing. More...

Functions
char *	create_mixed_ascii_frame_for_client (uint32_t target_client_id, unsigned short width, unsigned short height, bool wants_stretch, size_t *out_size, bool *out_grid_changed, int *out_sources_count)
	Generate personalized ASCII frame for a specific client.
int	queue_audio_for_client (client_info_t *client, const void *audio_data, size_t data_size)
	Queue ASCII frame for delivery to specific client.
bool	any_clients_sending_video (void)
	Check if any connected clients are currently sending video.

Detailed Description

🎬 Multi-client video mixer: frame generation, ASCII conversion, and per-client personalized rendering

1. Collect video frames from all active clientsclientsCreate composite video layouts (single client, 2x2, 3x3 grids))Generate client-specific ASCII art with terminal capability awarenessawarenessProcess latest frames from double-buffer system for real-time performanceperformanceManage memory efficiently with buffer pools and zero-copy operationsoperationsSupport advanced rendering modes (half-block, color, custom palettes) VIDEO MIXING ARCHITECTURE:

The mixing system operates in several stages:

1. FRAME COLLECTION:
   • Scans all active clients for available video frames
   • Uses per-client double-buffer system for smooth frame handling
   • Implements aggressive frame dropping to maintain real-time performance
   • Always uses the latest available frame for professional-grade performance
2. LAYOUT CALCULATION:
   • Determines grid size based on number of active video sources
   • Supports: single (1x1), side-by-side (2x1), 2x2, 3x2, 3x3 layouts
   • Calculates aspect-ratio preserving dimensions for each cell
3. COMPOSITE GENERATION:
   • Creates composite image with proper aspect ratio handling
   • Places each client's video in appropriate grid cell
   • Supports both normal and half-block rendering modes
4. ASCII CONVERSION:
   • Converts composite to ASCII using client-specific capabilities
   • Respects terminal color support, palette preferences, UTF-8 capability
   • Generates ANSI escape sequences for color output
5. PACKET GENERATION:
   • Wraps ASCII frames in protocol packets with metadata
   • Includes checksums, dimensions, and capability flags
   • Queues packets for delivery via client send threads

PER-CLIENT CUSTOMIZATION:

Unlike traditional video mixing that generates one output, this system creates personalized frames for each client:

TERMINAL CAPABILITY AWARENESS:

• Color depth: 1-bit (mono), 8-color, 16-color, 256-color, 24-bit RGB
• Character support: ASCII-only vs UTF-8 box drawing
• Render modes: foreground, background, half-block (2x resolution)
• Custom ASCII palettes: brightness-to-character mapping

PERFORMANCE OPTIMIZATIONS:

• Per-client palette caches (avoid repeated initialization)
• Double-buffer system for smooth frame delivery
• Buffer pool allocation to reduce malloc/free overhead
• Aggressive frame dropping under load
• Zero-copy operations where possible

THREADING AND CONCURRENCY:

This module operates in a high-concurrency environment:

THREAD SAFETY MECHANISMS:

• Double-buffer thread safety (atomic operations)
• Reader-writer locks on client manager (allows concurrent reads)
• Buffer pool thread safety (lock-free where possible)
• Atomic snapshot operations for client state

PERFORMANCE CHARACTERISTICS:

• Supports 60fps per client with linear scaling
• Handles burst traffic with frame buffer overruns
• Minimal CPU overhead with double-buffer system
• Memory usage scales with number of active clients

FRAME BUFFER MANAGEMENT:

DOUBLE-BUFFER STRATEGY: Each client uses a double-buffer system for smooth frame delivery:

• Always provides the latest available frame
• No frame caching complexity or stale data concerns
• Professional-grade real-time performance like Zoom/Google Meet
• Memory managed via buffer pools

BUFFER OVERFLOW HANDLING: When clients send frames faster than processing:

• Aggressive frame dropping (keep only latest)
• Logarithmic drop rate based on buffer occupancy
• Maintains real-time performance under load

INTEGRATION WITH OTHER MODULES:

• render.c: Calls frame generation functions at 60fps per client
• client.c: Provides client state and capabilities
• protocol.c: Stores incoming video frames in per-client buffers
• video/: Performs actual RGB-to-ASCII conversion
• palette.c: Manages client-specific character palettes

WHY THIS MODULAR DESIGN:

The original server.c contained all video mixing logic inline, making it:

• Impossible to understand the video pipeline
• Difficult to optimize performance bottlenecks
• Hard to add new rendering features
• Challenging to debug frame generation issues

This separation provides:

• Clear video processing pipeline
• Isolated performance optimization
• Easier feature development
• Better error isolation and debugging

MEMORY MANAGEMENT PHILOSOPHY:

• All allocations use buffer pools where possible
• Frame data is reference-counted (avoid copies)
• Double-buffer system eliminates need for frame copies
• Automatic cleanup on client disconnect
• Graceful degradation on allocation failures

Author

Zachary Fogg me@zf.nosp@m.o.gg

Date

September 2025

Version

2.0 (Post-Modularization)

See also

render.c For frame generation scheduling

video/ For RGB-to-ASCII conversion implementation

palette.c For client Character Palettes management

Definition in file stream.c.

Function Documentation

any_clients_sending_video()

bool any_clients_sending_video

(

void

)

Check if any connected clients are currently sending video.

This function scans all active clients to determine if at least one is sending video frames. Used by render threads to avoid generating frames when no video sources are available (e.g., during webcam warmup).

LOCK OPTIMIZATION: Uses atomic reads only, no rwlock acquisition This is safe because client_id, active, and is_sending_video are all atomics

Returns

true if at least one client has is_sending_video flag set, false otherwise

Definition at line 1305 of file stream.c.

                                     {
  // LOCK OPTIMIZATION: Don't acquire rwlock - all fields we access are atomic
  // client_id, active, is_sending_video are all atomic variables
 
  // Iterate through all client slots
  for (int i = 0; i < MAX_CLIENTS; i++) {
    client_info_t *client = &g_client_manager.clients[i];
 
    // Skip uninitialized clients (atomic read)
    if (atomic_load(&client->client_id) == 0) {
      continue;
    }
 
    // Check if client is active and sending video (both atomic reads)
    bool is_active = atomic_load(&client->active);
    bool is_sending = atomic_load(&client->is_sending_video);
 
    if (is_active && is_sending) {
      return true;
    }
  }
 
  return false;
}

References client_manager_t::clients, and g_client_manager.

Referenced by client_video_render_thread().

create_mixed_ascii_frame_for_client()

char * create_mixed_ascii_frame_for_client	(	uint32_t	target_client_id,
		unsigned short	width,
		unsigned short	height,
		bool	wants_stretch,
		size_t *	out_size,
		bool *	out_grid_changed,
		int *	out_sources_count
	)

Generate personalized ASCII frame for a specific client.

This is the core video mixing function that creates customized ASCII art frames for individual clients. It collects video from all active clients, creates an appropriate grid layout, and converts to ASCII using the target client's terminal capabilities.

ALGORITHM OVERVIEW:

1. FRAME COLLECTION PHASE:
   • Scan all active clients for available video frames
   • Use double-buffer system to get latest frames
   • Implement aggressive frame dropping under load (maintains real-time)
   • Always use the most current frame for professional-grade quality
2. LAYOUT CALCULATION PHASE:
   • Determine grid dimensions based on active client count
   • Calculate cell sizes with aspect ratio preservation
   • Handle special cases: single client, multiple clients
3. COMPOSITE GENERATION PHASE:
   • Create composite image with appropriate dimensions
   • Place each client's video in grid cell with proper scaling
   • Support both normal and half-block rendering modes
4. ASCII CONVERSION PHASE:
   • Convert composite to ASCII using client capabilities
   • Apply client-specific palette and color settings
   • Generate ANSI escape sequences if supported

PERFORMANCE OPTIMIZATIONS:

FRAME BUFFER MANAGEMENT:

• Aggressive frame dropping: always read latest available
• Double-buffer system: ensures smooth frame delivery
• Buffer pool allocation: reduces malloc/free overhead
• Zero-copy operations where possible

CONCURRENCY OPTIMIZATIONS:

• Reader locks on client manager (allows parallel execution)
• Double-buffer atomic operations (lock-free access)
• Atomic snapshots of client state
• Thread-safe buffer operations

CLIENT CAPABILITY INTEGRATION:

TERMINAL AWARENESS:

• Respects client's color depth capabilities
• Uses client's preferred ASCII palette
• Handles UTF-8 vs ASCII-only terminals
• Supports half-block mode for 2x vertical resolution

RENDERING MODES SUPPORTED:

• RENDER_MODE_FOREGROUND: Traditional ASCII art
• RENDER_MODE_BACKGROUND: Inverted colors for better contrast
• RENDER_MODE_HALF_BLOCK: Unicode half-blocks for 2x resolution

ERROR HANDLING STRATEGY:

• Invalid dimensions: Logged and frame generation aborted
• Memory allocation failures: Graceful degradation, cleanup resources
• Client disconnections: Skip client in current frame, continue
• Buffer overruns: Frame dropping maintains real-time performance
• Capability missing: Wait for capabilities before generating frames

MEMORY MANAGEMENT:

• All image operations use buffer pool allocation
• Frame data is properly freed after use
• Cached frames use deep copies (prevent corruption)
• Automatic cleanup on function exit

Parameters

target_client_id	Client who will receive this customized frame
width	Terminal width in characters for this client
height	Terminal height in characters for this client
wants_stretch	Unused parameter (aspect ratio always preserved)
out_size	Pointer to store resulting ASCII frame size

Returns

Allocated ASCII frame string (caller must free), or NULL on error

Note

This function is called at 60fps per client by render threads

Generated frame is customized for target client's capabilities

Memory allocated from buffer pool - caller must free properly

Warning

Function performs extensive validation - invalid input returns NULL

Client capabilities must be received before frame generation

See also

ascii_convert_with_capabilities() For ASCII conversion implementation

framebuffer_read_multi_frame() For frame buffer operations

calculate_fit_dimensions_pixel() For aspect ratio calculations

Definition at line 955 of file stream.c.

                                                                  {
  (void)wants_stretch; // Unused - we always handle aspect ratio ourselves
 
  uint64_t frame_gen_start_ns = time_get_ns();
 
  // Initialize output parameters
  if (out_grid_changed) {
    *out_grid_changed = false;
  }
  if (out_sources_count) {
    *out_sources_count = 0;
  }
 
  if (!out_size || width == 0 || height == 0) {
    SET_ERRNO(ERROR_INVALID_PARAM,
              "Invalid parameters for create_mixed_ascii_frame_for_client: width=%u, height=%u, out_size=%p", width,
              height, out_size);
    return NULL;
  }
 
  // Collect all active clients and their image sources
  image_source_t sources[MAX_CLIENTS];
  uint64_t collect_start_ns = time_get_ns();
  int source_count = collect_video_sources(sources, MAX_CLIENTS);
  uint64_t collect_end_ns = time_get_ns();
 
  // Count sources that actually have video data
  int sources_with_video = 0;
  for (int i = 0; i < source_count; i++) {
    if (sources[i].has_video && sources[i].image) {
      sources_with_video++;
    }
  }
 
  static uint64_t last_detailed_log = 0;
  uint64_t now_ns = collect_end_ns;
  if (now_ns - last_detailed_log > 333 * NS_PER_MS_INT) { // Log every 333ms (3x per second)
    last_detailed_log = now_ns;
    log_info("FRAME_GEN_START: target_client=%u sources=%d collect=%.1fms", target_client_id, sources_with_video,
             (collect_end_ns - collect_start_ns) / NS_PER_MS);
  }
 
  // Return the source count for debugging/tracking
  if (out_sources_count) {
    *out_sources_count = sources_with_video;
  }
 
  // GRID LAYOUT CHANGE DETECTION:
  // Check if the number of active video sources has changed
  // NOTE: We only UPDATE the count and SIGNAL the change via out parameter
  // Broadcasting CLEAR_CONSOLE must happen AFTER the new frames are written to buffers
  // to prevent race condition where CLEAR arrives before new frame is ready
  int previous_count = atomic_load(&g_previous_active_video_count);
  if (sources_with_video != previous_count) {
    // Use compare-and-swap to ensure only ONE thread detects the change
    if (atomic_compare_exchange_strong(&g_previous_active_video_count, &previous_count, sources_with_video)) {
      log_dev_every(
          LOG_RATE_DEFAULT,
          "Grid layout changing: %d -> %d active video sources - caller will broadcast clear AFTER buffering frame",
          previous_count, sources_with_video);
      if (out_grid_changed) {
        *out_grid_changed = true; // Signal to caller
      }
    }
  }
 
  // No active video sources - don't generate placeholder frames
  image_t *composite = NULL;
 
  if (sources_with_video == 0) {
    *out_size = 0;
    // No active video sources for client - this isn't an error.
    // Return NULL to indicate no frame should be sent
    return NULL;
  }
 
  if (sources_with_video == 1) {
    // Single source handling - create composite and convert to ASCII
    // Note: create_single_source_composite returns a reference to sources[i].image
    // which could be modified by other threads. Make a copy to prevent concurrent
    // modification during ascii_convert_with_capabilities.
    image_t *single_source = create_single_source_composite(sources, source_count, target_client_id, width, height);
    if (single_source) {
      composite = image_new_copy(single_source);
      if (!composite) {
        SET_ERRNO(ERROR_MEMORY, "Failed to copy single source composite");
        *out_size = 0;
        return NULL;
      }
    }
  } else {
    // Multiple sources - create grid layout
    composite =
        create_multi_source_composite(sources, source_count, sources_with_video, target_client_id, width, height);
  }
 
  char *out = NULL;
 
  if (!composite) {
    SET_ERRNO(ERROR_INVALID_STATE, "Per-client %u: Failed to create composite image", target_client_id);
    *out_size = 0;
    out = NULL;
  }
 
  // Convert composite to ASCII using client capabilities
  // Pass terminal dimensions so the frame can be padded to full width
  char *ascii_frame = convert_composite_to_ascii(composite, target_client_id, width, height);
 
  if (ascii_frame) {
    // The frame should have been null-terminated by the padding functions.
    // Use strlen() which is optimized and reliable
    size_t ascii_len = strlen(ascii_frame);
 
    // Safety check: don't accept unreasonably large frames (10MB limit)
    if (ascii_len > 10 * 1024 * 1024) {
      log_error("Frame size exceeds 10MB safety limit (possible buffer overflow)");
      SET_ERRNO(ERROR_INVALID_PARAM, "Frame size exceeds 10MB");
      return NULL;
    }
 
    // Ensure frame ends with a reset sequence to avoid garbage at terminal
    // This prevents color codes from leaking into uninitialized memory
    const char reset_seq[] = "\033[0m";
    const size_t reset_len = 4;
 
    if (ascii_len >= reset_len) {
      // Check if frame already ends with reset
      const char *frame_end = ascii_frame + ascii_len - reset_len;
      if (strncmp(frame_end, reset_seq, reset_len) == 0) {
        // Frame properly ends with reset, use full length
        *out_size = ascii_len;
      } else {
        // Frame doesn't end with reset - this is the REAL bug!
        // Truncate to the last occurrence of reset sequence
        const char *last_reset = NULL;
        for (const char *p = ascii_frame + ascii_len - reset_len; p >= ascii_frame; p--) {
          if (strncmp(p, reset_seq, reset_len) == 0) {
            last_reset = p;
            break;
          }
        }
 
        if (last_reset) {
          // Include the reset sequence and truncate after it
          *out_size = (size_t)(last_reset - ascii_frame) + reset_len;
          ascii_frame[*out_size] = '\0'; // Ensure null termination
          log_warn("Frame was missing reset at end (had garbage), truncated from %zu to %zu bytes", ascii_len,
                   *out_size);
        } else {
          // No reset found, use full length as fallback
          *out_size = ascii_len;
          log_warn("Frame has no reset sequences, sending full %zu bytes", ascii_len);
        }
      }
    } else {
      // Frame too short to have a reset, use as-is
      *out_size = ascii_len;
    }
 
    log_dev_every(LOG_RATE_SLOW, "create_mixed_ascii_frame_for_client: Final frame size=%zu bytes for client %u",
                  *out_size, target_client_id);
 
    // Debug: Log the last 50 bytes of the frame to see what's really there
    if (*out_size >= 50) {
      char hex_buf[300] = {0};
      size_t hex_len = 0;
      const uint8_t *last_bytes = (const uint8_t *)ascii_frame + (*out_size - 50);
      for (int i = 0; i < 50 && hex_len < sizeof(hex_buf) - 5; i++) {
        hex_len += snprintf(hex_buf + hex_len, sizeof(hex_buf) - hex_len, "%02X ", last_bytes[i]);
      }
      log_dev_every(4500 * US_PER_MS_INT, "FRAME_LAST_50_BYTES (hex): %s", hex_buf);
 
      // Also log as ASCII for readability
      char ascii_buf[100] = {0};
      for (int i = 0; i < 50 && i < (int)sizeof(ascii_buf) - 1; i++) {
        if (last_bytes[i] >= 32 && last_bytes[i] < 127) {
          ascii_buf[i] = (char)last_bytes[i];
        } else if (last_bytes[i] == '\n') {
          ascii_buf[i] = 'N';
        } else if (last_bytes[i] == '\0') {
          ascii_buf[i] = '0';
        } else {
          ascii_buf[i] = '.';
        }
      }
      log_dev_every(4500 * US_PER_MS_INT, "FRAME_LAST_50_ASCII: %s", ascii_buf);
    }
 
    out = ascii_frame;
  } else {
    SET_ERRNO(ERROR_TERMINAL, "Per-client %u: Failed to convert image to ASCII", target_client_id);
    *out_size = 0;
  }
 
  if (composite) {
    // For single source, composite is a malloc-allocated copy, not from pool
    // Check alloc method to determine correct destroy function
    if (composite->alloc_method == IMAGE_ALLOC_POOL) {
      image_destroy_to_pool(composite);
    } else {
      image_destroy(composite);
    }
  }
  for (int i = 0; i < source_count; i++) {
    if (sources[i].image) {
      image_destroy_to_pool(sources[i].image);
    }
  }
 
  uint64_t frame_gen_end_ns = time_get_ns();
  uint64_t frame_gen_duration_ns = frame_gen_end_ns - frame_gen_start_ns;
  if (frame_gen_duration_ns > 10 * NS_PER_MS_INT) { // Log if > 10ms
    char duration_str[32];
    format_duration_ns((double)frame_gen_duration_ns, duration_str, sizeof(duration_str));
    log_warn("SLOW_FRAME_GENERATION: Client %u full frame gen took %s", target_client_id, duration_str);
  }
 
  return out;
}

References format_duration_ns(), image_destroy(), image_destroy_to_pool(), image_new_copy(), and time_get_ns().

Referenced by client_video_render_thread().

queue_audio_for_client()

int queue_audio_for_client	(	client_info_t *	client,
		const void *	audio_data,
		size_t	data_size
	)

Queue ASCII frame for delivery to specific client.

Packages a generated ASCII frame into a protocol packet and queues it for asynchronous delivery to the target client. This function handles all protocol details including headers, checksums, and metadata.

PACKET STRUCTURE CREATED:

• ascii_frame_packet_t header containing:
  • width, height: Client terminal dimensions
  • original_size: Uncompressed frame size
  • compressed_size: Compressed size (currently 0)
  • checksum: CRC32 of frame data for integrity
  • flags: Capability flags (color support, etc.)
• Frame data: Complete ASCII art string with ANSI codes

PROTOCOL INTEGRATION:

• Converts dimensions to network byte order
• Generates CRC32 checksum for error detection
• Sets appropriate capability flags based on client
• Uses PACKET_TYPE_ASCII_FRAME packet type

DELIVERY MECHANISM:

• Queues packet in client's video packet queue
• Send thread delivers packet asynchronously
• Queue overflow handling prevents memory exhaustion
• Error conditions are logged but don't affect other clients

MEMORY MANAGEMENT:

• Creates temporary packet buffer for header+data combination
• packet_queue_enqueue() copies data (safe to free temp buffer)
• No memory leaks on success or failure paths
• Buffer pool allocation not used here (temporary buffer)

ERROR HANDLING:

• Invalid parameters: Return -1, no side effects
• Memory allocation failure: Logged, return -1
• Queue overflow: Logged as debug (expected under load)
• Client shutdown: Graceful failure without error spam

Parameters

client	Target client for frame delivery
ascii_frame	Generated ASCII art string (null-terminated)
frame_size	Size of ASCII frame in bytes

Returns

0 on successful queuing, -1 on error

Note

Frame delivery happens asynchronously via send thread

Queue overflow drops frames to maintain real-time performance

Checksums enable client-side integrity verification

See also

packet_queue_enqueue() For underlying queue implementation

asciichat_crc32() For checksum generation

Queue audio data for delivery to specific client

Queues mixed audio data for delivery to a specific client. This is a simple wrapper around the packet queue system for audio delivery.

AUDIO PIPELINE INTEGRATION:

• Called by audio mixing threads after combining multiple client streams
• Audio data is already in final format (float samples, mixed)
• No additional processing required at this stage

PACKET DETAILS:

• Uses PACKET_TYPE_AUDIO_BATCH packet type
• Audio data is raw float samples bundled together
• Batch format reduces packet overhead ~32x
• Sample rate and format determined by audio system

DELIVERY CHARACTERISTICS:

• Higher priority than video packets (lower latency)
• Uses client's audio packet queue
• Send thread prioritizes audio over video
• Queue overflow drops oldest audio to maintain real-time

ERROR HANDLING:

• Invalid parameters: Return -1 immediately
• Queue overflow: Handled by packet queue (drops old data)
• Client shutdown: Graceful failure

Parameters

client	Target client for audio delivery
audio_data	Mixed audio samples (float format)
data_size	Size of audio data in bytes

Returns

0 on successful queuing, -1 on error

Note

Audio has priority over video in send thread

Real-time audio requires queue overflow handling

See also

packet_queue_enqueue() For underlying queue implementation

mixer.c For Audio mixing implementation

Definition at line 1285 of file stream.c.

                                                                                            {
  if (!client || !client->audio_queue || !audio_data || data_size == 0) {
    return -1;
  }
 
  return packet_queue_enqueue(client->audio_queue, PACKET_TYPE_AUDIO_BATCH, audio_data, data_size, 0, true);
}

References packet_queue_enqueue().