stream.c File Reference

🎬 Multi-client video mixer: frame generation, ASCII conversion, and per-client personalized rendering More...

Go to the source code of this file.

Data Structures
struct	image_source_t
	One client's video input as seen by the compositor. More...

Macros
#define	SERVER_STREAM_HEX_BUFFER_SIZE   300
#define	SERVER_STREAM_ASCII_BUFFER_SIZE   100

Functions
char *	create_mixed_ascii_frame_for_client (const char *target_client_id, unsigned short width, unsigned short height, bool wants_stretch, size_t *out_size, bool *out_grid_changed, int *out_sources_count, acip_render_settings_t *out_rendering)
	Generate personalized ASCII frame for a specific client.
bool	any_clients_sending_video (void)
	Queue ASCII frame for delivery to specific client.

Detailed Description

🎬 Multi-client video mixer: frame generation, ASCII conversion, and per-client personalized rendering

1. Collect video frames from all active clientsclientsCreate composite video layouts (single client, 2x2, 3x3 grids))Generate client-specific ASCII art with terminal capability awarenessawarenessProcess latest frames from double-buffer system for real-time performanceperformanceManage memory efficiently with owned buffers and zero-copy operationsoperationsSupport advanced rendering modes (half-block, color, custom palettes) VIDEO MIXING ARCHITECTURE:

The mixing system operates in several stages:

1. FRAME COLLECTION:
   • Scans all active clients for available video frames
   • Uses per-client double-buffer system for smooth frame handling
   • Implements aggressive frame dropping to maintain real-time performance
   • Always uses the latest available frame for professional-grade performance
2. LAYOUT CALCULATION:
   • Determines grid size based on number of active video sources
   • Supports: single (1x1), side-by-side (2x1), 2x2, 3x2, 3x3 layouts
   • Calculates aspect-ratio preserving dimensions for each cell
3. COMPOSITE GENERATION:
   • Creates composite image with proper aspect ratio handling
   • Places each client's video in appropriate grid cell
   • Supports both normal and half-block rendering modes
4. ASCII CONVERSION:
   • Converts composite to ASCII using client-specific capabilities
   • Respects terminal color support, palette preferences, UTF-8 capability
   • Generates ANSI escape sequences for color output
5. PACKET GENERATION:
   • Wraps ASCII frames in protocol packets with metadata
   • Includes checksums, dimensions, and capability flags
   • Queues packets for delivery via client send threads

PER-CLIENT CUSTOMIZATION:

Unlike traditional video mixing that generates one output, this system creates personalized frames for each client:

TERMINAL CAPABILITY AWARENESS:

• Color depth: 1-bit (mono), 8-color, 16-color, 256-color, 24-bit RGB
• Character support: ASCII-only vs UTF-8 box drawing
• Render modes: foreground, background, half-block (2x resolution)
• Custom ASCII palettes: brightness-to-character mapping

PERFORMANCE OPTIMIZATIONS:

• Per-client palette caches (avoid repeated initialization)
• Double-buffer system for smooth frame delivery
• buffer domain allocation to reduce malloc/free overhead
• Aggressive frame dropping under load
• Zero-copy operations where possible

THREADING AND CONCURRENCY:

This module operates in a high-concurrency environment:

THREAD SAFETY MECHANISMS:

• Double-buffer thread safety (atomic operations)
• Reader-writer locks on client manager (allows concurrent reads)
• buffer domain thread safety (lock-free where possible)
• Atomic snapshot operations for client state

PERFORMANCE CHARACTERISTICS:

• Supports 60fps per client with linear scaling
• Handles burst traffic with frame buffer overruns
• Minimal CPU overhead with double-buffer system
• Memory usage scales with number of active clients

FRAME BUFFER MANAGEMENT:

DOUBLE-BUFFER STRATEGY: Each client uses a double-buffer system for smooth frame delivery:

• Always provides the latest available frame
• No frame caching complexity or stale data concerns
• Professional-grade real-time performance like Zoom/Google Meet
• Memory managed via owned buffers

BUFFER OVERFLOW HANDLING: When clients send frames faster than processing:

• Aggressive frame dropping (keep only latest)
• Logarithmic drop rate based on buffer occupancy
• Maintains real-time performance under load

INTEGRATION WITH OTHER MODULES:

• render.c: Calls frame generation functions at 60fps per client
• client.c: Provides client state and capabilities
• protocol.c: Stores incoming video frames in per-client buffers
• video/: Performs actual RGB-to-ASCII conversion
• palette.c: Manages client-specific character palettes

WHY THIS MODULAR DESIGN:

The original server.c contained all video mixing logic inline, making it:

• Impossible to understand the video pipeline
• Difficult to optimize performance bottlenecks
• Hard to add new rendering features
• Challenging to debug frame generation issues

This separation provides:

• Clear video processing pipeline
• Isolated performance optimization
• Easier feature development
• Better error isolation and debugging

MEMORY MANAGEMENT PHILOSOPHY:

• All allocations use owned buffers where possible
• Frame data is reference-counted (avoid copies)
• Double-buffer system eliminates need for frame copies
• Automatic cleanup on client disconnect
• Graceful degradation on allocation failures

Author

Zachary Fogg me@zf.nosp@m.o.gg

Date

September 2025

Version

2.0 (Post-Modularization)

See also

render.c For frame generation scheduling

video/ For RGB-to-ASCII conversion implementation

palette.c For client Character Palettes management

Definition in file src/server/stream.c.

Macro Definition Documentation

SERVER_STREAM_ASCII_BUFFER_SIZE

#define SERVER_STREAM_ASCII_BUFFER_SIZE   100

Definition at line 133 of file src/server/stream.c.

Referenced by create_mixed_ascii_frame_for_client().

SERVER_STREAM_HEX_BUFFER_SIZE

#define SERVER_STREAM_HEX_BUFFER_SIZE   300

Definition at line 132 of file src/server/stream.c.

Referenced by create_mixed_ascii_frame_for_client().

Function Documentation

any_clients_sending_video()

bool any_clients_sending_video

(

void

)

Queue ASCII frame for delivery to specific client.

Packages a generated ASCII frame into a protocol packet and queues it for asynchronous delivery to the target client. This function handles all protocol details including headers, checksums, and metadata.

PACKET STRUCTURE CREATED:

• ascii_frame_packet_t header containing:
  • width, height: Client terminal dimensions
  • original_size: Uncompressed frame size
  • compressed_size: Compressed size (currently 0)
  • checksum: CRC32 of frame data for integrity
  • flags: Capability flags (color support, etc.)
• Frame data: Complete ASCII art string with ANSI codes

PROTOCOL INTEGRATION:

• Converts dimensions to network byte order
• Generates CRC32 checksum for error detection
• Sets appropriate capability flags based on client
• Uses PACKET_TYPE_ASCII_FRAME packet type

DELIVERY MECHANISM:

• Queues packet in client's video packet queue
• Send thread delivers packet asynchronously
• Queue overflow handling prevents memory exhaustion
• Error conditions are logged but don't affect other clients

MEMORY MANAGEMENT:

• Creates temporary packet buffer for header+data combination
• packet_queue_push_copy() copies data (safe to free temp buffer)
• No memory leaks on success or failure paths
• buffer domain allocation not used here (temporary buffer)

ERROR HANDLING:

• Invalid parameters: Return -1, no side effects
• Memory allocation failure: Logged, return -1
• Queue overflow: Logged as debug (expected under load)
• Client shutdown: Graceful failure without error spam

Parameters

client	Target client for frame delivery
ascii_frame	Generated ASCII art string (null-terminated)
frame_size	Size of ASCII frame in bytes

Returns

0 on successful queuing, -1 on error

Note

Frame delivery happens asynchronously via send thread

Queue overflow drops frames to maintain real-time performance

Checksums enable client-side integrity verification

See also

packet_queue_push_copy() For underlying queue implementation

asciichat_crc32() For checksum generation

Check if any connected clients are currently sending video

This function scans all active clients to determine if at least one is sending video frames. Used by render threads to avoid generating frames when no video sources are available (e.g., during webcam warmup).

LOCK OPTIMIZATION: Uses atomic reads only, no rwlock acquisition This is safe because client_id, active, and is_sending_video are all atomics

Returns

true if at least one client has is_sending_video flag set, false otherwise

Definition at line 1283 of file src/server/stream.c.

                                     {
  // LOCK OPTIMIZATION: Don't acquire rwlock - all fields we access are atomic
  // client_id, active, is_sending_video are all atomic variables
 
  // Iterate through all client slots
  for (int i = 0; i < MAX_CLIENTS; i++) {
    client_info_t *client = &g_client_manager.clients[i];
 
    // Skip uninitialized clients (atomic read)
    if (client->client_id[0] == '\0') {
      continue;
    }
 
    // Check if client is active and sending video (both atomic reads)
    bool is_active = atomic_load_bool(&client->active);
    bool is_sending = atomic_load_bool(&client->is_sending_video);
 
    if (is_active && is_sending) {
      return true;
    }
  }
 
  return false;
}

References client_info::active, client_info::client_id, g_client_manager, client_info::is_sending_video, and MAX_CLIENTS.

Referenced by client_video_render_thread().

create_mixed_ascii_frame_for_client()

char * create_mixed_ascii_frame_for_client	(	const char *	target_client_id,
		unsigned short	width,
		unsigned short	height,
		bool	wants_stretch,
		size_t *	out_size,
		bool *	out_grid_changed,
		int *	out_sources_count,
		acip_render_settings_t *	out_rendering )

Generate personalized ASCII frame for a specific client.

This is the core video mixing function that creates customized ASCII art frames for individual clients. It collects video from all active clients, creates an appropriate grid layout, and converts to ASCII using the target client's terminal capabilities.

ALGORITHM OVERVIEW:

1. FRAME COLLECTION PHASE:
   • Scan all active clients for available video frames
   • Use double-buffer system to get latest frames
   • Implement aggressive frame dropping under load (maintains real-time)
   • Always use the most current frame for professional-grade quality
2. LAYOUT CALCULATION PHASE:
   • Determine grid dimensions based on active client count
   • Calculate cell sizes with aspect ratio preservation
   • Handle special cases: single client, multiple clients
3. COMPOSITE GENERATION PHASE:
   • Create composite image with appropriate dimensions
   • Place each client's video in grid cell with proper scaling
   • Support both normal and half-block rendering modes
4. ASCII CONVERSION PHASE:
   • Convert composite to ASCII using client capabilities
   • Apply client-specific palette and color settings
   • Generate ANSI escape sequences if supported

PERFORMANCE OPTIMIZATIONS:

FRAME BUFFER MANAGEMENT:

• Aggressive frame dropping: always read latest available
• Double-buffer system: ensures smooth frame delivery
• buffer domain allocation: reduces malloc/free overhead
• Zero-copy operations where possible

CONCURRENCY OPTIMIZATIONS:

• Reader locks on client manager (allows parallel execution)
• Double-buffer atomic operations (lock-free access)
• Atomic snapshots of client state
• Thread-safe buffer operations

CLIENT CAPABILITY INTEGRATION:

TERMINAL AWARENESS:

• Respects client's color depth capabilities
• Uses client's preferred ASCII palette
• Handles UTF-8 vs ASCII-only terminals
• Supports half-block mode for 2x vertical resolution

RENDERING MODES SUPPORTED:

• RENDER_MODE_FOREGROUND: Traditional ASCII art
• RENDER_MODE_BACKGROUND: Inverted colors for better contrast
• RENDER_MODE_HALF_BLOCK: Unicode half-blocks for 2x resolution

ERROR HANDLING STRATEGY:

• Invalid dimensions: Logged and frame generation aborted
• Memory allocation failures: Graceful degradation, cleanup resources
• Client disconnections: Skip client in current frame, continue
• Buffer overruns: Frame dropping maintains real-time performance
• Capability missing: Wait for capabilities before generating frames

MEMORY MANAGEMENT:

• All image operations use buffer domain allocation
• Frame data is properly freed after use
• Cached frames use deep copies (prevent corruption)
• Automatic cleanup on function exit

Parameters

target_client_id	Client who will receive this customized frame
width	Terminal width in characters for this client
height	Terminal height in characters for this client
wants_stretch	Unused parameter (aspect ratio always preserved)
out_size	Pointer to store resulting ASCII frame size

Returns

Allocated ASCII frame string (caller must free), or NULL on error

Note

This function is called at 60fps per client by render threads

Generated frame is customized for target client's capabilities

Memory allocated from buffer domain - caller must free properly

Warning

Function performs extensive validation - invalid input returns NULL

Client capabilities must be received before frame generation

See also

ascii_convert_with_capabilities() For ASCII conversion implementation

framebuffer_read_multi_frame() For frame Owned Buffers operations

calculate_fit_dimensions_pixel() For aspect ratio calculations

Definition at line 1024 of file src/server/stream.c.

                                                                                                         {
  (void)wants_stretch; // Unused - we always handle aspect ratio ourselves
 
  // Register the stream counter once even when per-client render workers enter
  // this function concurrently.
  static _Atomic bool stream_atomics_registered = false;
  if (!atomic_exchange_explicit(&stream_atomics_registered, true, memory_order_acq_rel)) {
    NAMED_REGISTER_ATOMIC(&g_previous_active_video_count, "server_video_source_count_for_layout_detection", NULL);
  }
 
  uint64_t frame_gen_start_ns = time_get_ns();
 
  // Initialize output parameters
  if (out_grid_changed) {
    *out_grid_changed = false;
  }
  if (out_sources_count) {
    *out_sources_count = 0;
  }
 
  if (!out_size || width == 0 || height == 0) {
    SET_ERRNO(ERROR_INVALID_PARAM, "Invalid parameters: width=%u, height=%u, out_size=%p", width, height, out_size);
    return NULL;
  }
 
  // Collect all active clients and their image sources
  image_source_t sources[MAX_CLIENTS];
  uint64_t collect_start_ns = time_get_ns();
  int source_count = collect_video_sources(sources, MAX_CLIENTS);
  uint64_t collect_end_ns = time_get_ns();
 
  // Count sources that actually have video data
  int sources_with_video = 0;
  for (int i = 0; i < source_count; i++) {
    if (sources[i].has_video && sources[i].image) {
      sources_with_video++;
    }
  }
 
  log_dev_every(333 * NS_PER_MS_INT, "FRAME_GEN_START: target_client=%s sources=%d collect=%.1fms", target_client_id,
                sources_with_video, (collect_end_ns - collect_start_ns) / NS_PER_MS);
 
  // Return the source count for debugging/tracking
  if (out_sources_count) {
    *out_sources_count = sources_with_video;
  }
 
  // GRID LAYOUT CHANGE DETECTION:
  // Check if the number of active video sources has changed
  // NOTE: We only UPDATE the count and SIGNAL the change via out parameter
  // Broadcasting CLEAR_CONSOLE must happen AFTER the new frames are written to buffers
  // to prevent race condition where CLEAR arrives before new frame is ready
  uint64_t previous_count = atomic_load_u64(&g_previous_active_video_count);
  if (sources_with_video != (int)previous_count) {
    // Use compare-and-swap to ensure only ONE thread detects the change
    if (atomic_cas_u64(&g_previous_active_video_count, &previous_count, (uint64_t)sources_with_video)) {
      log_dev_every(
          LOG_RATE_DEFAULT,
          "Grid layout changing: %d -> %d active video sources - caller will broadcast clear AFTER buffering frame",
          previous_count, sources_with_video);
      if (out_grid_changed) {
        *out_grid_changed = true; // Signal to caller
      }
    }
  }
 
  // No active video sources - don't generate placeholder frames
  image_t *composite = NULL;
 
  if (sources_with_video == 0) {
    *out_size = 0;
    // No active video sources for client - this isn't an error.
    // Return NULL to indicate no frame should be sent
    return NULL;
  }
 
  if (sources_with_video == 1) {
    // Single source handling - create composite and convert to ASCII
    // Note: create_single_source_composite returns a reference to sources[i].image
    // which could be modified by other threads. Make a copy to prevent concurrent
    // modification during ascii_convert_with_capabilities.
    image_t *single_source = create_single_source_composite(sources, source_count, target_client_id, width, height);
    if (single_source) {
      composite = image_new_copy(single_source);
      if (!composite) {
        SET_ERRNO(ERROR_MEMORY, "Failed to copy single source composite");
        *out_size = 0;
        return NULL;
      }
    }
  } else {
    // Multiple sources - create grid layout
    composite =
        create_multi_source_composite(sources, source_count, sources_with_video, target_client_id, width, height);
  }
 
  char *out = NULL;
 
  if (!composite) {
    SET_ERRNO(ERROR_INVALID_STATE, "Per-client %s: Failed to create composite image", target_client_id);
    *out_size = 0;
    return NULL;
  }
 
  // Convert composite to ASCII using client capabilities
  // Pass terminal dimensions so the frame can be padded to full width
  char *ascii_frame = convert_composite_to_ascii(composite, target_client_id, width, height, out_rendering, sources,
                                                 source_count, sources_with_video);
 
  if (ascii_frame) {
    // The frame should have been null-terminated by the padding functions.
    // Use strlen() which is optimized and reliable
    size_t ascii_len = strlen(ascii_frame);
 
    // Safety check: don't accept unreasonably large frames (10MB limit)
    if (ascii_len > 10 * 1024 * 1024) {
      log_error("Frame size exceeds 10MB safety limit (possible buffer overflow)");
      SET_ERRNO(ERROR_INVALID_PARAM, "Frame size exceeds 10MB");
      SAFE_FREE(ascii_frame);
      image_destroy(composite);
      *out_size = 0;
      return NULL;
    }
 
    *out_size = ascii_len; // Renderer owns complete rows; never truncate to a color reset.
 
    log_dev_every(LOG_RATE_SLOW, "Final frame size=%zu bytes for client %s", *out_size, target_client_id);
 
    // Debug: Log the last 50 bytes of the frame to see what's really there
    if (*out_size >= 50) {
      char hex_buf[SERVER_STREAM_HEX_BUFFER_SIZE] = {0};
      size_t hex_len = 0;
      const uint8_t *last_bytes = (const uint8_t *)ascii_frame + (*out_size - 50);
      for (int i = 0; i < 50 && hex_len < sizeof(hex_buf) - 5; i++) {
        hex_len += snprintf(hex_buf + hex_len, sizeof(hex_buf) - hex_len, "%02X ", last_bytes[i]);
      }
      log_dev_every(4500 * US_PER_MS_INT, "FRAME_LAST_50_BYTES (hex): %s", hex_buf);
 
      // Also log as ASCII for readability
      char ascii_buf[SERVER_STREAM_ASCII_BUFFER_SIZE] = {0};
      for (int i = 0; i < 50 && i < (int)sizeof(ascii_buf) - 1; i++) {
        if (last_bytes[i] >= 32 && last_bytes[i] < 127) {
          ascii_buf[i] = (char)last_bytes[i];
        } else if (last_bytes[i] == '\n') {
          ascii_buf[i] = 'N';
        } else if (last_bytes[i] == '\0') {
          ascii_buf[i] = '0';
        } else {
          ascii_buf[i] = '.';
        }
      }
      log_dev_every(4500 * US_PER_MS_INT, "FRAME_LAST_50_ASCII: %s", ascii_buf);
    }
 
    out = ascii_frame;
  } else {
    SET_ERRNO(ERROR_TERMINAL, "Per-client %s: Failed to convert image to ASCII", target_client_id);
    *out_size = 0;
  }
 
  if (composite) {
    image_destroy(composite);
  }
  for (int i = 0; i < source_count; i++) {
    if (sources[i].image) {
      image_destroy(sources[i].image);
    }
  }
 
  uint64_t frame_gen_end_ns = time_get_ns();
  uint64_t frame_gen_duration_ns = frame_gen_end_ns - frame_gen_start_ns;
  if (frame_gen_duration_ns > 10 * NS_PER_MS_INT) { // Log if > 10ms
    char duration_str[TIME_DURATION_BUFFER_SIZE];
    time_pretty((uint64_t)((double)frame_gen_duration_ns), -1, duration_str, sizeof(duration_str));
    log_warn_every(LOG_RATE_DEFAULT, "SLOW_FRAME_GENERATION: Client %s full frame gen took %s", target_client_id,
                   duration_str);
  }
 
  return out;
}

References ERROR_INVALID_PARAM, ERROR_INVALID_STATE, ERROR_MEMORY, ERROR_TERMINAL, height, image_destroy(), image_new_copy(), log_dev_every, log_error, LOG_RATE_DEFAULT, LOG_RATE_SLOW, log_warn_every, MAX_CLIENTS, NAMED_REGISTER_ATOMIC, NS_PER_MS, NS_PER_MS_INT, out, SAFE_FREE, SERVER_STREAM_ASCII_BUFFER_SIZE, SERVER_STREAM_HEX_BUFFER_SIZE, SET_ERRNO, TIME_DURATION_BUFFER_SIZE, time_pretty(), US_PER_MS_INT, and width.

Referenced by client_video_render_thread().