stream.h File Reference

Multi-client video mixing and ASCII frame generation. More...

Go to the source code of this file.

Functions
char *	create_mixed_ascii_frame_for_client (const char *target_client_id, unsigned short width, unsigned short height, bool wants_stretch, size_t *out_size, bool *out_grid_changed, int *out_sources_count)
	Generate personalized ASCII frame for a specific client.
bool	any_clients_sending_video (void)
	Check if any connected clients are currently sending video.
int	queue_audio_for_client (client_info_t *client, const void *audio_data, size_t data_size)
	Queue ASCII frame for delivery to specific client.

Variables
atomic_t	g_should_exit
	Global application exit flag (shared across all modes)

Detailed Description

Multi-client video mixing and ASCII frame generation.

Definition in file stream.h.

Function Documentation

any_clients_sending_video()

bool any_clients_sending_video

(

void

)

Check if any connected clients are currently sending video.

This function scans all active clients to determine if at least one is sending video frames. Used by render threads to avoid generating frames when no video sources are available (e.g., during webcam warmup).

LOCK OPTIMIZATION: Uses atomic reads only, no rwlock acquisition This is safe because client_id, active, and is_sending_video are all atomics

Returns

true if at least one client has is_sending_video flag set, false otherwise

Definition at line 1335 of file stream.c.

                                     {
  // LOCK OPTIMIZATION: Don't acquire rwlock - all fields we access are atomic
  // client_id, active, is_sending_video are all atomic variables
 
  // Iterate through all client slots
  for (int i = 0; i < MAX_CLIENTS; i++) {
    client_info_t *client = &g_client_manager.clients[i];
 
    // Skip uninitialized clients (atomic read)
    if (client->client_id[0] == '\0') {
      continue;
    }
 
    // Check if client is active and sending video (both atomic reads)
    bool is_active = atomic_load_bool(&client->active);
    bool is_sending = atomic_load_bool(&client->is_sending_video);
 
    if (is_active && is_sending) {
      return true;
    }
  }
 
  return false;
}

References client_info::active, atomic_load_bool(), client_info::client_id, client_manager_t::clients, g_client_manager, client_info::is_sending_video, and MAX_CLIENTS.

Referenced by client_video_render_thread().

create_mixed_ascii_frame_for_client()

char * create_mixed_ascii_frame_for_client	(	const char *	target_client_id,
		unsigned short	width,
		unsigned short	height,
		bool	wants_stretch,
		size_t *	out_size,
		bool *	out_grid_changed,
		int *	out_sources_count
	)

Generate personalized ASCII frame for a specific client.

This is the core video mixing function that creates customized ASCII art frames for individual clients. It collects video from all active clients, creates an appropriate grid layout, and converts to ASCII using the target client's terminal capabilities.

ALGORITHM OVERVIEW:

1. FRAME COLLECTION PHASE:
   • Scan all active clients for available video frames
   • Use double-buffer system to get latest frames
   • Implement aggressive frame dropping under load (maintains real-time)
   • Always use the most current frame for professional-grade quality
2. LAYOUT CALCULATION PHASE:
   • Determine grid dimensions based on active client count
   • Calculate cell sizes with aspect ratio preservation
   • Handle special cases: single client, multiple clients
3. COMPOSITE GENERATION PHASE:
   • Create composite image with appropriate dimensions
   • Place each client's video in grid cell with proper scaling
   • Support both normal and half-block rendering modes
4. ASCII CONVERSION PHASE:
   • Convert composite to ASCII using client capabilities
   • Apply client-specific palette and color settings
   • Generate ANSI escape sequences if supported

PERFORMANCE OPTIMIZATIONS:

FRAME BUFFER MANAGEMENT:

• Aggressive frame dropping: always read latest available
• Double-buffer system: ensures smooth frame delivery
• Buffer pool allocation: reduces malloc/free overhead
• Zero-copy operations where possible

CONCURRENCY OPTIMIZATIONS:

• Reader locks on client manager (allows parallel execution)
• Double-buffer atomic operations (lock-free access)
• Atomic snapshots of client state
• Thread-safe buffer operations

CLIENT CAPABILITY INTEGRATION:

TERMINAL AWARENESS:

• Respects client's color depth capabilities
• Uses client's preferred ASCII palette
• Handles UTF-8 vs ASCII-only terminals
• Supports half-block mode for 2x vertical resolution

RENDERING MODES SUPPORTED:

• RENDER_MODE_FOREGROUND: Traditional ASCII art
• RENDER_MODE_BACKGROUND: Inverted colors for better contrast
• RENDER_MODE_HALF_BLOCK: Unicode half-blocks for 2x resolution

ERROR HANDLING STRATEGY:

• Invalid dimensions: Logged and frame generation aborted
• Memory allocation failures: Graceful degradation, cleanup resources
• Client disconnections: Skip client in current frame, continue
• Buffer overruns: Frame dropping maintains real-time performance
• Capability missing: Wait for capabilities before generating frames

MEMORY MANAGEMENT:

• All image operations use buffer pool allocation
• Frame data is properly freed after use
• Cached frames use deep copies (prevent corruption)
• Automatic cleanup on function exit

Parameters

target_client_id	Client who will receive this customized frame
width	Terminal width in characters for this client
height	Terminal height in characters for this client
wants_stretch	Unused parameter (aspect ratio always preserved)
out_size	Pointer to store resulting ASCII frame size

Returns

Allocated ASCII frame string (caller must free), or NULL on error

Note

This function is called at 60fps per client by render threads

Generated frame is customized for target client's capabilities

Memory allocated from buffer pool - caller must free properly

Warning

Function performs extensive validation - invalid input returns NULL

Client capabilities must be received before frame generation

See also

ascii_convert_with_capabilities() For ASCII conversion implementation

framebuffer_read_multi_frame() For frame buffer operations

calculate_fit_dimensions_pixel() For aspect ratio calculations

Definition at line 973 of file stream.c.

                                                                  {
  (void)wants_stretch; // Unused - we always handle aspect ratio ourselves
 
  // Register the stream counter once even when per-client render workers enter
  // this function concurrently.
  static _Atomic bool stream_atomics_registered = false;
  if (!atomic_exchange_explicit(&stream_atomics_registered, true, memory_order_acq_rel)) {
    NAMED_REGISTER_ATOMIC(&g_previous_active_video_count, "server_video_source_count_for_layout_detection", NULL);
  }
 
  uint64_t frame_gen_start_ns = time_get_ns();
 
  // Initialize output parameters
  if (out_grid_changed) {
    *out_grid_changed = false;
  }
  if (out_sources_count) {
    *out_sources_count = 0;
  }
 
  if (!out_size || width == 0 || height == 0) {
    SET_ERRNO(ERROR_INVALID_PARAM,
              "Invalid parameters for create_mixed_ascii_frame_for_client: width=%u, height=%u, out_size=%p", width,
              height, out_size);
    return NULL;
  }
 
  // Collect all active clients and their image sources
  image_source_t sources[MAX_CLIENTS];
  uint64_t collect_start_ns = time_get_ns();
  int source_count = collect_video_sources(sources, MAX_CLIENTS);
  uint64_t collect_end_ns = time_get_ns();
 
  // Count sources that actually have video data
  int sources_with_video = 0;
  for (int i = 0; i < source_count; i++) {
    if (sources[i].has_video && sources[i].image) {
      sources_with_video++;
    }
  }
 
  log_dev_every(333 * NS_PER_MS_INT, "FRAME_GEN_START: target_client=%s sources=%d collect=%.1fms", target_client_id,
               sources_with_video, (collect_end_ns - collect_start_ns) / NS_PER_MS);
 
  // Return the source count for debugging/tracking
  if (out_sources_count) {
    *out_sources_count = sources_with_video;
  }
 
  // GRID LAYOUT CHANGE DETECTION:
  // Check if the number of active video sources has changed
  // NOTE: We only UPDATE the count and SIGNAL the change via out parameter
  // Broadcasting CLEAR_CONSOLE must happen AFTER the new frames are written to buffers
  // to prevent race condition where CLEAR arrives before new frame is ready
  uint64_t previous_count = atomic_load_u64(&g_previous_active_video_count);
  if (sources_with_video != (int)previous_count) {
    // Use compare-and-swap to ensure only ONE thread detects the change
    if (atomic_cas_u64(&g_previous_active_video_count, &previous_count, (uint64_t)sources_with_video)) {
      log_dev_every(
          LOG_RATE_DEFAULT,
          "Grid layout changing: %d -> %d active video sources - caller will broadcast clear AFTER buffering frame",
          previous_count, sources_with_video);
      if (out_grid_changed) {
        *out_grid_changed = true; // Signal to caller
      }
    }
  }
 
  // No active video sources - don't generate placeholder frames
  image_t *composite = NULL;
 
  if (sources_with_video == 0) {
    *out_size = 0;
    // No active video sources for client - this isn't an error.
    // Return NULL to indicate no frame should be sent
    return NULL;
  }
 
  START_TIMER("composite_create");
  if (sources_with_video == 1) {
    // Single source handling - create composite and convert to ASCII
    // Note: create_single_source_composite returns a reference to sources[i].image
    // which could be modified by other threads. Make a copy to prevent concurrent
    // modification during ascii_convert_with_capabilities.
    image_t *single_source = create_single_source_composite(sources, source_count, target_client_id, width, height);
    if (single_source) {
      composite = image_new_copy(single_source);
      if (!composite) {
        SET_ERRNO(ERROR_MEMORY, "Failed to copy single source composite");
        *out_size = 0;
        return NULL;
      }
    }
  } else {
    // Multiple sources - create grid layout
    composite =
        create_multi_source_composite(sources, source_count, sources_with_video, target_client_id, width, height);
  }
  STOP_TIMER_AND_LOG(dev, 500000, "composite_create", "Composite creation (single/multi)");
 
  char *out = NULL;
 
  if (!composite) {
    SET_ERRNO(ERROR_INVALID_STATE, "Per-client %s: Failed to create composite image", target_client_id);
    *out_size = 0;
    return NULL;
  }
 
  // Convert composite to ASCII using client capabilities
  // Pass terminal dimensions so the frame can be padded to full width
  START_TIMER("ascii_convert");
  char *ascii_frame = convert_composite_to_ascii(composite, target_client_id, width, height);
  STOP_TIMER_AND_LOG(dev, 500000, "ascii_convert", "ASCII conversion (composite to ASCII)");
 
  if (ascii_frame) {
    // The frame should have been null-terminated by the padding functions.
    // Use strlen() which is optimized and reliable
    size_t ascii_len = strlen(ascii_frame);
 
    // Safety check: don't accept unreasonably large frames (10MB limit)
    if (ascii_len > 10 * 1024 * 1024) {
      log_error("Frame size exceeds 10MB safety limit (possible buffer overflow)");
      SET_ERRNO(ERROR_INVALID_PARAM, "Frame size exceeds 10MB");
      SAFE_FREE(ascii_frame);
      image_destroy(composite);
      *out_size = 0;
      return NULL;
    }
 
    // Ensure frame ends with a reset sequence to avoid garbage at terminal
    // This prevents color codes from leaking into uninitialized memory
    const char reset_seq[] = "\033[0m";
    const size_t reset_len = 4;
 
    if (ascii_len >= reset_len) {
      // Check if frame already ends with reset
      const char *frame_end = ascii_frame + ascii_len - reset_len;
      if (strncmp(frame_end, reset_seq, reset_len) == 0) {
        // Frame properly ends with reset, use full length
        *out_size = ascii_len;
      } else {
        // Frame doesn't end with reset - this is the REAL bug!
        // Truncate to the last occurrence of reset sequence
        const char *last_reset = NULL;
        for (const char *p = ascii_frame + ascii_len - reset_len; p >= ascii_frame; p--) {
          if (strncmp(p, reset_seq, reset_len) == 0) {
            last_reset = p;
            break;
          }
        }
 
        if (last_reset) {
          // Include the reset sequence and truncate after it
          *out_size = (size_t)(last_reset - ascii_frame) + reset_len;
          ascii_frame[*out_size] = '\0'; // Ensure null termination
          log_warn("Frame was missing reset at end (had garbage), truncated from %zu to %zu bytes", ascii_len,
                   *out_size);
        } else {
          // No reset found, use full length as fallback
          *out_size = ascii_len;
          log_dev_every(LOG_RATE_SLOW, "Frame has no reset sequences, sending full %zu bytes", ascii_len);
        }
      }
    } else {
      // Frame too short to have a reset, use as-is
      *out_size = ascii_len;
    }
 
    log_dev_every(LOG_RATE_SLOW, "create_mixed_ascii_frame_for_client: Final frame size=%zu bytes for client %s",
                  *out_size, target_client_id);
 
    // Debug: Log the last 50 bytes of the frame to see what's really there
    if (*out_size >= 50) {
      char hex_buf[300] = {0};
      size_t hex_len = 0;
      const uint8_t *last_bytes = (const uint8_t *)ascii_frame + (*out_size - 50);
      for (int i = 0; i < 50 && hex_len < sizeof(hex_buf) - 5; i++) {
        hex_len += snprintf(hex_buf + hex_len, sizeof(hex_buf) - hex_len, "%02X ", last_bytes[i]);
      }
      log_dev_every(4500 * US_PER_MS_INT, "FRAME_LAST_50_BYTES (hex): %s", hex_buf);
 
      // Also log as ASCII for readability
      char ascii_buf[100] = {0};
      for (int i = 0; i < 50 && i < (int)sizeof(ascii_buf) - 1; i++) {
        if (last_bytes[i] >= 32 && last_bytes[i] < 127) {
          ascii_buf[i] = (char)last_bytes[i];
        } else if (last_bytes[i] == '\n') {
          ascii_buf[i] = 'N';
        } else if (last_bytes[i] == '\0') {
          ascii_buf[i] = '0';
        } else {
          ascii_buf[i] = '.';
        }
      }
      log_dev_every(4500 * US_PER_MS_INT, "FRAME_LAST_50_ASCII: %s", ascii_buf);
    }
 
    out = ascii_frame;
  } else {
    SET_ERRNO(ERROR_TERMINAL, "Per-client %s: Failed to convert image to ASCII", target_client_id);
    *out_size = 0;
  }
 
  START_TIMER("image_cleanup");
  if (composite) {
    // For single source, composite is a malloc-allocated copy, not from pool
    // Check alloc method to determine correct destroy function
    if (composite->alloc_method == IMAGE_ALLOC_POOL) {
      image_destroy_to_pool(composite);
    } else {
      image_destroy(composite);
    }
  }
  for (int i = 0; i < source_count; i++) {
    if (sources[i].image) {
      image_destroy_to_pool(sources[i].image);
    }
  }
  STOP_TIMER_AND_LOG(dev, 500000, "image_cleanup", "Image cleanup (composite + sources)");
 
  uint64_t frame_gen_end_ns = time_get_ns();
  uint64_t frame_gen_duration_ns = frame_gen_end_ns - frame_gen_start_ns;
  if (frame_gen_duration_ns > 10 * NS_PER_MS_INT) { // Log if > 10ms
    char duration_str[32];
    time_pretty((uint64_t)((double)frame_gen_duration_ns), -1, duration_str, sizeof(duration_str));
    log_warn_every(LOG_RATE_DEFAULT, "SLOW_FRAME_GENERATION: Client %s full frame gen took %s", target_client_id,
                   duration_str);
  }
 
  return out;
}

References image_t::alloc_method, atomic_cas_u64(), atomic_load_u64(), ERROR_INVALID_PARAM, ERROR_INVALID_STATE, ERROR_MEMORY, ERROR_TERMINAL, IMAGE_ALLOC_POOL, image_destroy(), image_destroy_to_pool(), image_new_copy(), log_dev_every, log_error, LOG_RATE_DEFAULT, LOG_RATE_SLOW, log_warn, log_warn_every, MAX_CLIENTS, NAMED_REGISTER_ATOMIC, NS_PER_MS, NS_PER_MS_INT, SAFE_FREE, SET_ERRNO, START_TIMER, STOP_TIMER_AND_LOG, time_get_ns(), time_pretty(), and US_PER_MS_INT.

Referenced by client_video_render_thread().

queue_audio_for_client()

int queue_audio_for_client	(	client_info_t *	client,
		const void *	audio_data,
		size_t	data_size
	)

Queue ASCII frame for delivery to specific client.

Packages a generated ASCII frame into a protocol packet and queues it for asynchronous delivery to the target client. This function handles all protocol details including headers, checksums, and metadata.

PACKET STRUCTURE CREATED:

• ascii_frame_packet_t header containing:
  • width, height: Client terminal dimensions
  • original_size: Uncompressed frame size
  • compressed_size: Compressed size (currently 0)
  • checksum: CRC32 of frame data for integrity
  • flags: Capability flags (color support, etc.)
• Frame data: Complete ASCII art string with ANSI codes

PROTOCOL INTEGRATION:

• Converts dimensions to network byte order
• Generates CRC32 checksum for error detection
• Sets appropriate capability flags based on client
• Uses PACKET_TYPE_ASCII_FRAME packet type

DELIVERY MECHANISM:

• Queues packet in client's video packet queue
• Send thread delivers packet asynchronously
• Queue overflow handling prevents memory exhaustion
• Error conditions are logged but don't affect other clients

MEMORY MANAGEMENT:

• Creates temporary packet buffer for header+data combination
• packet_queue_enqueue() copies data (safe to free temp buffer)
• No memory leaks on success or failure paths
• Buffer pool allocation not used here (temporary buffer)

ERROR HANDLING:

• Invalid parameters: Return -1, no side effects
• Memory allocation failure: Logged, return -1
• Queue overflow: Logged as debug (expected under load)
• Client shutdown: Graceful failure without error spam

Parameters

client	Target client for frame delivery
ascii_frame	Generated ASCII art string (null-terminated)
frame_size	Size of ASCII frame in bytes

Returns

0 on successful queuing, -1 on error

Note

Frame delivery happens asynchronously via send thread

Queue overflow drops frames to maintain real-time performance

Checksums enable client-side integrity verification

See also

packet_queue_enqueue() For underlying queue implementation

asciichat_crc32() For checksum generation

Queue audio data for delivery to specific client

Queues mixed audio data for delivery to a specific client. This is a simple wrapper around the packet queue system for audio delivery.

AUDIO PIPELINE INTEGRATION:

• Called by audio mixing threads after combining multiple client streams
• Audio data is already in final format (float samples, mixed)
• No additional processing required at this stage

PACKET DETAILS:

• Uses PACKET_TYPE_AUDIO_BATCH packet type
• Audio data is raw float samples bundled together
• Batch format reduces packet overhead ~32x
• Sample rate and format determined by audio system

DELIVERY CHARACTERISTICS:

• Higher priority than video packets (lower latency)
• Uses client's audio packet queue
• Send thread prioritizes audio over video
• Queue overflow drops oldest audio to maintain real-time

ERROR HANDLING:

• Invalid parameters: Return -1 immediately
• Queue overflow: Handled by packet queue (drops old data)
• Client shutdown: Graceful failure

Parameters

client	Target client for audio delivery
audio_data	Mixed audio samples (float format)
data_size	Size of audio data in bytes

Returns

0 on successful queuing, -1 on error

Note

Audio has priority over video in send thread

Real-time audio requires queue overflow handling

See also

packet_queue_enqueue() For underlying queue implementation

mixer.c For Audio mixing implementation

Definition at line 1315 of file stream.c.

                                                                                            {
  if (!client || !client->audio_queue || !audio_data || data_size == 0) {
    return -1;
  }
 
  return packet_queue_enqueue(client->audio_queue, PACKET_TYPE_AUDIO_BATCH, audio_data, data_size, 0, true);
}

References client_info::audio_queue, packet_queue_enqueue(), and PACKET_TYPE_AUDIO_BATCH.

Variable Documentation

g_should_exit

atomic_t g_should_exit

extern

Global application exit flag (shared across all modes)

Global shutdown flag referenced by lib/logging.c and lib/debug/lock.c In production builds, this is defined in main.c and set during shutdown. For tests, we stub it out as false (tests don't perform shutdown).

Definition at line 34 of file globals.c.

{0};

Referenced by client_audio_render_thread(), client_dispatch_thread(), client_receive_thread(), client_send_thread_func(), client_video_render_thread(), handle_image_frame_h265_packet(), handle_image_frame_packet(), server_main(), should_exit(), signal_exit(), stats_logger_thread(), and stop_client_render_threads().