Table of Contents

VAST.Image Library

The VAST.Image library provides image processing, MJPEG streaming, and video mixing capabilities for VASTreaming applications. It supports MJPEG HTTP sources/sinks, static image overlays, real-time video compositing with multiple layers, and GPU-accelerated image processing.

Overview

Feature Description
MJPEG Server Serve MJPEG streams over HTTP to clients
MJPEG Client Receive MJPEG streams from cameras and servers
Image Source Static images and text overlays as media sources
Mixing Source Combine multiple video/audio sources with compositing
Video Effects Opacity, crop, rotation, blur, chroma key, fade animations
Audio Mixing Mix multiple audio tracks with volume control

Requirements

MJPEG Server

Note

MjpegServer is not available as a standalone server. It operates as part of the VAST.Network.StreamingServer infrastructure and is instantiated by the StreamingServer automatically when enabled.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                 VAST.Network.StreamingServer                │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │               VAST.Network.PublishingPoint              │ │
│ │                    /mjpeg/stream1                       │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │                  Live Media Source                  │ │ │
│ │ │           (Camera, File, Network Stream)            │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ │                            ↓                            │ │
│ │ ┌─────────────────────────────────────────────────────┐ │ │
│ │ │             MjpegSink (internal class)              │ │ │
│ │ │      (Receives MJPEG frames, sends to clients)      │ │ │
│ │ └─────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
│                             ↓                               │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │              MjpegServer (internal class)               │ │
│ │              (client context management)                │ │
│ └─────────────────────────────────────────────────────────┘ │
│                             ↓                               │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │                       HttpServer                        │ │
│ │            (HTTP request/response management,           │ │
│ │    based on System.Net.HttpListener or ASP.NET Core)    │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
                              ↓
             Multiple HTTP Clients (browsers, players)

Enabling MJPEG Server

var server = new VAST.Network.StreamingServer();

// Enable MJPEG over HTTP
server.EnableMjpeg = true;
server.MjpegServerParameters = new VAST.Image.JPEG.MjpegServerParameters
{
    MjpegPath = "/mjpeg/"
};

server.Start();

Client Access

Clients connect via HTTP GET requests:

http://server:port/mjpeg/stream-name

The server responds with a continuous MJPEG stream using multipart/x-mixed-replace content type.

MJPEG HTTP Source

The MjpegHttpSource class receives MJPEG video streams over HTTP from cameras and streaming servers.

Supported Formats

Format Content Type Description
Multipart multipart/x-mixed-replace Standard MJPEG with boundary separators
Continuous image/jpeg Borderless stream of JPEG images

Usage

var source = new VAST.Image.JPEG.MjpegHttpSource();
source.Uri = "http://camera.local/mjpeg/stream";

source.NewStream += (sender, e) =>
{
    Console.WriteLine($"Stream: {e.MediaType}");
    Console.WriteLine($"Resolution: {e.MediaType.Width}x{e.MediaType.Height}");
};

source.NewSample += (sender, e) =>
{
    // Process MJPEG frame
    ProcessFrame(e.Sample);
};

source.StateChanged += (sender, state) =>
{
    if (state == MediaState.Opened)
    {
        source.Start();
    }
};

source.Open();

Authentication

For cameras requiring authentication:

source.Uri = "http://username:password@camera.local/mjpeg/stream";
Note

If the username or password contains special characters, they must be URL-encoded.

Connection Timeout

The source automatically reconnects if no data is received within 60 seconds.

Image Source

The ImageSource class provides static images and text overlays as media sources for use with the mixing source or as standalone video sources.

ImageSource utilizes SkiaSharp in .NET Core or System.Drawing in .NET Framework projects.

Loading Images

var source = new VAST.Image.ImageSource();

// Load from file stream
using (var stream = File.OpenRead("overlay.png"))
{
    source.SetImage(stream);
}

source.StateChanged += (sender, state) =>
{
    if (state == MediaState.Opened)
    {
        source.Start();
    }
};

source.Open();

Async Loading

await source.SetImageAsync(imageStream);

Text Overlays

Render text as an image with full styling support:

var overlay = new VAST.Image.OverlayText
{
    Text = "Live Stream",
    Location = new VAST.Common.Rect(0, 0, 400, 100),
    FontFamily = "Arial",
    FontSize = 24,
    FontBold = true,
    FontColor = "#FFFFFF",
    BackgroundColor = "#80000000",
    OutlineWidth = 2,
    OutlineColor = "#000000",
    HorizontalAlignment = VAST.Image.HorizontalAlignment.Center,
    VerticalAlignment = VAST.Image.VerticalAlignment.Center
};

source.SetText(overlay);

Supported Image Formats

Format Support
JPEG Full
PNG Full (with transparency)
BMP Full
GIF First frame only
WebP .NET Core only

Mixing Source

The MixingSource class combines multiple video and audio sources into a single output stream with real-time compositing, layer management, and audio mixing.

Operating Modes

MixingSource operates in one of two modes based on the AllowVideoProcessing setting:

Mode AllowVideoProcessing Description
Compositing true GPU-accelerated compositing with layers, effects, and multiple visible sources
Switching false Single source displayed at a time, sources can be switched between

Compositing Mode (AllowVideoProcessing = true)

When video processing is enabled, the mixing source uses GPU-accelerated compositing to combine multiple sources:

  • Multiple sources can be visible simultaneously using layers
  • Video effects (opacity, crop, rotation, blur, chroma key) are available
  • Fade animations between layer states
  • Higher resource usage

Switching Mode (AllowVideoProcessing = false)

When video processing is disabled, the mixing source operates in a simpler switching mode:

  • Only one source is displayed at a time
  • VideoMixingType must be set to Single
  • Sources are switched between using SourceIndex
  • Fallback source support via FallbackSourceIndex
  • All sources are converted to the output resolution(s)
  • Lower resource usage, suitable for simple source switching scenarios

Basic Usage

var mixingSource = new VAST.Image.Mixing.MixingSource();

// Configure the mixing scene
var descriptor = new VAST.Image.Mixing.Descriptor
{
    AllowVideoProcessing = true, // enable video compositing
    Sources = new List<VAST.Image.Mixing.Source>
    {
        new VAST.Image.Mixing.Source
        {
            Uri = "rtsp://camera1.local/stream",
        },
        new VAST.Image.Mixing.Source
        {
            Uri = "rtsp://camera2.local/stream",
        }
    },
    Processing = new VAST.Image.Mixing.Processing
    {
        VideoProcessing = new VAST.Image.Mixing.VideoProcessing
        {
            Mixing = new VAST.Image.Mixing.VideoMixing
            {
                Type = Image.Mixing.VideoMixingType.All,
                Layers = new List<VAST.Image.Mixing.Layer>
                {
                    new VAST.Image.Mixing.Layer
                    {
                        Sources = new List<int> { 0 },
                        Location = new VAST.Common.Rect(0, 0, 1920, 1080)
                    },
                    new VAST.Image.Mixing.Layer
                    {
                        Sources = new List<int> { 1 },
                        Location = new VAST.Common.Rect(1500, 50, 1870, 330),
                        Opacity = 0.9f
                    }
                },
            },
            Tracks = new List<VAST.Image.Mixing.VideoTrack>
            {
                new VAST.Image.Mixing.VideoTrack
                {
                    Index = 0,
                    Width = 1280,
                    Height = 720,
                    Framerate = new VAST.Common.Rational(30),
                    KeyframeInterval = 30,
                    Bitrate = 3000000,
                    Codec = VAST.Common.Codec.H264,
                    Profile = 66
                }
            }
        },
        AudioProcessing = new VAST.Image.Mixing.AudioProcessing
        {
            Mixing = new VAST.Image.Mixing.AudioMixing
            {
                Type = VAST.Image.Mixing.AudioMixingType.Single,
                SourceIndex = 0,
            },
            Tracks = new List<VAST.Image.Mixing.AudioTrack>
            {
                new VAST.Image.Mixing.AudioTrack
                {
                    Index = 1,
                    SampleRate = 44100,
                    Channels = 2,
                    Bitrate = 128000,
                    Codec = VAST.Common.Codec.AAC,
                }
            }
        }
    }
};

mixingSource.Update(descriptor);

mixingSource.NewStream += (sender, e) =>
{
    Console.WriteLine($"Output stream: {e.MediaType}");
};

mixingSource.NewSample += (sender, e) =>
{
    // Process mixed output
    ProcessSample(e.Sample);
};

mixingSource.StateChanged += (sender, state) =>
{
    if (state == MediaState.Opened)
    {
        mixingSource.Start();
    }
};

mixingSource.Open();

Mixing Configuration Levels

Video and audio mixing settings can be configured at two levels:

Level Classes Description
Common VideoProcessing, AudioProcessing Shared mixing settings applied to all output tracks
Track-specific VideoTrack, AudioTrack Per-track mixing settings that override common settings

When mixing is defined at the processing level (common), all output tracks share the same mixing configuration. When mixing is defined at the track level, each track can have its own independent mixing settings.

var processing = new VAST.Image.Mixing.Processing
{
    VideoProcessing = new VAST.Image.Mixing.VideoProcessing
    {
        // Common mixing for all video tracks
        Mixing = new VAST.Image.Mixing.VideoMixing { ... },

        Tracks = new List<VAST.Image.Mixing.VideoTrack>
        {
            new VAST.Image.Mixing.VideoTrack
            {
                Index = 0,
                // Track-specific mixing (overrides common)
                Mixing = new VAST.Image.Mixing.VideoMixing { ... }
            }
        }
    }
};

Sources

Source position in the Sources list defines source index.

Using Existing Media Sources

Instead of specifying a Uri, you can pass an existing IMediaSource object via the MediaSource property. This is useful when you want to use a capture device, VirtualNetworkSource, or any other pre-configured source:

var descriptor = new VAST.Image.Mixing.Descriptor
{
    Sources = new List<VAST.Image.Mixing.Source>
    {
        new VAST.Image.Mixing.Source
        {
            MediaSource = captureSource
        }
    }
};

Source Formats

The Format property specifies how to interpret the source content:

Format Description
text Text overlay rendered from Content property containing text string using Decoration settings
image In-memory image data (via Content property with image bytes)
imagefile Static image loaded from file path or uri (PNG, JPG, BMP, GIF, TIFF)
mediafile Media file loaded from file path or uri (MP4, MOV, TS, MP3, WAV, playlists)

Format is auto-detected from file extension when not explicitly specified.

Video Layers

Layers are composited in z-order (first layer is background, last layer is foreground).

Layer Properties

Property Description
Sources List of source indices (their indices) to display
Layout Manual (arranged by user) or Auto (arranged automatically)
Location Position and size on output canvas
Crop Crop source video (left, top, right, bottom margins)
Rotation Rotation angle in degrees (negative = horizontal mirror)
Opacity 0.0 (transparent) to 1.0 (opaque)
Stretch How content fills the layer area
FillColor The fill color for empty areas

Stretch Modes

Mode Description
Original Keep original image size, clip if outside boundaries
Preserve Scale to fit preserving aspect ratio, whole image visible (letterbox/pillarbox)
Fill Stretch to fill boundaries without preserving aspect ratio
Zoom Scale to fill preserving aspect ratio, clip edges if needed

Video Effects

Apply real-time effects to layers:

var layer = new VAST.Image.Mixing.Layer
{
    Sources = new List<int> { 0 },
    Location = new VAST.Common.Rect(0, 0, 1920, 1080),

    // Brightness/Contrast (-10 to 10)
    BrightnessAdjustment = 1.0f,
    ContrastAdjustment = -2.0f,

    // Blur effect
    BlurSize = 5,

    // Chroma key (green screen)
    ChromaKeyColor = "#00FF00",
    ChromaKeyThreshold = 0.1f,
    ChromaKeySmoothing = 0.05f
};

Fade Animations

Smooth transitions when layers appear or disappear. Specify either FadeInMsec or FadeOutMsec, not both simultaneously:

// Fade in a new layer
var layer = new VAST.Image.Mixing.Layer
{
    Sources = new List<int> { 1 },
    Location = new VAST.Common.Rect(100, 100, 500, 400),
    FadeInMsec = 500   // 500ms fade in
};

mixingSource.Update(descriptor);

After the animation completes, update the descriptor again with the fade property removed to finalize the transition:

// After fade completes, remove FadeInMsec
layer.FadeInMsec = null;
mixingSource.Update(descriptor);

Audio Mixing

Mix audio from multiple sources with individual volume control:

var descriptor = new VAST.Image.Mixing.Descriptor
{
    Processing = new VAST.Image.Mixing.Processing
    {
        AudioProcessing = new VAST.Image.Mixing.AudioProcessing
        {
            Mixing = new VAST.Image.Mixing.AudioMixing
            {
                Type = VAST.Image.Mixing.AudioMixingType.All,
                Filters = new List<VAST.Image.Mixing.Filter>
                {
                    new VAST.Image.Mixing.Filter
                    {
                        Sources = new List<int> { 0 },
                        Volume = 1.0f // Full volume
                    },
                    new VAST.Image.Mixing.Filter
                    {
                        Sources = new List<int> { 1 },
                        Volume = 0.3f // 30% volume (background)
                    }
                }
            },
        }
    }
};

Output Tracks

MixingSource supports multiple output video and audio tracks, enabling scenarios like adaptive bitrate streaming or multiple audio languages.

Important

VideoTrack.Index and AudioTrack.Index properties must form a consecutive sequence starting from 0 (e.g., 0, 1, 2, ...).

Video Track Properties

Property Description
Index Track index (must be consecutive starting from 0)
Width Output width in pixels (0 = use source width)
Height Output height in pixels (0 = use source height)
Framerate Output framerate (null = use source framerate)
Bitrate Bitrate in bits per second
Codec Video codec (H264, H265, etc.)
KeyframeInterval GOP size in frames
Profile Encoder profile
Mixing Track-specific mixing settings

Audio Track Properties

Property Description
Index Track index (must be consecutive starting from 0)
SampleRate Sample rate in Hz (0 = use source rate)
Channels Number of channels (0 = use source channels)
Bitrate Bitrate in bits per second
Codec Audio codec (AAC, MP3, etc.)
Mixing Track-specific mixing settings

Multiple Output Tracks Example

var processing = new VAST.Image.Mixing.Processing
{
    VideoProcessing = new VAST.Image.Mixing.VideoProcessing
    {
        Tracks = new List<VAST.Image.Mixing.VideoTrack>
        {
            new VAST.Image.Mixing.VideoTrack
            {
                Index = 0,
                Width = 1920, Height = 1080,
                Bitrate = 5000000, Codec = Codec.H264
            },
            new VAST.Image.Mixing.VideoTrack
            {
                Index = 1,
                Width = 1280, Height = 720,
                Bitrate = 2500000, Codec = Codec.H264
            },
            new VAST.Image.Mixing.VideoTrack
            {
                Index = 2,
                Width = 640, Height = 360,
                Bitrate = 800000, Codec = Codec.H264
            }
        }
    },
    AudioProcessing = new VAST.Image.Mixing.AudioProcessing
    {
        Tracks = new List<VAST.Image.Mixing.AudioTrack>
        {
            new VAST.Image.Mixing.AudioTrack
            {
                Index = 3,
                SampleRate = 48000, Channels = 2,
                Bitrate = 128000, Codec = Codec.AAC
            }
        }
    }
};

Audio Level Monitoring

Monitor real-time audio levels from input sources:

mixingSource.AudioNotification += (sender, e) =>
{
    Console.WriteLine($"Source {e.SourceUri}: Peak={e.PeakAudioLevel:P0}, Avg={e.AverageAudioLevel:P0}");
};

Dynamic Scene Updates

Update the mixing scene at runtime:

// Keep current descriptor
var currentDescriptor = new ...

// Modify and apply
currentDescriptor.Processing.VideoProcessing.Mixing.Layers[1].Opacity = 0.5f;
mixingSource.Update(currentDescriptor);

Fallback Sources

Configure fallback sources for automatic failover when video compositing is disabled. Use FallbackSourceIndex to specify which source to display when the primary source becomes unavailable:

var descriptor = new VAST.Image.Mixing.Descriptor
{
    AllowVideoProcessing = false,  // Required for fallback to work
    Sources = new List<VAST.Image.Mixing.Source>
    {
        new VAST.Image.Mixing.Source { Uri = "rtsp://primary-camera/stream" },
        new VAST.Image.Mixing.Source { Uri = "fallback.mp4" }  // Local file recommended
    },
    Processing = new VAST.Image.Mixing.Processing
    {
        VideoProcessing = new VAST.Image.Mixing.VideoProcessing
        {
            Mixing = new VAST.Image.Mixing.VideoMixing
            {
                Type = VAST.Image.Mixing.VideoMixingType.Single,
                SourceIndex = 0,           // Primary source
                FallbackSourceIndex = 1    // Fallback when primary unavailable
            }
        }
    }
};
Note

Fallback source switching is only available when AllowVideoProcessing is false and VideoMixingType is Single. It is recommended to use a locally stored image or video file as the fallback source to ensure availability.

Performance Options

Option Description
AllowVideoProcessing Enable/disable GPU compositing
ForceVideoResampling Downscale inputs to output resolution early
EnableSmoothOutputBuffer Buffer output for consistent frame delivery
EnableZeroLatency Minimize processing delay by sacrificing precise synchronization and smoothness 
var descriptor = new VAST.Image.Mixing.Descriptor
{
    AllowVideoProcessing = true,
    ForceVideoResampling = true,  // Recommended for many sources
    EnableSmoothOutputBuffer = true
};

Preview Output

The mixed video output can be rendered to a video renderer for preview purposes using the Renderer property:

// Assign a video renderer for preview
mixingSource.Renderer = videoRenderer;

The renderer displays the mixed output in addition to outputting samples via the NewSample event.

By default, the renderer is cleared (filled with black) when the mixing source stops. To keep the last rendered frame visible after stopping, set CleanupOnStop to false:

mixingSource.CleanupOnStop = false;

Image Processor Parameters

Configure the image processing backend:

var parameters = new VAST.Image.ImageProcessorParameters
{
    ImageFramework = VAST.Image.ImageFramework.OpenGL
};

var mixingSource = new VAST.Image.Mixing.MixingSource(parameters);

Available Frameworks

Framework Description
Auto Automatically detected image framework
Builtin Default image processing framework of current OS
OpenGL GPU-accelerated compositing
ImageSharp Software compositing

Troubleshooting

Common Issues

Issue Cause Solution
MJPEG not playing Wrong content type Verify camera sends multipart/x-mixed-replace
Black output No sources connected Check source URIs and network
High CPU usage Software rendering Enable GPU image processor
Audio out of sync Mismatched sample rates Ensure consistent audio format
Layer not visible Wrong z-order Check layer order (last = top)

Debugging

Enable debug logging to diagnose issues:

VAST.Common.Log.Level = VAST.Common.Log.LogLevel.Debug;

See Also