Table of Contents

VAST.Common.Ext.iOS

The VAST.Common.Ext.iOS assembly provides Apple platform-specific functionality for VASTreaming applications, including hardware-accelerated video encoding/decoding via VideoToolbox, audio encoding/decoding via AudioToolbox, camera capture via AVFoundation, and video rendering via OpenGL, Metal and AVSampleBufferDisplayLayer.

Overview

Feature Technology Description
Video Decoding VideoToolbox Hardware-accelerated H.264/H.265/MJPEG decoding
Video Encoding VideoToolbox Hardware-accelerated H.264/H.265/VP9/MJPEG encoding
Audio Decoding AudioToolbox AAC/MP3/AC3/G.711 decoding with resampling
Audio Encoding AudioToolbox AAC/MP3/G.711 encoding with resampling
Video Capture AVFoundation Camera access with multi-camera support
Audio Capture AVFoundation Microphone and external audio input
Audio Playback AudioUnit Low-latency audio output
Video Rendering AVSampleBufferDisplayLayer, OpenGL Hardware-accelerated video display
Image Processing Metal, OpenGL GPU-accelerated image processing

Requirements

  • Minimum iOS: iOS 12.2
  • Recommended: iOS 14.5+ for low-latency encoding mode
  • Architecture: ARM64
  • Dependencies: VAST.Common

Hardware Video Decoding

VAST.Common.Ext.iOS uses Apple's VideoToolbox framework with VTDecompressionSession for hardware-accelerated video decoding. The decoder outputs either CMSampleBuffer for efficient rendering or raw pixel data for processing.

Supported Codecs

Codec Support Notes
H.264/AVC All devices Universal support
H.265/HEVC A9+ chip (iOS 11+) Device-dependent hardware support
MJPEG All devices Hardware-accelerated

Usage

Hardware decoding is automatically selected when available:

var parameters = new VAST.Media.DecoderParameters
{
    PreferHardwareAcceleration = true
};

var decoder = VAST.Media.DecoderFactory.Create(
    encodedMediaType,
    decodedMediaType,
    parameters);

Native Buffer Output

When decoding for display, the decoder can output CMSampleBuffer directly, enabling zero-copy rendering to AVSampleBufferDisplayLayer for optimal performance.

Hardware Video Encoding

VideoToolbox provides hardware-accelerated encoding via VTCompressionSession for streaming and recording applications.

Supported Codecs

Codec Support Typical Resolutions
H.264/AVC All devices Up to 4K on modern devices
H.265/HEVC A10+ chip Up to 4K on supported devices
VP9 Apple Silicon Limited device support
MJPEG All devices Hardware-accelerated

Low-Latency Mode

On iOS 14.5+, the encoder supports low-latency mode for real-time streaming applications with reduced encoding delay.

Usage

var encodedMediaType = new VAST.Common.MediaType
{
    ContentType = ContentType.Video,
    CodecId = Codec.H264,
    Bitrate = 4_000_000, // 4 Mbps
    Width = 1920,
    Height = 1080,
    Framerate = new VAST.Common.Rational(30),
};
encodedMediaType.Metadata.Add("KeyframeInterval", "30"); // 1 second

var encoder = VAST.Media.EncoderFactory.Create(
    rawMediaType,
    encodedMediaType,
    encoderParams);

Built-in Framerate Conversion

The encoder includes a built-in framerate converter to adjust input frame timing when the source framerate differs from the target encoding framerate.

Audio Decoding

VAST.Common.Ext.iOS uses AudioToolbox's AudioConverter for audio decoding with built-in resampling support.

Supported Codecs

Codec Support Notes
AAC All devices Hardware-accelerated
MP3 All devices Software decoding
AC-3 (Dolby Digital) All devices Software decoding
G.711 A-law All devices Telephony codec
G.711 μ-law All devices Telephony codec

Resampling

The decoder includes built-in sample rate and channel count conversion using AudioConverter's resampling capabilities.

Audio Encoding

VAST.Common.Ext.iOS uses AudioToolbox's AudioConverter for audio encoding with built-in resampling support.

Supported Codecs

Codec Support Notes
AAC All devices AAC-LC profile, software encoding
MP3 All devices Software encoding
G.711 A-law All devices Telephony codec
G.711 μ-law All devices Telephony codec

Built-in Resampling

The encoder includes automatic sample rate, channel count, and sample format conversion. Input audio is automatically resampled to match the target encoding format requirements. Supported input sample formats include:

  • S16 (16-bit signed integer, interleaved)
  • S16P (16-bit signed integer, planar)
  • S32 (32-bit signed integer, interleaved)
  • S32P (32-bit signed integer, planar)
  • FLT (32-bit float, interleaved)
  • FLTP (32-bit float, planar)

Usage

var encodedMediaType = new VAST.Common.MediaType
{
    ContentType = ContentType.Audio,
    CodecId = Codec.AAC,
    Bitrate = 128_000, // 128 kbps
    SampleRate = 44100,
    Channels = 2,
};

var encoder = VAST.Media.EncoderFactory.Create(
    rawAudioMediaType,
    encodedMediaType,
    encoderParams);

Video Capture

AVFoundation provides comprehensive camera access with support for advanced features on compatible devices.

Permissions

Add camera usage description to your Info.plist:

<key>NSCameraUsageDescription</key>
<string>This app requires camera access for video streaming</string>

Request authorization before accessing the camera:

var status = AVCaptureDevice.GetAuthorizationStatus(AVAuthorizationMediaType.Video);
if (status == AVAuthorizationStatus.NotDetermined)
{
    await AVCaptureDevice.RequestAccessForMediaTypeAsync(AVAuthorizationMediaType.Video);
}

Enumerating Cameras

var devices = await VAST.Capture.VideoDeviceEnumerator.Enumerate();

foreach (var device in devices)
{
    Console.WriteLine($"Camera: {device.Name}");
    Console.WriteLine($"  ID: {device.DeviceId}");

    foreach (var captureMode in device.CaptureModes)
    {
        Console.WriteLine($"  Capture Mode: {captureMode}");
    }
}

Capturing Video

var captureMode = selectedDevice.CaptureModes[selectedDevice.DefaultCaptureMode];
var videoSource = VAST.Media.SourceFactory.CreateVideoCapture(selectedDevice.DeviceId, captureMode);

videoSource.StateChanged += (sender, e) =>
{
    switch (e)
    {
        case VAST.Media.MediaState.Opened:
            videoSource.Start();
            break;
    }
};

videoSource.NewSample += (sender, e) =>
{
    // Process captured frame
    ProcessFrame(e.Sample);
};

videoSource.Open();

Multi-Camera Support

On supported iOS devices (iPhone XS and later), the library supports simultaneous capture from multiple cameras using AVCaptureMultiCamSession.

Camera Features

The Camera API supports advanced features on compatible devices:

  • Auto-focus and manual focus control
  • Auto-exposure and manual exposure
  • White balance adjustment
  • Optical and digital zoom
  • Torch/flash control

These functions are exposed via the ICameraControl interface, available from the CameraControl property after capture starts.

Audio Capture

Access device microphones using AVFoundation's audio capture.

Permissions

Add microphone usage description to your Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>This app requires microphone access for audio streaming</string>

Capturing Audio

var audioSource = VAST.Media.SourceFactory.CreateAudioCapture(selectedAudioDevice.DeviceId);

audioSource.StateChanged += (sender, e) =>
{
    switch (e)
    {
        case VAST.Media.MediaState.Opened:
            audioSource.Start();
            break;
    }
};

audioSource.NewSample += (sender, e) =>
{
    // Process captured audio
    ProcessAudio(e.Sample);
};

audioSource.Open();

Audio Playback

VAST.Common.Ext.iOS provides low-latency audio rendering through AudioUnit and RemoteIO.

Volume and Balance Control

The audio renderer supports volume (0.0 to 1.0) control.

Video Rendering

VAST.Common.Ext.iOS provides video rendering via AVSampleBufferDisplayLayer for optimal performance with hardware-decoded video.

AVSampleBufferDisplayLayer

When using hardware decoding, the renderer can accept CMSampleBuffer directly from the decoder for zero-copy display, minimizing memory bandwidth and CPU usage.

Metal Support

The library includes Metal-based image processing for GPU-accelerated format conversion and effects processing via the MetalImageProcessor and MetalRenderer components.

OpenGL ES Support

For efficient presentation of OpenGL image processor output, OpenGL ES rendering is also supported via the IosOpenGlRenderer component (iOS only).

.NET MAUI Integration

For .NET MAUI applications, use the VAST.UI library with platform-specific handlers:

// In your MAUI page
var mediaPlayer = new VAST.Controls.MediaPlayerControl();
// TODO: add player control to your page

mediaPlayer.Source = "rtsp://...";
mediaPlayer.Play();

Troubleshooting

Common Issues

Issue Cause Solution
Choppy playback Thermal throttling Reduce resolution or frame rate
Camera access denied Missing permission Check Info.plist and request authorization
Encoder fails to start Unsupported codec Check device capabilities, use H.264 fallback
No audio output Audio session inactive Activate AVAudioSession
Low-latency mode unavailable Older iOS version Requires iOS 14.5+

Memory Management

When working with CMSampleBuffer and CVPixelBuffer objects, ensure proper disposal to avoid memory leaks:

using (var sample = decoder.Decode(encodedPacket))
{
    // Process the sample
    renderer.Process(sample);
}

See Also