Key Concepts and Definitions

Learn the principles of building multithreaded filterchains

Pipelines, streams, frames

A media streaming pipeline consists of independently running threads that produce and consume frames.

A stream consists of a continuous stream of frames.

Frames can be almost anything:

  • Encoded images (say, H264 or H265 video)

  • Decoded images, i.e. just bitmaps in various formats (interleaved, planar)

  • Encoded or decoded audio (i.e., mp3 or wav, etc.)

  • Metadata about the frames creating a stream

  • Messages / signals

In Limef we emphasize this universal aspect of frames: they can be metadata, commands, almost any object that has an id number and a timestamp.

The available frame types are identified by the Limef::frame::FrameClass enumeration. Each frame class corresponds to a subclass of Limef::frame::Frame:

FrameClass

Class

FrameClass::Packet

Limef::frame::PacketFrame — encoded video/audio packets

FrameClass::Decoded

Limef::frame::DecodedFrame — raw bitmaps or PCM audio

FrameClass::Codec

Limef::frame::CodecFrame — codec parameters

FrameClass::Signal

Limef::frame::SignalFrame — thread control commands

FrameClass::Stream

Limef::frame::StreamFrame — stream metadata for encoders

FrameClass::SDP

Limef::frame::SDPFrame — SDP for RTSP

FrameClass::RTPPacket

Limef::frame::RTPPacketFrame — RTP/RTCP packets

FrameClass::Tensor

Limef::frame::TensorFrame — N-plane tensor, CPU or GPU

Threads

A thread consumes certain kind of frames and produces another and/or different kind of frames.

Producer threads only produce, while terminal threads only consume.

An example of a producing thread is Limef::thread::MediaFileThread, which reads a media file and pushes Limef::frame::PacketFrames downstream.

A terminal thread is for example the Limef::rtsp::RTSPServerThread that serves the media stream over the internet.

Each thread exposes an input and output framefilter via Limef::thread::Thread::getInput() and Limef::thread::Thread::getOutput(). These are the connection points where threads plug into the filterchain.

Framefilters

A framefilter takes an input frame and produces an output frame. The input and output frames can be of the same or of a different frame type.

An example of the simplest framefilter is the Limef::ff::DumpFrameFilter that dumps information about the frame into the terminal.

Framefilters can be chained with the cc method (“connect chain”):

InfoFrameFilter     info("info");
DumpFrameFilter     dump("dump");
DecodingFrameFilter decode("decode");

// Chain them: info -> dump -> decode
info.cc(dump).cc(decode);

// Start the cascade with a single go() call:
info.go(frame);

The cascade calls go() on each filter in turn. Each filter may transform the frame, then calls pass() to forward it downstream.

For a one-to-many split, use Limef::ff::SplitFrameFilter:

SplitFrameFilter split("split");
DumpFrameFilter  branch1("branch1");
DumpFrameFilter  branch2("branch2");

split.cc(branch1);
split.cc(branch2);        // or: split.cm({&branch1, &branch2})
// later: split.dc(branch1);  // disconnect one branch at runtime

Limef::ff::SplitFrameFilter is thread-safe — you can call cc / dc after threads have started.

Filterchains

Upstream and downstream

upstream      downstream
ff1 -> ff2 -> ff3

When ff1.go(frame) is called, these rules are strictly followed:

  • Frames received from upstream are never modified by downstream framefilters — they can be copied, read, or passed on, but not mutated

  • A framefilter may own internal frames which it passes downstream, but their ownership stays with that framefilter

Connecting threads

A thread starts a filterchain at its output and the chain ends at the next thread’s input:

ProducerThread.getOutput() ──► ff1 ──► ff2 ──► ConsumerThread.getInput()

Use Limef::thread::Thread::getOutput() and Limef::thread::Thread::getInput() to wire threads into the chain.

Almost any kind of information flows in a filterchain. Data frames (Limef::frame::PacketFrame, Limef::frame::DecodedFrame, etc.) carry media payload, while Limef::frame::SignalFrame carries control commands (start, stop, flush, parameter changes) — all through the same go() call, no separate control bus needed.

Diagram notation

Diagrams follow these conventions:

  • Thread — rectangle [Name], abbreviated ...TR. Incoming edge uses an arrow (-->) to mark the thread boundary: the frame is copied into a FrameFifo before the thread processes it.

  • FrameFilter — rounded rectangle (Name), abbreviated ...FF. Incoming edge uses a plain line (---) to show synchronous, same-chain forwarding — no copy, no boundary.

  • Block — subroutine box [[Name]]. Always uses an arrow on incoming edges since blocks always contain internal threads.

        flowchart LR
    ff1(UpstreamFF) --- ff2(AnotherFF) --> thr[ConsumerTR]

    classDef thread fill:#4a90d9,stroke:#2c5f8a,color:#fff
    classDef ff     fill:#5ba85a,stroke:#3d6e3d,color:#fff
    class thr thread
    class ff1,ff2 ff

FF→FF: no arrow (same chain). FF→Thread: arrow (FrameFifo boundary).

YAML declarations

Pipelines can be described in YAML. Inline nesting reflects the filterchain topology directly — each node’s downstream is nested under its connect: key:

source:              # top-level named node
    class: MediaFileThread
    connect:
        split:       # inline anonymous node, nested under source
            class: SplitFrameFilter
            connect:
                decode:
                    class: DecodingFrameFilter
                    connect:
                        analyzer:
                            class: DumpFrameFilter
                muxer:       # second branch of the split
                    class: MuxerFrameFilter

connect: reference rules:

  • bare name → top-level named node: connect: encoderblock

  • ./name → sibling member within the same block: connect: ./cuda2dec

  • ../name → sibling of the parent (one level up): connect: ../output (used from inside switch branches)

A complete example

Encoded frames (H265) come from a file-reading thread. The stream is forked: one branch is decoded for analysis, the other goes to a muxer.

        flowchart TD
    src[MediaFileTR]
    split(SplitFF)
    decode(DecodingFF)
    analyze(DumpFF)
    mux(MuxerFF)

    src --- split
    split --- decode
    decode --- analyze
    split --- mux

    classDef thread fill:#4a90d9,stroke:#2c5f8a,color:#fff
    classDef ff     fill:#5ba85a,stroke:#3d6e3d,color:#fff
    class src thread
    class split,decode,analyze,mux ff

In C++:

DumpFrameFilter     analyzer("analyzer");
MuxerFrameFilter    muxer("muxer", mux_ctx);
DecodingFrameFilter decode("decode");
SplitFrameFilter    split("split");

decode.cc(analyzer);
split.cc(decode);
split.cc(muxer);

MediaFileThread source("source", media_ctx);
source.getOutput().cc(split);

source.start();

Python Guidelines

Limef has Python bindings, so filter chains can be built entirely from Python. The same pipeline as above looks like this:

import limef

analyze = limef.DumpFrameFilter("analyzer")
muxer   = limef.MuxerFrameFilter("muxer", mux_ctx)
decode  = limef.DecodingFrameFilter("decode")
split   = limef.SplitFrameFilter("split")

decode.cc(analyze)
split.cc(decode)
split.cc(muxer)

source = limef.MediaFileThread("source", media_ctx)
source.getOutput().cc(split)

source.start()

However, some rules must be followed when developing with python.

Warning

cc() stores raw C++ pointers — Python’s garbage collector has no visibility into these references. Any Python object whose refcount drops to zero will be destroyed even if a C++ chain still points at it, causing a crash or a “pure virtual method called” abort

Important

FrameFilter or Thread object that participates in a chain must be kept alive by a Python variable for the entire duration of the pipeline run.

The safe pattern when a pipeline is assembled inside a helper function is to own all objects as instance attributes of a class:

class MyPipeline:
    def __init__(self, media_ctx, mux_ctx):
        self._analyze = limef.DumpFrameFilter("analyzer")
        self._muxer   = limef.MuxerFrameFilter("muxer", mux_ctx)
        self._decode  = limef.DecodingFrameFilter("decode")
        self._split   = limef.SplitFrameFilter("split")
        self._source  = limef.MediaFileThread("source", media_ctx)

        self._decode.cc(self._analyze)
        self._split.cc(self._decode)
        self._split.cc(self._muxer)
        self._source.getOutput().cc(self._split)

        self.chain = self._split   # expose tail for downstream wiring

    def start(self): self._source.start()
    def stop(self):  self._source.stop()

All objects live exactly as long as the MyPipeline instance.

Avoid these:

  • Inline temporaries — ff1.cc(limef.SomeFilter()).cc(ff2): the filter has no variable holding it and is GC’d immediately after the statement.

  • Only returning the tail — a helper that builds A → B → C and returns only C silently destroys A and B when the function returns.

  • Reassigning chain — chain = chain.cc(x) drops the reference to whatever chain pointed to before; if that was the only reference, the object is GC’d.

  • Context temporaries — SomeThread(SomeCtx()) is only safe when the C++ constructor copies the context by value