Messaging

Message containers, subscriptions, indexing and record/replay. More...

Detailed Description

Message containers, subscriptions, indexing and record/replay.

Everything that flows between Psyclone components is a message. DataMessage carries user payloads, ControlMessage carries internal control commands, Subscriptions.h defines the trigger/retrieve/post specifications, and MessageIndex / MessagePlayer provide in-memory indexing and record/replay.

Why these objects exist

A Psyclone system is a mesh of components living in different threads, processes and machines. They cannot share pointers or C++ objects directly, so every piece of information that crosses a boundary — sensor readings, commands, results, log records — is packed into a DataMessage — a single flat block of memory containing a fixed header (type, sender, receiver, timestamps, TTL, priority) followed by any number of self-describing typed entries (strings, integers, floats, times, raw binary, attached sub-messages, and arrays/maps of these) addressed by string key.

Why a custom container instead of JSON/protobuf/std::map

Because the in-memory layout IS the wire format, a DataMessage needs no serialisation step at any boundary: the same bytes are memcpy'd into a shared-memory queue, written to a socket by the Networking layer, or saved to disk for replay. There is no schema compiler, no external dependency, and behaviour is identical on Windows, Linux and macOS. Entries are self-describing, so readers and writers do not need to agree on a schema at compile time — a component can read the two fields it cares about and forward the rest untouched. Format evolution is handled explicitly via CURRENTDATAMESSAGEVERSION and DataMessage::ConvertDataFromOlderMessageFormat().

How the pieces work together

A producer builds a DataMessage and posts it; subscriptions (Subscriptions.h) decide which components receive it; MessageIndex can keep a queryable in-memory index of recent messages, and MessagePlayer can record a stream to disk and replay it later at original or accelerated speed.

// Producer: build a message and fill in typed fields
DataMessage* msg = new DataMessage(myType, myComponentID); // PsyType + sender id
msg->setString("camera", "front-left");
msg->setInt("width", 1920);
msg->setDouble("exposure", 0.0125);
msg->setTime("captured", GetTimeNow());
msg->setData("pixels", pixelBuffer, pixelBytes);   // raw binary block
 
// The flat block can now cross any boundary without serialisation,
// e.g. via a shared-memory queue (see the Memory group):
MessageQueue::AddMessageToQueue(qid, msg);          // copied into shared memory
delete msg;                                         // caller keeps ownership
 
// Consumer (possibly another process): read the fields back
DataMessage* in = MessageQueue::WaitForMessageQueue(qid, 100); // 100 ms wait
if (in) {
    const char* cam = in->getString("camera");      // pointer into the message
    int64 w         = in->getInt("width");
    uint64 when     = in->getTime("captured");
    uint32 n; const char* px = in->getData("pixels", n);
    // ... use values before deleting the message ...
    delete in;
}

Note

Pointers returned by getString()/getData() point into the message's own memory block — they are valid only while the message is alive and unmodified; copy the data out if you need it longer.

Warning

The header structs in DataMessage.h are a de-facto wire/shared- memory ABI. Never change their size or field order without bumping CURRENTDATAMESSAGEVERSION and providing a converter, and always rebuild all binaries together (see the Shared-Memory Subsystem group for the same rule on shared-memory structs).

See also

DataMessage, ControlMessage, MessageIndex, MessagePlayer, Requests, Shared-Memory Subsystem

Files
file	ControlMessage.h
	Lightweight internal control/command message used by the Psyclone runtime.
file	DataMessage.h
	The binary DataMessage container — the central data-exchange object of Psyclone/CMSDK.
file	MessageIndex.h
	In-memory, multi-key index over DataMessages with TTL-based expiry — the storage engine behind whiteboards.
file	MessagePlayer.h
	Record and replay of DataMessage streams to/from disk.
file	ObjectIDs.h
	Object type ids used to tag and verify every binary structure in CMSDK memory.
file	Subscriptions.h
	Subscription, trigger, retrieval, query, post and signal specifications — the declarative wiring of a Psyclone system.
file	Types.h
	Core fixed-width scalar typedefs and the PsyType / PsyContext hierarchical identifiers.
file	ControlMessage.cpp
	Implementation of the ControlMessage wrapper and its self test.
file	DataMessage.cpp
	Implementation of DataMessage: construction, flat-block entry management, format conversion and (de)serialisation to JSON/XML/CSV.
file	MessageIndex.cpp
	Implementation of the multi-key DataMessage index: per-key multimaps, EOL-driven eviction and range queries.
file	MessagePlayer.cpp
	Implementation of DataMessage recording and timed replay, including index/metadata files and CSV export.
file	Subscriptions.cpp
	Implementation of the named lookups on TriggerSpec tails and TriggerSpec XML serialisation.

Classes
struct	cmlabs::ControlMessageHeader
	Wire/shared-memory header of a ControlMessage; any payload follows immediately after. More...
class	cmlabs::ControlMessage
	Wrapper class managing a single flat ControlMessage memory block. More...
struct	cmlabs::DataMessageEntryHeader
	Header preceding every user entry inside a DataMessage's flat memory block. More...
struct	cmlabs::DataMessageHeader_Old
	Legacy (pre-version-10) DataMessage wire header, kept only for format conversion. More...
struct	cmlabs::DataMessageHeader
	The current (version 10) DataMessage wire/shared-memory header. More...
class	cmlabs::DataMessage
	The central Psyclone data container: a self-contained binary message with typed, named user entries. More...
class	cmlabs::MessageIndex
	Multi-key in-memory index and lifetime manager for DataMessages. More...
struct	cmlabs::ReplayIndexEntry
	Fixed-size on-disk index record describing one recorded message. More...
class	cmlabs::MessagePlayer
	Records DataMessage streams to disk and replays them with original timing. More...
struct	cmlabs::FilterSpec
	One filter condition attached to a TriggerSpec: a test on a user entry of the candidate message. More...
struct	cmlabs::QuerySpec
	Specification of a query against a catalog/service/feed, attached to a TriggerSpec. More...
struct	cmlabs::RetrieveSpec
	Specification of a whiteboard retrieval attached to a TriggerSpec. More...
struct	cmlabs::PostSpec
	Specification of a message to post when a trigger fires (or a crank completes). More...
struct	cmlabs::SignalSpec
	Specification of a signal (a typed notification without payload) emitted when a trigger fires. More...
struct	cmlabs::TriggerSpec
	A complete trigger definition: what fires a component's crank and what happens then. More...
struct	cmlabs::ComponentSetup
	Complete static setup for one component, as parsed from the system specification. More...
struct	cmlabs::CustomView
	A custom UI view registered for a component (local configuration; never transmitted). More...
class	cmlabs::ComponentSpec
	In-memory aggregate of a component's setup, triggers and custom views. More...
struct	PsyType
	Hierarchical message type identifier — the key used for publish/subscribe matching in Psyclone. More...
struct	PsyContext
	Hierarchical execution-context identifier. More...