Queue Type Reference

Queue Types

The One-to-One queue delivers each message to exactly one consumer. A visibility key ensures that only one message per key group can be in-flight at any time, preventing duplicate processing. Messages are deduplicated by primary key hash on publish. Consumers must acknowledge (or nack) each message before receiving the next one for that visibility key. If a lease expires, the message is automatically requeued by the reaper.

Settings

Publish Flow

Consume Flow

Dead Letter Flow

Key Behaviors

• Visibility key enforcement — only one message per visibility key value can be in-flight (leased) at a time. This prevents a single consumer from monopolising the queue and ensures ordered processing per key group.
• Primary key deduplication — the primary key fields are hashed with SHA-256 on publish. If a message with the same hash already exists in the queue, the publish is rejected with a 409 Conflict.
• Overflow policies — when max_queue_depth is set and the queue is full, "reject" returns a 409 error to the publisher, while "drop_oldest_append" silently removes the oldest message to make room.
• Lease-based consumption — each consume creates a time-limited lease. If the consumer crashes or fails to ack/nack, the reaper automatically requeues the message after consume_ttl_seconds.
• Auto-ack mode — when enabled, messages are immediately deleted on consume with no explicit acknowledgement required. Nacks are rejected in this mode.

The One-to-Many queue delivers independent copies of each message to multiple consumers. Each consumer (identified by client_id) tracks their own position in the queue and must acknowledge a message before receiving the next one. There is no visibility key — consumption is tracked per consumer, not per message field. This is ideal for fan-out patterns where multiple services need to process the same data independently.

Settings

Publish Flow

Consume Flow

Dead Letter Flow

Key Behaviors

• Independent consumer tracking — each consumer (client_id) maintains their own position. Consumer A can be on message 5 while Consumer B is still on message 2.
• No visibility key — unlike One-to-One, there is no field-based routing. Every consumer sees every message in queue order.
• Per-consumer leasing — each consumer gets their own lease. If Consumer A's lease expires, only Consumer A's copy is requeued.
• Primary key deduplication — same as One-to-One; duplicate primary key hashes are rejected on publish.
• Fan-out pattern — publish once, consume independently by many services. Perfect for event distribution.

The Key-Value queue is a state snapshot and lookup store. Items are keyed by their primary key and can be retrieved individually, in bulk, or as a delta since the consumer's last request. There is no consume/ack/nack cycle and no dead letter queue — this is a persistent store that consumers read from at any time. It also serves as the lookup source for Stream-to-One and Stream-to-Many enrichment queues.

Settings

Publish Flow

Consume Flows

Delete Flow

Key Behaviors

• State snapshots — consumers can read the full state at any time with GET /items. No consume/ack cycle means no message loss from crashes.
• Delta queries — POST /items/updates returns only items modified since the provided timestamp, enabling efficient polling without re-fetching everything.
• Sort key ordering — when a sort_key is configured, GET /items returns items sorted by that field for consistent ordering.
• On-duplicate handling — "reject" (default) returns 409 if the primary key already exists; "replace" overwrites the existing item with the new payload.
• No DLQ — since there is no consume/ack cycle, there is no dead letter queue. Expired items (via max_message_age_seconds) are simply removed.
• Enrichment source — Key-Value queues serve as the lookup source for Stream-to-One and Stream-to-Many queues.

The Time-Series queue aggregates numeric data points into clock-aligned time buckets. Each bucket computes OHLC (open, high, low, close) statistics along with sum, count, and average for every configured aggregation field. Buckets are aligned to clock boundaries (e.g., a 5-minute interval starting at the top of the hour) for consistent reporting. Gap filling ensures continuous time ranges even when no data arrives.

Settings

Publish Flow

Query Flows

Key Behaviors

• Clock-aligned buckets — bucket boundaries snap to clock intervals (e.g., 10:00, 10:05, 10:10 for 5-minute intervals), not relative to the first data point.
• OHLC aggregation — each field in every bucket tracks open (first value), high (maximum), low (minimum), close (most recent), sum, count, and average.
• Gap filling — when fill_gaps is true, empty buckets are included in range queries with zero counts, ensuring a continuous time axis for charting.
• Multi-series support — when primary_key is set, data is partitioned into separate series. Pass the primary_key query parameter to retrieve a specific series, and GET /buckets/keys to list the active series.
• No consume/ack cycle — like Key-Value, Time-Series is a read-only store. There is no dead letter queue.
• Automatic eviction — when max_buckets is exceeded, the oldest buckets are dropped to keep storage bounded.
• Clear buckets — the queue owner can reset the series with DELETE /queues/{id}/buckets (tenant-admin auth), removing every stored bucket for the queue.

The Stream-to-One queue combines the single-consumer delivery model of One-to-One with automatic data enrichment. On publish, the API performs a foreign key lookup against a linked Key-Value queue and merges the lookup data into the message payload before encryption and storage. If the lookup fails, the message is either rejected (with a DLQ entry) or deferred for later enrichment, depending on the missing_lookup policy. Consumption follows the same visibility-key-enforced, lease-based pattern as One-to-One.

Settings

Publish + Enrichment Flow

Consume Flow

Key Behaviors

• Automatic enrichment — the API transparently joins incoming data with lookup data before the consumer ever sees the message. No client-side joins needed.
• Missing lookup: reject — if the foreign key is not found in the Key-Value queue, the message is sent to the dead letter queue with reason "key_not_found" and a 422 is returned.
• Missing lookup: defer — the message is stored with status "pending_enrichment". When the matching key-value item is later published, deferred messages are automatically enriched and moved to the active queue.
• Visibility key enforcement — same as One-to-One, only one message per visibility key can be in-flight at a time.
• Lookup queue validation — at creation the lookup_queue must reference an existing Key-Value queue owned by the same tenant (rejected with a 400 otherwise). The foreign key's value is matched against that queue's primary key at publish time; the field names need not be identical.

The Stream-to-Many queue is the multi-consumer variant of Stream-to-One. It performs the same automatic Key-Value lookup enrichment on publish, but delivers independent copies to multiple consumers — just like One-to-Many. There is no visibility key; each consumer independently tracks their position and acknowledges messages at their own pace.

Settings

Publish + Enrichment Flow

Consume Flow

Key Behaviors

• Same enrichment as Stream-to-One — identical foreign key lookup and merge behaviour on publish, including reject and defer modes for missing lookups.
• Multi-consumer delivery — like One-to-Many, each consumer independently tracks position and acknowledges at their own pace.
• No visibility key — unlike Stream-to-One, there is no field-based in-flight exclusivity. All consumers receive all messages.
• Independent leasing — each consumer's lease is independent. A nack or lease expiry for one consumer does not affect others.
• Deferred enrichment — when missing_lookup is "defer" and the key-value item is later published, all deferred messages matching that key are enriched and promoted to the active queue for all consumers.

The Document queue is a repository for whole files. Unlike the message queue types, an uploaded XML, CSV, or JSON document is stored intact as a single item — it is never split into per-record messages. Clients list the stored documents, download an individual document in the format they ask for, and delete it. There is no consume/ack/nack cycle and no dead letter queue: a download is non-destructive, and a document stays until it is explicitly deleted or expires.

Settings

Upload

POST /queues/{id}/documents with the raw file as the body and Content-Type set to the file's format (application/json, text/csv, or application/xml). Two headers are required: X-Document-Name and X-Document-Description. The document must be convertible to JSON or the upload is rejected. Maximum size is 1 MB. Duplicate filenames are allowed — each upload is an independent document with its own id.

List & Download

GET /queues/{id}/documents returns a paginated list (filename, description, size, upload time) with optional filename-prefix and uploaded-at filters. GET /queues/{id}/documents/{doc_id} downloads one document; the Accept header selects the format (JSON / CSV / XML) and the broker converts on the fly — the original format is returned byte-exact. The name and description come back as X-Document-Name / X-Document-Description response headers.

Delete

DELETE /queues/{id}/documents/{doc_id} removes a document. Only consumers may delete — upload requires the publish permission; list, download, and delete require the consume permission.

Cross-Tenant Fan-Out

A document queue can be a subscription source (allow_subscriptions). Each uploaded document is fanned out to subscriber tenants' document queues, re-encrypted under each subscriber's own key. Delivered copies are independent — a source-side delete or TTL expiry does not cascade to subscribers, and re-delivery is idempotent.

Key Behaviors

• Whole-file storage — a document is one item, never split into messages.
• Format conversion on download — stored once, served as JSON, CSV, or XML per the Accept header (the source must be JSON-convertible at upload).
• Non-destructive reads — downloading never removes a document; only an explicit delete or TTL does.
• Duplicate names allowed — documents are addressed by id and distinguished by upload time.

The Text Document queue is a repository for plain-text files. It is the text/plain sibling of the Document queue: an uploaded document is stored intact as a single item and returned exactly as it was uploaded. Unlike the Document queue, text is treated as an opaque blob — it is never parsed, canonicalised to JSON, or converted between formats. Everything else (listing, download, delete, retention, cross-tenant fan-out, permissions) works exactly like the Document queue.

Settings

Upload

POST /queues/{id}/documents with the raw file as the body and Content-Type: text/plain. Two headers are required: X-Document-Name and X-Document-Description. The body is stored verbatim — there is no JSON/CSV/XML validation, so any UTF-8 text is accepted. A non-text/plain Content-Type is rejected with 415. Maximum size is 1 MB; duplicate filenames are allowed (each upload is an independent document with its own id).

List & Download

GET /queues/{id}/documents returns the same paginated list as the Document queue. GET /queues/{id}/documents/{doc_id} downloads one document and always returns it verbatim as text/plain — the Accept header is ignored and no conversion is attempted, because the stored text is not structured.

Delete

DELETE /queues/{id}/documents/{doc_id} removes a document. Only consumers may delete — upload requires the publish permission; list, download, and delete require the consume permission.

Cross-Tenant Fan-Out

A text-document queue can be a subscription source (allow_subscriptions). Each uploaded document is fanned out to subscriber tenants' text-document queues, re-encrypted under each subscriber's own key. Delivered copies are independent — a source-side delete or TTL expiry does not cascade to subscribers, and re-delivery is idempotent.

Key Behaviors

• Verbatim storage — the exact bytes uploaded are stored and returned; no parsing or canonicalisation.
• No format conversion — download always returns text/plain; the Accept header is ignored.
• text/plain only — a structured Content-Type (JSON/CSV/XML) is rejected with 415; use the Document queue for those.
• Otherwise identical to Document — same TTL, depth cap, fan-out, and permission model.

Cross-Cutting Features