Components

Components are the building blocks of streams. Each stream uses one input, zero or more processors, and one output.

Inputs

Inputs define where data enters the pipeline.

Component	Description
Generate	Generates synthetic messages for testing
HTTP Client	Pulls data via HTTP requests
HTTP Server	Accepts incoming HTTP requests (webhooks)
Kafka	Consumes messages from Kafka topics
Broker	Combines multiple inputs into one stream
MySQL Replication	CDC from MySQL/MariaDB binlog
Shopify	Fetches data from Shopify stores

Processors transform, validate, or route messages within the pipeline.

Component	Description
Mapping	Bloblang transformations
JSON Schema	Validates messages against a JSON schema
Catch	Error handling — runs processors on failure
Switch	Conditional processing based on message content
Schema Registry Decode	Decodes Avro messages via Schema Registry

Outputs define where data is delivered.

Component	Description
HTTP Client	Sends data via HTTP requests
Kafka	Produces messages to Kafka topics
SQL Insert	Inserts rows into SQL databases
Sync Response	Returns response to HTTP Server input
Switch	Conditional routing to different outputs
Broker	Routes messages to multiple outputs

Caches provide state storage for components like Shopify position tracking.

Cache	Description
Memory	In-memory with TTL
Redis	Distributed caching
Memcached	Distributed memory caching
File	File-based persistence
LRU	Least Recently Used with capacity limits
TTLRU	TTL-aware LRU
Ristretto	High-performance cache

Rate Limit	Description
Coordinator	Distributed rate limiting across workers

Buffer	Description
Memory	In-memory buffering
SQLite	Persistent at-least-once delivery
System Window	Time-based windowing