How Mirexa works

A stream is named by three strings: exchange, symbol and timeframe, for example XNAS / AAPL / 1m. The names travel in the payload of every request and become directory names on disk, so each part is limited to letters, digits, . _ -, 1 to 64 bytes, and never . or ... That check is the security boundary for the file layout.

A candle is a timestamp and five 8-byte numbers: open, high, low, close, volume. On the wire it is 49 bytes with a flag byte; in the WAL it is 48. A tick is 24 bytes on the wire and grows to 32 in the WAL, because each entry repeats its stream's hash so it can be recovered alone.

On disk a stream's files sit under its partition's directory. What a partition is, and why all timeframes of one symbol share one, belongs to clustering: see Partitions.

Each stream can carry a trading calendar in its configuration. When the flusher writes a chunk, it classifies every missing interval in that chunk against the calendar and files the result in the stream's gap index. A hole is classified once, at flush time, and never revisited. Correcting a calendar later changes how future holes are filed and leaves history alone.

The cache keeps five parallel columns (open, high, low, close, volume) plus a gap bitmap. A live candle stream holds up to 50,000 slots and keeps 25,000 after the flusher compacts it. Overwriting a slot already published goes through a small version overlay, so readers that hold no lock never see a torn value.

A cache's starting timestamp lives only in memory. After a restart, a floor at the stream's newest flushed chunk stands in for it, so a write cannot land on top of data already in a chunk. Every live batch holds the cache from its admission check to its last candle, and compaction waits for it.

A correction to a bar that is already in a chunk is written as a patch chunk that names its base. A later correction folds the earlier one in, so a read is always the base plus one patch. Each chunk carries CRC-32C checksums over its data and footer, checked on every read.

An index write is always a full rewrite: temporary file, fsync, rename, then an fsync of the parent directory. Without the last step a rename can be lost on XFS or btrfs while looking fine on ext4. A build-time test fails if a rename appears anywhere without its directory sync.

The partition count is fixed once a cluster agrees on it. Routing is hash % count, so changing the count would move every stream at once, with reads finding nothing in the meantime. The engine refuses the change in its replicated state machine rather than trusting operators not to try. Partitions move between nodes through a directory, one level up, which is why consistent hashing was not needed.

When a cluster first forms, the leader places every partition in one Raft entry, spreading them by count across the online nodes. A node that joins a fully placed cluster gets nothing at first; moving data to it is a rebalance, which is a separate and opt-in decision (Copies and moves).

Each directory entry names one primary, a list of copies, the subset of copies that are in sync, and an epoch: the Raft log index of the entry that installed the current primary. The epoch only moves when the primary changes, so adding a copy never disturbs writes to a partition.

Three reasons keep data out of consensus. A quorum commit per batch would add a network round trip and a second fsync to an ingest path that acks in well under a millisecond. The same bytes would be written durably twice, once in the Raft log and again in the WAL and chunks. And candle ingest does not need the total order consensus provides, because it is idempotent by stream and timestamp.

The price is fencing. In a Raft-replicated design an old leader simply cannot commit. Here a directory entry naming a new primary does nothing to the old one by itself, which is why the engine has epochs on writes and leases on primaries. And because copies are asynchronous, a primary that crashes can lose writes it acked after its copies' last pull.

Every node runs the same binary. A new node joins through a seed list, is admitted to the Raft group as a non-voter, and replicates the directory locally, so routing a request is a map lookup on the node itself. The leader promotes non-voters toward a fixed number of voters, three by default, never to an even count.

Raft holds three things only: the node registry, the partition directory and a few cluster-wide settings (partition count, codec, liveness timeout). A heartbeat does not write to the log; only the decision it justifies does, such as fencing a node that went silent.

A seed is a contact point, not a leader address: any member can say who leads. A node configured with a member list bootstraps a new cluster instead, and seeds itself, so there is one registration path for everyone. Once running, a node reaches the cluster through the members its own state knows, leader first, and falls back to its seeds only after them.

Voter membership changes on failure and departure only. A draining voter counts as already gone, so a replacement is promoted before it is removed, and the group never shrinks from 3 to 2. A leader that is being decommissioned hands leadership to another voter before it is removed, so the group is never leaderless.

Heartbeats run every third of the liveness timeout and never write to the log. A node silent for the full timeout is fenced; a fence and a drain are independent facts, so a drain survives the node crashing.

Each partition's directory entry carries an epoch: the Raft log index of the entry that installed its current primary. It changes only when the primary changes. Clients send the epoch they believe with every write, and a primary refuses an older one, answering with a redirect that carries the current epoch so the client's cached route corrects itself. The epoch stops stale clients. A primary that is itself out of date is stopped by its lease: see When a node dies.

A copy proposes itself in sync once it reaches the primary's durable position. It leaves the in-sync set on its own if it stays behind for longer than two liveness timeouts plus two seconds, but only while the primary is still believed alive: when the primary dies, every copy stops keeping up at once, and none of them fell behind.

An operator moves a partition with one request; reconcilers carry out the steps. A planner can also suggest moves that even out bytes across nodes, and acting on them without approval is opt-in. Draining a node moves its primaries first, then its copies, never dropping a copy to do it.

Copies are byte-identical by construction, so the primary is the reference and no vote is needed. The question reconciliation answers is narrower: what can make a copy hold something else, and how would anyone notice?

Rot is found only where something reads it, and a seed never re-reads what a copy already holds. That is why every node also scrubs. At the default rate, 100 GB takes about 3.3 hours per pass (arithmetic, not measured). Nothing deletes a quarantine automatically; it stays as evidence until an operator removes it.

Besides the SDK there is mirexa-cli, which ingests CSV and queries to CSV over the same client, mirexa-token, which mints signed tokens offline, and the web dashboard, which holds one token for all its users and talks to a node through the same protocol.

Retrying a write is safe only if the server provably did not keep it. Overloaded is raised before the WAL, so the batch was never stored and a resend is a first attempt. A timeout or dropped connection mid-request is different: the batch may have landed with only the ack lost. The client reports that as ambiguous and the caller reconciles by reading back. Ingest is also idempotent by stream and timestamp while a bar is in the hot cache, but the retry rule rests on the shed guarantee, not on that.

Overloaded is retried for you, three attempts in total by default, honouring the server's hint. Reads are retried across a reconnect. A Batcher moves the receipt: Add returning nil means buffered, a failed flush poisons the batcher, and only Close is guaranteed to report it.

A dropped batch is gone, not late. The server drops when a subscriber's queue is full, and after drop_limit drops in total it evicts the subscriber with a SlowConsumer error. Recover with a query or a backfilling re-subscribe; a ResumingSubscriber does the second on its own after a disconnect. One subscription per connection, because the push stream is per connection.

A connection opens with INIT: an 84-byte frame carrying an Ed25519-signed token with tenant, expiry and a permission bitmap. The server clamps the grant to its own ceiling and returns it in INIT_ACK. Expiry is enforced during the session as well: a writer is refused on its next frame, a subscriber is disconnected by a timer.

Only Overloaded is retryable, and the Go SDK retries it for you. A redirect is not an error; the routing client follows it to the named node, which is a first attempt there because the refusing node dispatched nothing.