Shipping Reliable Crypto Order Execution

Most crypto trading products sell speed and a chart. The hard part isn't drawing the candles. It's being able to answer, at any millisecond, a deceptively simple question: what is the true state of my order right now? When we started building MadaiOps, our internal orders and trading-operations app, that question organized everything. Speed is a feature. Truth is the product. Crypto order execution reliability is what you have when the app can answer that question honestly even while the venue is degrading, and an app that shows a fill that didn't happen — or hides one that did — destroys trust in a way no latency win can buy back.

This post is about the engineering that makes an orders app trustworthy: the state machine, idempotency, reconciliation over WebSockets, and the failure modes that only show up under load. We're opinionated about all of it, because the boring choices are the ones that keep you solvent.

Model the order as an explicit state machine

The single best decision we made was to stop treating an order as a row you UPDATE and start treating it as a finite state machine with named, enumerated transitions. An order lives in states like PENDING_NEW, WORKING, PARTIALLY_FILLED, FILLED, CANCEL_REQUESTED, CANCELED, REJECTED, and EXPIRED. Every transition is a function of the current state plus an inbound event, and illegal transitions are rejected loudly instead of absorbed silently.

The payoff: ambiguity becomes a validation-time concern instead of a 2 a.m. incident. A FILLED order can never go back to WORKING. A fill event arriving for a CANCELED order isn't "impossible" — it's a real race, and the state machine forces you to decide what it means. Usually it means the cancel lost the race, the order actually filled, and your local CANCELED was wrong. Model orders as mutable rows and that event just overwrites something; you never notice until reconciliation — or a user — catches it.

The takeaway

Enumerate your states before you write a single API call. If you can't draw the diagram on a whiteboard, you don't understand your domain yet, and the exchange will teach you the hard way. Store the transitions, not just the current state. An append-only event log of what happened beats a single mutable status column, because the log is replayable and the column is a guess.

Idempotency is not optional

Networks retry. Your own retry logic retries. A user double-taps "Buy." Any of these can send the same order twice, and on an exchange a duplicate order is real money. The defense is a client-generated idempotency key — most exchanges call it clientOrderId or newClientOrderId — attached to every order the moment the user commits, before the request ever leaves the device or server.

The rule we enforce: the key is generated once, persisted locally before the network call, and reused verbatim on every retry. If the first attempt timed out with no response, the retry carries the same key, and the exchange either creates the order once or returns the existing one. Either way you converge on a single order. Generate the key after a failure and you've built a duplicate-order machine.

This extends to cancels and modifications. A cancel request should also be idempotent: cancelling an already-canceled order is a no-op that returns success, not an error that triggers another retry loop. We treat "the desired end state is already true" as success everywhere. It sounds trivial. It eliminates an entire category of retry storms.

The takeaway

Persist the idempotency key before you act, not after you succeed. The gap between "I sent it" and "I know it worked" is exactly where money disappears, and the key is the only thing that closes it.

Reconcile relentlessly: the WebSocket is a hint, not the truth

Exchanges push order updates over WebSockets, and they're wonderful when they work: sub-second fills, live partial-fill quantities, cancel confirmations. But a WebSocket is a stream you can miss. You will get disconnected. Messages arrive out of order, get dropped during a reconnect, or simply never come because the venue is degraded. If your order state is only whatever the socket last told you, your app is confidently wrong the moment the socket hiccups.

So we run two loops. The fast loop consumes the WebSocket and applies events optimistically. The slow loop polls the REST API — GET /order and the open-orders endpoints — on an interval and after every reconnect, then reconciles that snapshot against local state. REST is the source of truth; the socket is a low-latency hint. When they disagree, REST wins and we log the divergence, because a divergence is a bug report the system wrote for you.

Sequencing matters. Most exchanges stamp updates with a monotonic sequence number or an updateTime. We drop any event older than the last applied one for that order, which makes out-of-order delivery harmless. On reconnect the correct move is: resubscribe, immediately pull a full REST snapshot to backfill the gap, then trust the live stream again. Skip the snapshot and you miss the fill that happened during the two seconds you were offline.

The takeaway

Never let a real-time feed be your only writer of state. The socket makes you fast; the reconcile loop makes you correct. You need both, and when they conflict, correct beats fast every time.

Partial fills and disconnects: the two that expose you

Partial fills are where naive apps quietly lie. An order for 1.0 BTC that fills 0.3, then 0.4, then hangs is not "open" and it is not "done." It's PARTIALLY_FILLED with 0.7 executed and 0.3 remaining, and every one of those numbers has to be tracked from the individual fill events, not inferred from a single status field. We accumulate executed quantity and volume-weighted average price from the discrete fills, so the position is always reconstructable from primitives. A user who sees "filled" needs to know filled how much, at what average price. Anything less is a half-truth.

Disconnects compound this. Drop offline mid-fill and the WebSocket never told you about the 0.4, so your local state is stale. This is precisely why the reconnect-then-snapshot discipline exists: the REST snapshot returns cumulative executed quantity, and the reconcile loop closes the gap without the user ever seeing a wrong number. The failure mode we design against isn't "the app crashed" — crashes are honest. It's "the app kept running and showed something false."

The takeaway

Track cumulative executed quantity as the primary fact and derive everything else from it. Never store "remaining" as an independent field you mutate. Derive it (ordered − executed) so it can't drift out of sync with reality.

Truthful under pressure beats clever

There's a persistent temptation to be clever: predict fills before confirmation, hide "transient" errors, smooth over a reconnect so the UI never flickers. We've learned to resist most of it. A trading operator doesn't want a smooth lie; they want to know, honestly, that the connection just dropped and state is reconciling. A visible "reconnecting, verifying orders" state is worth more than a seamless UI that turns out to be showing yesterday's truth.

Concretely, we surface uncertainty instead of papering over it. When local state and the exchange haven't reconciled recently, the app says so. When an order is CANCEL_REQUESTED but not yet confirmed canceled, it shows exactly that, not a premature CANCELED. Optimistic UI is fine for a to-do app. For money, optimism is a liability you pay for in support tickets and lost trust.

The takeaway

When in doubt, show the doubt. The most reassuring thing a financial app can do under stress is refuse to guess.

Closing thought

None of this is glamorous. There's no demo where "our idempotency keys are persisted before the network call" gets applause. But reliability isn't a feature you add later. It's the set of decisions you make early and refuse to compromise, then defend every time a shortcut looks tempting. Build the system that tells the truth when everything is on fire, and the fast, pretty parts take care of themselves.