All projects

Cloud & delivery engineering / EdgeOps

When the connection fails,
keep the readings.

A fleet-reliability project connecting a durable device outbox with AWS telemetry ingestion and a controlled deployment workflow.

Scope
Synthetic fleet · Real AWS deployment
Status
Deployed project · Public source (MIT)
Focus
AWS · Terraform · Jenkins
Result
Rollback rehearsed, hashes matched

Why this project

My focus here is the boundary between device reliability and cloud operations: what happens when a connection drops, what an acknowledgement actually proves, and how a deployment change gets checked before it ships.

The problem

Intermittent connections make telemetry easy to lose or to replay. A successful HTTP response may only mean a queue accepted the batch while processing is still pending. The system has to keep unsent readings, make that boundary explicit, and stop old readings from overwriting newer device state.

The approach

A Python device agent stores readings in a SQLite outbox before transmission. In the AWS path, authenticated ingestion sends telemetry to SQS; a separate Lambda worker updates DynamoDB. The dashboard presents fleet state and a recovery simulation. Terraform defines the cloud stack, and a local Jenkins pipeline checks an explicitly approved, constrained deployment plan.

Key decisions

Persist before sending; define the acknowledgement

The agent retains readings until the expected acknowledgement arrives. A cloud HTTP 202 response confirms queue acceptance, not completed processing. Failed or malformed responses leave the batch available for a later retry.

Protect current state from replay

Event checks and persistent per-device sequence tracking prevent duplicate or older readings from replacing newer state. This deliberately favors a trustworthy current view over retaining every late arrival; it is not an exactly-once delivery claim.

Deploy only the reviewed plan

The pipeline verifies the deployment identity and region, restricts updates to existing application code and frontend assets, and checks the saved plan and its input hashes before apply. Infrastructure creation, deletion and permission changes are outside that deployment path.

Evidence & limits

A recorded release, rollback and restore

The retained rehearsal records a new release, a rollback to the previous source checkpoint, and restoration of the new release through Jenkins and Terraform. Delivered frontend and Lambda artifact hashes matched the selected checkpoints, and the recorded fleet state remained unchanged. When the older API lacked release metadata, the loaded newer interface showed an unknown API version instead of claiming a match.

This was a controlled source-snapshot redeployment on 6 October 2026, not a database rollback or a Lambda-alias switch. The records were inspected for this case study; the deployment was not rerun.

Inspected implementation and retained records cover local outbox replay, duplicate suppression, authenticated AWS requests, asynchronous queue processing, and deployment-policy checks. A recorded cloud smoke exercise observed accepted telemetry reaching fleet state and repeated batches being suppressed. The release rehearsal retains both the rollback and the restored release. 105 tests run on every change; a recorded release → rollback → restore took 177.5 s.

What this doesn’t prove

The fleet is synthetic and shared by authorized demo operators. Physical-device behavior, production load, uptime and multi-tenant isolation have not been tested. The agent retries only when invoked, and late out-of-order readings are discarded rather than kept as full history. Dashboard recovery is a server-side simulation, so it doesn’t prove the device agent went through a real network outage. Internal validation reports remain private, and the operator sign-in isn’t offered as a public demo.

Evidence last reviewed 7 October 2026. This is a high-level case study: results taken from project records are kept separate from checks rerun for this review, and Chisel's code and internal reports are not published.

Takeaway

Reliability depends on explicit boundaries: saved locally, accepted by a queue, processed into current state, and deployed from a reviewed artifact. Each state should be visible without claiming more than its check demonstrates.