It also contains ten dashboards for a display its manufacturer stopped supporting, which is not what I set out to build.

MIT licensed.

Source → github.com/isaacrowntree/vesc-workbench (MIT)

If you tune a VESC, this is for you

Everyone who runs a VESC ends up in the same loop. Change a current limit. Ride. Change it back. Change the throttle curve. Ride. Was that better, or was it a headwind? What did you actually have it set to three weeks ago, before the thing you’re now trying to undo?

The tooling does not help you here. VESC Tool is a good GUI, but it is a GUI: you click through tabs, you hope you typed the number into the field you meant, and when you are done there is no record of what changed. Exporting an XML backup is a manual step you take instead of riding, so nobody does it every time — and a backup you only take before something scary isn’t a history. And on a lot of builds — a sealed skate enclosure, a scooter deck, an ebike downtube — the USB port is behind screws, so you are doing all of this on a phone, standing in a driveway.

This repo is that loop, scripted:

make pull      # read both motor sides' configs to XML
make apply     # write them back, then verify by reading them again

Your settings are now text files. You can git diff a tuning session, review it before it goes near the motors, and revert it in one command. apply reads back after writing, so a setting that didn’t take is something you find out about at the bench rather than at speed — which matters, because setMcconf(false) accepts a write and silently discards it.

And it runs over Bluetooth, from your laptop, with the board sitting where it is.

The connection trick, because nobody documents it

This is the part worth the post on its own, so it has its own page in the repo.

VESC Tool ships a CLI. It looks like it should solve everything, and then it doesn’t: --vescPort calls connectSerial() and takes a serial device, full stop. Hand it an IP and it refuses. Bridge the TCP socket to a socat PTY and it opens the port and then never completes the handshake. If your controller isn’t reachable over USB, the documented CLI is a dead end.

I wrote that down as impossible. It isn’t.

VESC Tool also accepts --loadQml, and QML loaded that way runs inside the application, with the VescIf singleton in scope. VescIf is the whole connection and configuration API — and VescIf.connectTcp() is invokable from it. The phone app has a Wireless Bridge to Computer (TCP) mode sitting right there on its Start page.

So the path is:

your laptop  --TCP-->  phone (VESC Tool app)  --BLE-->  ESC

The entire minimum viable version:

import QtQuick 2.7

Item {
    id: root
    property int ticks: 0

    Component.onCompleted: VescIf.connectTcp("192.168.1.100", 65102)

    Timer {
        interval: 500; running: true; repeat: true
        onTriggered: {
            root.ticks++
            if (root.ticks > 60) { console.log("timeout"); Qt.quit() }
            if (!VescIf.isPortConnected()) return

            // Firmware params arrive AFTER the socket connects. Until they do,
            // getFirmwareNow() returns "x.x" and every config read is garbage.
            var fw = VescIf.getFirmwareNow()
            if (fw.indexOf("x.x") >= 0) return

            console.log("connected, fw " + fw)
            VescIf.disconnectPort()
            Qt.quit()
        }
    }
}

"/Applications/VESC Tool.app/Contents/MacOS/VESC Tool" --offscreen --loadQml connect.qml

That’s it. --offscreen keeps the GUI away, console.log goes to stdout, and from there VescIf.mcConfig(), VescIf.appConfig() and VescIf.commands() are all yours. You do not need the rest of my repo to use this — take the file.

That comment about "x.x" is not decoration: the socket connects several seconds before the firmware parameters arrive, and a config read in that window returns defaults that look exactly like a controller which has wiped itself.

Diagnostics that answer the actual question

Half of tuning is not tuning, it’s working out what is wrong. A GUI shows you a number; it rarely tells you why the number is that.

make ppm-watch    # live remote readout — prints only on change
make ppm-cal      # guided calibration: neutral, full throttle, full brake
make probe        # connect, report firmware and LispBM state
make check        # is the bridge up? is desktop VESC Tool holding it?

ppm-watch distinguishes the remote is not transmitting from the decoder is not running. In VESC Tool those look identical — a still bar — and they have completely different fixes. make check does the same thing for the connection: it tells you which failure you have in a second, instead of leaving you to interpret a two-minute timeout.

And when you’re working on a board that is powered up:

make motors-off   # kill motor output, no config write, display stays live
make motors-on

app-disable-output rather than a configuration change, so nothing needs undoing afterwards and nothing gets left in a weird state if you walk away.

The findings are the other half of the repo

docs/findings.md is the document I wanted to find and could not. The pattern in all of them is the same: the ESC does something reasonable, reports it accurately, and the accurate report points at the wrong thing.

There’s a correction in there too. The esk8 forums will tell you VESC’s built-in traction control interferes with braking and is dangerous. I enabled it, then read app_ppm.c to understand the failure mode: it lives entirely in the non-brake branch, braking never reaches the traction-control code, and it self-disengages on any fault. The warning is real for some other control paths; for PPM on current firmware it’s repeated folklore. Documented with the file and the branch, so you can check my reasoning rather than trusting either of us.

Where it came from: a display that refused to grow up

All of this exists because of a much smaller problem.

I have a LaCroix Nazaré — an electric skateboard with a FOCBOX Unity sealed in the enclosure, which I ride on grass at a golf course, a surface demanding enough that the tuning genuinely matters. I wanted it on firmware 7 for the FOC improvements and LispBM. The DAVEGA X display bolted to the deck refused to run on it:

Unsupported VESC FW: 7.01

Supported VESC FW versions: 3.48 - 6.x

DAVEGA is discontinued. Everyone downgrades.

I checked whether that was really true rather than assuming it. The vendor’s update index still lists a v5.07rc3, dated 2025-03-11 — newer than the v5.06 the public changelog stops at, released after the shop closed and never announced. It carries the same constant as every version before it. No firmware DAVEGA ever shipped accepts VESC 7.

But the telemetry protocol did not change. COMM_GET_VALUES returns the same twenty-five fields, same order, same scaling, in 6.00 and in 7.x. The display isn’t failing to parse anything — it reads two version bytes, sees a 7, and declines to have the conversation.

VESC firmware 6 embeds LispBM, and from 6.06 it exposes the firmware’s own command decoder to it as cmds-proc. So a script on the ESC can take the display’s UART line, hand every packet to the real handler, and rewrite the reply on the way out:

(defun fixfw (d) {
    (var n (bufget-u8 d 1))
    (if (and (= (bufget-u8 d 0) 2) (= (bufget-u8 d 2) 0)) {
        (bufset-u8 d 3 6)                    ; FW_VERSION_MAJOR -> 6
        (bufset-u8 d 4 0)                    ; FW_VERSION_MINOR -> 0
        (bufcpy p 0 d 2 n)                   ; crc16 always starts at index 0
        (var c (crc16 p n))
        (bufset-u8 d (+ 2 n) (shr c 8))
        (bufset-u8 d (+ 3 n) (bitwise-and c 255))})})

Twenty-seven lines for the rewrite, 178 for the whole shim once you count the UART reader and its debug counters. Everything else passes through untouched. The display sees a 6.00 controller; it’s talking to a 7.00 controller; both are telling the truth about the only thing that matters.

You shouldn’t take my word for the payload being unchanged, so tests/protocol-diff.sh checks out both firmware versions from upstream and diffs the serialisation on every CI run. If Vedder ever changes the layout, the test goes red and the shim is wrong in a way you find out about immediately rather than at 40 km/h.

If you have a DAVEGA and want to keep its own firmware, that shim is the whole answer and you can stop reading here.

Writing LispBM without bricking your throttle

If you’re doing anything custom on a VESC, LispBM is where it happens, and the repo treats it as a real development environment:

make upload-lisp LISP=path/to/script.lisp   # upload and run
make lisp-stats                             # heap, CPU, globals
make lisp-stop / lisp-erase                 # stop, or back to stock
make test-lisp                              # run it in the real interpreter

That last one matters more than it sounds. Scripts are tested by running them in the upstream LispBM REPL in Docker with stubs for the VESC extensions — not by transcribing them into Python and testing the transcription. LispBM symbols are case-insensitive, t is a special symbol that never resolves from the environment (so (var t ...) silently kills the context that uses it), and uart-read’s timeout argument is in seconds and isn’t the argument you’d guess. Running the actual interpreter catches all three. A reimplementation catches none of them.

And make lisp-erase is always one command from stock behaviour, which is the thing that makes experimenting on a board you ride tolerable.

Then the display stopped being a black box

The shim treats the DAVEGA as something to lie to. That stopped being true once I went looking properly.

Its firmware is not open source and the shop closed in 2024, but the vendor’s own installer still names its endpoints, and they still resolve. More usefully: the X is an ESP32 running MicroPython, and it will give you a REPL. Hold up and down while it boots and it raises its own access point.

From there its filesystem is right in front of you. Settings are a plain /data/config.json — which is how I found the display had been configured for 175 mm wheels on 72/16 gearing when the board runs 200 mm on 84/20. It had been under-reading my speed by 18%, silently, for as long as I had owned it. One command fixed it.

It also means the screen is programmable. The firmware runs a user start.py before the stock app starts, so a replacement dashboard is not a firmware build — it is one file you can delete. Hold UP at boot and it steps aside.

At which point the shim stops being necessary at all. A dashboard I wrote has no version gate to fail, so it reads firmware 7 directly, and the LispBM context on the ESC is free for something better than lying about a version byte — it runs a flight recorder now.

So the display got ten dashboards

Ten themes, each with its own layout rather than its own colours: a full analogue tachometer, hexagonal shards on a diagonal split, three hairline arcs, one enormous thin numeral, concentric rings with a single red hand, a shift-light rail you read peripherally, a power-flow meter that treats current as more important than speed. The default takes the best idea from each and throws the rest away.

Those are not mockups. Every picture in this post is the real screen code run through a host-side stand-in for the panel, rendering the pixels it actually produced.

The interesting part is not the pictures. It is that the panel cannot draw a curve.

The display driver offers fill_rectangle, pixel and writeblock. There is no line, no circle, no polygon. And measured on the device, the costs are not what you would guess:

A draw call costs the same whether it covers nine pixels or nine thousand. So the obvious way to draw an arc — one thin rectangle per column, the way you would rasterise it — costs 481 ms for a 96 px radius. That is four times the budget for an entire frame, to draw one gauge.

The way through is the third primitive. Compose the curve into a memory buffer and push it in a single transfer — about 6 ms for a 240×20 strip, and the maths in between is free because it never touches the bus. There isn’t enough RAM to hold the whole picture, so it goes in horizontal strips with one buffer reused down the screen, and the drawing code works in absolute screen coordinates while each strip quietly discards what falls outside it.

That is the entire trick, and it is what makes a tachometer possible on a panel with no line primitive. The dial face, its bezel and its twenty-one tick marks are composed once as furniture; the sweep is composed the same way into a band that covers only the dial.

I did try the cleverer version first — repaint only the wedge between the old angle and the new. It is cheaper, and it is wrong: an arc drawn in three pieces lands on different pixels from the same arc drawn in one, because each piece quantises its own start angle. A partial renderer that disagrees with a full repaint is worse than one that costs a few more milliseconds, so the arcs repaint whole. Everything that is a straight line still repaints only its tip.

Steady-state cost across all ten: 23 to 88 ms a frame, against 5 Hz telemetry.

The harness is the reason any of it works

A 2.8″ screen bolted to a deck is a terrible place to iterate a design. So before the first dashboard there was a host-side stand-in for the ILI9341 that records every draw call, rasterises to PNG, and models the panel’s cost. Screens are pure functions of a telemetry frame, so the same code runs on the device and in CI, and 642 checks run with no hardware attached.

Four things it catches that looking at the screen cannot:

• Anything that leaves the 240×320 frame fails the build, rather than being clipped where you might not notice.
• Golden images for every frame in the envelope — standstill, full throttle, hard regen, thermal derate, battery empty, battery full, a fault at speed. Those frames are generated from the board’s verified configuration, so the extremes a real ride rarely produces are covered on purpose.
• Partial redraw against a full repaint, across all 810 transitions between envelope frames and themes. Partial redraw’s failure mode is stale pixels — a value that got shorter, a gauge that receded — and the two paths must agree exactly, pixel for pixel.
• Chrome labels against live regions. A label is drawn once; a region repaints whenever its value changes; the panel’s font is opaque. So a label sitting inside a region’s box is erased the first time that value moves and never comes back. It looks right at a standstill and wrong thirty seconds into a ride.

That last one is the test I’m most pleased with, because the harness was flattering the design until I fixed it. It drew glyphs without their backgrounds, while the real font fills the whole character cell — so labels that punched holes through their own gauges looked perfect in every render. Modelling the panel honestly made four themes fail immediately.

Every theme goes through all of it, and a parity test asserts each of the 64 declared colours actually appears in the rendered output, so a palette nobody has looked at cannot ship.

And then the same code, in the interpreter that runs it

All of that is CPython, which is fast and convenient and does not tell you whether the code will start on the board. So there is a second suite that builds MicroPython 1.14 — the version the DAVEGA reports — with framebuf compiled in, and drives the real modules through a whole ride: boot, telemetry replayed from frames recorded off the Unity, every screen, the buttons through their real 50 ms debounce, the menu, a controller that stops answering, and the recovery. Same argument as testing LispBM in the upstream REPL: test the code in the interpreter that will execute it, not in a host language that resembles it.

It earned its keep on the first run. MicroPython stores an RGB565 pixel with a native 16-bit write, so on a little-endian MCU the low byte lands first — and the ILI9341 wants the high byte first, with the transfer streaming the buffer untouched. Every curve would have reached the panel with its colours reversed. Red drawing blue. It could not show up off-device, because the pure-Python stand-in writes big-endian directly: the harness and the hardware disagreed and only the hardware was right. The two now render the same drawing side by side and compare byte for byte.

The other thing it watches is allocation, because 98 kB of heap and a two-hour ride is a bad combination for anything that leaks. It measures two consecutive 400-frame windows, because one window cannot tell a leak from a warm-up — caches fill and tweens settle in the first, and none of it repeats. Absolute byte counts are reported rather than asserted: this is the unix port on a 64-bit host, where every reference is twice the width, so only the drift carries.

Which is exactly the limit that bit me later. The board would not start one morning, stuck on the stock firmware’s “initializing” — a MemoryError asking for ten kilobytes to draw the first curve. Everything passed on the host, where the heap is twenty times larger. The fix was to claim that buffer during boot while the heap is still whole, take the largest band available rather than one fixed size, and have the loop fall back to the lightest layout instead of handing the screen back. A dead screen on a deck is worse than the wrong colours.

Where it’s at

Telemetry is live and the board has been out on it: a DAVEGA X running a dashboard I wrote, reading a FOCBOX Unity on firmware 7.00 directly, no version spoofing in the path. A Unity is two controllers in one case, and the standard reply only carries whichever one answered — its own temperature, its own tachometer — so the local ESC is asked over the wire and the second is asked through it over CAN, and the two are combined the way DAVEGA’s own Unity code does it: pack current and energy summed, per-motor figures averaged, distance counted once. Temperature is the one place I diverge and take the hotter of the two, because an average hides the controller that is about to derate behind the one that is fine.

The board is tuned for grass, 80 A a side, and traction control is on and confirmed after a hard run round a golf course. Themes and the dashboard both install over WiFi in one command, as precompiled bytecode — MicroPython compiles a .py every time it imports it, and on this ESP32 that compile was most of the wait between switching the board on and seeing a number.

And it has been ridden. Speed, current, charge and range on the glass, both controllers read, at 40 km/h on grass — which is the only test that was ever going to settle it, and the one the whole harness exists to earn the right to run.

Hardware coverage is honest — FOCBOX Unity, one board, one display — but the connection layer and the config workflow aren’t Unity-specific at all. vesc/profiles/ holds one file per known-good setup, and deliberately holds connection details only: not current limits, not gearing. Copying a stranger’s motor tuning is how packs and motors get damaged. Run the detection wizard, then use this to keep track of what you changed.

If you have a VESC and a scripting habit, start with docs/connecting.md. Even if you use nothing else, having your board’s configuration in git is worth the twenty minutes.

Source → github.com/triptechtravel/clawdwatch · Docs → triptechtravel.github.io/clawdwatch

The line this draws

There is an obvious, tempting version of an AI-era monitoring tool where a model looks at the response and decides whether you have a problem. clawdwatch deliberately does not do that. Detection is a threshold, a state machine, and a maintenance window — code you can read in an afternoon and reason about at 3am. What the AI gets is the other half of the job: the part after “this is broken,” where somebody has to work out why.

That split is the whole design. Everything below is either “make the detection boring and trustworthy” or “make the handoff useful.”

Detection is a state machine

One check, one result, one transition — pure, with the clock injected as a parameter rather than read from Date.now():

unknown   → healthy     first success                      — silent
healthy   → degraded    failure, threshold not yet met     — silent
degraded  → unhealthy   consecutive failures >= threshold  — OPENED
unhealthy → unhealthy   still failing, reminder due        — REMINDER
unhealthy → healthy     first success again                — RECOVERED
degraded  → healthy     recovered before opening           — silent

The unhealthy → unhealthy edge is the one that matters and the one the previous generation of this tool could not express at all: it returned nothing forever, which meant a multi-day outage alerted exactly once, on day one, and then went quiet while still being down. Reminders are only sayable if your state machine has a name for “still broken, and it has been a while.”

Alerts are batched per run, so a ten-endpoint outage is one notification rather than ten.

One storage system

D1. That is the entire persistence story — checks, hot state, results, deliveries, incidents.

The version before this one split hot state into an R2 JSON blob and history into an Analytics Engine dataset that nothing ever read. Two storage systems, two consistency stories, and a whole class of “which one is right?” bugs, in exchange for a dataset nobody queried. Collapsing it into D1 removed more code than it added, and the integration suite now applies the shipped migration and runs in workerd — so the SQL is genuinely exercised rather than mocked.

Public code, private config

Checks live in the database and are editable through the UI. That is only safe if a secret can never land in a row, so:

{ "headers": { "X-Api-Key": "${MY_API_KEY}" } }

The reference is what is stored. Substitution happens at exactly one point — building the outbound request — and every path leaving the system (API responses, alert payloads, logs, config exports) goes through redaction first. Writing a check that contains a literal secret value is rejected with a 400, and a property test asserts that no resolved value appears in any outbound representation.

That single guard is what makes a UI-editable, database-backed monitor safe to open-source at all.

For a token that belongs to a whole domain rather than one check — a WAF bypass, say — there are headerRules keyed by host pattern.

Keeping the thing that explains the outage

By default, response bodies are read to evaluate assertions and then discarded; what gets stored is the assertion failure message, truncated to 256 characters. A monitored endpoint that returns personal data does not leak it into the monitoring database.

But “expected 200, got 500” throws away the one thing that usually explains the outage. The motivating case was a health endpoint whose 500 body named the failing dependency by name — and the alert carried none of it. So a check can opt in:

{ "id": "api-health", "captureBodyOnFailure": true }

The excerpt is capped at 512 characters, taken only from textual content types, run through the same secret scrubber as everything else before truncation (so a secret straddling the cut is masked, not half-printed), never taken from a passing check, and deliberately never posted to Slack. It reaches the webhook notifier and the dashboard — both of which are already trusted with the monitoring database. Slack is not.

Handing an incident to an agent

An agent inbox is just a URL, so webhook() is the whole integration. Nothing is installed into the agent:

• Every alert carries a links object — incident, ack, annotate, run-now — as short-lived signed URLs. The receiving agent can act on the alert it was handed without holding any standing credential. If the alert leaks, what leaked expires.
• GET /api/agent.md describes the full API, generated from the route table so it cannot drift. Point an agent at that URL and the setup is done.
• An agent that has triaged something writes its findings back with POST /api/incidents/:id/annotate, and the note appears on the incident in the dashboard next to the human comments.

The agent.md route replaced an earlier idea: shipping a skill file for agents to install. A static file that duplicates a live API drifts, and in practice never gets installed — the previous version’s skill claimed 90-day retention for a system that kept 48 hours, and had never actually been copied into the container it was meant for. Generating the document from the route table, with a test asserting the two never diverge, is the version that stays true.

There is a worked receiver: thinkbot, an ops agent that takes clawdwatch alerts and correlates them against GitHub, Datadog, Sentry and Rollbar to report what changed.

When the receiver is a Worker, skip the webhook

If the thing receiving alerts lives on the same Cloudflare account, a service binding beats HTTP: the platform authenticates the call, so there is no shared HMAC secret to distribute or rotate, and no public inbox to defend.

notifiers: [rpc({ binding: (env) => env.AGENT })]

One sharp edge worth stating out loud, because it is the kind of thing that quietly becomes a vulnerability: an RPC call carries no signature. Authenticity comes from the binding itself. Keep signature verification in your HTTP handler, and do not let shared downstream code assume it ran.

Service bindings are same-account only, so webhook() with hmac() remains the option for everyone else.

A versioned payload, so a release is not an outage

Every alert carries schemaVersion, exported as ALERT_SCHEMA_VERSION, and the contract is narrow enough to be useful:

• adding an optional field does not bump it — bodySnippet was added exactly this way;
• removing or renaming a field, or changing its meaning or type, does.

Receivers should ignore unknown fields and must not hard-fail on a version higher than they know. A receiver that rejects unknown versions turns every clawdwatch release into a monitoring outage, which is a genuinely embarrassing way to lose visibility.

Auth, and the thing it refuses to do

Mount it behind Cloudflare Access. Reads are open by default; writes always require a principal — a signed-in person, a service token, or a capability link.

There is no query-parameter API key, on purpose. URLs leak into logs, analytics, and referrer headers, and a shared static secret has no identity, no expiry, and no revocation. Arriving through Access is also not treated as evidence in itself: JWTs are verified against the team’s JWKS, the expected aud, the issuer, and the clock — because anyone who learns the Worker’s direct route bypasses the Access edge entirely.

No identity is stored. The JWT is verified for the authorization decision and discarded; there are no email, IP, or user-agent columns.

A dashboard you will actually read

One mark per check run. A five-minute cron looks like five-minute samples, not a smoothed line — because the smoothing is exactly where a two-run blip goes to hide. Every delivery is recorded too, so the dashboard can answer the question monitoring tools are worst at: did the last alert actually arrive?

Getting it running

npm create cloudflare@latest my-monitor -- \
  --template clawdwatch/clawdwatch/examples/worker
cd my-monitor

wrangler d1 create clawdwatch          # paste the id into wrangler.jsonc
npm run migrate
wrangler secret put SLACK_WEBHOOK_URL   # optional, and all Slack needs
npm run deploy

Or use it as a library, if you would rather it be a route inside a Worker you already run:

import { createMonitor, slack } from 'clawdwatch';

const monitor = createMonitor<Env>({
  d1: (env) => env.MONITORING_DB,
  secrets: (env) => ({ SLACK_WEBHOOK_URL: env.SLACK_WEBHOOK_URL }),
  notifiers: [slack({ webhook: '${SLACK_WEBHOOK_URL}' })],
});

export default { fetch: monitor.fetch, scheduled: monitor.scheduled };

Assertions cover status code, headers, body, response time, and jsonPath — the last of which is what turns a health endpoint that returns {"db":"ok","queue":"degraded"} into an actual signal rather than a 200.

Why it exists

Triptech runs production APIs on Cloudflare Workers, and the monitoring options are roughly: a SaaS that bills per check and knows nothing about your account, or a Worker you write yourself and then never quite finish. clawdwatch is the finished version of the second one — small enough to read, deterministic where it needs to be, and built so the interesting half of an incident can be handed to something that will actually go and look.

Read the source, or start at getting started.

This post replaces three older ones about the same clients. Those described sites that no longer exist, and they described them in the language of a brochure. This is what actually got built.

What was there before

The two sister bedding sites each carried a single monolithic Jekyll layout — 1,843 lines on Breeder’s Choice, 1,380 on Superior Shavings — pulling Bootstrap 5.3.3, jQuery 3.7.1, jQuery CSV, AOS 2.3.1 and a Font Awesome kit from five different CDN hosts before either rendered a word. Mower Corner was one page — one index.md — reaching for jsDelivr, cdnjs, unpkg, Google Fonts and Cloudinary.

None of that was unreasonable in 2015. It’s just that the entire budget of every page went to downloading generic scaffolding, and what came back out the other side looked like every other Bootstrap site, because it was one.

What replaced it, per site:

Hand-written, no framework, no jQuery, no AOS, no icon kit. Google Fonts and Analytics still load, and Mower Corner still serves its imagery through Cloudinary — I’m not going to claim a purity I didn’t ship.

Four businesses, four registers

The temptation with four clients at once is one design system in four colourways. These are four different trades in two states, and two of them are sister brands owned by the same person — which makes differentiating them the actual design problem, not a nice-to-have.

Mower Corner — “spec plate.” Their mark is a condensed wordmark standing in a strip of grass, unchanged since the 90s, and it’s genuine equity in Colac. So the grass stayed and got redrawn as a crisp silhouette; the sky gradient and the clip-art operator silhouettes that dated it did not. The structural device is the riveted spec plate off the side of every machine in the shop, carrying the shop’s own data — phone, hours, address. Cast iron #14181A, safety orange #F2600C, the traced grass green #2B7134.

Breeder’s Choice — “the bale and the spec.” A family business that has done one thing since 1994. A 14 kg bale with a moisture spec is a real product, so the product leads: real photography of their own pallets, and a kraft bale-label tag carrying the spec the way the printed wrap does. Logo greens #0B4423 / #0F6B36 on cream and kraft.

Superior Shavings — “mill-side.” The sister plant, in NSW softwood country, where the pitch is proximity — from the saw into the bale with minimal handling. So it got a deliberately opposite register: iron greys and the logo’s red #BF2025, a numbered mill line, and the product spec set as a mono bale ticket. Same structure as Breeder’s Choice, same ethos, different plant. The two sites cross-link rather than compete.

AC Service & Parts — refine, not replace. The paddock palette and harvest photography were already working. Rebuilding would have been the expensive way to end up somewhere similar, so the budget went where it counted: contrast fixed to AA on every ground, marketing claims softened to what’s verifiable, structured data repaired, heroes self-hosted, and the logo’s teal finally let out of quarantine.

The stockist finder belongs to the client

Breeder’s Choice and Superior Shavings are sold through produce stores, saddleries and pet retailers — 407 and 540 of them respectively. The owner maintains both lists as CSVs in his own Dropbox, and has for years.

The obvious engineering move is to pull that CSV at build time: it server-renders, it’s fast, it’s cacheable. I built it the other way on purpose. The CSV is still fetched in the browser at page load, because a build-time fetch means his edits don’t appear until something triggers a deploy — and the moment his workflow depends on my CI, I’ve taken his list away from him. He edits a spreadsheet; the site updates. That’s the whole contract.

What did change is what it costs. jQuery plus jQuery CSV became about forty lines of native fetch and a CSV parser that handles quoted fields properly. The state matching runs two passes — strict uppercase abbreviation first, then a case-insensitive recovery pass — because the live CSVs mix VIC, Vic and Victoria, and a strict parse silently drops those rows. That’s not defensive programming for its own sake; it’s the difference between a stockist appearing and a stockist not existing.

The list itself renders as indexable HTML across nine pages per site — an index plus eight states — with the Google map as an enhancement on top. The previous implementation drew everything into a map and nothing else, which meant several hundred Australian retail locations were invisible to search.

Shipping a redesign onto a live domain nobody has approved yet

This is the part I’d reuse anywhere.

Both bedding sites belong to one owner, and a redesign he hasn’t seen can’t go live — but a preview he has to take my word for isn’t a preview either. Staging on a .github.io URL means he’s reviewing something that behaves subtly differently from the real thing: different domain, different SSL, no CNAME, no analytics.

So the new design shipped to his own live domain, at /preview/, months before he approved it — while the 2013 site kept serving / untouched. Four things made that safe:

1. Every preview page carried noindex: true and sitemap: false in front matter, so Google never saw a half-approved site or a duplicate of the live one.
2. The old layout stayed the default. The new pages opted into new layouts by path; / kept rendering the old one until the day of the flip.
3. Config keys were deliberately duplicated. The old layout read stockistscsv and phone_international; the new templates read stockists_csv and phone_intl. Both sets lived in _config.yml, each commented with which layout owned it, so cleaning up the new names could never quietly break the page that was actually serving customers.
4. The 404 page was rebuilt standalone — its own markup, its own inline CSS, linking only home and the phone number. An earlier version inherited the redesign’s header, which meant any bad URL on the live domain leaked the unreleased site and offered navigation to routes that didn’t exist yet.

The flip itself, once he signed off, was mechanical: move preview/* to the root, repoint the URL front matter, drop the two release gates, delete the old layout and the legacy aliases it alone read. One commit per site. The design had already been running on the real domain for weeks, so there was nothing to discover at go-live.

Mower Corner, specifically

Mower Corner went live back in early August and got the most functional work, because a shop has state a bedding manufacturer doesn’t.

The header carries a live open/closed readout — “Open now · until 5pm”, or “Closed · opens tomorrow 8:30am” — computed client-side against the shop’s real trading hours in Melbourne time, with today’s row marked in the hours plate. It’s the single most-asked question about a physical shop and it was previously answered by a static table you had to cross-reference against your own watch.

Four service pages were written against real local search demand rather than invented: chainsaw service and sharpening, mower servicing and repairs, spare parts, and pickup and delivery. The range picked up STIHL’s iMOW robotic mowers, and Click & Collect is branded STIHL-only throughout — they’re a STIHL dealer, and the online-order pipeline is STIHL’s, so implying you could get a Victa through it would have generated exactly the wrong phone calls.

Two implementation notes I’d defend anywhere:

• The mobile menu is a <details> disclosure. It opens and closes with no JavaScript at all. The 28 lines of JS behind it only add the niceties — Escape to close, outside-click, closing on same-page anchors — so a script failure degrades to a working menu rather than a hamburger that does nothing.
• Reveal-on-scroll replaced AOS with an IntersectionObserver, and it checks prefers-reduced-motion first. If the visitor has asked for less motion, or the browser lacks the API, everything is simply visible.

Source → github.com/triptechtravel/thinkbot · Docs → triptechtravel.github.io/thinkbot

Why it exists

A health endpoint returned a 500 whose body named the failing dependency. It did that for five hours. The alert that fired carried expected 200, got 500 and nothing else, because the monitor read the body to evaluate the assertion and then threw it away.

The cause was sitting in a response nobody kept, and nobody was awake to read it.

clawdwatch now keeps that body. thinkbot is the half that reads it at 3am — the step monitoring genuinely cannot do on its own. A check can tell you an endpoint returned 500. It cannot tell you that a pull request merged eleven minutes earlier, that a Sentry exception first appeared inside that window, and that a Datadog metric stepped rather than wobbled at the same moment.

The division of labour

The clawdwatch post made the case that detection should be deterministic — a threshold, a state machine, a maintenance window, and no model anywhere near the decision about whether you have a problem. thinkbot is what that decision buys you. It is the other side of a line drawn on purpose:

Detection is code you can read at 3am. Explanation is the part where a model earns its keep.

Once something is definitely broken, the work is reading five systems and noticing what they have in common. That is genuinely well-suited to a model, and it is nobody’s favourite job at 3am.

What triage actually looks like

The system prompt is an ordered procedure, not a vibe:

1. Confirm it is still failing. runCheckNow is cheap and stops you explaining an outage that has already passed.
2. Look at the history. A single blip and a sustained outage call for different responses.
3. Look for what changed. Most outages follow a deploy, so recent merged PRs on the relevant repository is usually the highest-value call.
4. Look for corroboration. An exception that started inside the same window, or a metric that stepped rather than wobbled.
5. Check whether this endpoint has failed before, and what was concluded last time.

Then it says what it found, in one short paragraph, citing the specific thing — the PR number, the exception, the ratio. And if the evidence does not support a cause, it says the cause is unclear and lists what it ruled out.

That last clause is the one that matters:

A confident wrong answer sends someone to the wrong service and costs more than an honest “unclear”.

Conclusions worth keeping get written back to the incident with annotateIncident, using the short-lived signed links that arrived with the alert — so the agent records what it concluded without holding any standing credential.

Silence is a valid outcome

If triage found nothing that explains the failure, thinkbot replies with the single word NOTHING, and the channel gets nothing at all.

This is the design decision I would defend hardest. An agent that always produces a paragraph will always produce a paragraph — and under an incident someone is actually trying to read, filler is worse than absence. It reads as commentary. It buries the alert. An empty channel is information: it means the automated pass found nothing, and a human should look.

Getting this right required saying it three times over — in the prompt, in the return contract (text.trim() empty means “nothing worth saying”), and in each channel’s decision about whether to post. It is easy to accidentally build an agent that cannot shut up.

Two transports, one triage path

Service binding, preferred. If clawdwatch runs on the same Cloudflare account, it calls thinkbot’s AlertInbox entrypoint directly:

"services": [
  { "binding": "AGENT", "service": "thinkbot", "entrypoint": "AlertInbox" }
]

The platform authenticates the call. No shared secret to rotate, no public endpoint to defend.

Signed webhook, for anything that cannot use a binding. POST /hooks/clawdwatch verifies an HMAC over timestamp.body using clawdwatch’s own verifySignature rather than a local reimplementation — because two implementations of one signature scheme is a bug with a delay fuse.

And the sharp edge, stated in both repos because it is exactly the sort of thing that quietly becomes a vulnerability: an RPC call carries no signature. Authenticity comes from the binding. So the shared triage path never assumes one was checked.

Failing CI runs get triaged the same way

POST /hooks/e2e takes a signed report from a CI runner when an end-to-end suite fails.

A service binding is not available here — bindings are same-account only, and a GitHub runner is not on the account — so this is HMAC under thinkbot’s own header names, keyed by E2E_WEBHOOK_SECRET. That is deliberately a different key from the monitoring inbox: a CI runner is a different sender in a different trust domain, and leaking one key must not grant the other.

The payload is deliberately not a clawdwatch AlertEvent. A test run is not a synthetic check — there is no incident to annotate and no signed links to act on — so forging one would hand the agent a prompt telling it to call annotateIncident against an incident that does not exist. It carries the repo, the commit, the run URL, and the failures the reporter saw.

The split is the point. The runner holds evidence no Worker can reach — which specs failed and what they asserted. thinkbot holds the credentials the runner should not, and answers what changed around that commit.

Two details earned the hard way:

• loadError with no failures is a distinct incident. The suite never ran, so it says nothing about whether the site is healthy. Reporting that as “0 tests failed” is how a real two-night outage read as noise.
• This path always posts, unlike monitoring triage. There is no second notifier behind it, so silence would mean a failing nightly suite simply disappears. The headline is the floor; the triage paragraph is what gets added on top.

Exercising the path that is not exercised

Removing the CI workflow’s own Slack step left exactly one delivery path and no way to tell whether it still works. You find that out during the outage the alert was meant to announce.

So a report may set "probe": true. It travels the same route and posts the same way — a probe down a different code path proves that path works and nothing whatsoever about the one a real failure takes — but it is labelled first and unmistakably, and it skips triage entirely.

That last part is not an optimisation. An agent asked to explain a non-event will invent one.

A thing worth copying: keep the dumb notifier

From the production configuration on the clawdwatch side:

notifiers: [
  slack({ webhook: '${SLACK_WEBHOOK_URL}' }),
  rpc({ name: 'thinkbot', binding: (env) => env.AGENT }),
]

Slack is listed first and deliberately kept. If the assistant is the only alert path, then an assistant outage is a monitoring outage — and you find that out at the worst possible moment. A failing notifier cannot affect the other one, so the plain, stupid, model-free Slack message always goes out. The agent’s paragraph is an addition to it, never a replacement for it.

If you take one idea from either of these projects, take that one.

Static assets are a second front door

thinkbot holds a GitHub PAT, Datadog and Sentry keys, and write access to monitoring incidents. workers_dev and preview_urls are off, and every inbound route verifies its caller before doing any work.

That was true of the routes and still not true of the deployment, and the gap is worth writing down because it generalises to any Worker with a static bundle in front of it. Assets are matched before the Worker runs. So an asset bundle answers requests that no route guard ever sees — and while one was deployed here, that single fact produced three separate bugs:

• an unauthenticated GET to the hostname was served from the bundle, in a route guard’s blind spot;
• /health answered 200 with index.html instead of the handler — a liveness endpoint that cannot fail, reporting healthy straight through an outage;
• and every signed inbox returned 405, because the asset handler rejects non-GET rather than falling through. Monitoring never noticed, because it reaches thinkbot over the RPC binding, which does not pass through assets at all.

The fix is not another path in run_worker_first — that just moves the boundary and leaves the rest of it in place. It is having no assets at all: with the bundle gone, every request lands on the code that checks its caller.

There is a local trap attached, worth knowing if you ever mix Vite and Wrangler: vite build writes .wrangler/deploy/config.json, which redirects wrangler at dist/. Both are gitignored, so a stale redirect will keep deploying the old bundle while silently ignoring your edits to wrangler.jsonc. That is how a bundle you thought you had removed stays in production.

It is not about our estate

Every source is optional, and one with no token configured reports that it is not configured rather than failing the triage. There are no built-in defaults for the GitHub owner or the Sentry org — a default would mean an unconfigured deployment quietly querying somebody else’s organisation.

What a deployment knows about itself lives in ESTATE_NOTES: free-form prose appended to the system prompt describing which repositories matter, what the Sentry projects are called, which service owns what. It is read by a model, not parsed, so plain sentences are fine.

Baking one organisation’s inventory into the prompt is what makes an otherwise general tool unusable by anyone else — and stale for its original owner the first time the estate changes.

Try it

git clone https://github.com/triptechtravel/thinkbot
npm install && npm test

Then give it tokens for whichever sources you use, tell it who you are with GITHUB_OWNER and SENTRY_ORG, and point clawdwatch at it. SETUP.md is the short version; SECURITY.md is worth reading first if you plan to turn on captureBodyOnFailure for endpoints whose error paths can return personal data.

Source → github.com/isaacrowntree/sustain (MIT) · clone it and pnpm dev — there’s no hosted demo, and the app never sends your data anywhere.

Why this exists

I played trombone at a decent level and then stopped for years. I also snore, enthusiastically, and the suggestion that came back was: play the didgeridoo.

That sounds like folk advice. It isn’t. Puhan et al., BMJ 2006 is a randomised controlled trial: 25 people with moderate obstructive sleep apnoea, four months, and the didgeridoo group’s apnoea–hypopnoea index dropped by about half, with significant improvements in daytime sleepiness and in how much their partners were disturbed. It later won the 2017 Ig Nobel Peace Prize, which is the kind of accolade that makes people assume the science is a joke. The science is fine. The mechanism is unglamorous: circular breathing and sustained droning are resistance training for the upper airway, and the same effect shows up in the myofunctional-therapy literature with no instrument involved at all.

Here’s the part nobody puts on the poster. The trial’s actual dose was at least 20 minutes a day, at least five days a week, for four months — and the participants beat it, averaging about 25 minutes across nearly six days. That is roughly 120 sessions. The intervention isn’t the didgeridoo; the intervention is turning up 120 times.

So the problem I actually had was not “how do I play the didgeridoo.” It was “what gets me to session 87.”

Did this already exist?

Mostly, and it’s worth knowing about before you read another word of mine.

Silent Sleep Training (~US$349) is the closest thing, and its pedigree is remarkable: it’s built by Alex Suarez’s company — Suarez is the didgeridoo teacher who taught the participants in the BMJ trial. It bundles a purpose-made silicone instrument with an app that listens and gamifies a 15-minute daily session. Didge For Sleep (US$167) is the other half of the idea: a real travel didgeridoo plus a sleep-apnoea-specific video curriculum, but no app, no tracking, no session engine.

What doesn’t exist is the overlap: a real didgeridoo, a 30–40 minute guided session, and a microphone that verifies you actually played. The snoring-adjacent apps (SnoreGym, SnoreFree, Airway Gym, Soundly) do daily arcs with their own mouth exercises. The mic-verified music trainers (Yousician, tonestro) need pitched repertoire and can’t score an unpitched drone. And a GitHub search for a didgeridoo trainer or a circular-breathing trainer returns nothing usable — the repos named “didgeridoo” are an acoustic-impedance calculator and, I promise, an operating system.

So: build. But build the general thing, because a drone hold and a trombone long tone are the same exercise wearing different hats.

No levels, no XP, no points

The easiest version of this app is a Duolingo skin: streaks, experience points, a little celebration when the bar fills. I deliberately didn’t build that, and the reason is that the didgeridoo already keeps score, honestly. Your longest unbroken drone is a real number about your actual throat. Wrapping fake currency around a real measurement doesn’t add motivation — the overjustification research suggests it displaces it.

So progression is the curriculum:

• Records, not scores. Longest drone. Longest unbroken sound. That’s the whole reward economy, and every figure on screen is a measurement of something you did.
• Unlock by demonstration. The next drill appears when you show you can do the one before it, not when a counter fills. Circular breathing on the instrument stays locked until you’ve done the water-and-straw drills and can keep the bubbles unbroken through a sniff.
• Perfect weeks, not daily streaks. The evidence protocol is five days a week, so rest days are drawn into the calendar as scheduled, not as failures. There is no chain to break — the habit-formation work found a single missed day doesn’t impair habit formation, and an app that punishes you for it is inventing a problem to sell you the cure.
• Missing a day is recoverable, not fatal. Miss a weekday and the weekend offers a make-up session that puts the week back to whole. This is the feature I’d argue hardest for: nearly every habit app treats the miss as the punishable event, when the miss is just Tuesday being Tuesday. The thing worth protecting is the return.
• You can start on a Wednesday. Join mid-week and week one asks only for the sessions that were still ahead of you — a four-day week counts as perfect if you did four. Nothing is more discouraging than a program that opens by scoring you against days that had already gone by the time you downloaded it.

That bar chart at the bottom is the sixteen-week journey, one bar per week, height being sessions completed and colour being the phase you were in. It’s the only “gamified” element in the app and it’s just a picture of the truth.

What a session looks like

Thirty-five minutes of staring at a countdown is how a practice habit dies. So a session is a lane of timed segments flowing toward you — the Guitar Hero sustain bar stretched to the length of a breath.

Most of what’s in that screen is stolen, on purpose, from software that solved these problems years ago:

• The strike line is a physical object, not a marker. Every rhythm game since Guitar Hero converges the lane into something that reacts — so segment boundaries flare as they’re consumed, and the ember is the sharpest, warmest point on screen.
• The played trail rides behind it, an edge-on seismograph of what the microphone actually heard: bright where the drone lived, ash where it dropped. Guitar Hero’s real insight about long notes is that a hold is never static — you get continuous proof the thing is still listening.
• Rests get a breathing halo, paced like the Apple Watch Breathe animation, because in a didgeridoo program the rests genuinely are breathing exercises.
• The next segment brightens about one breath before it arrives, with a soft chime — telegraphing, not a countdown, so you prepare instead of react.
• A long unbroken run escalates the world: the stars brighten, the dust thickens. Flower and Tetris Effect both make progress an environmental change rather than a meter, and it turns out that’s exactly right for something you do with your lungs.
• The HUD fades after about ten seconds of stable playing and comes back on any input. The research on this is unambiguous that auto-hiding chrome without a way to summon it back just annoys people.

The microphone is optional, and local

Verification is tiered, and the app is fully usable at tier one:

1. Timer. Honour system. No microphone at all. This is how the BMJ trial ran — those participants had a paper diary.
2. Energy. Raw loudness, about twenty lines of code, enough to make the scene breathe with you.
3. Pitch. pitchy’s McLeod Pitch Method over a 4096-sample window, gated on clarity, frequency band and pitch stability, with hysteresis so a wobble doesn’t stop the clock. This is what turns “I think I held it about half a minute” into a measured record.

Three practical things I learned building tier three. You must ask for the microphone with echoCancellation, noiseSuppression and autoGainControl all off, because WebRTC’s noise suppressor classifies a sustained tone as background noise and quietly attenuates the exact signal you’re trying to measure. Don’t use FFT bin peak-picking — at a normal FFT size the bins are about 23 Hz wide, which is useless against a 60–90 Hz fundamental. And it doesn’t matter that laptop microphones roll off below ~100 Hz, because autocorrelation-family algorithms recover the period from the harmonic stack even when the fundamental is physically missing from the recording.

All of it runs in the browser. Progress lives in IndexedDB, recordings live in IndexedDB, and there is no account, no server and no telemetry. The one network request the app makes is for a webfont.

Multi-instrument by construction

An instrument pack is data, not code: metrics, a declarative curriculum of phases and drills, prerequisites, and an analyzer spec. The engine compiles a pack plus today’s date into a session. The didgeridoo pack encodes the trial’s protocol — drone holds, the circular-breathing drill ladder, lip and vocal-tract work — across sixteen weeks in four phases.

The useful accident is that the pitch analyzer is instrument-agnostic. A trombone pack is the same detector with a different frequency band, and it gets more out of it, because a detector that reports exact hertz can measure long-tone steadiness in cents. Guitar is the genuinely hard one — chord detection is a different problem — and it’s marked help-wanted in the repo rather than pretended at. You don’t need to write code to contribute a pack; a curriculum written out as text is the actual work.

Where it’s at

It runs. The engine, the didgeridoo pack, all three microphone tiers and the web app work end to end, with tests across the packages. I’m running it locally for now rather than hosting it — for a local-first app with no backend, pnpm dev is the deployment.

The instrument is sorted too: a PVC didgeridoo from African Drumming. Plastic is not a compromise here — the BMJ trial handed its participants plastic instruments precisely because they’re easier to learn on, and the therapeutic effect comes from what your airway is doing, not from what the tube is made of.

So there’s a curriculum, a coach, an instrument, and an empty progress file. The piece I’m most looking forward to is the one I can do least about, because it takes exactly sixteen weeks: the app records thirty seconds of you on day one and plays it back to back with day one hundred and twenty. Ask me in December.

A note worth making plainly: the didgeridoo is an Aboriginal Australian instrument, known as the yidaki to the Yolŋu people of north-east Arnhem Land, with a ceremonial history that long predates its use as a snoring intervention. Sustain is a practice timer for a breathing exercise. It makes no claim on any of that.

Header photo by Wil Stewart on Unsplash.

Live demo (drop a 3D model, watch it become a drawing) → isaacrowntree.com/linework · Source → github.com/isaacrowntree/linework (MIT)

Why this exists

I’ve been building a “will this part actually fit your bike?” planner — the sort of tool that tells you whether a given rear shock, dropper, or ebike motor bolts onto your frame before you spend the money. It grew out of my earlier bike-part-planner engine, and the thing that makes it click is the drawing: an exploded, service-manual-style diagram of the bike with every part ballooned to a parts list, that you can grab and rotate to see how the pieces sit in space.

So I needed to render exploded technical diagrams. Rotatable ones. With dimension lines, callout balloons, and a title block — the whole drafting vocabulary. And I went looking for a library, fully expecting to find three of them.

There aren’t any.

The empty category

The 3D tooling world splits cleanly in two, and neither half wants this job.

On one side is three.js and the WebGL universe. It’s superb — at shaded surfaces, lights, materials, cameras. But everything that makes a technical drawing read well is a fight there: crisp 1px strokes (WebGL lines are a known pain), dashed hidden-lines, line-weight hierarchy, and text callouts that have to live in a separate DOM overlay layer bolted on top of the canvas. I’d be spending all my effort making a rendering engine stop doing the thing it’s good at. It’s a cannon for a job that needs a technical pen.

On the other side are the friendly pseudo-3D toys — and there’s really only one that fits the aesthetic, Zdog. Lovely flat-shaded look, tiny, designer-friendly. But its last release was in 2019, it’s self-described as a beta, and text is a third-party plugin. Dimension lines, balloons, and leader text aren’t a nice-to-have for an exploded diagram — they’re half the drawing. A renderer that can’t do text can’t do this.

Nothing in between ships parametric technical illustration: exploded views, callouts, paper-space annotations, styling you control. So I wrote it.

What it actually is

The pipeline is the textbook one, implemented honestly rather than faked:

3D points → yaw/pitch rotation → perspective projection → painter’s-algorithm depth sort → SVG strings.

The important word is sort. Paint order is recomputed every frame from the actual depth of every shape, so when you rotate the model the near stay passes in front of the far one correctly — it’s real occlusion, not hand-authored layering that falls apart at a new angle. Circles in a plane project to correct ellipses (wheels, bearing races), boxes cull their back faces, and cylinders keep their end-caps readable rather than thinning to invisible slivers edge-on. That last one is a deliberate lie in service of legibility, which is exactly the kind of call a drawing library gets to make and a physics-accurate one doesn’t.

Here’s a bearing assembly, exploded — and this exact SVG was rendered by the library, in Node, with zero client-side JavaScript:

Point it at a model you already have

That pillow block is hand-authored — but the bigger unlock is that linework will import. Hand it a glTF, OBJ or STL mesh (or a three.js BufferGeometry) and it recovers the drawing for you. A shaded 3D model carries no lines; its form lives in where the surface bends. So linework/import keeps only the feature edges — the outline and the hard creases, nothing from the smooth interior of a face — which is precisely the set of lines a draftsperson would ink. One call does it:

import { parseGLB, meshToShapes } from "linework/import";
const meshes = parseGLB(await file.arrayBuffer());   // parseOBJ · parseSTL · fromBufferGeometry
const shapes = meshToShapes(meshes, { angle: 25, fit: { cx: 400, cy: 300, size: 440 } });

Even STEP and IGES work — not by embedding a CAD kernel (tessellating trimmed-NURBS B-reps is a job for OpenCASCADE, not 180 lines), but because linework consumes meshes: run the file through occt-import-js and a ~10-line fromOcct() adapter maps the result straight in. The heavyweight kernel stays an optional peer; the core stays tiny.

Here’s a CC0 street lantern — 5,394 triangles of shaded mesh — reduced to ~2,500 feature edges and rendered as a rotatable line drawing, again entirely in Node:

That reframe is what made this worth releasing. “Hand-code a 3D scene” is a small audience; “bring any model you already have and get a technical drawing” is a large one — and every 3D tool on earth exports glTF. Drop your own .glb on the live demo and watch it happen; it runs in your browser, nothing uploaded.

Why SVG strings, specifically

This is the whole bet, and it’s what neither neighbour can offer. Because the output is just markup:

• You style it. The library emits class names you define — it never bakes in a colour, a stroke width, or a font. Theme it with CSS variables, restyle it for dark mode, print it.
• It’s crawlable and accessible. Text callouts are real text. A screen reader and Google both see them.
• It renders on a server. render() is a pure function from shapes to a string — no canvas, no DOM, no browser. A static-site generator can emit finished 3D-looking diagrams at build time with nothing shipped to the client. The hero image in the repo is generated this way; the live demo uses the same code to orbit under your pointer.

Making it nice to author

A renderer you dread writing scenes for is a renderer you don’t use, so linework ships a small authoring layer on top of the raw primitives. It reads like drafting instead of like assembling tuples — context blocks scope which animated part, which CSS tag, and which layering everything inside them belongs to:

import { sketch } from "linework/sketch";

const s = sketch({ yaw: 0.5, pitch: 0.16, f: 1500, cx: 460, cy: 320 });

s.box(300, 380, 320, 42, 40, 80);                 // base plate

s.part("housing", 'class="prt"', () => {          // one <g>, one sort unit
  s.cyl([460, 218], 70, 34, -34, "ink");          // bearing body
  s.bias(0.6, () => s.cap([460, 218], 34, 32, "ink"));
});

s.note("320 mm", [460, 452]);                      // paper-space annotation
el.innerHTML = s.render();                         // depth-sorted SVG

And because grabbing a drawing to spin it is the same twenty lines of pointer-capture boilerplate every single time, that’s a one-liner too — linework/orbit handles the drag, the clamps, the reset, and a gentle idle sway so an embed is alive before anyone touches it. Install and kick the tyres:

npm i linework          # on npm — ESM, types included, zero runtime deps
npm test                # projection, parallax, sorting, culling, edge extraction

The lesson

There’s a reflex, when a problem looks 3D, to reach for the big 3D engine — and then spend a week teaching a flight simulator to draw a neat labelled diagram. The actual need was smaller and more specific than the tool everyone reaches for, and nobody had written the small specific thing because it sits in the crack between “serious 3D” and “cute toy.”

That crack is where a lot of good little libraries live. This one is ~180 lines, has no dependencies to rot, comes with tests so a wrong drawing fails loudly instead of just looking plausible, and it’s MIT on GitHub — take it, theme it, drop in your own model and draw with it. The bike planner it was born for is coming next.

Docs & live demo → isaacrowntree.com/durable-sync · Source → github.com/isaacrowntree/durable-sync (MIT) · npm → durable-sync

Why this exists

If you want offline-first sync in 2026, the credible engines — Zero, Electric, PowerSync — all want Postgres and a long-running container. On an all-Cloudflare stack that’s a second backend to run, forever. They’re genuinely better than this at almost everything; that’s not the point. The point is the tax.

Meanwhile the platform already hands you the hard part for free: a Durable Object is a single-threaded ordering point, per user. That’s the exact thing everyone else reaches for Postgres to get. Give each user their own DO with idFromName(userKey) and write-ordering stops being a distributed-systems problem and becomes a language feature. Once you have that, the rest of “sync” is just an outbox, a cursor, and a great deal of care about what “success” means.

So durable-sync is small on purpose. It’s a primitive, not a database. No conflict resolution — ops are immutable facts appended to a log, and if your writes are events (“this workout happened”) rather than edits, you don’t have a conflict problem. HTTP, not WebSocket — a socket is useless in a basement, and gyms are Faraday cages, so sync happens when the app is open and the network exists. Bring your own IndexedDB.

The four things that are easy to get wrong

Here’s the honest reason this is a package and not a gist. Sync that looks like it works and sync that actually works are separated by a handful of failures that are individually obvious and collectively brutal — because every one of them fails silently. Each of these shipped to production in the app this came from:

• res.ok is not evidence. Behind an auth proxy like Cloudflare Access, an expired session redirects to a same-origin login page — which fetch quietly follows and reports as a 200. Drain your outbox on res.ok and you’ve just deleted writes that never reached the server. durable-sync validates that a reply actually came from the journal before it touches the queue.
• A cursor only means something against the log that issued it. Reset the log and every client is holding a sequence number from a log that no longer exists — usually pointing past the rebuilt one, so seq > cursor matches nothing and the client silently never syncs again. Every pull carries an epoch; a client that sees a new one replays from zero. Apply is idempotent, so replay is cheap.
• Pulling is not always safe. In Rampset, a remote op landing mid-workout rewrote the working weight that the finish logic reads back — and silently dropped a 5×5 to a 3×5. Pushing is always safe; pulling isn’t. A canPull() gate defers the pull while the user is in the middle of something, and eventual consistency makes waiting free.
• The gate has to be the only door. The journal is addressed by a server-side identity while the ops carry whatever the client selected — so writing as the wrong one files data under someone else, permanently, in an append-only log. The engine has a canWrite() gate; the transport that could bypass it simply isn’t exported. There is one way in.

None of these are things you’d think to test for until they’ve bitten you. That scar tissue is the actual product.

What you write

The server is a Durable Object you extend and export. Its methods are native DO RPC — you call them on the stub, typed, with no request-building in between:

// worker.ts
import { SyncJournal } from "durable-sync/server";
export class Journal extends SyncJournal {}

// your route — forward exactly the methods a client should reach
export async function POST(req: Request) {
  const { ops } = await req.json();
  const journal = env.JOURNAL.get(env.JOURNAL.idFromName(userKey));
  return Response.json(await journal.push(ops));
}

Note what you don’t forward: reset(). The DO has it, but a client reaches only the methods you wire to a route — that’s the whole access model, no router second-guessing you. The client is local-first: commit to IndexedDB, queue the op, let the network catch up.

import { createSync, localStorageCursor } from "durable-sync/client";

export const sync = createSync({
  endpoint: "/api/sync",
  outbox: dexieOutbox(db.outbox),          // durable — it may hold the only copy
  cursor: localStorageCursor("myapp.cursor"),
  async apply(op) { /* idempotent: an op can arrive twice */ },
  canPull: async () => !(await somethingInProgress()),
});

There’s a runnable example in the repo — a Worker plus a browser client with a real IndexedDB outbox — and a live demo on the docs site where you can cut the network, watch writes pile up in the outbox with no sequence number, then reconnect and watch them drain. That widget is the library.

When to use something else

Being clear so you don’t adopt it and find out: no conflict resolution, no pagination (a pull returns everything after the cursor in one response — fine for thousands of ops, not millions), no live push, no auth, and nothing runs while the app is closed because Safari has no Background Sync. If you need those, you want a real sync engine and the Postgres that comes with it — there’s an honest comparison against Zero, Electric, PowerSync and Cloudflare’s own partysync on the docs site. If you need two people editing the same record to merge, you want a CRDT — Yjs or Automerge.

The slice where durable-sync is the right answer is narrow: you’re all-Cloudflare, your writes are events, and the app has to keep working on a train. Inside that slice, it’s a few kilobytes and no second backend.

The lesson

Extracting a library from a working app is a good forcing function: it makes you separate the parts that are genuinely reusable from the parts that were only ever about your problem. What surprised me is how little of the value was in the code. The op log is a hundred lines. The outbox is an interface. Anyone could write those in an afternoon — and then spend the next year discovering, one silent data-loss bug at a time, the four things above.

That year is what’s in the package. The code is small; the scar tissue is the product.

Source → github.com/isaacrowntree/sovereign-ibkr-fund (Apache-2.0 OR MIT) · Built on bezant

Why it exists

I had a portfolio problem that a lot of people quietly have: one winner had run so hard it became 40%+ of the book. Great on the way up, a single point of failure on the way down. Fixing that by hand — trimming the concentration, funding a diversified target, doing it without fat-fingering a real order — is exactly the kind of repetitive, high-stakes, easy-to-get-wrong work you should not be doing manually at 11pm.

So the fund isn’t a get-rich bot. It’s a discipline engine: hold a target allocation, notice when reality drifts from it, and make the smallest correct trades to close the gap — with enough guardrails that a bad market-data tick or a logic bug can’t do real damage.

bezant already gave every language typed access to IBKR’s Client Portal API. Sovereign is what happens when you build an actual fund on that foundation and take the “it’s real money” part seriously.

Nine agents, no LLM calls

The agents are plain TypeScript --once processes — deterministic, no model calls in the loop. That’s deliberate: I want a trade decision to be a pure function of the inputs, reproducible and auditable, not a sample from a distribution.

Under the hood there’s a real quant toolbox: Hierarchical Risk Parity, Black-Litterman, risk-parity, Ledoit-Wolf shrinkage covariance, a regime overlay, and vol targeting — all backtestable.

The guardrails are the point

Anyone can write a loop that places market orders. The interesting engineering is everything that stops it from doing something stupid with real money:

• Validation-first execution. Before it will batch a rebalance, the executor proves a live fill on the single smallest order and confirms it against IBKR’s own execution records. No confirmed fill, no batch.
• Executions are authoritative. Fills are reconciled against IBKR’s execution log, and the trade ledger is idempotent — the same fill can never be recorded twice, even if a confirmation arrives by two paths.
• Hard caps. Absolute backstops on per-order notional, per-order % of NAV, and per-run notional — independent of the sizing math. A garbled input can’t size a monster order.
• Data-sanity gates. If NAV or a price tick looks impossible (zeroed, or a 100× move with no cash flow), the strategist refuses to generate orders that cycle rather than trade against garbage.
• Drawdown control. De-risk and hard-stop thresholds that pull exposure down when the book is bleeding.
• State lives outside the checkout. Positions and ledger sit in STATE_DIR, never entangled with the code — so a deploy can never clobber your holdings.

Most of these exist because the honest way to build this is to assume your own code will misbehave and make sure the blast radius is bounded when it does.

Standalone, or under whatever you run

Every agent is just node dist/agents/<name>.js --once. That one contract means Sovereign doesn’t care how it’s scheduled:

• Built-in scheduler — npm start runs the whole fund on a cadence, zero dependencies.
• cron / systemd — point timers at the --once scripts; examples in deploy/.
• Any orchestrator — set ENABLE_SCHEDULER=false and let your platform drive the same scripts.

The core never imports the scheduler, so it genuinely runs both ways.

Your book stays yours

The public repo ships a generic sample portfolio (a diversified ETF template) and runs against paper out of the box. Your real allocation goes in a gitignored src/portfolios/local.ts that takes precedence automatically and never leaves your machine. Everything else — caps, thresholds, cadences, optimizer choice — is environment-driven. Nothing about your positions lives in the source.

Backtesting

There’s a full backtest engine (HRP, risk-parity, Black-Litterman, regime overlay, vol targeting). The historical dataset is gitignored — you generate it yourself from Yahoo Finance with npm run fetch-data, and the backtest suites skip cleanly until it exists, so a fresh clone is green on the first npm test.

Status and licensing

• v0.1 — runs end-to-end against IBKR paper accounts; the API and agent set will evolve.
• Dual-licensed Apache-2.0 OR MIT.
• Requires a running bezant gateway — that’s how it talks to IBKR.
• Not affiliated with Interactive Brokers. This is not financial advice. It places real trades — start on paper, and understand every guardrail before you flip TRADING_MODE=live.

If bezant was about making IBKR programmable, Sovereign is about making a portfolio governable — trading on rules you can read, with brakes you can trust. If you’re running money through IBKR and you’d rather it followed a written policy than your 11pm instincts, clone it and point it at your own bezant. Contributions welcome — especially on the optimizers and the risk engine.

Site & guide → isaacrowntree.github.io/rampset · Source → github.com/isaacrowntree/rampset (MIT)

Why this exists

I was a StrongLifts user from 2015 to 2026 — 1,221 logged workouts, a grandfathered subscription I’d held since 2018, and honestly no complaints for over a decade. It did the one thing a 5×5 app has to do: tell me what to lift today and remember what I lifted yesterday.

Then a sign-in bug locked me out of my own subscription. I did the support dance — reinstalled the app, removed and re-added every Google account on the phone, followed every step of every linked help article, sent screenshots. After a morning of back-and-forth, support resolved it their way: they cancelled my subscription. I hadn’t asked for that. The legacy plan I’d been quietly grandfathered on for years ceased to exist the moment it was cancelled, and the only path back was the shiny new trial at roughly six times my old price.

Well — screw that. I write software for a living, my workout history exports to CSV, and it’s 2026: with Claude Code in the loop, “I’ll just build it myself” is no longer a threat you mutter into an email draft and delete. It’s a Saturday.

Three ways to train

The interesting design constraint wasn’t me — it was that my wife trains too, and her program is nothing like mine. That forced a better architecture than a clone would have had, and it’s why Rampset ships with three shapes on one engine:

• Program mode — classic StrongLifts-style 5×5, A/B, with variants from Lite to Ultra Max. The app prescribes: tap a plate-shaped circle to log a set, warm-ups ramp with per-side plate math, +2.5 kg linear progression, graduated deloads after time off, and the full stall protocol (5×5 → 3×5 → 1×5).
• Madcow mode — weekly ramped 5×5: heavy / light / intensity days, back-off sets, and a Friday PR set that sets next week’s top weight.
• Routine mode — a Strong-style layout: exercise cards with set rows (previous · kg · reps · ✓), everything prefilled from the last session, per-exercise rest timers, timed and bodyweight sets. You prescribe; the app remembers.

Same schema, same logging engine, same charts — the mode is just a flag on the program. Both of our full histories came in through CSV importers for the two incumbent apps’ export formats, auto-detected and idempotent, verified against the real files. And export gives it all back whenever you want — no lock-in is the whole point.

The parts I’m fond of

• Offline-first for real. Gyms are Faraday cages. IndexedDB is the source of truth on-device; the service worker keeps the shell alive; a screen wake-lock holds the display on mid-workout; and a rest timer keeps counting on a wall-clock deadline even when the phone locks, then rings a WebAudio bell it synthesises itself. No signal just means sync happens later.
• Sync without a database server. Each person gets their own SQLite-backed Durable Object — a single-threaded, per-user journal that makes write ordering a non-problem instead of a distributed-systems problem. Finished workouts push as idempotent ops; fresh devices pull. R2 holds dated snapshots underneath as the disaster layer.
• No auth code, no personal data in the repo. Identity is handled entirely by Cloudflare Access (a free zero-trust login) — the app itself contains zero auth code. Lifters are defined by an env var that’s gitignored; the committed code carries only generic program templates. Nothing about you or your training lives in the source.
• TDD the whole way. 240 tests: the progression engine, both CSV dialects, the sync journal, backup round-trips, even “a rest timer survives navigating away mid-countdown.” The engines are pure functions in src/lib/, each with a failing-test-first workflow waiting — because editing them is the point.
• It feels like an app. OLED-black UI, floating dock, haptic ticks on set logging, sheets that close with the Android back gesture, splash screens, home-screen shortcuts. The bar for “doesn’t feel like a website” is a hundred small details, and a multi-agent review fleet is very good at finding all hundred.

Run your own

Rampset is built to be yours. Clone it, point it at your lifters, and either use it locally or deploy your own copy to Cloudflare:

git clone https://github.com/isaacrowntree/rampset
cd rampset
npm install
cp .env.example .env.local     # your lifters, units, starting weights
npm run dev                    # → http://localhost:3000
npm test                       # the engines, importers, store, components

To put it online, npm run deploy ships it to Cloudflare Workers via the OpenNext adapter; then you add a Cloudflare Access application in front of your Worker and allow your lifters’ emails. Workers and Access are free for a household. The one paid gate is R2 for cloud backups — the data sits inside R2’s 10 GB free tier, but Cloudflare wants a card on file to enable it. Don’t want to add one? Drop the BACKUPS binding: the app is fully offline-first on local IndexedDB, and Settings → Export hands you a complete CSV backup whenever you want.

The lesson

There’s a quiet lesson in this for anyone running a subscription product: your most loyal decade-long customers aren’t locked in by your data-export button being hard to find anymore. They’re one bad support interaction away from an export file, a coding agent, and a very productive Saturday afternoon.

And here’s the part that should really keep you up at night. This time they don’t even have to build it. The export importers, the two proven programs, the offline sync, the plate math — it’s all sitting in a public repo under a licence that says make it yours. Take it to the gym. Take it somewhere no subscription can reach you.

This is the story of how a tool I adopted became a tool I help maintain: how I found it, ran it in production, watched it fall over for a reason that wasn’t really its fault, went looking for help from a company that no longer existed, and ended up forking it, fixing it, and extending it.

Why I reached for Sequin

The problem Sequin solves is the boring, load-bearing kind. The POI, review, and translation data lives in Postgres (on Neon). Search runs on Typesense. Several other services — a translation workflow, a delta cache for the mobile app — need to know the instant a row changes. The naive version of this is a spray of application-level hooks and cron jobs that drift out of sync the moment anything fails silently.

CDC inverts that. Postgres already writes every change to its write-ahead log; Sequin reads the log and fans each change out reliably, with retries and backfills, so the index and the database can’t disagree for long. The whole thing is a dozen sinks driven by a single declarative sequin.yaml, version-controlled and applied from CI:

   Postgres (Neon)  →  Sequin  →  Typesense collections (POIs, reviews, translations)
   logical slot        transforms →  Webhooks (translation pipeline, delta sync)

The transforms are the nice part. A small Elixir function denormalises a POI’s feature list into boolean columns, builds language-suffixed fields (description_en, description_mi) from a tall translations table, and filters out non-public rows before they ever reach the public index. It’s the right amount of logic in the right place, and for a good while it just worked.

The night Dragonfly took Sequin with it

Then came outages that made no sense. Sequin would be humming along, and then — with no deploy, no traffic spike, no obvious cause — every consumer would stop at once. The sinks would go cold, the index would fall behind, and the only fix was a restart.

The correlation, once I traced it, was infuriating: it happened whenever Dragonfly — the Redis-compatible store Sequin uses for coordination, running as a managed service on Railway — redeployed itself. A routine maintenance update to a dependency was taking down the whole engine.

The mechanism is a textbook Erlang supervision-tree footgun. Sequin uses a Redis-backed mutex to elect a single leader across nodes — a MutexOwner GenServer that holds a lock and refreshes it before it expires. When Dragonfly restarted, the connection blipped, the mutex refresh failed, and MutexOwner did the most literal possible thing: it crashed. And because it sits under a supervisor with a :one_for_all restart strategy, its death took every sibling down with it — the entire runtime supervisor, all consumers, all sinks. A two-second Redis blip became a total outage that only a human restart would clear.

MutexedSupervisor (:one_for_all)
  └── MutexOwner  ✗  Redis blips → mutex refresh fails → crash
        ⇒ :one_for_all fires ⇒ every consumer & sink is torn down with it

That’s not really Sequin doing something wrong so much as an assumption — “Redis is always there” — that doesn’t hold on a platform where your Redis can redeploy under you at any moment.

Upstream had gone dark

So I did the normal thing: went to fix it upstream, or at least to ask. And found the lights off. The project had moved into maintenance mode, the company behind Sequin had wound down, and there was no one on the other end of the issue tracker to merge a patch or ship a release. A tool at the centre of a data pipeline had, effectively, been orphaned.

This is the quiet risk of building on someone else’s open source: the licence guarantees you the code, but nothing guarantees you a maintainer. When the maintainer disappears, you have exactly two choices — rip the dependency out, or adopt it.

So I adopted it

Ripping out CDC and rebuilding the sink pipeline from scratch would have been weeks of work to end up back where I started. Adoption was the better trade. The project got forked into github.com/triptechtravel/sequin, with its own build and release pipeline, and from then on it was treated as what it now was: code I own.

That meant the unglamorous infrastructure of ownership. A GitHub Actions workflow that builds a patched image and pushes it to GHCR. Deployment to Railway. Fixing the parts of the build that assumed a corporate CI — a missing cmake for a Kafka NIF, a Sentry DSN that was baked in at build time and now had to be optional. None of it is exciting. All of it is the price of being the maintainer instead of a user.

Fixing the crash properly

With the fork in hand, the Dragonfly bug got fixed at the root. MutexOwner no longer treats a Redis error as fatal. While it holds the mutex and Redis becomes unreachable, it now retries indefinitely with exponential backoff (capped at an hour) instead of crashing — Redis going down should degrade Sequin gracefully, never take it out. When Redis comes back, it re-acquires the lock and resumes as if nothing happened. The invalid GenStateMachine stop value that caused the original crash got corrected too, and the LiveView metrics pages were hardened so a Redis blip renders an empty chart instead of a MatchError.

The part I’m most pleased with is the test. It’s easy to write a unit test that mocks a Redis error; it’s much more convincing to simulate the actual failure. The integration test uses iptables to REJECT traffic to Redis mid-run — a real network partition, the same thing a Dragonfly redeploy looks like from the process’s point of view — and asserts that the MutexOwner survives the outage and recovers when the rule is dropped. That’s the difference between “handles a mocked error” and “survives the thing that was actually paging me.”

The outages stopped.

Then it went behind Cloudflare Access

Once you own a fork, you stop just patching it and start shaping it. The most recent addition: real SSO on the admin console.

Out of the box, self-hosted Sequin authenticates users with an email-and-password login sitting in its own Postgres table. For an internal tool that a whole team touches, that’s the wrong model — there’s already Google identity and Cloudflare Zero Trust in front of everything else. So the Sequin console went behind Cloudflare Access, and Sequin learned to trust it.

The mechanics, mirroring the same trusted-header pattern I’ve used for a Payload CMS:

• Cloudflare Access gates the console with a Google-SSO policy. On every request it forwards, it injects a signed JWT in the Cf-Access-Jwt-Assertion header.
• A new Elixir plug verifies that JWT — fetching the Access application’s public keys (JWKS), checking the signature, issuer, audience, and expiry — and then transparently signs the user in. First time through, it provisions the user just-in-time from the verified email, adopting any existing account so nobody lands in an empty instance. You never see Sequin’s own login screen.
• The tricky part is machine traffic. The search config is applied from CI, which authenticates with a token rather than a browser SSO session. Cloudflare Access lets you scope policies so interactive console traffic goes through Google while automated, token-authenticated callers are validated on their own credentials — so the SSO gate never breaks the deploy pipeline.
• Finally, the settings UI now reflects reality: it shows which identity provider you authenticated with and disables the email/password fields for SSO users, rather than presenting a dead form.

The whole thing is feature-flagged, so the upstream password-login behaviour is still the default for anyone else running the code. It ships as a normal patched image through the same GHCR-to-Railway pipeline as every other fix.

Owning a fork you didn’t write

There’s a version of this story that reads as a cautionary tale about depending on startups. I don’t think that’s the lesson. Sequin was — is — a genuinely good piece of engineering, and the fact that it could be adopted, read, fixed, and extended with SSO is entirely because it was open source. A closed SaaS that shut down would have left nothing but a migration deadline.

The real lesson is that “using open source” and “owning open source” are different commitments, and the gap between them can close overnight. When it does, the codebases you can actually take over are the ones written clearly enough to understand under pressure. Sequin was. You read the supervision tree, see why a Redis blip cascaded, and fix it — in someone else’s code that has quietly become yours.

If you’re travelling Australia or New Zealand, the search that lands you at the right campsite is riding on this pipeline. Grab CamperMate on iOS or Android — free, no account required.

Header photo by v2osk on Unsplash.