We had lived with it for a year. What changed was that we started using Claude Fable 5.1 on the codebase and had it read the SDK sources instead of the docs, and then build something we had never had: a test rig on each platform that puts real fingers on real pins and reads the pixels back to see what actually happened. That rig is the story. Our own theories about the bug were wrong twice, and it was the rig, not the reasoning, that said so. Along the way it gave us a tap rule in JavaScript that decides against the pin as drawn and in the order it is painted, three bugs in react-native-maps on the new architecture, an Android tap that the app now hears about 22 ms after the finger lifts instead of 649 ms, and, when the same rule was ported to the website’s Mapbox map, a warning that our port had got draw order wrong while every test passed.

The question the SDKs do not answer

The app’s map is react-native-maps 1.29 on Fabric, React Native’s newer rendering layer. On iOS the map is MapKit and every pin is a live React Native view inside an MKAnnotationView, MapKit’s container for a pin. On Android it is Google Maps and every pin is a bitmap, drawn natively from the same geometry so the two look identical. The website is mapbox-gl 3.29, with the pins drawn by Mapbox itself from a list of points (a symbol layer over a GeoJSON source).

When a finger lands on a cluster, the app needs one answer: which pin did the user put their finger on? Each SDK answers a different question.

• MapKit tests the whole rectangle the pin is drawn in. A pin is a circle on a pointer, so most of that rectangle is empty space either side of the pointer. Tap the empty corner beside pin A’s pointer and MapKit reports A, even when your finger is visibly on the head of pin B behind it.
• Google Maps tests a zone that does not line up with the pin at all. For custom marker images the zone is bigger than the image and shifted upward, a long-standing SDK defect that is worse on real phones than on the emulator. Measured later on a Samsung A13 with real touches: a finger placed dead-centre on pin b was reported as the pin beside it, and a tap on empty map well above a pin counted as a tap on that pin.
• Mapbox GL tests the rectangle it uses to stop labels overlapping, the “collision box”. For our pins that is the whole image the pin is drawn in, including the transparent margin reserved for badges, so the clickable area is half as wide again as the pin you can see.

Three SDKs, three rectangles. The react-native-maps documentation says a marker has an onPress. MapKit’s says an annotation view is selected on tap. Google’s says onMarkerClick fires for the marker that was clicked. Mapbox’s says queryRenderedFeatures returns the features at a point. All true. All silent on the shape being tested. That silence was the whole bug.

Deciding it ourselves

The fix is the same on every platform: stop asking the SDK which pin, ask it where the finger was, project the nearby pins to the screen, and decide against the shape we actually draw, in the order we actually paint.

That needs two things the SDKs do not hand over by default: the finger position and an agreed shape.

The finger position. On Android, the library’s marker-click event reports the marker’s position, not the tap’s. The patch reports the tap location instead, which the touch handler already captures, the way presses on polygons and lines already do. On iOS it took two small patches, both upstream bugs on the Fabric path:

• RNMapsMarkerView.mm built the press event’s position by reading an x and a y out of a dictionary that only holds latitude and longitude. There is no x or y in it, so every marker press on the new architecture reported its position as zero, zero: the top-left corner of the screen.
• AIRMapMarker.m measured the finger relative to the pin’s callout view. With no callout open that view does not exist, which also gives zero, zero. It now measures the finger relative to the map ([recognizer locationInView:marker.map]).

On the web the position is free: the click event has it. Mapbox is only asked for the list of pins drawn in a box around the click, a little bigger than a pin so that a pin whose edge is near the click is included, and the decision is made from there.

The shape. One small file describing the pin as a circle sitting on a pointer, every measurement taken from the tip, which is the point that sits on the map coordinate:

{
  "circleCenterAboveTip": 26,
  "circleRadius": 15,
  "pointerTopAboveTip": 17,
  "pointerHalfWidth": 12,
  "containerWidth": 40,
  "containerHeight": 45
}

The JavaScript hit test reads it. The Android code that draws the pin is tested against it pixel by pixel. The iOS view is rendered into a bitmap and compared to it. The website’s pinGeometry.json now carries it verbatim, and a browser test (Playwright) draws the web pin on a real canvas and checks every pixel’s transparency against the same shape rule. The file exists because the first version of the hit test shipped with a hand-copied rectangle describing a pin the app no longer drew: the renderer had moved the circle two pixels and nothing tied the two together. Now they cannot disagree without a test failing.

The silhouette test is small:

export function isInsideMarker(dx: number, dy: number, g = MARKER_PIN_GEOMETRY): boolean {
  const fromCentreY = dy + g.circleCenterAboveTip
  if (dx * dx + fromCentreY * fromCentreY <= g.circleRadius * g.circleRadius) {
    return true
  }
  if (dy <= 0 && dy >= -g.pointerTopAboveTip) {
    const taper = -dy / g.pointerTopAboveTip // 1 at the top edge, 0 at the tip
    return Math.abs(dx) <= g.pointerHalfWidth * taper
  }
  return false
}

A circle plus a tapering pointer, not a rectangle drawn around the whole thing. A rectangle is exactly how you get “the touch target is wider than the marker”.

The other half of the rule is which pin wins when silhouettes overlap. The original code picked the nearest tip. That is structurally wrong: a pin’s circle sits about 26 pixels above its own tip, so when you tap the circle you can see, you lose to whichever pin has its tip under your finger, and that pin’s circle is usually hidden behind the one you aimed at. That was Android’s “selects the one underneath” in its entirety. The rule is now: among pins whose silhouette contains the tap, the one painted on top wins, because that is the one the user can see. What decides “on top” is the search itself: every place comes back from search with a relevance score, the more relevant place is drawn over the less relevant one, and the same number that sets the drawing order is the number the tap rule reads. If those two ever disagreed, the user would tap the pin they can see and get the one behind it.

If the finger misses every pin outline, the nearest pin within about twelve pixels wins. That allowance exists because a fingertip is about 7 mm wide and the pointer is only three pixels wide near its tip, so a tap that is clearly at a pin often lands just outside its outline.

The first version made it worse

The first build with the new rule went out to a phone and came back with one line: “categorically worse behaviour.” Pins with nothing near them opened a different pin on Android, and some pins on iOS stopped responding at all. The rule was right. Four things it depended on were not.

None of these were visible in the unit tests, because the unit tests encoded the same assumptions we had. Every test we could write would agree with us. We needed a judge that could not: the pixels on the screen, and real touches, on the real app.

The test rig: trust nothing but the screen

The ask was explicit: “set up a test harness against native code and JS code together without needing a sim.” The requirement underneath that message is the one that matters. Every layer between our code and the screen had an opinion about what a tap meant, and every one of them had been wrong at some point. The rig had to run the real app, use the real SDK, put a real finger down, and take its answer from the only place none of those layers could edit: the screenshot.

It ended up as four layers of tests, each catching something the one below could not.

1. Draw the Android pin on a laptop and check every pixel. No phone, no emulator. The code that draws the Android pin is run on an ordinary computer using Robolectric, a tool that lets Android code run outside a phone, and the result is compared pixel by pixel with the shape file at the three screen densities our test phones have. A second test goes further: it saves the drawn pins as images and runs the tap rule itself over them, so the code that draws and the code that decides are checked against each other in one place.

2. Run Apple’s real map code inside the real app. Apple’s map classes, and the layer of react-native-maps that sits on them, cannot be tested on their own; they only compile as part of the app. So the iOS tests live in a separate project that loads the actual CamperMate build and drives its map objects directly: create a pin, remove it, reuse it, tap it, ask where its frame is. Three more tests go one level up and mount a second, test-only screen inside the running app’s own React Native instance, then read back what JavaScript received. Every patch we made to the SDK is checked by undoing it and confirming exactly one test fails.

3. Put real fingers on real pins, where the test says they are. On iOS a test places three pins on the live map, writes their on-screen positions to a file, and waits. A script reads the file and uses Maestro, a tool that taps a phone’s screen from a script, to tap the map above pin a, the centre of pin b, and the empty corner beside pin c. The test then checks what JavaScript was told. One thing this taught us: touches never arrive at a test window unless it has a root view controller, the iOS object that owns what a window shows.

4. Real fingers on Android, inside the running app, judged by a screenshot. An Android test mounts the same test-only screen inside the app, works out where each pin’s image is from Google’s projection and the pin’s anchor, and injects real touches at those points. Android has no way to ask which pin it drew on top, so the test reads the screen: it takes a screenshot, looks at the pixel at the centre of the selected pin, and checks that it is the brand yellow before and after the tap. Three things this taught us: the standard automation click never reaches this window, the app has no keyboard focus for about ten seconds after the screen mounts, and two touches on one spot within 0.7 s count as a double-tap zoom.

Asked at the end of that day whether all this was “a sad indictment” of the developer experience, the summary we wrote down was: three layers of software that never agreed on what a tap meant, and the only description of the truth was the pixels on the screen.

What the screen showed

The rig disproved our best theory and found the real one. That sentence is the reason this post exists, so here is exactly how it happened.

The report was vivid: “I’m literally clicking one POI a very long way away and an iOS marker on the other side of a mountain range turns on.” The theory was view recycling: Fabric reuses the native view behind a pin for a different pin, and if the old pin’s annotation was still on the map, a tap on it would be reported under the new pin’s name. We patched the recycling hook, prepareForRecycle, to disconnect the old annotation and take it off the map. Plausible, checked by reverting it and watching a test fail, and wrong. A test that creates and destroys pins twelve times over through the real renderer, with that patch reverted, left no stale annotations behind. The patch stays in as a precaution.

The test that checks where each pin’s frame sits found the actual bug. On Apple Maps the anchor prop does nothing, and the docs do say so in one line we had never read: “iOS: Google Maps only. For Apple Maps, see the centerOffset prop.” We had passed anchor on both platforms for a year and nothing at runtime warned. We had asked for the pin to hang from its bottom centre (anchor {0.5, 1}). The Fabric marker in react-native-maps ignores that and reads only centerOffset, which was left at zero, so every pin, 40 points wide and 45 tall, sat centred on its coordinate with its tip drawn 22.5 points too low. The JavaScript hit test assumed the tip was on the coordinate, so the shape it checked floated 22.5 points above the pin the user could see. Empty map above a pin selected it. A tap on the drawn circle mostly missed and fell through to the nearest-pin fallback. The info window’s iOS lift had been “tuned on device” against those low-drawn pins. The fix is one constant, {0, -containerHeight / 2}: shift the view up by half its own height. Every iOS pin moved up to where its coordinate is, in line with Android.

Other findings, in the order the screen gave them up:

• Every pin blinked on every selection (iOS). A frame-difference filter over a cluster far from the tapped pin showed every other pin blanked for two frames. Fabric treats a change to a pin’s stacking number (zIndex) as a reorder of siblings, which it does by removing and re-inserting the view, and MapKit lays out every annotation again on any add or remove. Raising the selected pin by 1,000 instead of 999,999 flashed just the same. A separate overlay marker mounted on top did not. Base pins now never change props when selected; the highlight is its own marker on both platforms.
• The peek took forever to disappear. JavaScript dismissed it within 50 ms of the map reporting a press. The wait came before that: on both platforms the single-tap recogniser waits to see whether a second tap follows before it reports the first. The peek now hides the instant a finger touches the map, and the selection clears once the tap is confirmed.
• The invisible strip under the card. The tappable area of the info window included an invisible strip, the full width of the card, between the card and the pin, so tapping the gap opened the place on both platforms. The tappable area is now the card itself.
• On Android, showing an info window overrides the stacking order. react-native-maps calls showInfoWindow on every tapped marker when moveOnMarkerPress is off, and Google draws the marker whose window is showing above every other marker whatever its stacking number. An empty window still counts. That is why the selection overlay had been rendering perfectly and invisibly, underneath. The patch shows a window only for a marker that has one; proven by reading the pixel at the circle centre.
• A tap on empty sea opened a random POI (Android). When the finger missed every pin outline and no pin was within the twelve-pixel allowance, the code still selected whichever marker Google had named. Android’s finger is exact now, so a miss there is a background tap. iOS keeps its fallback because MapKit’s frame is at least the pin’s own box.
• An invisible dev-only control. A 96-by-72-pixel button for triggering end-to-end tests rendered in every dev build, transparent, in the left-middle of the map, eating taps. “This popup broke marker clicks” was true and the popup was innocent.

Dense packs, with real data

Three pins on a test screen is not what users see. Around every popular tourism town, Queenstown, Wellington, Byron Bay, CamperMate shows a dense pack of pins all the time, and that is where the wrong-pin bug lived. So the tests use the real data: a script pulled the 250 places nearest to each of six such towns, 1,266 in all, straight from the production search index. A test lays each scene out on a phone screen at four zoom levels, in the app’s paint order, then simulates a tap every three pixels across the whole pack, tens of thousands of taps per scene. The hit test must return the pin painted at every pixel; the geographic pre-filter must never drop it; every pin with any ink must be tappable; one tap over the densest pack stays under 2 ms.

It found a real flaw on its first run. The nearest-pin fallback kept a running best and let near-ties drift along a chain, B to C to A, each within the tie band of the last, until it could answer a pin farther than the nearest, or one beyond the twelve-pixel allowance entirely. Ties are now relative to the minimum distance and the slop is a hard limit on the answer.

On device, 160 real Wellington places at suburb zoom with four real touches: iOS four of four, Android four of four, each checked against the pin the operating system actually painted on top. On iOS that is decided by each layer’s stacking position; the system’s own hit-testing walks the view tree in a different order, so it cannot be used as the answer key.

Then Android was correct and slow

With both platforms resolving the right pin, the next report was about feel: “on Android it’s nowhere near as fast as iOS when switching between info windows. Noticeably bad. There’s a huge lag.”

Every Android tap was waiting on Google. When a finger lifts, the Google Maps SDK holds the event for 300 ms to see whether a second tap follows, because two taps mean zoom. Only then does it run its own marker hit test and call the app. iOS pins have their own tap recogniser and never waited. Measured on the arm64 emulator, from the moment the finger touched the screen to the moment the app’s JavaScript heard about it:

The patch listens for the finger lifting on the same gesture detector that already sees every touch before Google does, and tells the app right then, with the finger position. Google’s own callbacks for that same lift are then ignored, using a flag set when the fast path fires and cleared when the next finger comes down. A long press or a drag Google consumed is never also a tap. The tap’s screen position is converted to a map position once and reused, where before it crossed between native code and JavaScript three times in a row.

The same change made Android pins fade in over 250 ms once their bitmap lands, which also hides Google’s red default pin while a cold bitmap decodes. The old native app had the same entrance.

Code review caught what the harness had not: a pin whose image failed to load could be left fully transparent, invisible but still tappable, so both failure paths now fall back to Google’s default pin. And Google’s My Location button became a map press in the location picker, dropping the pin under the button; the screen has its own button, so Google’s is off.

One accepted behaviour change: a double-tap to zoom with a peek open now closes the peek on the first tap. Same as iOS, new on Android.

The website had the same bug, and two of its own

The website’s map had the identical complaint, and the ask was direct: “look at how we did marker hit box and cleanup fixes for 5.6.1 in campermate-react-native. we are struggling with the same tap targets on mapbox in mobile / desktop views on campermate.com.” Followed three minutes later by: “clicking on different markers shows the infowindow photo for the previous marker.”

Reading mapbox-gl 3.29 rather than its docs produced the same class of defect. Mapbox resolves a symbol-layer click against the icon’s collision rectangle, the whole padded canvas including the transparent badge margins, so a tap in the dead corner of one pin’s box over the visible head of a lower pin selected the wrong pin. Worse, two handlers answered every click: the layer’s handler set the selection, then a map-level handler re-queried one pixel and cleared it if the query disagreed. A finger 5 px off a 36 px head was a background click that closed the panel. The stale photo was simpler: the panel’s image element had no React key, so React reused the same element and only changed its address, and the browser kept painting the previous place’s photo until the new one downloaded.

The port was almost verbatim. lib/Maps/markerHitTest.ts on the web is the app’s hit test; pinGeometry.json started as a model of the app’s geometry and, a day later, became a copy of it, with the app’s Android rasteriser constants alongside so the web pin is the app’s pin to the pixel. One click handler owns the click. Mapbox lists the pins drawn in a box around the click, each is converted to a screen position, and the same rule decides: the pin whose outline holds the click, topmost first, then the nearest pin within about twelve pixels, otherwise the click was on the background, exactly as on Android, because a mouse pointer is exact. The dense-pack sweep was ported with its six fixtures. A Playwright test bundles the real renderer, draws the pin on a real canvas and checks every pixel’s transparency against the shape rule, the web’s counterpart of the Android laptop test. Another drives the real desktop map with a mouse: on the Queenstown pack there are 44 points where Mapbox’s rectangle names the wrong pin and the silhouette names the right one.

Then the web had two surprises the apps did not.

Mapbox’s query order is not its paint order. The first version trusted the order queryRenderedFeatures returns as the draw order, and the PR description said so, “verified in the 3.29 source”. That holds only for layers Mapbox sorts by screen position, top to bottom. With a sort key set, which the pin layer uses so that the same search relevance as in the app decides what draws on top, mapbox-gl never records its draw order at all and returns results in the reverse of the order the data arrived, unrelated to what it drew. So the wrong pin could still win whenever results arrived most-relevant first, which is the common case, and the test’s answer key shared the assumption, which is why everything passed. Paint order now comes from each feature’s own sort key with the query index as the source-order tiebreak, the test’s answer key applies that rule independently, and the pointer cursor is driven by the same resolver, because until then the cursor promised a click on the rectangle’s corners that the silhouette then read as background.

map.project returns one world copy. A pin drawn on a wrapped copy of the world, any low zoom with copies rendered or a Chatham Islands viewport, projected a world away from the click and read as background. Each candidate is now projected three times, at its own longitude and one full world width to the east and to the west, and the copy nearest the click is kept. The info window’s own placement had the same seam bug, fixed by reusing the helper.

The port also exposed a regression nobody had noticed: the info window’s tail, the small triangle pointing at the pin, had disappeared in April when the card stopped being part of a DOM marker and became an absolutely positioned element lifted a flat 18 px, over the head of its own pin. Tapping pins had been unreliable enough that nobody saw where the card landed. The tail is back, the lift comes from the shared geometry, and the placement rule is the app’s: above, below when there is no room, beside at the edges, tail always on the pin.

One thing did not port. icon-size is a layout property in Mapbox, not a per-feature one, so the app’s iOS sprout animation is not available on a symbol layer. Opacity is per feature, so the web pins fade in over the app’s 150 ms, Android’s entrance.

Would this have been easier without React Native in the way?

A fair question, because the six upstream issues below are all react-native-maps bugs, and it would be easy to read this post as “React Native made maps hard”. The honest split is about a third and two thirds.

The part the SDKs own, which native would have hit just the same. The CamperMate apps that came before React Native used the Google Maps SDK on both platforms, and both took the SDK’s answer as the answer: the iOS app’s didTapMarker: and the Android app’s OnMarkerClickListener each opened whichever marker Google named, with no hit test of their own. So the Google Maps hit zone, bigger than the icon and shifted upward, was in those apps for years. Google’s 300 ms wait before reporting a single tap, to rule out a double tap, was there too, since the native apps waited on the same callback. Arbitrary draw order for two markers at one coordinate, showInfoWindow pulling a marker above every stacking number, and the fact that no SDK will tell you which marker it painted on top: all of that is the SDK, and a native app that wanted the right pin would have needed the same silhouette, the same paint-order rule, the same finger-up dispatch, and a screenshot to check its work.

The part React Native added. Every bug in the “finger position” half of the fix was the bridge: a press position read out of a dictionary that only holds latitude and longitude, a finger measured against a callout view that did not exist, Android reporting the marker’s position instead of the finger’s. A native app gets the touch point from the gesture recogniser or the motion event and never has this problem. anchor doing nothing on Apple Maps is a leak in an abstraction; the old iOS app set groundAnchor itself, in one line. Whole-number stacking numbers, the annotation remount and blink when one pin’s zIndex changes, the tap reported twice: all artefacts of the layer in the middle. And most of the days went on the rig, because with three layers between the finger and the SDK, the only way to test was to mount a screen inside the running app and inject touches from outside it. Natively that is an ordinary XCTest or Espresso test with the views in hand.

One caveat cuts the other way. Native would have made it easier to not notice. The hit-test defect was tolerable in the native apps for years precisely because there was no framework amplifying it into “a marker on the other side of a mountain range turns on”. React Native made it bad enough to force the fix; the fix itself, silhouette plus paint order plus finger position, is the same on every platform and would have been worth doing natively too.

What we would tell someone starting this

• Build the rig before the theory. Every explanation we reasoned our way to was wrong, including one we had already patched and mutation-tested. The test that took a screenshot and looked at a pixel was right every time, and the one web test that shared our assumption instead of checking the screen passed for a day while being wrong. When the layers below you disagree with each other, the screen is the only witness they cannot argue with.
• No map SDK answers “which pin did I tap”. MapKit tests a frame, Google tests an offset region, Mapbox tests a collision rectangle. Their documentation describes the event, not the shape. Read the source. Then decide it yourself, from the finger position, against the shape you draw, in the order you paint.
• Put the pin’s geometry in one file and test every renderer against it, pixel by pixel. Our first hit box described a pin nobody drew any more.
• Paint order is part of the hit test. Make it unique, make it an integer, read it from the same field on both sides, and never take a query’s result order as the draw order without checking which sort the layer uses.
• On Apple Maps, anchor does nothing and centerOffset is the whole story. The docs say it in one line; nothing at runtime does. Changing zIndex remounts the annotation and lays out every other one again. Highlight with a separate marker.
• Mutation-test the patches. A hunk that stays green when reverted is decoration. One of ours was.

The react-native-maps changes that are general rather than CamperMate-specific are now filed upstream with their diffs: the Fabric press position and the nil callout view (#6002), the callout z-position boost that never applied (#6003), the blink when one pin’s zIndex changes (#6004), Android reporting the marker’s position instead of the finger’s (#6005), showInfoWindow raising pins above every zIndex (#6006), and the finger-up dispatch with its latency table (#6007). The Mapbox finding went onto the existing report of query order ignoring symbol-sort-key (mapbox-gl-js #10800). Everything else here is the shape of the work: a year of following the docs, then a few days of reading what the SDKs actually do and building something that could tell us when we were wrong.

This post walks through the mechanisms: why VESC Tool’s command line cannot connect over the network and why a script loaded into it can; a 178-line script in LispBM, the small Lisp interpreter that runs on the controller, that rewrites those two bytes and recomputes the checksum; a serial-port call that silently switches the throttle input off and saves that to permanent storage; the display’s own interactive prompt and the settings file that had it under-reading speed by 18%; a screen where a draw call costs 2.9 ms whether it covers 4 pixels or 9,600; and the test that caught every curve about to reach the screen with its colours reversed.

MIT licensed.

Source → github.com/isaacrowntree/vesc-workbench (MIT)

Change a number, ride, forget what it was

Everyone who runs a VESC ends up in the same loop. Change a current limit. Ride. Change the throttle curve. Ride. Was that better, or was it a headwind? What was it set to three weeks ago?

VESC Tool is a graphical program. You click through tabs, hope you typed the number into the field you meant, and when you are done there is no record of what changed. Exporting an XML backup is a manual step you take instead of riding, so nobody does it every time. And on a sealed skateboard enclosure the USB port is behind screws, so all of this happens on a phone in a driveway.

The repo is that loop, scripted:

make pull      # read both motor sides' configs to XML
make apply     # write them back, then verify by reading them again

Settings are text files. Run git diff on a tuning session, review it before it goes near the motors, revert it in one command. apply reads the settings back after writing them, because the write call with its verify flag off (setMcconf(false)) accepts a write and discards it; only the flagged version asks the controller to check and confirm. That one took an afternoon.

The command line only talks over a cable; the scripting runtime does not

VESC Tool ships a command-line mode, and I expected it to solve this. Its port option (--vescPort) goes straight to the serial-connection code and takes a serial device, full stop. An address and port (--vescPort 192.168.1.100:65102) is rejected. A fake serial port bridged to the network socket (a socat pseudo-terminal) opens, and then the handshake never completes: VESC Tool drives the port directly and the fake does not behave like a real one. I wrote that down as impossible.

It isn’t. VESC Tool also accepts --loadQml, which loads a script written in QML, Qt’s user-interface language, and a script loaded that way runs inside the application, with VescIf in scope. VescIf is the one object that holds the whole connection and configuration interface, and its connectTcp() method can be called from the script. The phone app has a Wireless Bridge to Computer (TCP) mode on its Start page, listening on port 65102, one client at a time. The laptop talks to the phone over Wi-Fi and the phone talks to the controller over Bluetooth, the link it always uses.

The entire minimum viable version, from docs/connecting.md:

import QtQuick 2.7

Item {
    id: root
    property int ticks: 0

    Component.onCompleted: VescIf.connectTcp("192.168.1.100", 65102)

    Timer {
        interval: 500; running: true; repeat: true
        onTriggered: {
            root.ticks++
            if (root.ticks > 60) { console.log("timeout"); Qt.quit() }
            if (!VescIf.isPortConnected()) return

            // Firmware params arrive AFTER the socket connects. Until they do,
            // getFirmwareNow() returns "x.x" and every config read is garbage.
            var fw = VescIf.getFirmwareNow()
            if (fw.indexOf("x.x") >= 0) return

            console.log("connected, fw " + fw)
            VescIf.disconnectPort()
            Qt.quit()
        }
    }
}

"/Applications/VESC Tool.app/Contents/MacOS/VESC Tool" --offscreen --loadQml connect.qml

--offscreen keeps the window from opening and console.log goes to standard output. From there the motor configuration, the app configuration and the command interface (VescIf.mcConfig(), VescIf.appConfig() and VescIf.commands()) are all yours. You do not need the rest of the repo; take the file.

The "x.x" check is load-bearing. The connected flag goes true as soon as the socket is up, several seconds before the controller has sent its parameters, and a configuration read in that window returns defaults that look exactly like a controller that has wiped itself. Every target in vesc/qml/ waits for a real firmware string for that reason.

Diagnostics that tell two failures apart

Half of tuning is working out what is wrong, and a still bar in the GUI does not say why it is still.

make ppm-watch    # live remote readout — prints only on change
make ppm-cal      # guided calibration: neutral, full throttle, full brake
make probe        # connect, report firmware and LispBM state
make check        # is the bridge up? is desktop VESC Tool holding it?

PPM is the pulse signal the hand remote sends, and ppm-watch distinguishes the remote is not transmitting from the controller is not decoding it, which look identical in VESC Tool and have different fixes. make check is a port probe against the bridge plus a check for a running desktop VESC Tool, which would be occupying the bridge’s single client slot; it answers in a second instead of a two-minute timeout.

make motors-off   # kill motor output, no config write, display stays live
make motors-on

motors-off tells the controller to stop driving the motors (app-disable-output), which is not a configuration change, so nothing needs undoing and nothing is left in a strange state if you walk away.

The controller reports accurately and points at the wrong thing

docs/findings.md is the document I wanted and could not find. Every entry has the same shape: the firmware does something reasonable, reports it correctly, and the report sends you somewhere else.

The first one is the one to know before running any script. From bldc/lispBM/lispif_vesc_extensions.c, ext_uart_start:

if (appconf->app_to_use == APP_UART ||
        appconf->app_to_use == APP_PPM_UART ||
        appconf->app_to_use == APP_ADC_UART) {
    appconf->app_to_use = APP_NONE;
    conf_general_store_app_configuration(appconf);   // written to FLASH
    app_set_configuration(appconf);
}

The setting app_to_use says which input drives the motor. If it is any of the three modes that include the serial port, “serial only” (3), “remote and serial” (4) or “analogue and serial” (5), then uart-start writes “none” to permanent storage and the throttle is gone. It also explains config writes that “do not persist”: they persisted fine, and the next uart-start overwrote them. make upload-lisp now searches the script for uart-start, reads the setting, and changes “remote and serial” to “remote only” (4 to 1) and “analogue and serial” to “analogue only” (5 to 2) before uploading, since the firmware is going to clear the serial half either way. Verified on the board: set to “remote only”, the decoder reads a live pulse of between 1.4990 and 1.5020 ms; set to “remote and serial”, it reads nothing at all.

There is a correction in there too. The esk8 forums say VESC traction control interferes with braking. I enabled it and then read applications/app_ppm.c: traction control lives entirely in the non-brake branch, mc_interface_set_brake_current is called before that branch is reached, and it disengages on any fault. What it does do is taper drive current smoothly to zero as the two wheels’ speeds drift apart, reaching zero at a configured difference (tc_max_diff), measured in electrical revolutions per minute, the motor speed as the controller counts it. On this board, with seven magnet pole pairs, a 4.2-to-1 gear ratio and 200 mm wheels, the default of 3,000 is about 3.8 km/h of difference between the wheels, tight enough that normal grass slip keeps triggering the taper, which is the “power surging” people describe and probably where the folklore comes from. Doubling it to 6,000 (about 7.7 km/h) is what runs here, confirmed in motion.

Two bytes stood between the display and firmware 7

All of this exists because of a smaller problem. I have a LaCroix Nazaré electric skateboard with a FOCBOX Unity sealed in the enclosure, ridden on grass at a golf course, and I wanted it on firmware 7 for its newer motor-control code and for LispBM scripting. The DAVEGA X display refused:

Unsupported VESC FW: 7.01

Supported VESC FW versions: 3.48 - 6.x

DAVEGA closed in May 2024. Everyone downgrades. I checked whether that was true first: the vendor’s update index at davega.eu/fw/index_v5.json still lists a v5.07rc3, dated 2025-03-11, newer than the v5.06 the changelog stops at, released after the shop closed and never announced. It carries the same gate strings as every version before it. No firmware DAVEGA shipped accepts VESC 7. docs/firmware-archive.md records all eight images and their SHA-256s, since the endpoints will not answer forever.

What did not change is the protocol. lisp/tests/protocol-diff.sh fetches the two firmware source files that define the telemetry reply (comm/commands.c and datatypes.h) from the upstream bldc repository at tag 6.00 and at master, extracts the lines that append each field to the reply, and compares them: 25 fields, same types, same order. The display is not failing to parse anything. It reads two version bytes, sees a 7, and declines to have the conversation.

One thing I got wrong on the way. I claimed the firmware-version reply was structurally identical too. It is not: 7.x appends a four-byte hardware checksum (buffer_append_uint32(send_buffer, main_calc_hw_crc(), &ind)) that 6.00 lacks. It does not break anything, because the display reads the major and minor version at fixed positions, bytes 1 and 2, and the length byte accounts for the extra word, but my first version of the comparison script only matched three of the four ways the firmware writes into a reply and missed it. The script now compares the appended fields in that reply too, and the test fails the build only on the packet that matters.

VESC firmware 6.06 lets a LispBM script hand a packet to the firmware’s own command decoder (cmds-proc) and receive the reply as an event (event-cmds-data-tx). So a script on the controller can take over the serial line the display is wired to, hand every packet to the real handler, and rewrite one reply on the way out. From lisp/src/proxy.lisp:

(defun fixfw (d) {
    (var n (bufget-u8 d 1))
    (if (and (= (bufget-u8 d 0) 2) (= (bufget-u8 d 2) 0)) {
        (bufset-u8 d 3 6)               ; FW_VERSION_MAJOR -> 6
        (bufset-u8 d 4 0)               ; FW_VERSION_MINOR -> 0
        (bufcpy p 0 d 2 n)
        (var c (crc16 p n))
        (bufset-u8 d (+ 2 n) (shr c 8))
        (bufset-u8 d (+ 3 n) (bitwise-and c 255))
    })
    d
})

The rewrite is 27 lines; the whole shim, serial reader and debug counters included, is 178. The checksum function takes no start offset, which is why the payload is copied into its own buffer first. Everything else passes through untouched.

Three things made it harder than the listing suggests. Thirteen commands (numbers 62, 66 to 72, 80, 83, 90, 116, 125 and 158) are handled on a separate blocking thread and reply through a different path, never through the event the shim listens to; forward one and no reply ever comes, and the firmware is left marked as busy. So the shim answers those itself, and answers the ping over the CAN bus (the wire between the Unity’s two halves) with a precomputed reply. The first read loop asked for both header bytes in one call and dropped any payload that arrived short, which cost whole round trips: reading the header and then the payload separately took the reply rate from about 6% to about 70%. And the safe reader was not free until it was made so. Measured on the Unity over three windows of 40 to 45 s each:

lisp/tests/bench_reader.lisp counts serial-port calls, which is what the interpreter pays for. The shipped reader was spending three reads per frame when the length byte already said how much was coming; fetching the body in one call took it to two, and the CPU cost of the framing guarantees came back to noise.

If you have a DAVEGA and want to keep its own firmware, that shim is the whole answer and you can stop reading here.

Test the Lisp in the interpreter that runs it

If you do anything custom on a VESC, LispBM is where it happens:

make upload-lisp LISP=path/to/script.lisp   # upload and run
make lisp-stats                             # heap, CPU, globals
make lisp-stop / lisp-erase                 # stop, or back to stock
make test-lisp                              # run it in the real interpreter

test-lisp runs the scripts in the upstream LispBM interpreter’s own command prompt, in Docker, with stand-ins for the VESC-specific functions, rather than in a Python re-implementation. LispBM names are case-insensitive, so a function W collides with an alias w and silently calls itself. t is a special symbol that never resolves from the environment, so declaring a variable named t kills the script that uses it. The serial read function takes its timeout as the fifth argument, and the serial write function takes no length and writes the whole array. The real interpreter catches all of these; a reimplementation catches none.

Upload happens in 384-byte chunks with a one-second timeout per chunk, which is why lisp/minify.py exists and why the shim is written as tightly as it is.

The shim is retired now. The script slot on the controller runs a flight recorder instead: the session’s peak values, the first fault along with the voltage, motor speed, duty cycle and current that explain it, and traction-control engagements, which nothing on the controller reports today. It runs for hours on a board someone rides, so “probably fine” was not an answer. The whole ride plays through the real interpreter at three lengths and the heap is compared: identical after 500 ticks and after 25,000, about eighty minutes, with every high-water branch taken. The soak also found uptime accumulating 0.2 per tick into a float and reading 5000.18 s after exactly 5000; it is derived from the sample count now.

The display is a small computer you can log into

The shim treats the DAVEGA as something to lie to. That stopped being true once I looked properly.

Its firmware is closed, but the X is an ESP32, the same Wi-Fi microcontroller found in a lot of hobby electronics, running MicroPython, a small Python for microcontrollers, with a USB-to-serial chip behind the micro USB port. Hold UP and DOWN at boot and the startup script enters WebREPL, an interactive Python prompt over Wi-Fi, before any user code runs. From there the filesystem is in front of you. Settings are a plain /config.json, which is how I found the display configured for 175 mm wheels on a 72-tooth to 16-tooth gear pair when the board runs 200 mm wheels on 84 to 20. It had been under-reading speed by 18% for as long as I had owned it. make davega-settings fixed it.

It also means the screen is programmable. The startup script names its boot steps in order, and one of them runs a user-supplied /start.py before the stock app starts. A replacement dashboard is one file you can delete, and holding UP at boot stands it down. The version check lives in the stock app, not in the class that talks to the controller. A dashboard I wrote reads firmware 7 directly, and the shim stops being necessary at all.

Ten dashboards for a panel that cannot draw a curve

Ten themes, each with its own layout rather than its own palette: a full analogue tachometer, hexagonal shards on a diagonal split, three hairline arcs, one enormous numeral, concentric rings with a single red hand, a shift-light rail you read peripherally, a power-flow meter that treats current as more important than speed. The default takes one idea from each. Every picture in this post is the real screen code run through a host stand-in for the panel, rendering the pixels it produced.

The pictures are not the interesting part. The panel cannot draw a curve. The driver for its controller chip (an ILI9341) offers three operations, fill a rectangle, set a pixel and write a block of pixels, and no line, circle or polygon. I expected cost to scale with pixels. Measured on the device:

A draw call costs the same whether it covers nine pixels or nine thousand. So the obvious way to draw an arc, one thin rectangle per column, costs 481 ms for an arc 96 pixels in radius, four times the budget for a whole frame, to draw one gauge.

The way through is the third operation. davega/gui/bands.py draws the curve into an in-memory pixel buffer and pushes it to the panel in one block write. There is not enough memory for a buffer the size of the screen, so the screen is painted in horizontal bands with one small buffer reused all the way down.

The dial face, bezel and tick marks are composed once, as furniture; the moving sweep is composed the same way into a band that covers only the dial.

I tried the cleverer version first: repaint only the wedge between the old angle and the new. It is cheaper and it is wrong. An arc drawn in three pieces lands on different pixels from the same arc drawn in one, because each piece rounds its own start angle to a pixel, and a partial repaint that disagrees with a full repaint is worse than one that costs a few more milliseconds. Arcs repaint whole; everything straight still repaints only its tip. All 810 transitions between test frames and themes now agree exactly.

Measured on the panel with five screens live, before the curved layouts landed, against telemetry arriving five times a second:

The panel’s own number-drawing routine is native and about three times cheaper than drawing digits as text characters; that change alone took a settled frame from 82 ms to 25 ms.

The harness lied first

A 2.8-inch screen bolted to a deck is a bad place to iterate a design. So before the first dashboard there was davega/harness/display.py: a stand-in for the panel that records every draw call, writes the result out as a PNG using nothing but Python’s standard library, and models the panel’s measured cost per call. A screen is a function of three inputs, the display, the current telemetry frame and the board’s configuration, with no hidden state, so the same code runs on the device and in continuous integration, and make gui-test runs 642 checks with no hardware.

Four things it catches that looking at the screen cannot:

• Anything drawn outside the 240-by-320 screen fails the build rather than being clipped where you might not notice.
• Reference images for every frame in the test set: standstill, full throttle, hard regenerative braking, a thermal power cut, battery empty, battery full, a fault at speed. The frames are generated from the board’s verified configuration (a battery of twelve cells in series and four in parallel, 17 Ah; 80 A per motor; 30 A out of and 8 A back into the battery per side; 4.2-to-1 gearing; 200 mm wheels), so extremes a ride rarely produces are covered on purpose.
• Partial redraw against a full repaint across all 810 transitions. Partial redraw’s failure mode is stale pixels, and the two paths must agree byte for byte. This caught the fault banner staying on the screen after the controller recovered.
• Fixed labels against live regions. A label is drawn once; a region repaints when its value changes; the panel’s font paints its own background. A label inside a region’s box is erased the first time that value moves and never comes back.

That last test exists because the harness was flattering the design. Its print drew glyphs without their backgrounds, while the real font fills the whole character cell, so labels that punched holes through their own gauges looked perfect in every render. Modelling the panel honestly made four themes fail at once. A parity test also asserts every colour declared in themes.py appears in the rendered mockups; it immediately found five fault colours that existed only in code.

Running the same code in the display’s own Python

All of that runs under desktop Python, which does not tell you whether the code will start on the board. make test-device builds MicroPython 1.14, the version the DAVEGA reports, with the pixel-buffer module compiled in, and drives the real gui package through a whole ride: boot, telemetry replayed from two frames recorded off the Unity, every screen, the buttons through their real 50 ms press-settling delay, the menu, a controller that stops answering, and the recovery.

It earned its keep on the first run. A colour on this panel is two bytes. MicroPython writes those two bytes in the processor’s native order, low byte first, and the panel expects the high byte first, with the block write streaming the buffer to it unchanged. Every curve would have reached the panel with its colours reversed. Red drawing blue. It could not show up off the device because the pure-Python stand-in writes the high byte first directly: the harness and the hardware disagreed and only the hardware was right. The two bytes are now swapped on the way into the native drawing path, the byte order is probed rather than assumed, and the two paths render side by side and compare byte for byte.

The other thing it watches is memory use, because 98 kB of free memory and a two-hour ride is a bad combination. It measures two consecutive windows of 400 frames each, since one window cannot tell a leak from a warm-up; caches fill and animations settle in the first. Absolute byte counts are reported rather than asserted, because this is MicroPython built for a 64-bit laptop, where every reference is twice the width it is on the board. Only the drift between the two windows counts.

Which is exactly the limit that bit me twice on the board. Switching theme killed the dashboard and handed the screen to the stock app’s “initializing”; the error file on the display named the allocation, 19,200 bytes in the band painter, claimed fresh on every call at the moment memory was at its worst. Then, with that fixed, a cold boot failed the same way asking for 9,600 bytes at the first curve, because the menu now remembers the rider’s choice and it had been left on a layout with a semicircular gauge. The buffer is now claimed at boot, before any screen exists, and the reserve routine takes the largest band it can get, falling back through 20, 12, 8 and 4 rows. A layout too heavy for what is left drops back to the default arrangement and writes that choice down, so the next boot does not fail the same way. Verified on the board from a clean start: every layout boots with about 31 kB to spare. A dead screen on a deck is worse than the wrong colours.

What the board does now

Telemetry is live. A DAVEGA X running a dashboard I wrote reads a FOCBOX Unity on firmware 7.00 directly, no version spoofing in the path. Two things the reference firmware could not have told me: the serial port transmits on pin 17 and receives on pin 16, the reverse of what the reference code’s attribute names suggest, and a firmware-7 reply is 79 bytes, not the 70 the reference buffer allows. And MicroPython’s serial read blocks for the port’s whole timeout when nothing has arrived, so asking whether anything is waiting before reading made a real difference:

A Unity is two controllers in one case and the standard reply carries only the one that answered, so davega/gui/vesc.py asks the near side over the wire and the far side through it over the CAN bus, then combines them the way DAVEGA’s own Unity code does: battery current and energy summed, per-motor figures averaged, distance counted once. Temperature is the one divergence; it takes the hotter, because an average hides the controller about to cut its power.

Charge and range use an internal-resistance model rather than intuition. The pack’s resistance is measured as the slope of voltage against current once 8 A of spread has been seen, and the method recovers 0.0400 ohms on a simulated 40-milliohm pack. A twelve-cell pack resting at 48 V and pulling 28 A through 40 milliohms reads 46.9 V at the terminals; a gauge that reads the terminal voltage calls that 41% and the truth is 51%. That ten-point swing is the whole “battery drops when I accelerate” complaint. Range is worked out by draining the modelled pack step by step rather than dividing remaining energy by today’s consumption rate: at a measured 18.7 Wh per km from 45.8 V that is 15.2 km rather than 18.0, shorter, which is the uncomfortable direction and the right one.

The dashboard ships pre-compiled. MicroPython compiles a .py file every time it imports it, and on this ESP32 that compile was most of the wait between switching on and seeing a number: 154 kB of source against 62 kB of compiled .mpy files. start.py stays plain source, because an escape hatch that depends on the build tools having run is not one.

The board is tuned for grass at 80 A a side, traction control on, and it has been ridden. Speed, current, charge and range on the screen, both controllers read, at 40 km/h on grass. Hardware coverage is one FOCBOX Unity, one board, one display; the connection layer and the config workflow are not Unity-specific, but nobody has proved that on other hardware yet. vesc/profiles/ holds connection details only, deliberately not current limits or gearing. Copying a stranger’s motor tuning is how packs and motors get damaged.

Three things I would tell myself at the start. Read the dependency before probing it: the VESC serial protocol is written down in the reference DAVEGA code (janpom/davega), and an evening guessing at its function signatures was worth ten minutes of reading. Test in the interpreter that will run the code, because the two bugs that would have reached the deck, reversed colours and a 9,600-byte out-of-memory error, were invisible under desktop Python by construction. And when a device you have not touched disagrees with a value you typed in, the device is the better witness; the DAVEGA’s battery-capacity setting said 17,000 mAh for months before I stopped believing the pack had six cells in parallel.

If you have a VESC and a scripting habit, start with docs/connecting.md. Even if you use nothing else, having your board’s configuration in git is worth the twenty minutes.

This post walks through what the v3 rewrite changed and why: the “still unhealthy” transition the old state machine could not express, why three storage systems became one (the rewrite commit deleted 2,386 lines), how a secret can be referred to by name in a database anyone with the dashboard can edit without its value ever landing in a row, why a failing response body is scrubbed of secrets before it is cut short, the test that keeps the API document an agent reads in step with the routes that exist, and the thing the docs claim about signed action links that the code does not yet do.

Source → github.com/triptechtravel/clawdwatch · Docs → triptechtravel.github.io/clawdwatch

Where the model is allowed, and where it is not

There is an obvious version of an AI-era monitoring tool where a model looks at the response and decides whether you have a problem. clawdwatch does not do that. Detection is a threshold, a state machine and a maintenance window — code you can read in an afternoon and reason about at 3am. What the AI gets is the other half of the job: the part after “this is broken”, where somebody has to work out why.

Everything below is either “make the detection boring and trustworthy” or “make the handoff useful.”

The state machine could not say “still down”

The transition function is 118 lines in src/engine/transition.ts, and its signature is the design:

export function computeTransition(
  state: CheckState,
  result: CheckResult,
  check: CheckConfig,
  now: number,
  nextIncidentId: string,
): { state: CheckState; transition: Transition } {

The current time is passed in as a parameter and never read from the system clock; the incident id is supplied by the caller. That is what lets the test suite walk a check through a week of results in 273 lines without a clock. Four states, unknown, healthy, degraded and unhealthy, and the moves between them:

The threshold defaults to three consecutive failures. The move that matters is the loop from unhealthy back to unhealthy. The file header records the bug it fixes: in v2 that case returned nothing forever, so a multi-day outage alerted once on day one and then went quiet while still down. Reminders can only be sent if the state machine has a name for “still broken, and it has been a while”:

  // Already unhealthy: remind only when the cadence has elapsed.
  const interval = check.reminderIntervalMs;
  const lastAlert = state.lastAlertAt ? Date.parse(state.lastAlertAt) : null;
  const downSince = state.downSince ? Date.parse(state.downSince) : now;

  if (interval !== null && lastAlert !== null && now - lastAlert >= interval) {
    next.lastAlertAt = at;
    return {
      state: next,
      transition: { kind: 'reminder', downSinceMs: Math.max(0, now - downSince) },
    };
  }

The default cadence is one hour; null turns reminders off. The test drives a sustained failure through the function and asserts the sequence ['opened', 'reminder', 'none', 'reminder'].

Two more v2 defects were structural, and the orchestrator’s header names them: checks ran one after another, so ten checks with a 10 s timeout and retries could use up the time the scheduled run was allowed, and alerts fired one per check inside that loop, so a ten-endpoint outage was ten notifications. Now checks run six at a time, transitions are collected per run into opened, recovered and reminder events plus a single summary, and a quiet run emits nothing at all.

Three storage systems became one

D1, Cloudflare’s hosted SQLite. Six tables — checks, check_state, check_results, incidents, maintenance_windows, notifier_deliveries — in one migration whose header says what it replaced:

-- One storage system: D1. (v2 split hot state into an R2 JSON blob and
-- history into Analytics Engine that nothing ever read.)

That is not a figure of speech. The v2 state module opens with “R2 state persistence for monitoring … History is stored in Analytics Engine” (R2 is Cloudflare’s file storage and Analytics Engine its time-series store), and the orchestrator wrote a data point to Analytics Engine on every result. Nothing in v2 ever read that data back; the rewrite commit lists it as “Analytics Engine (write-only)”. Two storage systems, two consistency stories, a whole class of “which one is right?” bugs, for a dataset with no reader.

The rewrite commit touched 23 files: 3,058 insertions, 2,386 deletions. db.ts (370 lines) and state.ts (69) went; transition.ts (118), secrets.ts (195) and store/d1.ts (398) arrived. The integration suite applies the shipped migration and runs in workerd against a local stand-in for D1, so the SQL is exercised rather than mocked — 40 tests in 1.94 s.

One schema decision worth copying: check_state deliberately has no foreign key to checks(id). A run writes every check’s result and state in one batch, and if a check were deleted mid-run a foreign-key violation would fail the batch and silently lose that tick for every other check too.

Results are kept 48 hours (historyRetentionHours: 48) and pruned every run. The pruning of notifier_deliveries keeps MAX(id) per notifier, because alerts are sparse and deleting a notifier’s only row would make it vanish from the dashboard.

Public code, private config

Checks live in the database and are editable through the UI. That is only safe if a secret can never land in a row, so a check stores a reference:

{ "headers": { "X-Api-Key": "${MY_API_KEY}" } }

A reference is a name in ${...} braces, and it is turned into the real value at exactly one point, where the runner builds the request headers, throwing an error if a name is missing rather than sending an empty header. Every path leaving the system goes through scrub() first, which does the substitution in reverse — each literal value is replaced with its ${NAME}:

export function scrub(text: string, secrets: SecretMap): string {
  let out = text;
  for (const [name, value] of Object.entries(secrets)) {
    if (!value || value.length < MIN_SECRET_LENGTH) continue;
    out = out.split(value).join(`\${${name}}`);
  }
  return out;
}

MIN_SECRET_LENGTH is 8; shorter values are too common to treat as secrets. Alert payloads go further and carry a CheckSummary with headers and body dropped entirely, on the principle that what is absent cannot leak.

Writing a check that contains a literal secret value is rejected with a 400 whose message names the reference you should have used. The test posts a header of hc-super-secret-value and asserts the error contains ${HEALTHCHECK_SECRET}. And there is a test named property: no resolved secret escapes — not a generated property-based test, a hand-written exhaustive loop over five placements (a header, a Bearer authorisation header, a JSON body, a query string, all three at once) times every secret — asserting first that the write guard throws, and then that if the value had somehow reached storage, the API view, the alert summary and a log line are all clean.

That guard is what makes a dashboard-editable, database-backed monitor safe to open-source at all. For a token that belongs to a whole domain rather than one check — one that lets the monitor past a web application firewall, say — header rules keyed by host pattern carry the same references.

Keeping the thing that explains the outage

By default a response body is read to evaluate assertions and discarded; what is stored is the assertion message, truncated to 256 characters. A monitored endpoint that returns personal data does not leak it into the monitoring database.

But “expected 200, got 500” throws away the one thing that usually explains the outage. The motivating case was a health endpoint whose 500 body named the failing dependency, for five hours, while the alert carried none of it. So a check can opt in with "captureBodyOnFailure": true, and buildBodySnippet() is deliberately ordered:

export function buildBodySnippet(raw: string, secrets: SecretMap): string | null {
  const collapsed = scrub(raw, secrets).replace(/\s+/g, ' ').trim();
  if (collapsed === '') return null;
  return collapsed.length <= MAX_BODY_SNIPPET_LENGTH
    ? collapsed
    : `${collapsed.slice(0, MAX_BODY_SNIPPET_LENGTH - 1)}…`;
}

Scrub before truncating, so a secret straddling the cut is masked rather than half-printed. The cap is 512 characters — long enough for a JSON error or an HTML title, short enough that it cannot become a bulk exfiltration channel. Capture only runs when assertions failed, only for textual content types (the regex accepts text/*, JSON, XML, problem+json and *+json, and treats an absent content-type as textual because many error paths omit it).

The snippet reaches the webhook notifier and the dashboard, both already trusted with the monitoring database. It never reaches Slack: bodySnippet does not appear anywhere in slack.ts, and a test sets one to connection refused to db-primary and asserts neither string is in the rendered payload. Slack is a wide, retained audience.

Handing an incident to an agent

An agent inbox is just a URL, so webhook() is the whole integration. Every alert carries a links object — incident, ack, annotate, runNow, maintenance, capabilities — and GET /api/agent.md describes the API so an agent can act on what it was handed.

Here is where I have to correct the documentation, including my own. The docs say the action links “expire after about an hour”. The signing machinery exists: a function mints a link by signing the action and its expiry time with the same keyed hash the webhook notifier uses, a verifier checks the expiry and compares the signature in constant time, and the authentication layer accepts such a signature in the query string, scoped to one action. It is tested. But the orchestrator’s link builder emits plain URLs, and the minting function has no production call site. As shipped, an agent that receives an alert still needs an identity — a Cloudflare Access service token — to write back. The signed-link path is the next thing to wire in, and until then the docs overstate it.

What is real is agent.md. It is generated from a 17-entry ROUTES table in src/routes/agent-md.ts, and two tests keep it honest in both directions:

    const mounted = app(fakeDb().db).routes
      .filter((r) => r.path.startsWith('/api/'))
      .map((r) => `${r.method} ${r.path}`);

    const documented = new Set(ROUTES.map((r) => `${r.method} ${r.path}`));
    const undocumented = [...new Set(mounted)].filter((m) => !documented.has(m));
    expect(undocumented).toEqual([]);

and the mirror, documents no route that does not exist. The route replaced an earlier idea: shipping a skill file for agents to install. The file header says why — a static file that duplicates a live API drifts, and in practice never gets installed; the previous generation’s skill claimed 90-day retention for a system that kept 48 hours, and was never copied into the container it was meant for.

An agent that has triaged something writes its findings back with POST /api/incidents/:id/annotate, and the note appears on the incident next to the human comments. There is a worked receiver: thinkbot, which takes these alerts and correlates them against GitHub, Datadog and Sentry.

When the receiver is a Worker, skip the webhook

If the receiver lives on the same Cloudflare account, a service binding beats HTTP: the platform authenticates the call, so there is no shared HMAC secret to distribute or rotate, no public inbox to defend, and no replay window to reason about. The payload crosses as a structured value rather than a JSON string.

notifiers: [rpc({ binding: (env) => env.AGENT })]

The receiver is a WorkerEntrypoint with an alert(event) method. rpc() throws, rather than returning quietly, when the binding is missing — dispatch records delivery outcomes, and an unconfigured binding must show up there as a failure. A silent success is how a dead alert path stays invisible.

One sharp edge, stated in both repos: an RPC call carries no signature. Authenticity comes from the binding. Keep signature verification in the HTTP handler and do not let shared downstream code assume it ran. Service bindings are same-account only, so webhook() with hmac() remains the option for everyone else; its verifier rejects signatures older than DEFAULT_TOLERANCE_MS, five minutes.

A versioned payload, so a release is not an outage

Every alert carries schemaVersion, exported as ALERT_SCHEMA_VERSION, currently 1. The rule receivers can rely on is in the type’s comment: adding an optional field does not bump it — bodySnippet was added this way — and removing, renaming or retyping one does. Receivers should ignore unknown fields and must not hard-fail on a version higher than they know. A receiver that rejects unknown versions turns every clawdwatch release into a monitoring outage.

Arriving through the login proxy is not proof of identity

Mount it behind Cloudflare Access, Cloudflare’s login proxy. Reads are open by default; writes always require an identity.

There is no API key in the query string. URLs leak into logs, analytics and referrers, and a shared static secret has no identity, no expiry and no revocation. And arriving through Access is not treated as proof in itself, because anyone who learns the Worker’s direct address bypasses the proxy. So src/auth/jwt.ts fetches the team’s public signing keys from Access (cached for five minutes), refuses any signing algorithm but RS256, verifies the token’s signature against the named key, and checks its expiry, its not-before time, that it was issued for this application and that it was issued by this team. A token minted for another Access application must not work here; that is what keeps a leaked token scoped to this app.

The detail the previous implementation got wrong: a service token, the kind of identity a machine caller gets, carries no email address in its token. Its identity is in the common_name field. Code that only looks for an email silently rejects every machine caller, which is exactly the caller an agent integration needs. Both shapes are handled, and verification failures are returned as a flat 403 without the reason, because the reason tells an attacker which check failed.

No identity is stored. The JWT is verified for the decision and discarded; there are no email, IP or user-agent columns.

A dashboard that does not interpolate

One mark per check run. The tick-strip component’s header explains the choice: a scheduled check produces separate samples, and a smoothed line between them would assert measurements that were never taken. Colour is status, height is response time against the check’s own ceiling, and the marks distinguish “responded badly” (a status code came back) from “unreachable” (no status code at all). Healthy marks are quieted in the stylesheet so amber and red carry the page.

Every delivery is recorded too, so the dashboard can answer the question monitoring tools are worst at: did the last alert actually arrive? That panel had a bug worth owning. Delivery rows are only written when an alert fires, so a failed delivery from initial setup sat red for six days with everything healthy — there was no next alert to overwrite it. The fix ages rows out: after 24 hours a chip renders grey as “last delivery failed”, which says what we actually know. The same commit found that pruneDeliveries was exported and unit-tested but had no production call site, so the table had grown for the life of the deploy.

Getting it running

npm create cloudflare@latest my-monitor -- \
  --template clawdwatch/clawdwatch/examples/worker
cd my-monitor

wrangler d1 create clawdwatch          # paste the id into wrangler.jsonc
npm run migrate
wrangler secret put SLACK_WEBHOOK_URL   # optional, and all Slack needs
npm run deploy

Or as a library, a route inside a Worker you already run:

import { createMonitor, slack } from 'clawdwatch';

const monitor = createMonitor<Env>({
  d1: (env) => env.MONITORING_DB,
  secrets: (env) => ({ SLACK_WEBHOOK_URL: env.SLACK_WEBHOOK_URL }),
  notifiers: [slack({ webhook: '${SLACK_WEBHOOK_URL}' })],
});

export default { fetch: monitor.fetch, scheduled: monitor.scheduled };

Defaults: a check every five minutes, a 10 s timeout, one retry after 5 s, three failures to open an alert, reminders hourly. Assertions cover status code, headers, body, response time and a value picked out of a JSON body — the last of which turns a health endpoint that returns {"db":"ok","queue":"degraded"} into a signal rather than a 200.

What this cost and what it bought

Triptech runs production APIs on Cloudflare Workers, and the monitoring options are a SaaS that bills per check and knows nothing about your account, or a Worker you write yourself and never quite finish. v2 was the second one: three storage systems, a state machine with no word for “still down”, and a skill file describing retention the system did not have.

Three things I would carry to the next project. Inject the clock and the id generator, and the state machine becomes 273 lines of tests that run in milliseconds. Do the secret scrub before the truncation, not after. And generate the document an agent reads from the same table that mounts the routes, with a test in each direction — it is the only kind of documentation that was still true a month later. The capability links are the open item; the docs describe them, the code signs them, and the orchestrator does not yet attach them.

Read the source, or start at getting started.

This post walks through what the rebuild actually consists of: the line counts before and after; the stockist finder that still reads the owner’s Dropbox CSV at page load and the two-pass state matcher that recovered 74 of 407 retailers a strict regex was dropping; the four things that made it safe to ship an unapproved redesign to /preview/ on the client’s live domain; a Liquid filter chain that doubled a URL; and the <details> menu, Intl.DateTimeFormat readout and reduced-motion reveal on Mower Corner.

This post replaces three older ones about the same clients. Those described sites that no longer exist, and described them in the language of a brochure. This is what got built.

What was there before

Breeder’s Choice’s old _layouts/default.html pulled, in order: a Font Awesome kit from kit.fontawesome.com, Bootstrap 5.3.3 CSS from jsDelivr, AOS 2.3.1 from unpkg, and at line 1233 jQuery 3.7.1 from code.jquery.com, Bootstrap’s bundle from jsDelivr again, jQuery CSV 1.0.40 from cdnjs, and AOS’s script from unpkg. Superior Shavings had the same Font Awesome kit, the same Bootstrap and jQuery, jQuery CSV from jsDelivr, and Montserrat from Google Fonts — no AOS, which I had misremembered until I read the blob. Five CDN hosts each, a different five. Mower Corner’s externals were in _includes/styles.html and _includes/scripts.html: Bootstrap 5.3.2, Font Awesome 6.4.2 from cdnjs, AOS from unpkg, Oswald and Open Sans from Google Fonts, and every photo from Cloudinary.

None of that was unreasonable in 2015. It is just that the whole budget of every page went to generic scaffolding, and what came out looked like every other Bootstrap site, because it was one.

What replaced it, verified with wc -l:

AC Service & Parts is a one-page site whose 698 lines of CSS are inline in _layouts/home.html; it had a full redesign in April and a refinement pass in August, below. Google Fonts and Analytics still load on all four, and Mower Corner still serves its imagery through Cloudinary, because the 2017 shop photography is flat and underexposed and e_improve:outdoor plus e_auto_color on the actual pixels beats any CSS filter. I am not going to claim a purity I did not ship.

Four businesses, four registers

The temptation with four clients at once is one design system in four colourways. These are four trades in two states, and two are sister brands owned by the same person, which makes differentiating them the actual design problem.

Mower Corner — “spec plate.” Their mark is a condensed wordmark standing in a strip of grass, unchanged since the 90s, and it is genuine equity in Colac. The grass stayed and was redrawn as a silhouette; the sky gradient and clip-art operators did not. The structural device is the riveted spec plate off the side of every machine in the shop, carrying the shop’s own data. main.css declares --ink: #14181a (cast iron), --orange: #f2600c (safety orange, the equity colour), --grass: #2b7134 (the traced green from the wordmark’s grass strip), and --radius: 2px; /* pressed metal, not soft plastic */.

Breeder’s Choice — “the bale and the spec.” A family business that has done one thing since 1994. A 14 kg bale with a moisture spec is a real product, so the product leads: their own pallet photography, and a kraft bale-label tag carrying the spec the way the printed wrap does. --forest: #0B4423 and --green: #0F6B36 from the logo, on cream and kraft, --radius: 6px; /* softer than the industrial sibling */. The stylesheet header calls it “the daylight sibling of Superior Shavings’ iron-dark industrial register.”

Superior Shavings — “mill-side.” The sister plant, in NSW softwood country, where the pitch is proximity: from the saw into the bale with minimal handling. So it got the opposite register: --iron: #1F2224, the logo’s --red: #BF2025, a numbered mill line, the spec set as a mono bale ticket. Same structure as Breeder’s Choice, same ethos, different plant. Each site links the other from the story section (“In NSW? Meet Superior Shavings”), the footer, and a Brand node in the JSON-LD.

AC Service & Parts — refine, not replace. The paddock palette and harvest photography were already working, so the budget went to what a review found: terracotta text on bone was 3.1:1, and moving it to --terracotta-dark gave 4.7:1 on bone — and then failed on --paper and --bone-dim, so the token darkened again from #A8562F to #9A4E2A for 4.5:1 on every ground. Search Console had flagged 30 “Either offers, review, or aggregateRating should be specified” errors, because each catalogue <li> was typed as a schema.org Product with only a name and image; the Contentful model has no price or SKU, so those could never validate, and the catalogue is now an OfferCatalog on the LocalBusiness node. The two Unsplash heroes were self-hosted, the desktop nav lost an aria-hidden="true" that had been hiding it from screen readers, “no middlemen” was dropped because SpareX is a distributor, and a second, ungated gtag block at the end of <body> was removed before it could double-count.

The stockist finder belongs to the client

Breeder’s Choice and Superior Shavings are sold through produce stores, saddleries and pet retailers: 407 and 540 of them, counted from the live CSVs today. The owner maintains both lists in his own Dropbox, and has for years.

The obvious engineering move is to pull the CSV at build time: it server-renders, it’s fast, it’s cacheable. I built it the other way on purpose, and the header of stockists.js says so:

/* Stockist finder — fully dynamic by design.
   The CSV lives in the owner's Dropbox and is fetched in the browser at page
   load, so edits go live with NO site deploy. Do not move this to build time.
   No jQuery, no jquery-csv: ~40 lines of native fetch + parse. */

A build-time fetch means his edits do not appear until something triggers a deploy, and the moment his workflow depends on my CI, I have taken his list away from him. He edits a spreadsheet; the site updates. The URL reaches the script as data-csv="{{ site.stockists_csv }}", the parser is 23 lines and handles quoted fields and embedded commas, and if the fetch fails the page says so and shows the phone number.

The state matching is where the real bug was. The first version was one regex, \b(NSW|VIC|QLD|SA|WA|TAS|NT|ACT)\b, against the address fields. The review pass counted what it dropped: 74 of Breeder’s Choice’s 407 rows, and Victoria went from 185 stockists to 258 once they were recovered. The live CSVs mix VIC, Vic and Victoria, and a strict parse silently loses those rows. So it is two passes:

  function stateOf(s) {
    var hay = (s['Address 2'] || '') + ' ' + (s['Address 1'] || '');
    var m = hay.match(/\b(NSW|VIC|QLD|SA|WA|TAS|NT|ACT)\b/);
    if (m) return m[1];
    m = hay.match(/\b(new south wales|victoria|queensland|south australia|western australia|tasmania|northern territory|australian capital territory|nsw|vic|qld|sa|wa|tas|nt|act)\b/i);
    if (!m) return '';
    var t = m[1].toLowerCase();
    return STATE_NAMES[t] || t.toUpperCase();
  }

That is not defensive programming for its own sake; it is the difference between a stockist appearing and a stockist not existing.

The list renders as indexable HTML across nine pages per site — stockists/index.md plus act, nsw, nt, qld, sa, tas, vic and wa — with the Google map as an enhancement on top. The previous implementation drew everything into a map and nothing else, which meant several hundred Australian retail locations were invisible to search.

Shipping a redesign onto a live domain nobody has approved yet

This is the part I would reuse anywhere.

Both bedding sites belong to one owner, and a redesign he has not seen cannot go live — but a preview he has to take my word for is not a preview either. Staging on a .github.io URL means he reviews something that behaves differently from the real thing: different domain, different certificate, no CNAME, no analytics.

So the new design shipped to his own live domain at /preview/ on 7 August, two weeks before he approved it, while the old site kept serving / untouched.

Four things made that safe:

1. Every preview page carried two release gates in front matter. noindex: true and sitemap: false, on the home page and all nine stockist pages, so Google never saw a half-approved site or a duplicate of the live one. The og:image default was scoped to path: "preview" for the same reason: the old pages at / had to stay byte-identical.
2. The old layout stayed the default by doing nothing. There is no defaults: layout in _config.yml. The root index.md kept layout: default; every new page under preview/ opted into layout: home or layout: stockists explicitly.
3. Config keys were deliberately duplicated. The old layout read stockistscsv and phone_international, and jekyll-seo-tag appended site.tagline to the live homepage <title>. So the new templates read stockists_csv, phone_intl and brand_tagline, and both sets lived in _config.yml, each commented with which layout owned it. The comment on the alias reads “the OLD layout still serving at / reads this key. Remove when the preview is promoted.” It was added in its own commit, after the review pass found the live stockists page had lost its CSV.
4. The 404 page was rebuilt standalone, layout: null, its own markup and inline CSS, linking only home and the phone number. The CSS comment says why: “this page serves on the LIVE domain today, so it carries no nav into the unreleased design and no links that 404.” An earlier version inherited the redesign’s header, which meant any bad URL on the live domain leaked the unreleased site.

One Liquid bug is worth recording because it looks correct. The header built the stockists link as:

{%- assign stk = page.stockists_index | default: home | append: 'stockists/' -%}

Filters are a left-to-right pipeline, not a conditional. default: only substitutes when the left side is falsy, but append: runs regardless — so on a preview page where page.stockists_index was already /preview/stockists/, the link became /preview/stockists/stockists/. The fix is an explicit unless.

The flip itself, once he signed off, was mechanical: git mv preview/stockists/* stockists/, repoint home_url and stockists_index, drop the two gates, delete _layouts/default.html and the aliases only it read. One commit per site: 14 files, 45 insertions, 1,467 deletions on Superior Shavings; 47 and 1,932 on Breeder’s Choice. The design had been running on the real domain for two weeks, so there was nothing to discover at go-live.

Two things I learned from GitHub Pages in the same fortnight. It caches assets for four hours, which twice in one day left browsers running stale JS after a deploy; every asset URL on the bedding sites now carries ?v={{ site.time | date: '%s' }}. And the Pages action injects a base_path that silently breaks every asset URL if the custom-domain setting is ever dropped, so the workflow pins jekyll build --baseurl "". Google Analytics is emitted only when jekyll.environment == "production", which the workflow sets; a dev build has zero gtag loads, production exactly one.

Mower Corner, specifically

Mower Corner went live on 6 August and got the most functional work, because a shop has state a bedding manufacturer does not.

The header carries a live readout — “Open now · until 5pm”, or “Closed · opens tomorrow 9am” — computed in the shop’s timezone, not the visitor’s, so someone checking from interstate sees whether Colac is open right now. The hours live once, in _config.yml, and drive the schema.org openingHoursSpecification, the printed plate and the readout; the layout serialises them to window.MC_HOURS with jsonify. The JS uses Intl.DateTimeFormat('en-AU', { timeZone: 'Australia/Melbourne', … }).formatToParts(), normalises the hour with % 24 because en-AU can render midnight as 24, strips the weekday to three letters because CLDR data varies, and wraps the whole thing in try — formatToParts with an explicit timeZone can throw on old engines, and a status readout is never worth a blank page. It refreshes every 60 s.

Four service pages were written against local search demand rather than invented: chainsaw-service-sharpening, mower-servicing-repairs, spare-parts and pickup-and-delivery. A pickup process I had drafted was removed the same day in a commit titled “remove the pickup process I invented, keep only what’s sourced”. The range picked up STIHL’s iMOW robotic mowers, and Click & Collect is branded STIHL-only throughout, because the online-order pipeline is STIHL’s and implying you could get a Victa through it would generate exactly the wrong phone calls. site.webmanifest went from display: standalone to browser, because a brochure site for a local shop has no business prompting Chrome to install itself.

Two implementation notes I would defend anywhere:

• The mobile menu is a <details> disclosure. It opens and closes with no JavaScript. mobileMenu() is 28 lines and only adds Escape (returning focus to the <summary>), outside-click, closing on same-page anchors, and closing when the (min-width: 62rem) query flips to desktop — so a script failure degrades to a working menu rather than a hamburger that does nothing.
• Reveal-on-scroll replaced AOS with an IntersectionObserver at threshold: 0.1, staggered by Math.min(i % 4, 3) * 70 ms. It checks prefers-reduced-motion first and, if the visitor has asked for less motion or the browser lacks the API, marks everything visible. The hiding styles are armed by document.documentElement.classList.add('js') from the script itself, not the document head: if the file never loads, nothing is ever hidden.

The rebuild commit records two multi-agent review rounds over the code and 43 confirmed findings fixed, from ink-on-orange ::selection at 5.4:1 to a duplicate canonical. What I would take from the four together: put the client’s data where the client already edits it, count what a regex drops before trusting it, and stage on the real domain behind noindex rather than on a lookalike. No Lighthouse numbers were recorded, so I am not quoting any.

This post walks through the mechanisms and the failures that shaped them: the ordered triage procedure in the prompt and the rule that a failed tool may never read as an all-clear; the three ways a Worker agent fails silently that I found by running it against live incidents (background work cancelled with no error, a bundle of static files answering “method not allowed” to every inbox, and a reasoning model using up its whole output budget before writing a word); the side-by-side model comparison that changed the default model; and the tool that found a stalled site eight earlier triage turns had called a test flake.

Source → github.com/triptechtravel/thinkbot · Docs → triptechtravel.github.io/thinkbot

The division of labour

The clawdwatch post made the case that detection should be deterministic: a threshold, a state machine, a maintenance window, and no model anywhere near the decision about whether you have a problem. thinkbot is what that decision buys you.

Detection is code you can read at 3am. Explanation is the part where a model earns its keep.

Once something is definitely broken, the work is reading five systems and noticing what they have in common. A check can tell you an endpoint returned 500. It cannot tell you that a pull request merged eleven minutes earlier, that a Sentry exception first appeared inside that window, and that a Datadog metric stepped rather than wobbled at the same moment.

The procedure is in the prompt, in order

OPS_SYSTEM_PROMPT in src/agent-ops.ts is an ordered procedure, not a vibe:

When you are handed an alert, work in this order:
  1. Confirm it is still failing. runCheckNow is cheap and stops you
     explaining an outage that has already passed.
  2. Look at the check history. A single blip and a sustained outage call for
     different responses.
  3. Look for what changed. Most outages follow a deploy, so recentDeploys on
     the relevant repository is usually the highest-value call.
  4. Look for corroboration: a new Sentry error that started inside the same
     window, or a Datadog metric that stepped rather than wobbled.
  5. Check whether this endpoint has failed before and what was concluded.

A turn may make at most twelve tool calls and run for at most 120 s, and each tool result is trimmed to 1,500 characters before it goes back to the model. Then it says what it found, citing the pull request number, the exception, the ratio. If the evidence does not support a cause it says the cause is unclear and lists what it ruled out — a confident wrong answer sends someone to the wrong service and costs more than an honest “unclear”.

The clause that took a real incident to write is this one:

NEVER report the absence of evidence as evidence of absence. If a tool errored,
returned nothing, or was not configured, say so in those words — "Sentry was
unreachable" is useful and "no new Sentry errors" is a lie that reads exactly
like an all-clear.

Findings worth keeping are written back with annotateIncident to the clawdwatch API at MONITORING_URL, authenticated with a Cloudflare Access service token. The signed per-incident action links described in the clawdwatch docs are not yet attached to alerts, so this is the standing credential thinkbot does hold — read scope on the sources, write on annotations.

Silence is a valid outcome

If triage found nothing that explains the failure, the model replies NOTHING and the channel gets nothing. The return contract is result.text.trim(): an empty string means “nothing worth saying” and the caller decides whether that becomes silence or a log line. triageAlert logs [alert] triage found nothing to report and returns.

An agent that always produces a paragraph will always produce a paragraph, and under an incident someone is trying to read, filler is worse than absence. Getting this right meant saying it three times over: in the prompt, in the return contract, and in each channel’s decision to post. Chat is the one exception, on purpose — a person waiting on a thread cannot tell a silent bot from a broken one, so answerChat always replies, with “I could not finish that one” if it must.

Two transports, one triage path

Service binding, preferred. A service binding is a direct call from one Worker to another on the same Cloudflare account, and clawdwatch calls the AlertInbox entry point that way:

"services": [
  { "binding": "AGENT", "service": "thinkbot", "entrypoint": "AlertInbox" }
]

The platform authenticates the call. No shared secret, no public endpoint.

Signed webhook, for anything that cannot bind. POST /hooks/clawdwatch verifies a keyed hash (an HMAC) over the timestamp and the body, using clawdwatch’s own exported verification function and header names rather than a local reimplementation, because two implementations of one signature scheme is a bug with a delay fuse.

And the sharp edge, stated in both repos: a direct Worker-to-Worker call carries no signature. The AlertInbox comment says there is correspondingly no check there, that the check belongs to the HTTP adapter, and that triageAlert is written not to assume it ran.

Failing CI runs get triaged the same way

POST /hooks/e2e takes a signed report from a GitHub Actions runner when the nightly browser test suite (Playwright) fails. A binding is not available — bindings are same-account only and a runner is not on the account — so this is a keyed hash under thinkbot’s own header names, x-thinkbot-signature and x-thinkbot-timestamp, with its own key, E2E_WEBHOOK_SECRET. That is a different key from the monitoring inbox: a CI runner is a different sender in a different trust domain, and leaking one key must not grant the other.

The payload is deliberately not a clawdwatch AlertEvent. A test run is not a synthetic check; there is no incident to annotate and no links to act on, so forging one would hand the agent a prompt telling it to call annotateIncident against an incident that does not exist. It carries repo, sha, runUrl, loadError, failures, passed and skipped, and the type guard checks only that repo and sha are strings — the signature proves key possession, not sensible content.

Two details earned the hard way:

• loadError with no failures is a distinct incident. The prompt branch says so in capitals: “E2E SUITE FAILED TO RUN … No tests executed, so this says nothing about whether the site is healthy.” Reporting that as “0 tests failed” is how a real two-night outage read as noise.
• This path always posts. There is no second notifier behind it and nobody opens the Actions tab. The headline is the floor; the paragraph is what gets added.

Exercising the path that is not exercised

Removing the CI workflow’s own Slack step left one delivery path and no way to tell whether it worked. So a report may set "probe": true. It travels the same route and posts the same way — a probe down a different code path proves nothing about the one a real failure takes — but the headline is 🧪 … E2E alert path probe — delivery works, nothing is wrong, first and unmistakably, and the prompt short-circuits to “Reply with the single word NOTHING. Do not investigate.”

That last part is not an optimisation. An agent asked to explain a non-event will invent one.

The first real report answered 200 and posted nothing

The probe worked. Then the first real failure came through: 200 to the runner, no error in the logs, nothing in Slack, fifteen minutes on. The difference was that a probe’s model turn returns immediately and a real one makes tool calls.

The message was built as headline plus finding, inside the background work the Worker is allowed to keep doing after it has replied (waitUntil). So a slow turn swallowed the alert. The first fix split them: post the headline in the request itself — one Slack call, milliseconds — and let the finding follow as a second message. Worst case became a headline with no explanation, never an explanation nobody receives.

That was not the whole bug. Running recorded reports through the live deployment, one turn finished in 28 s and the rest were killed:

waitUntil() tasks did not complete within the allowed time after invocation
end and have been cancelled.

Background work gets about thirty seconds after the reply. A triage turn that makes four or five tool calls regularly needs more, and the runtime cancels it with no error and no message. The monitoring inbox had the same shape, which means clawdwatch triage had been dropping silently whenever it ran long — invisible by construction, because the thing that failed was the thing that would have told you.

Every model turn now runs as a job on a Cloudflare queue instead:

  "queues": {
    "producers": [{ "queue": "thinkbot-triage", "binding": "TRIAGE_QUEUE" }],
    "consumers": [
      {
        "queue": "thinkbot-triage",
        "max_batch_size": 1,
        "max_batch_timeout": 0,
        "max_retries": 0,
        "dead_letter_queue": "thinkbot-triage-dlq"
      }
    ]
  },

Retries are off and every job is acked, including failed ones, because a retried turn may already have posted its finding, and a duplicate explanation under an incident someone is reading is worse than the missing one it recovers. The turn stays bounded at 120 s, below the consumer’s budget, so a hang leaves a log line saying so — the difference between “found nothing” and “never finished”.

Moving to a queue had a knock-on: everything an inbox receives has to survive being written out as JSON. The channel parsers used to hand back a reply function that remembered the Slack thread or Telegram chat it belonged to, and a function cannot go in a queue message. It is now a plain description of where to reply (the channel and either the Slack conversation and thread or the Telegram chat id), with a test that pins the one property that matters, that it survives the round trip through JSON.

Static assets are a second front door

thinkbot holds a GitHub personal access token, Datadog and Sentry keys, and write access to monitoring incidents. workers_dev and preview_urls are off and every route verifies its caller before doing any work. That was true of the routes and still not true of the deployment.

The Worker was scaffolded from Cloudflare’s agent starter template with a React chat page, and even after the page was disabled, deploy still built it and uploaded the bundle of static files. Static files are served before the Worker’s code runs. So the bundle answered requests no route guard ever saw, and that single fact produced three bugs:

• an unauthenticated GET to the hostname returned the chat page;
• /health answered 200 with the chat page’s HTML instead of the health handler — a liveness endpoint that cannot fail, reporting healthy through the outage it exists to catch;
• every signed inbox returned 405, “method not allowed”, because the static file handler rejects anything but GET rather than passing it on to the Worker. Verified against the live deployment: POST to /hooks/clawdwatch, /hooks/slack and /health alike. Monitoring never noticed, because it reaches thinkbot over the direct binding, which does not pass through the static files.

The first fix listed the paths that should reach the Worker first ("run_worker_first": ["/agents/*", "/oauth/*", "/hooks/*"]), then added /health to the list. That moves the boundary and leaves the rest in place. The real fix is having no static files at all: with the bundle gone, every request lands on the code that checks its caller. Deleting the chat page also dropped React, Tailwind, the Vite build and eleven packages; removing the now-unused chat agent’s Durable Object (Cloudflare’s single-instance stateful object) took two more packages with it, and the Worker bundle went from 2.8 MB to 1.2 MB.

One local trap, recorded only in the commit message and worth knowing if you mix Vite (the front-end build tool) and Wrangler (Cloudflare’s deploy tool): the front-end build writes a small config file under .wrangler/deploy/ that points Wrangler at the built dist/ folder instead of your source. Both are ignored by git, so a stale redirect keeps deploying the old bundle while ignoring your edits to wrangler.jsonc. Remove .wrangler/deploy and dist before the next deploy.

The model posted 256 exclamation marks

npm test mocks the model — 192 tests, 469 ms, free and offline. That is the right default, and it means the whole suite goes green whatever the model emits. On 2026-08-22 a collapsed generation put 256 exclamation marks into the alert channel under a live headline about three failing specs, where it read as the alerting itself having broken.

Nothing sat between the model’s output and Slack but an empty-string check and the word NOTHING. Now usableFinding() in src/triage-output.ts judges the output on properties, not quality: one character making up more than 40% of it, letters making up less than 35% of it, or fewer than 24 characters in total, and in each case it prefers silence. It cannot separate a correct diagnosis from a confident wrong one; no rule can.

The wall was exactly 256 characters, and ! is one token. That is Workers AI’s default completion cap, not a content limit, so maxOutputTokens is now set explicitly — which makes a collapse longer, and is only safe because the guard drops it first.

To exercise the model half at all, POST /hooks/e2e/dry-run runs the same prompt through the same turn behind the same signature and returns the result instead of posting it, and test/fixtures/ holds real payloads recovered from the workflow logs of the runs that produced them. Invented reports correlate with nothing. The integration test’s header is explicit about what it can check: that the output is prose, finished, hedged when the evidence is thin, and silent on a probe. Not that the diagnosis is right.

Two models on the same incident, and what neither found

scripts/triage-harness.mjs runs two models against the same recorded report. On 2026-08-22, six turns each:

A made-up negative is indistinguishable in an incident channel from a real all-clear, which decided it. The slower model is about twice as slow per turn, and this runs on a nightly, so the cost is nothing. DEFAULT_MODEL is @cf/moonshotai/kimi-k2.6 as of that run.

What the comparison did not show is either model finding the cause. Both concluded “test flake, site healthy” about an incident that was a server-side render stall, because no tool reached the place the evidence was. A better model would have been confidently wrong too.

The one source that would have been right

The site was stalling for 40–60 s on roughly a third of cold renders and truncating the HTML stream. Every source the agent could reach agreed it was healthy: nothing merged in the window, no new Sentry issue, no Datadog alert, and the uptime checks — which fetch a page and read the first byte — were green. A response that starts in 100 ms and never finishes is invisible to all of them. In the Worker’s own invocation telemetry it reads:

wallTime 59971ms, cpuTime 117ms, outcome canceled

A minute of elapsed time against a tenth of a second of processor time can only be a request stuck waiting on something else. So workerHealth in src/tools/workers.ts queries Cloudflare’s Worker telemetry and states the verdict rather than leaving it as arithmetic:

const STALL_WALL_MS = 10_000;
const STALL_RATIO = 20;

Over the real incident window it returns STALLED: p99 wall 52572ms against p99 CPU 1172ms — 45:1. That is a request waiting on I/O that never returns, NOT slow code. (the “p99” figures are the slowest one per cent of requests). Over the window after the fix: Healthy, with the slowest one per cent taking 6,538 ms. It returns totals and percentiles only, never log lines, which run to thousands a minute and carry request URLs and headers you then pay for in context on every later turn. Its first version asked the API for p50, which Cloudflare rejects with a failure flag and an empty error list, so it failed silently while every unit test passed; median is the supported spelling, and workers.itest.ts drives the real API to catch the next rename.

With the window supplied, triage reached the conclusion eight earlier turns had missed: “requests are starting and then stalling on I/O, not slow code. The uptime checks are all green because they measure first byte only. This is a failing site, not a failing test.”

The empty turns were the token ceiling

Even with the right tool, two runs in three did the most work — GitHub, Datadog, Worker telemetry across four or five steps — and then ended without writing the paragraph. Two fixes were reasoned to rather than measured: the first assumed a conversation-continuation problem, the second a malformed tool-call exchange. Both were plausible. Each was declared a fix on a sample too small to show otherwise.

Logging finishReason and usage settled it in one run:

finishReason=stop    outputTokens=153     1 turn
finishReason=length  outputTokens=2048    4 turns, both retries included

2,048 was the output budget. A reasoning model was spending the entire budget thinking and had nothing left to write, and the retry hit the same wall. Raised to 8,192: five runs after, five of five answered, no empty turns. The galling detail, now recorded beside MAX_OUTPUT_TOKENS so the number is never guessed at again: the comment on the old value already said the headroom was for “a reasoning model’s hidden tokens”. Right worry, wrong number.

A tool that cannot answer costs a step and a sentence

This deployment carried a Rollbar tool with an invalid token for months. Every triage turn spent one of its twelve steps to be told “forbidden”, and once the prompt required reporting sources that failed, every write-up carried “Rollbar could not be checked” as though it were a finding. Rollbar left the estate on 2026-08-22 and the tool went with it.

The general fix is structural. src/tools/registry.ts declares each provider as a name, the env vars it cannot work without, and a builder; a provider missing any of them is not offered at all — not offered and failing, not offered. GITHUB_OWNER and SENTRY_ORG have no defaults for the same reason a default would mean an unconfigured deployment quietly querying somebody else’s organisation. What a deployment knows about itself lives in ESTATE_NOTES, free-form prose appended to the system prompt and read by a model, not parsed. Baking one organisation’s inventory into the prompt is what makes a general tool unusable by anyone else, and stale for its owner the first time the estate changes.

Keep the dumb notifier

From the production configuration on the clawdwatch side:

notifiers: [
  slack({ webhook: '${SLACK_WEBHOOK_URL}' }),
  rpc({ name: 'thinkbot', binding: (env) => env.AGENT }),
]

Slack is listed first and deliberately kept. If the agent is the only alert path, an agent outage is a monitoring outage, and every failure above — the cancelled background work, the “method not allowed” from the static files, the exhausted token budget — would have been one. A failing notifier cannot affect the other, so the plain, model-free Slack message always goes out. If you take one idea from either project, take that one.

Try it

git clone https://github.com/triptechtravel/thinkbot
npm install && npm test

Then wrangler queues create thinkbot-triage-dlq and thinkbot-triage, give it tokens for whichever sources you use, tell it who you are with GITHUB_OWNER and SENTRY_ORG, and point clawdwatch at it. SETUP.md is the short version; SECURITY.md is worth reading first if you plan to turn on captureBodyOnFailure for endpoints whose error paths can return personal data.

What building it taught me, in order of cost: measure before diagnosing, because two of the fixes above were reasoned to and wrong; an agent’s worst failures are silent ones, so every path needs a floor that does not depend on the model; and the tool set should be the estate’s, not the author’s. The next open item is on the clawdwatch side — attaching the signed action links so annotateIncident can drop its standing credential.

This post walks through the parts I had to get wrong first: why the microphone has to be opened with WebRTC’s processing switched off, why pitch comes from a 4096-sample autocorrelation window and not an FFT, the 400 ms onset and 250 ms grace that stop a wobble ending your record, how a “perfect week” is prorated when you join on a Wednesday, and what the three.js lane does at 4 s, 3.2 s and 9 s.

Source → github.com/isaacrowntree/sustain (MIT) · clone it and pnpm dev — there’s no hosted demo, and the app never sends your data anywhere.

The intervention is turning up 120 times

I played trombone at a decent level and then stopped for years. I also snore, and the suggestion that came back was: play the didgeridoo.

That sounds like folk advice. It isn’t. Puhan et al., BMJ 2006 is a randomised controlled trial: 25 people with moderate obstructive sleep apnoea, four months, and the didgeridoo group’s apnoea–hypopnoea index dropped by about half, with significant improvements in daytime sleepiness and in how much their partners were disturbed. It later won the 2017 Ig Nobel Peace Prize, which makes people assume the science is a joke. The mechanism is unglamorous: circular breathing and sustained droning are resistance training for the upper airway, and the same effect shows up in the myofunctional-therapy literature with no instrument involved.

The part nobody puts on the poster is the dose. The protocol was at least 20 minutes a day, at least five days a week, for four months, and the participants beat it, averaging about 25 minutes across nearly six days. That is roughly 120 sessions. The intervention is not the didgeridoo; the intervention is turning up 120 times.

So the problem I had was not “how do I play the didgeridoo.” It was “what gets me to session 87.”

What already existed

Two things, worth knowing before reading mine. Silent Sleep Training is built by Alex Suarez’s company — Suarez taught the participants in the BMJ trial — and bundles a silicone instrument with an app that listens to a 15-minute daily session. Didge For Sleep is a real travel didgeridoo plus an apnoea-specific video curriculum, with no app and no tracking.

What did not exist was the overlap: a real didgeridoo, a 30–40 minute guided session, and a microphone that verifies you played. The mic-verified music trainers (Yousician, tonestro) need pitched repertoire and cannot score an unpitched drone. A GitHub search for a didgeridoo or circular-breathing trainer returns an acoustic-impedance calculator and an operating system.

So: build. But build the general thing, because a drone hold and a trombone long tone are the same exercise wearing different hats.

The instrument already keeps score

The easy version of this app is a Duolingo skin: streaks, XP, a celebration when the bar fills. I did not build that, because the didgeridoo already keeps score honestly. Your longest unbroken drone is a real number about your actual throat, and wrapping fake currency around a real measurement does not add motivation — the overjustification research suggests it displaces it.

So progression is the curriculum, and it is implemented as three mechanisms in packages/core.

Records, not scores. The pack declares two metrics, longest-drone and longest-unbroken, both direction: 'higher'. A record is a { value, date, verified } entry; verified is whether a microphone measured it or you typed it. The test in core.test.ts records 12, then 10, then 20: the first and third are personal records, the second is not, and the third reports previousBest: 12.

Unlock by demonstration. A drill lists prerequisites and unlockedDrills() in unlocks.ts is the whole gate:

    case 'metric': {
      const best = state.bests[req.metric];
      if (!best) return false;
      const def = pack.metrics.find((m) => m.id === req.metric);
      return def?.direction === 'lower' ? best.value <= req.gte : best.value >= req.gte;
    }
    case 'self-report':
      return state.selfReports[req.id] === true;
    case 'drill-completed':
      return (state.drillCompletions[req.drill] ?? 0) >= (req.times ?? 1);

The circular-breathing chain in the didgeridoo pack runs in a fixed order: cheek-puff breathing (three demonstrations), then water spray (two), then straw and glass (three, plus a self-report: “Can you keep the straw bubbles unbroken through a sniff?”), then circular breathing on the didgeridoo (two), then bridging the gap. After that, the unbroken-sets drill needs something measured rather than reported: a single unbroken note of 60 seconds. When nothing has been demonstrated yet, the session compiler does not skip the gate; it walks back down the chain, and the week-six test asserts that the session contains cheek-puff breathing and not circular breathing on the didgeridoo.

Perfect weeks, not daily streaks. The evidence protocol is five days a week, so rest days are scheduled, not failures; the habit-formation work found a single missed day does not impair habit formation. Two details in adherence.ts cost more thought than the rest of the file:

    // The join week only asks for the practice days that were still ahead.
    let target = fullTarget;
    if (wk === mondayOf(join)) {
      const missedDays = Math.min(daysBetween(wk, join), fullTarget);
      target = Math.max(1, fullTarget - missedDays);
    }

Join on a Wednesday, practise Wednesday to Friday, and the test asserts target is 3, the week is perfect, and the streak is 1. Nothing is more discouraging than a program that opens by scoring you against days that had gone by before you downloaded it.

The second is the make-up: compileSession() returns null on a rest day unless called with { makeup: true }, and the adherence test practises Monday to Thursday, misses Friday, plays Saturday, and asserts the week is perfect. Nearly every habit app treats the miss as the punishable event. The thing worth protecting is the return.

That bar chart is the sixteen-week journey, one bar per week, height being sessions completed and colour the phase. It is the only “gamified” element and it is a picture of the truth.

A session is a lane

Thirty-five minutes of staring at a countdown is how a practice habit dies. So a session is a lane of timed segments flowing toward you in a three.js night scene — the Guitar Hero sustain bar stretched to the length of a breath.

The whole scene is one 449-line class, PracticeScene in apps/web/src/scene/scene.ts. A segment is seg.seconds * SCALE units long with SCALE = 0.55, playing segments are 0.42 units tall and the full 2.2-unit lane width, rests are 0.1 tall and 45% width, and the lane group is translated by elapsed * SCALE each frame so the strike point never moves. Most of the rest is borrowed from software that solved these problems years ago:

• The next segment brightens when remaining < 4 seconds, ramping its opacity linearly over the 4 s — telegraphing, not a countdown. The audio pre-cue chime is separate and earlier-tuned: it fires at 3.2 s, and only for segments of 10 s or more, so a run of short rests does not chime continuously.
• Rests get a breathing halo on an 8 s cycle, 0.5 - 0.5 * Math.cos((t % 8) * (2π / 8)), paced like the Apple Watch Breathe animation, because in a didgeridoo program the rests are breathing exercises. Under prefers-reduced-motion it holds still.
• The HUD fades after 9,000 ms without a pointermove, pointerdown or keydown, by adding a quiet class; any input removes it.
• The played trail rides behind the strike point, a fill mesh whose scale.z grows with progress: bright where the drone lived, dim where it dropped.

One thing I got wrong and fixed in a nine-line commit: the camera was at (0, 2.4, 4.6) looking at (0, 0.5, -7), and from there the upcoming segments stacked into each other. Raising it to (0, 3.1, 4.4) and looking further down the lane at (0, 0.35, -8) made the lane ahead readable. Same commit dropped the bloom multiplier from 0.8 to 0.5; the ember was flaring the whole scene.

Listening without a server

Verification is tiered, and the app is fully usable at tier one. The engine consumes a PracticeSource and does not care which tier produced the frames:

 *  - 'timer'  — always playing while running (honor system, no mic)
 *  - 'energy' — mic loudness gate (reactive visuals, coarse credit)
 *  - 'pitch'  — pitch-verified playing (auto-measured metrics)

The timer is the default until the mic is granted; it is how the BMJ trial ran, with a paper diary. The energy tier is an eight-line class over an adaptive noise floor — gate() is max(0.01, noiseFloor * 3), and the floor is learned only from frames that are already quiet. The pitch tier is where the work was.

The browser fights you. Opening the microphone with default constraints silently attenuates a drone. mic.ts explains why in the comment above the call:

/**
 * Open the mic with browser audio processing disabled. WebRTC noise
 * suppression classifies sustained tones as background noise and attenuates
 * them, and auto-gain pumps a continuous drone — both must be off.
 */

So it is echoCancellation: false, noiseSuppression: false, autoGainControl: false, all three.

Not an FFT. Pitch is pitchy’s McLeod Pitch Method, an autocorrelation-family detector, over a time-domain buffer:

const WINDOW = 4096; // ~85 ms at 48 kHz — 5-7 periods of a 60-90 Hz drone

findPitch returns a frequency and a clarity, and a frame counts as playing when clarity >= minClarity && pitchHz >= minHz && pitchHz <= maxHz. The thresholds come from the pack, not the detector, and the didgeridoo pack sets minHz: 50, maxHz: 100, minClarity: 0.8, stabilitySemitones: 3, with a comment that a beginner’s drone wobbles and too strict a gate reads as “the app can’t hear me”. The 50–100 Hz band is what keeps speech and room noise out, so the other two can afford to be forgiving. Two further gates: over any 1 s window with at least five samples, a pitch spread wider than 3 semitones is not a drone; and a reading within 1 Hz of 50 or 60 Hz at RMS below 0.02 is mains hum, not you.

Hysteresis lives in the clock, not the detector. PlayingClock in clock.ts has two constants, onsetMs = 400 and graceMs = 250. A frame stream has to be “playing” for 400 ms before the clock starts, and then the whole qualifying window is credited retroactively — you were playing during it, the clock was just waiting to be sure. Once running, a gap shorter than 250 ms keeps crediting through the wobble. Per-frame credit is capped at 500 ms and a gap over 2,000 ms is treated as a discontinuity (a backgrounded tab), so a stall cannot mint minutes. Swapping sources resets the clocks, so timer time can never be reported as verified.

All of it runs in the browser. Progress and recordings live in two IndexedDB object stores, progress and recordings, through a 79-line hand-rolled wrapper — the web app’s only runtime dependency is three. There is no account, no server and no telemetry. The network requests the app makes are to Google Fonts, for Fraunces and Karla; nothing else.

Packs are data

An instrument pack is metrics, a curriculum of phases and drills with prerequisites, and an analyzer spec. The engine’s entry point is one function:

export function compileSession(
  pack: InstrumentPack,
  day: ProgramDay,
  unlockedDrillIds: Set<string>,
  options: CompileOptions = {},
): CompiledSession | null {

Session length ramps across a phase by linear interpolation on phaseProgress, which is week position within the phase’s [start, end]. The didgeridoo pack is daysPerWeek: 5, totalWeeks: 16 in four phases — Foundation (weeks 1–2), Breath mechanics (3–5), Connection (6–9), Endurance & voice (10–16) — each closed by a measured assessment on the phase’s last practice day: 120 s per metric plus 45 s rest. The third one is named “The loop” and its subtitle is “Sixty seconds is the gate.”

The useful accident is that the pitch analyzer is instrument-agnostic. A trombone pack is the same detector with a different minHz/maxHz, and gets more out of it, because a detector that reports hertz can measure long-tone steadiness in cents. Guitar needs a polyphonic analyzer, which is a different problem, and is marked help-wanted rather than pretended at.

Where it’s at

pnpm -r test is green: 42 tests (17 in core, 13 in apps/web, 5 each in audio and pack-sdk, 2 in the didgeridoo pack) cover the engine, the clock, and the web app’s storage; the scene is checked by looking at it. The pack’s baseline-recording on day one and summit-recording at week 16, both 30 s of MediaRecorder keyed ${pack.id}:${sessionDate}:${drillId}, play back to back from a Compare view on the home screen — earliest day one, latest summit, and “not recorded yet, week N of 16” until the summit exists. Export is a versioned bundle, { version: 1, exportedAt, progress, recordings }, one JSON file with every recording inline as base64, and import validates the version and the pack, writes the recordings back into IndexedDB, and will not replace existing progress without a confirmation. The engine, the didgeridoo pack, all three microphone tiers and the web app work end to end, and for a local-first app with no backend, pnpm dev is the deployment.

The instrument is a PVC didgeridoo from African Drumming. The BMJ trial handed its participants plastic instruments because they are easier to learn on, and the effect comes from what your airway is doing, not what the tube is made of.

So there is a curriculum, a coach, an instrument, and an empty progress file. What I learned building it: the browser’s audio defaults are tuned to remove exactly the signal a drone is, the clock needs more care than the detector, and the hard part of a habit app is the week you join and the day you miss. The day-one recording is on disk, and the summit will play back beside it when it lands. The trombone pack is next. Ask me in December.

A note worth making plainly: the didgeridoo is an Aboriginal Australian instrument, known as the yidaki to the Yolŋu people of north-east Arnhem Land, with a ceremonial history that long predates its use as a snoring intervention. Sustain is a practice timer for a breathing exercise. It makes no claim on any of that.

Header photo by Wil Stewart on Unsplash.

Live demo (drop a 3D model, watch it become a drawing): isaacrowntree.com/linework · Source: github.com/isaacrowntree/linework (MIT)

This post covers why no existing library does this, how render() sorts, how featureEdges() decides which edges of a shaded mesh become lines, how STEP files (the CAD exchange format) work without shipping a CAD geometry engine in the package, and what the 25 tests actually pin down.

Why this exists

I’ve been building a “will this part fit your bike?” planner, a tool that tells you whether a given rear shock, dropper post or ebike motor bolts onto your frame before you spend the money. It grew out of my earlier bike-part-planner engine, and the part that makes it useful is the drawing: an exploded, service-manual-style diagram, every part numbered with a callout balloon that points at a parts list, that you can grab and rotate.

So I needed exploded technical diagrams, rotatable, with dimension lines, callout balloons and a title block. I went looking for a library and expected to find three.

No library does this

The 3D tooling world splits in two and neither half wants this job.

On one side is three.js and WebGL. It is excellent at shaded surfaces, lights, materials and cameras. Everything that makes a technical drawing readable is a fight there: one-pixel strokes, dashed hidden lines, a hierarchy of line weights, and text callouts that have to live in a separate HTML layer on top of the canvas. I would have spent the effort making a renderer stop doing what it is good at.

On the other side is Zdog, the one pseudo-3D library with the right look. Its last npm release was v1.1.3 in January 2022; it sorts shapes by one depth value each, so shapes that pass through one another draw in the wrong order and you nudge their depth offsets by hand until they don’t; and it has no text. Dimension lines and leader text are half of an exploded diagram. A renderer that cannot do text cannot do this.

Nothing in between produces technical illustrations from parameters. So I wrote it.

Rotate, project, sort, paint

The pipeline is the textbook one. Rotate the 3D points about the vertical and horizontal axes; project them with perspective, so that a point’s on-screen scale is f/(f−z) and nearer points draw larger; sort every shape from farthest to nearest (the painter’s algorithm: paint the back first so the front covers it); then write SVG strings. The important word is sort, and the important property is that it happens every frame from the projected depth of every shape, never from a layer order the author supplies. This is the whole of render():

export function render(shapes: Shape[], view: View, parts: Record<string, PartDef> = {}): string {
  const X = xform(view);
  const slots: Slot[] = [];
  const byPart = new Map<string, { svgs: string[]; depths: number[] }>();
  for (const sh of shapes) {
    const e = emit(sh, X);
    if (!e.svg) continue;
    if (sh.part) { /* accumulate into byPart */ } else { slots.push(e); }
  }
  for (const [key, b] of byPart) {
    const depth = b.depths.reduce((a, c) => a + c, 0) / b.depths.length;
    slots.push({ svg: `<g ${parts[key]?.attrs ?? ""}>${b.svgs.join("")}</g>`, depth, part: key });
  }
  slots.sort((a, b) => a.depth - b.depth); // far → near
  return slots.map((s) => s.svg).join("");
}

Shapes that share a part key are averaged into one depth and wrapped in one <g> group, so an animated part moves as a unit. A box’s faces are four-sided polygons, and a face whose corners come out clockwise on screen is facing away from the viewer, so it is skipped (if (sh.cull && area <= 0)) and the back of a box never paints over its front. A disc projects to the correct ellipse for the plane it sits in, which is what wheels and bearing races need. A bias field nudges a shape’s depth by hand for the cases every painter’s renderer has, where two shapes sit in the same plane, and the README says so: three shapes that each overlap the next in a cycle cannot be sorted, so split one of them.

Here is a bearing assembly, exploded. This exact SVG was rendered by the library, in Node, with no client-side JavaScript:

A shaded mesh has no lines

The pillow block was written by hand. The larger unlock is linework/import. A mesh carries no lines; its form lives in where the surface bends, so the drawing has to be recovered from the triangles. featureEdges() does it in three passes:

1. Weld vertices by position. Meshes routinely split one geometric edge across two pieces with separately numbered vertices. Each vertex position is rounded to a grid one ten-thousandth of the bounding-box diagonal and mapped to a single representative, so a shared edge is seen as shared and not as two boundaries.
2. Collect the face directions at each edge. Each triangle contributes its surface normal (the direction it faces) to its three edges, keyed on the pair of welded vertex ids. Triangles with zero area are skipped; they would otherwise produce false creases.
3. Keep the edges a draftsperson would ink. The figure shows the three cases.

The crease threshold is the angle option, 25° by default. The tests say what that means concretely: a closed cube yields exactly its 12 edges and none of the diagonals across its faces; raising the threshold past 90° drops every interior edge; an open triangle is three boundary edges; vertices at the same position but numbered separately are welded. One call does the whole thing:

import { parseGLB, meshToShapes } from "linework/import";
const meshes = parseGLB(await file.arrayBuffer());   // parseOBJ · parseSTL · fromBufferGeometry
const shapes = meshToShapes(meshes, { angle: 25, fit: { cx: 400, cy: 300, size: 440 } });

parseGLB reads embedded buffers and node transforms; it does not support Draco compression, and it reads geometry only. Here is the lantern, 5,394 triangles of shaded mesh reduced to about 2,500 edges and rendered in Node by scripts/gen-import-demo.mjs:

That reframe is why this was worth releasing. “Hand-code a 3D scene” is a small audience. “Bring the model you already have” is a larger one. Drop a .glb on the live demo and it runs in your browser; nothing is uploaded.

STEP files without shipping a CAD engine

STEP and IGES files describe exact mathematical surfaces (boundary representations built from trimmed NURBS patches), not triangles, and turning those into triangles is a job for OpenCASCADE, an open-source CAD geometry engine, not for 171 lines. So linework does not embed one. fromOcct() maps the output of occt-import-js straight into the same Mesh type; the roughly 6 MB WebAssembly build of that engine stays an optional dependency of your app, and linework stays dependency-free. fromBufferGeometry() does the same for a three.js geometry by reading its typed arrays, without importing three.js.

Why SVG strings, specifically

• You style it. The library emits class names you choose and never bakes in a colour, a stroke width or a font. Theme it with CSS variables; restyle it for dark mode; print it.
• Text is text. Callouts and dimensions are real <text>; a screen reader and a crawler both see them.
• It renders on a server. render() is a pure function from shapes to a string. Both diagrams in this post came out of Node at build time. The live demo runs the same dist/ files under your pointer.

Making it nice to author

linework/sketch sits on top of the raw primitives so a scene reads like drafting instead of assembling arrays of numbers. Context blocks set which part, which CSS class and which depth bias everything inside them belongs to:

import { sketch } from "linework/sketch";

const s = sketch({ yaw: 0.5, pitch: 0.16, f: 1500, cx: 460, cy: 320 });

s.box(300, 380, 320, 42, 40, 80);                 // base plate

s.part("housing", 'class="prt"', () => {          // one <g>, one sort unit
  s.cyl([460, 218], 70, 34, -34, "ink");          // bearing body
  s.bias(0.6, () => s.cap([460, 218], 34, 32, "ink"));
});

s.note("320 mm", [460, 452]);                      // paper-space annotation
el.innerHTML = s.render();                         // depth-sorted SVG

Annotations in paper space (fixed to the page rather than to the model) are appended after the sorted scene and never rotate, exactly like the notes on a real drawing; sketch.pt(p, z) projects a model point to the page so a leader line can pin a note to a part. linework/orbit is the drag-to-rotate code as one call: pointer capture, limits on how far the view can turn, batching of redraws to the browser’s animation frame, double-click to reset, and an idle sway that switches off when the user has asked for reduced motion. It is the only entry point that needs a browser.

npm i linework          # ESM, types included, zero runtime deps
npm test                # projection, parallax, sorting, culling, edge extraction

Numbers

The README quotes about one millisecond to rebuild a scene of a few hundred shapes. That is a figure from the demo, not a benchmark I have recorded, and I have not measured the lantern’s per-frame cost.

The lesson

When a problem looks 3D there is a reflex to reach for the 3D engine and then spend a week teaching it to draw a labelled diagram. The actual need was smaller and more specific than the tool everyone reaches for, and nobody had written the small specific thing because it sits between “serious 3D” and “cute toy”. That gap is where a lot of good little libraries live. This one is MIT on GitHub: take it, theme it, drop in your own model. The bike planner it was built for is next.

This post walks through why a Durable Object already is the ordering point that sync engines run Postgres to get, the four silent data-loss bugs the code encodes (the res.ok one, a cursor pointing past a reset log, a pull landing mid-workout, and one user’s writes filed under another’s journal), a fifth found while writing tests for the SQL path that had never executed, the comparison table I got wrong in my own favour, and the 71 unit and 16 workerd tests that now stand in front of all of it.

Docs & live demo → isaacrowntree.com/durable-sync · Source → github.com/isaacrowntree/durable-sync (MIT) · npm → durable-sync

The ordering point is already there

The credible offline-first engines in 2026, Zero, Electric, PowerSync, all want Postgres and a service in front of it. On an all-Cloudflare stack that is a second backend, run forever, to get one property: a single place that decides the order of one user’s writes. A Durable Object is that place by construction. It is single-threaded, it has its own SQLite, and env.JOURNAL.idFromName(userKey) gives every user their own. The header comment in src/server/journal.ts is the whole design:

The Durable Object is the single-threaded consistency point that orders one user’s ops; this module is the pure logic, so it’s testable without a Workers runtime. Ops are opaque: { opId, kind, payload }. The log never interprets a payload, which is what keeps it a primitive rather than a database.

The schema is three tables:

CREATE TABLE IF NOT EXISTS ops (
  seq INTEGER PRIMARY KEY AUTOINCREMENT,
  op_id TEXT NOT NULL UNIQUE,
  kind TEXT NOT NULL,
  payload TEXT NOT NULL,
  ts INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS snapshot (id INTEGER PRIMARY KEY CHECK (id = 1), seq INTEGER NOT NULL, blob TEXT NOT NULL);

op_id is the idempotency key, so a client that is not sure a push landed retries it and the log ignores the duplicate. Because the DO is single-threaded the check-then-insert has no race. There is no conflict resolution on purpose: if writes are events (“this workout happened”) rather than edits, two devices appending do not conflict, they interleave. HTTP rather than WebSocket, because gyms are Faraday cages and a socket that cannot connect is not a feature. The server is a class you extend and a route you write; what you do not forward, a client cannot reach:

// examples/notes/worker.js
import { SyncJournal } from "durable-sync/server";
export class Journal extends SyncJournal {}

const journal = env.JOURNAL.get(env.JOURNAL.idFromName("demo"));
if (req.method === "POST") {
  const { ops } = await req.json();
  return Response.json(await journal.push(ops));
}

reset() exists on the DO. Nothing routes to it. That is the access model, and it is why 0.3.0 replaced the DO’s internal fetch router with native RPC methods: the router was a layer I did not write, sitting between the Worker and the log, special-casing DELETE.

Bug one: a successful status code is not evidence

The first version of the client in Rampset did what every example does: POST the outbox, and on res.ok delete the queued rows. Cloudflare Access sits in front of the Worker. When the Access session expired, the request was redirected to the login page on the same origin, fetch followed the redirect, and the response was a 200 with an HTML body. The outbox was drained against it. The fix in src/client/transport.ts is a type guard on the body, and the comment above it is the incident report:

/** Did this response really come from the journal?
 *
 * Behind an auth proxy (Cloudflare Access, and others), an expired session
 * redirects to a same-origin login page — which fetch FOLLOWS, and reports as
 * a 200. `res.ok` is therefore not evidence of anything. The outbox may hold
 * the only copy of a write, so it may only be drained against a reply the
 * journal demonstrably wrote. */
function isJournalReply<T extends { seq?: unknown }>(body: T | null): body is T & { seq: number } {
  return typeof body?.seq === "number";
}

The client also sends x-requested-with: XMLHttpRequest, which makes a well-behaved proxy answer an expired session with a 401 instead of a login page. That is belt and braces; the guard is what protects the outbox.

Bug two: a cursor only means something against the log that issued it

Each client stores the highest seq it has applied and pulls seq > cursor. During the incident I reset the journal. Every client was then holding a cursor from a log that no longer existed, usually pointing past the rebuilt one, so seq > cursor matched nothing, and every client silently never synced again. Now the meta table carries an epoch, a crypto.randomUUID() stamped on first use and replaced by reset(), and every pull reply carries it:

const epoch = body.epoch;
if (epoch && epoch !== current.epoch) {
  // The replay a reset forces is the most expensive pull there is, so
  // it's the one that most wants a fold.
  body = await fetchPull(0, !!snapshot);

apply is required to be idempotent, so a replay from zero is cheap and safe. There is a smaller bug hiding next to this one: when a reply omits the epoch the client must keep the one it knows, not write undefined, or the next reply that has one looks like a new generation and triggers a needless full replay. Rampset guarded that. The extraction regressed it, and the second commit of the day put it back.

Bug three: pulling is not always safe

Pushing never disturbs the device that pushes. Pulling applies someone else’s op to local state, and in Rampset a remote op landing mid-workout rewrote the working weight the finish logic reads back, and turned a logged 5×5 into a 3×5. The engine therefore has two gates and they are asymmetric:

async function run(): Promise<number> {
  const pushed = await transport.push();
  if (!(await canPull())) {
    // A clean push is as synced as we can honestly claim right now.
    record(pushed.ok ? { lastOkAt: now() } : { ...snapshot, lastError: UNREACHABLE });
    return 0;
  }
  const pulled = await transport.pull();

Returning false from canPull() pushes and tries again later. Eventual consistency makes the wait free.

Bug four: the gate has to be the only door

The journal is addressed by the server-side identity; the ops carry whatever lifter the client had selected. With canWrite() on the engine but createTransport exported, a caller could reach past the gate and push directly, which is exactly how Rampset posted one user’s workouts to another’s journal, got a valid ack, and drained the outbox against it. In an append-only log that is permanent. The fix is not a check, it is a missing export. The top of src/client/index.ts:

// createTransport is deliberately NOT exported. Reaching past createSync to
// the transport bypasses canWrite() — which is exactly how the app this came
// from posted one user's workouts to another's journal and drained the outbox
// against the ack. The gate is the only door.

sync.now({ force }) skips the ten-second throttle and never the gate.

Bug five arrived while writing tests for bug one

0.1.0 shipped with 33 tests, every one of them against MemoryOpStore. SqlOpStore, the class every production journal actually runs, had no tests, so no constraint or upsert in it had ever been executed by the suite. Writing them (against real SQLite through node:sqlite) surfaced a bug in the reply format. The journal refuses an op with a blank opId or kind and reported that in accepted, but the client drained the whole outbox against any well-formed reply and never looked. A count cannot carry the distinction: accepted: 1 of two ops means one stored and one refused, or one stored and one duplicate, and those need opposite handling. 0.2.0 changed push to return stored: string[], the ids the log now holds, and the client drains exactly those:

const stored = Array.isArray(reply.stored) ? new Set(reply.stored as unknown[]) : null;
const drain = stored ? rows.filter((r) => stored.has(r.opId)) : rows;
await outbox.remove(drain.map((r) => r.id));

A Worker older than the client sends no stored, and the client keeps the old behaviour rather than stall a rollout where the PWA leads the Worker. The same bug as trusting res.ok, one layer in.

Proving the SQL runs where it ships

The unit suite proves the logic and that the SQL parses. It cannot prove the SQL behaves the same on the DO’s own SQLite, which is the one place a divergence would only ever surface at runtime, in production. So there is an opt-in integration suite that boots workerd through @cloudflare/vitest-pool-workers and drives a real SyncJournal: push and the stored[] ack, pull, snapshot upsert, reset dropping the snapshot, the stale-epoch refusal.

npx vitest run
 Test Files  4 passed (4)
      Tests  71 passed (71)
   Duration  174ms

npx vitest run --config vitest.integration.config.ts
 Test Files  1 passed (1)
      Tests  16 passed (16)
   Duration  1.41s

It is npm run test:integration, not npm test, and its CI workflow is workflow_dispatch only, because it is an order of magnitude slower and never needs to block a publish. The integration worker imports src/, not dist/, so a regression is caught before a publish.

What I got wrong about the competition

0.1.2 has no code changes. The docs site carries a comparison against Zero, Electric, PowerSync and Cloudflare’s partysync, and when I fact-checked it against their primary docs three of the four rival rows were wrong, every error in my favour. partysync is delta sync with an IndexedDB read cache, so the gap is a write queue and nothing else; Zero rejects writes while disconnected, which is an explicit non-goal rather than a shortcoming; PowerSync’s “conflict resolution: yes” had been overstated. The commit is titled “Correct the rival claims — every error flattered us”. The table now says, in its own caveat, that it was written by someone with an obvious interest.

What you write

import { createSync, localStorageCursor } from "durable-sync/client";

export const sync = createSync({
  endpoint: "/api/sync",
  outbox: dexieOutbox(db.outbox),          // durable — it may hold the only copy
  cursor: localStorageCursor("myapp.cursor"),
  async apply(op) { /* idempotent: an op can arrive twice */ },
  canPull: async () => !(await somethingInProgress()),
});

The outbox must be IndexedDB, not memory: until a push is acknowledged it may be the only copy of the write in existence, and iOS kills backgrounded PWAs. start() syncs on visibilitychange, focus, online and pageshow (a tab restored from the bfcache does not fire visibilitychange), with a minIntervalMs of 10 s to collapse the burst a single foregrounding produces, and a 5-minute poll as a safety net that iOS will freeze anyway. There is a runnable example in the repo, a Worker plus a browser client with a real IndexedDB outbox, and a live demo whose Offline button flips window.fetch so you can watch ops queue with no sequence number and then drain.

When to use something else

A pull returns everything after the cursor in one response, fine for thousands of ops and not for millions. There is no live push; the other device converges on its next foreground. There is no auth: the DO key is your isolation boundary. One createSync per identity, memoised, because the throttle lives in the closure. And nothing runs while the app is closed, because Safari has no Background Sync and pretending otherwise would be a lie. If two people need to edit the same record and have it merge, you want a CRDT, Yjs or Automerge. If you already run Postgres, Zero, Electric and PowerSync are better engineered for the general case. The slice where durable-sync is the right answer is narrow: all-Cloudflare, writes that are events, and an app that has to keep working on a train.

What the extraction taught me

The op log is a hundred lines and the outbox is an interface; anyone could write those in an afternoon. Every non-obvious line in the package is a bug that shipped, and each of them failed silently with lastOkAt set and the status row green. The one that cost a workout was a redirect fetch followed on my behalf. The one that cost every client’s sync was a reset() I ran myself. The fifth was found only because the class that runs in production finally had a test, which is the sentence I would put at the top of the README if I were starting again. The code is small; the scar tissue is the product.

Source: github.com/isaacrowntree/sovereign-ibkr-fund (Apache-2.0 OR MIT) · Built on bezant

This post covers the executor’s rule that one small probe order must fill before it trusts the connection with a batch, and why that probe is removed from the queue by object reference rather than by position; the three caps in absolute dollars that sit outside the order-sizing maths; and the three bugs found by the pre-launch risk register, a written review of every way the executor could lose money: net asset value (the account’s total worth, “NAV” from here on) read in the wrong currency and counted twice, a confirmation timeout that could place the same trade twice, and a tax-lot matcher that paired sales with purchases by record and ignored how many shares each held. All three are fixed in v0.1.0; the register that found them ships in docs/execution-risk-register.md.

Nine agents and not one language-model call

Every agent is a plain process: node dist/agents/<name>.js --once. There are no calls to a large language model anywhere in the loop. A trade decision is a function of its inputs (positions, prices, the target portfolio, the risk state) and has to be reproducible from the state file, so it cannot come from a model that gives a different answer each time it is asked.

Underneath is a set of ways to turn price history into target weights: Hierarchical Risk Parity, Black-Litterman, risk parity, and Ledoit-Wolf shrinkage for estimating the covariance matrix, plus a market-regime overlay and volatility targeting, each with its own test file under src/portfolio/, src/quant/ and src/risk/.

Prove one fill before trusting the connection

The executor never sends a batch of orders on a session until one live order has been confirmed as filled. planExecution in src/execution/staging.ts puts it in validate mode: one order, the smallest-value sell (or the smallest order at all if there are no sells), and nothing else until IBKR’s execution records confirm the fill.

if (!validated && pending.length > 0) {
  const pool = sells.length > 0 ? sells : buys;
  const probe = [...pool].sort((a, b) => a.estimatedValue - b.estimatedValue)[0];
  return {
    mode: 'validate',
    orders: [probe],
    // Exclude the probe by identity, NOT by position: the smallest-value
    // probe is frequently not queue index 0 (the strategist emits sells
    // loss-first, not cheapest-first), so `slice(1)` would drop the real
    // index-0 order AND re-queue the probe that just executed — a
    // duplicate order on a live account.
    deferred: pending.filter(o => o !== probe),
  };
}

That comment is there because the first version used slice(1), which drops whatever is first in the queue. The figure shows why that is the wrong thing to drop.

A duplicate order on a live account is the whole class of failure this mode exists to prevent.

Caps that do not trust the maths

Order sizes are worked out as a percentage of NAV. If the NAV is garbage, the percentage is garbage, so there is a second gate in absolute dollars that does not know about the sizing at all. orderCapViolation in src/risk/data-sanity.ts checks every order against three caps set by environment variables, and the executor halts the run if any is breached:

In front of that, navSanityViolation and priceSanityViolations refuse to size anything when the numbers look impossible: a NAV of zero or less, a NAV below a floor, a NAV that has moved more than a configured percentage since the last cycle, or a price that has jumped more than a configured percentage. The strategist logs, alerts and gives up rather than generating orders against a NAV that reads zero or a price that reads a hundred times too low. On top of that, loss thresholds first reduce exposure and then stop trading altogether, and everything stateful lives in STATE_DIR, outside the checkout, so a deploy cannot overwrite the ledger.

The NAV was in the wrong currency

The risk register is a review of the executor against the live account before unpausing it, and its first finding was a units error. The account is held with Interactive Brokers Australia, with Australian dollars as its base currency. The API’s /summary endpoint reports totalcashvalue and netliquidation in the base currency, so the cash check was comparing US-dollar purchase costs against Australian-dollar cash, and the strategist was sizing US shares as a fraction of a NAV stated in Australian dollars.

I expected /summary to be the account’s cash. What the live /ledger endpoint showed was a separate balance for each currency, which IBKR does not convert: a US-dollar cash balance of $0, US-dollar settled cash of $0, and most of the NAV held in US stock. A cash account cannot borrow US dollars, so US-dollar purchases have to be funded by the proceeds of the US-dollar sales that run first, which the sells-before-buys ordering already guarantees. The fix is gateway.getUsdBalances(), which reads /ledger and returns US-dollar cash and US-dollar NAV, skipping the BASE row (IBKR’s total in the base currency rather than a real currency). That last clause matters: with BASE included, the NAV came out roughly doubled.

Re-running the strategist on the corrected numbers, verified against the live account (US-dollar cash $0.00, US-dollar NAV $29,155.64), showed the old quantities were oversized by the exchange rate:

The weights now sum to about 100%, and the concentrated holding’s true weight was 41.6%, not the 29% the mixed-currency maths had reported.

A timeout is not a cancel

Second finding. When the wait for a fill confirmation timed out, the executor dropped the order rather than requeue it, on the stated assumption that the strategist would rebuild the queue from live positions. The strategist reads positions only; it knows nothing about orders still open at the broker. If the timed-out order was still working at IBKR, the drift persisted, the strategist’s next run generated the same order again, and the executor submitted it a second time.

The fix was already one route away: bezant exposes DELETE /accounts/{id}/orders/{orderId}, and it just was not wired up. gateway.cancelOrder() now runs, on a best-effort basis, after a confirmation timeout, an error on the confirmation stream, or a partial fill that timed out, before the run halts. A failed cancel leaves the risk that was already there; it never throws.

Matching sales by record is not matching them by share

Third finding. The original tax-lot matcher paired each sale with exactly one earliest unmatched purchase record and ignored quantities. A sale that spanned two purchase lots priced the whole quantity off one lot, which could flip the sign of the realised profit or loss. That matters because only a sale at a loss opens a wash-sale window (the rule that disallows a tax loss if the same stock is bought back within a set number of days), so a loss that was wrongly reported as a gain let the strategist buy the stock back inside 31 days. And a sale smaller than its lot marked the whole lot consumed, so the next sale of the same symbol found no cost basis at all.

matchSellFifo in src/tax/fifo.ts now replays the history, reduces each purchase lot by the shares that earlier sales already consumed, and consumes the current sale oldest lot first, producing a weighted cost basis, a longTermQty (the shares held over a year, which qualify for the Australian capital-gains discount), and a matchedLots[] array on the trade record. Eight unit tests cover it, including the multi-lot sign flip.

IBKR’s execution records win

Fills reach the ledger by two paths: the executor records what the WebSocket stream confirmed, and src/execution/reconcile.ts backfills from IBKR’s execution history. The reconciler records each execution id once however many times it sees it, with a fallback key for records written before execution ids were captured, so a fill that arrives by both paths is recorded once. If the stream said an order did not fill and IBKR’s executions say it did, the executions win and the ledger is corrected with a logged RECONCILED line.

Run it under any scheduler

npm start runs the built-in scheduler and a status server, with each agent on the cadence above (SCHED_*_SEC to tune). Set ENABLE_SCHEDULER=false and any scheduler (cron, systemd timers, something else) can drive the same --once scripts; deploy/ has systemd unit and timer files. The core never imports the scheduler.

Your real portfolio stays out of git

The public repo ships src/portfolios/sample.ts, a template of eight exchange-traded funds, and runs against a paper account out of the box:

Weights must sum to 100 and validateTargets() enforces it. A real allocation goes in src/portfolios/local.ts, which is gitignored and takes precedence automatically. Caps, thresholds, cadences and the choice of optimiser are all environment variables; nothing about a real position is in source.

The backtest engine runs on a Yahoo Finance daily price dataset that is also gitignored: npm run fetch-data writes it, and the backtest suites skip until it exists, so a fresh clone is green on the first npm test.

Status and licensing

• v0.1.0: runs end-to-end against IBKR paper accounts; the set of agents will evolve.
• Dual-licensed under Apache-2.0 or MIT.
• Requires a running bezant gateway, on http://localhost:8080 by default.
• Not affiliated with Interactive Brokers. This is not financial advice. It places real trades: start on a paper account, and read every guardrail before setting TRADING_MODE=live.

The lesson from the register is that the dangerous bugs were not in the optimiser. They were units, identity and ordering: Australian dollars where US dollars were assumed, a queue position where an object reference was needed, a record where a share count was needed. Each was caught by reading the live account’s actual responses against what the code assumed, and each now has a test. If bezant made IBKR programmable, Sovereign is the part that makes a portfolio follow a written policy, with brakes that are independent of the policy. Contributions welcome, especially on the optimisers and the risk engine.

This post covers the engine (linear progression; the stall protocol that steps a lift from five sets of five down to three sets of five and then one set of five; weight reductions scaled to how long you were away; Madcow’s ramp fractions), the offline mechanics (a three-second network timeout in the service worker, a rest timer based on a saved start time rather than a countdown, a bell synthesised from two sine tones), and the production incident on 14 and 15 July in which a finished workout was erased, which produced four fixes that later became the durable-sync library.

Site and guide: isaacrowntree.github.io/rampset · Source: github.com/isaacrowntree/rampset (MIT)

Three ways to train, one engine

The constraint that shaped the architecture was not me. My wife trains too, and her program looks nothing like mine. So the Program row carries a mode, and finishWorkout branches on it:

export type ProgramMode = "progression" | "routine" | "madcow";

• Program mode is StrongLifts-style training: five sets of five, alternating two workouts (A and B), with the family of variants (Plus, Lite, Mini, Ultra, Ultra Max). The engine in src/lib/progression.ts is a pure function. A successful workout adds incrementKg to the lift: 2.5 kg on the main lifts, 5 kg on the deadlift, nothing on the assistance lifts. Three failures in a row subtract deloadPct, 10%, rounded to the nearest 2.5 kg. The stall protocol lives on the program slot as deloadCount: after the first deload the lift stays at five sets of five, after the second it drops to three sets of five, after the third to one set of five. Warm-up sets ramp up to the working weight like this:

  Warm-up set	Fraction of the working weight	Reps
  1	one half	5
  2	two thirds	5
  3	five sixths	3
  4	eleven twelfths	2

  Plate math works out the plates for one side of the bar, largest first, from the 20, 15, 10, 5, 2.5 and 1.25 kg plates, and returns null rather than lie when a weight cannot be loaded exactly.

• Madcow mode is five sets of five ramped up over the week. src/lib/madcow.ts encodes the three days directly:

  Day	What it does
  Heavy	four sets climbing through 50%, 62.5%, 75% and 87.5% of the top weight, then a top set of five
  Light	the same ramp, stopping at 75%
  Intensity	climbs to a personal-record set of three reps at the top weight plus one increment, then a back-off set of eight at 75%

  If the personal-record set succeeds, next week’s top weight moves up.

• Routine mode uses the layout of the Strong app: exercise cards with a row per set, everything prefilled from the last session by src/lib/prefill.ts, a rest time per exercise, timed sets and bodyweight sets. You prescribe; the app remembers.

Coming back after time off is handled by src/lib/deload.ts, and it is the whole file:

export function deloadPctForLayoff(daysAway: number): number {
  if (daysAway >= 56) return 0.3;
  if (daysAway >= 28) return 0.2;
  if (daysAway >= 14) return 0.1;
  return 0;
}

Both of our histories came in through CSV importers for the two incumbent apps’ export formats, detected from the header row ("Set Order" meant Strong; "Workout,Workout Name" or "Date (yyyy/mm/dd)" meant StrongLifts) and deduplicated by their order within each date and workout label, because two “Workout A”s on one day is common in real exports. Once the migration was done the importers, 633 lines including tests, were deleted; export gives everything back as CSV from Settings.

Offline is the default state, not an error state

Gyms are Faraday cages. IndexedDB, the browser’s on-device database (used through the Dexie library), is the source of truth on the device, and the network is an optimisation. The service worker in public/sw.js caches the app shell and five routes, and the one constant in it that matters is this:

/** Gym dead zones don't fail fetches — they hang them. Anything
 * network-first must give up quickly and fall back to cache. */
const NETWORK_TIMEOUT_MS = 3000;

The rest timer is the other place where “offline” really means “the phone locked in your pocket”. src/components/RestTimer.tsx does not count down; it saves the moment the rest started under liftlog.restTimer and, on every tick, works out how many seconds have passed since then (Math.floor((Date.now() - startTs) / 1000)), so sending the app to the background, navigating away, and a full reload cannot lose it. Closing the timer clears the tick but deliberately not the saved start time. A saved rest older than 30 minutes is treated as stale and discarded. When it rings, it rings with a bell the app synthesises, because a bundled audio file is one more thing to fail offline:

for (const [freq, at, dur] of [
  [1318.5, 0, 0.9], // E6
  [880, 0.12, 1.1], // A5
] as const) {
  const osc = ctx.createOscillator();
  osc.type = "sine";
  osc.frequency.value = freq;
  gain.gain.exponentialRampToValueAtTime(0.5, t0 + at + 0.015);
  gain.gain.exponentialRampToValueAtTime(0.0001, t0 + at + dur);

Two tones per strike, struck twice, half a second apart. The AudioContext is created inside the tap that logs the set, because browsers only allow sound to start from a user gesture, and resumed before striking, because long workouts suspend it. The screen is kept awake while a workout is open, and the phone gives a 10-millisecond vibration on every logged set.

Sync without a database server

Each lifter gets their own Durable Object, addressed by their lower-cased email, which comes from the cf-access-authenticated-user-email header that Cloudflare Access sets. The Durable Object handles one request at a time and has its own SQLite database, which makes it the place where that user’s writes are put in order, with no Postgres anywhere. Finished workouts are pushed as operations keyed by workout id, so applying one twice changes nothing; fresh devices pull. Underneath, src/server/backupStore.ts writes a dated snapshot to R2 under snapshots/<user>/<stamp>.json and merges a latest.json pointer with a write that only succeeds if the file has not changed since it was read (an etag check), retried three times, because two devices can finish a backup at once.

Identity is handled entirely by Access, so the app contains no authentication code, and the list of lifters is an environment variable (NEXT_PUBLIC_LIFTLOG_USERS in .env.example, a LIFTLOG_USERS Worker secret in production) that never enters the repo.

The week it lost a workout

On 14 July a real workout was erased from the R2 bucket. docs/sync-repair-plan.md is the write-up; the commit trail is the short version, and every step of it is a bug that looked like success.

The backup route overwrote latest.json instead of merging it, so a device with an older local copy wiped out a newer one (#17). The journal only holds workouts finished since it existed, so years of imported history lived in latest.json and the dated snapshots alone, and snapshots expire after 90 days. One overwrite plus three months is unrecoverable. That is why the merge is now a union, written with the etag check.

Then the outbox, the queue of writes waiting to be sent. Behind Cloudflare Access an expired session returns the login page with a success status (200) from the app’s own domain, and flushOutbox emptied the queue as soon as the response looked successful, deleting writes the journal never saw (#11). The fix checks the shape of the response body before emptying the queue, and sends an X-Requested-With header so Access answers with a 401 instead of a login page. The same trap is in the service worker, which now caches a response only if it succeeded, was not redirected, and came from the app’s own origin, because caching a login page poisons the offline shell.

The same commit fixed the reason pulls happen at all during a workout: applyOp writes workingWeightKg and finishWorkout reads it back to decide deloads, so a remote operation landing mid-session could step a lift from five sets of five to three sets of five. Sync is now push-only while a workout is active:

canPull: async () => !(await getActiveWorkout(user.id)),

Repairing the journal needed a reset (#12), and the reset shipped its own bug (#14). Each device remembers how far through the journal it has read as a cursor, the sequence number of the last entry it saw. Rebuilding the journal restarted its sequence numbers at 1 while every device still held a cursor from the old journal.

The journal now carries an epoch number, and a device that sees a new one replays from zero.

Last, identity (#20). The identity check was on cloudBackup and syncNow, but finishFlow called flushOutbox directly, so finishing a workout as the other lifter on a shared phone posted it to the signed-in lifter’s journal, got a valid acknowledgement, and emptied the outbox. In an append-only log that is permanent. The fix moved the check to the one door every write passes through, and #21 stopped an unreliable /api/me call (a two-second timeout that a cold mobile network blows routinely) from being read as “no identity, allow everything”: the app now remembers the last identity it actually resolved and falls back to that.

Those four failure modes are what got extracted as durable-sync:

• a successful-looking response is not evidence that the write landed;
• a cursor only means something against the epoch it came from;
• pulling remote changes is not always safe;
• the identity check must be the only door.

Rampset now uses the package; its own 329-line copy of the journal was deleted (#22), and the class name and SQL schema were kept identical so the live Durable Object carried over without a migration.

Tests

 Test Files  34 passed (34)
      Tests  256 passed (256)
   Duration  5.22s

The engines are pure functions in src/lib/ and each has a test file beside it: progression, deloads, Madcow, plates, estimated one-rep max (one case pins five reps at 115 kg to an estimate of 133.3 kg, a value taken from the real export), the sync journal, backup round-trips, and the Sheet component whose browser-history handling was the root cause of a finish-flow bug in #11 (router.replace overwrote the history entry the sheet had pushed, so consuming it on close walked the user back into a fresh workout). The count was 240 at the initial release and has moved with the incident fixes and the importer deletion; 256 is what runs today.

Run your own

git clone https://github.com/isaacrowntree/rampset
cd rampset
npm install
cp .env.example .env.local     # your lifters, units, starting weights
npm run dev                    # → http://localhost:3000
npm test                       # vitest — the engines, store, sync, components

npm run deploy builds with the OpenNext adapter (which packages a Next.js app for Cloudflare) and ships to Cloudflare Workers; wrangler.jsonc declares the SYNC_JOURNAL Durable Object (a new_sqlite_classes migration) and the BACKUPS R2 bucket. Put a Cloudflare Access application in front of the Worker and allow your lifters’ emails. Workers and Access are free for a household. The one paid gate is R2: the data sits well inside the 10 GB free tier, but Cloudflare wants a card on file to enable it. Without it, drop the BACKUPS binding; the app works fully offline on the local IndexedDB store, and the Export button in Settings hands you a complete CSV whenever you want.

What I would tell the subscription app

The migration cost me a Saturday and the incident cost me a week, and the week was the useful part. Every bug in it returned a 200. A loyal customer with an export file is no longer locked in by anything, and with a coding agent in the loop “I’ll build it myself” is a weekend, not a threat. The two proven programs, the plate math and the sync are all in a public repo under a licence that says make it yours. Take it somewhere no subscription can reach you.