eRPC Kept Evicting Its Fastest Upstream on Arbitrum
On Arbitrum, a 30-second head poll made eRPC read a healthy provider as stale. Three config fixes in 45 minutes, one per layer, and what the metrics show a month later.
On staging, one of our four Arbitrum upstreams kept getting thrown out of rotation. It was Alchemy, the fastest of the four, and the thing throwing it out was the selection policy of eRPC, the RPC load balancer that sits in front of our providers. By 2026-10-13 the exclusion counter for it stood at about 17,000 after six days.
eRPC compares each upstream’s last polled head with a network head built from all of them. On a chain that makes four blocks a second, a head polled every 30 seconds goes stale fast, and eRPC read that staleness as lag. The default policy evicts an upstream more than 16 blocks behind, which on Arbitrum is four seconds.
I fixed it that evening in three commits over 45 minutes. Each one exposed the next place the same number was read: the exclusion gate, then a second gate in seconds, then the ranking score and the stickiness rule. Over the last seven days, Alchemy has not been excluded once. Earlier I wrote about why we picked eRPC and Ponder for Frak’s on-chain reward flow; this is what it took to keep one of its defaults from routing around our best provider.
The Setup
The project is nexus-rpc, the RPC endpoint behind our wallet. For Arbitrum One and Arbitrum Sepolia it reads from four upstreams, Alchemy, dRPC, Dwellir and BlockPi, all on free tiers with rate-limit budgets in packages/erpc/src/rateLimits.ts. A fifth, Pimlico, only answers the ERC-4337 bundler methods. Every request goes through retry, hedging and a 5 s timeout, and a selection policy decides which upstreams are eligible and in what order.
The image is ghcr.io/erpc/erpc:0.3.0, and the config is TypeScript that Bun bundles into erpc.js. Nothing in it sets statePollerInterval, so each upstream’s head is polled at the default of 30 seconds. The metrics agree: 120 polls an hour per upstream.
How eRPC 0.3.0 Decides Who Is Behind
I read this at the 0.3.0 tag of erpc/erpc. The docs describe the policy; the lag arithmetic is only in the code.
The Network Head
Each time an upstream reports its latest block, Tracker.SetLatestBlockNumber in health/tracker.go stores the number and derives a network head with corroboratedNetworkHead:
// health/tracker.go (0.3.0), trimmed
// corroboratedNetworkHead derives the network head for one axis from every
// upstream that has reported on the network: the second-highest once two have
// reported, the only value while one has.
func (t *Tracker) corroboratedNetworkHead(...) (int64, common.Upstream) {
// ...
pick := common.PickServedTip(heads)
// ...
corroborated := pick.Sorted[min(1, pick.Inputs-1)]
// ...
}
Taking the second-highest is deliberate: one upstream answering for the wrong chain cannot drag everybody else behind. The inputs are each upstream’s last stored head, whatever its age.
The Lag
blocksBehind(ntwBn, upsHead) gives the lag: network head minus the upstream’s stored head, floored at zero. When the network head moves, the lag is recomputed for every upstream on the network and stored in TrackedMetrics.BlockHeadLag, which the policy reads. The same function sets the erpc_upstream_block_head_lag gauge, so the gauge shows what the policy sees.
The Default Policy
With no selectionPolicy configured, eRPC runs internal/policy/default_policy.js. Three lines matter here:
// internal/policy/default_policy.js (0.3.0), trimmed
// Block-head lag: drop if behind tip by ≥16 blocks or ≥30s.
.excludeIf(any(blockNumberLagAbove(16), blockSecondsLagAbove(30)))
// ...
.sortByScore(PREFER_FASTEST)
// ...
.stickyPrimary({ hysteresis: 0.30, minSwitchInterval: '30s' })
The comment says “≥16” and the predicate is strict: blockNumberLagAbove is u.metrics.blockHeadLag > blocks (internal/policy/stdlib/stdlib.js:1544).
blockSecondsLagAbove(30) reads blockHeadLagSeconds, which readUpstreamMetrics in internal/policy/eval.go computes as BlockHeadLag × GetNetworkBlockTime(). The block time is an exponential moving average (EMA, a smoothed running estimate) kept by updateBlockTimeSample: alpha 0.1, one sample per head advance, on-chain timestamp delta in whole seconds divided by the block gap.
Putting It Together
Here is a constructed example; the numbers are mine. Three providers sit at the same real head on a chain making four blocks a second, and their polls land eight seconds apart:
A is exactly as fresh as B and C. At 16 seconds its stored head is eight seconds older than B’s, the second-highest, and eRPC reads those eight seconds as 32 blocks of lag. Any upstream whose stored head is more than four seconds older than the second-highest one crosses the default gate.
That also answers what looked like a contradiction in my own commit message, which says Alchemy “runs 2-3s ahead of the others”. A lead lowers an upstream’s lag by the length of the lead and no more. If Alchemy’s poll fired earlier in the 30-second cycle than the others’ by more than its lead, its stored head was still the oldest when they reported, and it read as behind on every cycle. Being the fastest did not get Alchemy evicted. Its poll timing did, and being the fastest is why the eviction hurt.
Layer 1: The Block Gate
The first commit, 25498fd, replaced the implicit default with an explicit copy of it and changed one line:
// packages/erpc/src/index.ts (25498fd)
// 240 blocks (~60s on Arbitrum) = two poll intervals, only trips on a
// genuinely stuck upstream.
selectionPolicy: {
evalFunc: (upstreams) =>
upstreams
.removeCordoned()
// ... error rate, throttle and latency excludes, unchanged
.excludeIf(
any(blockNumberLagAbove(240), blockSecondsLagAbove(60))
)
.whenEmpty(() => upstreams)
// ... preferTag, sortByScore(PREFER_FASTEST), stickyPrimary, probeExcluded
},
Thirty seconds of polling at four blocks a second is 120 blocks. Two intervals is 240, about a minute of chain time, so an upstream’s head has to stand still for about a minute, a whole missed poll on top of the normal gap, before it trips. The price is that a stuck upstream now serves for up to a minute before the gate drops it, with retries and hedging as the only cover.
Arbitrum One went clean. Arbitrum Sepolia did not: 13 exclusions in the next 10 minutes, all from the seconds gate I had left in place.
Layer 2: The Seconds Gate
blockSecondsLagAbove(60) does not measure seconds. It multiplies the block lag by eRPC’s block-time estimate, and on Arbitrum the estimate was off. The commit 031c92f records it at 400 to 930 ms, against a real block time of about a quarter of a second. eRPC exports the estimate as the gauge erpc_network_dynamic_block_time_milliseconds, and it has not settled since: over the last seven days it ranged from 226 to 866 ms, with a median between 309 and 352 ms depending on the network.
At 400 to 930 ms a block, a 60-second gate trips somewhere between 150 and 65 blocks of lag, well under the 240 the first gate had just been raised to. Once the estimate passed 500 ms, the trip point fell below the 120 blocks a stored head ages in one normal poll window, so the seconds gate became the one that fired.
I could not find why the estimate runs high by reading updateBlockTimeSample, whose own comment cites Arbitrum, “1s / 4 blocks = 250ms”, as the case it was built for. I did not need to. A block count does not depend on the estimate, and 240 blocks is about 60 s on both networks, so I dropped the seconds gate:
// packages/erpc/src/index.ts (031c92f)
// No seconds-based gate: erpc 0.3.0's block-time estimate reads 400-930ms
// on Arbitrum (real: ~260ms), so `blockSecondsLagAbove` trips on plain poll
// jitter too.
.excludeIf(blockNumberLagAbove(240))
Layer 3: The Score and the Sticky Scope
With nothing excluded, I expected Alchemy to take over as primary. It sat last on both networks, with a p70 of 37 ms, according to the third commit, 4434acc.
The ranking reads the same lag. sortByScore(PREFER_FASTEST) turns a weighted penalty into a score, with the preset’s weights in stdlib.js:
// internal/policy/stdlib/stdlib.js (0.3.0), lines 641 and 741
PREFER_FASTEST: { errorRate: 4, respLatency: 15, throttledRate: 4,
blockHeadLag: 1, finalizationLag: 0, misbehaviors: 2 },
// ...
u.score = overallVal / (1 + penalty);
respLatency multiplies the p70 in seconds by 15, and blockHeadLag multiplies the lag in blocks by 1. One block of lag costs as much as 67 ms of p70. Alchemy at 37 ms with no other penalty scores 0.64. Give it 100 blocks of poll age and the penalty becomes 100.555, for a score of 0.0098 (1 / 101.555), which is the “~0.01” in the commit. The gate had been loosened, and the ranking was still charging poll age at a rate where 100 blocks weighed as much as 6.7 seconds of latency.
stickyPrimary made it stick. It keeps the current primary until a challenger scores 1.3 times the incumbent (hysteresis: 0.3) and 30 seconds have passed since the last switch. Its scope defaults to 'network' (stdlib.js line 816), and for that scope useWildcard is true (line 844). The wildcard path ignores the slot’s score and recomputes one from network-wide metrics with hardcoded weights:
// internal/policy/stdlib/stdlib.js (0.3.0), lines 855-856
const penalty = computeWeightedPenalty(m, PRESETS.PREFER_FASTEST, 'p70ResponseSeconds');
return 1 / (1 + penalty);
Whatever weights I gave sortByScore, the sticky check judged challengers with PREFER_FASTEST, lag included, and a challenger scoring 0.01 never beats anyone by 30 percent.
internal/policy/sticky.go defines four scopes. network and network-finality score from the wildcard aggregate. network-method and network-method-finality read the slot’s own score. I picked network-method, so the sticky check reads whatever sortByScore produced:
// packages/erpc/src/index.ts (4434acc)
// Rank on PREFER_FASTEST's weights minus head lag (staleness is handled
// by the gate above), and keep stickyPrimary on these slot scores.
.sortByScore({
errorRate: 4,
respLatency: 15,
throttledRate: 4,
blockHeadLag: 0,
finalizationLag: 0,
misbehaviors: 2,
})
.stickyPrimary({
scope: "network-method",
hysteresis: 0.3,
minSwitchInterval: "30s",
})
blockHeadLag: 0 is the only weight that changed. Without the scope change it would have had no effect on who stays primary.
The scope’s name suggests one primary per method, and that is not what happens here. The policy still evaluates at eRPC’s default grain, one slot per network, because the config leaves evalScope unset, so every method shares the primary that slot picks. Within that slot, the scope only decides which score the sticky check reads.
| Commit | Time (2026-10-13, +0200) | Change | What was left |
|---|---|---|---|
25498fd | 22:34 | Own policy, lag gate at 240 blocks / 60 s | Sepolia still flapping on the seconds gate |
031c92f | 22:49 | Remove the seconds gate | Alchemy still ranked last on staging |
4434acc | 23:19 | Rank without blockHeadLag, network-method sticky scope | End of the session |
A Month Later
Staging and production both run eRPC 0.3.0 with these overrides, which reached main on 2026-10-15 (0797875). Both pods have been up since the night of 31 October. Prometheus keeps seven days, so every number here covers 13 to 20 November. The October data, including the run that produced the 17,000, is gone.
First, what that counter counts. erpc_selection_exclusion_total goes up once per evaluation tick for every upstream the tick excludes (internal/policy/slot.go), and the tick is 15 seconds unless the config sets evalInterval (common/defaults.go), which ours does not. So the counter measures time. 17,000 ticks is about 70 hours of exclusion inside a 144-hour window.
Over the last seven days, across both Arbitrum networks in both environments, the counter went up 34 times. None of them is Alchemy. dRPC on Sepolia accounts for 24 and Dwellir for 10, all for block lag above 240. The lag gauge shows those were real stalls, with peaks between 332 and 666 blocks, the kind of event the gate is for.
Alchemy’s lag gauge read 0 at every sample in the window. Every one, on both networks, in both environments. In production the others stayed at 3 blocks or less at the 99th percentile, apart from dRPC on Sepolia, so this week the polls land close together and Alchemy’s lead covers the age of its stored head. I did not expect that. It means the default 16-block gate would not have tripped on Alchemy either, and this week’s numbers cannot show that the overrides still do the work they did in October. My guess is that the poll phases changed with the restart on 31 October; I have not read how the poller schedules them.
On routing, the score now does what I wanted. On Arbitrum One production, 720 of the 753 first-attempt eth_call requests of the week went to Alchemy and 33 to dRPC, and the network-wide primary switched 8 times in seven days.
Latency needs a caveat about the histogram behind it. erpc_network_request_duration_seconds has bucket edges at 50 ms, 500 ms, 5 s and 30 s, and a p95 from histogram_quantile is a straight-line interpolation inside one of them. The October commit compared the slower fallbacks with Alchemy as “p95 ~410ms vs ~48ms”, and that says less than it seems to. On the fallbacks, more than 5% of requests took over 50 ms. With Alchemy, at least 95% finished under 50 ms. This week, 95.0% of the 936 requests on Arbitrum One production and 97.8% of the 399 on Sepolia staging finished under 50 ms. Those are small samples. Most of what our upstreams answer is eRPC’s own polling, about 40,000 head requests per upstream per week, since every 30-second poll asks for both the latest and the finalized block, against a few hundred user calls.
What Upstream Did Next
My commit message said the problem was not fixed upstream as of 0.4.0. The tags agree: 0.4.0 is dated 2026-10-11 and its default_policy.js still has blockNumberLagAbove(16), blockSecondsLagAbove(30). Between 0.3.0 and 0.4.0 the tracker changed by seven lines, from PR #1208 (shared head updates and block-time sampling).
Version 0.5.0, tagged 2026-10-27, includes PR #1296, merged on 2026-10-21, eight days after my commits: “credit a head’s age since its last poll before counting lag”. Its comment on headReport in health/tracker.go describes the same failure:
// health/tracker.go (0.5.0), trimmed
// A stored head ages between polls by design; counting that age as lag made
// a healthy upstream polled every 10s on a 0.4s chain read 25 blocks behind
// late in each poll window ...
// So for one poll period after a report, what the network head made since
// counts as made by the upstream too: its lag stays what it was at the report.
Past one poll period the credit stops and a missing report counts as lag again, so a stuck upstream still gets caught. I never filed an issue for this, so I don’t know whether #1296 came from the same observation.
We still run 0.3.0. The upgrade would let me delete the gate override, though not the rest: 0.5.0’s default policy keeps blockSecondsLagAbove(30) on top of a block-time estimate that this week ranged from 226 to 866 ms, and PREFER_FASTEST still charges one point per block of lag. Both would need re-checking against this config before I remove anything.
What I Would Carry to Another Project
The gate, the score and the sticky tiebreaker each read BlockHeadLag through a different code path. Fixing one left the other two on the same bad number. Each needed its own commit. When I tune a threshold on a metric now, I first list every place that reads the metric.
The second lesson came from writing this up. My commit messages quoted a p95 and an exclusion count, and both turned out to mean something narrower than they seemed: a p95 interpolated inside a 50-to-500 ms bucket, and a count of 15-second ticks. Next time I quote a metric in a commit message, I will say what it counts.