12031
Sanjay
@sanjay #12031
@farcaster protocol dev
4086 Follower 210 Following
I was originally leaning towards account ordering. But happy with where we ended up. The biggest issue with blockchains is managing block state growth. Thanks to @cassie for inspiring the solution on how to handle this in snapchains.
Also, special thanks to @vrypan.eth for the App Ordering idea.
Also, special thanks to @vrypan.eth for the App Ordering idea.
If you enjoy coming up with novel distributed systems algorithms, we have just the challenge for you.
Please upgrade Hubble to 1.14.4 which implements storage changes defined in https://github.com/farcasterxyz/protocol/discussions/191
We'll min version on Monday at the latest so all hubs will be ready for Aug 28 when storage units were originally scheduled to expire.
If you are manually calculating storage, you'll need to update your logic. You may find the new helper functions implemented in the hub-nodejs package useful https://github.com/farcasterxyz/hub-monorepo/blob/main/packages/core/src/limits.ts
We'll min version on Monday at the latest so all hubs will be ready for Aug 28 when storage units were originally scheduled to expire.
If you are manually calculating storage, you'll need to update your logic. You may find the new helper functions implemented in the hub-nodejs package useful https://github.com/farcasterxyz/hub-monorepo/blob/main/packages/core/src/limits.ts
Since this day, we're at 50x the message count, 180x the peer count, ~40x the db size. Perf metrics are harder to compare since it was during an incident but currently p95 merge latency is ~30ms and p95 gossip delay is <1s (vs 2000ms and ~2.5hrs during the incident)
Hub message disruption today was caused by our hub losing gossip connectivity to all other hubs (unclear why exactly due to a logging bug). It was unable to regain connection due a bad interaction with a libp2p upgrade.
Released 1.14.2 with a fix.
Released 1.14.2 with a fix.
Hubble 1.14 is out. It includes a bunch of fixes around follows (consistency issues and large compaction events breaking event streams).
If you are using shuttle, make sure you’re on the latest version before upgrading hubs, there’s a breaking api change for events.
If you are using shuttle, make sure you’re on the latest version before upgrading hubs, there’s a breaking api change for events.
Message processing was broken today because some events exceeded grpc client default size. If you're using shuttle, please upgrade to 0.4.1 to get the fix. If you are constructing hub clients manually and listening to events, make sure to pass in the following param
This one was tricky to track down. There were a lot of dead ends. Special thanks to
@wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂
@cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need.
And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.
@wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂
@cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need.
And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.
Reminder that the replicator is deprecated by shuttle. We’re going to remove the replicator from the hub codebase by end of next week to avoid any confusion. If you’re still using the replicator, please migrate to the shuttle package.
Let me know if you have any questions around migration.
Let me know if you have any questions around migration.
We’re seeing message processing delays again. The team is working on scaling our systems to be able to handle it.
alpha version of the package is out https://github.com/farcasterxyz/hub-monorepo/tree/main/packages/hub-shuttle
@ted @gregan the RWA thesis is finally playing out. When does the goldfinch goat pool open? There's clear investor demand https://x.com/NeerajKA/status/1775881228656181312?s=20
Please reach out if you’re interested in using this, or have any feedback
We're planning to min version the hubs to 1.11.2 later today to improve network health. This version includes the fix to have all hubs use snapshots to catch up if they are too far behind. If you're on an older version, please run `./hubble.sh upgrade` to get up to date.
https://warpcast.com/sanjay/0x4a7d7839
https://warpcast.com/sanjay/0x4a7d7839
We've released 1.11.1. @wazzymandias.eth made it so that hubs will now automatically use snapshot sync to catch up if they are too many messages behind.
Note that this will reset the db, if you're running a replicator you can disable this to be safe, by setting `CATCHUP_SYNC_WITH_SNAPSHOT=false` in your .env file
Note that this will reset the db, if you're running a replicator you can disable this to be safe, by setting `CATCHUP_SYNC_WITH_SNAPSHOT=false` in your .env file
This is a very cool talk. Interestingly, it's almost exactly the same algorithm the hubs use right now. Prolly trees are more efficient since it collapses the number of levels required (we use timestamp prefix tries, so it always at least 10 levels), but apart from that it's exactly the same.
Our existing sync architecture for Hubs is running into scaling issues. Been thinking about a new design that can scale to billions of messages and millions of fids. If you enjoy complex distributed systems problems, I would appreciate feedback
https://warpcast.notion.site/Sync-V2-a9c0fd81d7b245a0b3fbd51e6007909f
https://warpcast.notion.site/Sync-V2-a9c0fd81d7b245a0b3fbd51e6007909f
Just min versioned to 1.10.5. This should improve gossip performance for the whole network. Please upgrade via “./hubble.sh upgrade” if you haven’t already.
Thank you @wazzymandias.eth for the fix.
Thank you @wazzymandias.eth for the fix.
Hubble 1.10 is out, please upgrade. Note that this version requires a minimum of 16GB RAM of ram to ensure Hubs will be capable of keeping up with traffic. Hubs with less than 16GB will fail the startup check.
1.9.8 released with more performance improvements. Please upgrade. Also recommend switching your instance to 4 vcpus and 16GB of ram.
I'd love to have the ability to vote for snapshot.org proposals through frames. If anyone from the snapshot team is interested, I would be happy to help build this.
It would greatly improve participation. Start with showing proposals in frame and then delegate votes to your fid eth address and vote with a FrameAction.
It would greatly improve participation. Start with showing proposals in frame and then delegate votes to your fid eth address and vote with a FrameAction.
Hubble 1.9.7 is out with performance improvements. Please upgrade. We're aware of hub sync delays. This release should help, and more fixes coming soon.
I'm firefighting hubs at the moment, so I didn't have time to build a POC. But here's an idea that could be powerful: After you validateMessage on your frame server, you can sign an EIP-712 message with (userId, casterId, userVerifiedAddress). This will enable your smart contracts to use trusted fc data.
I live for days like this. Stressful in the moment, but nothing like working with people at the top of their game solving hard problems.