Ponder
/ponder-sh202
An open-source backend framework for crypto apps
Goal: By the end of 2025, even the largest Ponder app backfills will be fast enough to run in CI. Seconds or minutes, not hours or days.
It should feel like a build or reload, not a "backfill".
It should feel like a build or reload, not a "backfill".
Ponder 0.9.3 adds a new event ordering strategy designed for low-latency apps that index 10+ chains.
The current ordering strategy is “omnichain” - events across all chains share a single queue, with blocks ordered by (block timestamp, chain ID, block number).
The current ordering strategy is “omnichain” - events across all chains share a single queue, with blocks ordered by (block timestamp, chain ID, block number).
Introducing Ponder 0.9
We built a SQL-based alternative to GraphQL, with live query support and a TanStack Query integration.
Apps & websites that use Ponder for indexing are about to feel much faster →
We built a SQL-based alternative to GraphQL, with live query support and a TanStack Query integration.
Apps & websites that use Ponder for indexing are about to feel much faster →
Fun debugging rabbit hole from this week.
Last week, a few Ponder users opened issues for an "invalid byte sequence for encoding "UTF8": 0x00" error. Most software ignores or removes null characters (\0, \x00, or \u0000), but the Postgres TEXT data type rejects them with that error.
We reproduced the issue with an ENS NameRegistered event log from 2022 (s/o @greg). Turns out, some EVM event logs contain null characters within decoded `string` parameter values.
Now, the framework removes null characters from all decoded ABI parameters before passing them to userland. We could have instead fixed this on the PG driver/Drizzle side, but it seemed more fragile (are we wrong?).
We're not certain why this only appeared recently - we haven't changed any relevant codepaths in recent releases. My best guess is that Viem used to remove them but stopped. Viem is a peer dep, so the version can change independently from Ponder's version.
PR: https://github.com/ponder-sh/ponder/pull/1414
Last week, a few Ponder users opened issues for an "invalid byte sequence for encoding "UTF8": 0x00" error. Most software ignores or removes null characters (\0, \x00, or \u0000), but the Postgres TEXT data type rejects them with that error.
We reproduced the issue with an ENS NameRegistered event log from 2022 (s/o @greg). Turns out, some EVM event logs contain null characters within decoded `string` parameter values.
Now, the framework removes null characters from all decoded ABI parameters before passing them to userland. We could have instead fixed this on the PG driver/Drizzle side, but it seemed more fragile (are we wrong?).
We're not certain why this only appeared recently - we haven't changed any relevant codepaths in recent releases. My best guess is that Viem used to remove them but stopped. Viem is a peer dep, so the version can change independently from Ponder's version.
PR: https://github.com/ponder-sh/ponder/pull/1414
Two of the top 20 built with ponder, possibly more
602
Jason Goldberg Ⓜ️ 💜
@betashop.eth·20:55 30/12/2024
these apps had the highest daily average # of transactions the past 7 days on Base
we're bringing the world onchain in 2025
we're bringing the world onchain in 2025
Introducing Ponder 0.8
Native transfer indexing, `ponder` NPM package, support for 16 new Viem actions, and a simpler pattern for Direct SQL →
Native transfer indexing, `ponder` NPM package, support for 16 new Viem actions, and a simpler pattern for Direct SQL →
Built with ponder - great execution of a data-intensive, realtime EVM app
Good first issue re: ABI encoding, factory indexing. Will unblock real use cases. Happy to help you get started / jam on it - get in touch :)
https://github.com/ponder-sh/ponder/issues/1323
https://github.com/ponder-sh/ponder/issues/1323
The ponder client library will be very, very good
Write queries in your frontend codebase, type-safe with no codegen, realtime/reactive by default
Coming soon to a browser near you
Write queries in your frontend codebase, type-safe with no codegen, realtime/reactive by default
Coming soon to a browser near you
We're hosting a Ponder holiday meetup in NYC next Thursday!
This is a developer-focused event for users, contributors and friends interested in crypto app development, TypeScript tooling, web frameworks, open-source, the holiday spirit, etc
Reply here or send me a DC if you'd like to join
This is a developer-focused event for users, contributors and friends interested in crypto app development, TypeScript tooling, web frameworks, open-source, the holiday spirit, etc
Reply here or send me a DC if you'd like to join
Enjoyed working with @0age on this!
Most actors using The Compact need an indexer. It’s a great use case for Ponder - multichain, non-trivial indexing logic, 100% open-source
Use our endpoint (link in readme) or host an instance yourself
Most actors using The Compact need an indexer. It’s a great use case for Ponder - multichain, non-trivial indexing logic, 100% open-source
Use our endpoint (link in readme) or host an instance yourself
4262
0age
@0age·17:03 02/12/2024
open-sourced an indexer for The Compact built alongside the gigabrains at
ponder.sh: https://github.com/Uniswap/the-compact-indexer
very clutch for anyone building a frontend, tracking analytics, or running a filler or an allocator (link to hosted version in the README)
h/t @typedarray.eth @kjs
ponder.sh: https://github.com/Uniswap/the-compact-indexer
very clutch for anyone building a frontend, tracking analytics, or running a filler or an allocator (link to hosted version in the README)
h/t @typedarray.eth @kjs
Just shipped 0.7.7 which adds a `totalCount` field to the GraphQL page type. This is actually pretty handy for some paginated UI patterns, and is also probably the fastest way to check how big the database is
https://github.com/ponder-sh/ponder/releases/tag/%40ponder%2Fcore%400.7.7
https://github.com/ponder-sh/ponder/releases/tag/%40ponder%2Fcore%400.7.7
Introducing Ponder 0.7
PGlite, Drizzle, raw SQL, portable table objects, and more. It’s our biggest release since 0.1 →
PGlite, Drizzle, raw SQL, portable table objects, and more. It’s our biggest release since 0.1 →
✅ shipping next week
651
Kevin
@typedarray.eth·16:01 13/08/2024
Should Ponder replace SQLite with PGlite (pglite.dev) as the dev database?
Pros:
1) Less magic/astonishment ("wait, what database am I using rn?")
2) Simpler schema definition API (currently stuck with the subset of stuff that both PG and SQLite support)
Pros:
1) Less magic/astonishment ("wait, what database am I using rn?")
2) Simpler schema definition API (currently stuck with the subset of stuff that both PG and SQLite support)
https://ponder.sh/docs/indexing/call-traces#register-an-indexing-function how does this work if you're trying to index eth xfrs to an EOA (or CREATE2 address that hasn't been deployed)?
@typedarray.eth in what order do the events / call handlers execute in ponder?
biggest critique of ponder right now is the logging immediately jumps to the table and I can't scroll up to see any logs that I might have emitted from the indexing function bodies.
As a serial console.log debugger this is terrible 😩
As a serial console.log debugger this is terrible 😩
Should Ponder replace SQLite with PGlite (pglite.dev) as the dev database?
Pros:
1) Less magic/astonishment ("wait, what database am I using rn?")
2) Simpler schema definition API (currently stuck with the subset of stuff that both PG and SQLite support)
Pros:
1) Less magic/astonishment ("wait, what database am I using rn?")
2) Simpler schema definition API (currently stuck with the subset of stuff that both PG and SQLite support)
This is my favorite weird edge case in our codebase. If you manage to crash a Ponder app with this error, I'll give you $100.
github.com/ponder-sh/ponder/blob/c19358158d3be13e0c8bab8450ba16ab3f71011d/packages/core/src/sync-store/postgres/store.ts#L222-L230
github.com/ponder-sh/ponder/blob/c19358158d3be13e0c8bab8450ba16ab3f71011d/packages/core/src/sync-store/postgres/store.ts#L222-L230
If you noticed "missing events" in your Ponder app recently, you may have been affected by this (rare) bug. Upgrade to 0.5.3 for the fix.
It's more common on low-throughput chains where bloom filters are mostly empty. S/o @kjs and Mateus - was a team effort!
https://x.com/devjoshstevens/status/1816567954521383339
It's more common on low-throughput chains where bloom filters are mostly empty. S/o @kjs and Mateus - was a team effort!
https://x.com/devjoshstevens/status/1816567954521383339
Introducing API functions
With today's 0.5.0 release, you can build custom API endpoints directly in your Ponder app.
Use cases
- Replace GraphQL with tRPC or REST
- Build SQL queries with Drizzle
- Merge onchain & offchain data
With today's 0.5.0 release, you can build custom API endpoints directly in your Ponder app.
Use cases
- Replace GraphQL with tRPC or REST
- Build SQL queries with Drizzle
- Merge onchain & offchain data
We're hiring backend, infrastructure, and web framework engineers at Ponder (ponder.sh).
We have hundreds of production users, growing 40% per month.
NYC required. Team is 3 people, all engineers. DM with interest or referrals.
We have hundreds of production users, growing 40% per month.
NYC required. Team is 3 people, all engineers. DM with interest or referrals.
Anyone writing tests for your ponder indexer?
How are you doing it?
How are you doing it?
In solidity events uint256 types are returned as BigInts.
Is it best to convert these to strings when storing?
Not good to convert to int type?
Is it best to convert these to strings when storing?
Not good to convert to int type?
Just shipped a *very exciting* experimental feature in the Ponder telegram
Ponder is now 10-50x faster during historical indexing
0.4.37 introduces batched database writes & an in-memory cache for frequently accessed records
Apps that use a remote database will be much faster
0.4.37 introduces batched database writes & an in-memory cache for frequently accessed records
Apps that use a remote database will be much faster
Ponder 0.4.35 improves support for Foundry/Anvil
With a few lines of code, you can integrate Ponder with any Foundry development workflow using `forge script`
Build your indexer alongside your smart contracts, all on your local machine
With a few lines of code, you can integrate Ponder with any Foundry development workflow using `forge script`
Build your indexer alongside your smart contracts, all on your local machine
New in Ponder: JSON columns
Use p.json() for
1) Solidity structs
2) Storing raw log/trace/receipt objects alongside application data
3) Any schema-less KV data
Include a custom TypeScript type to enjoy autocomplete + type checking across your app.
Use p.json() for
1) Solidity structs
2) Storing raw log/trace/receipt objects alongside application data
3) Any schema-less KV data
Include a custom TypeScript type to enjoy autocomplete + type checking across your app.
Shipped: Call trace indexing
This was a top requested feature from teams looking to migrate their subgraphs to Ponder.
Why use call traces?
This was a top requested feature from teams looking to migrate their subgraphs to Ponder.
Why use call traces?
Ponder 0.4.16 introduces block indexing
Block indexing is great for cron / scheduled tasks:
- Read the latest price from an oracle contract every minute to build a price chart
- Aggregate data every hour to update protocol-wide statistics
Block indexing is great for cron / scheduled tasks:
- Read the latest price from an oracle contract every minute to build a price chart
- Aggregate data every hour to update protocol-wide statistics
Ponder 0.4.15 improves crash recovery behavior
Before, if a fully indexed app crashed (often due to a short-lived RPC reliability issue) it would need to reindex from scratch
Now, Ponder apps periodically checkpoint their indexing progress. After a crash, you can restart the app and it will pick up where it left off
Before, if a fully indexed app crashed (often due to a short-lived RPC reliability issue) it would need to reindex from scratch
Now, Ponder apps periodically checkpoint their indexing progress. After a crash, you can restart the app and it will pick up where it left off
In the next version of Ponder, error logs for SQL constraint violations are more human-friendly
0.4.9 adds support for transaction receipts and speeds up historical indexing by ~50%, thanks to @kjs
https://x.com/kyscott18/status/1782826942908997653
https://x.com/kyscott18/status/1782826942908997653
With today's release, Ponder runs on Windows via PowerShell and command prompt (cmd.exe).
Huge thanks to Mateus (not on FC yet) who found the key bug!
We’re also running the entire test suite on Windows runners in CI.
PR: github.com/ponder-sh/ponder/pull/806
Huge thanks to Mateus (not on FC yet) who found the key bug!
We’re also running the entire test suite on Windows runners in CI.
PR: github.com/ponder-sh/ponder/pull/806
It's never been easier to build and self-host a custom EVM indexer
Docker, DigitalOcean, Railway, local - Ponder runs nearly anywhere, using web technologies you already know
Docker, DigitalOcean, Railway, local - Ponder runs nearly anywhere, using web technologies you already know
Ponder 0.4.0
- Better direct SQL experience
- Faster indexing for apps with >10 contracts
- New terminal UI
- Express -> Hono
- Fixed annoying "there is a newer version of the record" bug
Migration guide: ponder.sh/docs/migration-guide#040
- Better direct SQL experience
- Faster indexing for apps with >10 contracts
- New terminal UI
- Express -> Hono
- Fixed annoying "there is a newer version of the record" bug
Migration guide: ponder.sh/docs/migration-guide#040
>100 open-source Ponder apps on GitHub as of today
Underrated advantage of Ponder over closed-source providers or rolling your own indexer: You get dozens of high-quality example apps that you can fork, run locally, browse for best practices, etc
github.com/ponder-sh/ponder/network/dependents
Underrated advantage of Ponder over closed-source providers or rolling your own indexer: You get dozens of high-quality example apps that you can fork, run locally, browse for best practices, etc
github.com/ponder-sh/ponder/network/dependents
This remains unsolved. I think it's impossible - if anyone wants to learn more about traces and double check my work, would gladly chat
651
Kevin
@typedarray.eth·16:34 28/03/2024
Fun `trace_` API problem:
Write a function that accepts a single trace object and returns a uint256 representing its execution index within the transaction.
The index doesn't need to be absolutely accurate, but when used as a sort key, it must accurately order _any_ subset of traces within one transaction.
Write a function that accepts a single trace object and returns a uint256 representing its execution index within the transaction.
The index doesn't need to be absolutely accurate, but when used as a sort key, it must accurately order _any_ subset of traces within one transaction.
