Can any big brain devs help me figure out how to speed up this import?

Just saving blocks and txns into a db...

But at the current pace it's gonna take two weeks to finish

https://github.com/utxo-one/bitcoin-analytics-api/blob/master/src/import/import.service.ts

Reply to this note

Please Login to reply.

Discussion

πŸ‘€

πŸ₯š head 🧠 dead ☠️

❀️ loves me

πŸ«‚ hugging together all ways

β€οΈβ€πŸ©Ή we heal all brokenhearted

πŸ’” apart or

πŸ’–β€οΈπŸ˜˜πŸ˜πŸ₯°πŸ’•β™₯οΈπŸ’™πŸ’œπŸ’”πŸ’—πŸ–€πŸ’›πŸ’–πŸ’žπŸ’šβ£οΈπŸ’“πŸ˜»πŸ§‘πŸ’˜πŸ’πŸ‘©β€β€οΈβ€πŸ‘¨πŸ’πŸ’ŒπŸ’πŸ‘©β€β€οΈβ€πŸ’‹β€πŸ‘©πŸ’ŸπŸ‘©β€β€οΈβ€πŸ’‹β€πŸ‘¨πŸ’‘πŸ‘©β€β€οΈβ€πŸ‘©πŸ‘¨β€β€οΈβ€πŸ‘¨πŸ‘¨β€β€οΈβ€πŸ’‹β€πŸ‘¨πŸ€ŽπŸ€πŸ©·πŸ«€πŸ©ΆπŸ«ΆπŸ©΅β€οΈβ€πŸ”₯πŸ©πŸ«°πŸ’’β€οΈβ€πŸ©ΉπŸ˜­πŸ˜’πŸ˜”β˜ΉοΈπŸ˜žπŸ˜•πŸ™πŸ˜ͺ😿 πŸ’œ attack and stroke in my stomach

#artwork #humanity #LoveIS

Can you rework your β€œsave” so that you batch insert the txs?

You’re probably io bound so either:

- get faster storage

- write out to disk less frequently

🍻

Ty ty u r right

Are you inserting one transaction at a time in the database? Doing inserts in batches should help with performance.

Yep I am good suggestion I will try

Great.

- it looks synchronous. parallelizing the block processing should quickly speed it up

- looks like your orm is only doing single inserts. bulk inserts would be faster

*normal brain here but have used these techniques successfully. also I typically avoid orms in favor of sql for greater control over the db, but ymmv.

Ok great many things to improve ty good sir 🫑

excited to see what you’re cookin up! πŸ€™

I hope everyone is proud that I dumped PHP

It wasn't easy, it's still the best language

Cc nostr:npub137c5pd8gmhhe0njtsgwjgunc5xjr2vmzvglkgqs5sjeh972gqqxqjak37w

php > js/ts

πŸƒ

🐢🐾😭😭😭

Just pushed new commit it's now 15x faster 🫑

LFG nice! 🫑