Hivemind is Live!
Hello Steemians, we are excited to announce that Steemit’s APIs are now powered by Hivemind! By routing most of the social APIs to Hivemind, we are able to dramatically shrink the size of the full nodes we run. Because there are so many social applications sharing information on Steem, full nodes had become extremely large and expensive to run especially because they are not optimized for use cases that are not consensus-critical.
Hivemind is a service that syncs a traditional database with the blockchain. An added benefit of Hivemind is that since it’s written in Python, developers can modify its behavior and APIs with ease, instead of relying on C++ developers and modifying code which can interact with critical consensus logic. Implementing features like unreblogging becomes trivial with Hivemind while simultaneously enabling node operators to dramatically shrink the size (and cost) of their servers.
Testing
In our tests we were able to route all api.steem.com APIs to the new configuration—Hivemind for social APIs and everything else to “slimmed down” steemd nodes—which performed well. In addition, these nodes were running on-disk, as opposed to RAM; a major win which was only made possible thanks to AppBase.
We are extremely excited to see Hivemind running in production, on-disk, and utilizing AppBase. After completing this task, we have decreased our steemd instance sizes from 488GB RAM to 61GB RAM instances. We are doing further analysis and may be able to decrease this usage even further. These are major improvements to Steem that will make it easier and more cost efficient for anyone to run a full node, and all without requiring a hardfork.
Bug Hunting
We have so far identified one minor stability issue and will be rolling out an update shortly. If you are experiencing any issues using Steemit.com please leave a note in the comments.
The Steemit Team
Awesome. Great job delivering this guys! I was very glad to see this route taken as opposed to just 'sun-setting' the most popular and widely used app in all of crypto :)
This is fantastic work! I wondered whether you’d really be able to get RAM requirements down below 64Gb, but you’ve done it.
Next question: who wants to help me set up a full node. I have the hardware to meet the 2x64Gb + 1 x 32Gb requirements and have very fast internet 300/150 Mb/s.
Posted using Partiko iOS
@apshamiton
please contact me, @surfyogi on telegram messenger
A full node with only 61gb of ram is the dream ! Congrats team ! Can you guys share more specs about your instances ? How much disk, ssd or nvme, which cpu etc ?
We run two types of Steem nodes - one that handles account history, and one that handles everything else except for the tags and follows plugins (because Hivemind handles these plugins itself now). Both types of nodes run on instances in AWS that have 61GB RAM and a single physically attached nVME drive. The shared memory files are stored only on the physically attached drive, no longer in RAM as was previously required. Hivemind itself uses a postgres database in AWS RDS on instances with 32GB RAM. Hivemind's 'app servers' use much smaller instances with only 4GB RAM. All of this has been a phase in our plans to streamline infrastructure for cost effectiveness and there may be further improvements still yet to be made.
So, how many moving parts has Hivemid API instance?
right?
Yes, hivemind requires a postgres database. You need a steemd node or nodes that contain all plugins except for tags/follows. We use jussi for routing, which is a custom reverse proxy and caching layer.
Everything has official docker images available and could be set up to run together with
docker-compose.Well, I’m not a developer guy, I just wanna figure out how much you save on old rpc node vs new set of nodes delta.
With the cost of instances with lots of RAM being very high and fast disk being relatively cheap, quite a lot.
How much 1Gb RAM costs now?
It's not really a full node for only 61GB of ram, it's each node is only 61GB (there are multiple (3) that make up a full node).
Two types of Steem nodes, not three. See below reply to @howo :)
Servers/nodes same thing :)
Just wanted to be clear it wasn’t a single 61gb server making up a full node.
So it kinda like sharding?
Kind of but not really. In sharding you have the same tech but split into chunks.
With this you are splitting into chunks but using different tech (steemd, hivemind, rocksdb) in each section. L
What is rocksdb? I saw mention of postgres above...
Full nodes used to run with all data stored in ram. Rocksdb is a disk database that was added to recent versions of steem to allow account history to be stored on disk thus reducing memory requirements in half.
HiveMind uses a Postgres database.
5 comments in the first 10 minutes... This place is lively... 🕸️🍃
I ran into the gateway error when trying to load steemit for some minutes after the post went up. Could be why.
We've missed you. Things have been happening as you can see. Lots of drama, but some of us are still having fun.
Nice to see you @beanz! :D
Howdy Stranger.
Don't worry, it's been very lively around here in the past few weeks.
Posted using Partiko Android
😂
It only looks lively for people who haven't been around lately :-)
Shhh everyone! Beanz is back.
I hope you are doing well and all things are well with you. To quote @steevc:
That is a very positive news to say the least!
We left a note earlier today on a prior update by @steemitdev, whereby we believe one related functionality in api.steemit.com nodes is broken, as follows:
hey there,
We noticed at the time of rollout of changes (as of Jan 10) that the post's rshares data is now returning undefined
Is this an intended behaviour?
We are for example pulling post data using
steem.api.getDiscussionsByCreatedand then accessing post'spost.vote_rsharesThis is only no longer working on api.steemit.com and api.steemitdev.com
Would be great if you can provide some insights or look into this.
Thanks!
Yes I am also very pleased and want to see communities man, i just want subreddits , just something simple like chainbb, when it finally happens and we have an instant reliable faucet for creating new accounts then steem can finally just start growing organically on its own JUST from all the reddit users who will Migrate over just for the censorship resistance AND the payouts! In fact I think the censorship question will be enough to bring Millions of users over since that is what happened when Digg users first migrated to Reddit and then reddit users migrated to Voat and gab.ai etc
We are some of the first wave of reddit to steem immigrants and we are like the paratroopers who were dropped in over enemy lines into Normandy to capture strategic bridges etc and soon we will have a Normandy type invasion from the Bitcoin and crypto subreddits first into steem and then all sorts of other subreddits like r/trees r/TheDonald and all sorts of political subreddits, photography subreddits looking to get some money, meme crafters who post normall on reddit and 4chan for nopthing will find Dmania and take it over and also build their own versions to monetize memetics
its going to be a bright future for steem, we have riden out the bear market and its actually made us much stronger
my god I hope we don't get attack by the redit crowd....
they are nasty at times....
LOL
See the issue about it: https://github.com/steemit/hivemind/issues/161
My guess is that this is included inside the minor update, but upvoted for visibility.
Based on @roadscape's comment on the issue, I don't think this will be addressed.
Edit: Looks like I misread the problem.
Are you referring to rshares per vote, or per post? If the latter, the
net_rsharesfield is the canonical source.Yes the per post one.
So are you saying the value of vote_rshares has been dropped, while the net_rshares still has value? I am unable to test this out on api.steemit.com or api.steemitdev.com as I keep getting an RPC error atm
Try again, there was a small hiccup ;)
works now, and yes net_rshares is returning value, thank you.
This doesn't sound right. :D
This is a great update, though. Thanks for all the work on Hivemind, that sound like some major size decrease!
dereblogging?
de/unsteeming :P
I think that upvote ===> un-vote.
Resteem ==> un-resteem.
So it is un-resteeming.
Lol
I like unsteeming
Dis-ReSteeen
Discombobulate?
Antidisestablishmentarianism?
I could wrap my head around deresteeming. I think.
Is that what Secretary Clinton did? I gotta learn how to do that kind of high tech stuff.
That's probably why it was, rarely, a bit funky using Steemit.com in the last few days, but anyway: Great job!
488GB RAM to 61GB on-Disk is a huge improvement!
You still need multiple servers to make up a full node, but with this change, you are able to use 64GB instances. (Minimum of 3 servers like this to make up a full node).
To mimic our setup on a smaller scale, it would be 2 x 61GB servers and 1 x 32GB for Hivemind's postgres. Not 3 x 64GB.
So does this mean I could run the 2 x 61Gb on a 128Gb HEDT machine and the 1 x 32 Gb on another machine on the same local gigabit Ethernet? Because I have this physical hardware already and would like to run a full node. I also have very fast internet.
Posted using Partiko iOS
With a configuration like this I would run a single node that includes all of the plugins except for tags/follows in one on the machine with 128GB. The downside would be a longer replay/sync time, but it would likely be more performant than running two separate nodes on one machine (due to disk paging). Fast disk would be required, such as nVME SSD. The two machines talking over local gigabit ethernet would work just fine yes.
Thanks. Would it be better to move 64Gb to another computer and run 3 separate machines. How fast does the processor need to be on the 32Gb machine? Is recent Pentium 4560 dual core 4 thread enough for the third machine?
Posted using Partiko iOS
I hope you did not start to sweat because of the funky responses :D
Posted using Partiko Android
I'm hoping the 61GB RAM is just a typo, and should be "on disk" since they mention above that the full-nodes were on-disk.
Previously it was necessary to keep all state in RAM for acceptable performance. With many of the non-consensus social features now supported in hivemind instead of needing to be directly in Steem, it is now possible to keep all state for Steem on disk. That is why the RAM requirement has been dramatically reduced.
Ah I gotcha. So -- You'd still need about 64GB RAM to run a full-node? Huge improvement, no doubt -- but it's still a bit big....
Or am I getting mixed up with the requirements of Full Nodes and Witness Nodes? Witness nodes are smaller, yeah?
No, you can use 64GB servers, not a single 64GB server.
Yes, the requirements for a witness node are much lower.
If you haven't seen our post about MIRA, you should check that out - that is a feature in development that is aimed to reduce requirements for all nodes, even the smaller consensus nodes (such as witnesses and seeds).
Great work! Hope this will make it feasible for dApps, or collaborations of some projects, to run their own full nodes in the near future:). Certainly a top priority from my point of view. Well delivered!
Encountered the typical 504 Gateway Error for maybe 5-10 min prior to reading this post (so roughly 40 minutes after it was published) if that is of any help. No other issues experienced yet.
@emrebeyler did beat you to that.
https://steemit.com/hivemind/@emrebeyler/introducing-a-public-hivemind-node
:)
Good that means it is working yeah how can I sleep now🙈 I'm all fired up on this. 🤣😂
Posted using Partiko Android
_
How’s that “secure wallet” powerdown going?
Posted using Partiko iOS