Updates on steem-python, SteemData, and the node situation
steem-python
Users of the Python Steem library should upgrade to 0.18.93 as soon as possible. This version comes with new fixes, one of which prevents undefined behavior when all steemd nodes fail. It will now throw a SteemdNoResponse exception after exhausting automated failover and all of 200 retries (10 node failovers, 20 retries per node).
SteemData
I have been receiving a fair amount of complaints about SteemData state falling behind.
There are two ways to remedy this issue:
1.) Write a high performance http -> websocket proxy [1], as well as perform internal
state reconstruction to avoid querying steemd.
2.) Deploy a dedicated, self-healing steemd cluster.
Needless to say, both solutions are time and resource intensive, and unfortunately, I am not
abundant in either right now.
Big thanks to @gtg for letting me use his personal steemd node(s), which, when online, keep SteemData in sync.
Call for help
Full rpc [2] steemd nodes are a challenge to run. Currently, they require expensive servers with 128GB of RAM, and regular baby-sitting. To my knowledge, there are no open solutions for managing steemd clusters .
As rpc nodes became harder and more expensive to run, community powered nodes disappeared.
Aside from @gtg's node, I am not aware of any public nodes. As far as I know, two of the best community developers, @jesta and @good-karma run their own. Steemit's nodes are generally a bit slow, and occasionally unreliable [3].
If we'd like the Steem ecosystem to support third party apps and developers, we need better infrastructure.
We need to create a node cluster that is:
- worldwide distributed, with geo-location based load balancing
- scalable, openly monitored, and self-healing (trough automated remediation workflows)
- capable of running on all servers, and not locked down to a single vendor like AWS
There many talented people in our community, and I hope we can tackle this challenge.
[1] Steemit is already working on one, jussi
[2] Nodes with all plugins/api's enabled, serving at high throughput.
[3] Based on my experience with extended usage of their nodes for SteemData and Witness Tools.
https://rpc.steemliberator.com is a full RPC node ran and managed by yours truly.
So there is an issue after all with running a node? @elfspice was complaining about having problem with his.
It depends on what kind of node. Seed, wallet and witness configurations are generally working well.
The full rpc nodes however are expensive and somewhat troublesome to run.
@furion, your witness support would help me ensure I have the ability to keep my public full RPC node online indefinitely. I am going to pay out of pocket for whatever I don't raise on SteemIt, but something as simple as witness support can make a huge difference.
Btw, thanks for this kind of awareness post and thanks for SteemData.
So RPC node is just all these three combined. That doesn't sound too bad. But isn't running even one of the three getting increasingly harder as blockchain bloats? I hope Steemit Inc distributes funds towards this "issue of the tomorrow".
They're simply expensive; Full RPC nodes also work well, otherwise this site wouldn't be functioning.
Thank you very much for the update @furion. And great job on the decentralized video app for Steemit! Looking forward to using your service for uploading videos instead of having to rely on YouTube. Keep up the great work - as long as there are talented people like you in the community that are constantly improving it, Steemit will always be in great hands!
Thank you for supporting us at the early stage of our journey.
I would upvote this myself furion but i am trying to instead give out tokens for it.
Please email me your bts addy bro
Hey @fuzzyvest, would it be crazy to ask for your witness support so I can more easily pay for the full RPC node I have been running? Up to you kind sir.
You're very welcome @furion!
Hi @furion I've published a post about you, check it out if you can, thanks.
30 Best Steemit Bloggers Of The Day To Follow 3rd August 2017
https://steemit.com/steemit/@jzeek/30-best-steemit-bloggers-of-the-day-to-follow-3rd-august-2017
good jobs
So, we need a healthy and well organized community run network.
In my opinion, Steemit Inc. should have a top priority in establishing this network. With planning, support, providing resources and rewarding participants. Full package!
Ps: self-upvoted to push the comment up.
Which?
We already pay for and run a high-availability cluster of steemd nodes, at https://steemd.steemit.com.
What exactly are you proposing we do?
What I do miss are clear and precise instructions / documentation on how to set up and run a community node. Hardware resources needed, recommended, software needed, set up procedure, day to day operations needed. The n the why 's of running a node - benefits, not only financial.
Perhaps all this is already written and published somewhere?
All this to distribute the load from your cluster to the community.
Next, looking into the future - one word comes to mind - sharded.
If it really is a requirement to have 128 GB of RAM to run a node then that's way too much. Somehow the load should be lowered to come into a reasonable range. I am not telling you anything new, I guess.
OK, I realize that my suggestions are not very constructive. I can't propose an out of the box solution. I can only voice my concern. I believe that you are working on a long term solution to keep the platform functional and scalable.
Good luck!
I'm starting to understand the architecture more (I think), and am feeling more confident about the long-term future than I was, but here are some of my thoughts about what you could do:
a) Allow it to be run to provide only the last n days worth of vote, comment and transfer data. This is all I would need for running my services on it, and I expect many others could also manage with less historic data. A smaller index surely reduces the RAM requirements significantly.
b) Allow it to run in a 'ledger only' mode, where the index could be smaller (as it would exclude all other operations). This is almost all the exchanges would need I think.
I realise this will need some work on steemd, but over time, these increasingly demanding running requirements are centralising the network unnecessarily, making it more vulnerable, stifling community innovation and leading to doubts about scalability.
Thanks for your time.
They are working on several pieces of infrastructure to address these issues. I hope that the community can also help, and add some resilience to it.
Thanks for the update. I'll have a look at jussi.
For how long will 128GB be sufficient RAM for rpc steemd nodes?
I am currently watching vids on View.ly!!!!! Thanks for all your great work on it!
My concerns for Steemit continue to grow, so I hope much support comes your way so that you can address these issues.
resteemed.
Way to seek help it's awesome to see the call for the community to rally on this.
How openly dare we discuss this, for fear of being tarred with the FUD brush? I get the feeling like @elfspice that this may be a very serious situation, and it almost feels that if the community isn't strong enough to be able to be open about these issues, it is too weak to survive.
Your feelings are based on his FUD-spreading nonsense;
steemdruns fine. If it didn't, you wouldn't be seeing this comment.@gtg, @netuoso and @furion have been very helpful in allowing me to better understand the situation. Thanks.
It's not the community, the choice to show our metal is being taken away it seems by the politics at the top. I think the way Steemit.inc have behaved is very bad.
Couldn't agree more lol!
Yup, this :D
argh its already posted
Yes, I noticed that yesterday when trying to create a witness account. Kept getting errors until I restarted from scratch by installing steempy again and saw some updates.
So this applies to all the newbie witnesses too who are just setting up. thanks!
good job :)