AWS usage projections, DevOps solutions, reduction of Staging and Testing nodes, and eliminating redundancies.
Have you guys thought much about leveraging a more decentralized back-end? I recently launched a full node, and have been talking a lot about the decentralization of APIs with people. Notably, my node is quite underutilized -- I'd be happy to help by drawing some traffic.
On a bigger scale, I feel like if we convince witnesses to offer more API support, we could extend this to a fully decentralized scale. I'm sure many other witnesses would be on board providing Devops and infrastructure. We'd need to improve things like jussi, and get some better lines of communication with whomever is currently doing devops for Steemit, but I know in the long run we can do a lot better than AWS.
Changing the culture towards this seems logical to me: decentralized the source of the data, and decentralize the costs.
I'm in favour of Witnesses taking more responsibilities regarding APIs.
However, just a side note: Steemit Inc.'s API & infrastructure costs are in the millions of dollars per year. A "normal" API node costs max 1000€ per month, so roughly 12,000€ per year. If we really take this approach seriously, this will require a lot of funds, which might be difficult to come by in this bear market.
12,000€ (CAN$18,000) per year is more than @anyx paid to buy a 512 Gb full node server with an Xeon Gold CPU!!
Why would you pay more per year than the total capital cost of the item?!
Not to mention my server was absolute overkill.
However, it does have a monthly cost for collocation hosting, but this is a fixed cost of only a couple hundred/mo, but that also buys things like redundant power, cooling, etc. But indeed, in the long run, purchased assets are the way to go.
The last 5 items are only necessary because of huge centralized data centers. They are not needed when running less than 10 machines as part of a decentralized system. Redundant internet is easy and cheap to add and redundant power supply is only necessary in places with regular power outages. Once in ten years is typical for central Tel Aviv where I live. 24 hour monitoring is automatic if server is in your home and you also work from there.
Posted using Partiko iOS
Well, I have around 120 desktop (in my office), software routers instead of hardware routers etc. All of them custom made - so I tend to agree to what you say. But with these expensive hardware and network speeds (read QoS, BGP poisoning avoidance, DDoS protection) etc, its not quite possible to handle them the servers in our home. For example @anyx I am sure can handle most of the hard ware/CPU/memory related aspects. I can't. But I can handle lot of the network aspects - which others may not be able to. The witnesses are a mix of sysadmins, researchers, programmers and marketers - so we can't expect them to run the 17 + 2 + 2 servers in their home office. Even if everything including BGP is taken care of, I don't think DDoS is something we can monitor and retaliate that easily (unless we have a OpenBSD box and we are very handy with firewall rules).
Being said all this, I don't think we need to really host all the witness nodes in our backyard to save costs. Proper AWS cost optimizations can save a lot of costs. Personally I don't see any reason why the full nodes are on AWS where every disk access is billed. Once the development is done, those instances can be moved to data centers. AWS and cloud as we know is for elastic needs (CAPEX) and in the case of steem full nodes that kind of "elastic scaling" is not possible. (due the architecture.) May be sharding like NEAR protocol ( https://nearprotocol.com) is the way to reach there - but its not in the near term.
To conclude, its very much possible to run the infrastructure in traditional data centers at much lower costs than AWS. Steemit.com's workload is not something that needs cloud computing.
If Steem Monsters was able to run a successful crowdfunding campaign, both on Kickstarter and Fundition, why wouldn't that be possible for Steemit? They can easily give out delegations as rewards, which wouldn't have too big of an impact of the Steem price.
Also, won't Steem receive around 20 million USD in funding from Global Blockchain Technologies? Source
Look, I obviously haven't spent that much time thinking about this, it just seems that budget problems are somewhat easy to solve in this day and age.
Yeah.... A small part of me wants to believe that firing 70% of the team was done because they realized they hired the wrong people and the low market cap was just a good excuse.
This is insane and I guarantee you it's due to massive inefficiency. No website with this level of interaction costs that much. Yes -- we need to improve second layer (and primary layer) solutions to take the load off actual steem machines, but it's really not that hard.
The problem is the high cost of a single node, and the fact you need lots for scalability and redundancy, then you need a few that do nothing but create state files for the active ones and you need backups and staging ones. It adds up quickly.
It would be far Simpler if it was just a typical database.
Not even. The problem is poor first and second layer solutions on top of a full node.
Yeah, full nodes also suck. A lot. But the whole stack is worse.
No Arguments here. I hope this crypto drought encourages some good innovation that handles this. Sometimes loss is required to kickstart innovation.
Agreed - though maybe it depends on where the line is drawn around what is and isn't classed as infrastructure.. Or maybe someone is misleading ned.
Few mis configured EBS volumes with crazy IOPS values, Cloudfront, heavy usage of Docker based builds etc can be the reason for the high costs. AWS is not the right solution for something like steemit.com which has no per-visitor or per-user revenue model or challenges with capacity.
i'm in favor too. and eventually maybe more than APIs.
What do you have in mind?
nothing specific lol i'm just a gamedeveloper.
as for the cost of running a node i have no idea i'm sorry. but maybe that cost should've been steemit inc's only priority instead of trying to do everything on their own.
I very much agree with you @anyx that there needs to be more decentralisation of the back end and I congratulate you for launching a full node.
I also think that witnesses need to be owning their own hardware for witness nodes rather than renting it on the centralised "cloud". While @anyx's Xeon server with 512 Gb cost you CAN$16,000, its is possible to create a 128Gb witness & API server using HEDT motherboard and sub $500 CPU for around US$2000.
Owning your own witness server is somewhat problematic.
Other than bandwidth issues, (unless you have fiber based internet).
A power outage, internet issues (which I'm sure are common not only in Israel) and even a piece of hardware that stops working can make you miss far too many blocks, and cause small malfunctions all across the blockchain, not so small if you're in the top 20.
Maybe it would be best if different witnesses would host their server on different server providers, but it seems there are many providers to begin with.
Those are my 2 cents on the issue.
Thanks. I've addressed these issues in more detail here. https://steempeak.com/@apshamilton/properly-decentralising-steem-and-cutting-costs-by-witnesses-running-their-own-servers-and-apis
All these things can happen at a centralised hosted solution which is fundamentally more vulnerable because of its centralisation.
The whole point of decentralisation and having 20 witnesses and 100+ backups is that there is a quick failover to the next witness.
Also, for each witness having HEDT servers each with a once off capital cost of $2000 rather than an annual cost of $2000 forever means you can have more backup nodes and more redundancy.
It simply does not make sense to be renting when you can buy for around 1 year's rent.
A server provider usually operates in several countries, so unless someone is cutting an underwater internet cable, a witness can reactivate their server within 24 hours (much less if he/she is more experienced).
While a server in their home/office can easily be shut down for 3+ days, if their ISP is shitty in their area.
If there was an infrustrcture to allow witnesses to share their servers with each other (if that's even possible), to could allow more decentralisation without the drawbacks I meantioned. But I don't know how feasible such a project is.
Posted using Partiko Android
I’m not suggesting no redundancy. A witness should obviously have their backup server node in another location with another internet provider. Also it’s easy to have have a backup internet provider at your home or office. Everything an outsourced provider can do can be done for much less cost yourself when you are talking about single digit number of servers, which is what decentralized solutions involve.
Posted using Partiko iOS
That's unless we have someone as huge as YouTube's Linus Tech Tips on @dtube who runs servers themselves...
ned would have to give up most of his steem for any real whale to come in and start dealing with the costs ;)
While you're right, you can have redundancies like a UPS(duh), a generator for back-up power and do the same thing for internet connections.
Witnesses typically scale up over time. If you're a backup, you can front a fixed cost of less than $1000 and run it on your home internet. This is reasonably stable enough nowadays. When you get higher in the witness ranks you can deploy more nodes and improve redundancy and performance.
Missed blocks from one entity on the network won't impact things significantly anyways, we have built in redundancy at the system layer (the blockchain itself) so it really isn't a practical concern.
HEDT are a great way to go for limited API support (you can't fit all the api's, but you can serve a large amount of requests locally and route the ones you can't serve appropriately).
Notably, you can also build a consensus machine (e.g. 'witness node') for less than 1000$. My purchase price for a full api was overkill -- we can bring the costs down by optimizing for tasks rather than "do it all super fast on the same giant machine".
It's great if you can use different API endpoints; basically this is what a decentralized infrastructure should offer, anyway. However, when it comes to running API endpoints and other services reliably and at scale - so that other (professional) services can depend on them - it's not a simple operational task and it certainly isn't cheap.
I suppose it's important that we start stressing actual business models on top of Steem. If anybody is running an actual business that doesn't take its revenues from the reward pool, that person or organization would have an intrinsic motivation to have its own infrastructure under control, therefore could run it and potentially offer to outsiders, as well. Is anybody out there?
The next question then is: would anybody be willing to pay for API services to access and interact with the Steem blockchain? Would that make a business model of its own?
Hi @grintsch! We're building @dstors and I can tell you that our revenue model is not based on taking out of the reward pool. On the contrary, the objective is to distribute the existing reward pool to those on the blockchain who perform any and all economic activities related to their sales through their own @dstors. You can peruse the executive summary of the whitepaper that was released a few days ago on our blog. Yes, we're out there! Focused exclusively on giving back to the users as much as possible while introducing new models of interaction with the blockchain beyond what steemit.com initially offered through blogging.
Right, thanks for pointing that out. Great to see that, and I'll definitely check out your site and material.
Would witnesses really want to subsidize operations of Steemit.com? Steemit.com remains the central pillar for Steem, drawing in the vast majority of actual users and usage, so there's some argument to be made for it. However, Steemit.com itself has proven to be unsustainable. I remain skeptical that a large scale social network will ever be sustainable on a blockchain without centralized layers. This entire post and all of Ned's comments below are firm evidence.
PS: Before someone says it - no, speculation is not a sustainable financial model.
just because steemit inc/steemit.com is unsustainable it doesn't mean steem is. ned is the biggest whale here but one man's opinions has got nothing to do with a social network being sustainable or not.
if a central body is absolutely necessary for social, we should ditch steem and stick to facebook. but i thought people (especially crypto lovers and steemians) were so sick of it?
this is more like an opportunity for witnesses to actually step up and start making changes for more decentralization. it could be one of many solutions to come that shows steem's potential is not restricted to steemit.com or steemit inc or ned.
Never said anything about "central body being absolutely necessary for social". Furthermore, it's counterproductive to cite extreme and irrelevant examples such as Facebook.
The world doesn't work in extreme blacks and whites. The reality is more nuanced. To make Steem sustainable, we need centralized layers like Hivemind. This will retain the decentralized nature of the Steem blockchain, whilst streamlining computing resources required and thus greatly reducing costs. It's not just Steem, top blockchains like Bitcoin and Ethereum are also working on centralized layers of some kind for scalability and sustainability. It is clear this is the future of public blockchains, and does not mean "central body is absolutely necessary".
PS: I'm not interested in arguing semantics of "centralized layer", so don't bother.
if that's not what you meant then i stand corrected. facebook is not extreme and irrelevant because if this community remains dependent on one individual who owns a private company, it's just as extreme.
to say that we need centralized layers like hivemind is probably more black and white than anything i said. it could be one of many possible solutions. i'm guessing when you say the reality is nuanced we shouldn't look at hivemind as the only choice for sustainability. but sustanable for who?
ps. i think i understand what you mean by a 'centralized layer'. the only thing i wanna point out is that it doesn't have to come from steemit inc. and one company being unsustainable doesn't prove its necessity either.
Sure, there can be other solutions, but Hivemind is our best option that can be implemented in the near term for significantly improving sustainability. For whom? For everyone that runs an app or service on Steem, and requires Steem API nodes. Thus, eventually, for the end user. We want Steem to survive and thrive, and for that, improved scalability is essential. I understand you don't like centralized bodies, but almost all of Steem exists today, as it is, because of Steemit Inc. It has been two and a half years, and precious little has been contributed to Steem outside of Steemit Inc. Sure, it doesn't have to be that way, but that's the base reality. Till the time that other organisations and individuals start contributing significantly to Steem, Steemit Inc's survival is important to Steem.
yes steemit inc was the creator. but it doesn't mean they should hang on to control or should have any more weight in deciding where to go from here. it's a sad reality that after 2 and a half years this supposed 'decentralized' and 'censorship-resistant' platform still requires a central body/layer/whatever to stay running. if ned never used those words then everything i said would be irrelevant.
i have to disagree with you because steemit inc's survival is no longer important. even if steemit.com and api.steemit.com suddenly disappeared we'd still have other full nodes run by witnesses (or anybody who thinks the cost is worth it?). a lot of people will be confused and there'll be chaos. and i guess prices would drop too. (believe me i wouldn't like it. dan gave me a 100% downvote for saying most people would be locked out if steemit.com disappeared)
but the blockchain will stay and devs would simply switch servers. if steem breaks down due to one company/one man, then it doesn't deserve to stay. but i almost know that wouldn't be the case. or hope.. :)
I don't see it as being specifically for Steemit Inc., I see it as a path towards providing a decentralized data backbone for all of steem, which any app can use. Steemit is just one app on the steem platform.
Sure, but realistically, it's not "just one app". It's the vast majority of Steem - that was my concern. Ned says it costs $2 million per year to sustain Steemit Inc's infrastructure. Whilst some of it is the frontend, it's implied much of it is running Steem API nodes - their primary focus now is to reduce costs on them. I don't have numbers, but judging by activity, witnesses will have to take up an order of magnitude or more greater load. At the current prices, all top witnesses combined well have witness revenues of under $750k. How will we support a $2 million infrastructure? Yes, witnesses can go for cheaper infrastructure than AWS, streamline things considerably etc., but the sustainability seems very questionable. Not to mention, Steem is tiny now. We'll need to scale 100x to be relevant to the mainstream, and 1000x to compete with Reddit.
It's not adding up.
Yes, the price may increase, and it likely will, but history has showed us that price does not scale with activity - it is its own speculative bubble, and not a sustainable revenue model.
To be clear - I'm all for witnesses co-operating for the betterment of Steem, but I'm concerned about the harsh realities of the situation. Is it even feasible?
This screams inefficiency. Very few database providers require costs in this range. It's time we take a serious look at providing infrastructure instead of just "buying more AWS instances" because they have cash to blow.
I've been digging into the API flow of steem recently and quite frankly I'm shocked at how poorly maintained things are. My rough estimates are at least one order of magnitude improvement with relatively minimal changes. This isn't theoretical, I have actionable plans for improvements.
So is it feasible? Yes, absolutely. Sure, we have to encourage development and deprecate old API formats, but it will be worth it in the long run.
Lets take our heads out of the bull-market induced sands. They call the period after a market bubble the "plateau of productivity"... lets embrace that, and buckle down and work.
@anyx us investors also have skills and connections, I and probably other stakeholders would be willing to use our many skills and connections to help @ned with getting the liabilities down and the assets up etc etc.
@ned has earned reputation and goodwill from Fyrstikken Inc. So all we need to do is to get together virtually and talk and do things like a scheduled team/external board members & advisors.
I normally charge a lot for this important service, but because of Goodwill, I will give @Steemit Inc. or @ned personal an ONE-year agreement for ONE DOLLAR for ONE dedicated hour per week over 52 weeks.
May I suggest Sundays since that is the day most leaders sit alone in their home offices and plan for the battles of the week a day before the non-leaders arrive on Mondays, so Sundays it is a good day to meet with very little distractions, 100% focused & 100% dedicated like stakeholders/co-owners of the STEEM blockchain.
You game?
Thanks for the insight, much appreciated. Like I mentioned, I had figured things could be streamlined significantly, but didn't expect 10x overspending from Steemit Inc. It doesn't sound possible, but then again, laying off 70% of your workforce months after going on a hiring spree seems far fetched too.
All the best, I hope both Steemit Inc. and fellow top witnesses pay attention to your plan and implement them.
I full support that mentality!
Logic isn't everything. You need a good heart, too. The question is: Where can you buy one?
like that idea! I am in!
Fully agree with this
This is along the lines of what I was thinking.
This, 1000%. Steem blockchain is really not decentralized at all in actual practice if everything grinds to a halt when there are issues with the steemit api nodes. Many top witnesses already run full nodes and most if not all are barely utilized at all.