Did We Learn Any Lessons From HF20?

in #steem6 years ago

ats-witness.jpg



Let’s not mince words or beat around the bush...

Hard Fork 20 has not been a “success.”

Yes, the hard fork was accepted and there seemed to be no trouble updating to version 0.20.2. However, immediately after updating, the actual effect on nearly all users of the Steem blockchain was disastrous. Twice in just over a week, the blockchain was essentially rendered useless for 12+ hours, the second time lasting about 24 hours.

Both instances could have been avoided. If not avoided entirely, effective downtime could have been mitigated or at least reasonably expected and dealt with accordingly.

Regarding the actual update to version 0.20.2, we were given an 11th-hour admission by the Steemit, Inc. communication team that they essentially had no idea what was going to happen once HF20 was live. It shouldn’t be surprising to know that the uncertainty likely stemmed from the fact that the negative RC balance problem which caused the user issues was known during the testing period.


Despite knowing about the negative RC balance problem during testing, the planned fork went ahead as scheduled.


Coupled with this irresponsibility was the fact that both @ned and the @steemitblog account publicly declared that the hard fork was a “success” – while nearly the entire user base was left unable to comment to the contrary and looked on in absolute amazement at the wholesale detachment from reality by Steemit, Inc.

This is not the first time that a hard fork has left users in a state of disbelief – or even the first time that a near-complete reset of blockchain functions/abilities/rewards has occurred. We witnessed a nearly identical result after HF18 went live, where post rewards were reset to zero...without warning or the least bit of preparation. There was a “Whoops! Sorry, everyone!” from the Steemit, Inc. team and the usual excuses that we have all unfortunately grown accustomed to accepting.

Without getting too far into all of the arguments again, I just want to say...

We deserve better.


“Why didn’t the witnesses prevent this???”


I’ve seen this asked everywhere the last few days. And I don’t really have a great answer to the question, but I do have some answers, so I’ll try my best to communicate what could have been done, what should have been done, and what actually happens.

Yes, witnesses are to blame, but they are not solely to blame and not all witnesses share the same burden of blame, if they share any at all. What I mean by this is – there is a pecking order for witness influence and how hard forks are proposed and accepted. Here’s my best take on how things really work, according to my own observations for over two years and based on conversations and feedback from other witnesses.

The top-20 witnesses – by design – clearly hold all of the cards when it comes to hard forks. Hard forks require acceptance from a super-majority of the top-20 witnesses in order for the proposed fork to be accepted as the new version of the Steem blockchain going forward. If the top-20 either rejects or accepts the fork, the rest of the witnesses must then operate their nodes accordingly on the old version (if rejected) or the new version (if accepted). If they are not operating on the proper version, then their witness node is disabled and they do not sign blocks.

So regardless of whether or not a witness is competent enough to review and test code, if they are not in the top-20, then their vote on approving or rejecting a hard fork essentially does not matter.

Because the top-20 witnesses are theoretically supposed to be the people/organizations within the Steem community with the highest level of support and confidence to do their job, and because of the responsibilities bestowed upon them to do their job and to do it properly, these witnesses are compensated much more than the rest of the witnesses in the community.

With that and with their ability to reject or accept hard forks as a collective group kept in mind, it is wholly incumbent upon these top-20 witnesses to seriously and thoroughly review and to robustly test any protocol changes proposed as a hard fork. After all, it is the sake of the network that depends on their actions and decisions during these processes – the very network that pays them and where they are presumably invested themselves.

From all accounts that I’ve seen regarding HF20, this review and testing by our top witnesses largely did not occur. Despite not testing the new code, all of them updated to version 0.20.2 by Tuesday’s fork date.

But it’s at this point that we need to discuss some extenuating circumstances.

As pointed out by myself and many others over at least the past year and a half, the proposed protocol changes in the recent hard forks (17-20) were mostly too large to adequately review and there was not a proper testing grounds for the new code. The former issue has been mentioned directly to Steemit, Inc. many times. They continually propose massive amounts of protocol changes in hard forks, and witnesses – even if they could review it because of their knowledge of C++ code – could not possibly go line-by-line in a reasonable amount of time, then also run formal tests afterwards to an adequate degree that would allow them to make an informed decision about whether or not the code is safe for acceptance.

Granted, with the rewards that the top-20 witnesses have received and do receive, they could hire some help to review and test. However, the better solution would be to simply reduce the number of simultaneous changes and the size of them as much as possible. If that were to happen, then there would be absolutely no excuse for not reviewing and testing any new code.

But on this front, the repeated requests from witnesses to reduce the size and scope of hard forks has fallen on Steemit, Inc.’s deaf ears.

So what can be done by witnesses when these types of hard forks are proposed?

That’s an easy answer: Reject them.

Reject the hard fork as proposed and ask for a “clean fork” that can be both easily reviewed and tested, and/or that represents an immediate need, and/or that involves one major protocol change at a time. Yes, there may be changes in a large hard fork proposal that will improve the blockchain and user experience, but we should not forgo our duties as witnesses simply because we’re afraid to say, “No.”

And this brings us to another problem for witnesses: Pressure to tow the line.

I’m not saying that this happens with all witnesses, but there is a culture of cheerleading and shunning that occurs in this community and has occurred since day one. It has permeated every aspect of the platform, including – and especially – among witnesses. If you are critical of Steemit, Inc. or popular blockchain protocols (regardless of whether or not you believe that either are actually beneficial to the blockchain and community), then you can quickly find yourself on the outside looking in.

But the worst part of this culture is that bad ideas, faulty code, and poor behavior is often blindly supported and accepted because criticism is not handled well by those being criticized and because those who criticize are targeted and shunned by many different people and groups within the community...including by Steemit, Inc. themselves. It doesn’t even matter if the criticism is articulated well, if it’s pointed/accurate, and if it’s being done in the best interest of the Steem blockchain and its users/investors.

Regardless of whether or not you like the people who are critical, it’s no reason to support and accept those bad ideas, faulty code, and poor behavior. Even if it’s not intentional, this support and acceptance still happens far too often.

For those who are critical and spend a lot of time articulating their positions, if you say the wrong thing to the wrong people or if they see that you’re not supportive of their plans, then you will not be supported, you will be denied access to discussion groups where critical information and updates are shared, and you are pretty much guaranteed to never see the top-20 as a witness. Again – looking out for the best interest of the blockchain and its users, instead of a corporate entity’s best interests, may leave you working your ass off for pennies.

The fear of losing a top witness position is certainly enough to keep people “in line” or the incentive to reach one of those spots is certainly enough to encourage witnesses to “get in” it. It is partly because of this reason that hard forks are almost never examined critically and especially publicly by witnesses. But it’s not the only reason.

This leads us to another serious problem with many of our witnesses, the ones at the very top notwithstanding: Many do not understand enough about the blockchain (and coding), economics (and behavior), and social media to make informed decisions.

Some are skilled with C++ coding (but very few). Some know a few things about economics. Some know a few things about social media. But how many know enough about all of these things in order to make rational, informed decisions regarding the dynamics of each proposed protocol change? It’s not even a question about code review and testing. It’s a question about whether or not the concepts and the rationale for any proposed changes actually makes sense within the context of the Steem blockchain and the general socio-economic vision for it.

I’m definitely not saying that everyone needs to be well-versed in socio-economics in order to be a witness. However, it would be nice to see an effort by witnesses to at least better understand basic concepts of economics and monetary policy in order to properly manage things like Steem Dollars, APR, and bias. Many of our witnesses never even bother setting these parameters...or know their purpose.

I’m also not saying that every witness ought to be a professional behavioral psychologist. But it would help if they better understood incentivized behavior...if they had a better understanding of things that shape behavior, like antecedents and consequences, so that when behavioral changes are discussed (as with our latest hard fork), it would allow for some careful consideration of the protocol changes in question.

The Steem protocols are part of a complex system that includes making a lot of assumptions about user behavior and incentives. If we have no idea about things like profit motives and interest rates, or what a time sink is, then it’s going to be exponentially more difficult to comprehend protocol changes and witness parameters where these concepts are already implied.

So, all of that being said...

In the end, we’re left with a far less-than-ideal culture that largely punishes critical voices while encouraging and rewarding yes-men, excessively compensates the unskilled, inexperienced, and/or unmotivated, and allows and often excuses the same repeated mistakes from the same people/entity.

It’s no surprise then that hard forks are proposed, accepted, and fail to deliver on promises...or become implementation nightmares. This happens for all of the aforementioned reasons – but it shouldn’t and doesn’t have to be that way.

I would encourage all users – witnesses, whales, and everyone else – to please give more consideration to your witness votes. Take a minute to look around, ask around, and contact individual witnesses directly to find out more about us and what we are currently doing on/with/for the Steem blockchain. If you’re not satisfied with the way things are going, then you are the ones with the ability to change direction.

(And that includes you, @freedom. You are the single largest voter of witnesses. I would hope that you take that vote seriously.)

This last hard fork should not have been accepted. It was simply too large, included too many moving parts, was clearly under-tested, and the dev team and testers had at least one major concern about it. Witnesses dropped the ball by accepting the fork and the Steemit, Inc. dev team was not open enough with witnesses and the community about what was likely to occur...again.

We are now on our fourth iteration of HF20 but it has only been officially in production for three days. And we can’t even be sure that another patch or two won’t be necessary soon.

This is unacceptable.


My promise to you as a Steem witness.


I recently mentioned some of this on a post from @remlaps, but I want to include it here, for the record.

If I were to find myself as a top-20 witness, I can promise you that I will do the following:

In addition to my usual availability and engagement with the community and my general reviews of hard fork proposals, I will...

  1. Operate a public RPC node.
  2. Have the code for all proposed protocol changes professionally audited.
  3. Direct my team to participate in any official testnet for new code.


Regarding specific proposals for protocol changes, I will accept them based on whether or not...

  1. There is a problem in need of a solution – OR – There is a protocol change that is persuasively an improvement upon existing code.
  2. The rationale for the code is coherent with the existing set of protocols.
  3. The proposal is presented in a size that is easily manageable/auditable.
  4. The code has been professionally audited.
  5. The code has been adequately/robustly tested.


If any proposals do not meet this criteria, then my default position will be to reject the proposal.

I would hope that other witnesses take a similar stance on protocol changes and actually stick to it. I understand the eagerness to try new things or to “move the blockchain forward,” but that does not necessarily mean that we need to constantly tinker with the economics of the platform. Stability can be just as attractive and productive than continual change. Forking the blockchain should not be so easy to accomplish. It should not be a given.


Share your concerns and give us feedback.


I know some of you out there will just complain that I’m being critical once again. I understand that.

I also don’t care.

We need more critical voices – more people willing to go against the status quo. It is the critical voice that makes you more adept at defending your ideas. It’s the people looking for and pointing out errors that pushes you to be more careful to not make them. You don’t harden your project by cheerleading, rubberstamping, and shutting out criticism. You don’t learn from people telling you that you’re never wrong.

So be critical. Find the things that you don’t like. Express your displeasure. Communicate your concerns to those people that you believe will actually listen and may be able to help.

But don’t just be critical. Highlight the things that you believe are good…and ask for more of them or propose ways to improve upon them.

As stakeholders, this blockchain is the responsibility of all of us. It’s time that we start acting like it instead of trying to prematurely kill it…either by fattening it up to morbid obesity through cake-eating and complacency, or by killing the proverbial golden egg-layer.

It's in our own hands.




Vote for

ats-witness_banner_small.jpg

Block-change you can believe in!


Sort:  

There was, and is, a lot of heated debate with regards to HF20 and its readiness for prime time. The top 20, and those of us just under, are not in lock-step with Steemit, Inc. While we try to work with them disagreements will happen and often do. As a witness I sincerely apologize to the entire Steemit community for the disruption in service.

Thanks for the insight on your remarks about this hard fork. I appreciate that you tried to do what was right.

I am by no means stating that everyone is in lock-step. I am simply stating that there is a lot of pressure to be on board with Steemit, Inc.'s plans for the blockchain - and it's not just from them. It's pretty hard to deny the culture that I mentioned in the post when it's something that has been observed by many longstanding members of the community and new members alike. Those who push back against these pressures and this culture are risking a lot, as I'm sure you've noticed over the past 2+ years.

Even with that pressure and the culture here, I can't imagine why any top witness would want to push this fork through, knowing that it was an extremely ambitious hard fork and knowing that there was limited testing...and knowing that there were in fact problems with negative RC balances. And specifically in light of last week's halt to the chain due to one line of code that contained an error, I have a hard time believing that there wasn't more doubt among witnesses that this fork was going to be implemented smoothly.

Under such conditions, with access to that kind of information, it's mind-blowing to me that all of our top-20 witnesses updated to the new version anyway. I can only conclude that they...

  1. Were not informed of the negative RC balances.
  2. Did not sufficiently audit and/or test the new protocols.
    And/Or
  3. Didn't care enough to be bothered with doing their witness duties and finding out about any potential issues with the code.

Any one of these three is a major concern for this blockchain and its users/investors. As a whole, the "most trusted" people in charge of Steem blockchain oversight and functionality were completely derelict in their duties. I think this is a huge signal to users and investors that there can be little actual trust in such a "trustless" network. And I think it also speaks volumes about the level of centralization that actually exists rather than the theoretical decentralization that everyone seemingly pretends we have.

Anyway - thanks for taking a stand against what many people believe was a failure of implementation, regardless of whether or not they were for or against the changes. I had already stated prior to the fork that I would like to try the RC system, but I was cautious/concerned about the scope of the fork and the implementation of it. I think my concern and the concerns from people like you and @drakos have been validated. I will continue to oppose these massive protocol change packages for as long as they continue to be proposed.

Frankly, I had such high hopes for Steemit. However, so far I am disappointed.

Hard truth is that all witnesses need to stand up to these bullies and do the right thing. Maybe next time your preached down to have some balls to pull off your own fork and go your own way. Because if ned and steemit blog can 11th hour everybody off a bridge than you better not accept the next glass of kool-aid. Does anyone even know who these people are? Look at EOS.

Steemit is awesome. All is Forgiven.

Posted using Partiko iOS

As well you know, there should not have been a hardfork with supermajority support if there is still heated debate. Again, as well every top 20 witness knows, the moment all top 20 witnesses start producing with a 20.x node, it means they support HF20, not that they are still debating it. That HF20 is still being debated after they elected to run a 20.x node and the chain forked is beyond me.

While time will heal the downtime, the lack of confidence will haunt unless there is some evidence that lessons were learned in all camps.

The top 20 witnesses are the last line of defense between the end-users and hacking, incompetence, or catastrophic errors. While I do not have a large investment here, it is enough I want to feel it is in good hands.

I still see more people fighting rather than assisting with the testnet, etc.

While I am waiting to see how this pans out, it is difficult to feel much can change with Pumpkin and Freedom.

I also want to acknowledge that I saw you communicating on the blockchain prior to the fork about your concerns. Although I don't always appreciate your communication style, kudos for pointing out the risks and doing what you could to bring awareness.

The buck stops with the witnesses, not SteemIt Inc.

Pumpkin (and thereby Freedom) do change their witness votes from time to time, so all is not lost. The question is, who controls pumpkin and what are their parameters for choosing witnesses?

The buck stops with the witnesses, but if they are all "Yes Men" (and women) then we will never see critical thinking step in and make reasonable requests/demands. Everyone is too afraid of losing those witness rewards for producing blocks.

So really, the buck stops with the witness voters. We do have enough cumulative power to outvote Freedom if we ALL started using out witness votes. Unfortunately, so many people don't even bother to vote for witnesses or even know what it means to vote, or be a witness, that we will not be seeing these changes any time soon.

Pumpkin (and thereby Freedom) do change their witness votes from time to time, so all is not lost. The question is, who controls pumpkin and what are their parameters for choosing witnesses?

That would be amazing to know.

So really, the buck stops with the witness voters. We do have enough cumulative power to outvote Freedom if we ALL started using out witness votes. Unfortunately, so many people don't even bother to vote for witnesses or even know what it means to vote, or be a witness, that we will not be seeing these changes any time soon.

Maybe we should make major pushes to encourage people to proxy their votes more. Some people really enjoy keeping their ears to the ground and shift their choices frequently depending on developments. In my experience I went from not even knowing 10 witnesses I'd like to support to knowing who most of the top 100 are and wishing I could vote more than 30.

It's really not that realistic to expect your average Steemit user to gather the information necessary to choose 30 witnesses unless they have specific interest in it. If we had a culture where proxying was more popular, even a tiny minnow who dedicated to staying up to date with witness activities could have a pretty big say in matters.

I like this idea, however, first you have to teach the average Joe Steem users that they should care who their witnesses are. I think there are a lot of people on Steem who do not care one way or the other.

Maybe this current situation could be explained in a way that encourages people to set a proxy, but who do we encourage them to proxy to? Asking people to proxy needs to be accompanied with a list of potential proxies for people to consider. Asking people to proxy to yourself can easily be read as a power play.

So convince the people who don't vote for witnesses to vote, make it easy for them to proxy, and give them a list of potential proxies to choose from. I think then you might have a SLIGHT chance of making a small difference in the witness rankings.

But maybe I am too pessimistic.

Maybe we should make major pushes to encourage people to proxy their votes more. I've been thinking about this also.

There are many, probably most who do not have the ability to vote on witnesses. Although I am well informed, I am likely on the cusp of this myself.

Those who are "yes men" because they fear losing money are bad actors, but I feel an inability to have any influence.

I'm waiting for the dust to settle, but I can't currently trust the system.

you have some good forking points there... ;)

****echo****

While I am waiting to see how this pans out, it is difficult to feel much can change with Pumpkin and Freedom.

I also would love to see evidence that lessons were learned... show us witnesses! stand up!

Steemit Inc could outvote Pumpkin and Freedom if they voted for witnesses.

Dude.

tl;dr

He said, all this ole borking hardforks is BS. Time to start holding witnesses responsible for more than just signing blocks..

tl;dr:
SteemIt Inc engineers aren't the best in the world.
Witnesses are circle-jerkers.
Vote @ats-witness for witness

I second that! I also think the proxy vote idea would be awesome, now trust of the individual holding that proxy power for the betterment of Steem comes into play.

I have thought it before why not push for @DTube proxy votes? I would do it! You have the team with sufficient knowledge available to do ample research, just a thought I’ve had for awhile seems a good time to share. The DTube community is one of the fastest growing in terms of SP and I personally see much potential in making corrective impacts having it used wisely.

I had considered this for @OneLoveDTube as I have spent a fair amount of time explaining to newbies about importance of these witness votes, I know witness votes are SP relative. Trying to simplify it was taking me down the proxy path yet I am still uncertain whom to bestow that responsibility on to.

Posted using Partiko iOS

I hope that the lessons learnt here will be put to good use for future hardfork. Let us learn and move on. Steemit, we expect better next time.

Posted using Partiko Android

It's not only about the code, but also about what is this code doing?

It's an inflation / dilution game. And because of this downtime, and these limits that are in place that will be finetuned but still will be in place, we the community / users are set back, and we will loose this game. In simple words, we will loose our investment, slowly but steady under the impression we have a fair chance to come out at the other end.

See my comment below about filing a complaint with the FTC.

I agree with the point about having less changes per fork - this approach of many changes together looks a lot like the way governments ram through masses of changes and bury hard to swallow details in there so deep that the reviewers have no chance to find them and comment. This is NOT a good look!

As a witness and a software engineer, with C++ background and experience of working in the banking and social networking industry - I am very frustrated with the situation and really need to see changes here. Yours is the only post from a witness I have EVER seen (other than mine) proposing to use top 20 funds to pay for professional testing. I raised this issue myself in several recent posts.

I have also just proposed that the witness reward pool gets more evenly distributed so that we don't lose millions into a black box that has no measure of return - plus also allowing other competent people to use their passion to work on the project in useful ways.

...this approach of many changes together looks a lot like the way governments ram through masses of changes...

I've made this exact comparison in the past. It's a really crappy way to push needed changes through that actually have super-majority consensus, not only among witnesses, but within the larger community as well.

Yours is the only post from a witness I have EVER seen (other than mine) proposing to use top 20 funds to pay for professional testing.

Unless we are well-versed and experienced with C++ and auditing code, I think hiring someone to do that job is the most responsible thing to do when collecting large sums of money to perform duties as a top witness. Anything short of that would be borderline reckless, in my opinion. How can you justify accepting protocol changes for a blockchain valued in the hundreds of millions of dollars, with hundreds/thousands of large investors, if you haven't bothered to have those protocols professionally audited and tested?

Going forward, I think more professional audits and testing is really the only way to proceed. If it can't be done before the official implementation dates, then the hard forks ought to be rejected. In light of recent events, that's the only reasonable path.

Yes, I agree. I actually am experienced in C++ and testing, so I know something of the complexity and size of the task (though I wouldn't know all the details without going more deeply into it all). Co-ordination goes a LONG way when it comes to testing complex systems and I really think we need to have a witness testing channel in chat as a minimum. That in itself would help expose what is and is not occurring.

I worked out earlier (after getting the maths wrong the first time around) that top 20 witnesses currently receive in the region of about $90k per year. Paying others to do professional audits would take a sizeable chunk of that - but if they all split the cost it would be doable. This then comes back to the issue of potentially redirecting some of the witness budget towards tasks such as this but then we have the politics of who gets the job and so the complexities of getting so many individuals to work together in a coherent way just spiral on. I suppose a way of rewarding code changes and bug hunting would go a long way towards helping but @utopian-io doesn't even currently have Steem whitelisted, for reasons I don't understand. :/

It is of note that witnesses could also choose to hire an organization to audit the code together. It's not necessarily required that witnesses all hire their own programmers to audit the code.

The hardfork went amazingly, now just to show you I am going to up-vote my own comment.

Considering the special occasion, I'm going to up-vote this comment as well ;)

Now, just one more time lol

What a disaster.. there accounts that lost millions of dollars to to this HF20

I hate to say this, but I think STINC calls this fork a success because it was for them. It achieved what they cared about and everything that went wrong they really don't care about.

I kept being perplexed at how they could be so incompetent at protecting user experience during the upgrade. I mean, are these not professionals with some concept of Change Management procedures?

After numerous conversations it has become clear to me. They don't care about Steemit or users on Steemit, except how we help them prepare for the release of SMTs. That's the business they want to be in, not providing end user experiences themselves.

So yeah, this is how they roll. And this is how it's going to be. And they don't care how many newbies leave along the way. They just need the data and enough of us here to present proof of concept.

So those of us who stay with that understanding can stop being disappointed. I would appreciate witnesses doing what you describe as your intention above, and will give you a vote. But ultimately STINC is going to keep pressuring witnesses to rubberstamp their decisions because that's all they want you to do. They don't necessarily have the same priorities as we might wish they had.

I hate to say this, but I think STINC calls this fork a success because it was for them.

This has been my suspicion for a while now, looking back at the last several hard forks. It seems to me that they have been setting up a way that they can maximize the profits from their ninja-mined stake rather than use that stake for the purposes of development, marketing, and onboarding new users as they had originally stated it would be used.

The ninja-mine has always been one of the largest attack vectors against the Steem blockchain, and instead of addressing this by doing something reasonable (since they are clearly not using it as "promised"), they are leveraging it to further enrich themselves by selling/delegating that stake, by creating Resource Credits that can potentially be bought/sold in a marketplace, and by not having to spend that stake on new user onboarding.

So, the development has been shoddy, the marketing is non-existent (and there is no marketing team at all, as far as I know), and the onboarding can essentially be free for them. In light of this...why did/do they need that massive ninja-mined stake that has caused so many issues both within and outside the Steem community?

The push for SMTs just seems like more centralized development for a vision that, to me, will likely be destructive for the blockchain.

Holy Shit @ats-david!!

This has been the most lucid and eloquently crystal clear comment describing and illustrating with outstanding transparency the current situation of the steem blockchain that I've read in months. };)

Let's see how many more are willing to swim deeper in it.