The day the blockchain stood still

---

Yesterday, the blockchain - Steem - stood still and in this post, I'm going to take a retrospective look at the events.

What happened?

Now, due to a bug introduced in the new version v0.20.0 and activated after an edge case scenario, the blockchain froze. (Steem works in that way, that if there is a critical problem, the blockchain just halts)

After a few minutes of being in the black, we got the news inside the #witness channel on steem.chat by @timcliff: the TOP 20 witnesses came to the consensus to rollback to 19.x.

---

---

I believe that this was the correct decision.

Problems: Low Communication from Steemit Inc. & No ETA

But what happened afterwards, was a very long period of waiting of over 10 hours without any ETA and not enough communication.

Now, please don't get me wrong. I can fully understand that fixing important bugs takes time and sometimes, giving ETAs can be difficult.

But as a witness and as someone who is responsible for a business running on the blockchain, I need to know what is going on. (And I'm sure that I wasn't the only one. Steem Monsters is running a kickstarter right now in addition with being on Token Fest. Not being able to tell the customers when they can buy packs again etc. - that's lost money and unhappy customers)

Getting the info: a difficult path

And the flow of communications was very supoptimal.

I had to use a combination of the #witness channel on steem.chat, steemspeak on discord and the official SteemNetwork twitter account to be in the loop of what was happening.

Finally, at 2am in the morning for me, (nearly 10 hours after tim wrote the message on steem.chat) I was made aware of a solution. But I only knew of it by being in the steemspeak discord and hearing @themarkymark say that a specific version was released on github.

4 hours later and my nodes were replayed and first block was produced.

My Lessons out of this

I'm a big believer of taking a step back and using experiences as a way to grow and learn.

And in retroperspective, I think I wouldn't have needed to replay one of my nodes, as it was already on the right version, which would have made my node one of the first 20 running on the chain.

But in this case, I was active rank #24. (up from the real rank #32). Still very good, as I'm estimating that the TOP 20 got a heads up with the new fix, to not undermine the security of the blockchain. Which I understand.

Besides that, I also learned how important it is to always have a different version on your backup node. Especially if it's such a huge upgrade as v0.20.0.

And regarding the problems I've explained above:

Steem has a market capitalization of over 230 Million USD and this is on the low end. We were easily over 2 Billion. And a company (or however you might want to call it) of that size, has to make sure there is an open communications flow, especially with witnesses or business owners who are relying on the blockchain.

So please, @ned, take this feedback seriously. I love Steem, I want it to succeed and that's why I'm writing these lines.

Didn't witnesses review the code?

Now, regarding the accusations that witnesses didn't review the code enough, otherwise the problem would have been known.

Well, yes and no. First of, reviewing Steem code is actually very difficult as it requires a very high level of programming knowledge of c++, in addition with deep knowledge about blockchains & Steem.

On top of that, the error was only visible in an edgecase scenario and would have needed extensive tests with specific circumstances (at least that's my understanding).

While I am totally in favour of witnesses reviewing the code, most of us don't have the necessary skills. Now, achieving these skills is on my to-do list, but it will require quite a lot of time and for me, Steem has also multiple other needs. For example: we need more dApps like Steem Monsters & Co.

And currently, I rather focus on my strengths and the skills I have already developed to bring value to Steem by being as effectiv as possible.

Is Steem not decentralised?

Last but not least, I also want to get into the accusation based on yesterdays freeze: Steem is not decentralised.

In my opinion, this is not correct.

Steem isn't being run on one server or by one organization, which would be centralisation, but on many, many different servers all around the world.

Now, you could argue that some servers are being hosted in the same data-centers and from the same providers, or that some witnesses outsource their hosting to other witnesses and yes - those are security risks everybody should take seriously! (!)

Which means: the fact that Steem has hundreds of witnesses running multiple servers shows that it is indeed decentralised.

However, there is a bottleneck/weak point in Steem, which caused yesterdays problem, but all other blockchains have this in common: the code.

And I don't mean some holy code or law code, I mean the written code by the developers.

If there is a bug or problem inside the code, we could have 1000% more witnesses, it wouldn't change anything.

Maybe far in the future, we will have an alternative, but for now - we need to be very careful about the code and make sure it's well audited.

---

With that said:

Have a great day!

therealwolf

---

^{Witness Infrastructure:
Primary Node: 128GB - v0.19.12 | Backup Node: 64GB - v0.19.12 | Seed Node: 64GB - v0.19.12}

^{Projects I've developed on Steem:
Smartsteem.com > Investment & Promotion Service on Steem
Steem Chat-Wallet & Witness Essentials > Github: https://github.com/therealwolf42}

^{If you believe that I'm of value for Steem, then please vote for me as witness. You can also set me as a proxy and I'll vote on great witnesses for you.}