How to serialize and sign Steem transactions using Javascript (without Steem Javascript libraries)

in #steem5 years ago (edited)

document-428334_640.jpg

When you are sending STEEM or SBD to another account, when you are writing a post or a comment, or when you are voting or downvoting a post, you are broadcasting transactions into the Steem blockchain. In other words, any action you take on Steem blockchain is a transaction.

What makes transactions so special?

A transaction needs to be signed by the account owner and verified by the blockchain before including in the blocks. Without a signature, the transaction will be rejected by the blockchain.

You may ask "why signing? isn't there another way?"
Without using signatures, blockchain needs a way to authenticate users before broadcasting transactions. And for authenticating users, there must be a key included in the request that sent to the blockchain.

As you know, sending keys over to a server that claims to be a blockchain node is risky. Since anyone can run a blockchain node and capture keys. Of course, we can't trust the blockchain that lacks security.

With the help of encryption, we can sign transactions on our own hardware and send signed transactions securely to any server. A signed transaction doesn't include any key. Just a signature that proves the transaction is coming from the account owner.

How to broadcast transactions?

Broadcasting a transaction is pretty easy.
But before broadcasting, the transaction must be signed.
And for signing a transaction, the transaction must be serialized.

Available documents:

I couldn't find anything else.

Available libraries(JS):

steem-js and dsteem both are working in most cases.
But I couldn't use steem-js nor dsteem on the Nativescript environment (framework for Android and IOS applications). So I created Steem-TX. A complete but light library. Steem-TX should work in almost every situation.

I will explain things that are not available directly anywhere else. If you don't understand any part, just comment below. (or maybe google it 🤷)


Step 1

Creating Transaction:

As you know, a transaction consists of 6 variables inside an object:

const transaction = {
  ref_block_num: Number,
  ref_block_prefix: Number,
  expiration: Date,
  operations: Array,
  extensions: Array,
  signatures: Array
}

We will fill this object by the end of the post.

@xeroc explained most of the properties. Check this post for explanation of variables.

Defining 4 of 6 needed variables:

const props = getDynamicGlobalProperties()
const refBlockNum = props.head_block_number & 0xffff
const refBlockPrefix = Buffer.from(props.head_block_id, 'hex').readUInt32LE(4)
const expireTime = 1000 * 60
const expiration = new Date(Date.now() + expireTime)
    .toISOString()
    .slice(0, -5)
const extensions = []

Without operations, there is no transaction!
You can include one or more operations in this array. Check Devportal for list of all available operations.

const operations = [
  [
    'vote',
    {
      voter: 'guest123',
      author: 'guest123',
      permlink: '20191107t125713486z-post',
      weight: 9900
    }
  ]
]

Transaction is created:

const transaction = {
  ref_block_num: refBlockNum,
  ref_block_prefix: refBlockPrefix,
  expiration,
  operations,
  extensions
}

The only missing part is signatures


Step 2

Serializing transaction:

Create a new ByteBuffer:

const ByteBuffer = require('bytebuffer')
const buffer = new ByteBuffer(
    ByteBuffer.DEFAULT_CAPACITY,
    ByteBuffer.LITTLE_ENDIAN
  )

Every variable should be serialized according to the data type of its value.

We will use the following serialization for transaction object:

  • ref_block_num => UInt16 (16bit unsigned integer)
  • ref_block_prefix & expiration => UInt32 (32bit unsigned integer)
  • operations:
    voter & author & permlink => VString (varint32 prefixed UTF8 encoded string)
    weight => Int16 (16bit signed integer)
  • extensions => VString

Also, you need operation_id of operation. operation_id is the position of the operation defined in the Steem blockchain. Here is the list of operations on the Steem blockchain.

vote_operation: 0
comment_operation: 1
transfer_operation: 2
...

The code:

buffer.writeUInt16(refBlockNum)
buffer.writeUInt32(refBlockPrefix)
buffer.writeUInt32(expiration)
buffer.writeVarint32(operations.length) // number of operations
buffer.writeVarint32(0) // operation id
buffer.writeVString(voter)
buffer.writeVString(author)
buffer.writeVString(permlink)
buffer.writeInt16(weight)
buffer.writeVarint32(extensions.length) // number of extensions
// in case of any extensions
// buffer.writeVString(extensions[0])

The serialized transaction is ready.


Step 3

Create Digest from Transaction:

First, convert byte buffer to the actual buffer:

buffer.flip() // set offset = 0
const transactionData = Buffer.from(buffer.toBuffer())

Create SHA256 hash of transactionData and chain id (digest):

const chainId = '0000000000000000000000000000000000000000000000000000000000000000'
const CHAIN_ID = Buffer.from(chainId, 'hex')

const input = Buffer.concat([CHAIN_ID, transactionData])

You have 2 options here:

  • crypto-js
const CryptoJS = require('crypto-js')
const wa = CryptoJS.lib.WordArray.create(input)
const digest = Buffer.from(
    CryptoJS.SHA256(wa).toString(CryptoJS.enc.Hex),
    'hex'
  )
  • crypto
const crypto = require('crypto')
const digest = crypto.createHash('sha256').update(input).digest()

USE ONLY ONE OF ABOVE (crypto-js or crypto)

I used crypto-js in Steem-TX but you can use whichever that works for you. The code for crypto is also commented out in Steem-TX which can be easily replaced with crypto-js.

Now prepare private key:

const bs58 = require('bs58')
const key = '5JRaypasxMx1L97ZUX7YuC5Psb5EAbF821kkAGtBj7xCJFQcbLg'
const keyBuffer = bs58.decode(key)
const privateKey = keyBuffer.slice(0, -4).slice(1)

With digest and private key, we can finally sign the transaction.


Step 4

Sign the Transaction:

We will use this function in the signing process to verify the signature:

const isCanonicalSignature = signature => {
  return (
    !(signature[0] & 0x80) &&
    !(signature[0] === 0 && !(signature[1] & 0x80)) &&
    !(signature[32] & 0x80) &&
    !(signature[32] === 0 && !(signature[33] & 0x80))
  )
}

We can sign the transaction with more than one key in the case of multi-signature accounts. Just repeat the process of signing for the signedTransaction.

const signedTransaction = { ...transaction } // copy transaction object
if (!signedTransaction.signatures) {
  signedTransaction.signatures = []
}

// repeat for multi-signature accounts
const secp256k1 = require('secp256k1')
let rv = {}
let attempts = 0
do {
  const input = Buffer.concat([digest, Buffer.alloc(1, ++attempts)])
  const wa = CryptoJS.lib.WordArray.create(input)
  const options = {
    data: Buffer.from(
      CryptoJS.SHA256(wa).toString(CryptoJS.enc.Hex),
      'hex'
    )
  }
  rv = secp256k1.sign(digest, privateKey, options)
} while (!isCanonicalSignature(rv.signature))
const signature = {signature: rv.signature, recovery: rv.recovery}

Finally, the transaction is signed!
But, we must convert the signature into the string and include in the transaction object:

const sigBuffer = Buffer.alloc(65)
sigBuffer.writeUInt8(signature.recovery + 31, 0)
signature.signature.copy(buffer, 1)
const stringSignature = sigBuffer.toString('hex')
signedTransaction.signatures.push(stringSignature)

The end.

Step 5

Broadcast the Transaction:

signedTransaction is ready to be broadcasted into the blockchain using Condenser Api and broadcast_transaction_synchronous method.

If you understand all the steps from 1 to 4, I think you should know how to make a post-call to a Steem node and broadcast the transaction. In case you don't, check Devportal

You can use any library. I used axios to broadcast transaction:

const axios = require('axios')
const node = 'https://api.steemit.com'
const call = async (method, params = []) => {
  const res = await axios.post(
    node,
    JSON.stringify({
      jsonrpc: '2.0',
      method,
      params,
      id: 1
    })
  )
  if (res && res.statusText === 'OK') {
    return res.data
  }
}

const result = await call('condenser_api.broadcast_transaction_synchronous', [
  signedTransaction
])
console.log(result)

Conclusion:

After reading or writing this code you will appreciate the software that is doing all this process behind the scene when you are clicking on the upvote button.

You don't have to write this code from scratch. Steem-TX is already coded in the same way. See the announcement post.

Feel free to test, use, and share. With the hope for better Steem!

Your witness, @mahdiyari


Image source: pixabay.com

Sort:  

Great work!
Have you done performance tests?
Like this: https://esteem.app/steemdev/@almost-digital/dsteem-vs-steem-js-round-1

100,000 transaction sign:

dsteem: 51376.950ms
1,946 tx/s

steemTx: 58886.287ms
1,698 tx/s

10,000 transaction sign:

dsteem: 5605.882ms
1,783 tx/s

steemTx: 6554.588ms 
1,525 tx/s

SteemJS is too slow for such huge tests
100 transaction sign:

steemJS: 7930.872ms
12.6 tx/s

I think the difference in the performance is because of the used dependencies.

Thanks, that's promising. Using with crypto-js ?

We moved to dsteem due to better performance and footprint, each package should be secure and fast with minimal dependencies. It is important for us at eSteem, so will be testing this out.

Yes
crypto-js

Something definitely helpful for me to sign transactions for my Steem Ledger hardware wallet project 😃

It is interesting

Can we just download and use in a simple html page for demonstration ?

It will be good, if we can have some simple tutorials that one can run ?

It's not possible to directly run these codes on the browser. You will need another third party module to parse/compile the Steem-TX for browser usage!

If you can guide on how to do that, it will be great. Alternatively, if this can be changed to be used independently it will also be great.

I did Javascript the plain old days and not been into front end dev since long, but I will refresh my skills on latest Javascript / UI techs to see, if I can build something using this. Some of the college guys approached me to do some project on blockchain and I think, they will pitch in some interesting ideas, to build and try.

I will try to make it compatible directly with browsers too.
Also, any contribution is appreciated if anyone is willing to help.

When using steemconnect, keychain, or using steemit.com are transactions signed on user’s browser or their servers? In other words do any of these services see the private keys?

No
Everything happens on your browser

Interesting

Posted using Partiko Android

To listen to the audio version of this article click on the play image.

Brought to you by @tts. If you find it useful please consider upvoting this reply.

Nice post gengs.
Hopefully u wanna follow me back,friend😉