Building a Better Bitcoin Fee Market

Jameson Lopp (@lopp)

Over the past year we have seen the fee market for bitcoin transactions evolve at a rapid pace. As transaction volume continues to increase, so does the demand for block space, which remains in limited supply of 1 megabyte (MB) approximately every 10 minutes.

Since we have debated the issue of increasing the supply of block space ad nauseum, this article will focus on the history and current state of bitcoin transaction fees.

The fact is that the pressures resulting from high contention for block space have degraded user experience and thus incentivized bitcoin wallets to make adjustments in order to keep their users happy by ensuring timely confirmation of transactions.

However, we're a long way from operating an optimal fee market.

The History of Transaction Fees

For the first several years of bitcoin’s existence, transaction fees were optional – they were considered a donation to miners.

Bitcoin Core Fee Settings
Wallets paid the same fee on every transaction – defaulting to whatever fee the wallet developer thought was appropriate.

Bitcoin Core's default fee changed several times over the years as the bitcoin exchange rate increased, from 0.01 BTC to 0.0005 BTC to 0.0001 BTC. There were also rules around "priority transactions" that enabled users to send transactions with no fee if the inputs were old and high value enough, though miners have mostly phased those out at this point.

We learned over the years that hard-coded static transaction fees are terrible for several reasons:

It's not the absolute fee that matters to miners, but rather the fee rate per bytes of transaction data. From the miner’s perspective, they only have 1 MB of space into which they want to insert as many transactions as possible in order to collect more fees. As such, a 200 byte transaction with a fee of 0.0001 BTC is preferable to a 1,000 byte transaction with a fee of 0.0001 BTC, because they can insert five of the former and collect five times as much in fees.
From the user’s perspective, if you always set a static fee, you’ll likely eventually create a large data size transaction (due to spending many low value inputs) with a very low fee rate that may never get confirmed.
Wallets with static fees can't adapt to quickly changing market conditions, resulting in users broadcasting transactions that are either overpaying or underpaying. The former won't get transactions confirmed much faster, while the latter will result in long confirmation times because miners pass them over in favor of more profitable transactions to confirm.
The release of Bitcoin 0.3.15 in November 2010 included a change to start calculating fees relative to the transaction’s data size, but not every wallet software followed suit and many users continued blindly setting the same static fee on every transaction. This was generally not a problem until we started bumping up against the max block size, because miners would confirm pretty much any valid transaction that was successfully relayed to them.

As blocks began to fill up in 2015, it became clear that the best practice is to use a dynamic fee algorithm because it can respond to changing conditions on the network.

Bitcoin Core started calculating dynamic fee estimates as of the 0.10 release in February 2015, and Alex Morcos has been steadily improving them since then. Core's fee estimate algorithm is rather complex; you can view its code here and the english explanation here.

A Fee Market Emerges

Antoine Le Calvez, developer of p2sh.info, provides a historical analysis of dynamic and static fees.

Here are the past two years:

Dynamic Transaction Fee Usage
Historic Dynamic Fee Usage, via P2SH.info
We can see significant jumps in dynamic fee usage during the network stress tests and attacks last fall.

However, I suspect that this was not due to normal users switching to dynamic fee wallets, but rather the attackers themselves paying fees that they intentionally set to be higher than the static fees being used by most wallets at the time.

Some of the more sophisticated users adjusted their hard-coded fees during the attacks, but these were likely a small minority of the total transactions. We can also see a jump at the beginning of March 2016 – this is likely partially a result of Blockchain's new wallet deploying dynamic fees about a month earlier.

Rusty Russell performed an excellent analysis of the emerging fee market in December 2015, which showed that more transactions are using dynamically calculated fees, and that the average value of a transaction is increasing as tiny payments are getting priced out of the blockchain.

tx-by-value

AJ Towns followed up on Rusty’s post with more in-depth analysis. He identified eight distinct fee market phases over the course of bitcoin’s history:
Screen Shot 2016-05-05 at 10.23.24 AM

Towns continued his investigation in a second post and came to several conclusions about the effects that the emerging fee market has had upon users.

These were:

A significant number of wallets are dynamically calculating fees, at a per-byte granularity.
Many wallets still don’t calculate fees dynamically, or even calculate fees at a per-byte level.
Market-driven fees will only be able to rise further with increased adoption of wallets that support dynamic fee estimates.
Significantly overpaying the market rate will not get your transaction confirmed any quicker.
There have been two fee events that have impacted wallets with static fees, and a third fee event is coming soon.
Wallets that dynamically calculate fees pay substantially lower fees on average than those that don’t.
I've been tracking Bitcoin Core's fee estimates with Statoshi; here you can see that they have tripled over the past six months as contention for block space continues to increase.

Bitcoin Core Fee Estimates
Bitcoin Core Fee Estimates, via Statoshi.info
Calvez also provides a dashboard with all of the publicly available fee estimate APIs and their historical estimate data:

HIstoric Fee Estimates for various services
Various services' fee estimates, via P2SH.info
Interestingly, it appears that 21, BitGo, and Blocktrail’s fee estimates appear to be the most responsive to changing market conditions while BitPay, Blockchain, and BlockCypher have less volatile estimates.

However, it would be negligent for me to broadly recommend everyone switch to using dynamically calculated transaction fees without first noting the dangers involved.

As the saying goes, every solution leads to new problems. Dynamic transaction fees are no exception.

Dynamic Difficulties

Dynamic fee estimates will never be perfect because they are an attempt to predict the (near) future.

As Danish physicist Niels Bohr once quipped: "prediction is very difficult, especially about the future". If a fee estimate algorithm fails to correctly predict the future state of the fee market then users get stuck in a "fee trap" as noted by Dr Washington Sanchez of OpenBazaar.

This can occur if you broadcast a transaction with a perfectly reasonable fee for the current market conditions, but immediately after doing so, many other higher fee transactions get broadcasted by other users, which essentially push your transaction to the back of miners’ priority queue. The problem is that you can't, without Replace By Fee (RBF), update your "bid" in order to compensate for the new market conditions.

Bugs in fee estimate algorithms also have potential to wreak havoc on the fee market. Take, for example, a recent user error that resulted in someone creating a transaction with a whopping 300 BTC fee. This was bad enough for the unfortunate user who likely fat-fingered the "fee" and "value" amounts into some poorly coded software, but it also had ripple effects:

View image on Twitter
View image on Twitter
Follow
Antoine Le Calvez @khannib
Hum, @BlockCypher may need to review their fee estimation algorithm to exclude outliers. http://p2sh.info/dashboard/db/fee-estimation?panelId=6&fullscreen …
6:34 PM - 26 Apr 2016
5 5 Retweets 6 6 likes
This is an educated guess, but it appears that BlockCypher’s "1 to 2 block target" is using a fee estimate algorithm based upon a weighted moving average of fees from the past two days of blocks. As a result, when the 300 BTC fee transaction was mined, the recent fee average and thus their estimate spiked by 800%.

But this is not meant to pick on BlockCypher, as we have seen many wallets have issues with transaction fees. Not even Bitcoin Core is exempt, as I recently observed an unexpected spike in the fee estimates from Bitcoin Core 0.12.0:

Bitcoin Core Fee Estimate Anomaly
Bitcoin Core Fee Estimate Anomaly, via Statoshi.info
I noticed that Statoshi.info’s "two block target" fee estimate surged from 44 to 112 satoshis per byte on 27th February for no apparent reason. My other bitcoin nodes did not report the same spike.

This could perhaps be explained by differences in mempool transactions since Core’s fee estimate algorithm only uses fees from transactions that were first received as unconfirmed.

It is concerning and is a demonstration of a downside to extrapolating the future based upon your node’s view of the network, which is not guaranteed to be the same as everyone else's.

Macro meltdown

While there are plenty of challenges presented to bitcoin developers on a per-transaction basis, we should also be cognizant of the effects upon the fee market as a whole.

I'm concerned by a couple potential scenarios that could drive fee rates up faster than necessary:

Poorly coded dynamic fee algorithms could create a feedback loop that drives the fee rate up with no ceiling as transactions continue to flood the network with no regard for the mempool backlog. This is not a huge concern if most transactions are being initiated by humans who are then seeing a fee displayed and deciding whether or not it is too high for their tastes. However, if many transactions are being created automatically without any human decision making and the algorithms creating them don’t have any sanity checks on the maximum fee paid, the market could run away until the engineers who wrote those algorithms notice and make changes to them.
Frustrated users (or lazy developers) who are still relying upon hard-coded fees may keep manually updating their wallet settings until their transactions start getting confirmed, likely overpaying significantly, but eventually forcing other static fee users to do the same in order to remain competitive. Dynamic fee users will be pulled upwards as a result. I've already observed the former happening, though not yet to the extent that it has affected dynamic fees significantly.
Follow
Jameson Lopp @lopp
I'm seeing more transactions with high hard coded fees from 0.0005 to 0.001. Overpaying by an order of magnitude! https://tradeblock.com/bitcoin/tx/3f52f946b66c409cc8a27f7bc346305b1272141ef89ea91d046e60fda6d8ff6e …
4:11 PM - 24 Mar 2016
Photo published for TradeBlock
TradeBlock
TradeBlock serves financial institutions with execution and analysis tools that capitalize on the potential of blockchain technologies.
tradeblock.com
4 4 Retweets 4 4 likes
One recent development that hasn’t received much attention despite the fact that it has the potential to affect the fee market is the development of secondary miner fee markets.

This can take the form of special customer incentive programs such as BTCC’s BlockPriority service or it can take the form of private prepaid block space purchases. This spells trouble for any developers who are writing fee estimate algorithms because now there are opaque fee markets that are invisible to the rest of the world.

To quote BTCC’s press release:

"BlockPriority prioritizes all BTCC’s customers’ transactions, including those who pay zero transaction fees."

This should not be a problem for Bitcoin Core’s fee estimate algorithm because it requires 95% of mempool transactions with a given fee rate to be confirmed in X blocks after being seen, but it could affect more naive algorithms. The lesson for developers is that just because you are seeing transactions at a low fee rate of X being confirmed no longer means that it’s safe for your service to broadcast transactions at that fee rate.

It may make economic sense for mining pools to sell prepaid block space contracts because it gives them a new predictable revenue stream.

However, my warning to pools is this: you won’t be able to hide this activity if it becomes a significant portion of your mined transactions. There are plenty of wallet engineers such as myself who are monitoring for this type of behavior, and if it becomes a problem, we will go public. I suspect that any public mining pools found to be participating in this behavior will not fare so well if the individual hashers discover that the pool has been mining lower fee transactions and not sharing the profits from the private block space contracts.

This could result in hashers moving to a pool that they think will be more profitable.

Segregated Witness will also likely have an effect upon the fee market. It will offer a 75% fee discount in an attempt to rebalance the costs of creating versus consuming unspent transaction outputs.

This is expected to encourage users to favor the use of transactions that minimize impact on the UTXO set in order to minimize fees and to encourage developers to design smart contracts and new features in a way that will also minimize the impact on the UTXO set.

David Harding wrote up a helpful breakdown of the data savings provided by SegWit.

Follow
Jameson Lopp @lopp
Unfortunate economics: it costs 5X-10X as much (data size & thus fees) to add an input (↓ UTXO set) vs an output (↑ UTXO set) to a BTC txn.
3:06 PM - 2 Mar 2016
6 6 Retweets 9 9 likes
I think this is the right path to take, though I’m not sure that 75% is the optimal number. It seems to me that the most fair discount would be dynamic and based upon the ratio of the output’s data size to the corresponding data size of the same output when it is spent as an input.

Though this would certainly be much more difficult to implement, if it’s even possible at all.

Moving forward

There are still many wallets and bitcoin services that have not implemented dynamic fees. You know who you are, and your inaction is likely resulting in a poor experience for some of your users. If you haven’t implemented dynamic fees, then every transaction you broadcast fits into one of two categories:

You're overpaying the market rate and not getting confirmed significantly faster.
You're underpaying and transactions are getting stuck, resulting in a poor user experience.
It's highly unlikely that a transaction broadcast with a hard-coded fee is hitting the sweet spot and paying the optimal rate to match current market conditions.

Wallet developers should add safety mechanisms such as sanity check thresholds at the micro level and circuit breakers at the macro level to prevent users from shooting themselves in the foot.

Wallets should set a minimum threshold that is higher than just the minimum relay fee. By observing the minimum fee transactions that make it into blocks, we can clearly see that as of today if you’re paying under five satoshis per byte, you’re going to have a bad time.

Bitcoin Mining Fee Distribution
Bitcoin Mining Fee Distribution, via BitcoinFees
Wallet developers should think adversarially about their fee estimate algorithms and write them to be robust against edge cases that could occur due to mistakes in other wallet software or malicious attacks by entities trying to manipulate the fee market. They should also monitor their fee estimate data and set alerts to notify them if the estimates become more volatile than expected.

But, we should recognize that there is no "one-size-fits-all" fee estimate algorithm – it really depends upon the use case your bitcoin transactions fall under.

Bitcoin Core's algorithm aims to be as generic and conservative as possible so that it's incredibly reliable, but for some use cases it may result in overpaying. Each wallet and service will need to decide what trade off they are willing to make to balance the risk of delayed confirmations against the reward of saving money on fees.

As such, I encourage all bitcoin wallets to make their fee estimates accessible via public APIs. I suspect that the fee estimate algorithms themselves will become a point of competition and often remain closed source, but it the estimates are public then we can more easily watch for abnormal activity. Perhaps someone will even build a "Bitcoin Average" aggregator for fee estimates!

We should recognize that the "fee market" isn’t exactly a market in the traditional sense of the word.

Users can place a "bid" by broadcasting a transaction, but miners don’t publicize their "asks" – wallets have to guess based upon the transactions miners have recently confirmed and by what transactions are currently waiting to be confirmed.

Konrad S Graf stated his thoughts on the "fee market" recently:

"Fees are paid; products and services are bought. So, this term already obscures the real product. Users submit transactions with a fee as an open bid in hopes of confirmation. I describe this as a market for transaction-inclusion services. Users bid to have miners include transactions in candidate blocks. Inclusion in more candidate blocks—especially in relation to the total hashrate mining for those candidates – raises odds of quicker confirmation. Users prefer quicker confirmation to slower, other things equal, so the time element of scarcity is key. It is a market for confirmation priority, a time market."

Perhaps if mining pools published public APIs with their mining policies and "going rates", then it could help wallet services make more informed decisions when calculating dynamic fees rather than forcing developers to rely upon guesswork. This could also help alleviate problems caused by miners who create opaque secondary fee markets via private block space contracts.

Bitcoin users should not need to keep track of the current state of block contention.

Wallets should be handling the complexities of the fee market under the covers, giving the user several simple options when sending a transaction. One potential route would be for the user to authorize a max fee depending upon the transaction urgency and have the wallet use RBF to ramp up the fee paid after each block that passes without it being confirmed.

Transaction fee values should be displayed to the user in terms of their preferred unit of account, such as dollars. It would probably also make sense to display the fee in terms of a percentage of the transaction's value if it exceeds a certain threshold, such as 1%. This will make it easier for users to decide if the current state of the fee market is too contentious and that they would prefer to delay making the transaction until they can safely do so at a lower fee rate.

When we are discussing transaction fees, bitcoin users should stop saying that they paid "X cents" or "Y bits" in transaction fees because this type of statement is nonsensical without knowing the size of the transaction. We should standardize using fee rates in discussions, preferably denominated in terms of satoshis per byte. It’s simpler for humans to write and keep track of "20 satoshis per byte" in comparison to "20,000 satoshis per kilobyte".

The evolution of bitcoin's transaction fee market has been a rocky road up to this point.

At first fees were set manually by developers, then manually by users, and now we're at a more chaotic and potentially dangerous point where developers are playing a more active role in steering the economics of this emerging market.

Kristov Atlas eloquently noted the risks we take by centrally planning economic changes to bitcoin without fully studying them; developers should keep this in mind when writing fee estimate algorithms. We should strive to ensure that the fee market remains driven by humans with the aid of machines, not the other way around. Wallet developers must be careful when building their fee logic so that we can provide a smooth user experience without taking away users' freedom of choice, which is necessary for maintaining a functioning fee market.

Image via Dan Nott for CoinDesk