Steemit is bad at connecting users to relevant content - We need Smart Feeds

It's a known fact to all Steemers that here we are having a hard time to find the content we like. We have some basic tools like tags and following to keep track of the content we are interested in but we are still missing tones of good articles and see a lot of crappy ones.

Facing the problem

The reason is that Steemers rarely keep posting about only one category of content with their account. Someone may write an article about the benefits of a multi-sig wallet in Bitcoin and the next day send a cute photo of his dog. Reach to non relevant content kills attraction and weaken the chances of users coming back into platform.

Also if we rely on following we are missing tones of articles that are written by new authors that we still do not follow. So times to times we should go to latest articles section and after ignoring dozens of article we may find a new author we might like (or do we?).

Tags are also killing us in another way. We like them because they categorize content but there is no sorting based on the quality or relevance of article. Hot and Trending section are mostly showing us authors with more popularity or friends in Steem platform. We already know that best articles in Steemit are not necessarily the trending articles. That's fine but we still need to reach to good articles to upvote them right? How to make good articles trending while we cannot reach to good articles at all?

All of this aside, recent growth is making it look like an emergency as new users are having a hard time to reach to their audience since they are simply ignored in the new tab due to the high volume of posts in the platform.

Please not like Facebook

We dislike how corporate tech companies like Facebook or Twitter make us to only see what we like to see. I agree, that's BS. We don't want to get under controlled media again but we want to have better reachability to content we are looking for. That's why we need a different model of personalized feed which probably is not implemented yet. We need Optional Personalized Feed.

How it should look like?

We can have two optional button on top of homepage so users can have the freedom to use the chronological timeline or smart timeline based on their interests. The settings of Smart timeline can appear the first time users click on it and can be accessible on settings menu.

How it could be?

1. Should be discretionary (optional)

Users should not be obliged to use this feature at all or in any way this feature should not replace the chronological feed for users.

2. Should be simply configurable

Users should be able to configure it in a clear and concise manner. A JSON configurations string (which can be generated for less advanced users by any UI) seems to be a good option.

3. The algorithm should be open source

The algorithm to generate the feed based on the configuration should be open source and well documented so it can be reviewed by anyone. Developers of other apps (like Busy or eSteem) could have the freedom to fork the algorithm and implement their own smart feed based on the same pre-defined configuration standard (in case of implementing it off-chain).

4. The dataset should be protected by encryption and be stored on user's device

The dataset of user's previous engagement and interactions, likes, dislikes, reading time etc should be stored and used by machine learning algorithms to make a real personalized experience for users. In companies like Facebook this dataset is usually stored at centralized databases and used for other commercial purposes like target marketing. It's crucial that users should be able to protect their privacy and reading habits by encrypting this data using their posting WIF and store it on their own devices.
Storing datasets on personal devices has many interoperability and synchronization issues that could be solved by providing centralized tools to access this data. For example UI platforms like Steemit, eSteem or Busy can provide cloud features to store the encrypted data for users in a centralized manner (without having access to keys to decrypt them) but it should be optional for users.

How it should be implemented? (technical)

There are two stages of making a smart and personalized feed. The first stage is to ask user what they want to see? For example users can indicated they want to see posts from a small list of users at first. They want to see posts related to books and politics. These are relatively simple but very important to implement. But the second stage is to gather information about what users actually like by observing their interactions on user interface. If they vote, read or ignore a post, the features of the post should be stored and flagged with the behavior so it can later be used in a classification machine learning algorithm to prioritize the feed for users.

Here I am suggesting some solutions which I think all have their own pros and cons. It would be very good if you can also share your own thoughts about this as I believe non of my ideas are perfect at all.

Off chain, solution 1:

We can keep the blockchain neutral and have user interfaces implement their own version of algorithm. The only problem is that it needs many centralized indexing applications to provide the feed off-chain and the centralized apps should receive the decrypted info of users's engagement data-sets to generate the data so you should trust the centralized app is not storing and using your private data later.

Off-chain, solution 2:

Machine learning algorithms can be implemented on front-end also. Your browser can handle the whole process of storing the engagement data and use it to show you relevant content. However it cannot index data and fetch content in the way you need it because it doesn't have access to the whole database of content to pick the articles you really like. Even so, it can change the sorting and emphasize the content you are interested in and ignore those you probably are not in the normal chronological feed.

This wireframe can be a good way to show how different it is:

On-chain

First of all, I'm not the right person to propose chain features since I'm honestly bad in it. However, I assume more knowledgable people can come up with a way to have a neutral smart-feed configuration on chain per users. Obviously we cannot store the whole user engagement data on-chain because it's heavy and expensive. However, users can address their datastore along with a hash to their smart feed config json to make it safely accessible to steem's blockchain to use it.

Having an embedded on-chain feature for smart-feeds has a huge advantage since there is no need for any centralized server to index data, and in the same time the chain has full access to chain and can provide feed APIs for all the apps. It omits the flexibility of other Steem UI apps (Busy, eSteem etc) to make their own algorithm of smart feeds, though make it very easy to implement on every platform for users.

Conclusion

Smart and personalized feeds are proved to increase the engagement for users and save their time significantly on internet. For example Google is a simple example of smart and personalized feed. Even so, they are not much popular these days because the algorithms are kept as secret and the data are used for other purposes outside the user's control. I believe we need it for Steemit as soon as possible to move the platform to the next level and I hope we can figure out how to implement this feature with a high respect to privacy and use it as a powerful tool to leverage the connectivity of platform for new users.

If you agree with this feature, please re-esteem so it would be noticed more. Thank you.