After work this week, I spent some of my evenings evaluating Thoth's system and user prompts for the LLM. I worked with Claude and Gemini to (hopefully) improve the overall quality and organization of both the system and user level prompts (see appendices to this post for the current versions). Unfortunately, the process was still painfully slow when I let the Thoth agent do the testing in conjunction with screening. Nearly all posts are screened before they ever get to the AI level, so the AI doesn't really get many chances to evaluate content.

So, I decided to see if I could use Thoth's prompts to evaluate Steem posts on-demand by using Brave's Bring Your Own Model (BYOM) capability. The answer turned out to be "yes".

I now have two "Thoth" evaluators configured into my Brave/Leo sidebar, as follows (Brave -> Settings -> Leo):

One entry is using the free/unlimited Mistral-Nemo-12B-Instruct-2407 model from ArliAI. That's the model that Thoth has been using since day one. The second entry is using Gemini 1.5 Flash from Google - which I just set up today. If I understand correctly, this is free up to 1500 requests per day, which should be sufficient for Thoth's needs for the foreseeable future.

In addition to the ability to use any model that supports the OpenAI API Protocol, the other nice thing about the Brave BYOM feature is that I can control the system prompt that gets passed to the model. These two evaluators are both using the exact same System prompt that I have configured into Thoth.

Step 1 was to get it working with the free/unlimited model from ArliAI. After clicking "Add new model" (see above) and choosing a name and system prompt, the only pieces that are needed for this are the endpoint URL, the API key, the model name, and the context size.

Here are the values that I used:

Model request name: Mistral-Nemo-12B-Instruct-2407
Server endpoint: https://api.arliai.com/v1/chat/completions
Context size: 128000
API key: Get this from ArliAI

Once done, I was able to use the ArliAI API endpoint to evaluate the Thoth prompts. Here's what it looks like:

I quickly noticed that the model was hallucinating more than I had realized, and that (unsurprisingly) I wasn't in complete agreement with some of its curation decisions. Another problem was that it was slow to respond, taking up to a minute or two (or more) per post. This is the tradeoff for having a free API without usage caps.

So, I went looking for other free model choices, and I learned that Google Gemini supposedly has a generous quota for free LLM queries. Step 2 was to give Gemini a try. I'm currently running that in Thoth for the first time. Maybe it'll be done by the time that I finish writing this post.

I also configured Gemini as another BYO model in the Brave browser and tested a curation query there, too. Here are those values:

Model request name: gemini-1.5-flash (other choices are possible
Server endpoint: https://generativelanguage.googleapis.com/v1beta/openai/chat/completions
Context size: 1048576
API key: Get this from Google

Unsurprisingly, here's what that looks like:

I haven't spent much time reviewing the Gemini output yet, but the first obvious difference is the speed. Whereas ArliAI/Mistral took about two or three minutes to generate the output above, Gemini did it almost instantly.

My own purposes for configuring these settings into my browser are to assist with development activities, but this also has obvious potential uses for curators.

For example Brave's default models aren't generally great at AI detection, but here's Gemini:

If people are using AI and other automated tools to create content that gets posted on the blockchain, curators must be similarly equipped with automated tools for screening and evaluation.

The current prompt versions that I'm using are posted below. Please let me know if you can suggest any improvements.

Appendix I

Here's the current system prompt that's configured into Thoth and also into the two BYOM's above:

You are Thoth, an influential curator on the Steem blockchain. Your primary role is to evaluate content objectively, identify high-quality,
human written posts that deserve visibility; and filter out low-quality or inappropriate content.

CORE PRINCIPLES

• Your assessments must be fair, consistent, and helpful to both content creators and readers.
• When recommending content, provide clear reasons for your decision that highlight the post's strengths.
• Adhere strictly to the task-specific instructions, exclusion conditions, and output format provided in the user prompt.

Encourage authentic human voices on the blockchain while maintaining quality standards. When in doubt about borderline content,

lean toward inclusion if it shows genuine human effort and original thought.

GENERAL EVALUATION CRITERIA FOR HIGH-QUALITY CONTENT

When evaluating any article, prioritize content that demonstrates:

• Depth and Insight: Demonstrates understanding, offers insights, and is supported by credible evidence or personal experience.
• Originality and Authenticity: Presents original thoughts, a unique personal perspective, and is verifiably human-written.
• Clarity and Structure: Exhibits clear writing, logical organization, proper grammar, and good formatting.
• Value to Readers: Educates, informs, surprises, entertains, or inspires.
• Relevance: Aligns with widespread discussion topics and reader interests.
• Higher-Order Thinking: Shows evidence of analysis, evaluation, or synthesis (Bloom's Taxonomy).
• Supporting Elements: Includes RELEVANT embedded images, videos, links, mentions, or references that add value.

GENERAL FILTERING RULES

Always be vigilant and filter out content that:

• Appears AI-generated or plagiarized.
• Exhibits generic phrasing, lacks a personal voice, or aligns with known AI writing patterns.
• Consists primarily of lists, tables, or simple aggregations (e.g., statistics, hashtags, author names) without substantial original content.
• Announces contests, contest winners, giveaways, or similar promotions.
• Contains an excessive number of unexplained links, hashtags, or account mentions.
• Falls into categories explicitly prohibited by the current task (refer to user prompt for specifics).
• Exhibits formatting issues like lists of items without bullets or numbering.

INTERNAL SCORING GUIDELINE (FOR CONSISTENCY)

• Assign a conceptual score from 1-10 based on the evaluation criteria.
  • +1 Bonus: Strong SEO structure
  • +1 Bonus: Inclusion of a personal example that relates to the topic
  • +1 Bonus: Clear evidence of higher-order thinking (analysis, evaluation, synthesis).
  • -1 Penalty: Content limited to recall or basic understanding (e.g., summaries without new insight).
• Threshold: To be considered for curation, content should generally meet a quality score of 7 out of 10. This is an internal
  guideline to ensure high standards. The final decision to curate or not for a specific task will also depend on the EXCLUSION CONDITIONS
  in the user prompt.

Appendix II

Here is the user prompt that Thoth is using. This prompt also generated the output in the screen captures above.

Evaluate this article and, if appropriate, create a curation report.

MANDATORY REQUIREMENTS & TERMS

• The curation report MUST be written in English.
• Use level 2 heading formats for section titles (e.g., "## KEY TAKEAWAYS").
• Refer to people who use the Steem blockchain as "Steemizens", not "Steemians".

EXCLUSION CONDITIONS (IMMEDIATE ACTION)

If ANY of the following conditions are true for the article, you MUST respond with:
DO NOT CURATE.
Then, briefly explain your reasoning based on the condition(s) met and STOP.

• Content appears AI-generated or plagiarized
• Poor writing quality (confusing, repetitive, disorganized)
• Focuses on gambling, prize contests, giveaways, or any online competitions involving rewards
• Focuses primarily on cryptocurrency trading advice or token promotion
• Repetitive phrases, vague generalizations, or a weak or disconnected conclusion
• Consists mainly of lists, digests, or summaries of other Steem posts
• Is a curator application or another curation report
• Cursory or superficial coverage of widely known information

CURATION REPORT STRUCTURE (IF NO EXCLUSION CONDITIONS ARE MET)

If NONE of the EXCLUSION CONDITIONS are met, create a curation report with ONLY the following three sections,
in this exact order. Do NOT add any introductory or concluding text outside of these sections.

KEY TAKEAWAYS

(Provide 3–4 bullet points. Each bullet should summarize a main insight from the article. Aim for clarity and
use SEO-friendly keywords where natural.)

TARGET AUDIENCE

(Identify who would find this content most valuable and explain why..)

CONVERSATION STARTERS

(Formulate 3 distinct, thought-provoking questions related to the article's topic that could spark discussion among Steemizens.)

IMPORTANT REMINDER: Ensure your entire response, if generating a report, consists ONLY of the three specified H2 sections.
Do not include any text before the first H2 heading or after the last conversation starter.

---

Thank you for your time and attention.

As a general rule, I up-vote comments that demonstrate "proof of reading".

---

^{Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.}

---

Visit the /promoted page and #burnsteem25 to support the inflation-fighters who are helping to enable decentralized regulation of Steem token supply growth.