> ## Content Index
> Fetch the complete content index at: https://blog.borodutch.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# A button beat a line of text six to one: what reply promos in my Telegram bots measured in ten days
- URL: https://blog.borodutch.com/telegram-bot-reply-promos-button-vs-line/
- Published: 2026-11-07T16:00:00.000Z
- Updated: 2026-11-07T16:00:00.000Z
- Description: Across 784 people shown a promo in my Telegram bots, 10.8% opened the other bot from a button and 1.8% from a caption line, and the safety rule that stopped three tests was right only once.
- Author: Nikita Kolmogorov
- Tags: Telegram, Bots, Experiments, #radiant

A button under a bot's reply brought people to my other bots at six times the rate of a line of text saying the same thing. Across 784 people shown a promo in my Telegram bots between 28 October and 7 November 2026, 43 of 397 who got a button opened the other bot (10.8%), against 7 of 387 who got a caption line (1.8%). I did not expect the gap to be that wide, and the way the safety rule behaved during the tests taught me more than the winner did.

The bots are [Voicy, which turns voice messages into text](https://voicybot.com/?ref=blog.borodutch.com), [Jev Antispam](https://jev.borodutch.com/?ref=blog.borodutch.com), [Randy](https://randy.borodutch.com/?ref=blog.borodutch.com), [Banofbot](https://banofbot.borodutch.com/?ref=blog.borodutch.com), Temply and [VideoDownloadBot](https://videodl.borodutch.com/?ref=blog.borodutch.com). All numbers below come from the reports my own tooling printed on 7 November 2026 (`python3 bin/promos.py report` and the log of each experiment).

## How a reply promo test is set up

Every bot carries the same small hook. When a bot is about to answer a person in a private chat, the hook hashes the campaign name together with the Telegram user id and uses the result to put that person in one of three arms: a caption line added to the reply, a URL button under the reply, or a holdout that never sees anything. The same person always lands in the same arm, so nobody sees a promo, loses it and sees it again. Seven bots have the hook live today.

The rules were the same in every test. Private chats only, one promo per reply, at most one per person every 14 days, and every text says it comes from the same maker. The line reads something like "From the makers of this bot: Jev Antispam quietly deletes spam in your groups". The button says "Jev: antispam for groups". Both point to a `t.me/<bot>?start=<campaign>-<variant>-<host code>` deep link, so Telegram opens the other bot directly and that bot can tell which campaign and arm sent the person.

Three numbers matter. A **shown** person had the promo rendered in a reply. An **arrival** is a shown person who then pressed Start on the target bot with that payload. A **conversion** is an arrival who then actually used the target bot within seven days. The holdout is there for one reason: it tells me how many people would have shown up anyway, and how many would have blocked the bot anyway.

## What the numbers say: the button wins, by a lot

Seven tests had shown both a line and a button to anyone, and I also ran one by hand (VideoDownloadBot pointing to Voicy). Pooled:

| Arm          | Shown    | Arrivals | Rate  |
| ------------ | -------- | -------- | ----- |
| Caption line | 387      | 7        | 1.8%  |
| URL button   | 397      | 43       | 10.8% |
| Holdout      | no promo | 0        | 0%    |

Fifty arrivals in total, and six conversions: four of them came from one button test, two from caption lines. The gap is not a fluke of pooling. Inside every test the arms were assigned by hash, so each test compares like with like, and a one-sided exact test on the pooled counts gives a p-value below 0.0001.

The three tests hosted by Voicy, which has by far the most traffic, show it best:

| Test                                              | Line    | Button   |
| ------------------------------------------------- | ------- | -------- |
| Voicy to VideoDownloadBot, "free" wording         | 2 of 60 | 13 of 42 |
| Voicy to VideoDownloadBot, "what it does" wording | 2 of 54 | 4 of 50  |
| Voicy to Jev Antispam                             | 1 of 36 | 15 of 68 |

Together that is 5 arrivals from 150 people shown a line and 32 from 160 shown a button. The reverse direction agrees: VideoDownloadBot pointing to Voicy got 1 arrival from 175 line impressions and 5 from 173 button impressions. In the five smaller bots (Banofbot, Jev, Randy, Temply, Todorant) the samples are tiny, 1 of 61 for the line and 6 of 64 for the button, and one test, Banofbot to Jev, produced no arrivals at all from 45 people shown either version. I would not read anything into those yet.

My best guess for why a button wins is boring: it is a thing you can press, it sits where your thumb already is, and a caption line is one more sentence in a reply you came for. I have not tested that guess, so treat it as a guess. What I did test is that the quiet version, the line, is almost indistinguishable from nothing.

## The safety rule fired three times, and it was right once

Each test has stop rules written down before it starts. The one that matters here watches new blocks: it counts people who blocked the bot after being counted in a test, in treatment and in the holdout, and stops the test and emails me if treatment is meaningfully worse. The first version of the rule had two clauses. The first stopped a test when treatment's block rate exceeded the holdout's by one point and a one-sided test said p below 0.1, after 100 people. The second stopped it when treatment's block rate reached 3%, no questions asked.

That second clause was a mistake for Voicy, and I only saw it on the first test. About a quarter of the Voicy users in a test block the bot during the test window whether or not they ever saw a promo: in the Voicy-to-Jev test the holdout had 264 blocks among 1,121 people (23.6%). I have not worked out why. A 3% ceiling on a number that sits at 24 percent stops everything, every time.

- **31 October, Voicy to VideoDownloadBot:** the test was stopped with 21.6% blocks in treatment (22 of 102). The holdout was at 23% (208 of 905). Treatment was actually lower, so this was a false alarm. I changed the clause to also require treatment to be a point above the holdout.
- **3 November, Voicy to VideoDownloadBot again, with new wording:** stopped with 35.6% blocks in treatment (37 of 104) against 26.2% in the holdout. A gap of nine points on a sample that size is about two standard errors, and I treated it as real. The test ended there. It brought 6 arrivals from 104 people, so the traffic did not justify finding out.
- **7 November, Voicy to Jev Antispam:** stopped at 25.2% (26 of 103) against 23.6%. That is 1.6 points over the holdout, enough to trip the "three percent and one point over" clause, and well inside noise for 103 people. The button had just got 15 arrivals from 68 people. Another false alarm.

So the rule did what I wanted exactly once, the second time, and the other two stops cost me a few days of data each. The fix I take from this is not a looser number. It is that the outright clause needs the same significance check as the first one, because a safety rule with an absolute threshold only works when the baseline is far below it. The code does not check significance on that clause yet, so until it does, a Voicy-hosted test can stop early on noise.

## What I changed because of this

Caption lines are retired. New tests are button against button, with two wordings or two targets, because a test that includes the line spends 5% of a bot's users on an arm I already know is weak. The first of those, Voicy to Randy, started on 7 November, with a button and a 95% holdout, and has shown a promo to one person so far.

The holdout stays large: 80 to 90% of a bot's users in these tests, and never below 50% as a rule. Without it I would not know that a quarter of Voicy's test users block the bot on their own, and I could not have told the 31 October stop (treatment no worse than the holdout) from the 3 November one (nine points worse).

## What you can copy

- Use a button, not a sentence. If your bot's reply already carries a keyboard, adding a button costs one line of code.
- Keep a holdout that never sees the promo, and compare blocks against the holdout, not against zero.
- Measure the baseline before you write a stop rule. If your normal block rate is 24%, a rule that says "stop at 3%" is not a safety rule, it is an off switch.
- Count arrivals and the use that follows separately. Fifty people pressed Start and six used the other bot within a week; that second number is the one that pays for the effort.

The tests are still running, one after another, and I will write down the next result when a button wins or loses against another button. If you want to see what these bots do before the promo shows up, [Voicy](https://voicybot.com/?ref=blog.borodutch.com) is the one with the most traffic, and [VideoDownloadBot](https://t.me/videodownloadbot?ref=blog.borodutch.com) is the one that most Voicy users were pointed to.

*Numbers read on 7 November 2026 from the bots' own event reports; arrivals are people who pressed Start on the target bot, conversions are people who used it within seven days. Tests started between 28 October and 7 November, so older ones have had more time to collect results.*