Distest¶
Distest makes it easy to write application tests for discord bots.
Distest uses a secondary bot to send commands to your bot and ensure that it responds as expected.
See the interface reference for a list of assertions this library is capable of.
Note
Two quick note about recent changes:
- You NEED to enable the
members
intent on the tester bot. For more information, see Member Intent- If you’re using the
ext.commands.Bot
system, you will need to patch yourBot
to allow it to listen to other discord bots, as usually commands ignore other bots. This is really easy, we provide the patching function, just take a look at the patching documentation page.
Quickstart¶
Installation¶
- Install the library with pip:
$ pip install distest
Distest works by using a second bot (the ‘tester’) to assert that your bot (the ‘target’) reacts to events appropriately. This means you will need to create a second bot account through the Discord Developer’s Portal and obtain the authorization token. You also have to invite the tester to your discord guild. Additionally, be sure to enable the
SERVER MEMBERS INTENT
option, see Member Intent docs for more info.Refer to the Example Test Suite for the syntax/function calls necessary to build your suite.
Usage¶
The tests can be run in one of two modes: interactive and command-line. In interactive mode, the bot will wait for you to initiate tests manually. In command-line mode, the bot will join a designated channel, run all designated tests, and exit with a code of 0 if all tests were successful and any other number if the one or more tests failed. This allows for automating your test suite, allowing you to implement Continuous Integration on your Discord bot!
No matter how you run your tester, the file must contain:
- A call to
run_dtest_bot
, which will handle all command line arguments and run the tester in the correct mode - A
TestCollector
, which will let the bot find and run the you specify - One or more
Test
, which should be decorated with theTestCollector
, and are the actual tests that are run.
Note
The error codes will currently be 0 on success or 1 on failure, but we plan to implement meaningful error codes
Interactive Mode¶
- Run the bot by running your test suite module directly (called example_tester.py here):
$ python example_tester.py TARGET_ID TESTER_TOKEN
Go to the channel you want to run your tests in and call the bot using the
::run
command. You can either designate specific tests to run by name or use::run all
See also
::help
command for more commands/options.
Command-Line Mode¶
For command-line you have to designate the ID of the channel you want to run tests in (preceded by the -c
flag). You must also designate which
tests to run (with the -r
flag). Your command should look something like this:
$ python example_tester.py TARGET_ID TESTER_TOKEN -c CHANNEL_ID -r all
The program will print test names to the console as it runs them, and then exit.
See also
readme.md on GitHub, which contains a more in-depth look at the command properties
Example Test Suite¶
This is the example_tester.py
file found in the root directory. It contains tests for every assertion in Interface. This suite is also used to test our library, in conjunction with the example_target.py
.
The easiest way to get started is to adapt this suite of tests so it’s specific to your bot, then run this module with
$ python example_tester.py ${TARGET_ID} ${TESTER_TOKEN}
where TARGET_ID
is the Discord ID of your bot, and TESTER_TOKEN
is the auth token for your testing bot.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | """
A functional demo of all possible test cases. This is the format you will want to use with your testing bot.
Run with:
python example_tests.py TARGET_NAME TESTER_TOKEN
"""
import asyncio
import sys
from distest import TestCollector
from distest import run_dtest_bot
from discord import Embed, Member, Status
from distest import TestInterface
# The tests themselves
test_collector = TestCollector()
created_channel = None
@test_collector()
async def test_ping(interface):
await interface.assert_reply_contains("ping?", "pong!")
@test_collector()
async def test_delayed_reply(interface):
message = await interface.send_message(
"Say some stuff, but at 4 seconds, say 'yeet'"
)
await interface.get_delayed_reply(5, interface.assert_message_equals, "yeet")
@test_collector()
async def test_reaction(interface):
await interface.assert_reaction_equals("React with \u2714 please!", u"\u2714")
@test_collector()
async def test_reply_equals(interface):
await interface.assert_reply_equals("Please say 'epic!'", "epic!")
@test_collector()
async def test_channel_create(interface):
await interface.send_message("Create a tc called yeet")
created_channel = await interface.assert_guild_channel_created("yeet")
# @test_collector
# async def test_pin_in_channel(interface):
# await interface.send_message("Pin 'this is cool' in yeet")
# await interface.assert_guild_channel_pin_content_equals(created_channel )
@test_collector()
async def test_channel_delete(interface):
await interface.send_message("Delete that TC bro!")
await interface.assert_guild_channel_deleted("yeet")
@test_collector()
async def test_silence(interface):
await interface.send_message("Shhhhh...")
await interface.ensure_silence()
@test_collector()
async def test_reply_contains(interface):
await interface.assert_reply_contains(
"Say something containing 'gamer' please!", "gamer"
)
@test_collector()
async def test_reply_matches(interface):
await interface.assert_reply_matches(
"Say something matching the regex `[0-9]{1,3}`", r"[0-9]{1,3}"
)
@test_collector()
async def test_ask_human(interface):
await interface.ask_human("Click the Check!")
@test_collector()
async def test_embed_matches(interface):
embed = (
Embed(
title="This is a test!",
description="Descriptive",
url="http://www.example.com",
color=0x00FFCC,
)
.set_author(name="Author")
.set_thumbnail(
url="https://upload.wikimedia.org/wikipedia/commons/4/40/Test_Example_%28cropped%29.jpg"
)
.set_image(
url="https://upload.wikimedia.org/wikipedia/commons/4/40/Test_Example_%28cropped%29.jpg"
)
)
# This image is in WikiMedia Public Domain
await interface.assert_reply_embed_equals("Test the Embed!", embed)
@test_collector()
async def test_embed_part_matches(interface):
embed = Embed(title="Testing Title.", description="Wrong Description")
await interface.assert_reply_embed_equals(
"Test the Part Embed!", embed, attributes_to_check=["title"]
)
@test_collector()
async def test_reply_has_image(interface):
await interface.assert_reply_has_image("Post something with an image!")
@test_collector()
async def test_reply_on_edit(interface):
message = await interface.send_message("Say 'Yeah, that cool!'")
await asyncio.sleep(1)
await interface.edit_message(message, "Say 'Yeah, that is cool!'")
await interface.assert_message_contains(message, "Yeah, that is cool!")
@test_collector()
async def test_send_message_in_channel(interface):
message = await interface.send_message("Say stuff in another channel")
await interface.wait_for_message_in_channel("here is a message in another channel", 694397509958893640)
# Actually run the bot
if __name__ == "__main__":
run_dtest_bot(sys.argv, test_collector)
|
The Member Intent¶
Discord recently changed what you have to do for bots to be able to access server member informaiton, meaning that without changes, calling guild.members
will return an empty list, which is no good!!
To fix this, we need to do two things:
- Enable the Privileged Gateway Intent for Server Members.
- To do this. go to the Discord developer portal and select your tester bot
- Select the bot tab
- Enable the
SERVER MEMBERS INTENT
and thePRESENCE INTENT`
sliders
- Update distest! There are also changes that need to be made on our side. They have been made, but be sure you update to 0.4.9 or newer to get the changes!
Now, you should be good to go. Have fun testing!
Quick note - For some godforsaken reason, the on_member_update
event is just horribly slow and unreliable. I’m not really sure what to do about this, but be forewarned if you want to use it!
Main Functions¶
-
distest.
run_dtest_bot
(sysargs, test_collector, timeout=5)[source]¶ This is the function you will call in your test suite’s
if __name__ == "__main__":
statement to get the bot started.Parameters: - sysargs (list) – The list returned by
sys.argv
, this function parses it and will handle errors in format - test_collector (TestCollector) – The Collector that has been used to decorate the tests
- timeout (int) – An optional parameter to override the amount of time to wait for responses before failing tests. Defaults to 5 seconds.
- sysargs (list) – The list returned by
-
distest.
run_command_line_bot
(target, token, tests, channel_id, stats, collector, timeout)[source]¶ Start the bot in command-line mode. The program will exit 1 if any of the tests failed.
Relies on
run_dtest_bot()
to parse the command line arguments and pass them here. Not really meant to be called by the user.Parameters: - target (str) – The display name of the bot we are testing.
- token (str) – The tester’s token, used to log in.
- tests (str) – List of tests to run.
- channel_id (int) – The ID of the channel in which to run the tests.
- stats (bool) – Determines whether or not to display stats after run.
- collector (TestCollector) – The collector that gathered our tests.
- timeout (int) – The amount of time to wait for responses before failing tests.
-
distest.
run_interactive_bot
(target_name, token, test_collector, timeout=5)[source]¶ Run the bot in interactive mode.
Relies on
run_dtest_bot()
to parse the command line arguments and pass them here. Not really meant to be called by the user.Parameters: - target_name (str) – The display name of the bot we are testing.
- token (str) – The tester’s token, used to log in.
- test_collector (TestCollector) – The collector that gathered our tests.
- timeout (int) – The amount of time to wait for responses before failing tests.
Interface¶
This is the most important class in the library for you, as it contains all the assertions and tools you need to interface with the library. Generally broken down into a few overall types:
- Message (i.e.
assert_message_contains
): Does not send it’s own message, so it require aMessage
to be passed in. - Reply (i.e.
assert_reply_contains
): Sends a message containing the text in contents and analyzes messages sent after that. - Use
get_delayed_reply
to wait an amount of time before checking for a reply
- Use
- Reply (i.e.
- Embed (i.e.
assert_embed_equals
): Sends a message then checks the embed of the response against a list of attributes - Other Tests (i.e.
ask_human
): Some tests do weird things and don’t have a clear category. - Interface Functions (i.e.
connect
,send_message
): Help other tests but also can be useful in making custom tests out of the other tests.
-
class
distest.
TestInterface
(client, channel, target)[source]¶ All the tests, and some supporting functions. Tests are designed to be run by the tester and mixed together in order to actually test the bot.
Note
In addition to the tests failing due to their own reasons, all tests will also fail if they timeout. This period is specified when the bot is run.
Note
Some functions (
send_message
andedit_message
) are helper functions rather than tests and serve to bring some of the functionality of the discord library onto the same level as the tests.Note
assert_reply_*
tests will send a message with the passed content, whileassert_message_*
tests require aMessage
to be passed to them. This allows for more flexibility when you need it and an easier option when you don’t.Parameters: - client (DiscordCliInterface) – The discord client of the tester.
- channel (discord.TextChannel) – The discord channel in which to run the tests.
- target (discord.Member) – The bot we’re testing.
-
ask_human
(query)¶ Ask a human for an opinion on a question using reactions.
Currently, only yes-no questions are supported. If the human answers ‘no’, the test will be failed. Do not use if avoidable, since this test is not really automateable. Will fail if the reaction is wrong or takes too long to arrive
Parameters: query (str) – The question for the human. Raises: HumanResponseTimeout, HumanResponseFailure
-
static
assert_embed_equals
(message: discord.message.Message, matches: discord.embeds.Embed, attributes_to_prove: list = None)¶ If
matches
doesn’t match the embed ofmessage
, fail the test.Checks only the attributes from
attributes_to_prove
.Parameters: - message – original message
- matches – embed object to compare to
- attributes_to_prove – a string list with the attributes of the embed, which are to compare This are all the Attributes you can prove: “title”, “description”, “url”, “color”, “author”, “video”, “image” and “thumbnail”.
Returns: message
Return type:
-
assert_guild_channel_created
(channel_name, timeout=None)¶ Assert that the next channel created matches the name given
Parameters: Returns: The channel that was created
Return type: Raises: NoResponseError
-
assert_guild_channel_deleted
(channel_name, timeout=None)¶ Assert that the next channel deleted matches the name given
TODO: check what the deleted channel actually returns
Parameters: Returns: The channel that was deleted
Return type: Raises: NoResponseError
-
static
assert_message_contains
(message, substring)¶ If message does not contain the given substring, fail the test.
Parameters: - message (discord.Message) – The message to test.
- substring (str) – The string to test message against.
Returns: message
Return type: Raises: ResponseDidNotMatchError
-
static
assert_message_equals
(message, matches)¶ If
message
does not match a string exactly, fail the test.Parameters: - message (discord.Message) – The message to test.
- matches (str) – The string to test message against.
Returns: message
Return type: Raises: ResponseDidNotMatchError
-
static
assert_message_has_image
(message)¶ Assert message has an attachment. If not, fail the test.
Parameters: message (discord.Message) – The message to test. Returns: message Return type: discord.Message Raises: UnexpectedResponseError
-
static
assert_message_matches
(message, regex)¶ If message does not match a regex, fail the test.
Requires a properly formatted Python regex ready to be used in the
re
functions.Parameters: - message (discord.Message) – The message to test.
- regex (str) – The regular expression to test message against.
Returns: message
Return type: Raises: ResponseDidNotMatchError
-
assert_reaction_equals
(contents, emoji)¶ Send a message and ensure that the reaction is equal to emoji. If not, fail the test.
Parameters: - contents (str) – The content of the trigger message. (A command)
- emoji (discord.Emoji) – The emoji that the reaction must equal.
Returns: The resultant reaction object.
Return type: Raises: ReactionDidNotMatchError
-
assert_reply_contains
(contents, substring)¶ Send a message and wait for a response. If the response does not contain the given substring, fail the test.
Parameters: Returns: The reply.
Return type: Raises: ResponseDidNotMatchError
-
assert_reply_equals
(contents, matches)¶ Send a message and wait for a response. If the response does not match the string exactly, fail the test.
Parameters: Returns: The reply.
Return type: Raises: ResponseDidNotMatchError
-
assert_reply_has_image
(contents)¶ Send a message consisting of contents and wait for a reply.
Check that the reply contains a
discord.Attachment
. If not, fail the test.Parameters: contents (str) – The content of the trigger message. (A command) Returns: The reply. Return type: discord.Message Raises: ResponseDidNotMatchError, NoResponseError
-
assert_reply_matches
(contents: str, regex)¶ Send a message and wait for a response. If the response does not match a regex, fail the test.
Requires a properly formatted Python regex ready to be used in the
re
functions.Parameters: Returns: The reply.
Return type: Raises: ResponseDidNotMatchError
-
connect
(channel)¶ Connect to a given VoiceChannel :param channel: The VoiceChannel to connect to. :return:
-
disconnect
()¶ Disconnect from the VoiceChannel; Doesn’t work if the Bot isn’t connected. :return:
-
static
edit_message
(message, new_content)¶ Modify a message. Most tests and
send_message
return thediscord.Message
they sent, which can be used here. Helper FunctionParameters: - message (discord.Message) – The target message. Must be a
discord.Message
- new_content (str) – The text to change message to.
Returns: message after modification.
Return type: - message (discord.Message) – The target message. Must be a
-
ensure_silence
()¶ Assert that the bot does not post any messages for some number of seconds.
Raises: UnexpectedResponseError, TimeoutError
-
get_delayed_reply
(seconds_to_wait, test_function, *args)¶ Get the last reply after a specific time and check it against a given test.
Parameters: - seconds_to_wait (float) – Time to wait in s
- test_function (method) – The function to call afterwards, without parenthesis (assert_message_equals, not assert_message_equals()!)
- args – The arguments to pass to the test, requires the same number of args as the test function. Make sur to pass in all args, including kwargs with defaults. NOTE: this policy may change if it becomes kinda stupid down the road.
Return type: Method
Raises: Returns: The instance of the test requested
-
send_message
(content)¶ Send a message to the channel the test is being run in. Helper Function
Parameters: content (str) – Text to send in the message Returns: The message that was sent Return type: discord.Message
-
wait_for_event
(event: str, check: Optional[Callable[[...], bool]] = None, timeout: float = None)¶ A wrapper for the discord.py function
wait_for
, tuned to be useful for distest.See https://discordpy.readthedocs.io/en/latest/api.html#event-reference for a list of events.
Parameters: - event – The discord.py event, as a string and with the
on_
removed from the beginning. - check (Callable[..,bool]) – A check function that all events of the type are ran against. Should return true when the desired event occurs, takes the event’s params as it’s params
- timeout (float) – How many seconds to wait for the event to occur.
Returns: The parameters of the event requested
Raises: NoResponseError
- event – The discord.py event, as a string and with the
-
wait_for_message
()¶ Wait for the bot the send any message. Will fail on timeout, but will ignore messages sent by anything other that the target.
Returns: The message we’ve been waiting for. Return type: discord.Message Raises: NoResponseError
-
wait_for_message_in_channel
(content, channel_id)¶ Send a message with
content
and returns the next message that the targeted bot sends. Used in many other tests.Parameters: Returns: The message we’ve been waiting for.
Return type: Raises: NoResponseError
-
wait_for_reaction
(message)¶ Assert that
message
is reacted to with any reaction.Parameters: message (discord.Message) – The message to test with Returns: The reaction object. Return type: discord.Reaction Raises: NoReactionError –
-
wait_for_reply
(content)¶ Send a message with
content
and returns the next message that the targeted bot sends. Used in many other tests.Parameters: content (str) – The text of the trigger message. Returns: The message we’ve been waiting for. Return type: discord.Message Raises: NoResponseError
Enumerations¶
The following enumeration (subclass of enum.Enum
) is used to indicate the result of a run test.
Bot¶
Contains the discord clients used to run tests.
DiscordBot
contains the logic for running tests and finding the target bot
DiscordInteractiveInterface
is a subclass of DiscordBot
and contains the logic to handle
commands sent from discord to run tests, display stats, and more
DiscordCliInterface
is a subclass of DiscordInteractiveInterface
and simply contains logic to
start the bot when it wakes up
-
class
distest.bot.
DiscordBot
(target_id)[source]¶ Discord bot used to run tests. This class by itself does not provide any useful methods for human interaction, and is just used as a superclass of the two interfaces,
DiscordInteractiveInterface
andDiscordCliInterface
Parameters: target_id (str) – The name of the target bot, used to ensure that the target user is actually present in the server. Good for checking for typos or other simple mistakes. -
run_test
(test: distest.TestInterface.Test, channel: discord.channel.TextChannel, stop_error=False) → distest.TestInterface.TestResult[source]¶ Run a single test in a given channel.
Updates the test with the result and returns it
Parameters: - test (Test) – The
Test
that is to be run - channel (discord.TextChannel) – The
- stop_error – Weather or not to stop the program on error. Not currently in use.
Returns: Result of the test
Return type: - test (Test) – The
-
-
class
distest.bot.
DiscordInteractiveInterface
(target_id, collector: distest.collector.TestCollector, timeout=5)[source]¶ A variant of the discord bot which commands sent in discord to allow a human to run the tests manually.
Does NOT support CLI arguments
Parameters: - target_id (str) – The name of the bot to target (Username, no discriminator)
- collector (TestCollector) – The instance of Test Collector that contains the tests to run
- timeout (int) – The amount of time to wait for responses before failing tests.
-
on_message
(message: discord.message.Message)[source]¶ Handle an incoming message, see
discord.event.on_message()
for event reference.Parse a message, can ignore it or parse the message as a command and run some tests or do one of the alternate functions (stats, list, or help)
Parameters: message (discord.Message) – The message being recieved, passed by discord.py
-
on_ready
()[source]¶ Report when the bot is ready for use and report the available tests to the console
-
run_tests
(channel: discord.channel.TextChannel, name: str)[source]¶ Helper function for choosing and running an appropriate suite of tests Makes sure only tests that still need to be run are run, also prints to the console when a test is run
Parameters: - channel (discord.TextChannel) – The channel in which to run the tests
- name (str) – Selector string used to determine what category of test to run
-
class
distest.bot.
DiscordCliInterface
(target_id, collector, test, channel_id, stats, timeout)[source]¶ A variant of the discord bot which is designed to be run off command line arguments.
Parameters: - target_id (str) – The name of the bot to target (Username, no discriminator)
- collector (TestCollector) – The instance of Test Collector that contains the tests to run
- test (str) – The name of the test option (all, specific test, etc)
- channel_id (int) – The ID of the channel to run the bot in
- stats (bool) – If true, run in hstats mode.
-
on_ready
()[source]¶ Run all the tests sequentially when the bot becomes awake and exit when the tests finish. The CLI should run all by itself without prompting, and this allows it to behave that way.
-
run
(token) → int[source]¶ Override of the default run() that returns failure state after completion. Allows the failure to cascade back up until it is processed into an exit code by
run_command_line_bot()
Parameters: token (str) – The tester bot token Returns: Returns 1 if the any test failed, otherwise returns zero. Return type: int
Collector¶
The TestCollector Class and some supporting code.
Each test function in the tester bot should be decorated with an instance of TestCollector(), and must have a unique name. The TestCollector() is then passed onto the bot, which runs the tests.
-
class
distest.collector.
TestCollector
[source]¶ Used to group tests and pass them around all at once.
Tests can be either added with
add
or by using@TestCollector
to decorate the function, as seen in the sample code below. Is very similar in function toCommand
from discord.py, which you might already be familiar with.1 2 3 4 5 6 7 8 9 10 11 12
@test_collector() async def test_reply_contains(interface): await interface.assert_reply_contains( "Say something containing 'gamer' please!", "gamer" ) @test_collector() async def test_reply_matches(interface): await interface.assert_reply_matches( "Say something matching the regex `[0-9]{1,3}`", r"[0-9]{1,3}"
-
add
(function, name=None, needs_human=False)[source]¶ Adds a test function to the group, if one with that name is not already present
Parameters: - function (func) – The function to add
- name (str) – The name of the function to add, defaults to the function name but can be overridden
with the provided name just like with
discord.ext.commands.Command
. See sample code above. - needs_human (bool) – Optional boolean, true if the test requires a human interaction
-
Exceptions¶
Stores all the Exceptions that can be called during testing.
Allows for a more through understanding of what went wrong. Not all of these are currently in use.
Patches¶
Contains the code required to patch out the fact that Bot
class ignores messages
from other bots.
This should be used if you have a target bot that uses the ext.commands.Bot
system, as otherwise it’s commands will ignore messages from the tester bot.
Usage¶
Simply put the below code into your main bot and then when testing, the bot will no longer ignore other bots!
1 2 3 4 5 6 bot = commands.Bot(command_prefix='$') # Do anything you want for this if, be it env vars, command line args, or the likes. if sys.argv[2] == "TESTING": from distest.patches import patch_target bot = patch_target(bot)
Docs¶
-
distest.patches.
patch_target
(bot)[source]¶ Patches the target bot. It changes the
process_commands
function to remove the check if the received message author is a bot or not.Parameters: bot (discord.ext.commands.Bot) – Returns: The patched bot.
Contributing¶
Hey! You’re here because you want to contribute to the bot, and that’s awesome! Here are some notes about contributions:
- Please open an issue for your contribution and tag it with contribution to discuss it. Because of my time constraints, I probably won’t have much time to implement new features myself, but if you make a feature and PR it in, I’ll be more than happy to spend a bit of time testing it out and add it to the project. The other thing is to make sure you check the github project to see if there is someone else already working on it who you can help.
- You may need to install the additional requirements from requirements-dev.txt. This is as simple as running pip install -r requirements-dev.txt. This larger list mostly includes things like black for formatting and sphinx for doc testing.
- If you are adding new test types, please make sure you test them well to make sure they work as intended, and please add a demo of them in use to the example_tests() for others to see. When you are done, please open a PR and I’ll add it in!
- I use Black for my code formatting, it would be appreciated if you could use it when contributing as well. I will remind you when you make a PR if you don’t, it is essential to make sure that diffs aren’t cluttered up with silly formatting changes. Additionally, CodeFactor should be tracking code quality and doing something with PRs. We will see soon exactly how that will work out.
- To build the docs for testing purposes, cd into the docs folder and run make testhtml. This will build to a set of HTML files you can view locally.
- Also, if you just want to propose an idea, create an issue and tag it with enhancement. The library is missing tons of features, so let me know what you want to see, and if I have time I’ll see about getting around to addign it. Thank you for your help!