Setting up the testing framework

Any Elixir project already comes with the testing framework ExUnit, but we could also use support from mocks and factories, and we can also install a code coverage tool, so that we can see what still needs to be tested. Additionally, since we're using a database to save our facts, we need to separate it from the development or production databases. First, we need some extra packages. ExMachina is the Elixir's equivalent of FactoryBot, ExCoveralls and the last one, Patch is a mocking library.

We'll add them in the mix.exs file just like the last time by adding the necessary entries to the defp deps:

{:ex_machina, "~> 2.7.0", only: :test},
{:excoveralls, "~> 0.10", only: :test},
{:patch, "~> 0.13", only: :test}

As you can see, they all are marked with only: :test, which as expected will skip compiling these packages unless we're using the test environment. We also need a little bit of extra configuration for the coverage package to work. This is also done in the same file. We're also adding an extra directory -

def project do
  [
    app: :cat_facts,
    version: "0.1.0",
    elixir: "~> 1.15",
    elixirc_paths: elixirc_paths(Mix.env()),
    start_permanent: Mix.env() == :prod,
    deps: deps(),
    test_coverage: [tool: ExCoveralls],
    preferred_cli_env: [
      coveralls: :test,
      "coveralls.detail": :test,
      "coveralls.post": :test,
      "coveralls.html": :test
    ]
  ]
end

defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_), do: ["lib"]

In the new

defmodule CatFacts.DataCase do
  use ExUnit.CaseTemplate

  using do
    quote do
      alias CatFacts.Repo

      import Ecto
      import Ecto.Changeset
      import Ecto.Query
      import CatFacts.DataCase
    end
  end

  setup tags do
    CatFacts.DataCase.setup_sandbox(tags)
    :ok
  end

  def setup_sandbox(tags) do
    pid = Ecto.Adapters.SQL.Sandbox.start_owner!(CatFacts.Repo, shared: not tags[:async])
    on_exit(fn -> Ecto.Adapters.SQL.Sandbox.stop_owner(pid) end)
  end
end

Before we start running any tests, we also need a bit of extra configuration for our databases - we don't want to test anything on our development database after all, so we need to introduce a separate database for testing. For that, we need to split up our development and test configs. Back in the config directory, make a new file test.exs for that:

import Config

config :cat_facts, CatFacts.Repo,
  username: "user",
  password: "password",
  hostname: "localhost",
  database: "cat_facts_test#{System.get_env("MIX_TEST_PARTITION")}",
  pool: Ecto.Adapters.SQL.Sandbox,
  pool_size: 10

config :logger, level: :warning

And similarly, we can move the database config for the development environment from config.exs to a new dev.exs file:

import Config

config :cat_facts, CatFacts.Repo,
  database: "cat_facts_dev",
  username: "user",
  password: "password",
  hostname: "localhost",
  stacktrace: true,
  show_sensitive_data_on_connection_error: true,
  pool_size: 10

After this, the main config.exs file should look like this, which will define the common values and load the appropriate database configuration for the right environment.

import Config

config :cat_facts, ecto_repos: [CatFacts.Repo]

config :nostrum,
  token: System.get_env("CATFACT_BOT_TOKEN"),
  gateway_intents: :all

config :logger, :console,
  metadata: [:shard, :guild, :channel]

# Import environment specific config. This must remain at the bottom
# of this file so it overrides the configuration defined above.
import_config "#{config_env()}.exs"

Last but not least, we need a test file. Make a folder in the test directory named cat_facts and in that folder a new file, facts_test.exs. Just like in Rails and RSpec, the tests need to have that specific _test.exs naming scheme and extension in order to be picked up by the testing suite.

defmodule CatFacts.FactsTest do
  use CatFacts.DataCase

  doctest CatFacts.Facts, import: true
end

This test file will be using our DataCase we introduced earlier - so that the data doesn't linger between different tests - and we'll be using doctests to test our application's Facts context.

Testing database interactions

Now we can run mix deps.get from the console and then mix coveralls.html. We should see some output from the coveralls package in the console and in the cover directory we should get an excoveralls.html file which shows all our project's files that get compiled and how much of the code in them is covered by our tests.

As you can see, pretty much nothing is tested at this point, except for the fact that our application does start correctly. Since we've already declared that we'll be using doctests in our FactsTest test file above, let's actually write some. To do so, we need to go back to our lib/cat_facts/facts.ex file - that's where doctests go, rather than being separated in the test file itself.

@doc """
Creates a fact for the specified Discord guild with the supplied content.

## Examples

    iex> {:ok, %CatFacts.Facts.Fact{} = result} = create_guild_fact(12345, "my cool fact")
    iex> result.guild_id == 12345
    true

    iex> {:ok, %CatFacts.Facts.Fact{} = result} =
    ...> create_guild_fact(12345, "my cool fact")
    iex> result.content == "my cool fact"
    true

"""
def create_guild_fact(guild_id, content) do
  %Fact{}
  |> Fact.changeset(%{guild_id: guild_id, content: content})
  |> Repo.insert()
end

What we've written here is a doctest - a documentation block for a function that not only describes what the function does, but also provides one or more code examples of the function's usage. Each of those examples will also be treated as a separate test case. Here we're declaring two tests - they will add a fact into the database and then check that the returned record does have the parameters we've passed to the function.

A doctest consists of two parts, just like a regular test does - we can see a few lines of setup, which are marked by the iex> (and ...> in case of multiline commands) prefixes. The result of those setup commands will be compared to the line without a prefix - in those cases, to true, since the last setup command is a comparison. We can add a similar test for the second function in our Facts context:

@doc """
Returns a randomly chosen fact for the specified guild ID or nil if there aren't any facts present.

## Examples

    iex> get_random_guild_fact(99887)
    nil

    iex> create_guild_fact(123, "coolest fact")
    iex> create_guild_fact(543, "this one's uncool")
    iex> result = get_random_guild_fact(123)
    iex> result.content == "coolest fact"
    true
"""
def get_random_guild_fact(guild_id) do
  query =
    from f in Fact,
    where: f.guild_id == ^guild_id,
    order_by: fragment("RANDOM()"),
    limit: 1

  Repo.one(query)
end

Now with the two doctests and four examples in them, we can run the mix coveralls.html tool again and see what else needs to be covered with tests.

If we open the generated html file, we can see that it found only 3 relevant lines of code in our Facts context and that our tests checked all of them. Looking at the right side of the report to see what files are left untested, we can see that we should move to testing the bot_consumer.ex file, which has most of the code in this project.

Writing regular tests

We'll put the new tests in a new file, which should be saved as test/cat_facts/bot_consumer_test.exs. At first, we can pick up the easiest test we have to write in the BotConsumer module - dealing with unhandled messages, to which our function should return a :noop atom.

defmodule CatFacts.BotConsumerTest do
  use CatFacts.DataCase
  use Patch

  alias CatFacts.BotConsumer
  alias Nostrum.Struct.{WSState, Event, Guild, Interaction, Message}

  describe "on unhandled events" do
    setup do
      reaction_event = {:MESSAGE_REACTION_ADD, %Event.MessageReactionAdd{}, %WSState{}}

      {:ok, event: reaction_event}
    end

    test "handler returns noop", %{event: event} do
      assert BotConsumer.handle_event(event) == :noop
    end
  end
end

Let's walk through this code before we move on to more advanced tests.

use CatFacts.DataCase
use Patch

alias CatFacts.BotConsumer
alias Nostrum.Struct.{WSState, Event, Guild, Interaction, Message}

First we got a block of - mostly unused for now - aliases and uses. Aliases are used for code readability, just like in regular code, but the two use macros are more interesting. First we make sure that we're using the DataCase we've introduced earlier into test/support/data_case.exs. This will make sure all the tests are separated from each other. The other use macro introduces the Patch package we added at the start of this article. It will allow us to test our code while not actually connecting to any actual Discord servers.

describe "on unhandled events" do
  setup do
    reaction_event = {:MESSAGE_REACTION_ADD, %Event.MessageReactionAdd{}, %WSState{}}

    {:ok, event: reaction_event}
  end

  test "handler returns noop", %{event: event} do
    assert BotConsumer.handle_event(event) == :noop
  end
end

Next we have our actual test. While generally similar to how we write tests in Rails, there are some differences. In Rails, we'd normally use RSpec the setup would look similar to this:

let(:event) { {:MESSAGE_REACTION_ADD, %Event.MessageReactionAdd{}, %WSState{}} }

And the event variable declared in this way would be instantly available in the test itself. Elixir is a much more explicit language though, so the test setup is also a little more explicit.

setup do
  reaction_event = {:MESSAGE_REACTION_ADD, %Event.MessageReactionAdd{}, %WSState{}}

  {:ok, event: reaction_event}
end

First we declare our event inside the setup block. In order for it to be available inside the test, this block needs to return a tuple starting with an :ok atom and followed with whatever variables we want to have available in the tests.

test "handler returns noop", %{event: event} do

This tuple is then passed as a context to the test itself, where we can use pattern matching to extract the variables we want.

assert BotConsumer.handle_event(event) == :noop

The test itself on the other hand is relatively straightforward - there are the expected assertions and refutals available to us, so just like the line above, we're asserting that our function will return the expected result. Next we can move on to other types of events to be handled.

describe "on INTERACTION event" do
  setup do
    interaction = %Interaction{
      guild_id: 12345,
      data: %{
        name: "fact",
        options: [
          %{name: "content", value: "very interesting fact"}
        ]
      }
    }
    interaction_event = {:INTERACTION_CREATE, interaction, %WSState{}}
    expected_response = %{
      data: %{
        flags: 64,
        content: "Your cat fact has been added to this server's list of cat facts."
      },
      type: 4
    }

    {:ok, event: interaction_event, response: expected_response}
  end

  test "handler creates a response", %{event: {_, interaction, _} = event, response: response} do
    patch(Nostrum.Api, :create_interaction_response, nil)

    BotConsumer.handle_event(event)
    assert_called Nostrum.Api.create_interaction_response(^interaction, ^response)
  end
end

Once again most of this new code is taken by the setup macro. We're creating an interaction_event tuple just like in the previous example, but we also prepare a map of a response that we expect to be used. We can pass it to the test context.

test "handler creates a response", %{event: {_, interaction, _} = event, response: response} do

Since we also need our initial interaction to be used in the assertion, we can use pattern matching to extract it from the passed event, rather than passing it separately.

patch(Nostrum.Api, :create_interaction_response, nil)

Next we'll patch the Nostrum library, so that the create_interaction_response function responds with a nil instead of whatever the function's code originally was. This way it doesn't call the real Discord API anymore. Now we can easily focus on testing only our own module, and not the external services or libraries - after all, we can safely assume the creators of those test them on their own, so it's not our responsibility.

BotConsumer.handle_event(event)
assert_called Nostrum.Api.create_interaction_response(^interaction, ^response)

Finally, we call our handle_event function and check with the assert_called if our function called the Nostrum API in the expected way. In this particular assertion, we need to remember to use variable pinning if we want to check for specific values, otherwise it would just compare types and number of arguments.

Similarly, we can write a test for the READY event - the code for that test is pretty much the same as here, so you can check it out in the github repository.

Using factories for database records

Let's move on to our final part of the BotConsumer module, which is testing the response for different messages. Since that function should behave differently depending on the existence of records in the database, this is a good opportunity to use a Factory. Before we do so, we need to make some extra changes to our test setup.

In the mix.exs file, we need to add the test/factories directory to be compiled in the test environment, similarly to what we did for test/support. This will allow us to have a nice and logical directory structure.

defp elixirc_paths(:test), do: ["lib", "test/support", "test/factories"]

Next we need to take a look at the test/test_helper.exs file and start ExMachina there. ExMachina is Elixir's equivalent of FactoryBot - a library that deals with factories.

ExUnit.start()
{:ok, _} = Application.ensure_all_started(:ex_machina)

We also need a place to list and manage our factories, and that will go into the test/support/factory.ex file. Here we can list all our factories and make sure all of them are imported into our tests. We could place all our factories in this file just as functions in this Factory module, but I think Rails has a good idea here to keep those in separate files, and we can do that easily enough.

defmodule CatFacts.Factory do
  use ExMachina.Ecto, repo: CatFacts.Repo

  use CatFacts.FactFactory
end

And finally, the factory file itself - test/factories/fact_factory.ex. The extra lines of defmacro allow us to handle the use macro in the code above.

defmodule CatFacts.FactFactory do
  alias CatFacts.Facts.Fact

  defmacro __using__(_opts) do
    quote do
      def fact_factory do
        %Fact{
          guild_id: 123,
          content: "a normal fact"
        }
      end
    end
  end
end

With our setup finally complete, we can write the last tests that we need to have full code coverage. Since most of this code is very similar to other tests in this article, we can focus only on the one that uses our new factory. You can always check out the full test in the github repository.

describe "on MESSAGE event" do
  # ...

  test "handles the !catfact command with existing fact", %{ok_event: event} do
    insert(:fact, guild_id: 1234, content: "cool fact")
    patch(Nostrum.Api, :create_message, nil)

    BotConsumer.handle_event(event)
    assert_called Nostrum.Api.create_message(456, "cool fact")
  end
end

The factory works exactly like we are used to with FactoryBot - we can insert() a :fact, and specify (or omit any of them, in which they would fall back to the defaults from the FactFactory module) the attributes of the database record we're inserting. Similarly to FactoryBot, we could also use the build function instead and have a record that's not yet saved - to check for validations for example.

Wrapping up

With this we can finally consider our little project complete. In the first two articles we have finished the code and now all our code is able to be automatically tested without a need to manually go on Discord. Since we have learned about doctests, in a next project we should make sure to write them together with the code, and we also have learned how to mock external libraries, so that our tests can be focused on testing our own code.

Obviously our Discord bot turned out to be pretty simple, so the tests aren't very complicated either, but the basic principles behind them will stay the same in pretty much any sized project.