Golden Testing with Hspec-golden

The value of software testing relies on the ability to check if an application behaves properly. Unit tests are the first layer of testing and help developers detect bugs early in the code by testing each function individually. For example, let's create a function that welcomes someone to this tutorial:

module HelloWorld where

sayHi :: String -> String
sayHi name = "Welcome to the Golden Tests tutorial " ++ name

Using hspec we can easily test our function :

import HelloWorld
import Test.Hspec

spec :: Spec
spec = do
    describe "sayHi" $ 
        it "shows a Hello Golden Testers string" $
            sayHi "John Doe" `shouldBe` "Welcome to the Golden Tests tutorial John Doe"

In a nutshell, we are comparing the output of the function with a string that is stored inside the body of our test. We can take a different approach and store this expected output in a separate file. This approach is known as golden testing and the file in which we store the expected output takes the name of "golden file".

So, now we know what golden testing is, but if unit tests and golden tests are so similar, when should we use golden tests instead of just regular unit tests?

The truth is that unit tests can get messy when evaluating large, complex outputs, and that’s where golden testing can make a difference. Readability is important for writing maintainable software and that is true for our tests, for example when we test large outputs that don’t change that often, it would not be very practical to put it inside the source code as we tend to add visual noise. Golden Tests helpfully fix this issue and come with these advantages:

• Large outputs are stored in a golden file reducing the visual noise in our code.

• No need to escape quotes or binary data in the expected output.

• Golden Files can be automatically generated.

• Golden test libraries also automatically update the golden file. So they don't need to be updated manually when the expected output changes.

• It is useful when working with JSON, HTML, or images as they normally are large outputs that do not change that often.

In this tutorial, we will use the hspec-golden library to build golden tests. We will also use the hspec-golden CLI to easily update them.

You can either code along with this tutorial or check out the completed code in GitHub.

Hspec-golden

The hspec-golden testing library is a Stack Builders open-source project written in Haskell, that helps users implement golden tests using the hspec testing framework. You can find its source code in Github, and feel free to submit your suggestions or contributions.

Getting Started

First things first, let's create a new Haskell project to work in:

stack new "hspec-golden-tests-tutorial"

and add the following required dependencies to the recently created file package.yaml.

For this project we are using snapshot 19.22

Project dependencies:

dependencies:
- base >= 4.7 && < 5
- aeson >= 2.0.3.0
- text
- bytestring
- blaze-html >= 0.9.1.2

Testing dependencies:

dependencies:
- hspec-golden-tests-tutorial
- hspec
- hspec-golden

Let’s install these dependencies by running:

stack build

A Simple Test

Now that we have all of the dependencies installed, let's get started with the golden tests.

First, let's create a FizzBuzz module to be tested and add some code. For convenience, I have created it inside a FizzBuzz directory.

module FizzBuzz where 

fizzBuzz :: [Int] -> [String]
fizzBuzz list = map fizzOrBuzz list

fizzOrBuzz :: Int -> String
fizzOrBuzz n | n `mod` 15 == 0  = "FizzBuzz"
             | n `mod` 3  == 0  = "Fizz"
             | n `mod` 5  == 0  = "Buzz"
             | otherwise        = show n

For simplicity, we will test a simple function fizzBuzz that replaces the multiples of 3 with “Fizz” and multiples of 5 with “Buzz”. In case that a number is a multiple of both, it replaces the number with “FizzBuzz”. Before moving on with testing, let's review some of hspec-golden documentation.

Hspec-golden provides us with a defaultGolden function which creates the golden files and compares it with the Subject Under Test (SUT) output. It takes two parameters:

• the name of the test
• the output of our SUT

Note : _It is important to give the tests unique names otherwise hspec-golden will overwrite the results from other tests.

defaultGolden :: String -> String -> Golden String

By default, this function searches for golden tests inside a .golden/ directory. This directory is created automatically when the test runs.

Okay, enough reading, let's create our tests. Let's first create a test module named FizzBuzzGoldenSpec under test/FizzBuzz/ directory.

module FizzBuzz.FizzBuzzGoldenSpec where

We need some imports:

import           Test.Hspec                   --Tests
import           Test.Hspec.Golden            --Golden Tests
import           FizzBuzz.FizzBuzz             -- SUT

Finally let's write the test:

spec :: Spec
spec = 
    describe "fizzBuzz" $
      it "Turns multiples of 3 to Fizz and multiples of 5 to Buzz" $
        defaultGolden "FizzBuzz" (show $ fizzBuzz [1,2,3,4,5,11,12,13,14,15])

We are now ready to test our module.

$ stack test
hspec-golden-tests> test (suite: hspec-golden-tests-test)
FizzBuzz.FizzBuzzGolden
  FizzBuzz
    Turns 3 multiples to fizz and 5 multiples to buzz
      First time execution. Golden file created.
Finished in 0.0006 seconds
2 examples, 0 failures
hspec-golden-tests> Test suite hspec-golden-tests-test passed

The test was successful. If we check the ".golden" directory we will find two files.

• golden : Stores the expected output.
• actual : Stores the real output from the SUT.

$ tree
.golden
 └── fizzbuzz
     ├── actual
     └── golden

The actual and golden will both have in its content the output of the fizzBuzz function.

["1","2","Fizz","4","Buzz","11","Fizz","13","14","FizzBuzz"]

The testing framework recognized that this was the first execution, therefore created the fizzbuzz test with the actual and golden files. The difference between these two files is that the golden file will stay the same unless we want to update it, while the actual file will be overwritten every time we run the test. This is useful when updating the tests, but we will see that later on in this tutorial.

A More Real Test Case

The FizzBuzz module was used to create a simple and didactic example but it could have been easily tested with unit tests. In practice, golden tests are often used when testing JSON, HTML or images which generate large outputs. We also haven't yet tested the entire functionality of the hspec-golden library. For example the hgold CLI for updating the golden files.

Let's create some new modules to demonstrate a real case in which golden tests are needed.

• Html.hs : Renders a HTML template.
• Json.hs : Encodes a data type into a JSON Bytestring.

$ tree src
src
├── FizzBuzz.hs
├── HelloWorld.hs
├── Html.hs
└── Json.hs

To make things easy and save time, let's add the following code into each module.

Html.hs

{-# LANGUAGE OverloadedStrings #-}
module HTML.Html  where

import           Text.Blaze.Html5               as H
import           Text.Blaze.Html5.Attributes    as A
import           Text.Blaze.Html.Renderer.Pretty        (renderHtml)

htmlRendered :: Html -> String
htmlRendered page = renderHtml page

somePage :: Html
somePage = html $ do
    H.head $ do
        H.title "Stack Builders Tutorial."
    body $ do
       "Hello Golden Testers."

Json.hs

{-# LANGUAGE DeriveGeneric #-}
module JSON.Json where

import            Data.Aeson                   (ToJSON, encode)
import            GHC.Generics                 (Generic)
import            Data.ByteString.Lazy         (ByteString)

data Country = Country
  { cname     :: String
  , continent :: String
  , ctag      :: Int
  } deriving (Generic, Show)

instance ToJSON Country

ecuador, germany, japan :: Country
ecuador = Country "Ecuador" "America" 1
germany = Country "Germany" "Europe" 2
japan = Country "Japan" "Asia" 3

countries :: [Country]
countries = [ecuador,germany,japan]

encodeCountries :: [Country] -> ByteString
encodeCountries = encode

Before coding our tests, let's create and organize HtmlGoldenSpec.hs and JsonGoldenSpec.hs modules into directories as well.

$ tree test
test
├── FizzBuzz
│   └── FizzBuzzGoldenSpec.hs
├── Hello
│   ├── HelloGoldenSpec.hs
│   └── HelloSpec.hs
├── Html
│   └── HtmlGoldenSpec.hs
├── Json
│   └── JsonGoldenSpec.hs
└── Spec.hs

We are now ready to start coding our golden tests.

HTML Golden tests

Let's start with the HTML module importing all the things we need:

import           Test.Hspec
import           Test.Hspec.Golden
import           HTML.Html

and writing our test:

spec :: Spec
spec =
    describe "renderHtml" $
      it "Renders an Html5 file " $
       defaultGolden "html" (htmlRendered somePage)

Just like the previous example, after running the test we will have a new directory .golden/html/ that contains our actual and golden file. But what happens if we change our HTML template?

In practice changes may include several lines and tags but to make a simple example, let's edit the body of our HTML to "Goodbye Golden Testers" and run the tests:

test/Html/HtmlGoldenSpec.hs:11:5:
  1) Html.HtmlGolden.renderHtml Renders an Html5 file
      expected: "   
       <body>
               Hello Golden Testers.
       </body>"
      but got: "
       <body>
               Goodbye Golden Testers.
       </body>"
Finished in 0.0013 seconds
3 examples, 1 failure

Our HTML test failed but no big deal, we can edit our golden file and things will work just fine. Pretty simple right?

Actually, it's not. Let's remember this is just a didactic example. In real life golden files store really large outputs and it would be a waste of time to update them manually. Fortunately, hspec-golden provides us with the hgold CLI tool that automatically updates golden files. Let's install it.

Using stack:

$stack install hspec-golden

using Cabal:

$cabal install hspec-golden

Once installed we can use the --help flag to see how the CLI works.

According to hspec-golden, when hgold is used without flags it updates the .golden/ directory by default. If we store our golden files in a different directory we should use the --update flag and specify the name of the directory as an argument.

Okay, let's update our golden files:

$ hgold
Replacing golden with actual...
  Replacing file: .golden/hello/golden with: .golden/hello/actual
  Replacing file: .golden/html/golden with: .golden/html/actual
Finish...

hgold replaces the golden file (expected output) with the content of the actual file (SUT output).

So now we are ready to test our updated golden files:

Hello.HelloGolden
  sayHi
    returns Hello Golden Testers string
      Golden and Actual output hasn't changed
Hello.Hello
  sayHi
    shows a Hello Golden Testers string
Html.HtmlGolden
  renderHtml
    Renders an Html5 file 
      Golden and Actual output hasn't changed
Finished in 0.0034 seconds
3 examples, 0 failures

And we can see, everything works fine.

JSON Golden Test

Until now, we have been dealing with test cases that return only strings, but what if our return type is different like a ByteString or Text? For example, let's analyze the encodeCountries function in the Json.hs module.

encodeCountries :: [Country] -> ByteString
encodeCountries = encode

This function returns a ByteString. If we try to assert the output with our defaultGolden function,

  describe "encodeCountries" $ do
   it "encodes a group of Countries into a JSON String " $
    defaultGolden "json" (encodeCountries countries)

tests won't even compile:

$stack test
• Couldn't match type ‘B.ByteString’ with ‘[Char]’
      Expected type: String
        Actual type: B.ByteString

Conveniently, hspec-golden exports a Golden data type. With this tool, we can configure our Golden test by specifying the functions hspec-golden should use for our test to work:

data Golden str =
  Golden {
    output        :: str, -- ^ Output
    encodePretty  :: str -> String, -- ^ Makes the comparison pretty when the test fails
    writeToFile   :: FilePath -> str -> IO (), -- ^ How to write into the golden file the file
    readFromFile  :: FilePath -> IO str, -- ^ How to read the file,
    goldenFile    :: FilePath, -- ^ Where to read/write the golden file for this test.
    actualFile    :: Maybe FilePath, -- ^ Where to save the actual file for this test. If it is @Nothing@ then no file is written.
    failFirstTime :: Bool -- ^ Whether to record a failure the first time this test is run
  }

For example, we can define a different directory for storing our golden files or how our test will be read and how the output should be written. Also, the encodePretty parameter determines how the prompt should print a readable output when tests fail. With this data-type we are now able to create a new function to assert our output. Let's start by importing some modules:

module Json.JsonGoldenSpec where
import           Test.Hspec
import           Test.Hspec.Golden
import           JSON.Json
import qualified Data.ByteString.Lazy as B    

Next, based on the Golden data type documentation , let's create our assert function:

goldenBytestring :: String -> B.ByteString -> Golden B.ByteString
goldenBytestring name actualOutput =
    Golden {
        output = actualOutput,
        encodePretty = show,
        writeToFile = B.writeFile,
        readFromFile = B.readFile,
        goldenFile = ".otherGolden/" <> name <> "-golden",
  actualFile = Just (".otherGolden/" <> name <> "-actual"),     
        failFirstTime = False

    }

Let's analyze a little bit our goldenBytestring function: - Its type is Golden ByteString. - To print our failed tests it makes use of haskell's show function. - To write the bytestring into a file, as well as reading it, we have used the writeFile and readFile functions from the Data.ByteString.Lazy module. - We will be saving the output into a different directory called .otherGolden/.

We are now ready to create our test,

spec :: Spec
spec =
  describe "encodeCountries" $ do
    it "encodes a group of Countries into a JSON bytestring " $
      goldenBytestring "json" (encodeCountries countries)

and run it to see if our assert function is working properly.

  encodeCountries
    encodes a group of Countries into a JSON bytestring 
      First time execution. Golden file created.
Finished in 0.0034 seconds
4 examples, 0 failures

Everything works fine.

If we check our directories, we will find that the .otherGolden directory has been created.

$ tree 
├── .golden
│   ├── hello
│   │   ├── actual
│   │   └── golden
│   └── html
│       ├── actual
│       └── golden
|
└── .otherGolden

Let's remember that we stored our golden files in a different directory. So in the case that we make an update we should use the hgold CLI with the --update flag, followed by the name of our golden tests directory; in this case .otherGolden/:

$ hgold --update ".otherGolden"
Replacing golden with actual...
  Replacing file: .otherGolden/json/golden with: .otherGolden/json/actual
Finish...

Final Words

Golden testing provides us with a practical way to store and test large outputs generated by our components. For example, testing an API that sends huge amounts of data or testing our HTML templates in the front-end. In Haskell, the hspec-golden testing tool makes it easy to implement golden tests in a project. This tool provides a module capable of asserting the outputs from our SUT with our golden files as well as creating them in case they don't exist. It is also useful when updating our tests. With its CLI, we can replace old golden files with the new output of our SUT. Using this tool, Haskell developers will be able to start using golden tests within applications with very little effort.