jfmengels / elm-lint / Lint.Test

Module that helps you test your linting rules, using elm-test.

import Lint.Test exposing (LintResult)
import Test exposing (Test, describe, test)
import The.Rule.You.Want.To.Test exposing (rule)

testRule : String -> LintResult
testRule string =
    Lint.Test.run rule string

-- In this example, the rule we're testing is `NoDebug`
tests : Test
tests =
    describe "NoDebug"
        [ test "should not report calls to normal functions" <|
            \() ->
                testRule """module A exposing (..)
a = foo n"""
                    |> Lint.Test.expectNoErrors
        , test "should report Debug.log use" <|
            \() ->
                testRule """module A exposing (..)
a = Debug.log "some" "message\""""
                    |> Lint.Test.expectErrors
                        [ Lint.Test.error
                            { message = "Remove the use of `Debug` before shipping to production"
                            , under = "Debug.log"
                            }
                        ]
        ]

Strategies for effective testing

Use Test-Driven Development

Writing a linting rules is a process that works really well with the Test-Driven Development process loop, which is:

Have a good title

A good test title explains

Ideally, by only reading through the test titles, someone else should be able to rewrite the rule you are testing.

What should you test?

You should test the scenarios where you expect the rule to report something. At the same time, you should also test when it shouldn't. I encourage writing tests to make sure that things that are similar to what you want to report are not reported.

For instance, if you wish to report uses of variables named foo, write a test that ensures that the use of variables named differently does not get reported.

Tests are pretty cheap, and in the case of linting rules, it is probably better to have too many tests rather than too few, since the behavior of a rule rarely changes drastically.

Design goals

If you are interested, you can read the design goals for this module.

Running tests


type LintResult

The result of running a rule on a String containing source code.

run : Lint.Rule.Rule -> String -> LintResult

Run a Rule on a String containing source code. You can then use expectNoErrors or expectErrors to assert the errors reported by the rule.

import Lint.Test exposing (LintResult)
import My.Rule exposing (rule)
import Test exposing (Test)

all : Test
all =
    test "test title" <|
        \() ->
            Lint.Test.run rule """module SomeModule exposing (a)
a = 1"""
                |> Lint.Test.expectNoErrors

The source code needs to be syntactically valid Elm code. If the code can't be parsed, the test will fail regardless of the expectations you set on it.

Note that to be syntactically valid, you need at least a module declaration at the top of the file (like module A exposing (..)) and one declaration (like a = 1). You can't just have an expression like 1 + 2.

Note: This is a simpler version of runWithProjectData. If your rule is interested in project related details, then you should use runWithProjectData instead.

runWithProjectData : Lint.Project.Project -> Lint.Rule.Rule -> String -> LintResult

Run a Rule on a String containing source code, with data about the project loaded, such as the contents of elm.json file.

import Lint.Project as Project exposing (Project)
import Lint.Test exposing (LintResult)
import My.Rule exposing (rule)
import Test exposing (Test)

all : Test
all =
    test "test title" <|
        \() ->
            let
                project : Project
                project =
                    Project.new
                        |> Project.withElmJson elmJsonToConstructManually
            in
            Lint.Test.runWithProjectData project rule """module SomeModule exposing (a)
a = 1"""
                |> Lint.Test.expectNoErrors

The source code needs to be syntactically valid Elm code. If the code can't be parsed, the test will fail regardless of the expectations you set on it.

Note that to be syntactically valid, you need at least a module declaration at the top of the file (like module A exposing (..)) and one declaration (like a = 1). You can't just have an expression like 1 + 2.

Note: This is a more complex version of run. If your rule is not interested in project related details, then you should use run instead.

Making assertions


type ExpectedError

An expectation for an error. Use error to create one.

expectErrors : List ExpectedError -> LintResult -> Expectation

Assert that the rule reported some errors, by specifying which one.

Assert which errors are reported using error. The test will fail if a different number of errors than expected are reported, or if the message or the location is incorrect.

The errors should be in the order of where they appear in the source code. An error at the start of the source code should appear earlier in the list than an error at the end of the source code.

import Lint.Test exposing (LintResult)
import Test exposing (Test, describe, test)
import The.Rule.You.Want.To.Test exposing (rule)

testRule : String -> LintResult
testRule string =
    Lint.Test.run rule string

-- In this example, the rule we're testing is `NoDebug`
tests : Test
tests =
    describe "NoDebug"
        [ test "should report Debug.log use" <|
            \() ->
                testRule """module A exposing (..)
a = Debug.log "some" "message\""""
                    |> Lint.Test.expectErrors
                        [ Lint.Test.error
                            { message = "Remove the use of `Debug` before shipping to production"
                            , under = "Debug.log"
                            }
                        ]
        ]

expectNoErrors : LintResult -> Expectation

Assert that the rule reported no errors. Note, this is equivalent to using expectErrors like expectErrors [].

import Lint.Test exposing (LintResult)
import Test exposing (Test, describe, test)
import The.Rule.You.Want.To.Test exposing (rule)

testRule : String -> LintResult
testRule string =
    Lint.Test.run rule string

-- In this example, the rule we're testing is `NoDebug`
tests : Test
tests =
    describe "NoDebug"
        [ test "should not report calls to normal functions" <|
            \() ->
                testRule """module A exposing (..)
a = foo n"""
                    |> Lint.Test.expectNoErrors
        ]

error : { message : String, details : List String, under : String } -> ExpectedError

Create an expectation for an error.

message should be the message you're expecting to be shown to the user.

under is the part of the code where you are expecting the error to be shown to the user. If it helps, imagine under to be the text under which the squiggly lines will appear if the error appeared in an editor.

tests : Test
tests =
    describe "NoDebug"
        [ test "should report Debug.log use" <|
            \() ->
                testRule """module A exposing (..)
a = Debug.log "some" "message\""""
                    |> Lint.Test.expectErrors
                        [ Lint.Test.error
                            { message = "Remove the use of `Debug` before shipping to production"
                            , under = "Debug.log"
                            }
                        ]
        ]

If there are multiple locations where the value of under appears, the test will fail unless you use atExactly to remove any ambiguity of where the error should be used.

atExactly : { start : { row : Basics.Int, column : Basics.Int }, end : { row : Basics.Int, column : Basics.Int } } -> ExpectedError -> ExpectedError

Precise the exact position where the error should be shown to the user. This is only necessary when the under field is ambiguous.

atExactly takes a record with start and end positions.

tests : Test
tests =
    describe "NoDebug"
        [ test "should report multiple Debug.log calls" <|
            \() ->
                testRule """module A exposing (..)
a = Debug.log "foo" z
b = Debug.log "foo" z
"""
                    |> Lint.Test.expectErrors
                        [ Lint.Test.error
                            { message = "Remove the use of `Debug` before shipping to production"
                            , under = "Debug.log"
                            }
                            |> Lint.Test.atExactly { start = { row = 4, column = 5 }, end = { row = 4, column = 14 } }
                        , Lint.Test.error
                            { message = "Remove the use of `Debug` before shipping to production"
                            , under = "Debug.log"
                            }
                            |> Lint.Test.atExactly { start = { row = 5, column = 5 }, end = { row = 5, column = 14 } }
                        ]
        ]

Tip: By default, do not use this function. If the test fails because there is some ambiguity, the test error will give you a recommendation of what to use as a parameter of atExactly, so you do not have to bother writing this hard-to-write argument yourself.

whenFixed : String -> ExpectedError -> ExpectedError

Create an expectation that the error provides fixes, meaning that it used the withFixes function) and an expectation of what the source code should be after the error's fixes have been applied.

In the absence of whenFixed, the test will fail if the error provides fixes. In other words: If the error provides fixes, you need to use withFixes, and if it doesn't, you should not use withFixes.

tests : Test
tests =
    describe "NoDebug"
        [ test "should report multiple Debug.log calls" <|
            \() ->
                testRule """module A exposing (..)
a = 1
b = Debug.log "foo" 2
"""
                    |> Lint.Test.expectErrors
                        [ Lint.Test.error
                            { message = "Remove the use of `Debug` before shipping to production"
                            , under = "Debug.log"
                            }
                            |> Lint.Test.whenFixed """module SomeModule exposing (b)
a = 1
b = 2
"""
                        ]
        ]