zsh-syntax-highlighting/tests
Daniel Shahaf d5d2f22013 Revert "test harness: Rewrite the columnar pretty-printer without external tools." and "travis: Remove bsdmainutils since column(1) has been removed, three commits ago."
This reverts commits ea7c165b59 and
3d81c83132.

When "have 6 expectations and 4 region_highlight entries", the pure-zsh
implementation printed them as follows:

    not ok 7 - cardinality check - have 6 expectations and 4 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 6 comment' $'8 13 comment' $'15 15 default' $'16 21 comment' $'22 22 default' )»
    «region_highlight=( $'0 1 builtin' $'2 6 comment' $'7 13 comment' $'14 22 default' )»
    # expected_region_highlight  '22 22 default'
    # '1 1 builtin'              region_highlight
    # '3 6 comment'              '0 1 builtin'
    # '8 13 comment'             '2 6 comment'
    # '15 15 default'            '7 13 comment'
    # '16 21 comment'            '14 22 default'

Whereas the column(1)-based implementation prints them as follows:

    not ok 7 - cardinality check - have 6 expectations and 4 region_highlight entries: «expected_region_highlight=( $'1 1 builtin' $'3 6 comment' $'8 13 comment' $'15 15 default' $'16 21 comment' $'22 22 default' )» «region_highlight=( $'0 1 builtin' $'2 6 comment' $'7 13 comment' $'14 22 default' )»
    # expected_region_highlight  region_highlight
    # '1 1 builtin'              '0 1 builtin'
    # '3 6 comment'              '2 6 comment'
    # '8 13 comment'             '7 13 comment'
    # '15 15 default'            '14 22 default'
    # '16 21 comment'
    # '22 22 default'

Ultimately, this difference is down to the pure-zsh implementation
getting the arguments as a single list, whereas paste(1) gets two
separate lists.
2020-03-20 00:03:01 +00:00
..
edit-failed-tests test harness: Update tests/edit-failed-tests for harness output changes in commit 2b3638a211, "test harness: Tweak quiet-test output". 2020-03-15 13:33:09 +00:00
generate.zsh dev tools: Allow specifying preamble code when generating test cases. 2020-01-16 17:19:25 +00:00
README.md test harness: Let tests fail early by exiting non-zero or by setting a flag. 2020-03-19 00:37:21 +00:00
tap-colorizer.zsh Bump copyright years. 2017-12-25 08:42:30 +00:00
tap-filter test harness: Tweak quiet-test output 2020-02-22 15:12:24 +00:00
test-highlighting.zsh Revert "test harness: Rewrite the columnar pretty-printer without external tools." and "travis: Remove bsdmainutils since column(1) has been removed, three commits ago." 2020-03-20 00:03:01 +00:00
test-perfs.zsh test harness: Output the time information to the same place the test name was printed to. 2020-03-19 19:17:41 +00:00

zsh-syntax-highlighting / tests

Utility scripts for testing zsh-syntax-highlighting highlighters.

The tests harness expects the highlighter directory to contain a test-data directory with test data files. See the main highlighter for examples.

Tests should set the following variables:

Each test should define the string $BUFFER that is to be highlighted and the array parameter $expected_region_highlight. The value of that parameter is a list of strings of the form "$i $j $style". or "$i $j $style $todo". Each string specifies the highlighting that $BUFFER[$i,$j] should have; that is, $i and $j specify a range, 1-indexed, inclusive of both endpoints. $style is a key of $ZSH_HIGHLIGHT_STYLES. If $todo exists, the test point is marked as TODO (the failure of that test point will not fail the test), and $todo is used as the explanation.

If a test sets $skip_test to a non-empty string, the test will be skipped with the provided string as the reason.

If a test sets $fail_test to a non-empty string, the test will be skipped with the provided string as the reason.

If a test sets unsorted=1 the order of highlights in $expected_region_highlight need not match the order in $region_highlight.

Normally, tests fail if $expected_region_highlight and $region_highlight have different numbers of elements. To mark this check as expected to fail, tests may set $expected_mismatch to an explanation string (like $todo); this is useful when the only difference between actual and expected is that actual has some additional, superfluous elements. This check is skipped if the $todo component is present in any regular test point.

Note: $region_highlight uses the same "$i $j $style" syntax but interprets the indexes differently.

Note: Tests are run with setopt NOUNSET WARN_CREATE_GLOBAL, so any variables the test creates must be declared local.

Isolation: Each test is run in a separate subshell, so any variables, aliases, functions, etc., it defines will be visible to the tested code (that computes $region_highlight), but will not affect subsequent tests. The current working directory of tests is set to a newly-created empty directory, which is automatically cleaned up after the test exits. For example:

setopt PATH_DIRS
mkdir -p foo/bar
touch foo/bar/testing-issue-228
chmod  +x foo/bar/testing-issue-228
path+=( "$PWD"/foo )

BUFFER='bar/testing-issue-228'

expected_region_highlight=(
  "1 21 command" # bar/testing-issue-228
)

Writing new tests

An experimental tool is available to generate test files:

zsh -f tests/generate.zsh 'ls -x' acme newfile

This generates a highlighters/acme/test-data/newfile.zsh test file based on the current highlighting of the given $BUFFER (in this case, ls -x).

This tool is experimental. Its interface may change. In particular it may grow ways to set $PREBUFFER to inject free-form code into the generated file.

Highlighting test

test-highlighting.zsh tests the correctness of the highlighting. Usage:

zsh test-highlighting.zsh <HIGHLIGHTER NAME>

All tests may be run with

make test

which will run all highlighting tests and report results in TAP format. By default, the results of all tests will be printed; to show only "interesting" results (tests that failed but were expected to succeed, or vice-versa), run make quiet-test (or make test QUIET=y).

Performance test

test-perfs.zsh measures the time spent doing the highlighting. Usage:

zsh test-perfs.zsh <HIGHLIGHTER NAME>

All tests may be run with

make perf