Merge branch 'feature/redrawhook'

* feature/redrawhook:
  docs: Track making the new codepath conditional upon the 'memo=' feature.
  On the feature/redrawhook branch, changelog: Add entries for issues fixed by this branch.
  On the feature/redrawhook branch, change the detection of the 'memo=' feature to avoid a catch-22.
  driver: Make the redrawhook codepath conditional upon the memo= feature.
  On the feature/redrawhook branch, move the changelog entry to the current release's section.
  driver: Fix a bug that prevented subsequent, third-party zle-line-pre-redraw hooks from running.
  driver: Do not pass widget arguments to _zsh_highlight
  driver: Clarify comment.  No functional change.
  driver: Allow for -U in autoloaded function definition
  driver: Use idiomatic module check
  driver: Make the shadowing $WIDGET read only.
  driver: Avoid a fork in the common case.
  test harness: Actually test the new code.
  driver: Rewrite without a state variable
  noop: Make a whitespace-only change to reduce noise in the next commit.
  docs: Rewrap.
  docs: Update FAQ answer per changes on this branch.
  redo _zsh_highlight__function_callable_p
  driver: Use a different way of checking whether add-zle-hook-widget is present.
  changelog: Use a more specific link.
  changelog: Note the effect of fixing #245/#90 and an alternative.
  driver: Pass zle-line-finish arguments on to _zsh_highlight.
  driver: Hook zle-line-finish.
  driver: Reimplement using 'add-zle-hook-widget zle-line-pre-redraw'
  wrappers: Reimplement using Mikachu's zle-line-pre-redraw hook (workers/36650).

README.md

@@ -44,11 +44,23 @@ FAQ
 
 ### Why must `zsh-syntax-highlighting.zsh` be sourced at the end of the `.zshrc` file?
 
-`zsh-syntax-highlighting.zsh` wraps ZLE widgets.  It must be sourced after all
-custom widgets have been created (i.e., after all `zle -N` calls and after
-running `compinit`).  Widgets created later will work, but will not update the
+zsh-syntax-highlighting works by hooking into the Zsh Line Editor (ZLE) and
+computing syntax highlighting for the command-line buffer as it stands at the
+time z-sy-h's hook is invoked.
+
+In zsh 5.2 and older,
+`zsh-syntax-highlighting.zsh` hooks into ZLE by wrapping ZLE widgets.  It must
+be sourced after all custom widgets have been created (i.e., after all `zle -N`
+calls and after running `compinit`) in order to be able to wrap all of them.
+Widgets created after z-sy-h is sourced will work, but will not update the
 syntax highlighting.
 
+In zsh newer than 5.8 (not including 5.8 itself),
+zsh-syntax-highlighting uses the `add-zle-hook-widget` facility to install
+a `zle-line-pre-redraw` hook.  Hooks are run in order of registration,
+therefore, z-sy-h must be sourced (and register its hook) after anything else
+that adds hooks that modify the command-line buffer.
+
 ### Does syntax highlighting work during incremental history search?
 
 Highlighting the command line during an incremental history search (by default bound to

changelog.md

@@ -1,6 +1,57 @@
 # Changes in HEAD
 
 
+## Changes fixed as part of the switch to zle-line-pre-redraw
+
+The changes in this section were fixed by switching to a `zle-line-pre-redraw`-based
+implementation.
+
+Note: The new implementation will only be used on future zsh releases,
+numbered 5.8.0.3 and newer, due to interoperability issues with other plugins
+(issues #418 and #579).  The underlying zsh feature has been available since
+zsh 5.2.
+
+Whilst under development, the new implementation was known as the
+"feature/redrawhook" topic branch.
+
+- Fixed: Highlighting not triggered after popping a buffer from the buffer stack
+  (using the `push-line` widget, default binding: `M-q`)
+  [#40]
+
+- Fixed: Invoking completion when there were no matches removed highlighting
+  [#90, #470]
+
+- Fixed: Two successive deletes followed by a yank only yanked the latest
+  delete, rather than both of them
+  [#150, #151, #160; cf. #183]
+
+- Presumed fixed: Completing `$(xsel)` results in an error message from `xsel`,
+  with pre-2017 versions of `xsel`.  (For 2017 vintage and newer, see the issue
+  for details.)
+  [#154]
+
+- Fixed: When the standard `bracketed-paste-magic` widget is in use, pastes were slow
+  [#295]
+
+- Fixed: No way to prevent a widget from being wrapped
+  [#324]
+
+- Fixed: No highlighting while cycling menu completion
+  [#375]
+
+- Fixed: Does not coexist with the `IGNORE_EOF` option
+  [#377]
+
+- Fixed: The `undefined-key` widget was wrapped
+  [#421]
+
+- Fixed: Does not coexist with the standard `surround` family of widgets
+  [#520]
+
+- Fixed: First completed filename doesn't get `path` highlighting
+  [#632]
+
+
 # Changes in 0.8.0-alpha1-pre-redrawhook
 
 ## Notice about an improbable-but-not-impossible forward incompatibility
@@ -19,6 +70,25 @@ added to zsh at z-sy-h's initiative.  The new feature is used in the fix
 to issue #418.
 
 
+## Incompatible changes:
+
+- An unsuccessful completion (a <kbd>⮀ Tab</kbd> press that doesn't change the
+  command line) no longer causes highlighting to be lost.  Visual feedback can
+  alternatively be achieved by setting the `format` zstyle under the `warnings`
+  tag, for example,
+
+        zstyle ':completion:*:warnings' format '%F{red}No matches%f'
+
+    Refer to the [description of the `format` style in `zshcompsys(1)`]
+    [zshcompsys-Standard-Styles-format].
+
+    (#90, part of #245 (feature/redrawhook))
+
+[zshcompsys-Standard-Styles]: http://zsh.sourceforge.net/Doc/Release/Completion-System.html#Standard-Styles
+[zshcompsys-Standard-Styles-format]: http://zsh.sourceforge.net/Doc/Release/Completion-System.html#index-format_002c-completion-style
+  
+
+
 ## Other changes:
 
 - Document `$ZSH_HIGHLIGHT_MAXLENGTH`.

tests/generate.zsh

@@ -31,6 +31,9 @@
 emulate -LR zsh
 setopt localoptions extendedglob
 
+# Required for add-zle-hook-widget.
+zmodload zsh/zle
+
 # Argument parsing.
 if (( $# * $# - 7 * $# + 12 )) || [[ $1 == -* ]]; then
   print -r -- >&2 "$0: usage: $0 BUFFER HIGHLIGHTER BASENAME [PREAMBLE]"

tests/test-highlighting.zsh

@@ -31,6 +31,9 @@
 
 setopt NO_UNSET WARN_CREATE_GLOBAL
 
+# Required for add-zle-hook-widget.
+zmodload zsh/zle
+
 local -r root=${0:h:h}
 local -a anon_argv; anon_argv=("$@")
 

tests/test-perfs.zsh

@@ -29,6 +29,9 @@
 # -------------------------------------------------------------------------------------------------
 
 
+# Required for add-zle-hook-widget.
+zmodload zsh/zle
+
 # Check an highlighter was given as argument.
 [[ -n "$1" ]] || {
   echo >&2 "Bail out! You must provide the name of a valid highlighter as argument."

zsh-syntax-highlighting.zsh

@@ -49,6 +49,52 @@ if true; then
   fi
 fi
 
+# This function takes a single argument F and returns True iff F is an autoload stub.
+_zsh_highlight__function_is_autoload_stub_p() {
+  if zmodload -e zsh/parameter; then
+    #(( ${+functions[$1]} )) &&
+    [[ "$functions[$1]" == *"builtin autoload -X"* ]]
+  else
+    #[[ $(type -wa -- "$1") == *'function'* ]] &&
+    [[ "${${(@f)"$(which -- "$1")"}[2]}" == $'\t'$histchars[3]' undefined' ]]
+  fi
+  # Do nothing here: return the exit code of the if.
+}
+
+# Return True iff the argument denotes a function name.
+_zsh_highlight__is_function_p() {
+  if zmodload -e zsh/parameter; then
+    (( ${+functions[$1]} ))
+  else
+    [[ $(type -wa -- "$1") == *'function'* ]]
+  fi
+}
+
+# This function takes a single argument F and returns True iff F denotes the
+# name of a callable function.  A function is callable if it is fully defined
+# or if it is marked for autoloading and autoloading it at the first call to it
+# will succeed.  In particular, if a function has been marked for autoloading
+# but is not available in $fpath, then this function will return False therefor.
+#
+# See users/21671 http://www.zsh.org/cgi-bin/mla/redirect?USERNUMBER=21671
+_zsh_highlight__function_callable_p() {
+  if _zsh_highlight__is_function_p "$1" &&
+     ! _zsh_highlight__function_is_autoload_stub_p "$1"
+  then
+    # Already fully loaded.
+    return 0 # true
+  else
+    # "$1" is either an autoload stub, or not a function at all.
+    #
+    # Use a subshell to avoid affecting the calling shell.
+    #
+    # We expect 'autoload +X' to return non-zero if it fails to fully load
+    # the function.
+    ( autoload -U +X -- "$1" 2>/dev/null )
+    return $?
+  fi
+}
+
 # -------------------------------------------------------------------------------------------------
 # Core highlighting update system
 # -------------------------------------------------------------------------------------------------
@@ -351,6 +397,49 @@ _zsh_highlight_call_widget()
   _zsh_highlight
 }
 
+# Decide whether to use the zle-line-pre-redraw codepath (colloquially known as
+# "feature/redrawhook", after the topic branch's name) or the legacy "bind all
+# widgets" codepath.
+#
+# We use the new codepath under two conditions:
+#
+# 1. If it's available, which we check by testing for add-zle-hook-widget's availability.
+# 
+# 2. If zsh has the memo= feature, which is required for interoperability reasons.
+#    See issues #579 and #735, and the issues referenced from them.
+#
+#    We check this with a plain version number check, since a functional check,
+#    as done by _zsh_highlight, can only be done from inside a widget
+#    function — a catch-22.
+#
+#    See _zsh_highlight for the magic version number.  (The use of 5.8.0.2
+#    rather than 5.8.0.3 as in the _zsh_highlight is deliberate.)
+if is-at-least 5.8.0.2 && _zsh_highlight__function_callable_p add-zle-hook-widget
+then
+  autoload -U add-zle-hook-widget
+  _zsh_highlight__zle-line-finish() {
+    # Reset $WIDGET since the 'main' highlighter depends on it.
+    #
+    # Since $WIDGET is declared by zle as read-only in this function's scope,
+    # a nested function is required in order to shadow its built-in value;
+    # see "User-defined widgets" in zshall.
+    () {
+      local -h -r WIDGET=zle-line-finish
+      _zsh_highlight
+    }
+  }
+  _zsh_highlight__zle-line-pre-redraw() {
+    # Set $? to 0 for _zsh_highlight.  Without this, subsequent
+    # zle-line-pre-redraw hooks won't run, since add-zle-hook-widget happens to
+    # call us with $? == 1 in the common case.
+    true && _zsh_highlight "$@"
+  }
+  _zsh_highlight_bind_widgets(){}
+  if [[ -o zle ]]; then
+    add-zle-hook-widget zle-line-pre-redraw _zsh_highlight__zle-line-pre-redraw
+    add-zle-hook-widget zle-line-finish _zsh_highlight__zle-line-finish
+  fi
+else
   # Rebind all ZLE widgets to make them invoke _zsh_highlights.
   _zsh_highlight_bind_widgets()
   {
@@ -417,6 +506,7 @@ _zsh_highlight_bind_widgets()
       esac
     done
   }
+fi
 
 # Load highlighters from directory.
 #