path: root/t/helper
diff options
authorÆvar Arnfjörð Bjarmason <>2021-01-24 17:28:13 (GMT)
committerJunio C Hamano <>2021-01-25 00:09:17 (GMT)
commit95ca1f987edd23389e3079d0a7fe6d0f89927b68 (patch)
tree848a3df16861c43f43e8161eeae352e597337ce6 /t/helper
parenta4fea08b6ebc2782891abdf6fd4bb9feeb21ff4d (diff)
grep/pcre2: better support invalid UTF-8 haystacks
Improve the support for invalid UTF-8 haystacks given a non-ASCII needle when using the PCREv2 backend. This is a more complete fix for a bug I started to fix in 870eea8166 (grep: do not enter PCRE2_UTF mode on fixed matching, 2019-07-26), now that PCREv2 has the PCRE2_MATCH_INVALID_UTF mode we can make use of it. This fixes the sort of case described in 8a5999838e (grep: stess test PCRE v2 on invalid UTF-8 data, 2019-07-26), i.e.: - The subject string is non-ASCII (e.g. "ævar") - We're under a is_utf8_locale(), e.g. "en_US.UTF-8", not "C" - We are using --ignore-case, or we're a non-fixed pattern If those conditions were satisfied and we matched found non-valid UTF-8 data PCREv2 might bark on it, in practice this only happened under the JIT backend (turned on by default on most platforms). Ultimately this fixes a "regression" in b65abcafc7 ("grep: use PCRE v2 for optimized fixed-string search", 2019-07-01), I'm putting that in scare-quotes because before then we wouldn't properly support these complex case-folding, locale etc. cases either, it just broke in different ways. There was a bug related to this the PCRE2_NO_START_OPTIMIZE flag fixed in PCREv2 10.36. It can be worked around by setting the PCRE2_NO_START_OPTIMIZE flag. Let's do that in those cases, and add tests for the bug. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Junio C Hamano <>
Diffstat (limited to 't/helper')
3 files changed, 14 insertions, 0 deletions
diff --git a/t/helper/test-pcre2-config.c b/t/helper/test-pcre2-config.c
new file mode 100644
index 0000000..5258fdd
--- /dev/null
+++ b/t/helper/test-pcre2-config.c
@@ -0,0 +1,12 @@
+#include "test-tool.h"
+#include "cache.h"
+#include "grep.h"
+int cmd__pcre2_config(int argc, const char **argv)
+ if (argc == 2 && !strcmp(argv[1], "has-PCRE2_MATCH_INVALID_UTF")) {
+ int value = PCRE2_MATCH_INVALID_UTF;
+ return !value;
+ }
+ return 1;
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index 9d6d14d..f97cd9f 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {
{ "parse-options", cmd__parse_options },
{ "parse-pathspec-file", cmd__parse_pathspec_file },
{ "path-utils", cmd__path_utils },
+ { "pcre2-config", cmd__pcre2_config },
{ "pkt-line", cmd__pkt_line },
{ "prio-queue", cmd__prio_queue },
{ "proc-receive", cmd__proc_receive},
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index a6470ff..28072c0 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -35,6 +35,7 @@ int cmd__online_cpus(int argc, const char **argv);
int cmd__parse_options(int argc, const char **argv);
int cmd__parse_pathspec_file(int argc, const char** argv);
int cmd__path_utils(int argc, const char **argv);
+int cmd__pcre2_config(int argc, const char **argv);
int cmd__pkt_line(int argc, const char **argv);
int cmd__prio_queue(int argc, const char **argv);
int cmd__proc_receive(int argc, const char **argv);