path: root/grep.c
diff options
authorÆvar Arnfjörð Bjarmason <>2019-07-26 15:08:16 (GMT)
committerJunio C Hamano <>2019-07-26 20:56:40 (GMT)
commit8a5999838e3cc24652f09670b6fe9461a789721b (patch)
treedce791cafa53e02344681f25b471023b2d31c446 /grep.c
parent09872f6418f6b6fc1b823d3b324907c02e9bc75b (diff)
grep: stess test PCRE v2 on invalid UTF-8 data
Since my b65abcafc7 ("grep: use PCRE v2 for optimized fixed-string search", 2019-07-01) we've been dying on invalid UTF-8 data when grepping for fixed strings if the following are all true: * The subject string is non-ASCII (e.g. "ævar") * We're under a is_utf8_locale(), e.g. "en_US.UTF-8", not "C" * We compiled with PCRE v2 * That PCRE v2 did not have JIT support The last of those is why this wasn't caught earlier, per pcre2jit(3): "unless PCRE2_NO_UTF_CHECK is set, a UTF subject string is tested for validity. In the interests of speed, these checks do not happen on the JIT fast path, and if invalid data is passed, the result is undefined." I.e. the subject being matched against our pattern was invalid, but we were lucky and getting away with it on the JIT path, but the non-JIT one is stricter. This patch does nothing to fix that, instead we sneak in support for fixed patterns starting with "(*NO_JIT)", this disables the PCRE v2 jit with implicit fixed-string matching for testing, see pcre2syntax(3) the syntax. This is technically a change in behavior, but it's so obscure that I figured it was OK. We'd previously consider this an invalid regular expression as regcomp() would die on it, now we feed it to the PCRE v2 fixed-string path. I thought this was better than introducing yet another GIT_TEST_* environment variable. We're also relying on a behavior of PCRE v2 that technically could change, but I think the test coverage is worth dipping our toe into some somewhat undefined behavior. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Junio C Hamano <>
Diffstat (limited to 'grep.c')
1 files changed, 10 insertions, 0 deletions
diff --git a/grep.c b/grep.c
index 6d60e2e..5bc0f4f 100644
--- a/grep.c
+++ b/grep.c
@@ -615,6 +615,16 @@ static void compile_regexp(struct grep_pat *p, struct grep_opt *opt)
die(_("given pattern contains NULL byte (via -f <file>). This is only supported with -P under PCRE v2"));
p->is_fixed = is_fixed(p->pattern, p->patternlen);
+#ifdef USE_LIBPCRE2
+ if (!p->fixed && !p->is_fixed) {
+ const char *no_jit = "(*NO_JIT)";
+ const int no_jit_len = strlen(no_jit);
+ if (starts_with(p->pattern, no_jit) &&
+ is_fixed(p->pattern + no_jit_len,
+ p->patternlen - no_jit_len))
+ p->is_fixed = 1;
+ }
if (p->fixed || p->is_fixed) {
opt->pcre2 = 1;