path: root/Makefile
AgeCommit message (Collapse)Author
2022-11-23Merge branch 'ab/coccicheck-incremental'Junio C Hamano
"make coccicheck" is time consuming. It has been made to run more incrementally. * ab/coccicheck-incremental: Makefile: don't create a ".build/.build/" for cocci, fix output spatchcache: add a ccache-alike for "spatch" cocci: run against a generated ALL.cocci cocci rules: remove <id>'s from rules that don't need them Makefile: copy contrib/coccinelle/*.cocci to build/ cocci: optimistically use COMPUTE_HEADER_DEPENDENCIES cocci: make "coccicheck" rule incremental cocci: split off "--all-includes" from SPATCH_FLAGS cocci: split off include-less "tests" from SPATCH_FLAGS Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" heading Makefile: have "coccicheck" re-run if flags change Makefile: add ability to TAB-complete cocci *.patch rules cocci rules: remove unused "F" metavariable from pending rule Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)
2022-11-18Merge branch 'vd/skip-cache-tree-update'Taylor Blau
Avoid calling 'cache_tree_update()' when doing so would be redundant. * vd/skip-cache-tree-update: rebase: use 'skip_cache_tree_update' option read-tree: use 'skip_cache_tree_update' option reset: use 'skip_cache_tree_update' option unpack-trees: add 'skip_cache_tree_update' option cache-tree: add perf test comparing update and prime
2022-11-11Makefile: don't create a ".build/.build/" for cocci, fix outputÆvar Arnfjörð Bjarmason
Fix a couple of issues in the recently merged 0f3c55d4c2b (Merge branch 'ab/coccicheck-incremental' into next, 2022-11-08): In copying over the "contrib/coccinelle/" rules to ".build/contrib/coccinelle/" we inadvertently ended up with a ".build/.build/contrib/coccinelle/" as well. We'd generate the per-file patches in the former, and keep the rule and overall result in the latter. E.g. running: make contrib/coccinelle/free.cocci.patch COCCI_SOURCES="attr.c grep.c" Would, per "tree -a .build" yield the following result: .build ├── .build │   └── contrib │   └── coccinelle │   └── free.cocci.patch │   ├── attr.c │   ├── attr.c.log │   ├── grep.c │   └── grep.c.log └── contrib └── coccinelle ├── FOUND_H_SOURCES ├── free.cocci └── free.cocci.patch Now we'll instead generate all of our files in ".build/contrib/coccinelle/". Fixing this required renaming the directory where we keep our per-file patches, as we'd otherwise conflict with the result. Now the per-file patch directory is named e.g. "free.cocci.d". And the end result will now be: .build └── contrib └── coccinelle ├── FOUND_H_SOURCES ├── free.cocci ├── free.cocci.d │   ├── attr.c.patch │   ├── attr.c.patch.log │   ├── grep.c.patch │   └── grep.c.patch.log └── free.cocci.patch The per-file patches now have a ".patch" file suffix, which fixes another issue reported against 0f3c55d4c2b: The summary output was confusing. Before for the "make" command above we'd emit: [...] MKDIR -p .build/contrib/coccinelle CP contrib/coccinelle/free.cocci .build/contrib/coccinelle/free.cocci GEN .build/contrib/coccinelle/FOUND_H_SOURCES MKDIR -p .build/.build/contrib/coccinelle/free.cocci.patch SPATCH .build/.build/contrib/coccinelle/free.cocci.patch/grep.c SPATCH .build/.build/contrib/coccinelle/free.cocci.patch/attr.c SPATCH CAT $^ >.build/contrib/coccinelle/free.cocci.patch CP .build/contrib/coccinelle/free.cocci.patch contrib/coccinelle/free.cocci.patch But now we'll instead emit (identical output at the start omitted): [...] MKDIR -p .build/contrib/coccinelle/free.cocci.d SPATCH grep.c >.build/contrib/coccinelle/free.cocci.d/grep.c.patch SPATCH attr.c >.build/contrib/coccinelle/free.cocci.d/attr.c.patch SPATCH CAT .build/contrib/coccinelle/free.cocci.d/**.patch >.build/contrib/coccinelle/free.cocci.patch CP .build/contrib/coccinelle/free.cocci.patch contrib/coccinelle/free.cocci.patch I.e. we have an "SPATCH" line that makes it clear that we're running against the "{attr,grep}.c" file. The "SPATCH CAT" is then altered to correspond to it, showing that we're concatenating the "free.cocci.d/**.patch" files into one generated "free.cocci.patch" at the end. Reported-by: SZEDER Gábor <> Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-11cache-tree: add perf test comparing update and primeVictoria Dye
Add a performance test comparing the execution times of 'prime_cache_tree()' and 'cache_tree_update(_, WRITE_TREE_SILENT | WRITE_TREE_REPAIR)'. The goal of comparing these two is to identify which is the faster method for rebuilding an invalid cache tree, ultimately to remove one when both are (reundantly) called in immediate succession. Both methods are fast, so the new tests in '' must call each tested function multiple times to ensure the reported times (to 0.01s resolution) convey the differences between them. The tests compare the timing of a 'test-tool cache-tree' run as a no-op (to capture a baseline for the overhead associated with running the tool), 'cache_tree_update()', and 'prime_cache_tree()' on four scenarios: - A completely valid cache tree - A cache tree with 2 invalid paths - A cache tree with 50 invalid paths - A completely empty cache tree Example results: Test this tree ----------------------------------------------------------- 0090.2: no-op, clean 1.27(0.48+0.52) 0090.3: prime_cache_tree, clean 2.02(0.83+0.85) 0090.4: cache_tree_update, clean 1.30(0.49+0.54) 0090.5: no-op, invalidate 2 1.29(0.48+0.54) 0090.6: prime_cache_tree, invalidate 2 1.98(0.81+0.83) 0090.7: cache_tree_update, invalidate 2 2.12(0.94+0.86) 0090.8: no-op, invalidate 50 1.32(0.50+0.55) 0090.9: prime_cache_tree, invalidate 50 2.10(0.86+0.89) 0090.10: cache_tree_update, invalidate 50 2.35(1.14+0.90) 0090.11: no-op, empty 1.33(0.50+0.54) 0090.12: prime_cache_tree, empty 2.04(0.84+0.87) 0090.13: cache_tree_update, empty 2.51(1.27+0.92) These timings show that, while 'cache_tree_update()' is faster when the cache tree is completely valid, it is equal to or slower than 'prime_cache_tree()' when there are any invalid paths. Since the redundant calls are mostly in scenarios where the cache tree will be at least partially invalid (e.g., 'git reset --hard'), 'prime_cache_tree()' will likely perform better than 'cache_tree_update()' in typical cases. Helped-by: SZEDER Gábor <> Signed-off-by: Victoria Dye <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: discuss SHAttered in *_SHA{1,256} discussionÆvar Arnfjörð Bjarmason
Let's mention the SHAttered attack and more generally why we use the sha1collisiondetection backend by default, and note that for SHA-256 the user should feel free to pick any of the supported backends as far as hashing security is concerned. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: document default SHA-1 backend on OSXÆvar Arnfjörð Bjarmason
Since [1] the default SHA-1 backend on OSX has been APPLE_COMMON_CRYPTO. Per [2] we'll skip using it on anything older than Mac OS X 10.4 "Tiger"[3]. When "DC_SHA1" was made the default in [4] this interaction between it and APPLE_COMMON_CRYPTO seems to have been missed in. Ever since DC_SHA1 was "made the default" we've still used Apple's CommonCrypto instead of sha1collisiondetection on modern versions of Darwin and OSX. 1. 61067954ce1 (cache.h: eliminate SHA-1 deprecation warnings on Mac OS X, 2013-05-19) 2. 9c7a0beee09 (config.mak.uname: set NO_APPLE_COMMON_CRYPTO on older systems, 2014-08-15) 3. We could probably drop "NO_APPLE_COMMON_CRYPTO", as nobody's likely to care about such on old version of OSX anymore. But let's leave that for now. 4. e6b07da2780 (Makefile: make DC_SHA1 the default, 2017-03-17) Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile & test-tool: replace "DC_SHA1" variable with a "define"Ævar Arnfjörð Bjarmason
Address the root cause of technical debt we've been carrying since sha1collisiondetection was made the default in [1]. In a preceding commit we narrowly fixed a bug where the "DC_SHA1" variable would be unset (in combination with "NO_APPLE_COMMON_CRYPTO=" on OSX), even though we had the sha1collisiondetection library enabled. But the only reason we needed to have such a user-exposed knob went away with [1], and it's been doing nothing useful since then. We don't care if you define DC_SHA1=*, we only care that you don't ask for any other SHA-1 implementation. If it turns out that you didn't, we'll use sha1collisiondetection, whether you had "DC_SHA1" set or not. As a result of this being confusing we had e.g. [2] for cmake and the recent [3] for ci/ setting "DC_SHA1" explicitly, even though this was always a NOOP. A much simpler way to do this is to stop having the Makefile and CMakeLists.txt set "DC_SHA1" to be picked up by the, let's instead add a trivial "test-tool sha1-is-sha1dc". It returns zero if we're using sha1collisiondetection, non-zero otherwise. 1. e6b07da2780 (Makefile: make DC_SHA1 the default, 2017-03-17) 2. c4b2f41b5f5 (cmake: support for testing git with ctest, 2020-06-26) 3. 1ad5c3df35a (ci: use DC_SHA1=YesPlease on osx-clang job for CI, 2022-10-20) Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: document SHA-1 and SHA-256 default and selection orderÆvar Arnfjörð Bjarmason
For the *_SHA1 and *_SHA256 flags we've discussed the various flags, but not the fact that when you define multiple flags we'll pick one. Which one we pick depends on the order they're listed in the Makefile, which differed from the order we discussed them in this documentation. Let's be explicit about how we select these, and re-arrange the listings so that they're listed in the priority order we've picked. I'd personally prefer that the selection was more explicit, and that we'd error out if conflicting flags were provided, but per the discussion downhtread of[1] the consensus was to keep theses semantics. This behavior makes it easier to e.g. integrate with autoconf-like systems, where the configuration can provide everything it can support, and Git is tasked with picking the first one it prefers. 1. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: document default SHA-256 backendÆvar Arnfjörð Bjarmason
Since 27dc04c5450 (sha256: add an SHA-256 implementation using libgcrypt, 2018-11-14) we've claimed to support a BLK_SHA256 flag, but there's no such SHA-256 backend. Instead we fall back on adding "sha256/block/sha256.o" to "LIB_OBJS" and adding "-DSHA256_BLK" to BASIC_CFLAGS. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: rephrase the discussion of *_SHA1 knobsÆvar Arnfjörð Bjarmason
In the preceding commit the discussion of the *_SHA1 knobs was left as-is to benefit from a smaller diff, but since we're changing these let's use the same phrasing we use for most other knobs. E.g. "define X", not "define X environment variable", and get rid of the "when running make to link with" entirely. Furthermore the discussion of DC_SHA1* options is now under a "Options for the sha1collisiondetection implementation" heading, so we don't need to clarify that these options go along with DC_SHA1=Y, so let's rephrase them accordingly. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: create and use sections for "define" flag listingÆvar Arnfjörð Bjarmason
Since the "Define ..." template of comments at the top of the Makefile was started in 5bdac8b3269 ([PATCH] Improve the compilation-time settings interface, 2005-07-29) we've had a lot more flags added, including flags that come in "groups". Not having any obvious structure to the >500 line comment at the top of the Makefile has made it hard to follow. This change is almost entirely a move-only change, the two paragraphs at the start of the first two sections are new, and so are the added sections themselves, but other than that no lines are changed, only moved. We now list Makefile-only flags at the start, followed by stand-alone flags, and then cover "optional library" flags in their respective groups, followed by SHA-1 and SHA-256 flags, and finally DEVELOPER-specific flags. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: correct DC_SHA1 documentationÆvar Arnfjörð Bjarmason
The claim that DC_SHA1 takes priority over other *_SHA1 knobs was true when it was added in [1], But that hasn't been the case since it was made the fallback default in [2]. We should be making it not only the default, but something that takes priority over other *_SHA1 knobs, but that's outside the scope of this change. For now let's correct the documentation to match reality. Let's also remove the "unconditionally enable" wording, per the above the enabling of "DC_SHA1" is conditional on these other flags. The "Define DC_SHA1" here is also a lie, actually it's "we don't care if you define DC_SHA1, just don't define anything else", but that's a more general issue that'll be addressed in a subsequent commit. Let's first stop pretending that this setting (which we actually don't even use) takes priority over anything else. 1. 8325e43b82d (Makefile: add DC_SHA1 knob, 2017-03-16) 2. e6b07da2780 (Makefile: make DC_SHA1 the default, 2017-03-17) Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-08Makefile: always (re)set DC_SHA1 on fallbackÆvar Arnfjörð Bjarmason
Fix an edge case introduced in in e6b07da2780 (Makefile: make DC_SHA1 the default, 2017-03-17), when DC_SHA1 was made the default fallback we started unconditionally adding to BASIC_CFLAGS and LIB_OBJS, so we'd use the sha1collisiondetection by default. But the "DC_SHA1" variable remained unset, so e.g.: make test DC_SHA1= T=t0013*.sh Would skip the sha1collisiondetection tests, as we'd write "DC_SHA1=''" to "GIT-BUILD-OPTIONS", but if we manually removed that test prerequisite we'd pass the test (which we couldn't if we weren't using sha1collisiondetection). So let's have the fallback assignment use the 'override' directive instead of the ":=" simply expanded variable introduced in e6b07da2780. In this case we explicitly want to override the user's choice. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03cocci: run against a generated ALL.cocciÆvar Arnfjörð Bjarmason
The preceding commits to make the "coccicheck" target incremental made it slower in some cases. As an optimization let's not have the many=many mapping of <*.cocci>=<*.[ch]>, but instead concat the <*.cocci> into an ALL.cocci, and then run one-to-many ALL.cocci=<*.[ch]>. A "make coccicheck" is now around 2x as fast as it was on "master", and around 1.5x as fast as the preceding change to make the run incremental: $ git hyperfine -L rev origin/master,HEAD~,HEAD -p 'make clean' 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' -r 3 Benchmark 1: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'origin/master Time (mean ± σ): 4.258 s ± 0.015 s [User: 27.432 s, System: 1.532 s] Range (min … max): 4.241 s … 4.268 s 3 runs Benchmark 2: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD~ Time (mean ± σ): 5.365 s ± 0.079 s [User: 36.899 s, System: 1.810 s] Range (min … max): 5.281 s … 5.436 s 3 runs Benchmark 3: make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD Time (mean ± σ): 2.725 s ± 0.063 s [User: 14.796 s, System: 0.233 s] Range (min … max): 2.667 s … 2.792 s 3 runs Summary 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD' ran 1.56 ± 0.04 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'origin/master' 1.97 ± 0.05 times faster than 'make coccicheck SPATCH=spatch COCCI_SOURCES="$(echo $(ls o*.c builtin/h*.c))"' in 'HEAD~' This can be turned off with SPATCH_CONCAT_COCCI, but as the beneficiaries of "SPATCH_CONCAT_COCCI=" would mainly be those developing the *.cocci rules themselves, let's leave this optimization on by default. For more information see my "Optimizing *.cocci rules by concat'ing them" (<>) on the mailing list. This potentially changes the results of our *.cocci rules, but as noted in that discussion it should be safe for our use. We don't name rules, or if we do their names don't conflict across our *.cocci files. To the extent that we'd have any inter-dependencies between rules this doesn't make that worse, as we'd have them now if we ran "make coccicheck", applied the results, and would then have (due to hypothetical interdependencies) suggested changes on the subsequent "make coccicheck". Our "coccicheck-test" target makes use of the ALL.cocci when running tests, e.g. when testing unused.{c,out} we test it against ALL.cocci, not unused.cocci. We thus assert (to the extent that we have test coverage) that this concatenation doesn't change the expected results of running these rules. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03Makefile: copy contrib/coccinelle/*.cocci to build/Ævar Arnfjörð Bjarmason
Change the "coccinelle" rule so that we first copy the *.cocci source in e.g. "contrib/coccinelle/strbuf.cocci" to ".build/contrib/coccinelle/strbuf.cocci" before operating on it. For now this serves as a rather pointless indirection, but prepares us for the subsequent commit where we'll be able to inject generated *.cocci files. Having the entire dependency tree live inside .build/* simplifies both the globbing we'd need to do, and any "clean" rules. It will also help for future targets which will want to act on the generated patches or the logs, e.g. targets to alert if we can't parse certain files (or, less so than usual) with "spatch", and e.g. a replacement for "ci/". Such a replacement won't care about placing the patches in the in-tree, only whether they're "OK" (and about the diff). Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03cocci: optimistically use COMPUTE_HEADER_DEPENDENCIESÆvar Arnfjörð Bjarmason
Improve the incremental rebuilding support of "coccicheck" by piggy-backing on the computed dependency information of the corresponding *.o file, rather than rebuilding all <RULE>/<FILE> pairs if either their corresponding file changes, or if any header changes. This in effect uses the same method that the "sparse" target was made to use in c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23), except that the dependency on the *.o file isn't a hard one, we check with $(wildcard) if the *.o file exists, and if so we'll depend on it. This means that the common case of: make make coccicheck Will benefit from incremental rebuilding, now changing e.g. a header will only re-run "spatch" on those those *.c files that make use of it: By depending on the *.o we piggy-back on COMPUTE_HEADER_DEPENDENCIES. See c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY, 2021-09-23) for prior art of doing that for the *.sp files. E.g.: make contrib/coccinelle/free.cocci.patch make -W column.h contrib/coccinelle/free.cocci.patch Will take around 15 seconds for the second command on my 8 core box if I didn't run "make" beforehand to create the *.o files. But around 2 seconds if I did and we have those "*.o" files. Notes about the approach of piggy-backing on *.o for dependencies: * It *is* a trade-off since we'll pay the extra cost of running the C compiler, but we're probably doing that anyway. The compiler is much faster than "spatch", so even though we need to re-compile the *.o to create the dependency info for the *.c for "spatch" it's faster (especially if using "ccache"). * There *are* use-cases where some would like to have *.o files around, but to have the "make coccicheck" ignore them. See: For those users a: make make coccicheck SPATCH_USE_O_DEPENDENCIES= Will avoid considering the *.o files. * If that *.o file doesn't exist we'll depend on an intermediate file of ours which in turn depends on $(FOUND_H_SOURCES). This covers both an initial build, or where "coccicheck" is run without running "all" beforehand, and because we run "coccicheck" on e.g. files in compat/* that we don't know how to build unless the requisite flag was provided to the Makefile. Most of the runtime of "incremental" runs is now spent on various compat/* files, i.e. we conditionally add files to COMPAT_OBJS, and therefore conflate whether we *can* compile an object and generate dependency information for it with whether we'd like to link it into our binary. Before this change the distinction didn't matter, but now one way to make this even faster on incremental builds would be to peel those concerns apart so that we can see that e.g. compat/mmap.c doesn't depend on column.h. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03cocci: make "coccicheck" rule incrementalÆvar Arnfjörð Bjarmason
Optimize the very slow "coccicheck" target to take advantage of incremental rebuilding, and fix outstanding dependency problems with the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the "--jobs" argument to "spatch(1)". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. The rule didn't use FORCE (or its equivalents) before, so a: make coccicheck make coccicheck Would report nothing to do on the second iteration. But all of our patch output depended on all $(COCCI_SOURCES) files, therefore e.g.: make -W grep.c coccicheck Would do a full re-run, i.e. a a change in a single file would force us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to pick up where we left off, or how to incrementally merge e.g. a "grep.c" change with an existing *.cocci.patch. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was initially added as a sort of poor man's dependency discovery. As we don't include other *.c files depending on other *.c files has always been broken, as could be trivially demonstrated e.g. with: make coccicheck make -W strbuf.h coccicheck However, depending on the corresponding *.c files has been doing something, namely that *if* an API change modified both *.c and *.h files we'd catch the change to the *.h we care about via the *.c being changed. For API changes that happened only via *.h files we'd do the wrong thing before this change, but e.g. for function additions (not "static inline" ones) catch the *.h change by proxy. Now we'll instead: * Create a <RULE>/<FILE> pair in the .build directory, E.g. for swap.cocci and grep.c we'll create .build/contrib/coccinelle/swap.cocci.patch/grep.c. That file is the diff we'll apply for that <RULE>-<FILE> combination, if there's no changes to me made (the common case) it'll be an empty file. * Our generated *.patch file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the <RULE>/<FILE> files for a given <RULE>. In the case discussed above of "grep.c" being changed we'll do the full "cat" every time, so they resulting *.cocci.patch will always be correct and up-to-date, even if it's "incrementally updated". See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. As before we'll: * End up generating a contrib/coccinelle/swap.cocci.patch, if we "fail" by creating a non-empty patch we'll still exit with a zero exit code. Arguably we should move to a more Makefile-native way of doing this, i.e. fail early, and if we want all of the "failed" changes we can use "make -k", but as the current "ci/" expects us to behave this way let's keep the existing behavior of exhaustively discovering all cocci changes, and only failing if spatch itself errors out. Further implementation details & notes: * Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. This could be mitigated by combining "make -jN" with SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06). There will be cases where getting rid of "SPATCH_BATCH_SIZE" makes things worse, but a from-scratch "make coccicheck" with the default of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference) is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed the CPU more work in a less staggered way. * Getting rid of "SPATCH_BATCH_SIZE" particularly helps in cases where the default of 1 yields parallelism under "make coccicheck", but then running e.g.: make -W contrib/coccinelle/swap.cocci coccicheck I.e. before that would use only one CPU core, until the user remembered to adjust "SPATCH_BATCH_SIZE" differently than the setting that makes sense when doing a non-incremental run of "make coccicheck". * Before the "make coccicheck" rule would have to clean "contrib/coccinelle/*.cocci.patch*", since we'd create "*+" and "*.log" files there. Now those are created in .build/contrib/coccinelle/, which is covered by the "cocciclean" rule already. Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". As noted upthread of [1] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/<file>.o.d and supply them with the "--include" option to spatch(1).q 1. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03cocci: split off "--all-includes" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason
Per the rationale in 7b63ea57500 (Makefile: remove mandatory "spatch" arguments from SPATCH_FLAGS, 2022-07-05) we have certain flags that are truly mandatory, such as "--sp-file" and "--patch .". The "--all-includes" flag is also critical, but per [1] we might want to ad-hoc tweak it occasionally for testing or one-offs. But being unable to set e.g. SPATCH_FLAGS="--verbose-parsing" without breaking how our "spatch" works isn't ideal, i.e. before this we'd need to know about the default include flags, and specify: SPATCH_FLAGS="--all-includes --verbose-parsing". If we were then to change the default include flag (e.g. to "--recursive-includes") in the future any such one-off commands would need to be correspondingly updated. Let's instead leave the SPATCH_FLAGS for the user, while creating a new SPATCH_INCLUDE_FLAGS to allow for ad-hoc testing of the include strategy itself. 1. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03cocci: split off include-less "tests" from SPATCH_FLAGSÆvar Arnfjörð Bjarmason
Amend the "coccicheck-test" rule added in f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) to stop using "--all-includes". The flags we'll need for the tests are different than the ones we'll need for our main source code. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03Makefile: split off SPATCH_BATCH_SIZE comment from "cocci" headingÆvar Arnfjörð Bjarmason
Split off the "; setting[...]" part of the comment added in In 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06), and restore what we had before that, which was a comment indicating that variables for the "coccicheck" target were being set here. When 960154b9c17 amended the heading to discuss SPATCH_BATCH_SIZE it left no natural place to add a new comment about other flags that preceded it. As subsequent commits will add such comments we need to split the existing comment up. The wrapping for the "SPATCH_BATCH_SIZE" is now a bit odd, but minimizes the diff size. As a subsequent commit will remove that feature altogether this is worth it. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03Makefile: have "coccicheck" re-run if flags changeÆvar Arnfjörð Bjarmason
Fix an issue with the "coccicheck" family of rules that's been here since 63f0a758a06 (add coccicheck make target, 2016-09-15), unlike e.g. "make grep.o" we wouldn't re-run it when $(SPATCH) or $(SPATCH_FLAGS) changed. To test new flags we needed to first do a "make cocciclean". This now uses the same (copy/pasted) pattern as other "DEFINES" rules. As a result we'll re-run properly. This can be demonstrated e.g. on the issue noted in [1]: $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c V=1 [...] SPATCH contrib/coccinelle/xcalloc.cocci $ make contrib/coccinelle/xcalloc.cocci.patch COCCI_SOURCES=promisor-remote.c SPATCH_FLAGS="--all-includes --recursive-includes" * new spatch flags SPATCH contrib/coccinelle/xcalloc.cocci SPATCH result: contrib/coccinelle/xcalloc.cocci.patch $ 1. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03Makefile: add ability to TAB-complete cocci *.patch rulesÆvar Arnfjörð Bjarmason
Declare the contrib/coccinelle/<rule>.cocci.patch rules in such a way as to allow TAB-completion, and slightly optimize the Makefile by cutting down on the number of $(wildcard) in favor of defining "coccicheck" and "coccicheck-pending" in terms of the same incrementally filtered list. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-11-03Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)Ævar Arnfjörð Bjarmason
In f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) we abbreviated "_TEST" to "_T" to have it align with the rest of the "="'s above it. Subsequent commits will add more QUIET_SPATCH_* variables, so let's stop abbreviating this, and indent it in preparation for adding more of these variables. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Taylor Blau <>
2022-10-31Merge branch 'ds/bundle-uri-3'Taylor Blau
Define the logical elements of a "bundle list", data structure to store them in-core, format to transfer them, and code to parse them. * ds/bundle-uri-3: bundle-uri: suppress stderr from remote-https bundle-uri: quiet failed unbundlings bundle: add flags to verify_bundle() bundle-uri: fetch a list of bundles bundle: properly clear all revision flags bundle-uri: limit recursion depth for bundle lists bundle-uri: parse bundle list in config format bundle-uri: unit test "key=value" parsing bundle-uri: create "key=value" line parsing bundle-uri: create base key-value pair parsing bundle-uri: create bundle_list struct and helpers bundle-uri: use plain string in find_temp_filename()
2022-10-31Merge branch 'jh/trace2-timers-and-counters'Taylor Blau
Two new facilities, "timer" and "counter", are introduced to the trace2 API. * jh/trace2-timers-and-counters: trace2: add global counter mechanism trace2: add stopwatch timers trace2: convert ctx.thread_name from strbuf to pointer trace2: improve thread-name documentation in the thread-context trace2: rename the thread_name argument to trace2_thread_start api-trace2.txt: elminate section describing the public trace2 API tr2tls: clarify TLS terminology trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx
2022-10-27Merge branch 'js/cmake-updates'Junio C Hamano
Update to build procedure with VS using CMake/CTest. * js/cmake-updates: cmake: increase time-out for a long-running test cmake: avoid editing t/ add -p: avoid ambiguous signed/unsigned comparison cmake: copy the merge tools for testing cmake: make it easier to diagnose regressions in CTest runs
2022-10-24trace2: add global counter mechanismJeff Hostetler
Add global counters mechanism to Trace2. The Trace2 counters mechanism adds the ability to create a set of global counter variables and an API to increment them efficiently. Counters can optionally report per-thread usage in addition to the sum across all threads. Counter events are emitted to the Trace2 logs when a thread exits and at process exit. Counters are an alternative to `data` and `data_json` events. Counters are useful when you want to measure something across the life of the process, when you don't want per-measurement events for performance reasons, when the data does not fit conveniently within a region, or when your control flow does not easily let you write the final total. For example, you might use this to report the number of calls to unzip() or the number of de-delta steps during a checkout. Signed-off-by: Jeff Hostetler <> Signed-off-by: Junio C Hamano <>
2022-10-24trace2: add stopwatch timersJeff Hostetler
Add stopwatch timer mechanism to Trace2. Timers are an alternative to Trace2 Regions. Regions are useful for measuring the time spent in various computation phases, such as the time to read the index, time to scan for unstaged files, time to scan for untracked files, and etc. However, regions are not appropriate in all places. For example, during a checkout, it would be very inefficient to use regions to measure the total time spent inflating objects from the ODB from across the entire lifetime of the process; a per-unzip() region would flood the output and significantly slow the command; and some form of post-processing would be requried to compute the time spent in unzip(). Timers can be used to measure a series of timer intervals and emit a single summary event (at thread and/or process exit). Signed-off-by: Jeff Hostetler <> Signed-off-by: Junio C Hamano <>
2022-10-19cmake: avoid editing t/test-lib.shJohannes Schindelin
In 7f5397a07c6c (cmake: support for testing git when building out of the source tree, 2020-06-26), we implemented support for running Git's test scripts even after building Git in a different directory than the source directory. The way we did this was to edit the file `t/` to override `GIT_BUILD_DIR` to point somewhere else than the parent of the `t/` directory. This is unideal because it always leaves a tracked file marked as modified, and it is all too easy to commit that change by mistake. Let's change the strategy by teaching `t/` to detect the presence of a file called `GIT-BUILD-DIR` in the source directory. If it exists, the contents are interpreted as the location to the _actual_ build directory. We then write this file as part of the CTest definition. To support building Git via a regular `make` invocation after building it using CMake, we ensure that the `GIT-BUILD-DIR` file is deleted (for convenience, this is done as part of the Makefile rule that is already run with every `make` invocation to ensure that `GIT-BUILD-OPTIONS` is up to date). Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2022-10-19Makefile: force -O0 when compiling with SANITIZE=leakJeff King
Compiling with -O2 can interact badly with LSan's leak-checker, causing false positives. Imagine a simplified example like: char *str = allocate_some_string(); if (some_func(str) < 0) die("bad str"); free(str); The compiler may eliminate "str" as a stack variable, and just leave it in a register. The register is preserved through most of the function, including across the call to some_func(), since we'd eventually need to free it. But because die() is marked with NORETURN, the compiler knows that it doesn't need to save registers, and just clobbers it. When die() eventually exits, the leak-checker runs. It looks in registers and on the stack for any reference to the memory allocated by str (which would indicate that it's not leaked), but can't find one. So it reports it as a leak. Neither system is wrong, really. The C standard (mostly section defines an abstract machine, and compilers are allowed to modify the program as long as the observable behavior of that abstract machine is unchanged. Looking at random memory values on the stack is undefined behavior, and not something that the optimizer needs to support. But there really isn't any other way for a leak checker to work; it inherently has to do undefined things like scouring memory for pointers. So the two things are inherently at odds with each other. We can't fix it by changing the code, because from the perspective of the program running in an abstract machine, there is no leak. This has caused real false positives in the past, like: - - - This patch makes those go away by forcing -O0 when compiling with LSan. There are a few ways we could do this: - we could just teach the linux-leaks CI job to set -O0. That's the smallest change, and means we wouldn't get spurious CI failures. But it doesn't help people looking for leaks manually or in a specific test (and because the problem depends on the vagaries of the optimizer, investigating these can waste a lot of time in head-scratching as the problem comes and goes) - we default to -O2 in CFLAGS; we could pull this out to a separate variable ("-O$(O)" or something) and modify "O" when LSan is in use. This is the most flexible, in that you could still build with "make O=2 SANITIZE=leak" if you really wanted to (say, for experimenting). But it would also fail to kick in if the user defines their own CFLAGS variable, which again leads to head-scratching. - we can just stick -O0 into BASIC_CFLAGS when enabling LSan. Since this comes after the user-provided CFLAGS, it will override any previous -O setting found there. This is more foolproof, albeit less flexible. If you want to experiment with an optimized leak-checking build, you'll have to put "-O2 -fsanitize=leak" into CFLAGS manually, rather than using our SANITIZE=leak Makefile magic. Since the final one is the least likely to break in normal use, this patch uses that approach. The resulting build is a little slower, of course, but since LSan is already about 2x slower than a regular build, another 10% slowdown isn't that big a deal. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2022-10-17Merge branch 'dd/document-runtime-prefix-better'Junio C Hamano
Update comment in the Makefile about the RUNTIME_PREFIX config knob. * dd/document-runtime-prefix-better: Makefile: clarify runtime relative gitexecdir
2022-10-17Merge branch 'ed/fsmonitor-on-networked-macos'Junio C Hamano
By default, use of fsmonitor on a repository on networked filesystem is disabled. Add knobs to make it workable on macOS. * ed/fsmonitor-on-networked-macos: fsmonitor: fix leak of warning message fsmonitor: add documentation for allowRemote and socketDir options fsmonitor: check for compatability before communicating with fsmonitor fsmonitor: deal with synthetic firmlinks on macOS fsmonitor: avoid socket location check if using hook fsmonitor: relocate socket file if .git directory is remote fsmonitor: refactor filesystem checks to common interface
2022-10-12bundle-uri: unit test "key=value" parsingÆvar Arnfjörð Bjarmason
Create a new 'test-tool bundle-uri' test helper. This helper will assist in testing logic deep in the bundle URI feature. This change introduces the 'parse-key-values' subcommand, which parses an input file as a list of lines. These are fed into bundle_uri_parse_line() to test how we construct a 'struct bundle_list' from that data. The list is then output to stdout as if the key-value pairs were a Git config file. We use an input file instead of stdin because of a future change to parse in config-file format that works better as an input file. Co-authored-by: Derrick Stolee <> Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Derrick Stolee <> Signed-off-by: Junio C Hamano <>
2022-10-08Merge branch 'ac/fuzzers'Junio C Hamano
Source file shuffling. * ac/fuzzers: fuzz: reorganise the path for existing oss-fuzz fuzzers
2022-10-08Merge branch 'ds/use-platform-regex-on-macos'Junio C Hamano
With a bit of header twiddling, use the native regexp library on macOS instead of the compat/ one. * ds/use-platform-regex-on-macos: grep: fix multibyte regex handling under macOS
2022-10-06Makefile: clarify runtime relative gitexecdirĐoàn Trần Công Danh
"git" built with RUNTIME_PREFIX flag turned on could figure out gitexecdir and other paths as relative to "git" executable. However, in the section specifies gitexecdir, RUNTIME_PREFIX wasn't mentioned, thus users may wrongly assume that "git" always locates gitexecdir as relative path to the executable. Let's clarify that only "git" built with RUNTIME_PREFIX will locate gitexecdir as relative path. Signed-off-by: Đoàn Trần Công Danh <> Signed-off-by: Junio C Hamano <>
2022-10-05fsmonitor: relocate socket file if .git directory is remoteEric DeCosta
If the .git directory is on a remote filesystem, create the socket file in 'fsmonitor.socketDir' if it is defined, else create it in $HOME. Signed-off-by: Eric DeCosta <> Signed-off-by: Junio C Hamano <>
2022-10-05fsmonitor: refactor filesystem checks to common interfaceEric DeCosta
Provide a common interface for getting basic filesystem information including filesystem type and whether the filesystem is remote. Refactor existing code for getting basic filesystem info and detecting remote file systems to the new interface. Refactor filesystem checks to leverage new interface. For macOS, error-out if the Unix Domain socket (UDS) file is on a remote filesystem. Signed-off-by: Eric DeCosta <> Signed-off-by: Junio C Hamano <>
2022-09-20version: fix builtin linking & documentationVictoria Dye
Like most builtins, 'version' is documented in a corresponding 'Documentation/git-version.txt' and can be invoked with 'git version'. However, the 'check-docs' Makefile target showed that it was "removed but documented: git-version." This was cause by the fact that it is not built as a standalone 'git-version' executable, therefore appearing "removed" to 'check-docs'. Without a precedent for documented builtins that aren't built into an executable *or* any clear reason why a standalone 'git-version' shouldn't exist, the 'check-docs' error appears to correctly identify an issue. To correct that mismatch, add 'git-version' to the 'BUILT_INS' list in the root Makefile (indicating that the 'cmd_version()' function appears in a file that is *not* 'builtin/version.c'). Additionally, to avoid the "no link" message in 'check-docs', list 'git-version' as an "ancilliaryinterrogator" (like 'git help') in 'command-list.txt'. Signed-off-by: Victoria Dye <> Signed-off-by: Junio C Hamano <>
2022-09-19Merge branch 'vd/scalar-to-main'Junio C Hamano
Hoist the remainder of "scalar" out of contrib/ to the main part of the codebase. * vd/scalar-to-main: Documentation/technical: include Scalar technical doc t/perf: add 'GIT_PERF_USE_SCALAR' run option t/perf: add Scalar performance tests scalar-clone: add test coverage scalar: add to 'git help -a' command list scalar: implement the `help` subcommand git help: special-case `scalar` scalar: include in standard Git build & installation scalar: fix command documentation section header
2022-09-19fuzz: reorganise the path for existing oss-fuzz fuzzersArthur Chan
In order to provide a better organisation for oss-fuzz fuzzers and to avoid top-level clustters in the git repository when more fuzzers are introduced, move the existing fuzzer-related sources to their own oss-fuzz/ hierarchy. Grouping the fuzzers into their own directory, separate their application on fuzz-testing from the core functionalities of the git code, prvides better and tidier structure the oss-fuzz fuzzing library to manage, locate, build and execute those fuzzers for fuzz-testing purposes in future development. Signed-off-by: Arthur Chan <> Signed-off-by: Junio C Hamano <>
2022-09-13Merge branch 'ab/submodule-helper-prep'Junio C Hamano
Code clean-up of "git submodule--helper". * ab/submodule-helper-prep: (33 commits) submodule--helper: fix bad config API usage submodule--helper: libify even more "die" paths for module_update() submodule--helper: libify more "die" paths for module_update() submodule--helper: check repo{_submodule,}_init() return values submodule--helper: libify "must_die_on_failure" code paths (for die) submodule--helper update: don't override 'checkout' exit code submodule--helper: libify "must_die_on_failure" code paths submodule--helper: libify determine_submodule_update_strategy() submodule--helper: don't exit() on failure, return submodule--helper: use "code" in run_update_command() submodule API: don't handle SM_..{UNSPECIFIED,COMMAND} in to_string() submodule--helper: don't call submodule_strategy_to_string() in BUG() submodule--helper: add missing braces to "else" arm submodule--helper: return "ret", not "1" from update_submodule() submodule--helper: rename "int res" to "int ret" submodule--helper: don't redundantly check "else if (res)" submodule--helper: refactor "errmsg_str" to be a "struct strbuf" submodule--helper: add "const" to passed "struct update_data" submodule--helper: add "const" to copy of "update_data" submodule--helper: add "const" to passed "module_clone_data" ...
2022-09-09Merge branch 'ab/retire-ppc-sha1'Junio C Hamano
Remove the assembly version of SHA-1 implementation for PPC. * ab/retire-ppc-sha1: Makefile: use $(OBJECTS) instead of $(C_OBJ) Makefile + hash.h: remove PPC_SHA1 implementation
2022-09-02scalar: include in standard Git build & installationVictoria Dye
Move 'scalar' out of 'contrib/' and into the root of the Git tree. The goal of this change is to build 'scalar' as part of the standard Git build & install processes. This patch includes both the physical move of Scalar's files out of 'contrib/' ('scalar.c', 'scalar.txt', and ''), and the changes to the build definitions in 'Makefile' and 'CMakelists.txt' to accommodate the new program. At a high level, Scalar is built so that: - there is a 'scalar-objs' target (similar to those created in 029bac01a8 (Makefile: add {program,xdiff,test,git,fuzz}-objs & objects targets, 2021-02-23)) for debugging purposes. - it appears in the root of the install directory (rather than the gitexecdir). - it is included in the 'bin-wrappers/' directory for use in tests. - it receives a platform-specific executable suffix (e.g., '.exe'), if applicable. - 'scalar.txt' is installed as 'man1' documentation. - the 'clean' target removes the 'scalar' executable. Additionally, update the root level '.gitignore' file to ignore the Scalar executable. Signed-off-by: Victoria Dye <> Signed-off-by: Junio C Hamano <>
2022-09-02submodule--helper: move "is-active" to a test-toolÆvar Arnfjörð Bjarmason
Create a new "test-tool submodule" and move the "is-active" subcommand over to it. It was added in 5c2bd8b77ae (submodule--helper: add is-active subcommand, 2017-03-16), since a452128a36c (submodule--helper: introduce add-config subcommand, 2021-08-06) it hasn't been used by Since we're creating a command dispatch similar to test-tool.c itself let's split out the "struct test_cmd" into a new test-tool-utils.h, which both this new code and test-tool.c itself can use. Signed-off-by: Ævar Arnfjörð Bjarmason <> Reviewed-by: Glen Choo <> Signed-off-by: Junio C Hamano <>
2022-09-01Merge branch 'ds/bundle-uri-clone'Junio C Hamano
Implement "git clone --bundle-uri". * ds/bundle-uri-clone: clone: warn on failure to repo_init() clone: --bundle-uri cannot be combined with --depth bundle-uri: add support for http(s):// and file:// clone: add --bundle-uri option bundle-uri: create basic file-copy logic remote-curl: add 'get' capability
2022-08-31Makefile: use $(OBJECTS) instead of $(C_OBJ)Ævar Arnfjörð Bjarmason
In the preceding commit $(C_OBJ) added in c373991375a (Makefile: list generated object files in OBJECTS, 2010-01-26) became synonymous with $(OBJECTS). Let's avoid the indirection and use the $(OBJECTS) variable directly instead. Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Junio C Hamano <>
2022-08-31Makefile + hash.h: remove PPC_SHA1 implementationÆvar Arnfjörð Bjarmason
Remove the PPC_SHA1 implementation added in a6ef3518f9a ([PATCH] PPC assembly implementation of SHA1, 2005-04-22). When this was added Apple consumer hardware used the PPC architecture, and the implementation was intended to improve SHA-1 speed there. Since it was added we've moved to using sha1collisiondetection by default, and anyone wanting hard-rolled non-DC SHA-1 implementation can use OpenSSL's via the OPENSSL_SHA1 knob. The PPC_SHA1 originally originally targeted 32 bit PPC, and later the 64 bit PPC 970 (a.k.a. Apple PowerPC G5). See 926172c5e48 (block-sha1: improve code on large-register-set machines, 2009-08-10) for a reference about the performance on G5 (a comment in block-sha1/sha1.c being removed here). I can't get it to do anything but segfault on both the BE and LE POWER machines in the GCC compile farm[1]. Anyone who's concerned about performance on PPC these days is likely to be using the IBM POWER processors. There have been proposals to entirely remove non-sha1collisiondetection implementations from the tree[2]. I think per [3] that would be a bit overzealous. I.e. there are various set-ups git's speed is going to be more important than the relatively implausible SHA-1 collision attack, or where such attacks are entirely mitigated by other means (e.g. by incoming objects being checked with DC_SHA1). But that really doesn't apply to PPC_SHA1 in particular, which seems to have outlived its usefulness. As this gets rid of the only in-tree *.S assembly file we can remove the small bits of logic from the Makefile needed to build objects from *.S (as opposed to *.c) The code being removed here was also throwing warnings with the "-pedantic" flag, it could have been fixed as 544d93bc3b4 (block-sha1: remove use of obsolete x86 assembly, 2022-03-10) did for block-sha1/*, but as noted above let's remove it instead. 1. Tested on gcc{110,112,135,203}, a mixture of POWER [789] ppc64 and ppc64le. All segfault in anything needing object hashing (e.g. t/ when compiled with PPC_SHA1=Y. 2. 3. Acked-by: brian m. carlson" <> Signed-off-by: Ævar Arnfjörð Bjarmason <> Signed-off-by: Junio C Hamano <>
2022-08-29Merge branch 'mt/rot13-in-c'Junio C Hamano
Test portability improvements. * mt/rot13-in-c: tests: use the new C rot13-filter helper to avoid PERL prereq t0021: implementation the script in C t0021: avoid grepping for a Perl-specific string at filter output
2022-08-26grep: fix multibyte regex handling under macOSDiomidis Spinellis
The commit 29de20504e (Makefile: fix default regex settings on Darwin, 2013-05-11) fixed under Darwin (macOS) by adopting Git's regex library. However, this library is compiled with NO_MBSUPPORT, which causes git-grep to work incorrectly on multibyte (e.g. UTF-8) files. Current macOS versions pass with the native macOS regex library, which also supports multibyte characters. Adjust the Makefile to use the native regex library, and call setlocale(3) to set CTYPE according to the user's preference. The setlocale call is required on all platforms, but in platforms supporting gettext(3), setlocale was called as a side-effect of initializing gettext. Therefore, move the CTYPE setlocale call from gettext.c to common-main.c and the corresponding locale.h include into git-compat-util.h. Thanks to the global initialization of CTYPE setlocale, the test-tool regex command now works correctly with supported multibyte regexes, and is used to set the MB_REGEX test prerequisite by assessing a platform's support for them. Signed-off-by: Diomidis Spinellis <> Signed-off-by: Junio C Hamano <>