summaryrefslogtreecommitdiff
path: root/t/test-lib.sh
AgeCommit message (Collapse)Author
2024-02-07refs: introduce reftable backendPatrick Steinhardt
Due to scalability issues, Shawn Pearce has originally proposed a new "reftable" format more than six years ago [1]. Initially, this new format was implemented in JGit with promising results. Around two years ago, we have then added the "reftable" library to the Git codebase via a4bbd13be3 (Merge branch 'hn/reftable', 2021-12-15). With this we have landed all the low-level code to read and write reftables. Notably missing though was the integration of this low-level code into the Git code base in the form of a new ref backend that ties all of this together. This gap is now finally closed by introducing a new "reftable" backend into the Git codebase. This new backend promises to bring some notable improvements to Git repositories: - It becomes possible to do truly atomic writes where either all refs are committed to disk or none are. This was not possible with the "files" backend because ref updates were split across multiple loose files. - The disk space required to store many refs is reduced, both compared to loose refs and packed-refs. This is enabled both by the reftable format being a binary format, which is more compact, and by prefix compression. - We can ignore filesystem-specific behaviour as ref names are not encoded via paths anymore. This means there is no need to handle case sensitivity on Windows systems or Unicode precomposition on macOS. - There is no need to rewrite the complete refdb anymore every time a ref is being deleted like it was the case for packed-refs. This means that ref deletions are now constant time instead of scaling linearly with the number of refs. - We can ignore file/directory conflicts so that it becomes possible to store both "refs/heads/foo" and "refs/heads/foo/bar". - Due to this property we can retain reflogs for deleted refs. We have previously been deleting reflogs together with their refs to avoid file/directory conflicts, which is not necessary anymore. - We can properly enumerate all refs. With the "files" backend it is not easily possible to distinguish between refs and non-refs because they may live side by side in the gitdir. Not all of these improvements are realized with the current "reftable" backend implementation. At this point, the new backend is supposed to be a drop-in replacement for the "files" backend that is used by basically all Git repositories nowadays. It strives for 1:1 compatibility, which means that a user can expect the same behaviour regardless of whether they use the "reftable" backend or the "files" backend for most of the part. Most notably, this means we artificially limit the capabilities of the "reftable" backend to match the limits of the "files" backend. It is not possible to create refs that would end up with file/directory conflicts, we do not retain reflogs, we perform stricter-than-necessary checks. This is done intentionally due to two main reasons: - It makes it significantly easier to land the "reftable" backend as tests behave the same. It would be tough to argue for each and every single test that doesn't pass with the "reftable" backend. - It ensures compatibility between repositories that use the "files" backend and repositories that use the "reftable" backend. Like this, hosters can migrate their repositories to use the "reftable" backend without causing issues for clients that use the "files" backend in their clones. It is expected that these artificial limitations may eventually go away in the long term. Performance-wise things very much depend on the actual workload. The following benchmarks compare the "files" and "reftable" backends in the current version: - Creating N refs in separate transactions shows that the "files" backend is ~50% faster. This is not surprising given that creating a ref only requires us to create a single loose ref. The "reftable" backend will also perform auto compaction on updates. In real-world workloads we would likely also want to perform pack loose refs, which would likely change the picture. Benchmark 1: update-ref: create refs sequentially (refformat = files, refcount = 1) Time (mean ± σ): 2.1 ms ± 0.3 ms [User: 0.6 ms, System: 1.7 ms] Range (min … max): 1.8 ms … 4.3 ms 133 runs Benchmark 2: update-ref: create refs sequentially (refformat = reftable, refcount = 1) Time (mean ± σ): 2.7 ms ± 0.1 ms [User: 0.6 ms, System: 2.2 ms] Range (min … max): 2.4 ms … 2.9 ms 132 runs Benchmark 3: update-ref: create refs sequentially (refformat = files, refcount = 1000) Time (mean ± σ): 1.975 s ± 0.006 s [User: 0.437 s, System: 1.535 s] Range (min … max): 1.969 s … 1.980 s 3 runs Benchmark 4: update-ref: create refs sequentially (refformat = reftable, refcount = 1000) Time (mean ± σ): 2.611 s ± 0.013 s [User: 0.782 s, System: 1.825 s] Range (min … max): 2.597 s … 2.622 s 3 runs Benchmark 5: update-ref: create refs sequentially (refformat = files, refcount = 100000) Time (mean ± σ): 198.442 s ± 0.241 s [User: 43.051 s, System: 155.250 s] Range (min … max): 198.189 s … 198.670 s 3 runs Benchmark 6: update-ref: create refs sequentially (refformat = reftable, refcount = 100000) Time (mean ± σ): 294.509 s ± 4.269 s [User: 104.046 s, System: 190.326 s] Range (min … max): 290.223 s … 298.761 s 3 runs - Creating N refs in a single transaction shows that the "files" backend is significantly slower once we start to write many refs. The "reftable" backend only needs to update two files, whereas the "files" backend needs to write one file per ref. Benchmark 1: update-ref: create many refs (refformat = files, refcount = 1) Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.4 ms, System: 1.4 ms] Range (min … max): 1.8 ms … 2.6 ms 151 runs Benchmark 2: update-ref: create many refs (refformat = reftable, refcount = 1) Time (mean ± σ): 2.5 ms ± 0.1 ms [User: 0.7 ms, System: 1.7 ms] Range (min … max): 2.4 ms … 3.4 ms 148 runs Benchmark 3: update-ref: create many refs (refformat = files, refcount = 1000) Time (mean ± σ): 152.5 ms ± 5.2 ms [User: 19.1 ms, System: 133.1 ms] Range (min … max): 148.5 ms … 167.8 ms 15 runs Benchmark 4: update-ref: create many refs (refformat = reftable, refcount = 1000) Time (mean ± σ): 58.0 ms ± 2.5 ms [User: 28.4 ms, System: 29.4 ms] Range (min … max): 56.3 ms … 72.9 ms 40 runs Benchmark 5: update-ref: create many refs (refformat = files, refcount = 1000000) Time (mean ± σ): 152.752 s ± 0.710 s [User: 20.315 s, System: 131.310 s] Range (min … max): 152.165 s … 153.542 s 3 runs Benchmark 6: update-ref: create many refs (refformat = reftable, refcount = 1000000) Time (mean ± σ): 51.912 s ± 0.127 s [User: 26.483 s, System: 25.424 s] Range (min … max): 51.769 s … 52.012 s 3 runs - Deleting a ref in a fully-packed repository shows that the "files" backend scales with the number of refs. The "reftable" backend has constant-time deletions. Benchmark 1: update-ref: delete ref (refformat = files, refcount = 1) Time (mean ± σ): 1.7 ms ± 0.1 ms [User: 0.4 ms, System: 1.2 ms] Range (min … max): 1.6 ms … 2.1 ms 316 runs Benchmark 2: update-ref: delete ref (refformat = reftable, refcount = 1) Time (mean ± σ): 1.8 ms ± 0.1 ms [User: 0.4 ms, System: 1.3 ms] Range (min … max): 1.7 ms … 2.1 ms 294 runs Benchmark 3: update-ref: delete ref (refformat = files, refcount = 1000) Time (mean ± σ): 2.0 ms ± 0.1 ms [User: 0.5 ms, System: 1.4 ms] Range (min … max): 1.9 ms … 2.5 ms 287 runs Benchmark 4: update-ref: delete ref (refformat = reftable, refcount = 1000) Time (mean ± σ): 1.9 ms ± 0.1 ms [User: 0.5 ms, System: 1.3 ms] Range (min … max): 1.8 ms … 2.1 ms 217 runs Benchmark 5: update-ref: delete ref (refformat = files, refcount = 1000000) Time (mean ± σ): 229.8 ms ± 7.9 ms [User: 182.6 ms, System: 46.8 ms] Range (min … max): 224.6 ms … 245.2 ms 6 runs Benchmark 6: update-ref: delete ref (refformat = reftable, refcount = 1000000) Time (mean ± σ): 2.0 ms ± 0.0 ms [User: 0.6 ms, System: 1.3 ms] Range (min … max): 2.0 ms … 2.1 ms 3 runs - Listing all refs shows no significant advantage for either of the backends. The "files" backend is a bit faster, but not by a significant margin. When repositories are not packed the "reftable" backend outperforms the "files" backend because the "reftable" backend performs auto-compaction. Benchmark 1: show-ref: print all refs (refformat = files, refcount = 1, packed = true) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.5 ms … 2.0 ms 1729 runs Benchmark 2: show-ref: print all refs (refformat = reftable, refcount = 1, packed = true) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.5 ms … 1.8 ms 1816 runs Benchmark 3: show-ref: print all refs (refformat = files, refcount = 1000, packed = true) Time (mean ± σ): 4.3 ms ± 0.1 ms [User: 0.9 ms, System: 3.3 ms] Range (min … max): 4.1 ms … 4.6 ms 645 runs Benchmark 4: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = true) Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.0 ms, System: 3.3 ms] Range (min … max): 4.2 ms … 5.9 ms 643 runs Benchmark 5: show-ref: print all refs (refformat = files, refcount = 1000000, packed = true) Time (mean ± σ): 2.537 s ± 0.034 s [User: 0.488 s, System: 2.048 s] Range (min … max): 2.511 s … 2.627 s 10 runs Benchmark 6: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = true) Time (mean ± σ): 2.712 s ± 0.017 s [User: 0.653 s, System: 2.059 s] Range (min … max): 2.692 s … 2.752 s 10 runs Benchmark 7: show-ref: print all refs (refformat = files, refcount = 1, packed = false) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.5 ms … 1.9 ms 1834 runs Benchmark 8: show-ref: print all refs (refformat = reftable, refcount = 1, packed = false) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.4 ms … 2.0 ms 1840 runs Benchmark 9: show-ref: print all refs (refformat = files, refcount = 1000, packed = false) Time (mean ± σ): 13.8 ms ± 0.2 ms [User: 2.8 ms, System: 10.8 ms] Range (min … max): 13.3 ms … 14.5 ms 208 runs Benchmark 10: show-ref: print all refs (refformat = reftable, refcount = 1000, packed = false) Time (mean ± σ): 4.5 ms ± 0.2 ms [User: 1.2 ms, System: 3.3 ms] Range (min … max): 4.3 ms … 6.2 ms 624 runs Benchmark 11: show-ref: print all refs (refformat = files, refcount = 1000000, packed = false) Time (mean ± σ): 12.127 s ± 0.129 s [User: 2.675 s, System: 9.451 s] Range (min … max): 11.965 s … 12.370 s 10 runs Benchmark 12: show-ref: print all refs (refformat = reftable, refcount = 1000000, packed = false) Time (mean ± σ): 2.799 s ± 0.022 s [User: 0.735 s, System: 2.063 s] Range (min … max): 2.769 s … 2.836 s 10 runs - Printing a single ref shows no real difference between the "files" and "reftable" backends. Benchmark 1: show-ref: print single ref (refformat = files, refcount = 1) Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.4 ms, System: 1.0 ms] Range (min … max): 1.4 ms … 1.8 ms 1779 runs Benchmark 2: show-ref: print single ref (refformat = reftable, refcount = 1) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.4 ms … 2.5 ms 1753 runs Benchmark 3: show-ref: print single ref (refformat = files, refcount = 1000) Time (mean ± σ): 1.5 ms ± 0.1 ms [User: 0.3 ms, System: 1.1 ms] Range (min … max): 1.4 ms … 1.9 ms 1840 runs Benchmark 4: show-ref: print single ref (refformat = reftable, refcount = 1000) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.5 ms … 2.0 ms 1831 runs Benchmark 5: show-ref: print single ref (refformat = files, refcount = 1000000) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.5 ms … 2.1 ms 1848 runs Benchmark 6: show-ref: print single ref (refformat = reftable, refcount = 1000000) Time (mean ± σ): 1.6 ms ± 0.1 ms [User: 0.4 ms, System: 1.1 ms] Range (min … max): 1.5 ms … 2.1 ms 1762 runs So overall, performance depends on the usecases. Except for many sequential writes the "reftable" backend is roughly on par or significantly faster than the "files" backend though. Given that the "files" backend has received 18 years of optimizations by now this can be seen as a win. Furthermore, we can expect that the "reftable" backend will grow faster over time when attention turns more towards optimizations. The complete test suite passes, except for those tests explicitly marked to require the REFFILES prerequisite. Some tests in t0610 are marked as failing because they depend on still-in-flight bug fixes. Tests can be run with the new backend by setting the GIT_TEST_DEFAULT_REF_FORMAT environment variable to "reftable". There is a single known conceptual incompatibility with the dumb HTTP transport. As "info/refs" SHOULD NOT contain the HEAD reference, and because the "HEAD" file is not valid anymore, it is impossible for the remote client to figure out the default branch without changing the protocol. This shortcoming needs to be handled in a subsequent patch series. As the reftable library has already been introduced a while ago, this commit message will not go into the details of how exactly the on-disk format works. Please refer to our preexisting technical documentation at Documentation/technical/reftable for this. [1]: https://public-inbox.org/git/CAJo=hJtyof=HRy=2sLP0ng0uZ4=S-DpZ5dR1aF+VHVETKG20OQ@mail.gmail.com/ Original-idea-by: Shawn Pearce <spearce@spearce.org> Based-on-patch-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-29test-lib: check for TEST_PASSES_SANITIZE_LEAKRubén Justo
TEST_PASSES_SANITIZE_LEAK must be set before sourcing test-lib.sh, as we say in t/README: GIT_TEST_PASSING_SANITIZE_LEAK=true skips those tests that haven't declared themselves as leak-free by setting "TEST_PASSES_SANITIZE_LEAK=true" before sourcing "test-lib.sh". This test mode is used by the "linux-leaks" CI target. GIT_TEST_PASSING_SANITIZE_LEAK=check checks that our "TEST_PASSES_SANITIZE_LEAK=true" markings are current. Rather than skipping those tests that haven't set "TEST_PASSES_SANITIZE_LEAK=true" before sourcing "test-lib.sh" this mode runs them with "--invert-exit-code". This is used to check that there's a one-to-one mapping between "TEST_PASSES_SANITIZE_LEAK=true" and those tests that pass under "SANITIZE=leak". This is especially useful when testing a series that fixes various memory leaks with "git rebase -x". In a recent commit we fixed a test where it was set after sourcing test-lib.sh, leading to confusing results. To prevent future oversights, let's add a simple check to ensure the value for TEST_PASSES_SANITIZE_LEAK remains unchanged at test_done(). Signed-off-by: Rubén Justo <rjusto@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-02t: introduce GIT_TEST_DEFAULT_REF_FORMAT envvarPatrick Steinhardt
Introduce a new GIT_TEST_DEFAULT_REF_FORMAT environment variable that lets developers run the test suite with a different default ref format without impacting the ref format used by non-test Git invocations. This is modeled after GIT_TEST_DEFAULT_OBJECT_FORMAT, which does the same thing for the repository's object format. Adapt the setup of the `REFFILES` test prerequisite to be conditionally set based on the default ref format. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-02setup: introduce "extensions.refStorage" extensionPatrick Steinhardt
Introduce a new "extensions.refStorage" extension that allows us to specify the ref storage format used by a repository. For now, the only supported format is the "files" format, but this list will likely soon be extended to also support the upcoming "reftable" format. There have been discussions on the Git mailing list in the past around how exactly this extension should look like. One alternative [1] that was discussed was whether it would make sense to model the extension in such a way that backends are arbitrarily stackable. This would allow for a combined value of e.g. "loose,packed-refs" or "loose,reftable", which indicates that new refs would be written via "loose" files backend and compressed into "packed-refs" or "reftable" backends, respectively. It is arguable though whether this flexibility and the complexity that it brings with it is really required for now. It is not foreseeable that there will be a proliferation of backends in the near-term future, and the current set of existing formats and formats which are on the horizon can easily be configured with the much simpler proposal where we have a single value, only. Furthermore, if we ever see that we indeed want to gain the ability to arbitrarily stack the ref formats, then we can adapt the current extension rather easily. Given that Git clients will refuse any unknown value for the "extensions.refStorage" extension they would also know to ignore a stacked "loose,packed-refs" in the future. So let's stick with the easy proposal for the time being and wire up the extension. [1]: <pull.1408.git.1667846164.gitgitgadget@gmail.com> Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-02t: introduce DEFAULT_REPO_FORMAT prereqPatrick Steinhardt
A limited number of tests require repositories to have the default repository format or otherwise they would fail to run, e.g. because they fail to detect the correct hash function. While the hash function is the only extension right now that creates problems like this, we are about to add a second extension for the ref format. Introduce a new DEFAULT_REPO_FORMAT prereq that can easily be amended whenever we add new format extensions. Next to making any such changes easier on us, the prerequisite's name should also help to clarify the intent better. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-12-18Merge branch 'js/update-urls-in-doc-and-comment'Junio C Hamano
Stale URLs have been updated to their current counterparts (or archive.org) and HTTP links are replaced with working HTTPS links. * js/update-urls-in-doc-and-comment: doc: refer to internet archive doc: update links for andre-simon.de doc: switch links to https doc: update links to current pages
2023-11-26doc: switch links to httpsJosh Soref
These sites offer https versions of their content. Using the https versions provides some protection for users. Signed-off-by: Josh Soref <jsoref@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-09-29Merge branch 'jk/test-pass-ubsan-options-to-http-test'Junio C Hamano
UBSAN options were not propagated through the test framework to git run via the httpd, unlike ASAN options, which has been corrected. * jk/test-pass-ubsan-options-to-http-test: test-lib: set UBSAN_OPTIONS to match ASan
2023-09-21test-lib: set UBSAN_OPTIONS to match ASanJeff King
For a long time we have used ASAN_OPTIONS to set abort_on_error. This is important because we want to notice detected problems even in programs which are expected to fail. But we never did the same for UBSAN_OPTIONS. This means that our UBSan test suite runs might silently miss some cases. It also causes a more visible effect, which is that t4058 complains about unexpected "fixes" (and this is how I noticed the issue): $ make SANITIZE=undefined CC=gcc && (cd t && ./t4058-*) ... ok 8 - git read-tree does not segfault # TODO known breakage vanished ok 9 - reset --hard does not segfault # TODO known breakage vanished ok 10 - git diff HEAD does not segfault # TODO known breakage vanished The tests themselves aren't that interesting. We have a known bug where these programs segfault, and they do when compiled without sanitizers. With UBSan, when the test runs: test_might_fail git read-tree --reset base it gets: cache-tree.c:935:9: runtime error: member access within misaligned address 0x5a5a5a5a5a5a5a5a for type 'struct cache_entry', which requires 8 byte alignment So that's garbage memory which would _usually_ cause us to segfault, but UBSan catches it and complains first about the alignment. That makes sense, but the weird thing is that UBSan then exits instead of aborting, so our test_might_fail call considers that an acceptable outcome and the test "passes". Curiously, this historically seems to have aborted, because I've run "make test" with UBSan many times (and so did our CI) and we never saw the problem. Even more curiously, I see an abort if I use clang with ASan and UBSan together, like: # this aborts! make SANITIZE=undefined,address CC=clang But not with just UBSan, and not with both when used with gcc: # none of these do make SANITIZE=undefined CC=gcc make SANITIZE=undefined CC=clang make SANITIZE=undefined,address CC=gcc Likewise moving to older versions of gcc (I tried gcc-11 and gcc-12 on my Debian system) doesn't abort. Nor does moving around in Git's history. Neither this test nor the relevant code have been touched in a while, and going back to v2.41.0 produces the same outcome (even though many UBSan CI runs have passed in the meantime). So _something_ changed on my system (and likely will soon on other people's, since this is stock Debian unstable), but I didn't track it further. I don't know why it ever aborted in the past, but we definitely should be explicit here and tell UBSan what we want to happen. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-09-05Merge branch 'jk/test-lsan-denoise-output'Junio C Hamano
Tests with LSan from time to time seem to emit harmless message that makes our tests unnecessarily flakey; we work it around by filtering the uninteresting output. * jk/test-lsan-denoise-output: test-lib: ignore uninteresting LSan output
2023-08-28test-lib: ignore uninteresting LSan outputJeff King
When I run the tests in leak-checking mode the same way our CI job does, like: make SANITIZE=leak \ GIT_TEST_PASSING_SANITIZE_LEAK=true \ GIT_TEST_SANITIZE_LEAK_LOG=true \ test then LSan can racily produce useless entries in the log files that look like this: ==git==3034393==Unable to get registers from thread 3034307. I think they're mostly harmless based on the source here: https://github.com/llvm/llvm-project/blob/7e0a52e8e9ef6394bb62e0b56e17fa23e7262411/compiler-rt/lib/lsan/lsan_common.cpp#L414 which reads: PtraceRegistersStatus have_registers = suspended_threads.GetRegistersAndSP(i, &registers, &sp); if (have_registers != REGISTERS_AVAILABLE) { Report("Unable to get registers from thread %llu.\n", os_id); // If unable to get SP, consider the entire stack to be reachable unless // GetRegistersAndSP failed with ESRCH. if (have_registers == REGISTERS_UNAVAILABLE_FATAL) continue; sp = stack_begin; } The program itself still runs fine and LSan doesn't cause us to abort. But test-lib.sh looks for any non-empty LSan logs and marks the test as a failure anyway, under the assumption that we simply missed the failing exit code somehow. I don't think I've ever seen this happen in the CI job, but running locally using clang-14 on an 8-core machine, I can't seem to make it through a full run of the test suite without having at least one failure. And it's a different one every time (though they do seem to often be related to packing tests, which makes sense, since that is one of our biggest users of threaded code). We can hack around this by only counting LSan log files that contain a line that doesn't match our known-uninteresting pattern. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-30tests: skip test_eval_ in internal chain-lintJeff King
To check for broken &&-chains, we run "fail_117 && $1" as a test snippet, and check the exit code. We use test_eval_ to do so, because that's the way we run the actual test. But we don't need any of its niceties, like "set -x" tracing. In fact, they hinder us, because we have to explicitly disable them. So let's skip that and use "eval" more directly, which is simpler. I had hoped it would also be faster, but it doesn't seem to produce a measurable improvement (probably because it's just running internal shell commands, with no subshells or forks). Note that there is one gotcha: even though we don't intend to run any of the commands if the &&-chain is intact, an error like this: test_expect_success 'broken' ' # this next line breaks the &&-chain true # and then this one is executed even by the linter return 1 ' means we'll "return 1" from the eval, and thus from test_run_(). We actually do notice this in test_expect_success, but only by saying "hey, this test didn't say it was OK, so it must have failed", which is not right (it should say "broken &&-chain"). We can handle this by calling test_eval_inner_() instead, which is our trick for wrapping "return" in a test snippet. But to do that, we have to push the trace code out of that inner function and into test_eval_(). This is arguably where it belonged in the first place, but it never mattered because the "inner_" function had only one caller. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-30tests: drop here-doc check from internal chain-linterJeff King
Commit 99a64e4b73c (tests: lint for run-away here-doc, 2017-03-22) tweaked the chain-lint test to catch unclosed here-docs. It works by adding an extra "echo" command after the test snippet, and checking that it is run (if it gets swallowed by a here-doc, naturally it is not run). The downside here is that we introduced an extra $() substitution, which happens in a subshell. This has a measurable performance impact when run for many tests. The tradeoff in safety was undoubtedly worth it when 99a64e4b73c was written. But since the external chainlint.pl learned to find these recently, we can just rely on it. By switching back to a simpler chain-lint, hyperfine reports a measurable speedup on t3070 (which has 1800 tests): 'HEAD' ran 1.12 ± 0.01 times faster than 'HEAD~1' Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-30tests: replace chainlint subshell with a functionJeff King
To test that we don't break the &&-chain, test-lib.sh does something like: (exit 117) && $test_commands and checks that the result is exit code 117. We don't care what that initial command is, as long as it exits with a unique code. Using "exit" works and is simple, but is a bit expensive since it requires a subshell (to avoid exiting the whole script!). This isn't usually very noticeable, but it can add up for scripts which have a large number of tests. Using "return" naively won't work here, because we'd return from the function eval-ing the snippet (and it wouldn't find &&-chain breakages). But if we further push that into its own function, it does exactly what we want, without extra subshell overhead. According to hyperfine, this produces a measurable improvement when running t3070 (which has 1800 tests, all of them quite short): 'HEAD' ran 1.09 ± 0.01 times faster than 'HEAD~1' Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-30tests: run internal chain-linter under "make test"Jeff King
Since 69b9924b875 (t/Makefile: teach `make test` and `make prove` to run chainlint.pl, 2022-09-01), we run a single chainlint.pl process for all scripts, and then instruct each individual script to run with the equivalent of --no-chain-lint, which tells them not to redundantly run the chainlint script themselves. However, this also disables the internal linter run within the shell by eval-ing "(exit 117) && $1" and confirming we get code 117. In theory the external linter produces a superset of complaints, and we don't need the internal one anymore. However, we know there is at least one case where they differ. A test like: test_expect_success 'should fail linter' ' false && sleep 2 & pid=$! && kill $pid ' is buggy (it ignores the failure from "false", because it is backgrounded along with the sleep). The internal linter catches this, but the external one doesn't (and teaching it to do so is complicated[1]). So not only does "make test" miss this problem, but it's doubly confusing because running the script standalone does complain. Let's teach the suppression in the Makefile to only turn off the external linter (which we know is redundant, as it was already run) and leave the internal one intact. I've used a new environment variable to do this here, and intentionally did not add a "--no-ext-chain-lint" option. This is an internal optimization used by the Makefile, and not something that ordinary users would need to tweak. [1] For discussion of chainlint.pl and this case, see: https://lore.kernel.org/git/CAPig+cQtLFX4PgXyyK_AAkCvg4Aw2RAC5MmLbib-aHHgTBcDuw@mail.gmail.com/ Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-03-01Merge branch 'ar/test-lib-remove-stale-comment'Junio C Hamano
Test library clean-up. * ar/test-lib-remove-stale-comment: test-lib: drop comment about test_description
2023-02-27test-lib: drop comment about test_descriptionAndrei Rybak
When a comment describing how each test file should start was added in commit [1], it was the second comment of t/test-lib.sh. The comment describes how variable "test_description" is supposed to be assigned at the top of each test file and how "test-lib.sh" should be used by sourcing it. However, even in [1], the comment was ten lines away from the usage of the variable by test-lib.sh. Since then, the comment has drifted away both from the top of the file and from the usage of the variable. The comment just sits in the middle of the initialization of the test library, surrounded by unrelated code, almost one hundred lines away from the usage of "test_description". Nobody has noticed this drift during evolution of test-lib.sh, which suggests that this comment has outlived its usefulness. The assignment of "test_description", sourcing of "test-lib.sh" by tests, and the process of writing tests in general are described in detail in "t/README". So drop the obsolete comment. An alternative solution could be to move the comment either to the top of the file, or down to the usage of variable "test_description". [1] e1970ce43a ("[PATCH 1/2] Test framework take two.", 2005-05-13) Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-02-06add: remove "add.interactive.useBuiltin" & Perl "git add--interactive"Ævar Arnfjörð Bjarmason
Since [1] first released with Git v2.37.0 the built-in version of "add -i" has been the default. That built-in implementation was added in [2], first released with Git v2.25.0. At this point enough time has passed to allow for finding any remaining bugs in this new implementation, so let's remove the fallback code. As with similar migrations for "stash"[3] and "rebase"[4] we're keeping a mention of "add.interactive.useBuiltin" in the documentation, but adding a warning() to notify any outstanding users that the built-in is now the default. As with [5] and [6] we should follow-up in the future and eventually remove that warning. 1. 0527ccb1b55 (add -i: default to the built-in implementation, 2021-11-30) 2. f83dff60a78 (Start to implement a built-in version of `git add --interactive`, 2019-11-13) 3. 8a2cd3f5123 (stash: remove the stash.useBuiltin setting, 2020-03-03) 4. d03ebd411c6 (rebase: remove the rebase.useBuiltin setting, 2019-03-18) 5. deeaf5ee077 (stash: remove documentation for `stash.useBuiltin`, 2022-01-27) 6. 9bcde4d5314 (rebase: remove transitory rebase.useBuiltin setting & env, 2021-03-23) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-23Merge branch 'ab/test-env-helper'Junio C Hamano
Remove "git env--helper" and demote it to a test-tool subcommand. * ab/test-env-helper: env-helper: move this built-in to "test-tool env-helper"
2023-01-15env-helper: move this built-in to "test-tool env-helper"Ævar Arnfjörð Bjarmason
Since [1] there has been no reason for keeping "git env--helper" a built-in. The reason it was a built-in to begin with was to support the GIT_TEST_GETTEXT_POISON mode removed in that commit. I.e. unlike the rest of "test-tool" it would potentially be called by the installed git via "git-sh-i18n.sh". As none of that applies since [1] we should stop carrying this technical debt, and move it to t/helper/*. As this mostly move-only change shows this has the nice bonus that we'll stop wasting time translating the internal-only strings it emits. Even though this was a built-in, it was intentionally never documented, see its introduction in [2]. It never saw use outside of the test suite, except for the "GIT_TEST_GETTEXT_POISON" use-case noted above. 1. d162b25f956 (tests: remove support for GIT_TEST_GETTEXT_POISON, 2021-01-20) 2. b4f207f3394 (env--helper: new undocumented builtin wrapping git_env_*(), 2019-06-21) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-08Merge branch 'js/drop-mingw-test-cmp'Junio C Hamano
Use `git diff --no-index` as a test_cmp on Windows. We'd probably need to revisit "do we really want to, and have to, lose CRLF vs LF?" later, at which time we may be able to further clean this up by replacing "git diff --no-index" with "diff -u". * js/drop-mingw-test-cmp: tests(mingw): avoid very slow `mingw_test_cmp`
2022-12-12tests(mingw): avoid very slow `mingw_test_cmp`Johannes Schindelin
When Git's test suite uses `test_cmp`, it is not actually trying to compare binary files as the name `cmp` would suggest to users familiar with Unix' tools, but the tests instead verify that actual output matches the expected text. On Unix, `cmp` works well enough for Git's purposes because only Line Feed characters are used as line endings. However, on Windows, while most tools accept Line Feeds as line endings, many tools produce Carriage Return + Line Feed line endings, including some of the tools used by the test suite (which are therefore provided via Git for Windows SDK). Therefore, `cmp` would frequently fail merely due to different line endings. To accommodate for that, the `mingw_test_cmp` function was introduced into Git's test suite to perform a line-by-line comparison that ignores line endings. This function is a Bash function that is only used on Windows, everywhere else `cmp` is used. This is a double whammy because `cmp` is fast, and `mingw_test_cmp` is slow, even more so on Windows because it is a Bash script function, and Bash scripts are known to run particularly slowly on Windows due to Bash's need for the POSIX emulation layer provided by the MSYS2 runtime. The commit message of 32ed3314c104 (t5351: avoid using `test_cmp` for binary data, 2022-07-29) provides an illuminating account of the consequences: On Windows, the platform on which Git could really use all the help it can get to improve its performance, the time spent on one entire test script was reduced from half an hour to less than half a minute merely by avoiding a single call to `mingw_test_cmp` in but a single test case. Learning the lesson to avoid shell scripting wherever possible, the Git for Windows project implemented a minimal replacement for `mingw_test_cmp` in the form of a `test-tool` subcommand that parses the input files line by line, ignoring line endings, and compares them. Essentially the same thing as `mingw_test_cmp`, but implemented in C instead of Bash. This solution served the Git for Windows project well, over years. However, when this solution was finally upstreamed, the conclusion was reached that a change to use `git diff --no-index` instead of `mingw_test_cmp` was more easily reviewed and hence should be used instead. The reason why this approach was not even considered in Git for Windows is that in 2007, there was already a motion on the table to use Git's own diff machinery to perform comparisons in Git's test suite, but it was dismissed in https://lore.kernel.org/git/xmqqbkrpo9or.fsf@gitster.g/ as undesirable because tests might potentially succeed due to bugs in the diff machinery when they should not succeed, and those bugs could therefore hide regressions that the tests try to prevent. By the time Git for Windows' `mingw-test-cmp` in C was finally contributed to the Git mailing list, reviewers agreed that the diff machinery had matured enough and should be used instead. When the concern was raised that the diff machinery, due to its complexity, would perform substantially worse than the test helper originally implemented in the Git for Windows project, a test demonstrated that these performance differences are well lost within the 100+ minutes it takes to run Git's test suite on Windows. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-10-27Merge branch 'js/cmake-updates'Junio C Hamano
Update to build procedure with VS using CMake/CTest. * js/cmake-updates: cmake: increase time-out for a long-running test cmake: avoid editing t/test-lib.sh add -p: avoid ambiguous signed/unsigned comparison cmake: copy the merge tools for testing cmake: make it easier to diagnose regressions in CTest runs
2022-10-26Merge branch 'ab/test-malloc-with-sanitize-leak' into maint-2.38Junio C Hamano
Test fix. * ab/test-malloc-with-sanitize-leak: test-lib: have SANITIZE=leak imply TEST_NO_MALLOC_CHECK
2022-10-19cmake: avoid editing t/test-lib.shJohannes Schindelin
In 7f5397a07c6c (cmake: support for testing git when building out of the source tree, 2020-06-26), we implemented support for running Git's test scripts even after building Git in a different directory than the source directory. The way we did this was to edit the file `t/test-lib.sh` to override `GIT_BUILD_DIR` to point somewhere else than the parent of the `t/` directory. This is unideal because it always leaves a tracked file marked as modified, and it is all too easy to commit that change by mistake. Let's change the strategy by teaching `t/test-lib.sh` to detect the presence of a file called `GIT-BUILD-DIR` in the source directory. If it exists, the contents are interpreted as the location to the _actual_ build directory. We then write this file as part of the CTest definition. To support building Git via a regular `make` invocation after building it using CMake, we ensure that the `GIT-BUILD-DIR` file is deleted (for convenience, this is done as part of the Makefile rule that is already run with every `make` invocation to ensure that `GIT-BUILD-OPTIONS` is up to date). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-10-10Merge branch 'ab/test-malloc-with-sanitize-leak'Junio C Hamano
Test fix. * ab/test-malloc-with-sanitize-leak: test-lib: have SANITIZE=leak imply TEST_NO_MALLOC_CHECK
2022-09-29test-lib: have SANITIZE=leak imply TEST_NO_MALLOC_CHECKÆvar Arnfjörð Bjarmason
Since 131b94a10a7 (test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34, 2022-03-04) compiling with SANITIZE=leak has missed reporting some leaks. The old MALLOC_CHECK method used before glibc 2.34 seems to have been (mostly?) compatible with it, but after 131b94a10a7 e.g. running: TEST_NO_MALLOC_CHECK=1 make SANITIZE=leak test T=t6437-submodule-merge.sh Would report a leak in builtin/commit.c, but this would not: TEST_NO_MALLOC_CHECK= make SANITIZE=leak test T=t6437-submodule-merge.sh Since the interaction is clearly breaking the SANITIZE=leak mode, let's mark them as explicitly incompatible. A related regression for SANITIZE=address was fixed in 067109a5e7d (tests: make SANITIZE=address imply TEST_NO_MALLOC_CHECK, 2022-04-09). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-09-01test-lib: replace chainlint.sed with chainlint.plEric Sunshine
By automatically invoking chainlint.sed upon each test it runs, `test_run_` in test-lib.sh ensures that broken &&-chains will be detected early as tests are modified or new are tests created since it is typical to run a test script manually (i.e. `./t1234-test-script.sh`) during test development. Now that the implementation of chainlint.pl is complete, modify test-lib.sh to invoke it automatically instead of chainlint.sed each time a test script is run. This change reduces the number of "linter" invocations from 26800+ (once per test run) down to 1050+ (once per test script), however, a subsequent change will drop the number of invocations to 1 per `make test`, thus fully realizing the benefit of the new linter. Note that the "magic exit code 117" &&-chain checker added by bb79af9d09 (t/test-lib: introduce --chain-lint option, 2015-03-20) which is built into t/test-lib.sh is retained since it has near zero-cost and (theoretically) may catch a broken &&-chain not caught by chainlint.pl. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-09-01test-lib: retire "lint harder" optimization hackEric Sunshine
`test_run_` in test-lib.sh "lints" the body of a test by sending it down a `sed chainlint.sed | grep` pipeline; this happens once for each test run by a test script. Although this pipeline may seem relatively cheap in isolation, it can become expensive when invoked 26800+ times by `make test`, once for each test run, despite the existence of only 16500+ test definitions across all tests scripts. This difference in the number of tests defined in the scripts (16500+) and the number of tests actually run by `make test` (26800+) is explained by the fact that some test scripts run a very large number of small tests, all driven by a series of functions/loops which fill in the test bodies. This means that certain test definitions are being linted repeatedly (tens or hundreds of times) unnecessarily. To avoid such unnecessary work, 2d86a96220 (t: avoid sed-based chain-linting in some expensive cases, 2021-05-13) added an optimization hack which allows individual scripts to manually suppress the unnecessary repeated linting of the same test definition. However, unlike chainlint.sed which checks a test body as the test is run, chainlint.pl checks each test definition just once, no matter how many times the test is run, thus the sort of optimization hack introduced by 2d86a96220 is no longer needed and can be retired. Therefore, revert 2d86a96220. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-15Merge branch 'pw/use-glibc-tunable-for-malloc-optim'Junio C Hamano
Avoid repeatedly running getconf to ask libc version in the test suite, and instead just as it once per script. * pw/use-glibc-tunable-for-malloc-optim: tests: cache glibc version check
2022-08-04tests: cache glibc version checkPhillip Wood
131b94a10a ("test-lib.sh: Use GLIBC_TUNABLES instead of MALLOC_CHECK_ on glibc >= 2.34", 2022-03-04) introduced a check for the version of glibc that is in use. This check is performed as part of setup_malloc_check() which is called at least once for each test. As the test involves forking `getconf` and `expr` cache the result and use that within setup_malloc_check() to avoid forking these extra processes for each test. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: have the "check" mode for SANITIZE=leak consider leak logsÆvar Arnfjörð Bjarmason
As noted in previous on-list discussions[1] we have various tests that will falsely report being leak-free because we're missing the relevant exit code from LSAN as summarized below. We should fix those issues, but in the meantime and as an additional sanity check we can and should consider our own ASAN logs before reporting that a test is leak-free. Before this compiling with SANITIZE=leak and running: ./t6407-merge-binary.sh Will exit successfully, now we'll get an error and an informative message on: GIT_TEST_SANITIZE_LEAK_LOG=true ./t6407-merge-binary.sh Even better, as noted in the updated t/README we'll now error out when combined with the "check" mode: GIT_TEST_PASSING_SANITIZE_LEAK=check \ GIT_TEST_SANITIZE_LEAK_LOG=true \ ./t4058-diff-duplicates.sh Why do we miss these leaks? Because: * We have leaks inside "test_expect_failure" blocks, which by design will not distinguish a "normal" failure from an abort() or segfault. See [1] for a discussion of it shortcomings. * We have "git" invocations outside of "test_expect_success", e.g. setup code in the main body of the test, or in test helper functions that don't use &&-chaining. * Our tests will otherwise catch segfaults and abort(), but if we invoke a command that invokes another command it needs to ferry the exit code up to us. Notably a command that e.g. might invoke "git pack-objects" might itself exit with status 128 if that "pack-objects" segfaults or abort()'s. If the test invoking the parent command(s) is using "test_must_fail" we'll consider it an expected "ok" failure. * run-command.c doesn't (but probably should) ferry up such exit codes, so for e.g. "git push" tests where we expect a failure and an underlying "git" command fails we won't ferry up the segfault or abort exit code. * We have gitweb.perl and some other perl code ignoring return values from close(), i.e. ignoring exit codes from "git rev-parse" et al. * We have in-tree shellscripts like "git-merge-one-file.sh" invoking git commands, they'll usually return their own exit codes on "git" failure, rather then ferrying up segfault or abort() exit code. E.g. these invocations in git-merge-one-file.sh leak, but aren't reflected in the "git merge" exit code: src1=$(git unpack-file $2) src2=$(git unpack-file $3) That case would be easily "fixed" by adding a line like this after each assignment: test $? -ne 0 && exit $? But we'd then in e.g. "t6407-merge-binary.sh" run into write_tree_trivial() in "builtin/merge.c" calling die() instead of ferrying up the relevant exit code. Let's remove "TEST_PASSES_SANITIZE_LEAK=true" from tests we were falsely marking as leak-free. In the case of t6407-merge-binary.sh it was marked as leak-free in 9081a421a6d (checkout: fix "branch info" memory leaks, 2021-11-16). I'd previously removed other bad "TEST_PASSES_SANITIZE_LEAK=true" opt-ins in the series merged in ea05fd5fbf7 (Merge branch 'ab/keep-git-exit-codes-in-tests', 2022-03-16). The case of t1060-object-corruption.sh is more subtle, and will be discussed in a subsequent commit. 1. https://lore.kernel.org/git/cover-0.7-00000000000-20220318T002951Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: add a GIT_TEST_PASSING_SANITIZE_LEAK=check modeÆvar Arnfjörð Bjarmason
Add a new "GIT_TEST_PASSING_SANITIZE_LEAK=check" mode to the test-lib.sh. As noted in the updated "t/README" this compliments the existing "GIT_TEST_PASSING_SANITIZE_LEAK=true" mode added in 956d2e4639b (tests: add a test mode for SANITIZE=leak, run it in CI, 2021-09-23). Rather than document this all in one (even more) dense paragraph split up the discussion of how it combines with --immediate into its own paragraph following the discussion of "GIT_TEST_SANITIZE_LEAK_LOG=true". Before the removal of "test_external" in a preceding commit we would have had to special-case t9700-perl-git.sh and t0202-gettext-perl.sh. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: simplify by removing test_externalÆvar Arnfjörð Bjarmason
Remove the "test_external" function added in [1]. This arguably makes the output of t9700-perl-git.sh and friends worse. But as we'll argue below the trade-off is worth it, since "chaining" to another TAP emitter in test-lib.sh is more trouble than it's worth. The new output of t9700-perl-git.sh is now: $ ./t9700-perl-git.sh ok 1 - set up test repository ok 2 - use t9700/test.pl to test Git.pm # passed all 2 test(s) 1..2 Whereas before this change it would be: $ ./t9700-perl-git.sh ok 1 - set up test repository # run 1: Perl API (perl /home/avar/g/git/t/t9700/test.pl) ok 2 - use Git; [... omitting tests 3..46 from t/t9700/test.pl ...] ok 47 - unquote escape sequences 1..47 # test_external test Perl API was ok # test_external_without_stderr test no stderr: Perl API was ok At the time of its addition supporting "test_external" was easy, but when test-lib.sh itself started to emit TAP in [2] we needed to make everything surrounding the emission of the plan consider "test_external". I added that support in [2] so that we could run: prove ./t9700-perl-git.sh :: -v But since then in [3] the door has been closed on combining $HARNESS_ACTIVE and -v, we'll now just die: $ prove ./t9700-perl-git.sh :: -v Bailout called. Further testing stopped: verbose mode forbidden under TAP harness; try --verbose-log FAILED--Further testing stopped: verbose mode forbidden under TAP harness; try --verbose-log So the only use of this has been that *if* we had failure in one of these tests we could e.g. in CI see which test failed based on the test number. Now we'll need to look at the full verbose logs to get that same information. I think this trade-off is acceptable given the reduction in complexity, and it brings these tests in line with other similar tests, e.g. the reftable tests added in [4] will be condensed down to just one test, which invokes the C helper: $ ./t0032-reftable-unittest.sh ok 1 - unittests # passed all 1 test(s) 1..1 It would still be nice to have that ":: -v" form work again, it never *really* worked, but even though we've had edge cases test output screwing up the TAP it mostly worked between d998bd4ab67 and [3], so we may have been overzealous in forbidding it outright. I have local patches which I'm planning to submit sooner than later that get us to that goal, and in a way that isn't buggy. In the meantime getting rid of this special case makes hacking on this area of test-lib.sh easier, as we'll do in subsequent commits. The switch from "perl" to "$PERL_PATH" here is because "perl" is defined as a shell function in the test suite, see a5bf824f3b4 (t: prevent '-x' tracing from interfering with test helpers' stderr, 2018-02-25). On e.g. the OSX CI the "command perl"... will be part of the emitted stderr. 1. fb32c410087 (t/test-lib.sh: add test_external and test_external_without_stderr, 2008-06-19) 2. d998bd4ab67 (test-lib: Make the test_external_* functions TAP-aware, 2010-06-24) 3. 614fe015212 (test-lib: bail out when "-v" used under "prove", 2016-10-22) 4. ef8a6c62687 (reftable: utility functions, 2021-10-07) Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: add a SANITIZE=leak logging modeÆvar Arnfjörð Bjarmason
Add the ability to run the test suite under a new "GIT_TEST_SANITIZE_LEAK_LOG=true" mode, when true we'll log the leaks we find an a new "test-results/<test-name>.leak" directory. That new path is consistent with the existing "test-results/<test-name>.<type>" results, except that those are all files, not directories. We also set "log_exe_name=1" to include the name of the executable in the filename. This gives us files like "trace.git.<pid>" instead of the default of "trace.<pid>". I.e. we'll be able to distinguish "git" leaks from "test-tool", "git-daemon" etc. We then set "dedup_token_length" to non-zero ("0" is the default) to succinctly log a token we can de-duplicate these stacktraces on. The string is simply a one-line stack-trace with only function names up to N frames, which we limit at "9999" as a shorthand for "infinite" (there appears to be no way to say "no limit"). With these combined we can now easily get e.g. the top 10 leaks in the test suite grouped by full stacktrace: grep -o -P -h '(?<=DEDUP_TOKEN: ).*' test-results/*.leak/trace.git.* | sort | uniq -c | sort -nr | head -n 10 Or add "grep -E -o '[^-]+'" to that to group by functions instead of stack traces: grep -o -P -h '(?<=DEDUP_TOKEN: ).*' test-results/*.leak/trace.git.* | grep -E -o '[^-]+' | sort | uniq -c | sort -nr | head -n 20 This new mode requires git to be compiled with SANITIZE=leak, rather than explaining that in the documentation let's make it self-documenting by bailing out if the user asks for this without git having been compiled with SANITIZE=leak, as we do with GIT_TEST_PASSING_SANITIZE_LEAK=true. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27t/README: reword the "GIT_TEST_PASSING_SANITIZE_LEAK" descriptionÆvar Arnfjörð Bjarmason
Reword the documentation added in 956d2e4639b (tests: add a test mode for SANITIZE=leak, run it in CI, 2021-09-23) for brevity. The comment added in the same commit was also misleading: We skip certain tests if SANITIZE=leak and GIT_TEST_PASSING_SANITIZE_LEAK=true, not if we're compiled with SANITIZE=leak. Let's just remove the comment, the control flow here is obvious enough that the code can speak for itself. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: add a --invert-exit-code switchÆvar Arnfjörð Bjarmason
Add the ability to have those tests that fail return 0, and those tests that succeed return 1. This is useful e.g. to run "--stress" tests on tests that fail 99% of the time on some setup, i.e. to smoke out the flaky run which yielded success. In a subsequent commit a new SANITIZE=leak mode will make use of this. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: fix GIT_EXIT_OK logic errors, use BAIL_OUTÆvar Arnfjörð Bjarmason
Change various "exit 1" checks that happened after our "die" handler had been set up to use BAIL_OUT instead. See 234383cd401 (test-lib.sh: use "Bail out!" syntax on bad SANITIZE=leak use, 2021-10-14) for the benefits of the BAIL_OUT function. The previous use of "error" here was not a logic error, but the "exit" without "GIT_EXIT_OK" would emit the "FATAL: Unexpected exit with code $code" message on top of the error we wanted to emit. Since we'd also like to stop "prove" in its tracks here, the right thing to do is to emit a "Bail out!" message. Let's also move the "GIT_EXIT_OK=t" assignments to just above the "exit [01]" in "test_done". It's not OK if we exit in e.g. finalize_test_output. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: don't set GIT_EXIT_OK before calling test_atexit_handlerÆvar Arnfjörð Bjarmason
Change the control flow in test_done so that we'll set GIT_EXIT_OK=t after we call test_atexit_handler(). This seems to have been a mistake in 900721e15c4 (test-lib: introduce 'test_atexit', 2019-03-13). It doesn't make sense to allow our "atexit" handling to call "exit" without us emitting the errors we'll emit without GIT_EXIT_OK=t being set. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27test-lib: use $1, not $@ in test_known_broken_{ok,failure}_Ævar Arnfjörð Bjarmason
Clarify that these two functions never take N arguments, they'll only ever receive one. They've needlessly used $@ over $1 since 41ac414ea2b (Sane use of test_expect_failure, 2008-02-01). In the future we might want to pass the test source to these, but now that's not the case. This preparatory change helps to clarify a follow-up change. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-27Merge branch 'ab/test-quoting-fix' into maintJunio C Hamano
Fixes for tests when the source directory has unusual characters in its path, e.g. whitespaces, double-quotes, etc. source: <cover-v2-0.3-00000000000-20220630T101646Z-avarab@gmail.com> * ab/test-quoting-fix: config tests: fix harmless but broken "rm -r" cleanup test-lib.sh: fix prepend_var() quoting issue tests: add missing double quotes to included library paths
2022-07-18Merge branch 'ab/test-without-templates'Junio C Hamano
Tweak tests so that they still work when the "git init" template did not create .git/info directory. * ab/test-without-templates: tests: don't assume a .git/info for .git/info/sparse-checkout tests: don't assume a .git/info for .git/info/exclude tests: don't assume a .git/info for .git/info/refs tests: don't assume a .git/info for .git/info/attributes tests: don't assume a .git/info for .git/info/grafts tests: don't depend on template-created .git/branches t0008: don't rely on default ".git/info/exclude"
2022-07-13Merge branch 'ab/test-quoting-fix'Junio C Hamano
Fixes for tests when the source directory has unusual characters in its path, e.g. whitespaces, double-quotes, etc. * ab/test-quoting-fix: config tests: fix harmless but broken "rm -r" cleanup test-lib.sh: fix prepend_var() quoting issue tests: add missing double quotes to included library paths
2022-06-30test-lib.sh: fix prepend_var() quoting issueÆvar Arnfjörð Bjarmason
Fix a quoting issue in the function introduced in b9638d7286f (test-lib: make $GIT_BUILD_DIR an absolute path, 2022-02-27), running the test suite where the git checkout was on a path with e.g. a space in it would fail. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-06-15tests: add LIBCURL prerequisite to tests needing libcurlÆvar Arnfjörð Bjarmason
Add and use a LIBCURL prerequisite for tests added in 6dcbdc0d661 (remote: create fetch.credentialsInUrl config, 2022-06-06). These tests would get as far as emitting a couple of the warnings we were testing for, but would then die as we had no "git-remote-https" program compiled. It would be more consistent with other prerequisites (e.g. PERL for NO_PERL) to name this "CURL", but since e9184b0789a (t5561: skip tests if curl is not available, 2018-04-03) we've had that prerequisite defined for checking of we have the curl(1) program. The existing "CURL" prerequisite is only used in one place, and we should probably name it "CURL_PROGRAM", then rename "LIBCURL" to "CURL" as a follow-up, but for now (pre-v2.37.0) let's aim for the most minimal fix possible. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-06-15add -i tests: mark "TODO" depending on GIT_TEST_ADD_I_USE_BUILTINÆvar Arnfjörð Bjarmason
Fix an issue that existed before 0527ccb1b55 (add -i: default to the built-in implementation, 2021-11-30), but which became the default with that change, we should not be marking tests that are known to pass as "TODO" tests. When GIT_TEST_ADD_I_USE_BUILTIN=1 was made the default we started passing the tests added in 0f0fba2cc87 (t3701: add a test for advanced split-hunk editing, 2019-12-06) and 1bf01040f0c (add -p: demonstrate failure when running 'edit' after a split, 2015-04-16). Thus we've been emitting this sort of output: $ prove ./t3701-add-interactive.sh ./t3701-add-interactive.sh .. ok All tests successful. Test Summary Report ------------------- ./t3701-add-interactive.sh (Wstat: 0 Tests: 70 Failed: 0) TODO passed: 45, 47 Files=1, Tests=70, 2 wallclock secs ( 0.03 usr 0.00 sys + 0.86 cusr 0.33 csys = 1.22 CPU) Result: PASS Which isn't just cosmetic, but due to issues with test_expect_failure (see [1]) we could e.g. be hiding something as bad as a segfault in the new implementation. It makes sense catch that, especially before we put out a release with the built-in "add -i", so let's generalize the check we were already doing in 0527ccb1b55 with a new "ADD_I_USE_BUILTIN" prerequisite. 1. https://lore.kernel.org/git/patch-1.7-4624abc2591-20220318T002951Z-avarab@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-06-07Merge branch 'js/ci-github-workflow-markup'Junio C Hamano
Update the GitHub workflow support to make it quicker to get to the failing test. * js/ci-github-workflow-markup: ci: call `finalize_test_case_output` a little later ci(github): mention where the full logs can be found ci: use `--github-workflow-markup` in the GitHub workflow ci(github): avoid printing test case preamble twice ci(github): skip the logs of the successful test cases ci: optionally mark up output in the GitHub workflow ci/run-build-and-tests: add some structure to the GitHub workflow output ci: make it easier to find failed tests' logs in the GitHub workflow ci/run-build-and-tests: take a more high-level view test(junit): avoid line feeds in XML attributes tests: refactor --write-junit-xml code ci: fix code style
2022-06-06tests: don't depend on template-created .git/branchesÆvar Arnfjörð Bjarmason
As noted in c8a58ac5a52 (Revert "Don't create the $GIT_DIR/branches directory on init", 2009-10-31) there was an attempt long ago in 0cc5691a8b0 (Don't create the $GIT_DIR/branches directory on init, 2009-10-30) to get rid of the legacy "branches" directory. We should probably get rid of its creation by removing the "templates/branches--" file. But whatever our default behavior, our tests should be tightened up to explicitly create the .git/branches directory if they rely on our default templates, to make the dependency on those templates clear. So let's amend the two tests that would fail if .git/branches wasn't created. To do this introduce a new "TEST_CREATE_REPO_NO_TEMPLATE" variable, which we'll set before sourcing test-lib.sh, and change the "git clone" and "git init" commands in the tests themselves to explicitly pass "--template=". This way they won't get a .git/branches in either their top-level .git, or in the ones they create. We can then amend the tests that rely on the ".git/branches" directory existing to create it explicitly, and to remove it after its creation. This new "TEST_CREATE_REPO_NO_TEMPLATE" variable is a less heavy-handed version of the "NO_SET_GIT_TEMPLATE_DIR" variable. See a94d305bf80 (t/t0001-init.sh: add test for 'init with init.templatedir set', 2010-02-26) for its implementation. Unlike "TEST_CREATE_REPO_NO_TEMPLATE", this new "TEST_CREATE_REPO_NO_TEMPLATE" variable is narrowly scoped to what the "git init" in test-lib.sh does, as opposed to the global effect of "NO_SET_GIT_TEMPLATE_DIR" and the setting of "GIT_TEMPLATE_DIR" in wrap-for-bin.sh. I experimented with adding a new "GIT_WRAP_FOR_BIN_VIA_TEST_LIB" variable set in test-lib.sh, which would cause wrap-for-bin.sh to not set GIT_TEMPLATE_DIR, GITPERLLIB etc, as we set those in test-lib.sh. I think that's a viable approach, but it would interact e.g. with the appending feature of GITPERLLIB added in 8bade1e12e2 (wrap-for-bin: make bin-wrappers chainable, 2013-07-04). Doing so would allow us to convert the tests in t0001-init.sh that now use "NO_SET_GIT_TEMPLATE_DIR" to simply unset "GIT_TEMPLATE_DIR" in a sub-shell before invoking "git init" or "git clone". I think that approach is worth pursuing, but let's table it for now. Some future wrap-for-bin.sh refactoring can try to address it. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-23Merge branch 'ab/valgrind-fixes'Junio C Hamano
A bit of test framework fixes with a few fixes to issues found by valgrind. * ab/valgrind-fixes: commit-graph.c: don't assume that stat() succeeds object-file: fix a unpack_loose_header() regression in 3b6a8db3b03 log test: skip a failing mkstemp() test under valgrind tests: using custom GIT_EXEC_PATH breaks --valgrind tests
2022-05-21ci: call `finalize_test_case_output` a little laterJohannes Schindelin
We used to call that function already before printing the final verdict. However, now that we added grouping to the GitHub workflow output, we will want to include even that part in the collapsible group for that test case. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>