summaryrefslogtreecommitdiff
path: root/git-compat-util.h
AgeCommit message (Collapse)Author
2022-06-18Merge branch 'cb/path-owner-check-with-sudo-plus'Junio C Hamano
"sudo git foo" used to consider a repository owned by the original user a safe one to access; it now also considers a repository owned by root a safe one, too (after all, if an attacker can craft a malicious repository owned by root, the box is 0wned already). * cb/path-owner-check-with-sudo-plus: git-compat-util: allow root to access both SUDO_UID and root owned
2022-06-17git-compat-util: allow root to access both SUDO_UID and root ownedCarlo Marcelo Arenas Belón
Previous changes introduced a regression which will prevent root for accessing repositories owned by thyself if using sudo because SUDO_UID takes precedence. Loosen that restriction by allowing root to access repositories owned by both uid by default and without having to add a safe.directory exception. A previous workaround that was documented in the tests is no longer needed so it has been removed together with its specially crafted prerequisite. Helped-by: Johanness Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-06-10Merge branch 'ab/bug-if-bug'Junio C Hamano
A new bug() and BUG_if_bug() API is introduced to make it easier to uniformly log "detect multiple bugs and abort in the end" pattern. * ab/bug-if-bug: cache-tree.c: use bug() and BUG_if_bug() receive-pack: use bug() and BUG_if_bug() parse-options.c: use optbug() instead of BUG() "opts" check parse-options.c: use new bug() API for optbug() usage.c: add a non-fatal bug() function to go with BUG() common-main.c: move non-trace2 exit() behavior out of trace2.c
2022-06-03Merge branch 'ds/bundle-uri'Junio C Hamano
Preliminary code refactoring around transport and bundle code. * ds/bundle-uri: bundle.h: make "fd" version of read_bundle_header() public remote: allow relative_url() to return an absolute url remote: move relative_url() http: make http_get_file() external fetch-pack: move --keep=* option filling to a function fetch-pack: add a deref_without_lazy_fetch_extended() dir API: add a generalized path_match_flags() function connect.c: refactor sending of agent & object-format
2022-06-03Merge branch 'ns/batch-fsync'Junio C Hamano
Introduce a filesystem-dependent mechanism to optimize the way the bits for many loose object files are ensured to hit the disk platter. * ns/batch-fsync: core.fsyncmethod: performance tests for batch mode t/perf: add iteration setup mechanism to perf-lib core.fsyncmethod: tests for batch mode test-lib-functions: add parsing helpers for ls-files and ls-tree core.fsync: use batch mode and sync loose objects by default on Windows unpack-objects: use the bulk-checkin infrastructure update-index: use the bulk-checkin infrastructure builtin/add: add ODB transaction around add_files_to_cache cache-tree: use ODB transaction around writing a tree core.fsyncmethod: batched disk flushes for loose-objects bulk-checkin: rebrand plug/unplug APIs as 'odb transactions' bulk-checkin: rename 'state' variable and separate 'plugged' boolean
2022-06-02usage.c: add a non-fatal bug() function to go with BUG()Ævar Arnfjörð Bjarmason
Add a bug() function to use in cases where we'd like to indicate a runtime BUG(), but would like to defer the BUG() call because we're possibly accumulating more bug() callers to exhaustively indicate what went wrong. We already have this sort of facility in various parts of the codebase, just in the form of ad-hoc re-inventions of the functionality that this new API provides. E.g. this will be used to replace optbug() in parse-options.c, and the 'error("BUG:[...]' we do in a loop in builtin/receive-pack.c. Unlike the code this replaces we'll log to trace2 with this new bug() function (as with other usage.c functions, including BUG()), we'll also be able to avoid calls to xstrfmt() in some cases, as the bug() function itself accepts variadic sprintf()-like arguments. Any caller to bug() can follow up such calls with BUG_if_bug(), which will BUG() out (i.e. abort()) if there were any preceding calls to bug(), callers can also decide not to call BUG_if_bug() and leave the resulting BUG() invocation until exit() time. There are currently no bug() API users that don't call BUG_if_bug() themselves after a for-loop, but allowing for not calling BUG_if_bug() keeps the API flexible. As the tests and documentation here show we'll catch missing BUG_if_bug() invocations in our exit() wrapper. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-06-02common-main.c: move non-trace2 exit() behavior out of trace2.cÆvar Arnfjörð Bjarmason
Change the exit() wrapper added in ee4512ed481 (trace2: create new combined trace facility, 2019-02-22) so that we'll split up the trace2 logging concerns from wanting to wrap the "exit()" function itself for other purposes. This makes more sense structurally, as we won't seem to conflate non-trace2 behavior with the trace2 code. I'd previously added an explanation for this in 368b5843158 (common-main.c: call exit(), don't return, 2021-12-07), that comment is being adjusted here. Now the only thing we'll do if we're not using trace2 is to truncate the "code" argument to the lowest 8 bits. We only need to do that truncation on non-POSIX systems, but in ee4512ed481 that "if defined(__MINGW32__)" code added in 47e3de0e796 (MinGW: truncate exit()'s argument to lowest 8 bits, 2009-07-05) was made to run everywhere. It might be good for clarify to narrow that down by an "ifdef" again, but I'm not certain that in the interim we haven't had some other non-POSIX systems rely the behavior. On a POSIX system taking the lowest 8 bits is implicit, see exit(3)[1] and wait(2)[2]. Let's leave a comment about that instead. 1. https://man7.org/linux/man-pages/man3/exit.3.html 2. https://man7.org/linux/man-pages/man2/wait.2.html Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-26Merge branch 'cb/path-owner-check-with-sudo'Junio C Hamano
With a recent update to refuse access to repositories of other people by default, "sudo make install" and "sudo git describe" stopped working. This series intends to loosen it while keeping the safety. * cb/path-owner-check-with-sudo: t0034: add negative tests and allow git init to mostly work under sudo git-compat-util: avoid failing dir ownership checks if running privileged t: regression git needs safe.directory when using sudo
2022-05-16dir API: add a generalized path_match_flags() functionÆvar Arnfjörð Bjarmason
Add a path_match_flags() function and have the two sets of starts_with_dot_{,dot_}slash() functions added in 63e95beb085 (submodule: port resolve_relative_url from shell to C, 2016-04-15) and a2b26ffb1a8 (fsck: convert gitmodules url to URL passed to curl, 2020-04-18) be thin wrappers for it. As the latter of those notes the fsck version was copied from the initial builtin/submodule--helper.c version. Since the code added in a2b26ffb1a8 was doing really doing the same as win32_is_dir_sep() added in 1cadad6f658 (git clone <url> C:\cygwin\home\USER\repo' is working (again), 2018-12-15) let's move the latter to git-compat-util.h is a is_xplatform_dir_sep(). We can then call either it or the platform-specific is_dir_sep() from this new function. Let's likewise change code in various other places that was hardcoding checks for "'/' || '\\'" with the new is_xplatform_dir_sep(). As can be seen in those callers some of them still concern themselves with ':' (Mac OS classic?), but let's leave the question of whether that should be consolidated for some other time. As we expect to make wider use of the "native" case in the future, define and use two starts_with_dot_{,dot_}slash_native() convenience wrappers. This makes the diff in builtin/submodule--helper.c much smaller. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-05-13git-compat-util: avoid failing dir ownership checks if running privilegedCarlo Marcelo Arenas Belón
bdc77d1d685 (Add a function to determine whether a path is owned by the current user, 2022-03-02) checks for the effective uid of the running process using geteuid() but didn't account for cases where that user was root (because git was invoked through sudo or a compatible tool) and the original uid that repository trusted for its config was no longer known, therefore failing the following otherwise safe call: guy@renard ~/Software/uncrustify $ sudo git describe --always --dirty [sudo] password for guy: fatal: unsafe repository ('/home/guy/Software/uncrustify' is owned by someone else) Attempt to detect those cases by using the environment variables that those tools create to keep track of the original user id, and do the ownership check using that instead. This assumes the environment the user is running on after going privileged can't be tampered with, and also adds code to restrict that the new behavior only applies if running as root, therefore keeping the most common case, which runs unprivileged, from changing, but because of that, it will miss cases where sudo (or an equivalent) was used to change to another unprivileged user or where the equivalent tool used to raise privileges didn't track the original id in a sudo compatible way. Because of compatibility with sudo, the code assumes that uid_t is an unsigned integer type (which is not required by the standard) but is used that way in their codebase to generate SUDO_UID. In systems where uid_t is signed, sudo might be also patched to NOT be unsigned and that might be able to trigger an edge case and a bug (as described in the code), but it is considered unlikely to happen and even if it does, the code would just mostly fail safely, so there was no attempt either to detect it or prevent it by the code, which is something that might change in the future, based on expected user feedback. Reported-by: Guy Maurel <guy.j@maurel.de> Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Helped-by: Randall Becker <rsbecker@nexbridge.com> Helped-by: Phillip Wood <phillip.wood123@gmail.com> Suggested-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-04-11Merge tag 'v2.35.2'Junio C Hamano
2022-04-06Merge branch 'bc/csprng-mktemps'Junio C Hamano
Build fix. * bc/csprng-mktemps: git-compat-util: really support openssl as a source of entropy
2022-04-06core.fsync: use batch mode and sync loose objects by default on WindowsNeeraj Singh
Git for Windows has defaulted to core.fsyncObjectFiles=true since September 2017. We turn on syncing of loose object files with batch mode in upstream Git so that we can get broad coverage of the new code upstream. We don't actually do fsyncs in the most of the test suite, since GIT_TEST_FSYNC is set to 0. However, we do exercise all of the surrounding batch mode code since GIT_TEST_FSYNC merely makes the maybe_fsync wrapper always appear to succeed. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-04-06Merge branch 'ns/core-fsyncmethod' into ns/batch-fsyncJunio C Hamano
* ns/core-fsyncmethod: configure.ac: fix HAVE_SYNC_FILE_RANGE definition core.fsyncmethod: correctly camel-case warning message core.fsync: fix incorrect expression for default configuration core.fsync: documentation and user-friendly aggregate options core.fsync: new option to harden the index core.fsync: add configuration parsing core.fsync: introduce granular fsync control infrastructure core.fsyncmethod: add writeout-only mode wrapper: make inclusion of Windows csprng header tightly scoped
2022-04-06git-compat-util: really support openssl as a source of entropyCarlo Marcelo Arenas Belón
05cd988dce5 (wrapper: add a helper to generate numbers from a CSPRNG, 2022-01-17), configure openssl as the source for entropy in NON-STOP but doesn't add the needed header or link options. Since the only system that is configured to use openssl as a source of entropy is NON-STOP, add the header unconditionally, and -lcrypto to the list of external libraries. An additional change is required to make sure a NO_OPENSSL=1 build will be able to work as well (tested on Linux with a modified value of CSPRNG_METHOD = openssl), and the more complex logic that allows for compatibility with APPLE_COMMON_CRYPTO or allowing for simpler ways to link (without libssl) has been punted for now. Reported-by: Randall Becker <rsbecker@nexbridge.com> Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-30trace2: add stats for fsync operationsNeeraj Singh
Add some global trace2 statistics for the number of fsyncs performed during the lifetime of a Git process. These stats are printed as part of trace2_cmd_exit_fl, which is presumably where we might want to print any other cross-cutting statistics. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-25Merge branch 'ns/core-fsyncmethod'Junio C Hamano
Replace core.fsyncObjectFiles with two new configuration variables, core.fsync and core.fsyncMethod. * ns/core-fsyncmethod: core.fsync: documentation and user-friendly aggregate options core.fsync: new option to harden the index core.fsync: add configuration parsing core.fsync: introduce granular fsync control infrastructure core.fsyncmethod: add writeout-only mode wrapper: make inclusion of Windows csprng header tightly scoped
2022-03-23Sync with 2.34.2Johannes Schindelin
* maint-2.34: Git 2.34.2 Git 2.33.2 Git 2.32.1 Git 2.31.2 GIT-VERSION-GEN: bump to v2.33.1 Git 2.30.3 setup_git_directory(): add an owner check for the top-level directory Add a function to determine whether a path is owned by the current user
2022-03-23Sync with 2.33.2Johannes Schindelin
* maint-2.33: Git 2.33.2 Git 2.32.1 Git 2.31.2 GIT-VERSION-GEN: bump to v2.33.1 Git 2.30.3 setup_git_directory(): add an owner check for the top-level directory Add a function to determine whether a path is owned by the current user
2022-03-23Sync with 2.32.1Johannes Schindelin
* maint-2.32: Git 2.32.1 Git 2.31.2 Git 2.30.3 setup_git_directory(): add an owner check for the top-level directory Add a function to determine whether a path is owned by the current user
2022-03-23Sync with 2.31.2Johannes Schindelin
* maint-2.31: Git 2.31.2 Git 2.30.3 setup_git_directory(): add an owner check for the top-level directory Add a function to determine whether a path is owned by the current user
2022-03-23Sync with 2.30.3Johannes Schindelin
* maint-2.30: Git 2.30.3 setup_git_directory(): add an owner check for the top-level directory Add a function to determine whether a path is owned by the current user
2022-03-21Add a function to determine whether a path is owned by the current userJohannes Schindelin
This function will be used in the next commit to prevent `setup_git_directory()` from discovering a repository in a directory that is owned by someone other than the current user. Note: We cannot simply use `st.st_uid` on Windows just like we do on Linux and other Unix-like platforms: according to https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/stat-functions this field is always zero on Windows (because Windows' idea of a user ID does not fit into a single numerical value). Therefore, we have to do something a little involved to replicate the same functionality there. Also note: On Windows, a user's home directory is not actually owned by said user, but by the administrator. For all practical purposes, it is under the user's control, though, therefore we pretend that it is owned by the user. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2022-03-17mingw: avoid fallback for {local,gm}time_r()Carlo Marcelo Arenas Belón
mingw-w64's pthread_unistd.h had a bug that mistakenly (because there is no support for the *lockfile() functions required[1]) defined _POSIX_THREAD_SAFE_FUNCTIONS and that was being worked around since 3ecd153a3b (compat/mingw: support MSys2-based MinGW build, 2016-01-14). The bug was fixed in winphtreads, but as a side effect, leaves the reentrant functions from time.h no longer visible and therefore breaks the build. Since the intention all along was to avoid using the fallback functions, formalize the use of POSIX by setting the corresponding feature flag and compile out the implementation for the fallback functions. [1] https://unix.org/whitepapers/reentrant.html Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10core.fsyncmethod: add writeout-only modeNeeraj Singh
This commit introduces the `core.fsyncMethod` configuration knob, which can currently be set to `fsync` or `writeout-only`. The new writeout-only mode attempts to tell the operating system to flush its in-memory page cache to the storage hardware without issuing a CACHE_FLUSH command to the storage controller. Writeout-only fsync is significantly faster than a vanilla fsync on common hardware, since data is written to a disk-side cache rather than all the way to a durable medium. Later changes in this patch series will take advantage of this primitive to implement batching of hardware flushes. When git_fsync is called with FSYNC_WRITEOUT_ONLY, it may fail and the caller is expected to do an ordinary fsync as needed. On Apple platforms, the fsync system call does not issue a CACHE_FLUSH directive to the storage controller. This change updates fsync to do fcntl(F_FULLFSYNC) to make fsync actually durable. We maintain parity with existing behavior on Apple platforms by setting the default value of the new core.fsyncMethod option. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-10wrapper: make inclusion of Windows csprng header tightly scopedNeeraj Singh
Including NTSecAPI.h in git-compat-util.h causes build errors in any other file that includes winternl.h. NTSecAPI.h was included in order to get access to the RtlGenRandom cryptographically secure PRNG. This change scopes the inclusion of ntsecapi.h to wrapper.c, which is the only place that it's actually needed. The build breakage is due to the definition of UNICODE_STRING in NtSecApi.h: #ifndef _NTDEF_ typedef LSA_UNICODE_STRING UNICODE_STRING, *PUNICODE_STRING; typedef LSA_STRING STRING, *PSTRING ; #endif LsaLookup.h: typedef struct _LSA_UNICODE_STRING { USHORT Length; USHORT MaximumLength; #ifdef MIDL_PASS [size_is(MaximumLength/2), length_is(Length/2)] #endif // MIDL_PASS PWSTR Buffer; } LSA_UNICODE_STRING, *PLSA_UNICODE_STRING; winternl.h also defines UNICODE_STRING: typedef struct _UNICODE_STRING { USHORT Length; USHORT MaximumLength; PWSTR Buffer; } UNICODE_STRING; typedef UNICODE_STRING *PUNICODE_STRING; Both definitions have equivalent layouts. Apparently these internal Windows headers aren't designed to be included together. This is an oversight in the headers and does not represent an incompatibility between the APIs. Signed-off-by: Neeraj Singh <neerajsi@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-22C99: remove hardcoded-out !HAVE_VARIADIC_MACROS codeÆvar Arnfjörð Bjarmason
Remove the "else" branches of the HAVE_VARIADIC_MACROS macro, which have been unconditionally omitted since 765dc168882 (git-compat-util: always enable variadic macros, 2021-01-28). Since were always omitted, anyone trying to use a compiler without variadic macro support to compile a git since version git v2.31.0 or later would have had a compilation error. 10 months across a few releases since then should have been enough time for anyone who cared to run into that and report the issue. In addition to that, for anyone unsetting HAVE_VARIADIC_MACROS we've been emitting extremely verbose warnings since at least ee4512ed481 (trace2: create new combined trace facility, 2019-02-22). That's because there is no such thing as a "region_enter_printf" or "region_leave_printf" format, so at least under GCC and Clang everything that includes trace.h (almost every file) emits a couple of warnings about that. There's a large benefit to being able to have a hard dependency rely on variadic macros, the code surrounding usage.c is hard to maintain if we need to write two implementations of everything, and by relying on "__FILE__" and "__LINE__" along with "__VA_ARGS__" we can in the future make error(), die() etc. log where they were called from. We've also recently merged d67fc4bf0ba (Merge branch 'bc/require-c99', 2021-12-10) which further cements our hard dependency on C99. So let's delete the fallback code, and update our CodingGuidelines to note that we depend on this. The added bullet-point starts with lower-case for consistency with other bullet-points in that section. The diff in "trace.h" is relatively hard to read, since we need to retain the existing API docs, which were comments on the code used if HAVE_VARIADIC_MACROS was not defined. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-22git-compat-util.h: clarify GCC v.s. C99-specific in commentÆvar Arnfjörð Bjarmason
Change a comment added in e208f9cc757 (make error()'s constant return value more visible, 2012-12-15). It's not correct that this is GCC-ism anymore, it's code that uses standard C99 features. The comment being changed here pre-dates the HAVE_VARIADIC_MACROS define, which we got in e05bed960d3 (trace: add 'file:line' to all trace output, 2014-07-12). The original implementation of an error() macro) in e208f9cc757 used a GCC-ism with the paste operator (see the commit message for mention of it), but that was dropped later by 9798f7e5f9 (Use __VA_ARGS__ for all of error's arguments, 2013-02-08), giving us the C99-portable version we have now. While we could remove the __GNUC__ define here, it might cause issues for other compilers or static analysis systems, so let's not. See 87fe5df365 (inline constant return from error() function, 2014-05-06) for one such issue. See also e05bed960d3 (trace: add 'file:line' to all trace output, 2014-07-12) for another comment about GNUC's handling of __VA_ARGS__. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-16Merge branch 'ab/auto-detect-zlib-compress2'Junio C Hamano
The build procedure has been taught to notice older version of zlib and enable our replacement uncompress2() automatically. * ab/auto-detect-zlib-compress2: compat: auto-detect if zlib has uncompress2()
2022-02-12Merge branch 'bc/csprng-mktemps'Junio C Hamano
Pick a better random number generator and use it when we prepare temporary filenames. * bc/csprng-mktemps: wrapper: use a CSPRNG to generate random file names wrapper: add a helper to generate numbers from a CSPRNG
2022-01-26compat: auto-detect if zlib has uncompress2()Ævar Arnfjörð Bjarmason
We have a copy of uncompress2() implementation in compat/ so that we can build with an older version of zlib that lack the function, and the build procedure selects if it is used via the NO_UNCOMPRESS2 $(MAKE) variable. This is yet another "annoying" knob the porters need to tweak on platforms that are not common enough to have the default set in the config.mak.uname file. Attempt to instead ask the system header <zlib.h> to decide if we need the compatibility implementation. This is a deviation from the way we have been handling the "compatiblity" features so far, and if it can be done cleanly enough, it could work as a model for features that need compatibility definition we discover in the future. With that goal in mind, avoid expedient but ugly hacks, like shoving the code that is conditionally compiled into an unrelated .c file, which may not work in future cases---instead, take an approach that uses a file that is independently compiled and stands on its own. Compile and link compat/zlib-uncompress2.c file unconditionally, but conditionally hide the implementation behind #if/#endif when zlib version is 1.2.9 or newer, and unconditionally archive the resulting object file in the libgit.a to be picked up by the linker. There are a few things to note in the shape of the code base after this change: - We no longer use NO_UNCOMPRESS2 knob; if the system header <zlib.h> claims a version that is more cent than the library actually is, this would break, but it is easy to add it back when we find such a system. - The object file compat/zlib-uncompress2.o is always compiled and archived in libgit.a, just like a few other compat/ object files already are. - The inclusion of <zlib.h> is done in <git-compat-util.h>; we used to do so from <cache.h> which includes <git-compat-util.h> as the first thing it does, so from the *.c codes, there is no practical change. - Until objects in libgit.a that is already used gains a reference to the function, the reftable code will be the only one that wants it, so libgit.a on the linker command line needs to appear once more at the end to satisify the mutual dependency. - Beat found a trick used by OpenSSL to avoid making the conditionally-compiled object truly empty (apparently because they had to deal with compilers that do not want to see an effectively empty input file). Our compat/zlib-uncompress2.c file borrows the same trick for portabilty. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Helped-by: Beat Bolli <dev+git@drbeat.li> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-17wrapper: add a helper to generate numbers from a CSPRNGbrian m. carlson
There are many situations in which having access to a cryptographically secure pseudorandom number generator (CSPRNG) is helpful. In the future, we'll encounter one of these when dealing with temporary files. To make this possible, let's add a function which reads from a system CSPRNG and returns some bytes. We know that all systems will have such an interface. A CSPRNG is required for a secure TLS or SSH implementation and a Git implementation which provided neither would be of little practical use. In addition, POSIX is set to standardize getentropy(2) in the next version, so in the (potentially distant) future we can rely on that. For systems which lack one of the other interfaces, we provide the ability to use OpenSSL's CSPRNG. OpenSSL is highly portable and functions on practically every known OS, and we know it will have access to some source of cryptographically secure randomness. We also provide support for the arc4random in libbsd for folks who would prefer to use that. Because this is a security sensitive interface, we take some precautions. We either succeed by filling the buffer completely as we requested, or we fail. We don't return partial data because the caller will almost never find that to be a useful behavior. Specify a makefile knob which users can use to specify one or more suitable CSPRNGs, and turn the multiple string options into a set of defines, since we cannot match on strings in the preprocessor. We allow multiple options to make the job of handling this in autoconf easier. The order of options is important here. On systems with arc4random, which is most of the BSDs, we use that, since, except on MirBSD and macOS, it uses ChaCha20, which is extremely fast, and sits entirely in userspace, avoiding a system call. We then prefer getrandom over getentropy, because the former has been available longer on Linux, and then OpenSSL. Finally, if none of those are available, we use /dev/urandom, because most Unix-like operating systems provide that API. We prefer options that don't involve device files when possible because those work in some restricted environments where device files may not be available. Set the configuration variables appropriately for Linux and the BSDs, including macOS, as well as Windows and NonStop. We specifically only consider versions which receive publicly available security support here. For the same reason, we don't specify getrandom(2) on Linux, because CentOS 7 doesn't support it in glibc (although its kernel does) and we don't want to resort to making syscalls. Finally, add a test helper to allow this to be tested by hand and in tests. We don't add any tests, since invoking the CSPRNG is not likely to produce interesting, reproducible results. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-01-10Merge branch 'ab/usage-die-message'Junio C Hamano
Code clean-up to hide vreportf() from public API. * ab/usage-die-message: config API: use get_error_routine(), not vreportf() usage.c + gc: add and use a die_message_errno() gc: return from cmd_gc(), don't call exit() usage.c API users: use die_message() for error() + exit 128 usage.c API users: use die_message() for "fatal :" + exit 128 usage.c: add a die_message() routine
2022-01-05Merge branch 'jc/flex-array-definition'Junio C Hamano
The conditions to choose different definitions of the FLEX_ARRAY macro for vendor compilers has been simplified to make it easier to maintain. * jc/flex-array-definition: flex-array: simplify compiler-specific workaround
2021-12-10Merge branch 'cb/mingw-gmtime-r'Junio C Hamano
Build fix on Windows. * cb/mingw-gmtime-r: mingw: avoid fallback for {local,gm}time_r()
2021-12-10Merge branch 'bc/require-c99'Junio C Hamano
Weather balloon to break people with compilers that do not support C99. * bc/require-c99: git-compat-util: add a test balloon for C99 support
2021-12-09flex-array: simplify compiler-specific workaroundJunio C Hamano
We use "type array[];" syntax for the flex-array member at the end of a struct under C99 or later, except when we are building with older SUNPRO_C compilers. As we find more vendor compilers that claim to grok C99 but not understand the flex-array syntax, the existing "If we are using C99, but not with these compilers..." conditional will keep growing. Make it more manageable by listing vendor-specific exceptions earlier, with the expectation that new exceptions will not be combined into existing ones to make the condition longer, and instead will be implemented as a new "#elif" in the cascade of similar to old SUNPRO_C, we can just add a single line #elif defined(_MSC_VER) immediately before "#elif defined(__GNUC__)" to cause us to fallback to the safer but a bit wasteful version. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-07config API: use get_error_routine(), not vreportf()Ævar Arnfjörð Bjarmason
Change the git_die_config() function added in 5a80e97c827 (config: add `git_die_config()` to the config-set API, 2014-08-07) to use the public callbacks in the usage.[ch] API instead of the the underlying vreportf() function. In preceding commits the rest of the vreportf() users outside of usage.c was migrated to die_message(), so we can now make it "static". Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-07usage.c + gc: add and use a die_message_errno()Ævar Arnfjörð Bjarmason
Change the "error: " output when we exit with 128 due to gc.log errors to use a "fatal: " prefix instead. To do this add a die_message_errno() a sibling function to the die_errno() added in a preceding commit. Before this we'd expect report_last_gc_error() to return -1 from error_errno() in this case. It already treated a status of 0 and 1 specially. Let's just document that anything that's not 0 or 1 should be returned. We could also retain the "ret < 0" behavior here without hardcoding 128 by returning -128, and having the caller do a "return -ret", but I think this makes more sense, and preserves the path from die_message*()'s return value to the "return" without hardcoding "128". Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-07usage.c: add a die_message() routineÆvar Arnfjörð Bjarmason
We have code in various places that would like to call die(), but wants to defer the exit(128) it would invoke, e.g. to print an additional message, or adjust the exit code. Add a die_message() helper routine to bridge this gap in the API. Functionally this behaves just like the error() routine, except it'll print a "fatal: " prefix, and it will return with 128 instead of -1, this is so that caller can pass the return value to "exit()", instead of having to hardcode "exit(128)". Note that as with the other routines the "die_message_builtin" needs to return "void" and otherwise conform to the "report_fn" signature. As we'll see in a subsequent commit callers will want to replace e.g. their default "die_routine" with a "die_message_routine". For now we're just adding the routine and making die_builtin() in usage.c itself use it. In order to do that we need to add a get_die_message_routine() function, which works like the other get_*_routine() functions in usage.c. There is no set_die_message_rotine(), as it hasn't been needed yet. We can add it if we ever need it. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-01git-compat-util: add a test balloon for C99 supportbrian m. carlson
The C99 standard was released in January 1999, now 22 years ago. It provides a variety of useful features, including variadic arguments for macros, declarations after statements, designated initializers, and a wide variety of other useful features, many of which we already use. We'd like to take advantage of these features, but we want to be cautious. As far as we know, all major compilers now support C99 or a later C standard, such as C11 or C17. POSIX has required C99 support as a requirement for the 2001 revision, so we can safely assume any POSIX system which we are interested in supporting has C99. Even MSVC, long a holdout against modern C, now supports both C11 and C17 with an appropriate update. Moreover, even if people are using an older version of MSVC on these systems, they will generally need some implementation of the standard Unix utilities for the testsuite, and GNU coreutils, the most common option, has required C99 since 2009. Therefore, we can safely assume that a suitable version of GCC or clang is available to users even if their version of MSVC is not sufficiently capable. Let's add a test balloon to git-compat-util.h to see if anyone is using an older compiler. We'll add a comment telling people how to enable this functionality on GCC and Clang, even though modern versions of both will automatically do the right thing, and ask people still experiencing a problem to report that to us on the list. Note that C89 compilers don't provide the __STDC_VERSION__ macro, so we use a well-known hack of using "- 0". On compilers with this macro, it doesn't change the value, and on C89 compilers, the macro will be replaced with nothing, and our value will be 0. For sparse, we explicitly request the gnu99 style because we've traditionally taken advantage of some GCC- and clang-specific extensions when available and we'd like to retain the ability to do that. sparse also defaults to C89 without it, so things will fail for us if we don't. Update the cmake configuration to require C11 for MSVC. We do this because this will make MSVC to use C11, since it does not explicitly support C99. We do this with a compiler options because setting the C_STANDARD option does not work in our CI on MSVC and at the moment, we don't want to require C11 for Unix compilers. In the Makefile, don't set any compiler flags for the compiler itself, since on some systems, such as FreeBSD, we actually need C11, and asking for C99 causes things to fail to compile. The error message should make it obvious what's going wrong and allow a user to set the appropriate option when building in the event they're using a Unix compiler that doesn't support it by default. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-11-29Merge branch 'mc/clean-smudge-with-llp64'Junio C Hamano
The clean/smudge conversion code path has been prepared to better work on platforms where ulong is narrower than size_t. * mc/clean-smudge-with-llp64: clean/smudge: allow clean filters to process extremely large files odb: guard against data loss checking out a huge file git-compat-util: introduce more size_t helpers odb: teach read_blob_entry to use size_t t1051: introduce a smudge filter test for extremely large files test-lib: add prerequisite for 64-bit platforms test-tool genzeros: generate large amounts of data more efficiently test-genzeros: allow more than 2G zeros in Windows
2021-11-29Merge branch 'jc/unsetenv-returns-an-int'Junio C Hamano
The compatibility implementation for unsetenv(3) were written to mimic ancient, non-POSIX, variant seen in an old glibc; it has been changed to return an integer to match the more modern era. * jc/unsetenv-returns-an-int: unsetenv(3) returns int, not void
2021-11-28mingw: avoid fallback for {local,gm}time_r()Carlo Marcelo Arenas Belón
mingw-w64's pthread_unistd.h had a bug that mistakenly (because there is no support for the *lockfile() functions required[1]) defined _POSIX_THREAD_SAFE_FUNCTIONS and that was being worked around since 3ecd153a3b (compat/mingw: support MSys2-based MinGW build, 2016-01-14). The bug was fixed in winphtreads, but as a side effect, leaves the reentrant functions from time.h no longer visible and therefore breaks the build. Since the intention all along was to avoid using the fallback functions, formalize the use of POSIX by setting the corresponding feature flag and compile out the implementation for the fallback functions. [1] https://unix.org/whitepapers/reentrant.html Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-11-03git-compat-util: introduce more size_t helpersJohannes Schindelin
We will use them in the next commit. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-29unsetenv(3) returns int, not voidJunio C Hamano
This compatilibity implementation has been returning a wrong type, ever since 731043fd (Add compat/unsetenv.c ., 2006-01-25) added to the system, yet nobody noticed it in the past 16 years, presumably because no code checks failures in their unsetenv() calls. Sigh. For now, make it always succeed. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-29wrapper: remove xunsetenv()Carlo Marcelo Arenas Belón
Remove the unused wrapper function. Reported-by: Randall S. Becker <rsbecker@nexbridge.com> Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-06Merge branch 'ab/repo-settings-cleanup'Junio C Hamano
Code cleanup. * ab/repo-settings-cleanup: repository.h: don't use a mix of int and bitfields repo-settings.c: simplify the setup read-cache & fetch-negotiator: check "enum" values in switch() environment.c: remove test-specific "ignore_untracked..." variable wrapper.c: add x{un,}setenv(), and use xsetenv() in environment.c
2021-09-23Merge branch 'cb/unix-sockets-with-windows'Junio C Hamano
Adjust credential-cache helper to Windows. * cb/unix-sockets-with-windows: git-compat-util: include declaration for unix sockets in windows credential-cache: check for windows specific errors t0301: fixes for windows compatibility
2021-09-22wrapper.c: add x{un,}setenv(), and use xsetenv() in environment.cÆvar Arnfjörð Bjarmason
Add fatal wrappers for setenv() and unsetenv(). In d7ac12b25d3 (Add set_git_dir() function, 2007-08-01) we started checking its return value, and since 48988c4d0c3 (set_git_dir: die when setenv() fails, 2018-03-30) we've had set_git_dir_1() die if we couldn't set it. Let's provide a wrapper for both, this will be useful in many other places, a subsequent patch will make another use of xsetenv(). The checking of the return value here is over-eager according to setenv(3) and POSIX. It's documented as returning just -1 or 0, so perhaps we should be checking -1 explicitly. Let's just instead die on any non-zero, if our C library is so broken as to return something else than -1 on error (and perhaps not set errno?) the worst we'll do is die with a nonsensical errno value, but we'll want to die in either case. Let's make these return "void" instead of "int". As far as I can tell there's no other x*() wrappers that needed to make the decision of deviating from the signature in the C library, but since their return value is only used to indicate errors (so we'd die here), we can catch unreachable code such as if (xsetenv(...) < 0) [...]; I think it would be OK skip the NULL check of the "name" here for the calls to die_errno(). Almost all of our setenv() callers are taking a constant string hardcoded in the source as the first argument, and for the rest we can probably assume they've done the NULL check themselves. Even if they didn't, modern C libraries are forgiving about it (e.g. glibc formatting it as "(null)"), on those that aren't, well, we were about to die anyway. But let's include the check anyway for good measure. 1. https://pubs.opengroup.org/onlinepubs/009604499/functions/setenv.html Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>