path: root/sideband.c
AgeCommit message (Collapse)Author
2019-02-05Merge branch 'jt/fetch-v2-sideband'Junio C Hamano
"git fetch" and "git upload-pack" learned to send all exchange over the sideband channel while talking the v2 protocol. * jt/fetch-v2-sideband: tests: define GIT_TEST_SIDEBAND_ALL {fetch,upload}-pack: sideband v2 fetch response sideband: reverse its dependency on pkt-line pkt-line: introduce struct packet_writer pack-protocol.txt: accept error packets in any context Use packet_reader instead of packet_read_line
2019-01-17{fetch,upload}-pack: sideband v2 fetch responseJonathan Tan
Currently, a response to a fetch request has sideband support only while the packfile is being sent, meaning that the server cannot send notices until the start of the packfile. Extend sideband support in protocol v2 fetch responses to the whole response. upload-pack will advertise it if the uploadpack.allowsidebandall configuration variable is set, and fetch-pack will automatically request it if advertised. If the sideband is to be used throughout the whole response, upload-pack will use it to send errors instead of prefixing a PKT-LINE payload with "ERR ". This will be tested in a subsequent patch. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2019-01-17sideband: reverse its dependency on pkt-lineJonathan Tan
A subsequent patch will teach struct packet_reader a new field that, if set, instructs it to interpret read data as multiplexed. This will create a dependency from pkt-line to sideband. To avoid a circular dependency, split recv_sideband() into 2 parts: the reading loop (left in recv_sideband()) and the processing of the contents (in demultiplex_sideband()), and move the former into pkt-line. This reverses the direction of dependency: sideband no longer depends on pkt-line, and pkt-line now depends on sideband. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2018-12-04sideband: color lines with keyword onlyStefan Beller
When bf1a11f0a1 (sideband: highlight keywords in remote sideband output, 2018-08-07) was introduced, it was carefully considered which strings would be highlighted. However 59a255aef0 (sideband: do not read beyond the end of input, 2018-08-18) brought in a regression that the original did not test for. A line containing only the keyword and nothing else ("SUCCESS") should still be colored. Signed-off-by: Stefan Beller <> Signed-off-by: Junio C Hamano <>
2018-08-18sideband: do not read beyond the end of inputJunio C Hamano
The caller of maybe_colorize_sideband() gives a counted buffer <src, n>, but the callee checked src[] as if it were a NUL terminated buffer. If src[] had all isspace() bytes in it, we would have made n negative, and then (1) made number of strncasecmp() calls to see if the remaining bytes in src[] matched keywords, reading beyond the end of the array (this actually happens even if n does not go negative), and/or (2) called strbuf_add() with negative count, most likely triggering the "you want to use way too much memory" error due to unsigned integer overflow. Fix both issues by making sure we do not go beyond &src[n]. In the longer term we may want to accept size_t as parameter for clarity (even though we know that a sideband message we are painting typically would fit on a line on a terminal and int is sufficient). Write it down as a NEEDSWORK comment. Helped-by: Jonathan Nieder <> Signed-off-by: Junio C Hamano <>
2018-08-08sideband: highlight keywords in remote sideband outputHan-Wen Nienhuys
The colorization is controlled with the config setting "color.remote". Supported keywords are "error", "warning", "hint" and "success". They are highlighted if they appear at the start of the line, which is common in error messages, eg. ERROR: commit is missing Change-Id The Git push process itself prints lots of non-actionable messages (eg. bandwidth statistics, object counters for different phases of the process). This obscures actionable error messages that servers may send back. Highlighting keywords in the sideband draws more attention to those messages. The background for this change is that Gerrit does server-side processing to create or update code reviews, and actionable error messages (eg. missing Change-Id) must be communicated back to the user during the push. User research has shown that new users have trouble seeing these messages. The highlighting is done on the client rather than server side, so servers don't have to grow capabilities to understand terminal escape codes and terminal state. It also consistent with the current state where Git is control of the local display (eg. prefixing messages with "remote: "). The highlighting can be configured using color.remote.<KEYWORD> configuration settings. Since the keys are matched case insensitively, we match the keywords case insensitively too. Finally, this solution is backwards compatible: many servers already prefix their messages with "error", and they will benefit from this change without requiring a server update. By contrast, a server-side solution would likely require plumbing the TERM variable through the git protocol, so it would require changes to both server and client. Helped-by: Duy Nguyen <> Signed-off-by: Han-Wen Nienhuys <> Signed-off-by: Junio C Hamano <>
2018-04-24Avoid multiple PREFIX definitionsPhilip Oakley
The short and sweet PREFIX can be confused when used in many places. Rename both usages to better describe their purpose. EXEC_CMD_PREFIX is used in full to disambiguate it from the nearby GIT_EXEC_PATH. The PREFIX in sideband.c, while nominally independant of the exec_cmd PREFIX, does reside within libgit[1], so the definitions would clash when taken together with a PREFIX given on the command line for use by exec_cmd.c. Noticed when compiling Git for Windows using MSVC/Visual Studio [1] which reports the conflict beteeen the command line definition and the definition in sideband.c within the libgit project. [1] the libgit functions are brought into a single sub-project within the Visual Studio construction script provided in contrib, and hence uses a single command for both exec_cmd.c and sideband.c. Signed-off-by: Philip Oakley <> Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2017-12-04refactor "dumb" terminal determinationLars Schneider
Move the code to detect "dumb" terminals into a single location. This avoids duplicating the terminal detection code yet again in a subsequent commit. Signed-off-by: Lars Schneider <> Signed-off-by: Junio C Hamano <>
2016-07-19Merge branch 'lf/recv-sideband-cleanup'Junio C Hamano
Code simplification. * lf/recv-sideband-cleanup: sideband.c: small optimization of strbuf usage sideband.c: refactor recv_sideband()
2016-07-06sideband.c: small optimization of strbuf usageNicolas Pitre
Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2016-07-01sideband.c: refactor recv_sideband()Lukas Fleischer
We used character buffer manipulations to split messages from the sideband at line breaks and insert "remote: " at the beginning of each line, using the packet size to determine the end of a message. However, since it is safe to assume that diagnostic messages from the sideband never contain NUL characters, we can also NUL-terminate the buffer, use strpbrk() for splitting lines and use format strings to insert the prefix, to make the code easier to read and maintain. A strbuf is used for accumulating the output which is then printed using a single write(2) call to ensure the atomicity of the output. See 9ac13ec (atomic write for sideband remote messages, 2006-10-11) for details. Helped-by: Jeff King <> Helped-by: Junio C Hamano <> Helped-by: Nicolas Pitre <> Signed-off-by: Lukas Fleischer <> Signed-off-by: Junio C Hamano <>
2016-06-16sideband.c: make send_sideband() return voidLukas Fleischer
The send_sideband() function uses write_or_die() for writing data which immediately terminates the process on errors. If no such error occurred, send_sideband() always returned the value that was passed as fourth parameter prior to this commit. This value is already known to the caller in any case, so let's turn send_sideband() into a void function instead. Signed-off-by: Lukas Fleischer <> Signed-off-by: Junio C Hamano <>
2015-09-25convert trivial sprintf / strcpy calls to xsnprintfJeff King
We sometimes sprintf into fixed-size buffers when we know that the buffer is large enough to fit the input (either because it's a constant, or because it's numeric input that is bounded in size). Likewise with strcpy of constant strings. However, these sites make it hard to audit sprintf and strcpy calls for buffer overflows, as a reader has to cross-reference the size of the array with the input. Let's use xsnprintf instead, which communicates to a reader that we don't expect this to overflow (and catches the mistake in case we do). Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2014-06-02sideband.c: do not use ANSI control sequence on non-terminalMichael Naumov
Diagnostic messages received on the sideband #2 from the server side are sent to the standard error with ANSI terminal control sequence "\033[K" that erases to the end of line appended at the end of each line. However, some programs (e.g. GitExtensions for Windows) read and interpret and/or show the message without understanding the terminal control sequences, resulting them to be shown to their end users. To help these programs, squelch the control sequence when the standard error stream is not being sent to a tty. NOTE: I considered to cover the case that a pager has already been started. But decided that is probably not worth worrying about here, though, as we shouldn't be using a pager for commands that do network communications (and if we do, omitting the magic line-clearing signal is probably a sane thing to do). Thanks-to: Erik Faye-Lund <> Thanks-to: Jeff King <> Signed-off-by: Michael Naumov <> Signed-off-by: Junio C Hamano <>
2013-02-24pkt-line: share buffer/descriptor reading implementationJeff King
The packet_read function reads from a descriptor. The packet_get_line function is similar, but reads from an in-memory buffer, and uses a completely separate implementation. This patch teaches the generic packet_read function to accept either source, and we can do away with packet_get_line's implementation. There are two other differences to account for between the old and new functions. The first is that we used to read into a strbuf, but now read into a fixed size buffer. The only two callers are fine with that, and in fact it simplifies their code, since they can use the same static-buffer interface as the rest of the packet_read_line callers (and we provide a similar convenience wrapper for reading from a buffer rather than a descriptor). This is technically an externally-visible behavior change in that we used to accept arbitrary sized packets up to 65532 bytes, and now cap out at LARGE_PACKET_MAX, 65520. In practice this doesn't matter, as we use it only for parsing smart-http headers (of which there is exactly one defined, and it is small and fixed-size). And any extension headers would be breaking the protocol to go over LARGE_PACKET_MAX anyway. The other difference is that packet_get_line would return on error rather than dying. However, both callers of packet_get_line are actually improved by dying. The first caller does its own error checking, but we can drop that; as a result, we'll actually get more specific reporting about protocol breakage when packet_read dies internally. The only downside is that packet_read will not print the smart-http URL that failed, but that's not a big deal; anybody not debugging can already see the remote's URL already, and anybody debugging would want to run with GIT_CURL_VERBOSE anyway to see way more information. The second caller, which is just trying to skip past any extra smart-http headers (of which there are none defined, but which we allow to keep room for future expansion), did not error check at all. As a result, it would treat an error just like a flush packet. The resulting mess would generally cause an error later in get_remote_heads, but now we get error reporting much closer to the source of the problem. Brown-paper-bag-fixes-by: Ramsay Jones <> Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: teach packet_read_line to chomp newlinesJeff King
The packets sent during ref negotiation are all terminated by newline; even though the code to chomp these newlines is short, we end up doing it in a lot of places. This patch teaches packet_read_line to auto-chomp the trailing newline; this lets us get rid of a lot of inline chomping code. As a result, some call-sites which are not reading line-oriented data (e.g., when reading chunks of packfiles alongside sideband) transition away from packet_read_line to the generic packet_read interface. This patch converts all of the existing callsites. Since the function signature of packet_read_line does not change (but its behavior does), there is a possibility of new callsites being introduced in later commits, silently introducing an incompatibility. However, since a later patch in this series will change the signature, such a commit would have to be merged directly into this commit, not to the tip of the series; we can therefore ignore the issue. This is an internal cleanup and should produce no change of behavior in the normal case. However, there is one corner case to note. Callers of packet_read_line have never been able to tell the difference between a flush packet ("0000") and an empty packet ("0004"), as both cause packet_read_line to return a length of 0. Readers treat them identically, even though Documentation/technical/protocol-common.txt says we must not; it also says that implementations should not send an empty pkt-line. By stripping out the newline before the result gets to the caller, we will now treat the newline-only packet ("0005\n") the same as an empty packet, which in turn gets treated like a flush packet. In practice this doesn't matter, as neither empty nor newline-only packets are part of git's protocols (at least not for the line-oriented bits, and readers who are not expecting line-oriented packets will be calling packet_read directly, anyway). But even if we do decide to care about the distinction later, it is orthogonal to this patch. The right place to tighten would be to stop treating empty packets as flush packets, and this change does not make doing so any harder. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: drop safe_write functionJeff King
This is just write_or_die by another name. The one distinction is that write_or_die will treat EPIPE specially by suppressing error messages. That's fine, as we die by SIGPIPE anyway (and in the off chance that it is disabled, write_or_die will simulate it). Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2009-11-05Smart push over HTTP: client sideShawn O. Pearce
The git-remote-curl backend detects if the remote server supports the git-receive-pack service, and if so, runs git-send-pack in a pipe to dump the command and pack data as a single POST request. The advertisements from the server that were obtained during the discovery are passed into git-send-pack before the POST request starts. This permits git-send-pack to operate largely unmodified. For smaller packs (those under 1 MiB) a HTTP/1.0 POST with a Content-Length is used, permitting interaction with any server. The 1 MiB limit is arbitrary, but is sufficent to fit most deltas created by human authors against text sources with the occasional small binary file (e.g. few KiB icon image). The configuration option http.postBuffer can be used to increase (or shink) this buffer if the default is not sufficient. For larger packs which cannot be spooled entirely into the helper's memory space (due to http.postBuffer being too small), the POST request requires HTTP/1.1 and sets "Transfer-Encoding: chunked". This permits the client to upload an unknown amount of data in one HTTP transaction without needing to pregenerate the entire pack file locally. Signed-off-by: Shawn O. Pearce <> CC: Daniel Barkalow <> Signed-off-by: Junio C Hamano <>
2009-03-11recv_sideband: Bands #2 and #3 always go to stderrJohannes Sixt
This removes the last parameter of recv_sideband, by which the callers told which channel bands #2 and #3 should be written to. Sayeth Shawn Pearce: The definition of the streams in the current sideband protocol are rather well defined for the one protocol that uses it, fetch-pack/receive-pack: stream #1: pack data stream #2: stderr messages, progress, meant for tty stream #3: abort message, remote is dead, goodbye! Since both callers of the function passed 2 for the parameter, we hereby remove it and send bands #2 and #3 to stderr explicitly using fprintf. This has the nice side-effect that these two streams pass through our ANSI emulation layer on Windows. Signed-off-by: Johannes Sixt <> Acked-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-09-03improve handling of sideband message displayNicolas Pitre
Currently the code looks for line break characters in order to prepend "remote: " to every line received as many lines can be sent in a single chunk. However the opposite might happen too, i.e. a single message line split amongst multiple chunks. This patch adds support for the later case to avoid displays like: remote: Compressing objeremote: cts: 100% (313/313), done. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-01-09recv_sideband: Do not use ANSI escape sequence on dumb terminals.Johannes Sixt
The "clear to end of line" sequence is used to nicely output the progress indicator without leaving garbage on the terminal. However, this works only on ANSI capable terminals. We use the same check as in color.c to find out whether the terminal supports this feature and use a workaround (a few spaces in a row) if it does not. [jc: as an old fashoned git myself, and given the fact that the possible prefix and suffix are small number of short constant strings, I actually prefer a simpler-and-more-stupid approach. This is with Nico's clean-up.] Signed-off-by: Johannes Sixt <> Signed-off-by: Junio C Hamano <>
2007-11-05sideband.c: ESC is spelled '\033' not '\e' for portability.Nicolas Pitre
Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-11-04fix display overlap between remote and local progressNicolas Pitre
It is possible for the remote summary line to be displayed over the local progress display line, and therefore that local progress gets bumped to the next line. However, if the progress line is long enough, it might not be entirely overwritten by the remote summary line. This creates a messed up display such as: remote: Total 310 (delta 160), reused 178 (delta 112)iB/s Receiving objects: 100% (310/310), 379.98 KiB | 136 KiB/s, done. So we have to clear the screen line before displaying the remote message to make sure the local progress is not visible anymore on the first line. Yet some Git versions on the remote side might be sending updates to the same line and terminate it with \r, and a separate packet with a single \n might be sent later when the progress display is done. This means the screen line must *not* be cleared in that case. Since the sideband code already has to figure out line breaks in the received packet to properly prepend the "remote:" prefix, we can easily determine if the remote line about to be displayed is empty. Only when it is not then a proper suffix is inserted before the \r or \n to clear the end of the screen line. Also some magic constants related to the prefix length have been replaced with a variable, making it similar to the suffix length handling. Since gcc is smart enough to detect that the variable is constant there is no impact on the generated code. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-10-17cope with multiple line breaks within sideband progress messagesNicolas Pitre
A single sideband packet may sometimes contain multiple lines of progress messages, but we prepend "remote: " only to the whole buffer which creates a messed up display in that case. Make sure that the "remote: " prefix is applied to every remote lines. Signed-off-by: Nicolas Pitre <> Signed-off-by: Shawn O. Pearce <>
2006-10-11atomic write for sideband remote messagesNicolas Pitre
It has been a few times that I ended up with such a confusing display: |remote: Generating pack... |remote: Done counting 17 objects. |remote: Result has 9 objects. |remote: Deltifying 9 objects. |remote: 100% (9/9) done |remote: Unpacking 9 objects |Total 9, written 9 (delta 8), reused 0 (delta 0) | 100% (9/9) done The confusion can be avoided in most cases by writing the remote message in one go to prevent interleacing with local messages. The buffer declaration has been moved inside recv_sideband() to avoid extra string copies. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2006-09-10Move sideband server side support into reusable form.Junio C Hamano
The server side support; this is just the very low level, and the caller needs to know which band it wants to send things out. Signed-off-by: Junio C Hamano <> (cherry picked from b786552b67878c7780c50def4c069d46dc54efbe commit)
2006-09-10Move sideband client side support into reusable form.Junio C Hamano
This moves the receiver side of the sideband support from fetch-clone.c to sideband.c and its header file, so that archiver protocol can use it. Signed-off-by: Junio C Hamano <>