path: root/pkt-line.h
AgeCommit message (Collapse)Author
2021-03-15pkt-line: add options argument to read_packetized_to_strbuf()Johannes Schindelin
Update the calling sequence of `read_packetized_to_strbuf()` to take an options argument and not assume a fixed set of options. Update the only existing caller accordingly to explicitly pass the formerly-assumed flags. The `read_packetized_to_strbuf()` function calls `packet_read()` with a fixed set of assumed options (`PACKET_READ_GENTLE_ON_EOF`). This assumption has been fine for the single existing caller `apply_multi_file_filter()` in `convert.c`. In a later commit we would like to add other callers to `read_packetized_to_strbuf()` that need a different set of options. Signed-off-by: Johannes Schindelin <> Signed-off-by: Jeff Hostetler <> Signed-off-by: Junio C Hamano <>
2021-03-15pkt-line: add PACKET_READ_GENTLE_ON_READ_ERROR optionJohannes Schindelin
Introduce PACKET_READ_GENTLE_ON_READ_ERROR option to help libify the packet readers. So far, the (possibly indirect) callers of `get_packet_data()` can ask that function to return an error instead of `die()`ing upon end-of-file. However, random read errors will still cause the process to die. So let's introduce an explicit option to tell the packet reader machinery to please be nice and only return an error on read errors. This change prepares pkt-line for use by long-running daemon processes. Such processes should be able to serve multiple concurrent clients and and survive random IO errors. If there is an error on one connection, a daemon should be able to drop that connection and continue serving existing and future connections. This ability will be used by a Git-aware "Builtin FSMonitor" feature in a later patch series. Signed-off-by: Johannes Schindelin <> Signed-off-by: Jeff Hostetler <> Signed-off-by: Junio C Hamano <>
2021-03-15pkt-line: do not issue flush packets in write_packetized_*()Johannes Schindelin
Remove the `packet_flush_gently()` call in `write_packetized_from_buf() and `write_packetized_from_fd()` and require the caller to call it if desired. Rename both functions to `write_packetized_from_*_no_flush()` to prevent later merge accidents. `write_packetized_from_buf()` currently only has one caller: `apply_multi_file_filter()` in `convert.c`. It always wants a flush packet to be written after writing the payload. However, we are about to introduce a caller that wants to write many packets before a final flush packet, so let's make the caller responsible for emitting the flush packet. Signed-off-by: Jeff Hostetler <> Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2020-07-07Merge branch 'bc/sha-256-part-2'Junio C Hamano
SHA-256 migration work continues. * bc/sha-256-part-2: (44 commits) remote-testgit: adapt for object-format bundle: detect hash algorithm when reading refs t5300: pass --object-format to git index-pack t5704: send object-format capability with SHA-256 t5703: use object-format serve option t5702: offer an object-format capability in the test t/helper: initialize the repository for test-sha1-array remote-curl: avoid truncating refs with ls-remote t1050: pass algorithm to index-pack when outside repo builtin/index-pack: add option to specify hash algorithm remote-curl: detect algorithm for dumb HTTP by size builtin/ls-remote: initialize repository based on fetch t5500: make hash independent serve: advertise object-format capability for protocol v2 connect: parse v2 refs with correct hash algorithm connect: pass full packet reader when parsing v2 refs Documentation/technical: document object-format for protocol v2 t1302: expect repo format version 1 for SHA-256 builtin/show-index: provide options to determine hash algo t5302: modernize test formatting ...
2020-05-27pkt-line: add a member for hash algorithmbrian m. carlson
Add a member for the hash algorithm currently in use to the packet reader so it can parse references correctly. Signed-off-by: brian m. carlson <> Signed-off-by: Junio C Hamano <>
2020-05-24pkt-line: define PACKET_READ_RESPONSE_ENDDenton Liu
In a future commit, we will use PACKET_READ_RESPONSE_END to separate messages proxied by remote-curl. To prepare for this, add the PACKET_READ_RESPONSE_END enum value. In switch statements that need a case added, die() or BUG() when a PACKET_READ_RESPONSE_END is unexpected. Otherwise, mirror how PACKET_READ_DELIM is implemented (especially in cases where packets are being forwarded). Signed-off-by: Denton Liu <> Signed-off-by: Junio C Hamano <>
2020-05-19pkt-line: extern packet_length()Denton Liu
In a future commit, we will be manually processing packets and we will need to access the length header. In order to simplify this, extern packet_length() so that the logic can be reused. Change the function parameter from `const char *linelen` to `const char lenbuf_hex[4]`. Even though these two types behave identically as function parameters, use the array notation to semantically indicate exactly what this function is expecting as an argument. Also, rename it from linelen to lenbuf_hex as the former sounds like it should be an integral type which is misleading. Signed-off-by: Denton Liu <> Signed-off-by: Junio C Hamano <>
2019-12-02pkt-line: fix a typoJohannes Schindelin
This typo was introduced in 2153d478b74 (pkt-line: introduce packet_read_with_status, 2018-03-14). Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2019-05-05*.[ch]: manually align parameter listsDenton Liu
In previous patches, extern was mechanically removed from function declarations without care to formatting, causing parameter lists to be misaligned. Manually format changed sections such that the parameter lists should be realigned. Viewing this patch with 'git diff -w' should produce no output. Signed-off-by: Denton Liu <> Signed-off-by: Junio C Hamano <>
2019-05-05*.[ch]: remove extern from function declarations using spatchDenton Liu
There has been a push to remove extern from function declarations. Remove some instances of "extern" for function declarations which are caught by Coccinelle. Note that Coccinelle has some difficulty with processing functions with `__attribute__` or varargs so some `extern` declarations are left behind to be dealt with in a future patch. This was the Coccinelle patch used: @@ type T; identifier f; @@ - extern T f(...); and it was run with: $ git ls-files \*.{c,h} | grep -v ^compat/ | xargs spatch --sp-file contrib/coccinelle/noextern.cocci --in-place Files under `compat/` are intentionally excluded as some are directly copied from external sources and we should avoid churning them as much as possible. Signed-off-by: Denton Liu <> Signed-off-by: Junio C Hamano <>
2019-03-03remote-curl: use post_rpc() for protocol v2 alsoJonathan Tan
When transmitting and receiving POSTs for protocol v0 and v1, remote-curl uses post_rpc() (and associated functions), but when doing the same for protocol v2, it uses a separate set of functions (proxy_rpc() and others). Besides duplication of code, this has caused at least one bug: the auth retry mechanism that was implemented in v0/v1 was not implemented in v2. To fix this issue and avoid it in the future, make remote-curl also use post_rpc() when handling protocol v2. Because line lengths are written to the HTTP request in protocol v2 (unlike in protocol v0/v1), this necessitates changes in post_rpc() and some of the functions it uses; perform these changes too. A test has been included to ensure that the code for both the unchunked and chunked variants of the HTTP request is exercised. Note: stateless_connect() has been updated to use the lower-level packet reading functions instead of struct packet_reader. The low-level control is necessary here because we cannot change the destination buffer of struct packet_reader while it is being used; struct packet_buffer has a peeking mechanism which relies on the destination buffer being present in between a peek and a read. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2019-01-17{fetch,upload}-pack: sideband v2 fetch responseJonathan Tan
Currently, a response to a fetch request has sideband support only while the packfile is being sent, meaning that the server cannot send notices until the start of the packfile. Extend sideband support in protocol v2 fetch responses to the whole response. upload-pack will advertise it if the uploadpack.allowsidebandall configuration variable is set, and fetch-pack will automatically request it if advertised. If the sideband is to be used throughout the whole response, upload-pack will use it to send errors instead of prefixing a PKT-LINE payload with "ERR ". This will be tested in a subsequent patch. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2019-01-17sideband: reverse its dependency on pkt-lineJonathan Tan
A subsequent patch will teach struct packet_reader a new field that, if set, instructs it to interpret read data as multiplexed. This will create a dependency from pkt-line to sideband. To avoid a circular dependency, split recv_sideband() into 2 parts: the reading loop (left in recv_sideband()) and the processing of the contents (in demultiplex_sideband()), and move the former into pkt-line. This reverses the direction of dependency: sideband no longer depends on pkt-line, and pkt-line now depends on sideband. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2019-01-15pkt-line: introduce struct packet_writerJonathan Tan
A future patch will allow the client to request multiplexing of the entire fetch response (and not only during packfile transmission), which in turn allows the server to send progress and keepalive messages at any time during the response. It will be convenient for a future patch if writing options (specifically, whether the written data is to be multiplexed) could be controlled from a single place, so create struct packet_writer to serve as that place, and modify upload-pack to use it. Currently, it only stores the output fd, but a subsequent patch will (as described above) introduce an option to determine if the written data is to be multiplexed. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2019-01-02pack-protocol.txt: accept error packets in any contextMasaya Suzuki
In the Git pack protocol definition, an error packet may appear only in a certain context. However, servers can face a runtime error (e.g. I/O error) at an arbitrary timing. This patch changes the protocol to allow an error packet to be sent instead of any packet. Without this protocol spec change, when a server cannot process a request, there's no way to tell that to a client. Since the server cannot produce a valid response, it would be forced to cut a connection without telling why. With this protocol spec change, the server can be more gentle in this situation. An old client may see these error packets as an unexpected packet, but this is not worse than having an unexpected EOF. Following this protocol spec change, the error packet handling code is moved to pkt-line.c. Implementation wise, this implementation uses pkt-line to communicate with a subprocess. Since this is not a part of Git protocol, it's possible that a packet that is not supposed to be an error packet is mistakenly parsed as an error packet. This error packet handling is enabled only for the Git pack protocol parsing code considering this. Signed-off-by: Masaya Suzuki <> Signed-off-by: Junio C Hamano <>
2018-03-15pkt-line: add packet_buf_write_len functionBrandon Williams
Add the 'packet_buf_write_len()' function which allows for writing an arbitrary length buffer into a 'struct strbuf' and formatting it in packet-line format. Signed-off-by: Brandon Williams <> Signed-off-by: Junio C Hamano <>
2018-03-14pkt-line: add delim packet supportBrandon Williams
One of the design goals of protocol-v2 is to improve the semantics of flush packets. Currently in protocol-v1, flush packets are used both to indicate a break in a list of packet lines as well as an indication that one side has finished speaking. This makes it particularly difficult to implement proxies as a proxy would need to completely understand git protocol instead of simply looking for a flush packet. To do this, introduce the special deliminator packet '0001'. A delim packet can then be used as a deliminator between lists of packet lines while flush packets can be reserved to indicate the end of a response. Documentation for how this packet will be used in protocol v2 will included in a future patch. Signed-off-by: Brandon Williams <> Signed-off-by: Junio C Hamano <>
2018-03-14pkt-line: allow peeking a packet line without consuming itBrandon Williams
Sometimes it is advantageous to be able to peek the next packet line without consuming it (e.g. to be able to determine the protocol version a server is speaking). In order to do that introduce 'struct packet_reader' which is an abstraction around the normal packet reading logic. This enables a caller to be able to peek a single line at a time using 'packet_reader_peek()' and having a caller consume a line by calling 'packet_reader_read()'. Signed-off-by: Brandon Williams <> Signed-off-by: Junio C Hamano <>
2018-03-14pkt-line: introduce packet_read_with_statusBrandon Williams
The current pkt-line API encodes the status of a pkt-line read in the length of the read content. An error is indicated with '-1', a flush with '0' (which can be confusing since a return value of '0' can also indicate an empty pkt-line), and a positive integer for the length of the read content otherwise. This doesn't leave much room for allowing the addition of additional special packets in the future. To solve this introduce 'packet_read_with_status()' which reads a packet and returns the status of the read encoded as an 'enum packet_status' type. This allows for easily identifying between special and normal packets as well as errors. It also enables easily adding a new special packet in the future. Signed-off-by: Brandon Williams <> Signed-off-by: Junio C Hamano <>
2017-10-17pkt-line: add packet_write functionBrandon Williams
Add a function which can be used to write the contents of an arbitrary buffer. This makes it easy to build up data in a buffer before writing the packet instead of formatting the entire contents of the packet using 'packet_write_fmt()'. Signed-off-by: Brandon Williams <> Signed-off-by: Junio C Hamano <>
2017-07-26sub-process: refactor handshake to common functionJonathan Tan
Refactor, into a common function, the version and capability negotiation done when invoking a long-running process as a clean or smudge filter. This will be useful for other Git code that needs to interact similarly with a long-running process. As you can see in the change to t0021, this commit changes the error message reported when the long-running process does not introduce itself with the expected "server"-terminated line. Originally, the error message reports that the filter "does not support filter protocol version 2", differentiating between the old single-file filter protocol and the new multi-file filter protocol - I have updated it to something more generic and useful. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2017-05-15pkt-line: annotate packet_writel with LAST_ARG_MUST_BE_NULLJeff King
packet_writel() takes a variable-sized list and reads to the first NULL. Let's let the compiler know so that it can help us catch mistakes in the callers. This should have been annotated similarly when it was a static function, but it's doubly important now that the function is available to the whole code-base. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2017-05-08convert: move packet_write_line() into pkt-line as packet_writel()Ben Peart
Add packet_writel() which writes multiple lines in a single call and then calls packet_flush_gently(). Update convert.c to use the new packet_writel() function from pkt-line. Signed-off-by: Ben Peart <> Signed-off-by: Junio C Hamano <>
2017-05-08pkt-line: add packet_read_line_gently()Ben Peart
Add packet_read_line_gently() to enable reading a line without dying on EOF. Signed-off-by: Ben Peart <> Signed-off-by: Junio C Hamano <>
2016-10-17pkt-line: add functions to read/write flush terminated packet streamsLars Schneider
write_packetized_from_fd() and write_packetized_from_buf() write a stream of packets. All content packets use the maximal packet size except for the last one. After the last content packet a `flush` control packet is written. read_packetized_to_strbuf() reads arbitrary sized packets until it detects a `flush` packet. Signed-off-by: Lars Schneider <> Signed-off-by: Junio C Hamano <>
2016-10-17pkt-line: add packet_flush_gently()Lars Schneider
packet_flush() would die in case of a write error even though for some callers an error would be acceptable. Add packet_flush_gently() which writes a pkt-line flush packet like packet_flush() but does not die in case of an error. The function is used in a subsequent patch. Signed-off-by: Lars Schneider <> Signed-off-by: Junio C Hamano <>
2016-10-17pkt-line: add packet_write_fmt_gently()Lars Schneider
packet_write_fmt() would die in case of a write error even though for some callers an error would be acceptable. Add packet_write_fmt_gently() which writes a formatted pkt-line like packet_write_fmt() but does not die in case of an error. The function is used in a subsequent patch. Signed-off-by: Lars Schneider <> Signed-off-by: Junio C Hamano <>
2016-10-17pkt-line: rename packet_write() to packet_write_fmt()Lars Schneider
packet_write() should be called packet_write_fmt() because it is a printf-like function that takes a format string as first parameter. packet_write_fmt() should be used for text strings only. Arbitrary binary data should use a new packet_write() function that is introduced in a subsequent patch. Suggested-by: Junio C Hamano <> Signed-off-by: Lars Schneider <> Signed-off-by: Junio C Hamano <>
2014-03-31comments: fix misuses of "nor"Justin Lebar
Signed-off-by: Justin Lebar <> Signed-off-by: Junio C Hamano <>
2013-02-24pkt-line: share buffer/descriptor reading implementationJeff King
The packet_read function reads from a descriptor. The packet_get_line function is similar, but reads from an in-memory buffer, and uses a completely separate implementation. This patch teaches the generic packet_read function to accept either source, and we can do away with packet_get_line's implementation. There are two other differences to account for between the old and new functions. The first is that we used to read into a strbuf, but now read into a fixed size buffer. The only two callers are fine with that, and in fact it simplifies their code, since they can use the same static-buffer interface as the rest of the packet_read_line callers (and we provide a similar convenience wrapper for reading from a buffer rather than a descriptor). This is technically an externally-visible behavior change in that we used to accept arbitrary sized packets up to 65532 bytes, and now cap out at LARGE_PACKET_MAX, 65520. In practice this doesn't matter, as we use it only for parsing smart-http headers (of which there is exactly one defined, and it is small and fixed-size). And any extension headers would be breaking the protocol to go over LARGE_PACKET_MAX anyway. The other difference is that packet_get_line would return on error rather than dying. However, both callers of packet_get_line are actually improved by dying. The first caller does its own error checking, but we can drop that; as a result, we'll actually get more specific reporting about protocol breakage when packet_read dies internally. The only downside is that packet_read will not print the smart-http URL that failed, but that's not a big deal; anybody not debugging can already see the remote's URL already, and anybody debugging would want to run with GIT_CURL_VERBOSE anyway to see way more information. The second caller, which is just trying to skip past any extra smart-http headers (of which there are none defined, but which we allow to keep room for future expansion), did not error check at all. As a result, it would treat an error just like a flush packet. The resulting mess would generally cause an error later in get_remote_heads, but now we get error reporting much closer to the source of the problem. Brown-paper-bag-fixes-by: Ramsay Jones <> Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: provide a LARGE_PACKET_MAX static bufferJeff King
Most of the callers of packet_read_line just read into a static 1000-byte buffer (callers which handle arbitrary binary data already use LARGE_PACKET_MAX). This works fine in practice, because: 1. The only variable-sized data in these lines is a ref name, and refs tend to be a lot shorter than 1000 characters. 2. When sending ref lines, git-core always limits itself to 1000 byte packets. However, the only limit given in the protocol specification in Documentation/technical/protocol-common.txt is LARGE_PACKET_MAX; the 1000 byte limit is mentioned only in pack-protocol.txt, and then only describing what we write, not as a specific limit for readers. This patch lets us bump the 1000-byte limit to LARGE_PACKET_MAX. Even though git-core will never write a packet where this makes a difference, there are two good reasons to do this: 1. Other git implementations may have followed protocol-common.txt and used a larger maximum size. We don't bump into it in practice because it would involve very long ref names. 2. We may want to increase the 1000-byte limit one day. Since packets are transferred before any capabilities, it's difficult to do this in a backwards-compatible way. But if we bump the size of buffer the readers can handle, eventually older versions of git will be obsolete enough that we can justify bumping the writers, as well. We don't have plans to do this anytime soon, but there is no reason not to start the clock ticking now. Just bumping all of the reading bufs to LARGE_PACKET_MAX would waste memory. Instead, since most readers just read into a temporary buffer anyway, let's provide a single static buffer that all callers can use. We can further wrap this detail away by having the packet_read_line wrapper just use the buffer transparently and return a pointer to the static storage. That covers most of the cases, and the remaining ones already read into their own LARGE_PACKET_MAX buffers. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: move LARGE_PACKET_MAX definition from sidebandJeff King
Having the packet sizes defined near the packet read/write functions makes more sense. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: teach packet_read_line to chomp newlinesJeff King
The packets sent during ref negotiation are all terminated by newline; even though the code to chomp these newlines is short, we end up doing it in a lot of places. This patch teaches packet_read_line to auto-chomp the trailing newline; this lets us get rid of a lot of inline chomping code. As a result, some call-sites which are not reading line-oriented data (e.g., when reading chunks of packfiles alongside sideband) transition away from packet_read_line to the generic packet_read interface. This patch converts all of the existing callsites. Since the function signature of packet_read_line does not change (but its behavior does), there is a possibility of new callsites being introduced in later commits, silently introducing an incompatibility. However, since a later patch in this series will change the signature, such a commit would have to be merged directly into this commit, not to the tip of the series; we can therefore ignore the issue. This is an internal cleanup and should produce no change of behavior in the normal case. However, there is one corner case to note. Callers of packet_read_line have never been able to tell the difference between a flush packet ("0000") and an empty packet ("0004"), as both cause packet_read_line to return a length of 0. Readers treat them identically, even though Documentation/technical/protocol-common.txt says we must not; it also says that implementations should not send an empty pkt-line. By stripping out the newline before the result gets to the caller, we will now treat the newline-only packet ("0005\n") the same as an empty packet, which in turn gets treated like a flush packet. In practice this doesn't matter, as neither empty nor newline-only packets are part of git's protocols (at least not for the line-oriented bits, and readers who are not expecting line-oriented packets will be calling packet_read directly, anyway). But even if we do decide to care about the distinction later, it is orthogonal to this patch. The right place to tighten would be to stop treating empty packets as flush packets, and this change does not make doing so any harder. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: provide a generic reading function with optionsJeff King
Originally we had a single function for reading packetized data: packet_read_line. Commit 46284dd grew a more "gentle" form, packet_read, that returns an error instead of dying upon reading a truncated input stream. However, it is not clear from the names which should be called, or what the difference is. Let's instead make packet_read be a generic public interface that can take option flags, and update the single callsite that uses it. This is less code, more clear, and paves the way for introducing more options into the generic interface later. The function signature is changed, so there should be no hidden conflicts with topics in flight. While we're at it, we'll document how error conditions are handled based on the options, and rename the confusing "return_line_fail" option to "gentle_on_eof". While we are cleaning up the names, we can drop the "return_line_fail" checks in packet_read_internal entirely. They look like this: ret = safe_read(..., return_line_fail); if (return_line_fail && ret < 0) ... The check for return_line_fail is a no-op; safe_read will only ever return an error value if return_line_fail was true in the first place. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: drop safe_write functionJeff King
This is just write_or_die by another name. The one distinction is that write_or_die will treat EPIPE specially by suppressing error messages. That's fine, as we die by SIGPIPE anyway (and in the off chance that it is disabled, write_or_die will simulate it). Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2013-02-20pkt-line: move a misplaced commentJeff King
The comment describing the packet writing interface was originally written above packet_write, but migrated to be above safe_write in f3a3214, probably because it is meant to generally describe the packet writing interface and not a single function. Let's move it into the header file, where users of the interface are more likely to see it. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2012-06-19remove the impression of unexpectedness when access is deniedHeiko Voigt
If a server accessed through ssh is denying access git will currently issue the message "fatal: The remote end hung up unexpectedly" as the last line. This sounds as if something really ugly just happened. Since this is a quite typical situation in which users regularly get we do not say that if it happens at the beginning when reading the remote heads. If its in the very first beginning of reading the remote heads it is very likely an authentication error or a missing repository. If it happens later during reading the remote heads we still indicate that it happened during this initial contact phase. Signed-off-by: Heiko Voigt <> Signed-off-by: Junio C Hamano <>
2009-10-31pkt-line: Add strbuf based functionsShawn O. Pearce
These routines help to work with pkt-line values inside of a strbuf, permitting simple formatting of buffered network messages. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-06-26Fix pkt-line.h to compile with a non-GCC compilerDennis Stosberg
pkt-line.h uses GCC's __attribute__ extension but does not include git-compat-util.h. So it will not compile with a compiler that does not support this extension. Signed-off-by: Junio C Hamano <>
2006-06-21upload-pack/fetch-pack: support side-band communicationJunio C Hamano
This implements a protocol extension between fetch-pack and upload-pack to allow stderr stream from upload-pack (primarily used for the progress bar display) to be passed back. Signed-off-by: Junio C Hamano <>
2005-08-10[PATCH] -Werror fixesTimo Sirainen
GCC's format __attribute__ is good for checking errors, especially with -Wformat=2 parameter. This fixes most of the reported problems against 2005-08-09 snapshot.
2005-06-30Make send/receive-pack be closer to doing something interestingLinus Torvalds