summaryrefslogtreecommitdiff
path: root/Documentation/technical/pack-protocol.txt
AgeCommit message (Collapse)Author
2018-02-13Merge branch 'jh/partial-clone'Junio C Hamano
The machinery to clone & fetch, which in turn involves packing and unpacking objects, have been told how to omit certain objects using the filtering mechanism introduced by the jh/object-filtering topic, and also mark the resulting pack as a promisor pack to tolerate missing objects, taking advantage of the mechanism introduced by the jh/fsck-promisors topic. * jh/partial-clone: t5616: test bulk prefetch after partial fetch fetch: inherit filter-spec from partial clone t5616: end-to-end tests for partial clone fetch-pack: restore save_commit_buffer after use unpack-trees: batch fetching of missing blobs clone: partial clone partial-clone: define partial clone settings in config fetch: support filters fetch: refactor calculation of remote list fetch-pack: test support excluding large blobs fetch-pack: add --no-filter fetch-pack, index-pack, transport: partial clone upload-pack: add object filtering for partial clone
2017-12-08upload-pack: add object filtering for partial cloneJeff Hostetler
Teach upload-pack to negotiate object filtering over the protocol and to send filter parameters to pack-objects. This is intended for partial clone and fetch. The idea to make upload-pack configurable using uploadpack.allowFilter comes from Jonathan Tan's work in [1]. [1] https://public-inbox.org/git/f211093280b422c32cc1b7034130072f35c5ed51.1506714999.git.jonathantanmy@google.com/ Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-17Documentation: document Extra ParametersJonathan Tan
Document the server support for Extra Parameters, additional information that the client can send in its first message to the server during a Git client-server interaction. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-08-23treewide: correct several "up-to-date" to "up to date"Martin Ågren
Follow the Oxford style, which says to use "up-to-date" before the noun, but "up to date" after it. Don't change plumbing (specifically send-pack.c, but transport.c (git push) also has the same string). This was produced by grepping for "up-to-date" and "up to date". It turned out we only had to edit in one direction, removing the hyphens. Fix a typo in Documentation/git-diff-index.txt while we're there. Reported-by: Jeffrey Manian <jeffrey.manian@gmail.com> Reported-by: STEVEN WHITE <stevencharleswhitevoices@gmail.com> Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-05-23Merge branch 'jt/push-options-doc'Junio C Hamano
The receive-pack program now makes sure that the push certificate records the same set of push options used for pushing. * jt/push-options-doc: receive-pack: verify push options in cert docs: correct receive.advertisePushOptions default
2017-05-10receive-pack: verify push options in certJonathan Tan
In commit f6a4e61 ("push: accept push options", 2016-07-14), send-pack was taught to include push options both within the signed cert (if the push is a signed push) and outside the signed cert; however, receive-pack ignores push options within the cert, only handling push options outside the cert. Teach receive-pack, in the case that push options are provided for a signed push, to verify that the push options both within the cert and outside the cert are consistent. This sets in stone the requirement that send-pack redundantly send its push options in 2 places, but I think that this is better than the alternatives. Sending push options only within the cert is backwards-incompatible with existing Git servers (which read push options only from outside the cert), and sending push options only outside the cert means that the push options are not signed for. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-18fetch-pack: show clearer error message upon ERRJonathan Tan
Currently, fetch-pack prints a confusing error message ("expected ACK/NAK") when the server it's communicating with sends a pkt-line starting with "ERR". Replace it with a less confusing error message. Also update the documentation describing the fetch-pack/upload-pack protocol (pack-protocol.txt) to indicate that "ERR" can be sent in the place of "ACK" or "NAK". In practice, this has been done for quite some time by other Git implementations (e.g. JGit sends "want $id not valid") and by Git itself (since commit bdb31ea: "upload-pack: report "not our ref" to client", 2017-02-23) whenever a "want" line references an object that it does not have. (This is uncommon, but can happen if a repository is garbage-collected during a negotiation.) Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-10Merge branch 'nd/shallow-deepen'Junio C Hamano
The existing "git fetch --depth=<n>" option was hard to use correctly when making the history of an existing shallow clone deeper. A new option, "--deepen=<n>", has been added to make this easier to use. "git clone" also learned "--shallow-since=<date>" and "--shallow-exclude=<tag>" options to make it easier to specify "I am interested only in the recent N months worth of history" and "Give me only the history since that version". * nd/shallow-deepen: (27 commits) fetch, upload-pack: --deepen=N extends shallow boundary by N commits upload-pack: add get_reachable_list() upload-pack: split check_unreachable() in two, prep for get_reachable_list() t5500, t5539: tests for shallow depth excluding a ref clone: define shallow clone boundary with --shallow-exclude fetch: define shallow boundary with --shallow-exclude upload-pack: support define shallow boundary by excluding revisions refs: add expand_ref() t5500, t5539: tests for shallow depth since a specific date clone: define shallow clone boundary based on time with --shallow-since fetch: define shallow boundary with --shallow-since upload-pack: add deepen-since to cut shallow repos based on time shallow.c: implement a generic shallow boundary finder based on rev-list fetch-pack: use a separate flag for fetch in deepening mode fetch-pack.c: mark strings for translating fetch-pack: use a common function for verbose printing fetch-pack: use skip_prefix() instead of starts_with() upload-pack: move rev-list code out of check_non_tip() upload-pack: make check_non_tip() clean things up on error upload-pack: tighten number parsing at "deepen" lines ...
2016-08-04Merge branch 'sb/pack-protocol-doc-nak'Junio C Hamano
A doc update. * sb/pack-protocol-doc-nak: Documentation: pack-protocol correct NAK response
2016-07-22Documentation: pack-protocol correct NAK responseStefan Beller
In the transport protocol we use NAK to signal the non existence of a common base, so fix the documentation. This helps readers of the document, as they don't have to wonder about the difference between NAK and NACK. As NACK is used in git archive and upload-archive, this is easy to get wrong. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-14receive-pack: implement advertising and receiving push optionsStefan Beller
The pre/post receive hook may be interested in more information from the user. This information can be transmitted when both client and server support the "push-options" capability, which when used is a phase directly after update commands ended by a flush pkt. Similar to the atomic option, the server capability can be disabled via the `receive.advertisePushOptions` config variable. While documenting this, fix a nit in the `receive.advertiseAtomic` wording. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-13upload-pack: support define shallow boundary by excluding revisionsNguyễn Thái Ngọc Duy
This should allow the user to say "create a shallow clone of this branch after version <some-tag>". Short refs are accepted and expanded at the server side with expand_ref() because we cannot expand (unknown) refs from the client side. Like deepen-since, deepen-not cannot be used with deepen. But deepen-not can be mixed with deepen-since. The result is exactly how you do the command "git rev-list --since=... --not ref". Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-13upload-pack: add deepen-since to cut shallow repos based on timeNguyễn Thái Ngọc Duy
This should allow the user to say "create a shallow clone containing the work from last year" (once the client side is fixed up, of course). In theory deepen-since and deepen (aka --depth) can be used together to draw the shallow boundary (whether it's intersection or union is up to discussion, but if rev-list is used, it's likely intersection). However, because deepen goes with a custom commit walker, we can't mix the two yet. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-05-06typofix: assorted typofixes in comments, documentation and messagesLi Peng
Many instances of duplicate words (e.g. "the the path") and a few typoes are fixed, originally in multiple patches. wildmatch: fix duplicate words of "the" t: fix duplicate words of "output" transport-helper: fix duplicate words of "read" Git.pm: fix duplicate words of "return" path: fix duplicate words of "look" pack-protocol.txt: fix duplicate words of "the" precompose-utf8: fix typo of "sequences" split-index: fix typo worktree.c: fix typo remote-ext: fix typo utf8: fix duplicate words of "the" git-cvsserver: fix duplicate words Signed-off-by: Li Peng <lip@dtdream.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-09-03pack-protocol: clarify LF-handling in PKT-LINE()Jeff King
The spec is very inconsistent about which PKT-LINE() parts of the grammar include a LF. On top of that, the code is not consistent, either (e.g., send-pack does not put newlines into the ref-update commands it sends). Let's make explicit the long-standing expectation that we generally expect pkt-lines to end in a newline, but that receivers should be lenient. This makes the spec consistent, and matches what git already does (though it does not always fulfill the SHOULD). We do make an exception for the push-cert, where the receiving code is currently a bit pickier. This is a reasonable way to be, as the data needs to be byte-for-byte compatible with what was signed. We _could_ make up some rules about signing a canonicalized version including newlines, but that would require a code change, and is out of scope for this patch. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-06-25Merge branch 'sb/pack-protocol-mention-smart-http' into maintJunio C Hamano
Doc updates. * sb/pack-protocol-mention-smart-http: Documentation/technical/pack-protocol: mention http as possible protocol
2015-06-16Merge branch 'sb/pack-protocol-mention-smart-http'Junio C Hamano
Doc updates. * sb/pack-protocol-mention-smart-http: Documentation/technical/pack-protocol: mention http as possible protocol
2015-06-02Documentation/technical/pack-protocol: mention http as possible protocolStefan Beller
Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-22doc: fix inconsistent spelling of "packfile"Patrick Steinhardt
Fix remaining instances where "pack-file" is used instead of "packfile". Some places remain where we still use "pack-file", This is the case when we explicitly refer to a file with a ".pack" extension as opposed to a data source providing a pack data stream. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-17pack-protocol.txt: fix insconsistent spelling of "packfile"Patrick Steinhardt
Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-10-08Merge branch 'jc/push-cert'Junio C Hamano
Allow "git push" request to be signed, so that it can be verified and audited, using the GPG signature of the person who pushed, that the tips of branches at a public repository really point the commits the pusher wanted to, without having to "trust" the server. * jc/push-cert: (24 commits) receive-pack::hmac_sha1(): copy the entire SHA-1 hash out signed push: allow stale nonce in stateless mode signed push: teach smart-HTTP to pass "git push --signed" around signed push: fortify against replay attacks signed push: add "pushee" header to push certificate signed push: remove duplicated protocol info send-pack: send feature request on push-cert packet receive-pack: GPG-validate push certificates push: the beginning of "git push --signed" pack-protocol doc: typofix for PKT-LINE gpg-interface: move parse_signature() to where it should be gpg-interface: move parse_gpg_output() to where it should be send-pack: clarify that cmds_sent is a boolean send-pack: refactor inspecting and resetting status and sending commands send-pack: rename "new_refs" to "need_pack_data" receive-pack: factor out capability string generation send-pack: factor out capability string generation send-pack: always send capabilities send-pack: refactor decision to send update per ref send-pack: move REF_STATUS_REJECT_NODELETE logic a bit higher ...
2014-09-17signed push: fortify against replay attacksJunio C Hamano
In order to prevent a valid push certificate for pushing into an repository from getting replayed in a different push operation, send a nonce string from the receive-pack process and have the signer include it in the push certificate. The receiving end uses an HMAC hash of the path to the repository it serves and the current time stamp, hashed with a secret seed (the secret seed does not have to be per-repository but can be defined in /etc/gitconfig) to generate the nonce, in order to ensure that a random third party cannot forge a nonce that looks like it originated from it. The original nonce is exported as GIT_PUSH_CERT_NONCE for the hooks to examine and match against the value on the "nonce" header in the certificate to notice a replay, but returned "nonce" header in the push certificate is examined by receive-pack and the result is exported as GIT_PUSH_CERT_NONCE_STATUS, whose value would be "OK" if the nonce recorded in the certificate matches what we expect, so that the hooks can more easily check. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-09-15signed push: add "pushee" header to push certificateJunio C Hamano
Record the URL of the intended recipient for a push (after anonymizing it if it has authentication material) on a new "pushee URL" header. Because the networking configuration (SSH-tunnels, proxies, etc.) on the pushing user's side varies, the receiving repository may not know the single canonical URL all the pushing users would refer it as (besides, many sites allow pushing over ssh://host/path and https://host/path protocols to the same repository but with different local part of the path). So this value may not be reliably used for replay-attack prevention purposes, but this will still serve as a human readable hint to identify the repository the certificate refers to. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-09-15signed push: remove duplicated protocol infoJunio C Hamano
With the interim protocol, we used to send the update commands even though we already send a signed copy of the same information when push certificate is in use. Update the send-pack/receive-pack pair not to do so. The notable thing on the receive-pack side is that it makes sure that there is no command sent over the traditional protocol packet outside the push certificate. Otherwise a pusher can claim to be pushing one set of ref updates in the signed certificate while issuing commands to update unrelated refs, and such an update will evade later audits. Finally, start documenting the protocol. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-09-15pack-protocol doc: typofix for PKT-LINEJunio C Hamano
Everywhere else we use PKT-LINE to denote the pkt-line formatted data, but "shallow/deepen" messages are described with PKT_LINE(). Fix them. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-08-28Document LF appearing in shallow command during send-pack/receive-packShawn Pearce
The implementation sends an LF, but the protocol documentation was missing this detail. Signed-off-by: Shawn Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-03-31Documentation: fix misuses of "nor"Justin Lebar
Signed-off-by: Justin Lebar <jlebar@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-02-10pack-protocol.txt: clarify 'obj-id' in the last ACK after 'done'Nguyễn Thái Ngọc Duy
It's introduced in 1bd8c8f (git-upload-pack: Support the multi_ack protocol - 2005-10-28) but probably better documented in the commit message of 78affc4 (Add multi_ack_detailed capability to fetch-pack/upload-pack - 2009-10-30). Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-12-11receive/send-pack: support pushing from a shallow cloneNguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-12-11make the sender advertise shallow commits to the receiverNguyễn Thái Ngọc Duy
If either receive-pack or upload-pack is called on a shallow repository, shallow commits (*) will be sent after the ref advertisement (but before the packet flush), so that the receiver has the full "shape" of the sender's commit graph. This will be needed for the receiver to update its .git/shallow if necessary. This breaks the protocol for all clients trying to push to a shallow repo, or fetch from one. Which is basically the same end result as today's "is_repository_shallow() && die()" in receive-pack and upload-pack. New clients will be made aware of shallow upstream and can make use of this information. The sender must send all shallow commits that are sent in the following pack. It may send more shallow commits than necessary. upload-pack for example may choose to advertise no shallow commits if it knows in advance that the pack it's going to send contains no shallow commits. But upload-pack is the server, so we choose the cheaper way, send full .git/shallow and let the client deal with it. Smart HTTP is not affected by this patch. Shallow support on smart-http comes later separately. (*) A shallow commit is a commit that terminates the revision walker. It is usually put in .git/shallow in order to keep the revision walker from going out of bound because there is no guarantee that objects behind this commit is available. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-29upload-pack: ignore 'shallow' lines with unknown obj-idsMichael Heemskerk
When the client sends a 'shallow' line for an object that the server does not have, the server currently dies with the error: "did not find object for shallow <obj-id>". The client may have truncated the history at the commit by fetching shallowly from a different server, or the commit may have been garbage collected by the server. In either case, this unknown commit is not relevant for calculating the pack that is to be sent and can be safely ignored, and it is not used when recomputing where the updated history of the client is cauterised. The documentation in technical/pack-protocol.txt has been updated to remove the restriction that "Clients MUST NOT mention an obj-id which it does not know exists on the server". This requirement is not realistic because clients cannot know whether an object has been garbage collected by the server. Signed-off-by: Michael Heemskerk <mheemskerk@atlassian.com> Reviewed-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-10-16Documentation/technical: convert plain text files to asciidocThomas Ackermann
These were not originally meant for asciidoc, but they are already so close. Mark them up in asciidoc. Signed-off-by: Thomas Ackermann <th.acker@arcor.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-18Doc: Improve shallow depth wordingPhilip Oakley
Avoid confusion in compound sentence about the start of the commit set and the depth measure. Use two sentences. Signed-off-by: Philip Oakley <philipoakley@iee.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-05-14pack-protocol: fix first-want separator in the examplesCarlos Martín Nieto
When sending the "want" list, the capabilities list is separated from the obj-id by a SP instead of NUL as in the ref advertisement. The text is correct, but the examples wrongly show the separator as NUL. Fix the example so it uses SP. Signed-off-by: Carlos Martín Nieto <cmn@elego.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-10-03pack-protocol: document "ERR" lineNguyen Thai Ngoc Duy
Since a807328 (connect.c: add a way for git-daemon to pass an error back to client), git client recognizes "ERR" line and prints a friendly message to user if an error happens at server side. Document this. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-06-09Document the underlying protocol used by shallow repositories and --depth ↵Alex Neronskiy
commands. Explain the exchange that occurs between a client and server when the client is requesting shallow history and/or is already using a shallow repository. Signed-off-by: Alex Neronskiy <zakmagnus@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-06-09Fix documentation of fetch-pack that implies that the client can disconnect ↵Alex Neronskiy
after sending wants. Specify conditions under which the client can terminate the connection early. Previously, an unintended behavior was possible which could confuse servers. Based-on-patch-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Alex Neronskiy <zakmagnus@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-04pack-protocol.txt: fix pkt-line lengthsTay Ray Chuan
Previously, the lengths were 4-bytes short. Fix it such that the lengths reflect the total length of the pkt-line, as per spec. Signed-off-by: Tay Ray Chuan <rctay89@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-04pack-protocol.txt: fix spellingTay Ray Chuan
s/paramater/parameter/. Signed-off-by: Tay Ray Chuan <rctay89@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-31Fix typos in technical documentation.Ralf Wildenhues
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-11-13Update packfile transfer protocol documentationScott Chacon
The current technical documentation for the packfile protocol is both sparse and incorrect. This documents the fetch-pack/upload-pack and send-pack/ receive-pack protocols much more fully. Add documentation from Shawn's upcoming http-protocol docs that is shared by the packfile protocol. protocol-common.txt describes ABNF notation amendments, refname rules and the packet line format. Add documentation on the various capabilities supported by the upload-pack and receive-pack protocols. protocol-capabilities.txt describes multi-ack, thin-pack, side-band[-64k], shallow, no-progress, include-tag, ofs-delta, delete-refs and report-status. Signed-off-by: Scott Chacon <schacon@gmail.com> Signed-off-by: Nanako Shiraishi <nanako3@lavabit.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2005-12-13Documentation: fix missing links to git(7)Junio C Hamano
Also move pack protocol description to technical/. Signed-off-by: Junio C Hamano <junkio@cox.net>