path: root/Documentation/gitremote-helpers.txt
AgeCommit message (Collapse)Author
2020-07-07Merge branch 'bc/sha-256-part-2'Junio C Hamano
SHA-256 migration work continues. * bc/sha-256-part-2: (44 commits) remote-testgit: adapt for object-format bundle: detect hash algorithm when reading refs t5300: pass --object-format to git index-pack t5704: send object-format capability with SHA-256 t5703: use object-format serve option t5702: offer an object-format capability in the test t/helper: initialize the repository for test-sha1-array remote-curl: avoid truncating refs with ls-remote t1050: pass algorithm to index-pack when outside repo builtin/index-pack: add option to specify hash algorithm remote-curl: detect algorithm for dumb HTTP by size builtin/ls-remote: initialize repository based on fetch t5500: make hash independent serve: advertise object-format capability for protocol v2 connect: parse v2 refs with correct hash algorithm connect: pass full packet reader when parsing v2 refs Documentation/technical: document object-format for protocol v2 t1302: expect repo format version 1 for SHA-256 builtin/show-index: provide options to determine hash algo t5302: modernize test formatting ...
2020-05-27docs: update remote helper docs for object-format extensionsbrian m. carlson
Update the remote helper docs to document the object-format extensions we will implement in remote-curl and the transport helper code shortly. Signed-off-by: brian m. carlson <> Signed-off-by: Junio C Hamano <>
2020-05-24stateless-connect: send response end packetDenton Liu
Currently, remote-curl acts as a proxy and blindly forwards packets between an HTTP server and fetch-pack. In the case of a stateless RPC connection where the connection is terminated before the transaction is complete, remote-curl will blindly forward the packets before waiting on more input from fetch-pack. Meanwhile, fetch-pack will read the transaction and continue reading, expecting more input to continue the transaction. This results in a deadlock between the two processes. This can be seen in the following command which does not terminate: $ git -c protocol.version=2 clone --shallow-since=20151012 Cloning into 'git'... whereas the v1 version does terminate as expected: $ git -c protocol.version=1 clone --shallow-since=20151012 Cloning into 'git'... fatal: the remote end hung up unexpectedly Instead of blindly forwarding packets, make remote-curl insert a response end packet after proxying the responses from the remote server when using stateless_connect(). On the RPC client side, ensure that each response ends as described. A separate control packet is chosen because we need to be able to differentiate between what the remote server sends and remote-curl's control packets. By ensuring in the remote-curl code that a server cannot send response end packets, we prevent a malicious server from being able to perform a denial of service attack in which they spoof a response end packet and cause the described deadlock to happen. Reported-by: Force Charlie <> Helped-by: Jeff King <> Signed-off-by: Denton Liu <> Signed-off-by: Junio C Hamano <>
2019-10-23Merge branch 'bc/smart-http-atomic-push'Junio C Hamano
The atomic push over smart HTTP transport did not work, which has been corrected. * bc/smart-http-atomic-push: remote-curl: pass on atomic capability to remote side
2019-10-17remote-curl: pass on atomic capability to remote sidebrian m. carlson
When pushing more than one reference with the --atomic option, the server is supposed to perform a single atomic transaction to update the references, leaving them either all to succeed or all to fail. This works fine when pushing locally or over SSH, but when pushing over HTTP, we fail to pass the atomic capability to the remote side. In fact, we have not reported this capability to any remote helpers during the life of the feature. Now normally, things happen to work nevertheless, since we actually check for most types of failures, such as non-fast-forward updates, on the client side, and just abort the entire attempt. However, if the server side reports a problem, such as the inability to lock a ref, the transaction isn't atomic, because we haven't passed the appropriate capability over and the remote side has no way of knowing that we wanted atomic behavior. Fix this by passing the option from the transport code through to remote helpers, and from the HTTP remote helper down to send-pack. With this change, we can detect if the server side rejects the push and report back appropriately. Note the difference in the messages: the remote side reports "atomic transaction failed", while our own checking rejects pushes with the message "atomic push failed". Document the atomic option in the remote helper documentation, so other implementers can implement it if they like. Signed-off-by: brian m. carlson <> Signed-off-by: Junio C Hamano <>
2019-08-30clarify documentation for remote helpersDavid Turner
Signed-off-by: David Turner <> Signed-off-by: Junio C Hamano <>
2019-04-22Merge branch 'js/check-docs-exe'Junio C Hamano
Dev support update. * js/check-docs-exe: check-docs: fix for setups where executables have an extension check-docs: do not expect guide pages to correspond to commands check-docs: really look at the documented commands again docs: do not document the `git remote-testgit` command docs: move gitremote-helpers into section 7
2019-04-01docs: do not document the `git remote-testgit` commandJohannes Schindelin
Since 7ded055401 (build: do not install git-remote-testgit, 2013-06-07), we do not install it. Therefore it makes no sense to document it, either. Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2019-04-01docs: move gitremote-helpers into section 7Johannes Schindelin
It is currently in section 1, but that section is intended for "Executable programs or shell commands". A more appropriate place is section 7: "Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7)". This issue should have been detected earlier by `make check-docs`, but was missed due to a bug in that Makefile target (that we are about to fix). Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2019-03-07Documentation: turn middle-of-line tabs into spacesMartin Ågren
These tabs happen to appear in columns where they don't stand out too much, so the diff here is non-obvious. Some of these are rendered differently by AsciiDoc and Asciidoctor (although the difference might be invisible!), which is how I found a few of them. The remainder were found using `git grep "[a-zA-Z.,)]$TAB[a-zA-Z]"`. Signed-off-by: Martin Ågren <> Signed-off-by: Junio C Hamano <>
2018-05-08Merge branch 'bw/protocol-v2'Junio C Hamano
The beginning of the next-gen transfer protocol. * bw/protocol-v2: (35 commits) remote-curl: don't request v2 when pushing remote-curl: implement stateless-connect command http: eliminate "# service" line when using protocol v2 http: don't always add Git-Protocol header http: allow providing extra headers for http requests remote-curl: store the protocol version the server responded with remote-curl: create copy of the service name pkt-line: add packet_buf_write_len function transport-helper: introduce stateless-connect transport-helper: refactor process_connect_service transport-helper: remove name parameter connect: don't request v2 when pushing connect: refactor git_connect to only get the protocol version once fetch-pack: support shallow requests fetch-pack: perform a fetch using v2 upload-pack: introduce fetch server command push: pass ref prefixes when pushing fetch: pass ref prefixes when fetching ls-remote: pass ref prefixes when requesting a remote's refs transport: convert transport_get_remote_refs to take a list of ref prefixes ...
2018-03-15transport-helper: introduce stateless-connectBrandon Williams
Introduce the transport-helper capability 'stateless-connect'. This capability indicates that the transport-helper can be requested to run the 'stateless-connect' command which should attempt to make a stateless connection with a remote end. Once established, the connection can be used by the git client to communicate with the remote end natively in a stateless-rpc manner as supported by protocol v2. This means that the client must send everything the server needs in a single request as the client must not assume any state-storing on the part of the server or transport. If a stateless connection cannot be established then the remote-helper will respond in the same manner as the 'connect' command indicating that the client should fallback to using the dumb remote-helper commands. A future patch will implement the 'stateless-connect' capability in our http remote-helper (remote-curl) so that protocol v2 can be used using the http transport. Signed-off-by: Brandon Williams <> Signed-off-by: Junio C Hamano <>
2017-12-05introduce fetch-object: fetch one promisor objectJonathan Tan
Introduce fetch-object, providing the ability to fetch one object from a promisor remote. This uses fetch-pack. To do this, the transport mechanism has been updated with 2 flags, "from-promisor" to indicate that the resulting pack comes from a promisor remote (and thus should be annotated as such by index-pack), and "no-dependents" to indicate that only the objects themselves need to be fetched (but fetching additional objects is nevertheless safe). Whenever "no-dependents" is used, fetch-pack will refrain from using any object flags, because it is most likely invoked as part of a dynamic object fetch by another Git command (which may itself use object flags). An alternative to this is to leave fetch-pack alone, and instead update the allocation of flags so that fetch-pack's flags never overlap with any others, but this will end up shrinking the number of flags available to nearly every other Git command (that is, every Git command that accesses objects), so the approach in this commit was used instead. This will be tested in a subsequent commit. Signed-off-by: Jonathan Tan <> Signed-off-by: Junio C Hamano <>
2017-05-01fix minor typosRené Genz
Helped-by: Stefan Beller <> Signed-off-by: René Genz <> Signed-off-by: Junio C Hamano <>
2017-02-15Merge branch 'jk/doc-remote-helpers-markup-fix'Junio C Hamano
Doc markup fix. * jk/doc-remote-helpers-markup-fix: docs/gitremote-helpers: fix unbalanced quotes
2017-02-13docs/gitremote-helpers: fix unbalanced quotesJeff King
Each of these options is missing the closing single-quote on the option name. This understandably confuses asciidoc, which ends up rendering a stray quote, like: option cloning {'true|false} Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2017-02-08push options: pass push options to the transport helperStefan Beller
When using non-builtin protocols relying on a transport helper (such as http), push options are not propagated to the helper. The user could ask for push options and a push would seemingly succeed, but the push options would never be transported to the server, misleading the users expectation. Fix this by propagating the push options to the transport helper. This is only addressing the first issue of (1) the helper protocol does not propagate push-option (2) the http helper is not prepared to handle push-option Once we fix (2), the http transport helper can make use of push options as well, but that happens as a follow up. (1) is a bug fix, whereas (2) is a feature, which is why we only do (1) here. Signed-off-by: Stefan Beller <> Signed-off-by: Junio C Hamano <>
2016-10-10Merge branch 'nd/shallow-deepen'Junio C Hamano
The existing "git fetch --depth=<n>" option was hard to use correctly when making the history of an existing shallow clone deeper. A new option, "--deepen=<n>", has been added to make this easier to use. "git clone" also learned "--shallow-since=<date>" and "--shallow-exclude=<tag>" options to make it easier to specify "I am interested only in the recent N months worth of history" and "Give me only the history since that version". * nd/shallow-deepen: (27 commits) fetch, upload-pack: --deepen=N extends shallow boundary by N commits upload-pack: add get_reachable_list() upload-pack: split check_unreachable() in two, prep for get_reachable_list() t5500, t5539: tests for shallow depth excluding a ref clone: define shallow clone boundary with --shallow-exclude fetch: define shallow boundary with --shallow-exclude upload-pack: support define shallow boundary by excluding revisions refs: add expand_ref() t5500, t5539: tests for shallow depth since a specific date clone: define shallow clone boundary based on time with --shallow-since fetch: define shallow boundary with --shallow-since upload-pack: add deepen-since to cut shallow repos based on time shallow.c: implement a generic shallow boundary finder based on rev-list fetch-pack: use a separate flag for fetch in deepening mode fetch-pack.c: mark strings for translating fetch-pack: use a common function for verbose printing fetch-pack: use skip_prefix() instead of starts_with() upload-pack: move rev-list code out of check_non_tip() upload-pack: make check_non_tip() clean things up on error upload-pack: tighten number parsing at "deepen" lines ...
2016-06-28doc: typeset HEAD and variants as literalMatthieu Moy
This is an application of the newly added CodingGuidelines to HEAD and variants like FETCH_HEAD. It was obtained with: perl -pi -e "s/'([A-Z_]*HEAD)'/\`\$1\`/g" *.txt Signed-off-by: Matthieu Moy <> Signed-off-by: Junio C Hamano <>
2016-06-28doc: typeset long command-line options as literalMatthieu Moy
Similarly to the previous commit, use backquotes instead of forward-quotes, for long options. This was obtained with: perl -pi -e "s/'(--[a-z][a-z=<>-]*)'/\`\$1\`/g" *.txt and manual tweak to remove false positive in ascii-art (o'--o'--o' to describe rewritten history). Signed-off-by: Matthieu Moy <> Signed-off-by: Junio C Hamano <>
2016-06-13fetch, upload-pack: --deepen=N extends shallow boundary by N commitsNguyễn Thái Ngọc Duy
In git-fetch, --depth argument is always relative with the latest remote refs. This makes it a bit difficult to cover this use case, where the user wants to make the shallow history, say 3 levels deeper. It would work if remote refs have not moved yet, but nobody can guarantee that, especially when that use case is performed a couple months after the last clone or "git fetch --depth". Also, modifying shallow boundary using --depth does not work well with clones created by --since or --not. This patch fixes that. A new argument --deepen=<N> will add <N> more (*) parent commits to the current history regardless of where remote refs are. Have/Want negotiation is still respected. So if remote refs move, the server will send two chunks: one between "have" and "want" and another to extend shallow history. In theory, the client could send no "want"s in order to get the second chunk only. But the protocol does not allow that. Either you send no want lines, which means ls-remote; or you have to send at least one want line that carries deep-relative to the server.. The main work was done by Dongcan Jiang. I fixed it up here and there. And of course all the bugs belong to me. (*) We could even support --deepen=<N> where <N> is negative. In that case we can cut some history from the shallow clone. This operation (and --depth=<shorter depth>) does not require interaction with remote side (and more complicated to implement as a result). Helped-by: Duy Nguyen <> Helped-by: Eric Sunshine <> Helped-by: Junio C Hamano <> Signed-off-by: Dongcan Jiang <> Signed-off-by: Nguyễn Thái Ngọc Duy <> Signed-off-by: Junio C Hamano <>
2016-06-13fetch: define shallow boundary with --shallow-excludeNguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <> Signed-off-by: Junio C Hamano <>
2016-06-13fetch: define shallow boundary with --shallow-sinceNguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <> Signed-off-by: Junio C Hamano <>
2016-06-08doc: change configuration variables formatTom Russello
This change configuration variables that where in italic style to monospace font according to the guideline. It was obtained with grep '[[:alpha:]]*\.[[:alpha:]]*::$' config.txt | \ sed -e 's/::$//' -e 's/\./\\\\./' | \ xargs -iP perl -pi -e "s/\'P\'/\`P\`/g" ./*.txt Signed-off-by: Tom Russello <> Signed-off-by: Erwan Mathoniere <> Signed-off-by: Samuel Groot <> Signed-off-by: Matthieu Moy <> Reviewed-by: Matthieu Moy <> Signed-off-by: Junio C Hamano <>
2016-06-08doc: change environment variables formatTom Russello
This change GIT_* variables that where in italic style to monospaced font according to the guideline. It was obtained with perl -pi -e "s/\'(GIT_.*?)\'/\`\1\`/g" *.txt One of the main purposes is to stick to the CodingGuidelines as possible so that people writting new documentation by mimicking the existing are more likely to have it right (even if they didn't read the CodingGuidelines). Signed-off-by: Tom Russello <> Signed-off-by: Erwan Mathoniere <> Signed-off-by: Samuel Groot <> Signed-off-by: Matthieu Moy <> Reviewed-by: Matthieu Moy <> Signed-off-by: Junio C Hamano <>
2015-08-19gitremote-helpers.txt: document pushcert optionDave Borowitz
Signed-off-by: Dave Borowitz <> Signed-off-by: Junio C Hamano <>
2015-05-13doc: drop backslash quoting of some curly bracesJeff King
Text like "{foo}" triggers an AsciiDoc attribute; we have to write "\{foo}" to suppress this. But when the "foo" is not a syntactically valid attribute, we can skip the quoting. This makes the source nicer to read, and looks better under Asciidoctor. With AsciiDoc itself, this patch produces no changes. Signed-off-by: Jeff King <> Signed-off-by: Junio C Hamano <>
2014-11-11doc: add some crossrefs between manual pagesMax Horn
In particular, git-fast-import and -export link to each other, and gitremote-helpers links to existing remote helpers, and vice versa. Also link to fast-import from the remote helper spec, as this is relevant for remote helpers using the fast-import format. Signed-off-by: Max Horn <> Signed-off-by: Junio C Hamano <>
2014-03-18Merge branch 'fc/transport-helper-fixes'Junio C Hamano
Updates transport-helper, fast-import and fast-export to allow the ref mapping and ref deletion in a way similar to the natively supported transports. * fc/transport-helper-fixes: remote-bzr: support the new 'force' option tests are now expected to pass transport-helper.c: do not overwrite forced bit transport-helper: check for 'forced update' message transport-helper: add 'force' to 'export' helpers transport-helper: don't update refs in dry-run transport-helper: mismerge fix
2013-12-11smart-http: support shallow fetch/cloneNguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <> Signed-off-by: Junio C Hamano <>
2013-11-12transport-helper: add 'force' to 'export' helpersFelipe Contreras
Otherwise they cannot know when to force the push or not (other than hacks). Tests-by: Richard Hansen <> Documentation-by: Richard Hansen <> Signed-off-by: Felipe Contreras <> Signed-off-by: Junio C Hamano <>
2013-09-12Merge branch 'mm/remote-helpers-doc'Junio C Hamano
* mm/remote-helpers-doc: Documentation/remote-helpers: document common use-case for private ref
2013-09-12Merge branch 'mm/mediawiki-dumb-push-fix'Junio C Hamano
* mm/mediawiki-dumb-push-fix: git-remote-mediawiki: no need to update private ref in non-dumb push git-remote-mediawiki: use no-private-update capability on dumb push transport-helper: add no-private-update capability git-remote-mediawiki: add test and check Makefile targets
2013-09-09Merge branch 'nd/clone-connectivity-shortcut'Junio C Hamano
* nd/clone-connectivity-shortcut: smart http: use the same connectivity check on cloning
2013-09-03transport-helper: add no-private-update capabilityMatthieu Moy
Since 664059fb (transport-helper: update remote helper namespace, 2013-04-17), a 'push' operation on a remote helper updates the private ref by default. This is often a good thing, but it can also be desirable to disable this update to force the next 'pull' to re-import the pushed revisions. Allow remote-helpers to disable the automatic update by introducing a new capability. Signed-off-by: Matthieu Moy <> Signed-off-by: Junio C Hamano <>
2013-08-26Documentation/remote-helpers: document common use-case for private refMatthieu Moy
The current documentation mentions the private ref namespace, but does not really explain why it can be useful. Signed-off-by: Matthieu Moy <> Signed-off-by: Junio C Hamano <>
2013-07-23smart http: use the same connectivity check on cloningNguyễn Thái Ngọc Duy
This is an extension of c6807a4 (clone: open a shortcut for connectivity check - 2013-05-26) to reduce the cost of connectivity check at clone time, this time with smart http protocol. Signed-off-by: Nguyễn Thái Ngọc Duy <> Signed-off-by: Junio C Hamano <>
2013-05-29Merge branch 'fc/transport-helper-error-reporting'Junio C Hamano
Update transport helper to report errors and maintain ref hierarchy used to keep track of remote helper state better. * fc/transport-helper-error-reporting: transport-helper: fix remote helper namespace regression test: remote-helper: add missing and t5801: "VAR=VAL shell_func args" is forbidden transport-helper: update remote helper namespace transport-helper: trivial code shuffle transport-helper: warn when refspec is not used transport-helper: clarify pushing without refspecs transport-helper: update refspec documentation transport-helper: clarify *:* refspec transport-helper: improve push messages transport-helper: mention helper name when it dies transport-helper: report errors properly
2013-04-18transport-helper: clarify pushing without refspecsFelipe Contreras
This has never worked, since it's inception the code simply skips all the refs, essentially telling fast-export to do nothing. Let's at least tell the user what's going on. Signed-off-by: Felipe Contreras <> Signed-off-by: Junio C Hamano <>
2013-04-18transport-helper: update refspec documentationFelipe Contreras
The refspec capability is not only used by 'import', also by 'export', and it's recommended in both. Signed-off-by: Felipe Contreras <> Signed-off-by: Junio C Hamano <>
2013-04-18transport-helper: clarify *:* refspecFelipe Contreras
The *:* refspec doesn't work, and never has, clarify the code and documentation to reflect that. This in effect reverts commit 9e7673e (gitremote-helpers(1): clarify refspec behaviour). Signed-off-by: Felipe Contreras <> Signed-off-by: Junio C Hamano <>
2013-04-15transport-helper: add 'signed-tags' capabilityJohn Keeping
This allows a remote helper using the 'export' protocol to specify that it supports signed tags, changing the handing from 'warn-strip' to 'verbatim'. Signed-off-by: John Keeping <> Signed-off-by: Junio C Hamano <>
2013-04-07gitremote-helpers(1): clarify refspec behaviourJohn Keeping
The documentation says that "If no 'refspec' capability is advertised, there is an implied `refspec *:*`" but this is only the case for the "import" command. Since there is a comment in transport-helper.c indicating that this default is for historical reasons, change the documentation to clarify that a refspec should always be specified. Signed-off-by: John Keeping <> Acked-by: Sverre Rabbelier <> Signed-off-by: Junio C Hamano <>
2013-02-07Merge branch 'jk/remote-helpers-doc'Junio C Hamano
"git help remote-helpers" did not work; 'remote-helpers' is not a subcommand name but a concept, so its documentation should have been in gitremote-helpers, not git-remote-helpers. * jk/remote-helpers-doc: Rename {git- => git}remote-helpers.txt
2013-02-01Rename {git- => git}remote-helpers.txtJohn Keeping
When looking up a topic via "git help <topic>", git-help prepends "git-" to topics that are the names of commands (either builtin or found on the path) and "git" (no hyphen) to any other topic name. "git-remote-helpers" is not the name of a command, so "git help remote-helpers" looks for "gitremote-helpers" and does not find it. Fix this by renaming "git-remote-helpers.txt" to "gitremote-helpers.txt". Signed-off-by: John Keeping <> Signed-off-by: Junio C Hamano <>