path: root/fetch.c
AgeCommit message (Collapse)Author
2007-09-19Modularize commit-walkerDaniel Barkalow
This turns the extern functions to be provided by the backend into a struct of pointers, renames the functions to be more namespace-friendly, and updates http-fetch to this interface. It removes the unused include from http-push.c. It makes git-http-fetch a builtin (with the implementation a separate file, accessible directly). Signed-off-by: Daniel Barkalow <> Signed-off-by: Junio C Hamano <>
2007-06-27Merge branch 'maint'Junio C Hamano
* maint: config: Change output of --get-regexp for valueless keys config: Complete documentation of --get-regexp cleanup merge-base test script Fix zero-object version-2 packs Ignore submodule commits when fetching over dumb protocols
2007-06-27Ignore submodule commits when fetching over dumb protocolsSven Verdoolaege
Without this patch, the code would look for the submodule commits in the superproject and (needlessly) fail when it couldn't find them. Signed-off-by: Sven Verdoolaege <> Acked-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2007-06-07War on whitespaceJunio C Hamano
This uses "git-apply --whitespace=strip" to fix whitespace errors that have crept in to our source files over time. There are a few files that need to have trailing whitespaces (most notably, test vectors). The results still passes the test, and build result in Documentation/ area is unchanged. Signed-off-by: Junio C Hamano <>
2007-03-21Initialize tree descriptors with a helper function rather than by hand.Linus Torvalds
This removes slightly more lines than it adds, but the real reason for doing this is that future optimizations will require more setup of the tree descriptor, and so we want to do it in one place. Also renamed the "desc.buf" field to "desc.buffer" just to trigger compiler errors for old-style manual initializations, making sure I didn't miss anything. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-12-20simplify inclusion of system header files.Junio C Hamano
This is a mechanical clean-up of the way *.c files include system header files. (1) sources under compat/, platform sha-1 implementations, and xdelta code are exempt from the following rules; (2) the first #include must be "git-compat-util.h" or one of our own header file that includes it first (e.g. config.h, builtin.h, pkt-line.h); (3) system headers that are included in "git-compat-util.h" need not be included in individual C source files. (4) "git-compat-util.h" does not have to include subsystem specific header files (e.g. expat.h). Signed-off-by: Junio C Hamano <>
2006-12-13Clarify fetch error for missing objects.Alex Riesen
Otherwise there're such things like: Cannot obtain needed none 9a6e87b60dbd2305c95cecce7d9d60f849a0658d while processing commit 0000000000000000000000000000000000000000. which while looks weird. What is the none needed for? Signed-off-by: Junio C Hamano <>
2006-09-27Clean-up lock-ref implementationJunio C Hamano
This drops "mustexist" parameter lock_ref_sha1() and lock_any_ref_forupdate() functions take. Signed-off-by: Junio C Hamano <>
2006-09-21Tell between packed, unpacked and symbolic refs.Junio C Hamano
This adds a "int *flag" parameter to resolve_ref() and makes for_each_ref() family to call callback function with an extra "int flag" parameter. They are used to give two bits of information (REF_ISSYMREF and REF_ISPACKED) about the ref. Signed-off-by: Junio C Hamano <>
2006-09-21Add callback data to for_each_ref() family.Junio C Hamano
This is a long overdue fix to the API for for_each_ref() family of functions. It allows the callers to specify a callback data pointer, so that the caller does not have to use static variables to communicate with the callback funciton. The updated for_each_ref() family takes a function of type int (*fn)(const char *, const unsigned char *, void *) and a void pointer as parameters, and calls the function with the name of the ref and its SHA-1 with the caller-supplied void pointer as parameters. The commit updates two callers, builtin-name-rev.c and builtin-pack-refs.c as an example. Signed-off-by: Junio C Hamano <>
2006-09-02Replace uses of strdup with xstrdup.Shawn Pearce
Like xmalloc and xrealloc xstrdup dies with a useful message if the native strdup() implementation returns NULL rather than a valid pointer. I just tried to use xstrdup in new code and found it to be missing. However I expected it to be present as xmalloc and xrealloc are already commonly used throughout the code. [jc: removed the part that deals with last_XXX, which I am finding more and more dubious these days.] Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-08-28free(NULL) is perfectly valid.Junio C Hamano
Jonas noticed some places say "if (X) free(X)" which is totally unnecessary. Signed-off-by: Junio C Hamano <>
2006-08-23Convert memcpy(a,b,20) to hashcpy(a,b).Shawn Pearce
This abstracts away the size of the hash values when copying them from memory location to memory location, much as the introduction of hashcmp abstracted away hash value comparsion. A few call sites were using char* rather than unsigned char* so I added the cast rather than open hashcpy to be void*. This is a reasonable tradeoff as most call sites already use unsigned char* and the existing hashcmp is also declared to be unsigned char*. [jc: Splitted the patch to "master" part, to be followed by a patch for merge-recursive.c which is not in "master" yet. Fixed the cast in the latter hunk to combine-diff.c which was wrong in the original. Also converted ones left-over in combine-diff.c, diff-lib.c and upload-pack.c ] Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-07-29Fix http-fetchJohannes Schindelin
With the latest changes in fetch.c, http-fetch crashed accessing write_ref[i], where write_ref was NULL. Signed-off-by: Johannes Schindelin <> Signed-off-by: Junio C Hamano <>
2006-07-28Teach git-local-fetch the --stdin switchPetr Baudis
This makes it possible to fetch many commits (refs) at once, greatly speeding up cg-clone. Signed-off-by: Petr Baudis <> Signed-off-by: Junio C Hamano <>
2006-07-28Make pull() support fetching multiple targets at oncePetr Baudis
pull() now takes an array of arguments instead of just one of each kind. Currently, no users use the new capability, but that'll change. Signed-off-by: Petr Baudis <> Signed-off-by: Junio C Hamano <>
2006-07-28Make pull() take some implicit data as explicit argumentsPetr Baudis
Currently it's a bit weird that pull() takes a single argument describing the commit but takes the write_ref from a global variable. This makes it take that as a parameter as well, which might be nicer for the libification in the future, but especially it will make for nicer code when we implement pull()ing multiple commits at once. Signed-off-by: Petr Baudis <> Signed-off-by: Junio C Hamano <>
2006-07-13Remove TYPE_* constant macros and use object_type enums consistently.Linus Torvalds
This updates the type-enumeration constants introduced to reduce the memory footprint of "struct object" to match the type bits already used in the packfile format, by removing the former (i.e. TYPE_* constant macros) and using the latter (i.e. enum object_type) throughout the code for consistency. Eventually we can stop passing around the "type strings" entirely, and this will help - no confusion about two different integer enumeration. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-06-18Shrink "struct object" a bitLinus Torvalds
This shrinks "struct object" by a small amount, by getting rid of the "struct type *" pointer and replacing it with a 3-bit bitfield instead. In addition, we merge the bitfields and the "flags" field, which incidentally should also remove a useless 4-byte padding from the object when in 64-bit mode. Now, our "struct object" is still too damn large, but it's now less obviously bloated, and of the remaining fields, only the "util" (which is not used by most things) is clearly something that should be eventually discarded. This shrinks the "git-rev-list --all" memory use by about 2.5% on the kernel archive (and, perhaps more importantly, on the larger mozilla archive). That may not sound like much, but I suspect it's more on a 64-bit platform. There are other remaining inefficiencies (the parent lists, for example, probably have horrible malloc overhead), but this was pretty obvious. Most of the patch is just changing the comparison of the "type" pointer from one of the constant string pointers to the appropriate new TYPE_xxx small integer constant. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-06-06ref-log: style fixes.Junio C Hamano
A few style fixes to get the code in line with the rest. - asterisk to make a type a pointer to something goes in front of the variable, not at the end of the base type. E.g. a pointer to an integer is "int *ip", not "int* ip". - open parenthesis for function parameter list, unlike syntactic constructs, comes immediately after the function name. E.g. "if (foo) bar();" not "if(foo) bar ();". - "else" does not come on the same line as the closing brace of corresponding "if". The style is mostly a matter of personal taste, and people may disagree, but consistency is important. Signed-off-by: Junio C Hamano <>
2006-06-04Merge branch 'lt/tree-2'Junio C Hamano
* lt/tree-2: fetch.c: do not call process_tree() from process_tree(). tree_entry(): new tree-walking helper function adjust to the rebased series by Linus. Remove "tree->entries" tree-entry list from tree parser Switch "read_tree_recursive()" over to tree-walk functionality Make "tree_entry" have a SHA1 instead of a union of object pointers Add raw tree buffer info to "struct tree" Remove last vestiges of generic tree_entry_list Convert fetch.c: process_tree() to raw tree walker Convert "mark_tree_uninteresting()" to raw tree walker Remove unused "zeropad" entry from tree_list_entry fsck-objects: avoid unnecessary tree_entry_list usage Remove "tree->entries" tree-entry list from tree parser builtin-read-tree.c: avoid tree_entry_list in prime_cache_tree_rec() Switch "read_tree_recursive()" over to tree-walk functionality Make "tree_entry" have a SHA1 instead of a union of object pointers Make "struct tree" contain the pointer to the tree buffer
2006-06-02fetch.c: do not call process_tree() from process_tree().Junio C Hamano
This function reads a freshly fetched tree object, and schedules the objects pointed by it for further fetching, so doing lookup_tree() and process_tree() recursively from there does not make much sense. We need to use process() on it to make sure we fetch it first, and leave the recursive processing to later stages. Signed-off-by: Junio C Hamano <>
2006-05-31fetch.c: do not pass uninitialized lock to unlock_ref().Junio C Hamano
Signed-off-by: Junio C Hamano <>
2006-05-31tree_entry(): new tree-walking helper functionLinus Torvalds
This adds a "tree_entry()" function that combines the common operation of doing a "tree_entry_extract()" + "update_tree_entry()". It also has a simplified calling convention, designed for simple loops that traverse over a whole tree: the arguments are pointers to the tree descriptor and a name_entry structure to fill in, and it returns a boolean "true" if there was an entry left to be gotten in the tree. This allows tree traversal with struct tree_desc desc; struct name_entry entry; desc.buf = tree->buffer; desc.size = tree->size; while (tree_entry(&desc, &entry) { ... use "entry.{path, sha1, mode, pathlen}" ... } which is not only shorter than writing it out in full, it's hopefully less error prone too. [ It's actually a tad faster too - we don't need to recalculate the entry pathlength in both extract and update, but need to do it only once. Also, some callers can avoid doing a "strlen()" on the result, since it's returned as part of the name_entry structure. However, by now we're talking just 1% speedup on "git-rev-list --objects --all", and we're definitely at the point where tree walking is no longer the issue any more. ] NOTE! Not everybody wants to use this new helper function, since some of the tree walkers very much on purpose do the descriptor update separately from the entry extraction. So the "extract + update" sequence still remains as the core sequence, this is just a simplified interface. We should probably add a silly two-line inline helper function for initializing the descriptor from the "struct tree" too, just to cut down on the noise from that common "desc" initializer. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-05-30Convert fetch.c: process_tree() to raw tree walkerLinus Torvalds
This leaves only the horrid code in builtin-read-tree.c using the old interface. Some day I will gather the strength to tackle that one too. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-05-30Remove "tree->entries" tree-entry list from tree parserLinus Torvalds
Instead, just use the tree buffer directly, and use the tree-walk infrastructure to walk the buffers instead of the tree-entry list. The tree-entry list is inefficient, and generates tons of small allocations for no good reason. The tree-walk infrastructure is generally no harder to use than following a linked list, and allows us to do most tree parsing in-place. Some programs still use the old tree-entry lists, and are a bit painful to convert without major surgery. For them we have a helper function that creates a temporary tree-entry list on demand. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-05-24Merge branch 'master' into sp/reflogJunio C Hamano
* master: (90 commits) fetch.c: remove an unused variable and dead code. Clean up sha1 file writing Builtin git-cat-file builtin format-patch: squelch content-type for 7-bit ASCII CMIT_FMT_EMAIL: Q-encode Subject: and display-name part of From: fields. add more informative error messages to git-mktag remove the artificial restriction tagsize < 8kb git-rebase: use canonical A..B syntax to format-patch git-format-patch: now built-in. fmt-patch: Support --attach fmt-patch: understand old <his> notation Teach fmt-patch about --keep-subject Teach fmt-patch about --numbered fmt-patch: implement -o <dir> fmt-patch: output file names to stdout Teach fmt-patch to write individual files. built-in tar-tree and remote tar-tree Builtin git-diff-files, git-diff-index, git-diff-stages, and git-diff-tree. Builtin git-show-branch. Builtin git-apply. ...
2006-05-24fetch.c: remove an unused variable and dead code.Junio C Hamano
Funnily enough, this variable was never assigned ever since it was introduced, and has been protecting some code that has never been executed. Signed-off-by: Junio C Hamano <>
2006-05-19Log ref updates made by fetch.Shawn Pearce
If a ref is changed by http-fetch, local-fetch or ssh-fetch record the change and the remote URL/name in the log for the ref. This requires loading the config file to check logAllRefUpdates. Also fixed a bug in the ref lock generation; the log file name was not being produced right due to a bad prefix length. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-05-18Improve abstraction of ref lock/write.Shawn Pearce
Created 'struct ref_lock' to contain the data necessary to perform a ref update. This change improves writing a ref as the file names are generated only once (rather than twice) and supports following symrefs (up to the maximum depth). Further the ref_lock structure provides room to extend the update API with ref logging. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2005-10-11[PATCH] Don't fetch objects that exist in the local repositoryNick Hengeveld
Be sure not to fetch objects that already exist in the local repository. The main process loop no longer performs this check, http-fetch now checks prior to starting a new request queue entry and when fetch_object() is called, and local-fetch now checks when fetch_object() is called. As discussed in this thread: Signed-off-by: Nick Hengeveld <>
2005-09-27[PATCH] Implement --recover for git-*-fetchDaniel Barkalow
With the --recover option, we verify that we have absolutely everything reachable from the target, not assuming that things reachable from refs will be complete. Signed-off-by: Daniel Barkalow <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Plug memory leak in process_tree()Sergey Vlasov
When freeing a tree entry, must free its name too. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Do not build object ref listsSergey Vlasov
The fetch code does not need object ref lists; by disabling them we can save some time and memory. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Remove call to parse_object() from process()Sergey Vlasov
The call to parse_object() in process() is not actually needed - if the object type is unknown, parse_object() will be called by loop(); if the type is known, the object will be parsed by the appropriate process_*() function. After this change blobs which exist locally are no longer parsed, which gives about 2x CPU usage improvement; the downside is that there will be no warnings for existing corrupted blobs, but detecting such corruption is the job of git-fsck-objects, not the fetch programs. Newly fetched objects are still checked for corruption in http-fetch.c and ssh-fetch.c (local-fetch.c does not seem to do it, but the removed parse_object() call would not be reached for new objects anyway). Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Clean up object flag definitionsSergey Vlasov
Remove holes left after deleting flags, and use shifts to emphasize that flags are single bits. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Remove redundant test of TO_SCAN in process()Sergey Vlasov
If the SEEN flag was not set, the TO_SCAN flag cannot be set, therefore testing it is pointless. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Remove some duplicated code in process()Sergey Vlasov
It does not matter if we call prefetch() or set the TO_SCAN flag before or after adding the object to process_queue. However, doing it before object_list_insert() allows us to kill 3 lines of duplicated code. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Remove redundant TO_FETCH flagSergey Vlasov
The TO_FETCH flag also became redundant after adding the SEEN flag - it was set and checked in process() to prevent adding the same object to process_queue multiple times, but now SEEN guards against this. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Remove redundant SCANNED flagSergey Vlasov
After adding the SEEN flag, the SCANNED flag became obviously redundant - each object can get into process_queue through process() only once, and therefore multiple calls to process_object() for the same object are not possible. Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Make process() look at each object only onceSergey Vlasov
The process() function is very often called multiple times for the same object (because lots of trees refer to the same blobs), but did not have a fast check for this, therefore a lot of useless calls to has_sha1_file() and parse_object() were made before discovering that nothing needs to be done. This patch adds the SEEN flag which is used in process() to make it look at each object only once. When testing git-local-fetch on the repository of GIT, this gives a 14x improvement in CPU usage (mainly because the redundant calls to parse_object() are now avoided - parse_object() always unpacks and parses the object data, even if it was already parsed before). Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-23[PATCH] fetch.c: Remove useless lookup_object_type() call in process()Sergey Vlasov
In all places where process() is called except the one in pull() (which is executed only once) the pointer to the object is already available, so pass it as the argument to process() instead of sha1 and avoid an unneeded call to lookup_object_type(). Signed-off-by: Sergey Vlasov <> Signed-off-by: Junio C Hamano <>
2005-09-18fetch() assumes we do not have the object.Junio C Hamano
Bugfix for the previous one. Signed-off-by: Junio C Hamano <>
2005-09-18Improve the safety check used in fetch.cJunio C Hamano
The recent safety check to trust only the commits we have made things impossibly slow and turn out to waste a lot of memory. This commit fixes it with the following improvements: - mark already scanned objects and avoid rescanning the same object again; - free the tree entries when we have scanned the tree entries; this is the same as b0d8923ec01fd91b75ab079034f89ced91500157 which reduced memory usage by rev-list; - plug memory leak from the object_list dequeuing code; - use the process_queue not just for fetching but for scanning, to make things tail recursive to avoid deep recursion; the deep recursion was especially prominent when we cloned a big pack. - avoid has_sha1_file() call when we already know we do not have that object. Signed-off-by: Junio C Hamano <>
2005-09-16[PATCH] fetch.c: cleanupsJunio C Hamano
Clean-ups suggested by Sergey Vlasov and acked by Daniel Barkalow. Signed-off-by: Junio C Hamano <>
2005-09-15Avoid wasting memory while keeping track of what we have during fetch.Junio C Hamano
Signed-off-by: Junio C Hamano <>
2005-09-15[PATCH] Fix fetch completeness assumptionsDaniel Barkalow
Don't assume that any commit we have is complete; assume that any ref we have is complete. Signed-off-by: Daniel Barkalow <> Signed-off-by: Junio C Hamano <>
2005-09-08Big tool rename.Junio C Hamano
As promised, this is the "big tool rename" patch. The primary differences since 0.99.6 are: (1) git-*-script are no more. The commands installed do not have any such suffix so users do not have to remember if something is implemented as a shell script or not. (2) Many command names with 'cache' in them are renamed with 'index' if that is what they mean. There are backward compatibility symblic links so that you and Porcelains can keep using the old names, but the backward compatibility support is expected to be removed in the near future. Signed-off-by: Junio C Hamano <>