path: root/builtin-pack-objects.c
AgeCommit message (Collapse)Author
2008-01-04Don't access line[-1] for a zero-length "line" from fgets.Jim Meyering
A NUL byte at beginning of file, or just after a newline would provoke an invalid buf[-1] access in a few places. * builtin-grep.c (cmd_grep): Don't access buf[-1]. * builtin-pack-objects.c (get_object_list): Likewise. * builtin-rev-list.c (read_revisions_from_stdin): Likewise. * bundle.c (read_bundle_header): Likewise. * server-info.c (read_pack_info_file): Likewise. * transport.c (insert_packed_refs): Likewise. Signed-off-by: Jim Meyering <> Signed-off-by: Junio C Hamano <>
2007-12-18Plug a resource leak in threaded pack-objects code.Johannes Sixt
A mutex and a condition variable is allocated for each thread and torn down when the thread terminates. However, for certain workloads it can happen that some threads are actually not started at all. In this case we would leak the mutex and condition variable. Now we allocate them only for those threads that are actually started. Signed-off-by: Johannes Sixt <> Signed-off-by: Junio C Hamano <>
2007-12-17threaded pack-objects: Use condition variables for thread communication.Johannes Sixt
In the threaded pack-objects code the main thread and the worker threads must mutually signal that they have assigned a new pack of work or have completed their work, respectively. Previously, the code used mutexes that were locked in one thread and unlocked from a different thread, which is bogus (and happens to work on Linux). Here we rectify the implementation by using condition variables: There is one condition variable on which the main thread waits until a thread requests new work; and each worker thread has its own condition variable on which it waits until it is assigned new work or signaled to terminate. As a cleanup, the worker threads are spawned only after the initial work packages have been assigned. Signed-off-by: Johannes Sixt <> Acked-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-12-11pack-objects: more threaded load balancing fix with often changed pathsNicolas Pitre
The code that splits the object list amongst work threads tries to do so on "path" boundaries not to prevent good delta matches. However, in some cases, a few paths may largely dominate the hash distribution and it is not possible to have good load balancing without ignoring those boundaries. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-12-08pack-objects: fix threaded load balancingNicolas Pitre
The current method consists of a master thread serving chunks of objects to work threads when they're done with their previous chunk. The issue is to determine the best chunk size: making it too large creates poor load balancing, while making it too small has a negative effect on pack size because of the increased number of chunk boundaries and poor delta window utilization. This patch implements a completely different approach by initially splitting the work in large chunks uniformly amongst all threads, and whenever a thread is done then it steals half of the remaining work from another thread with the largest amount of unprocessed objects. This has the advantage of greatly reducing the number of chunk boundaries with an almost perfect load balancing. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-12-08pack-objects: reverse the delta search sort listNicolas Pitre
It is currently sorted and then walked backward. Not only this doesn't feel natural for my poor brain, but it would make the next patch less obvious as well. So reverse the sort order, and reverse the list walking direction, which effectively produce the exact same end result as before. Also bring the relevant comment nearer the actual code and adjust it accordingly, with minor additional clarifications. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-12-08pack-objects: fix delta cache size accountingNicolas Pitre
The wrong value was substracted from delta_cache_size when replacing a cached delta, as trg_entry->delta_size was used after the old size had been replaced by the new size. Noticed by Linus. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-11-14Fix rev-list when showing objects involving submodulesLinus Torvalds
The function mark_tree_uninteresting() assumed that the tree entries are blob when they are not trees. This is not so. Since we do not traverse into submodules (yet), the gitlinks should be ignored. In general, we should try to start moving away from using the "S_ISLNK()" like things for internal git state. It was a mistake to just assume the numbers all were same across all systems in the first place. This implementation converts to the "object_type", and then uses a case statement. Noticed by Ilari on IRC. Test script taken from an earlier version by Dscho. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2007-11-04Merge branch 'np/pack'Junio C Hamano
* np/pack: pack-objects: get rid of an ugly cast make the pack index version configurable Conflicts: builtin-pack-objects.c
2007-11-02pack-objects: get rid of an ugly castNicolas Pitre
... when calling write_idx_file(). Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-11-02make the pack index version configurableNicolas Pitre
It is a good idea to use pack index version 2 all the time since it has proper protection against propagation of certain pack corruptions when repacking which is not possible with index version 1, as demonstrated in test t5302. Hence this config option. The default is still pack index version 1. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-10-30add throughput display to git-pushNicolas Pitre
This one triggers only when git-pack-objects is called with --all-progress and --stdout which is the combination used by git-push. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-10-30relax usage of the progress APINicolas Pitre
Since it is now OK to pass a null pointer to display_progress() and stop_progress() resulting in a no-op, then we can simplify the code and remove a bunch of lines by not making those calls conditional all the time. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-10-30make struct progress an opaque typeNicolas Pitre
This allows for better management of progress "object" existence, as well as making the progress display implementation more independent from its callers. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-10-19Change 'Deltifying objects' to 'Compressing objects'Shawn O. Pearce
Recently I was referred to the Grammar Police as the git-pack-objects progress message 'Deltifying %u objects' is considered to be not proper English to at least some small but vocal segment of the English speaking population. Techncially we are applying delta compression to these objects at this stage, so the new term is slightly more acceptable to the Grammar Police but is also just as correct. Signed-off-by: Shawn O. Pearce <>
2007-10-17fix const issues with some functionsNicolas Pitre
Two functions, namely write_idx_file() and open_pack_file(), currently return a const pointer. However that pointer is either a copy of the first argument, or set to a malloc'd buffer when that first argument is null. In the later case it is wrong to qualify that pointer as const since ownership of the buffer is transferred to the caller to dispose of, and obviously the free() function is not meant to be passed const pointers. Making the return pointer not const causes a warning when the first argument is returned since that argument is also marked const. The correct thing to do is therefore to remove the const qualifiers, avoiding the need for ugly casts only to silence some warnings. Signed-off-by: Nicolas Pitre <> Signed-off-by: Shawn O. Pearce <>
2007-10-17pack-objects.c: fix some global variable abuse and memory leaksNicolas Pitre
To keep things well layered, sha1close() now returns the file descriptor when it doesn't close the file. An ugly cast was added to the return of write_idx_file() to avoid a warning. A proper fix will come separately. Signed-off-by: Nicolas Pitre <> Signed-off-by: Shawn O. Pearce <>
2007-10-17pack-objects: no delta possible with only one object in the listNicolas Pitre
... so don't even try in that case, and save another useless line of progress display. Signed-off-by: Nicolas Pitre <> Signed-off-by: Shawn O. Pearce <>
2007-10-17more compact progress displayNicolas Pitre
Each progress can be on a single line instead of two. [sp: Changed "Checking files out" to "Checking out files" at Johannes Sixt's suggestion as it better explains the action that is taking place] Signed-off-by: Nicolas Pitre <> Signed-off-by: Shawn O. Pearce <>
2007-10-03Merge branch 'jc/autogc'Junio C Hamano
* jc/autogc: git-gc --auto: run "repack -A -d -l" as necessary. git-gc --auto: restructure the way "repack" command line is built. git-gc --auto: protect ourselves from accumulated cruft git-gc --auto: add documentation. git-gc --auto: move threshold check to need_to_gc() function. repack -A -d: use --keep-unreachable when repacking pack-objects --keep-unreachable Export matches_pack_name() and fix its return value Invoke "git gc --auto" from commit, merge, am and rebase. Implement git gc --auto
2007-09-17pack-objects --keep-unreachableJunio C Hamano
This new option is meant to be used in conjunction with the options "git repack -a -d" usually invokes the underlying pack-objects with. When this option is given, objects unreachable from the refs in packs named with --unpacked= option are added to the resulting pack, in addition to the reachable objects that are not in packs marked with *.keep files. Signed-off-by: Junio C Hamano <>
2007-09-15builtin-pack-objects.c: avoid bogus gcc warningsJunio C Hamano
These empty statement marcos can solicit bogus "statement with no effect" warnings; squelch them. Signed-off-by: Junio C Hamano <>
2007-09-12threaded delta search: proper locking for cache accountingNicolas Pitre
Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-10threaded delta search: add pack.threads config variableNicolas Pitre
Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-10fix threaded delta search lockingNicolas Pitre
Found by Jeff King. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-10threaded delta search: specify number of threads at run timeNicolas Pitre
This adds a --threads=<n> parameter to 'git pack-objects' with documentation. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-10threaded delta search: better chunck split pointNicolas Pitre
Try to keep object with the same name hash together. Suggested by Martin Koegler. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-10threaded delta search: refine work allocationNicolas Pitre
With this, each thread get repeatedly assigned the next available chunk of objects to process until the whole list is done. The idea is to have reasonably small chunks so that all CPUs remain busy with a minimum number of threads for as long as there is data to process. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-06basic threaded delta searchNicolas Pitre
this is still rough, hence it is disabled by default. You need to compile with "make THREADED_DELTA_SEARCH=1 ..." at the moment. Threading is done on different portions of the object list to be deltified. This is currently done by spliting the list into n parts and then a thread is spawned for each of them. A better method would consist of spliting the list into more smaller parts and have the n threads pick the next part available. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-06rearrange delta search progress reportingNicolas Pitre
This is to help threadification of the delta search code, with a bonus consistency check. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-06localize window memory usage accountingNicolas Pitre
This is to help threadification of delta searching. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-06straighten the list of objects to deltifyNicolas Pitre
Not all objects are subject to deltification, so avoid carrying those along, and provide the real count to progress display. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-09-02Keep last used delta base in the delta windowJunio C Hamano
This is based on Martin Koegler's idea to keep the object that was successfully used as the base of the delta when it is about to fall off the edge of the window. Instead of doing so only for the objects at the edge of the window, this makes the window a lru eviction mechanism. If an entry is used as a base, it is moved to the last of the queue to be evicted. This is a quick-and-dirty implementation, as it keeps the original implementation of the data structure used for the window. This originally was done as an array, not as an array of pointers, because it was meant to be used as a cyclic FIFO buffer and a plain array avoids an extra pointer indirection, while its FIFOness eant that we are not "moving" the entries like this patch does. The runtime from three versions were comparable. It seems to make the resulting chain even shorter, which can only be good. (stock "master") 15782196 bytes chain length = 1: 2972 objects chain length = 2: 2651 objects chain length = 3: 2369 objects chain length = 4: 2121 objects chain length = 5: 1877 objects ... chain length = 46: 490 objects chain length = 47: 515 objects chain length = 48: 527 objects chain length = 49: 570 objects chain length = 50: 408 objects (with your patch) 15745736 bytes (0.23% smaller) chain length = 1: 3137 objects chain length = 2: 2688 objects chain length = 3: 2322 objects chain length = 4: 2146 objects chain length = 5: 1824 objects ... chain length = 46: 503 objects chain length = 47: 509 objects chain length = 48: 536 objects chain length = 49: 588 objects chain length = 50: 357 objects (with this patch) 15612086 bytes (1.08% smaller) chain length = 1: 4831 objects chain length = 2: 3811 objects chain length = 3: 2964 objects chain length = 4: 2352 objects chain length = 5: 1944 objects ... chain length = 46: 327 objects chain length = 47: 353 objects chain length = 48: 304 objects chain length = 49: 298 objects chain length = 50: 135 objects [jc: this is with code simplification follow-up from Nico] Signed-off-by: Junio C Hamano <>
2007-08-31fix same sized delta logicNicolas Pitre
The code favoring shallower deltas when size is equal was triggered only when previous delta was also cached. There should be no relation between cached deltas and same sized deltas. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-08-25pack-objects: check return value from read_sha1_file()Junio C Hamano
Signed-off-by: Junio C Hamano <>
2007-08-19Make thin-pack generation subproject aware.Linus Torvalds
When a thin pack wants to send a tree object at "sub/dir", and the commit that is common between the sender and the receiver that is used as the base object has a subproject at that path, we should not try to use the data at "sub/dir" of the base tree as a tree object. It is not a tree to begin with, and more importantly, the commit object there does not have to even exist. Signed-off-by: Junio C Hamano <>
2007-08-16pack-objects: remove bogus arguments to delta_cacheable()Nicolas Pitre
Not only are they unused, but the order in the function declaration and the actual usage don't match. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-08-15Use xmkstemp() instead of mkstemp()Luiz Fernando N. Capitulino
xmkstemp() performs error checking and prints a standard error message when an error occur. Signed-off-by: Luiz Fernando N. Capitulino <> Signed-off-by: Junio C Hamano <>
2007-07-13Pack-objects: properly initialize the depth valueNicolas Pitre
Commit 5a235b5e was missing this little detail. Otherwise your pack will explode. Problem noted by Brian Downing. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-07-12reduce git-pack-objects memory usage a little moreNicolas Pitre
The delta depth doesn't have to be stored in the global object array structure since it is only used during the deltification pass. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-07-12Add pack-objects window memory usage limitBrian Downing
This adds an option (--window-memory=N) and configuration variable (pack.windowMemory = N) to limit the memory size of the pack-objects delta search window. This works by removing the oldest unpacked objects whenever the total size goes above the limit. It will always leave at least one object, though, so as not to completely eliminate the possibility of computing deltas. This is an extra limit on top of the normal window size (--window=N); the window will not dynamically grow above the fixed number of entries specified to fill the memory limit. With this, repacking a repository with a mix of large and small objects is possible even with a very large window. Cleaner and correct circular buffer handling courtesy of Nicolas Pitre. Signed-off-by: Brian Downing <> Acked-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-07-12Don't try to delta if target is much smaller than sourceBrian Downing
Add a new try_delta heuristic. Don't bother trying to make a delta if the target object size is much smaller (currently 1/32) than the source, as it's very likely not going to get a match. Even if it does, you will have to read at least 32x the size of the new file to reassemble it, which isn't such a good deal. This leads to a considerable performance improvement when deltifying a mix of small and large files with a very large window, because you don't have to wait for the large files to percolate out of the window before things start going fast again. Signed-off-by: Brian Downing <> Acked-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-07-12apply delta depth bias to already deltified objectsNicolas Pitre
We already apply a bias on the initial delta attempt with max_size being a function of the base object depth. This has the effect of favoring shallower deltas even if deeper deltas could be smaller, and therefore creating a wider delta tree (see commits 4e8da195 and c3b06a69). This principle should also be applied to all delta attempts for the same object and not only the first attempt. With this the criteria for the best delta is not only its size but also its depth, so that a shallower delta might be selected even if it is larger than a deeper one. Even if some deltas get larger, they allow for wider delta trees making the depth limit less quickly reached and therefore better deltas can be subsequently found, keeping the resulting pack size even smaller. Runtime access to the pack should also benefit from shallower deltas. Testing on different repositories showed slighter faster repacks, smaller resulting packs, and a much nicer curve for delta depth distribution with no more peak at the maximum depth level. Improvements are even more significant with smaller depth limits. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-07-09pack-objects: Prefer shallower deltas if the size is equalBrian Downing
Change "try_delta" so that if it finds a delta that has the same size but shallower depth than the existing delta, it will prefer the shallower one. This makes certain delta trees vastly less deep. Signed-off-by: Brian Downing <> Signed-off-by: Junio C Hamano <>
2007-06-07War on whitespaceJunio C Hamano
This uses "git-apply --whitespace=strip" to fix whitespace errors that have crept in to our source files over time. There are a few files that need to have trailing whitespaces (most notably, test vectors). The results still passes the test, and build result in Documentation/ area is unchanged. Signed-off-by: Junio C Hamano <>
2007-06-02Unify write_index_file functionsGeert Bosch
This patch unifies the write_index_file functions in builtin-pack-objects.c and index-pack.c. As the name "index" is overloaded in git, move in the direction of using "idx" and "pack idx" when refering to the pack index. There should be no change in functionality. Signed-off-by: Geert Bosch <> Acked-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-05-31fix repack with --max-pack-sizeNicolas Pitre
Two issues here: 1) git-repack -a --max-pack-size=10 on the GIT repo dies pretty quick. There is a lot of confusion about deltas that were suposed to be reused from another pack but that get stored undeltified due to pack limit and object size doesn't match entry->size anymore. This test is not really worth the complexity for determining when it is valid so get rid of it. 2) If pack limit is reached, the object buffer is freed, including when it comes from a cached delta data. In practice the object will be stored in a subsequent pack undeltified, but let's make sure no pointer to freed data subsists by clearing entry->delta_data. I also reorganized that code a bit to make it more readable. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-05-29builtin-pack-object: cache small deltasMartin Koegler
Signed-off-by: Martin Koegler <> Signed-off-by: Junio C Hamano <>
2007-05-29git-pack-objects: cache small deltas between big objectsMartin Koegler
Creating deltas between big blobs is a CPU and memory intensive task. In the writing phase, all (not reused) deltas are redone. This patch adds support for caching deltas from the deltifing phase, so that that the writing phase is faster. The caching is limited to small deltas to avoid increasing memory usage very much. The implemented limit is (memory needed to create the delta)/1024. Signed-off-by: Martin Koegler <> Signed-off-by: Junio C Hamano <>
2007-05-29builtin-pack-objects: don't fail, if delta is not possibleMartin Koegler
If builtin-pack-objects runs out of memory while finding the best deltas, it bails out with an error. If the delta index creation fails (because there is not enough memory), we can downgrade the error message to a warning and continue with the next object. Signed-off-by: Martin Koegler <> Signed-off-by: Junio C Hamano <>