path: root/pack-check.c
AgeCommit message (Collapse)Author
2010-04-20Extract verify_pack_index for reuse from verify_packShawn O. Pearce
The dumb HTTP transport should verify an index is completely valid before trying to use it. That requires checking the header/footer but also checking the complete content SHA-1. All of this logic is already in the front half of verify_pack, so pull it out into a new function that can be reused. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2009-06-06Don't expect verify_pack() callers to set pack_sizeMike Hommey
Since use_pack() will end up populating pack_size if it is not already set, we can just adapt the code in verify_packfile() such that it doesn't require pack_size to be set beforehand. This allows callers not to have to set pack_size themselves, and we can thus revert changes from 1c23d794 (Don't die in git-http-fetch when fetching packs). Signed-off-by: Mike Hommey <> Signed-off-by: Tay Ray Chuan <> Signed-off-by: Junio C Hamano <>
2008-10-03fix openssl headers conflicting with custom SHA1 implementationsNicolas Pitre
On ARM I have the following compilation errors: CC fast-import.o In file included from cache.h:8, from builtin.h:6, from fast-import.c:142: arm/sha1.h:14: error: conflicting types for 'SHA_CTX' /usr/include/openssl/sha.h:105: error: previous declaration of 'SHA_CTX' was here arm/sha1.h:16: error: conflicting types for 'SHA1_Init' /usr/include/openssl/sha.h:115: error: previous declaration of 'SHA1_Init' was here arm/sha1.h:17: error: conflicting types for 'SHA1_Update' /usr/include/openssl/sha.h:116: error: previous declaration of 'SHA1_Update' was here arm/sha1.h:18: error: conflicting types for 'SHA1_Final' /usr/include/openssl/sha.h:117: error: previous declaration of 'SHA1_Final' was here make: *** [fast-import.o] Error 1 This is because openssl header files are always included in git-compat-util.h since commit 684ec6c63c whenever NO_OPENSSL is not set, which somehow brings in <openssl/sha1.h> clashing with the custom ARM version. Compilation of git is probably broken on PPC too for the same reason. Turns out that the only file requiring openssl/ssl.h and openssl/err.h is imap-send.c. But only moving those problematic includes there doesn't solve the issue as it also includes cache.h which brings in the conflicting local SHA1 header file. As suggested by Jeff King, the best solution is to rename our references to SHA1 functions and structure to something git specific, and define those according to the implementation used. Signed-off-by: Nicolas Pitre <> Signed-off-by: Shawn O. Pearce <>
2008-06-25verify-pack: check packed object CRC when using index version 2Nicolas Pitre
To do so, check_pack_crc() moved from builtin-pack-objects.c to pack-check.c where it is more logical to share. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-06-25move show_pack_info() where it belongsNicolas Pitre
This is called when verify_pack() has its verbose argument set, and verbose in this context makes sense only for the actual 'git verify-pack' command. Therefore let's move show_pack_info() to builtin-verify-pack.c instead and remove useless verbose argument from verify_pack(). Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-06-25optimize verify-pack a bitNicolas Pitre
Using find_pack_entry_one() to get object offsets is rather suboptimal when nth_packed_object_offset() can be used directly. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-06-24call init_pack_revindex() lazilyNicolas Pitre
This makes life much easier for next patch, as well as being more efficient when the revindex is actually not used. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-06-02make verify-pack a bit more useful with bad packsNicolas Pitre
When a pack gets corrupted, its SHA1 checksum will fail. However, this is more useful to let the test go on in order to find the actual problem location than only complain about the SHA1 mismatch and bail out. Also, it is more useful to compare the stored pack SHA1 with the one in the index file instead of the computed SHA1 since the computed SHA1 from a corrupted pack won't match the one stored in the index either. Finally a few code and message cleanups were thrown in as a bonus. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-03-01add storage size output to 'git verify-pack -v'Nicolas Pitre
This can possibly break external scripts that depend on the previous output, but those script can't possibly be critical to Git usage, and fixing them should be trivial. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2008-03-01fix unimplemented packed_object_info_detail() featuresNicolas Pitre
Since commit eb32d236df0c16b936b04f0c5402addb61cdb311, there was a TODO comment in packed_object_info_detail() about the SHA1 of base object to OBJ_OFS_DELTA objects. So here it is at last. While at it, providing the actual storage size information as well is now trivial. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-06-06pack-check: Sort entries by pack offset before unpacking them.Alexandre Julliard
Because of the way objects are sorted in a pack, unpacking them in disk order is much more efficient than random access. Tests on the Wine repository show a gain in pack validation time of about 35%. Signed-off-by: Alexandre Julliard <> Signed-off-by: Junio C Hamano <>
2007-05-27Lazily open pack index files on demandShawn O. Pearce
In some repository configurations the user may have many packfiles, but all of the recent commits/trees/tags/blobs are likely to be in the most recent packfile (the one with the newest mtime). It is therefore common to be able to complete an entire operation by accessing only one packfile, even if there are 25 packfiles available to the repository. Rather than opening and mmaping the corresponding .idx file for every pack found, we now only open and map the .idx when we suspect there might be an object of interest in there. Of course we cannot known in advance which packfile contains an object, so we still need to scan the entire packed_git list to locate anything. But odds are users want to access objects in the most recently created packfiles first, and that may be all they ever need for the current operation. Junio observed in b867092f that placing recent packfiles before older ones can slightly improve access times for recent objects, without degrading it for historical object access. This change improves upon Junio's observations by trying even harder to avoid the .idx files that we won't need. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2007-05-26fixes to output of git-verify-pack -vNicolas Pitre
Now that the default delta depth is 50, it is a good idea to also bump MAX_CHAIN to 50. While at it, make the display a bit prettier by making the MAX_CHAIN limit inclusive, and display the number of deltas that are above that limit at the end instead of the beginning. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-04-10get rid of num_packed_objects()Nicolas Pitre
The coming index format change doesn't allow for the number of objects to be determined from the size of the index file directly. Instead, Let's initialize a field in the packed_git structure with the object count when the index is validated since the count is always known at that point. While at it let's reorder some struct packed_git fields to avoid padding due to needed 64-bit alignment for some of them. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-04-05clean up and optimize nth_packed_object_sha1() usageNicolas Pitre
Let's avoid the open coded pack index reference in pack-object and use nth_packed_object_sha1() instead. This will help encapsulating index format differences in one place. And while at it there is no reason to copy SHA1's over and over while a direct pointer to it in the index will do just fine. Signed-off-by: Nicolas Pitre <> Acked-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2007-03-17[PATCH] clean up pack index handling a bitNicolas Pitre
Especially with the new index format to come, it is more appropriate to encapsulate more into check_packed_git_idx() and assume less of the index format in struct packed_git. To that effect, the index_base is renamed to index_data with void * type so it is not used directly but other pointers initialized with it. This allows for a couple pointer cast removal, as well as providing a better generic name to grep for when adding support for new index versions or formats. And index_data is declared const too while at it. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2007-03-07Use off_t when we really mean a file offset.Shawn O. Pearce
Not all platforms have declared 'unsigned long' to be a 64 bit value, but we want to support a 64 bit packfile (or close enough anyway) in the near future as some projects are getting large enough that their packed size exceeds 4 GiB. By using off_t, the POSIX type that is declared to mean an offset within a file, we support whatever maximum file size the underlying operating system will handle. For most modern systems this is up around 2^60 or higher. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2007-03-07Use uint32_t for all packed object counts.Shawn O. Pearce
As we permit up to 2^32-1 objects in a single packfile we cannot use a signed int to represent the object offset within a packfile, after 2^31-1 objects we will start seeing negative indexes and error out or compute bad addresses within the mmap'd index. This is a minor cleanup that does not introduce any significant logic changes. It is roach free. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2007-02-27convert object type handling from a string to a numberNicolas Pitre
We currently have two parallel notation for dealing with object types in the code: a string and a numerical value. One of them is obviously redundent, and the most used one requires more stack space and a bunch of strcmp() all over the place. This is an initial step for the removal of the version using a char array found in object reading code paths. The patch is unfortunately large but there is no sane way to split it in smaller parts without breaking the system. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2006-12-29Loop over pack_windows when inflating/accessing data.Shawn O. Pearce
When multiple mmaps start getting used for all pack file access it is not possible to get all data associated with a specific object in one contiguous memory region. This limitation prevents simply passing a single address and length to SHA1_Update or to inflate. Instead we need to loop until we have processed all data of interest. As we loop over the data we are always interested in reusing the same window 'cursor', as the prior window will no longer be of any use to us. This allows the use_pack() call to automatically decrement the use count of the prior window before setting up access for us to the next window. Within each loop we need to make use of the available length output parameter of use_pack() to tell us how many bytes are available in the current memory region, as we cannot tell otherwise. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-12-29Replace use_packed_git with window cursors.Shawn O. Pearce
Part of the implementation concept of the sliding mmap window for pack access is to permit multiple windows per pack to be mapped independently. Since the inuse_cnt is associated with the mmap and not with the file, this value is in struct pack_window and needs to be incremented/decremented for each pack_window accessed by any code. To faciliate that implementation we need to replace all uses of use_packed_git() and unuse_packed_git() with a different API that follows struct pack_window objects rather than struct packed_git. The way this works is when we need to start accessing a pack for the first time we should setup a new window 'cursor' by declaring a local and setting it to NULL: struct pack_windows *w_curs = NULL; To obtain the memory region which contains a specific section of the pack file we invoke use_pack(), supplying the address of our current window cursor: unsigned int len; unsigned char *addr = use_pack(p, &w_curs, offset, &len); the returned address `addr` will be the first byte at `offset` within the pack file. The optional variable len will also be updated with the number of bytes remaining following the address. Multiple calls to use_pack() with the same window cursor will update the window cursor, moving it from one window to another when necessary. In this way each window cursor variable maintains only one struct pack_window inuse at a time. Finally before exiting the scope which originally declared the window cursor we must invoke unuse_pack() to unuse the current window (which may be different from the one that was first obtained from use_pack): unuse_pack(&w_curs); This implementation is still not complete with regards to multiple windows, as only one window per pack file is supported right now. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-12-29Refactor packed_git to prepare for sliding mmap windows.Shawn O. Pearce
The idea behind the sliding mmap window pack reader implementation is to have multiple mmap regions active against the same pack file, thereby allowing the process to mmap in only the active/hot sections of the pack and reduce overall virtual address space usage. To implement this we need to refactor the mmap related data (pack_base, pack_use_cnt) out of struct packed_git and move them into a new struct pack_window. We are refactoring the code to support a single struct pack_window per packfile, thereby emulating the prior behavior of mmap'ing the entire pack file. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-12-29Replace unpack_entry_gently with unpack_entry.Shawn O. Pearce
The unpack_entry_gently function currently has only two callers: the delta base resolution in sha1_file.c and the main loop of pack-check.c. Both of these must change to using unpack_entry directly when we implement sliding window mmap logic, so I'm doing it earlier to help break down the change set. This may cause a slight performance decrease for delta base resolution as well as for pack-check.c's verify_packfile(), as the pack use counter will be incremented and decremented for every object that is unpacked. Signed-off-by: Shawn O. Pearce <> Signed-off-by: Junio C Hamano <>
2006-09-23many cleanups to sha1_file.cNicolas Pitre
Those cleanups are mainly to set the table for the support of deltas with base objects referenced by offsets instead of sha1. This means that many pack lookup functions are converted to take a pack/offset tuple instead of a sha1. This eliminates many struct pack_entry usages since this structure carried redundent information in many cases, and it increased stack footprint needlessly for a couple recursively called functions that used to declare a local copy of it for every recursion loop. In the process, packed_object_info_detail() has been reorganized as well so to look much saner and more amenable to deltas with offset support. Finally the appropriate adjustments have been made to functions that depend on the above changes. But there is no functionality changes yet simply some code refactoring at this point. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2006-08-17Do not use memcmp(sha1_1, sha1_2, 20) with hardcoded length.David Rientjes
Introduces global inline: hashcmp(const unsigned char *sha1, const unsigned char *sha2) Uses memcmp for comparison and returns the result based on the length of the hash name (a future runtime decision). Acked-by: Alex Riesen <> Signed-off-by: David Rientjes <> Signed-off-by: Junio C Hamano <>
2006-06-20Remove all void-pointer arithmetic.Florian Forster
ANSI C99 doesn't allow void-pointer arithmetic. This patch fixes this in various ways. Usually the strategy that required the least changes was used. Signed-off-by: Florian Forster <> Signed-off-by: Junio C Hamano <>
2006-04-27verify-pack: check integrity in a saner order.Junio C Hamano
Check internal integrity to report corrupt pack or idx, and then check cross-integrity between idx and pack. Signed-off-by: Linus Torvalds <> Signed-off-by: Junio C Hamano <>
2006-03-05verify-pack -v: show delta-chain histogram.Junio C Hamano
Signed-off-by: Junio C Hamano <>
2006-02-15packed objects: minor cleanupJunio C Hamano
The delta depth is unsigned. Signed-off-by: Junio C Hamano <>
2006-02-10remove delta-against-self bitNicolas Pitre
After experimenting with code to add the ability to encode a delta against part of the deltified file, it turns out that resulting packs are _bigger_ than when this ability is not used. The raw delta output might be smaller, but it doesn't compress as well using gzip with a negative net saving on average. Said bit would in fact be more useful to allow for encoding the copying of chunks larger than 64KB providing more savings with large files. This will correspond to packs version 3. While the current code still produces packs version 2, it is made future proof so pack versions 2 and 3 are accepted. Any pack version 2 are compatible with version 3 since the redefined bit was never used before. When enough time has passed, code to use that bit to produce version 3 packs could be added. Signed-off-by: Nicolas Pitre <> Signed-off-by: Junio C Hamano <>
2005-08-10[PATCH] -Werror fixesTimo Sirainen
GCC's format __attribute__ is good for checking errors, especially with -Wformat=2 parameter. This fixes most of the reported problems against 2005-08-09 snapshot.
2005-07-07[PATCH] Typofix an error message in pack-check.cJunio C Hamano
The current error message does not make any sense. Signed-off-by: Junio C Hamano <> Signed-off-by: Linus Torvalds <>
2005-07-01[PATCH] Show more details of packfile with verify-pack -v.Junio C Hamano
This implements show_pack_info() function used in verify-pack command when -v flag is used to obtain something like unpack-objects used to give when it was first written. It shows the following for each non-deltified object found in the pack: SHA1 type size offset For deltified objects, it shows this instead: SHA1 type size offset depth base_sha1 In order to get the output in the order that appear in the pack file for debugging purposes, you can do this: $ git-verify-pack -v packfile | sort -n -k 4,4 Signed-off-by: Junio C Hamano <> Signed-off-by: Linus Torvalds <>
2005-07-01[PATCH] verify-pack updates.Junio C Hamano
Nico pointed out that having verify_pack.c and verify-pack.c was confusing. Rename verify_pack.c to pack-check.c as suggested, and enhances the verification done quite a bit. - Built-in sha1_file unpacking knows that a base object of a deltified object _must_ be in the same pack, and takes advantage of that fact. - Earlier verify-pack command only checked the SHA1 sum for the entire pack file and did not look into its contents. It now checks everything idx file claims to have unpacks correctly. - It now has a hook to give more detailed information for objects contained in the pack under -v flag. Signed-off-by: Junio C Hamano <> Signed-off-by: Linus Torvalds <>