path: root/convert.h
diff options
authorSteffen Prohaska <>2014-08-26 15:23:25 (GMT)
committerJunio C Hamano <>2014-08-28 17:25:15 (GMT)
commit9035d75a2be9d80d82676504d69553245017f6d4 (patch)
treea275b416bd102af42760a60edc77594744dbeb7e /convert.h
parentb29763aa9bcbb99a59aec3820e30ff1864cfa765 (diff)
convert: stream from fd to required clean filter to reduce used address space
The data is streamed to the filter process anyway. Better avoid mapping the file if possible. This is especially useful if a clean filter reduces the size, for example if it computes a sha1 for binary data, like git media. The file size that the previous implementation could handle was limited by the available address space; large files for example could not be handled with (32-bit) msysgit. The new implementation can filter files of any size as long as the filter output is small enough. The new code path is only taken if the filter is required. The filter consumes data directly from the fd. If it fails, the original data is not immediately available. The condition can easily be handled as a fatal error, which is expected for a required filter anyway. If the filter was not required, the condition would need to be handled in a different way, like seeking to 0 and reading the data. But this would require more restructuring of the code and is probably not worth it. The obvious approach of falling back to reading all data would not help achieving the main purpose of this patch, which is to handle large files with limited address space. If reading all data is an option, we can simply take the old code path right away and mmap the entire file. The environment variable GIT_MMAP_LIMIT, which has been introduced in a previous commit is used to test that the expected code path is taken. A related test that exercises required filters is modified to verify that the data actually has been modified on its way from the file system to the object store. Signed-off-by: Steffen Prohaska <> Signed-off-by: Junio C Hamano <>
Diffstat (limited to 'convert.h')
1 files changed, 5 insertions, 0 deletions
diff --git a/convert.h b/convert.h
index c638b33..d9d853c 100644
--- a/convert.h
+++ b/convert.h
@@ -44,6 +44,11 @@ static inline int would_convert_to_git(const char *path)
return convert_to_git(path, NULL, 0, NULL, 0);
+/* Precondition: would_convert_to_git_filter_fd(path) == true */
+extern void convert_to_git_filter_fd(const char *path, int fd,
+ struct strbuf *dst,
+ enum safe_crlf checksafe);
+extern int would_convert_to_git_filter_fd(const char *path);