summaryrefslogtreecommitdiff
path: root/Documentation/git-fast-import.txt
diff options
context:
space:
mode:
Diffstat (limited to 'Documentation/git-fast-import.txt')
-rw-r--r--Documentation/git-fast-import.txt1125
1 files changed, 1125 insertions, 0 deletions
diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
new file mode 100644
index 0000000..c2f483a
--- /dev/null
+++ b/Documentation/git-fast-import.txt
@@ -0,0 +1,1125 @@
+git-fast-import(1)
+==================
+
+NAME
+----
+git-fast-import - Backend for fast Git data importers
+
+
+SYNOPSIS
+--------
+frontend | 'git fast-import' [options]
+
+DESCRIPTION
+-----------
+This program is usually not what the end user wants to run directly.
+Most end users want to use one of the existing frontend programs,
+which parses a specific type of foreign source and feeds the contents
+stored there to 'git-fast-import'.
+
+fast-import reads a mixed command/data stream from standard input and
+writes one or more packfiles directly into the current repository.
+When EOF is received on standard input, fast import writes out
+updated branch and tag refs, fully updating the current repository
+with the newly imported data.
+
+The fast-import backend itself can import into an empty repository (one that
+has already been initialized by 'git-init') or incrementally
+update an existing populated repository. Whether or not incremental
+imports are supported from a particular foreign source depends on
+the frontend program in use.
+
+
+OPTIONS
+-------
+--date-format=<fmt>::
+ Specify the type of dates the frontend will supply to
+ fast-import within `author`, `committer` and `tagger` commands.
+ See ``Date Formats'' below for details about which formats
+ are supported, and their syntax.
+
+--force::
+ Force updating modified existing branches, even if doing
+ so would cause commits to be lost (as the new commit does
+ not contain the old commit).
+
+--max-pack-size=<n>::
+ Maximum size of each output packfile, expressed in MiB.
+ The default is 4096 (4 GiB) as that is the maximum allowed
+ packfile size (due to file format limitations). Some
+ importers may wish to lower this, such as to ensure the
+ resulting packfiles fit on CDs.
+
+--depth=<n>::
+ Maximum delta depth, for blob and tree deltification.
+ Default is 10.
+
+--active-branches=<n>::
+ Maximum number of branches to maintain active at once.
+ See ``Memory Utilization'' below for details. Default is 5.
+
+--export-marks=<file>::
+ Dumps the internal marks table to <file> when complete.
+ Marks are written one per line as `:markid SHA-1`.
+ Frontends can use this file to validate imports after they
+ have been completed, or to save the marks table across
+ incremental runs. As <file> is only opened and truncated
+ at checkpoint (or completion) the same path can also be
+ safely given to \--import-marks.
+
+--import-marks=<file>::
+ Before processing any input, load the marks specified in
+ <file>. The input file must exist, must be readable, and
+ must use the same format as produced by \--export-marks.
+ Multiple options may be supplied to import more than one
+ set of marks. If a mark is defined to different values,
+ the last file wins.
+
+--export-pack-edges=<file>::
+ After creating a packfile, print a line of data to
+ <file> listing the filename of the packfile and the last
+ commit on each branch that was written to that packfile.
+ This information may be useful after importing projects
+ whose total object set exceeds the 4 GiB packfile limit,
+ as these commits can be used as edge points during calls
+ to 'git-pack-objects'.
+
+--quiet::
+ Disable all non-fatal output, making fast-import silent when it
+ is successful. This option disables the output shown by
+ \--stats.
+
+--stats::
+ Display some basic statistics about the objects fast-import has
+ created, the packfiles they were stored into, and the
+ memory used by fast-import during this run. Showing this output
+ is currently the default, but can be disabled with \--quiet.
+
+
+Performance
+-----------
+The design of fast-import allows it to import large projects in a minimum
+amount of memory usage and processing time. Assuming the frontend
+is able to keep up with fast-import and feed it a constant stream of data,
+import times for projects holding 10+ years of history and containing
+100,000+ individual commits are generally completed in just 1-2
+hours on quite modest (~$2,000 USD) hardware.
+
+Most bottlenecks appear to be in foreign source data access (the
+source just cannot extract revisions fast enough) or disk IO (fast-import
+writes as fast as the disk will take the data). Imports will run
+faster if the source data is stored on a different drive than the
+destination Git repository (due to less IO contention).
+
+
+Development Cost
+----------------
+A typical frontend for fast-import tends to weigh in at approximately 200
+lines of Perl/Python/Ruby code. Most developers have been able to
+create working importers in just a couple of hours, even though it
+is their first exposure to fast-import, and sometimes even to Git. This is
+an ideal situation, given that most conversion tools are throw-away
+(use once, and never look back).
+
+
+Parallel Operation
+------------------
+Like 'git-push' or 'git-fetch', imports handled by fast-import are safe to
+run alongside parallel `git repack -a -d` or `git gc` invocations,
+or any other Git operation (including 'git-prune', as loose objects
+are never used by fast-import).
+
+fast-import does not lock the branch or tag refs it is actively importing.
+After the import, during its ref update phase, fast-import tests each
+existing branch ref to verify the update will be a fast-forward
+update (the commit stored in the ref is contained in the new
+history of the commit to be written). If the update is not a
+fast-forward update, fast-import will skip updating that ref and instead
+prints a warning message. fast-import will always attempt to update all
+branch refs, and does not stop on the first failure.
+
+Branch updates can be forced with \--force, but its recommended that
+this only be used on an otherwise quiet repository. Using \--force
+is not necessary for an initial import into an empty repository.
+
+
+Technical Discussion
+--------------------
+fast-import tracks a set of branches in memory. Any branch can be created
+or modified at any point during the import process by sending a
+`commit` command on the input stream. This design allows a frontend
+program to process an unlimited number of branches simultaneously,
+generating commits in the order they are available from the source
+data. It also simplifies the frontend programs considerably.
+
+fast-import does not use or alter the current working directory, or any
+file within it. (It does however update the current Git repository,
+as referenced by `GIT_DIR`.) Therefore an import frontend may use
+the working directory for its own purposes, such as extracting file
+revisions from the foreign source. This ignorance of the working
+directory also allows fast-import to run very quickly, as it does not
+need to perform any costly file update operations when switching
+between branches.
+
+Input Format
+------------
+With the exception of raw file data (which Git does not interpret)
+the fast-import input format is text (ASCII) based. This text based
+format simplifies development and debugging of frontend programs,
+especially when a higher level language such as Perl, Python or
+Ruby is being used.
+
+fast-import is very strict about its input. Where we say SP below we mean
+*exactly* one space. Likewise LF means one (and only one) linefeed.
+Supplying additional whitespace characters will cause unexpected
+results, such as branch names or file names with leading or trailing
+spaces in their name, or early termination of fast-import when it encounters
+unexpected input.
+
+Stream Comments
+~~~~~~~~~~~~~~~
+To aid in debugging frontends fast-import ignores any line that
+begins with `#` (ASCII pound/hash) up to and including the line
+ending `LF`. A comment line may contain any sequence of bytes
+that does not contain an LF and therefore may be used to include
+any detailed debugging information that might be specific to the
+frontend and useful when inspecting a fast-import data stream.
+
+Date Formats
+~~~~~~~~~~~~
+The following date formats are supported. A frontend should select
+the format it will use for this import by passing the format name
+in the \--date-format=<fmt> command line option.
+
+`raw`::
+ This is the Git native format and is `<time> SP <offutc>`.
+ It is also fast-import's default format, if \--date-format was
+ not specified.
++
+The time of the event is specified by `<time>` as the number of
+seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
+written as an ASCII decimal integer.
++
+The local offset is specified by `<offutc>` as a positive or negative
+offset from UTC. For example EST (which is 5 hours behind UTC)
+would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''.
+The local offset does not affect `<time>`; it is used only as an
+advisement to help formatting routines display the timestamp.
++
+If the local offset is not available in the source material, use
+``+0000'', or the most common local offset. For example many
+organizations have a CVS repository which has only ever been accessed
+by users who are located in the same location and timezone. In this
+case a reasonable offset from UTC could be assumed.
++
+Unlike the `rfc2822` format, this format is very strict. Any
+variation in formatting will cause fast-import to reject the value.
+
+`rfc2822`::
+ This is the standard email format as described by RFC 2822.
++
+An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git
+parser is accurate, but a little on the lenient side. It is the
+same parser used by 'git-am' when applying patches
+received from email.
++
+Some malformed strings may be accepted as valid dates. In some of
+these cases Git will still be able to obtain the correct date from
+the malformed string. There are also some types of malformed
+strings which Git will parse wrong, and yet consider valid.
+Seriously malformed strings will be rejected.
++
+Unlike the `raw` format above, the timezone/UTC offset information
+contained in an RFC 2822 date string is used to adjust the date
+value to UTC prior to storage. Therefore it is important that
+this information be as accurate as possible.
++
+If the source material uses RFC 2822 style dates,
+the frontend should let fast-import handle the parsing and conversion
+(rather than attempting to do it itself) as the Git parser has
+been well tested in the wild.
++
+Frontends should prefer the `raw` format if the source material
+already uses UNIX-epoch format, can be coaxed to give dates in that
+format, or its format is easily convertible to it, as there is no
+ambiguity in parsing.
+
+`now`::
+ Always use the current time and timezone. The literal
+ `now` must always be supplied for `<when>`.
++
+This is a toy format. The current time and timezone of this system
+is always copied into the identity string at the time it is being
+created by fast-import. There is no way to specify a different time or
+timezone.
++
+This particular format is supplied as its short to implement and
+may be useful to a process that wants to create a new commit
+right now, without needing to use a working directory or
+'git-update-index'.
++
+If separate `author` and `committer` commands are used in a `commit`
+the timestamps may not match, as the system clock will be polled
+twice (once for each command). The only way to ensure that both
+author and committer identity information has the same timestamp
+is to omit `author` (thus copying from `committer`) or to use a
+date format other than `now`.
+
+Commands
+~~~~~~~~
+fast-import accepts several commands to update the current repository
+and control the current import process. More detailed discussion
+(with examples) of each command follows later.
+
+`commit`::
+ Creates a new branch or updates an existing branch by
+ creating a new commit and updating the branch to point at
+ the newly created commit.
+
+`tag`::
+ Creates an annotated tag object from an existing commit or
+ branch. Lightweight tags are not supported by this command,
+ as they are not recommended for recording meaningful points
+ in time.
+
+`reset`::
+ Reset an existing branch (or a new branch) to a specific
+ revision. This command must be used to change a branch to
+ a specific revision without making a commit on it.
+
+`blob`::
+ Convert raw file data into a blob, for future use in a
+ `commit` command. This command is optional and is not
+ needed to perform an import.
+
+`checkpoint`::
+ Forces fast-import to close the current packfile, generate its
+ unique SHA-1 checksum and index, and start a new packfile.
+ This command is optional and is not needed to perform
+ an import.
+
+`progress`::
+ Causes fast-import to echo the entire line to its own
+ standard output. This command is optional and is not needed
+ to perform an import.
+
+`commit`
+~~~~~~~~
+Create or update a branch with a new commit, recording one logical
+change to the project.
+
+....
+ 'commit' SP <ref> LF
+ mark?
+ ('author' SP <name> SP LT <email> GT SP <when> LF)?
+ 'committer' SP <name> SP LT <email> GT SP <when> LF
+ data
+ ('from' SP <committish> LF)?
+ ('merge' SP <committish> LF)?
+ (filemodify | filedelete | filecopy | filerename | filedeleteall)*
+ LF?
+....
+
+where `<ref>` is the name of the branch to make the commit on.
+Typically branch names are prefixed with `refs/heads/` in
+Git, so importing the CVS branch symbol `RELENG-1_0` would use
+`refs/heads/RELENG-1_0` for the value of `<ref>`. The value of
+`<ref>` must be a valid refname in Git. As `LF` is not valid in
+a Git refname, no quoting or escaping syntax is supported here.
+
+A `mark` command may optionally appear, requesting fast-import to save a
+reference to the newly created commit for future use by the frontend
+(see below for format). It is very common for frontends to mark
+every commit they create, thereby allowing future branch creation
+from any imported commit.
+
+The `data` command following `committer` must supply the commit
+message (see below for `data` command syntax). To import an empty
+commit message use a 0 length data. Commit messages are free-form
+and are not interpreted by Git. Currently they must be encoded in
+UTF-8, as fast-import does not permit other encodings to be specified.
+
+Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`
+and `filedeleteall` commands
+may be included to update the contents of the branch prior to
+creating the commit. These commands may be supplied in any order.
+However it is recommended that a `filedeleteall` command precede
+all `filemodify`, `filecopy` and `filerename` commands in the same
+commit, as `filedeleteall`
+wipes the branch clean (see below).
+
+The `LF` after the command is optional (it used to be required).
+
+`author`
+^^^^^^^^
+An `author` command may optionally appear, if the author information
+might differ from the committer information. If `author` is omitted
+then fast-import will automatically use the committer's information for
+the author portion of the commit. See below for a description of
+the fields in `author`, as they are identical to `committer`.
+
+`committer`
+^^^^^^^^^^^
+The `committer` command indicates who made this commit, and when
+they made it.
+
+Here `<name>` is the person's display name (for example
+``Com M Itter'') and `<email>` is the person's email address
+(``cm@example.com''). `LT` and `GT` are the literal less-than (\x3c)
+and greater-than (\x3e) symbols. These are required to delimit
+the email address from the other fields in the line. Note that
+`<name>` is free-form and may contain any sequence of bytes, except
+`LT` and `LF`. It is typically UTF-8 encoded.
+
+The time of the change is specified by `<when>` using the date format
+that was selected by the \--date-format=<fmt> command line option.
+See ``Date Formats'' above for the set of supported formats, and
+their syntax.
+
+`from`
+^^^^^^
+The `from` command is used to specify the commit to initialize
+this branch from. This revision will be the first ancestor of the
+new commit.
+
+Omitting the `from` command in the first commit of a new branch
+will cause fast-import to create that commit with no ancestor. This
+tends to be desired only for the initial commit of a project.
+If the frontend creates all files from scratch when making a new
+branch, a `merge` command may be used instead of `from` to start
+the commit with an empty tree.
+Omitting the `from` command on existing branches is usually desired,
+as the current commit on that branch is automatically assumed to
+be the first ancestor of the new commit.
+
+As `LF` is not valid in a Git refname or SHA-1 expression, no
+quoting or escaping syntax is supported within `<committish>`.
+
+Here `<committish>` is any of the following:
+
+* The name of an existing branch already in fast-import's internal branch
+ table. If fast-import doesn't know the name, its treated as a SHA-1
+ expression.
+
+* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
++
+The reason fast-import uses `:` to denote a mark reference is this character
+is not legal in a Git branch name. The leading `:` makes it easy
+to distinguish between the mark 42 (`:42`) and the branch 42 (`42`
+or `refs/heads/42`), or an abbreviated SHA-1 which happened to
+consist only of base-10 digits.
++
+Marks must be declared (via `mark`) before they can be used.
+
+* A complete 40 byte or abbreviated commit SHA-1 in hex.
+
+* Any valid Git SHA-1 expression that resolves to a commit. See
+ ``SPECIFYING REVISIONS'' in linkgit:git-rev-parse[1] for details.
+
+The special case of restarting an incremental import from the
+current branch value should be written as:
+----
+ from refs/heads/branch^0
+----
+The `{caret}0` suffix is necessary as fast-import does not permit a branch to
+start from itself, and the branch is created in memory before the
+`from` command is even read from the input. Adding `{caret}0` will force
+fast-import to resolve the commit through Git's revision parsing library,
+rather than its internal branch table, thereby loading in the
+existing value of the branch.
+
+`merge`
+^^^^^^^
+Includes one additional ancestor commit. If the `from` command is
+omitted when creating a new branch, the first `merge` commit will be
+the first ancestor of the current commit, and the branch will start
+out with no files. An unlimited number of `merge` commands per
+commit are permitted by fast-import, thereby establishing an n-way merge.
+However Git's other tools never create commits with more than 15
+additional ancestors (forming a 16-way merge). For this reason
+it is suggested that frontends do not use more than 15 `merge`
+commands per commit; 16, if starting a new, empty branch.
+
+Here `<committish>` is any of the commit specification expressions
+also accepted by `from` (see above).
+
+`filemodify`
+^^^^^^^^^^^^
+Included in a `commit` command to add a new file or change the
+content of an existing file. This command has two different means
+of specifying the content of the file.
+
+External data format::
+ The data content for the file was already supplied by a prior
+ `blob` command. The frontend just needs to connect it.
++
+....
+ 'M' SP <mode> SP <dataref> SP <path> LF
+....
++
+Here `<dataref>` can be either a mark reference (`:<idnum>`)
+set by a prior `blob` command, or a full 40-byte SHA-1 of an
+existing Git blob object.
+
+Inline data format::
+ The data content for the file has not been supplied yet.
+ The frontend wants to supply it as part of this modify
+ command.
++
+....
+ 'M' SP <mode> SP 'inline' SP <path> LF
+ data
+....
++
+See below for a detailed description of the `data` command.
+
+In both formats `<mode>` is the type of file entry, specified
+in octal. Git only supports the following modes:
+
+* `100644` or `644`: A normal (not-executable) file. The majority
+ of files in most projects use this mode. If in doubt, this is
+ what you want.
+* `100755` or `755`: A normal, but executable, file.
+* `120000`: A symlink, the content of the file will be the link target.
+* `160000`: A gitlink, SHA-1 of the object refers to a commit in
+ another repository. Git links can only be specified by SHA or through
+ a commit mark. They are used to implement submodules.
+
+In both formats `<path>` is the complete path of the file to be added
+(if not already existing) or modified (if already existing).
+
+A `<path>` string must use UNIX-style directory separators (forward
+slash `/`), may contain any byte other than `LF`, and must not
+start with double quote (`"`).
+
+If an `LF` or double quote must be encoded into `<path>` shell-style
+quoting should be used, e.g. `"path/with\n and \" in it"`.
+
+The value of `<path>` must be in canonical form. That is it must not:
+
+* contain an empty directory component (e.g. `foo//bar` is invalid),
+* end with a directory separator (e.g. `foo/` is invalid),
+* start with a directory separator (e.g. `/foo` is invalid),
+* contain the special component `.` or `..` (e.g. `foo/./bar` and
+ `foo/../bar` are invalid).
+
+It is recommended that `<path>` always be encoded using UTF-8.
+
+`filedelete`
+^^^^^^^^^^^^
+Included in a `commit` command to remove a file or recursively
+delete an entire directory from the branch. If the file or directory
+removal makes its parent directory empty, the parent directory will
+be automatically removed too. This cascades up the tree until the
+first non-empty directory or the root is reached.
+
+....
+ 'D' SP <path> LF
+....
+
+here `<path>` is the complete path of the file or subdirectory to
+be removed from the branch.
+See `filemodify` above for a detailed description of `<path>`.
+
+`filecopy`
+^^^^^^^^^^^^
+Recursively copies an existing file or subdirectory to a different
+location within the branch. The existing file or directory must
+exist. If the destination exists it will be completely replaced
+by the content copied from the source.
+
+....
+ 'C' SP <path> SP <path> LF
+....
+
+here the first `<path>` is the source location and the second
+`<path>` is the destination. See `filemodify` above for a detailed
+description of what `<path>` may look like. To use a source path
+that contains SP the path must be quoted.
+
+A `filecopy` command takes effect immediately. Once the source
+location has been copied to the destination any future commands
+applied to the source location will not impact the destination of
+the copy.
+
+`filerename`
+^^^^^^^^^^^^
+Renames an existing file or subdirectory to a different location
+within the branch. The existing file or directory must exist. If
+the destination exists it will be replaced by the source directory.
+
+....
+ 'R' SP <path> SP <path> LF
+....
+
+here the first `<path>` is the source location and the second
+`<path>` is the destination. See `filemodify` above for a detailed
+description of what `<path>` may look like. To use a source path
+that contains SP the path must be quoted.
+
+A `filerename` command takes effect immediately. Once the source
+location has been renamed to the destination any future commands
+applied to the source location will create new files there and not
+impact the destination of the rename.
+
+Note that a `filerename` is the same as a `filecopy` followed by a
+`filedelete` of the source location. There is a slight performance
+advantage to using `filerename`, but the advantage is so small
+that it is never worth trying to convert a delete/add pair in
+source material into a rename for fast-import. This `filerename`
+command is provided just to simplify frontends that already have
+rename information and don't want bother with decomposing it into a
+`filecopy` followed by a `filedelete`.
+
+`filedeleteall`
+^^^^^^^^^^^^^^^
+Included in a `commit` command to remove all files (and also all
+directories) from the branch. This command resets the internal
+branch structure to have no files in it, allowing the frontend
+to subsequently add all interesting files from scratch.
+
+....
+ 'deleteall' LF
+....
+
+This command is extremely useful if the frontend does not know
+(or does not care to know) what files are currently on the branch,
+and therefore cannot generate the proper `filedelete` commands to
+update the content.
+
+Issuing a `filedeleteall` followed by the needed `filemodify`
+commands to set the correct content will produce the same results
+as sending only the needed `filemodify` and `filedelete` commands.
+The `filedeleteall` approach may however require fast-import to use slightly
+more memory per active branch (less than 1 MiB for even most large
+projects); so frontends that can easily obtain only the affected
+paths for a commit are encouraged to do so.
+
+`mark`
+~~~~~~
+Arranges for fast-import to save a reference to the current object, allowing
+the frontend to recall this object at a future point in time, without
+knowing its SHA-1. Here the current object is the object creation
+command the `mark` command appears within. This can be `commit`,
+`tag`, and `blob`, but `commit` is the most common usage.
+
+....
+ 'mark' SP ':' <idnum> LF
+....
+
+where `<idnum>` is the number assigned by the frontend to this mark.
+The value of `<idnum>` is expressed as an ASCII decimal integer.
+The value 0 is reserved and cannot be used as
+a mark. Only values greater than or equal to 1 may be used as marks.
+
+New marks are created automatically. Existing marks can be moved
+to another object simply by reusing the same `<idnum>` in another
+`mark` command.
+
+`tag`
+~~~~~
+Creates an annotated tag referring to a specific commit. To create
+lightweight (non-annotated) tags see the `reset` command below.
+
+....
+ 'tag' SP <name> LF
+ 'from' SP <committish> LF
+ 'tagger' SP <name> SP LT <email> GT SP <when> LF
+ data
+....
+
+where `<name>` is the name of the tag to create.
+
+Tag names are automatically prefixed with `refs/tags/` when stored
+in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
+use just `RELENG-1_0-FINAL` for `<name>`, and fast-import will write the
+corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
+
+The value of `<name>` must be a valid refname in Git and therefore
+may contain forward slashes. As `LF` is not valid in a Git refname,
+no quoting or escaping syntax is supported here.
+
+The `from` command is the same as in the `commit` command; see
+above for details.
+
+The `tagger` command uses the same format as `committer` within
+`commit`; again see above for details.
+
+The `data` command following `tagger` must supply the annotated tag
+message (see below for `data` command syntax). To import an empty
+tag message use a 0 length data. Tag messages are free-form and are
+not interpreted by Git. Currently they must be encoded in UTF-8,
+as fast-import does not permit other encodings to be specified.
+
+Signing annotated tags during import from within fast-import is not
+supported. Trying to include your own PGP/GPG signature is not
+recommended, as the frontend does not (easily) have access to the
+complete set of bytes which normally goes into such a signature.
+If signing is required, create lightweight tags from within fast-import with
+`reset`, then create the annotated versions of those tags offline
+with the standard 'git-tag' process.
+
+`reset`
+~~~~~~~
+Creates (or recreates) the named branch, optionally starting from
+a specific revision. The reset command allows a frontend to issue
+a new `from` command for an existing branch, or to create a new
+branch from an existing commit without creating a new commit.
+
+....
+ 'reset' SP <ref> LF
+ ('from' SP <committish> LF)?
+ LF?
+....
+
+For a detailed description of `<ref>` and `<committish>` see above
+under `commit` and `from`.
+
+The `LF` after the command is optional (it used to be required).
+
+The `reset` command can also be used to create lightweight
+(non-annotated) tags. For example:
+
+====
+ reset refs/tags/938
+ from :938
+====
+
+would create the lightweight tag `refs/tags/938` referring to
+whatever commit mark `:938` references.
+
+`blob`
+~~~~~~
+Requests writing one file revision to the packfile. The revision
+is not connected to any commit; this connection must be formed in
+a subsequent `commit` command by referencing the blob through an
+assigned mark.
+
+....
+ 'blob' LF
+ mark?
+ data
+....
+
+The mark command is optional here as some frontends have chosen
+to generate the Git SHA-1 for the blob on their own, and feed that
+directly to `commit`. This is typically more work than its worth
+however, as marks are inexpensive to store and easy to use.
+
+`data`
+~~~~~~
+Supplies raw data (for use as blob/file content, commit messages, or
+annotated tag messages) to fast-import. Data can be supplied using an exact
+byte count or delimited with a terminating line. Real frontends
+intended for production-quality conversions should always use the
+exact byte count format, as it is more robust and performs better.
+The delimited format is intended primarily for testing fast-import.
+
+Comment lines appearing within the `<raw>` part of `data` commands
+are always taken to be part of the body of the data and are therefore
+never ignored by fast-import. This makes it safe to import any
+file/message content whose lines might start with `#`.
+
+Exact byte count format::
+ The frontend must specify the number of bytes of data.
++
+....
+ 'data' SP <count> LF
+ <raw> LF?
+....
++
+where `<count>` is the exact number of bytes appearing within
+`<raw>`. The value of `<count>` is expressed as an ASCII decimal
+integer. The `LF` on either side of `<raw>` is not
+included in `<count>` and will not be included in the imported data.
++
+The `LF` after `<raw>` is optional (it used to be required) but
+recommended. Always including it makes debugging a fast-import
+stream easier as the next command always starts in column 0
+of the next line, even if `<raw>` did not end with an `LF`.
+
+Delimited format::
+ A delimiter string is used to mark the end of the data.
+ fast-import will compute the length by searching for the delimiter.
+ This format is primarily useful for testing and is not
+ recommended for real data.
++
+....
+ 'data' SP '<<' <delim> LF
+ <raw> LF
+ <delim> LF
+ LF?
+....
++
+where `<delim>` is the chosen delimiter string. The string `<delim>`
+must not appear on a line by itself within `<raw>`, as otherwise
+fast-import will think the data ends earlier than it really does. The `LF`
+immediately trailing `<raw>` is part of `<raw>`. This is one of
+the limitations of the delimited format, it is impossible to supply
+a data chunk which does not have an LF as its last byte.
++
+The `LF` after `<delim> LF` is optional (it used to be required).
+
+`checkpoint`
+~~~~~~~~~~~~
+Forces fast-import to close the current packfile, start a new one, and to
+save out all current branch refs, tags and marks.
+
+....
+ 'checkpoint' LF
+ LF?
+....
+
+Note that fast-import automatically switches packfiles when the current
+packfile reaches \--max-pack-size, or 4 GiB, whichever limit is
+smaller. During an automatic packfile switch fast-import does not update
+the branch refs, tags or marks.
+
+As a `checkpoint` can require a significant amount of CPU time and
+disk IO (to compute the overall pack SHA-1 checksum, generate the
+corresponding index file, and update the refs) it can easily take
+several minutes for a single `checkpoint` command to complete.
+
+Frontends may choose to issue checkpoints during extremely large
+and long running imports, or when they need to allow another Git
+process access to a branch. However given that a 30 GiB Subversion
+repository can be loaded into Git through fast-import in about 3 hours,
+explicit checkpointing may not be necessary.
+
+The `LF` after the command is optional (it used to be required).
+
+`progress`
+~~~~~~~~~~
+Causes fast-import to print the entire `progress` line unmodified to
+its standard output channel (file descriptor 1) when the command is
+processed from the input stream. The command otherwise has no impact
+on the current import, or on any of fast-import's internal state.
+
+....
+ 'progress' SP <any> LF
+ LF?
+....
+
+The `<any>` part of the command may contain any sequence of bytes
+that does not contain `LF`. The `LF` after the command is optional.
+Callers may wish to process the output through a tool such as sed to
+remove the leading part of the line, for example:
+
+====
+ frontend | git fast-import | sed 's/^progress //'
+====
+
+Placing a `progress` command immediately after a `checkpoint` will
+inform the reader when the `checkpoint` has been completed and it
+can safely access the refs that fast-import updated.
+
+Crash Reports
+-------------
+If fast-import is supplied invalid input it will terminate with a
+non-zero exit status and create a crash report in the top level of
+the Git repository it was importing into. Crash reports contain
+a snapshot of the internal fast-import state as well as the most
+recent commands that lead up to the crash.
+
+All recent commands (including stream comments, file changes and
+progress commands) are shown in the command history within the crash
+report, but raw file data and commit messages are excluded from the
+crash report. This exclusion saves space within the report file
+and reduces the amount of buffering that fast-import must perform
+during execution.
+
+After writing a crash report fast-import will close the current
+packfile and export the marks table. This allows the frontend
+developer to inspect the repository state and resume the import from
+the point where it crashed. The modified branches and tags are not
+updated during a crash, as the import did not complete successfully.
+Branch and tag information can be found in the crash report and
+must be applied manually if the update is needed.
+
+An example crash:
+
+====
+ $ cat >in <<END_OF_INPUT
+ # my very first test commit
+ commit refs/heads/master
+ committer Shawn O. Pearce <spearce> 19283 -0400
+ # who is that guy anyway?
+ data <<EOF
+ this is my commit
+ EOF
+ M 644 inline .gitignore
+ data <<EOF
+ .gitignore
+ EOF
+ M 777 inline bob
+ END_OF_INPUT
+
+ $ git fast-import <in
+ fatal: Corrupt mode: M 777 inline bob
+ fast-import: dumping crash report to .git/fast_import_crash_8434
+
+ $ cat .git/fast_import_crash_8434
+ fast-import crash report:
+ fast-import process: 8434
+ parent process : 1391
+ at Sat Sep 1 00:58:12 2007
+
+ fatal: Corrupt mode: M 777 inline bob
+
+ Most Recent Commands Before Crash
+ ---------------------------------
+ # my very first test commit
+ commit refs/heads/master
+ committer Shawn O. Pearce <spearce> 19283 -0400
+ # who is that guy anyway?
+ data <<EOF
+ M 644 inline .gitignore
+ data <<EOF
+ * M 777 inline bob
+
+ Active Branch LRU
+ -----------------
+ active_branches = 1 cur, 5 max
+
+ pos clock name
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ 1) 0 refs/heads/master
+
+ Inactive Branches
+ -----------------
+ refs/heads/master:
+ status : active loaded dirty
+ tip commit : 0000000000000000000000000000000000000000
+ old tree : 0000000000000000000000000000000000000000
+ cur tree : 0000000000000000000000000000000000000000
+ commit clock: 0
+ last pack :
+
+
+ -------------------
+ END OF CRASH REPORT
+====
+
+Tips and Tricks
+---------------
+The following tips and tricks have been collected from various
+users of fast-import, and are offered here as suggestions.
+
+Use One Mark Per Commit
+~~~~~~~~~~~~~~~~~~~~~~~
+When doing a repository conversion, use a unique mark per commit
+(`mark :<n>`) and supply the \--export-marks option on the command
+line. fast-import will dump a file which lists every mark and the Git
+object SHA-1 that corresponds to it. If the frontend can tie
+the marks back to the source repository, it is easy to verify the
+accuracy and completeness of the import by comparing each Git
+commit to the corresponding source revision.
+
+Coming from a system such as Perforce or Subversion this should be
+quite simple, as the fast-import mark can also be the Perforce changeset
+number or the Subversion revision number.
+
+Freely Skip Around Branches
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Don't bother trying to optimize the frontend to stick to one branch
+at a time during an import. Although doing so might be slightly
+faster for fast-import, it tends to increase the complexity of the frontend
+code considerably.
+
+The branch LRU builtin to fast-import tends to behave very well, and the
+cost of activating an inactive branch is so low that bouncing around
+between branches has virtually no impact on import performance.
+
+Handling Renames
+~~~~~~~~~~~~~~~~
+When importing a renamed file or directory, simply delete the old
+name(s) and modify the new name(s) during the corresponding commit.
+Git performs rename detection after-the-fact, rather than explicitly
+during a commit.
+
+Use Tag Fixup Branches
+~~~~~~~~~~~~~~~~~~~~~~
+Some other SCM systems let the user create a tag from multiple
+files which are not from the same commit/changeset. Or to create
+tags which are a subset of the files available in the repository.
+
+Importing these tags as-is in Git is impossible without making at
+least one commit which ``fixes up'' the files to match the content
+of the tag. Use fast-import's `reset` command to reset a dummy branch
+outside of your normal branch space to the base commit for the tag,
+then commit one or more file fixup commits, and finally tag the
+dummy branch.
+
+For example since all normal branches are stored under `refs/heads/`
+name the tag fixup branch `TAG_FIXUP`. This way it is impossible for
+the fixup branch used by the importer to have namespace conflicts
+with real branches imported from the source (the name `TAG_FIXUP`
+is not `refs/heads/TAG_FIXUP`).
+
+When committing fixups, consider using `merge` to connect the
+commit(s) which are supplying file revisions to the fixup branch.
+Doing so will allow tools such as 'git-blame' to track
+through the real commit history and properly annotate the source
+files.
+
+After fast-import terminates the frontend will need to do `rm .git/TAG_FIXUP`
+to remove the dummy branch.
+
+Import Now, Repack Later
+~~~~~~~~~~~~~~~~~~~~~~~~
+As soon as fast-import completes the Git repository is completely valid
+and ready for use. Typically this takes only a very short time,
+even for considerably large projects (100,000+ commits).
+
+However repacking the repository is necessary to improve data
+locality and access performance. It can also take hours on extremely
+large projects (especially if -f and a large \--window parameter is
+used). Since repacking is safe to run alongside readers and writers,
+run the repack in the background and let it finish when it finishes.
+There is no reason to wait to explore your new Git project!
+
+If you choose to wait for the repack, don't try to run benchmarks
+or performance tests until repacking is completed. fast-import outputs
+suboptimal packfiles that are simply never seen in real use
+situations.
+
+Repacking Historical Data
+~~~~~~~~~~~~~~~~~~~~~~~~~
+If you are repacking very old imported data (e.g. older than the
+last year), consider expending some extra CPU time and supplying
+\--window=50 (or higher) when you run 'git-repack'.
+This will take longer, but will also produce a smaller packfile.
+You only need to expend the effort once, and everyone using your
+project will benefit from the smaller repository.
+
+Include Some Progress Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Every once in a while have your frontend emit a `progress` message
+to fast-import. The contents of the messages are entirely free-form,
+so one suggestion would be to output the current month and year
+each time the current commit date moves into the next month.
+Your users will feel better knowing how much of the data stream
+has been processed.
+
+
+Packfile Optimization
+---------------------
+When packing a blob fast-import always attempts to deltify against the last
+blob written. Unless specifically arranged for by the frontend,
+this will probably not be a prior version of the same file, so the
+generated delta will not be the smallest possible. The resulting
+packfile will be compressed, but will not be optimal.
+
+Frontends which have efficient access to all revisions of a
+single file (for example reading an RCS/CVS ,v file) can choose
+to supply all revisions of that file as a sequence of consecutive
+`blob` commands. This allows fast-import to deltify the different file
+revisions against each other, saving space in the final packfile.
+Marks can be used to later identify individual file revisions during
+a sequence of `commit` commands.
+
+The packfile(s) created by fast-import do not encourage good disk access
+patterns. This is caused by fast-import writing the data in the order
+it is received on standard input, while Git typically organizes
+data within packfiles to make the most recent (current tip) data
+appear before historical data. Git also clusters commits together,
+speeding up revision traversal through better cache locality.
+
+For this reason it is strongly recommended that users repack the
+repository with `git repack -a -d` after fast-import completes, allowing
+Git to reorganize the packfiles for faster data access. If blob
+deltas are suboptimal (see above) then also adding the `-f` option
+to force recomputation of all deltas can significantly reduce the
+final packfile size (30-50% smaller can be quite typical).
+
+
+Memory Utilization
+------------------
+There are a number of factors which affect how much memory fast-import
+requires to perform an import. Like critical sections of core
+Git, fast-import uses its own memory allocators to amortize any overheads
+associated with malloc. In practice fast-import tends to amortize any
+malloc overheads to 0, due to its use of large block allocations.
+
+per object
+~~~~~~~~~~
+fast-import maintains an in-memory structure for every object written in
+this execution. On a 32 bit system the structure is 32 bytes,
+on a 64 bit system the structure is 40 bytes (due to the larger
+pointer sizes). Objects in the table are not deallocated until
+fast-import terminates. Importing 2 million objects on a 32 bit system
+will require approximately 64 MiB of memory.
+
+The object table is actually a hashtable keyed on the object name
+(the unique SHA-1). This storage configuration allows fast-import to reuse
+an existing or already written object and avoid writing duplicates
+to the output packfile. Duplicate blobs are surprisingly common
+in an import, typically due to branch merges in the source.
+
+per mark
+~~~~~~~~
+Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
+bytes, depending on pointer size) per mark. Although the array
+is sparse, frontends are still strongly encouraged to use marks
+between 1 and n, where n is the total number of marks required for
+this import.
+
+per branch
+~~~~~~~~~~
+Branches are classified as active and inactive. The memory usage
+of the two classes is significantly different.
+
+Inactive branches are stored in a structure which uses 96 or 120
+bytes (32 bit or 64 bit systems, respectively), plus the length of
+the branch name (typically under 200 bytes), per branch. fast-import will
+easily handle as many as 10,000 inactive branches in under 2 MiB
+of memory.
+
+Active branches have the same overhead as inactive branches, but
+also contain copies of every tree that has been recently modified on
+that branch. If subtree `include` has not been modified since the
+branch became active, its contents will not be loaded into memory,
+but if subtree `src` has been modified by a commit since the branch
+became active, then its contents will be loaded in memory.
+
+As active branches store metadata about the files contained on that
+branch, their in-memory storage size can grow to a considerable size
+(see below).
+
+fast-import automatically moves active branches to inactive status based on
+a simple least-recently-used algorithm. The LRU chain is updated on
+each `commit` command. The maximum number of active branches can be
+increased or decreased on the command line with \--active-branches=.
+
+per active tree
+~~~~~~~~~~~~~~~
+Trees (aka directories) use just 12 bytes of memory on top of the
+memory required for their entries (see ``per active file'' below).
+The cost of a tree is virtually 0, as its overhead amortizes out
+over the individual file entries.
+
+per active file entry
+~~~~~~~~~~~~~~~~~~~~~
+Files (and pointers to subtrees) within active trees require 52 or 64
+bytes (32/64 bit platforms) per entry. To conserve space, file and
+tree names are pooled in a common string table, allowing the filename
+``Makefile'' to use just 16 bytes (after including the string header
+overhead) no matter how many times it occurs within the project.
+
+The active branch LRU, when coupled with the filename string pool
+and lazy loading of subtrees, allows fast-import to efficiently import
+projects with 2,000+ branches and 45,114+ files in a very limited
+memory footprint (less than 2.7 MiB per active branch).
+
+
+Author
+------
+Written by Shawn O. Pearce <spearce@spearce.org>.
+
+Documentation
+--------------
+Documentation by Shawn O. Pearce <spearce@spearce.org>.
+
+GIT
+---
+Part of the linkgit:git[1] suite