Merge branch 'maint'

* maint:
  git-apply: fix whitespace stripping
  apply --index-info: fall back to current index for mode changes
  core-tutorial: minor cleanup
  documentation: replace Discussion section by link to user-manual chapter
  user-manual: todo updates and cleanup
  user-manual: fix introduction to packfiles
  user-manual: move packfile and dangling object discussion
  user-manual: rewrite object database discussion
  user-manual: reorder commit, blob, tree discussion
  user-manual: rewrite index discussion
  user-manual: create new "low-level git operations" chapter
  user-manual: rename "git internals" to "git concepts"
  user-manual: move object format details to hacking-git chapter
  user-manual: adjust section levels in "git internals"
  revision walker: --cherry-pick is a limited operation
  git-sh-setup: typofix in comments

5 files changed, 548 insertions, 1003 deletions

diff --git a/Documentation/Makefile b/Documentation/Makefile
index fbefe9a..39ec0ed 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -123,7 +123,7 @@ cmd-list.made: cmd-list.perl $(MAN1_TXT)
 	perl ./cmd-list.perl
 	date >$@
 
-git.7 git.html: git.txt core-intro.txt
+git.7 git.html: git.txt
 
 clean:
 	$(RM) *.xml *.xml+ *.html *.html+ *.1 *.5 *.7 *.texi *.texi+ howto-index.txt howto/*.html doc.dep
diff --git a/Documentation/core-intro.txt b/Documentation/core-intro.txt
deleted file mode 100644
index f3cc223..0000000
--- a/Documentation/core-intro.txt
+++ /dev/null
@@ -1,592 +0,0 @@
-////////////////////////////////////////////////////////////////
-
-	GIT - the stupid content tracker
-
-////////////////////////////////////////////////////////////////
-
-"git" can mean anything, depending on your mood.
-
- - random three-letter combination that is pronounceable, and not
-   actually used by any common UNIX command.  The fact that it is a
-   mispronunciation of "get" may or may not be relevant.
- - stupid. contemptible and despicable. simple. Take your pick from the
-   dictionary of slang.
- - "global information tracker": you're in a good mood, and it actually
-   works for you. Angels sing, and a light suddenly fills the room.
- - "goddamn idiotic truckload of sh*t": when it breaks
-
-This is a (not so) stupid but extremely fast directory content manager.
-It doesn't do a whole lot at its core, but what it 'does' do is track
-directory contents efficiently.
-
-There are two object abstractions: the "object database", and the
-"current directory cache" aka "index".
-
-The Object Database
-~~~~~~~~~~~~~~~~~~~
-The object database is literally just a content-addressable collection
-of objects.  All objects are named by their content, which is
-approximated by the SHA1 hash of the object itself.  Objects may refer
-to other objects (by referencing their SHA1 hash), and so you can
-build up a hierarchy of objects.
-
-All objects have a statically determined "type" aka "tag", which is
-determined at object creation time, and which identifies the format of
-the object (i.e. how it is used, and how it can refer to other
-objects).  There are currently four different object types: "blob",
-"tree", "commit" and "tag".
-
-A "blob" object cannot refer to any other object, and is, like the type
-implies, a pure storage object containing some user data.  It is used to
-actually store the file data, i.e. a blob object is associated with some
-particular version of some file.
-
-A "tree" object is an object that ties one or more "blob" objects into a
-directory structure. In addition, a tree object can refer to other tree
-objects, thus creating a directory hierarchy.
-
-A "commit" object ties such directory hierarchies together into
-a DAG of revisions - each "commit" is associated with exactly one tree
-(the directory hierarchy at the time of the commit). In addition, a
-"commit" refers to one or more "parent" commit objects that describe the
-history of how we arrived at that directory hierarchy.
-
-As a special case, a commit object with no parents is called the "root"
-object, and is the point of an initial project commit.  Each project
-must have at least one root, and while you can tie several different
-root objects together into one project by creating a commit object which
-has two or more separate roots as its ultimate parents, that's probably
-just going to confuse people.  So aim for the notion of "one root object
-per project", even if git itself does not enforce that.
-
-A "tag" object symbolically identifies and can be used to sign other
-objects. It contains the identifier and type of another object, a
-symbolic name (of course!) and, optionally, a signature.
-
-Regardless of object type, all objects share the following
-characteristics: they are all deflated with zlib, and have a header
-that not only specifies their type, but also provides size information
-about the data in the object.  It's worth noting that the SHA1 hash
-that is used to name the object is the hash of the original data
-plus this header, so `sha1sum` 'file' does not match the object name
-for 'file'.
-(Historical note: in the dawn of the age of git the hash
-was the sha1 of the 'compressed' object.)
-
-As a result, the general consistency of an object can always be tested
-independently of the contents or the type of the object: all objects can
-be validated by verifying that (a) their hashes match the content of the
-file and (b) the object successfully inflates to a stream of bytes that
-forms a sequence of <ascii type without space> + <space> + <ascii decimal
-size> + <byte\0> + <binary object data>.
-
-The structured objects can further have their structure and
-connectivity to other objects verified. This is generally done with
-the `git-fsck` program, which generates a full dependency graph
-of all objects, and verifies their internal consistency (in addition
-to just verifying their superficial consistency through the hash).
-
-The object types in some more detail:
-
-Blob Object
-~~~~~~~~~~~
-A "blob" object is nothing but a binary blob of data, and doesn't
-refer to anything else.  There is no signature or any other
-verification of the data, so while the object is consistent (it 'is'
-indexed by its sha1 hash, so the data itself is certainly correct), it
-has absolutely no other attributes.  No name associations, no
-permissions.  It is purely a blob of data (i.e. normally "file
-contents").
-
-In particular, since the blob is entirely defined by its data, if two
-files in a directory tree (or in multiple different versions of the
-repository) have the same contents, they will share the same blob
-object. The object is totally independent of its location in the
-directory tree, and renaming a file does not change the object that
-file is associated with in any way.
-
-A blob is typically created when gitlink:git-update-index[1]
-(or gitlink:git-add[1]) is run, and its data can be accessed by
-gitlink:git-cat-file[1].
-
-Tree Object
-~~~~~~~~~~~
-The next hierarchical object type is the "tree" object.  A tree object
-is a list of mode/name/blob data, sorted by name.  Alternatively, the
-mode data may specify a directory mode, in which case instead of
-naming a blob, that name is associated with another TREE object.
-
-Like the "blob" object, a tree object is uniquely determined by the
-set contents, and so two separate but identical trees will always
-share the exact same object. This is true at all levels, i.e. it's
-true for a "leaf" tree (which does not refer to any other trees, only
-blobs) as well as for a whole subdirectory.
-
-For that reason a "tree" object is just a pure data abstraction: it
-has no history, no signatures, no verification of validity, except
-that since the contents are again protected by the hash itself, we can
-trust that the tree is immutable and its contents never change.
-
-So you can trust the contents of a tree to be valid, the same way you
-can trust the contents of a blob, but you don't know where those
-contents 'came' from.
-
-Side note on trees: since a "tree" object is a sorted list of
-"filename+content", you can create a diff between two trees without
-actually having to unpack two trees.  Just ignore all common parts,
-and your diff will look right.  In other words, you can effectively
-(and efficiently) tell the difference between any two random trees by
-O(n) where "n" is the size of the difference, rather than the size of
-the tree.
-
-Side note 2 on trees: since the name of a "blob" depends entirely and
-exclusively on its contents (i.e. there are no names or permissions
-involved), you can see trivial renames or permission changes by
-noticing that the blob stayed the same.  However, renames with data
-changes need a smarter "diff" implementation.
-
-A tree is created with gitlink:git-write-tree[1] and
-its data can be accessed by gitlink:git-ls-tree[1].
-Two trees can be compared with gitlink:git-diff-tree[1].
-
-Commit Object
-~~~~~~~~~~~~~
-The "commit" object is an object that introduces the notion of
-history into the picture.  In contrast to the other objects, it
-doesn't just describe the physical state of a tree, it describes how
-we got there, and why.
-
-A "commit" is defined by the tree-object that it results in, the
-parent commits (zero, one or more) that led up to that point, and a
-comment on what happened.  Again, a commit is not trusted per se:
-the contents are well-defined and "safe" due to the cryptographically
-strong signatures at all levels, but there is no reason to believe
-that the tree is "good" or that the merge information makes sense.
-The parents do not have to actually have any relationship with the
-result, for example.
-
-Note on commits: unlike real SCM's, commits do not contain
-rename information or file mode change information.  All of that is
-implicit in the trees involved (the result tree, and the result trees
-of the parents), and describing that makes no sense in this idiotic
-file manager.
-
-A commit is created with gitlink:git-commit-tree[1] and
-its data can be accessed by gitlink:git-cat-file[1].
-
-Trust
-~~~~~
-An aside on the notion of "trust". Trust is really outside the scope
-of "git", but it's worth noting a few things.  First off, since
-everything is hashed with SHA1, you 'can' trust that an object is
-intact and has not been messed with by external sources.  So the name
-of an object uniquely identifies a known state - just not a state that
-you may want to trust.
-
-Furthermore, since the SHA1 signature of a commit refers to the
-SHA1 signatures of the tree it is associated with and the signatures
-of the parent, a single named commit specifies uniquely a whole set
-of history, with full contents.  You can't later fake any step of the
-way once you have the name of a commit.
-
-So to introduce some real trust in the system, the only thing you need
-to do is to digitally sign just 'one' special note, which includes the
-name of a top-level commit.  Your digital signature shows others
-that you trust that commit, and the immutability of the history of
-commits tells others that they can trust the whole history.
-
-In other words, you can easily validate a whole archive by just
-sending out a single email that tells the people the name (SHA1 hash)
-of the top commit, and digitally sign that email using something
-like GPG/PGP.
-
-To assist in this, git also provides the tag object...
-
-Tag Object
-~~~~~~~~~~
-Git provides the "tag" object to simplify creating, managing and
-exchanging symbolic and signed tokens.  The "tag" object at its
-simplest simply symbolically identifies another object by containing
-the sha1, type and symbolic name.
-
-However it can optionally contain additional signature information
-(which git doesn't care about as long as there's less than 8k of
-it). This can then be verified externally to git.
-
-Note that despite the tag features, "git" itself only handles content
-integrity; the trust framework (and signature provision and
-verification) has to come from outside.
-
-A tag is created with gitlink:git-mktag[1],
-its data can be accessed by gitlink:git-cat-file[1],
-and the signature can be verified by
-gitlink:git-verify-tag[1].
-
-
-The "index" aka "Current Directory Cache"
------------------------------------------
-The index is a simple binary file, which contains an efficient
-representation of a virtual directory content at some random time.  It
-does so by a simple array that associates a set of names, dates,
-permissions and content (aka "blob") objects together.  The cache is
-always kept ordered by name, and names are unique (with a few very
-specific rules) at any point in time, but the cache has no long-term
-meaning, and can be partially updated at any time.
-
-In particular, the index certainly does not need to be consistent with
-the current directory contents (in fact, most operations will depend on
-different ways to make the index 'not' be consistent with the directory
-hierarchy), but it has three very important attributes:
-
-'(a) it can re-generate the full state it caches (not just the
-directory structure: it contains pointers to the "blob" objects so
-that it can regenerate the data too)'
-
-As a special case, there is a clear and unambiguous one-way mapping
-from a current directory cache to a "tree object", which can be
-efficiently created from just the current directory cache without
-actually looking at any other data.  So a directory cache at any one
-time uniquely specifies one and only one "tree" object (but has
-additional data to make it easy to match up that tree object with what
-has happened in the directory)
-
-'(b) it has efficient methods for finding inconsistencies between that
-cached state ("tree object waiting to be instantiated") and the
-current state.'
-
-'(c) it can additionally efficiently represent information about merge
-conflicts between different tree objects, allowing each pathname to be
-associated with sufficient information about the trees involved that
-you can create a three-way merge between them.'
-
-Those are the three ONLY things that the directory cache does.  It's a
-cache, and the normal operation is to re-generate it completely from a
-known tree object, or update/compare it with a live tree that is being
-developed.  If you blow the directory cache away entirely, you generally
-haven't lost any information as long as you have the name of the tree
-that it described.
-
-At the same time, the index is at the same time also the
-staging area for creating new trees, and creating a new tree always
-involves a controlled modification of the index file.  In particular,
-the index file can have the representation of an intermediate tree that
-has not yet been instantiated.  So the index can be thought of as a
-write-back cache, which can contain dirty information that has not yet
-been written back to the backing store.
-
-
-
-The Workflow
-------------
-Generally, all "git" operations work on the index file. Some operations
-work *purely* on the index file (showing the current state of the
-index), but most operations move data to and from the index file. Either
-from the database or from the working directory. Thus there are four
-main combinations:
-
-1) working directory -> index
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You update the index with information from the working directory with
-the gitlink:git-update-index[1] command.  You
-generally update the index information by just specifying the filename
-you want to update, like so:
-
-	git-update-index filename
-
-but to avoid common mistakes with filename globbing etc, the command
-will not normally add totally new entries or remove old entries,
-i.e. it will normally just update existing cache entries.
-
-To tell git that yes, you really do realize that certain files no
-longer exist, or that new files should be added, you
-should use the `--remove` and `--add` flags respectively.
-
-NOTE! A `--remove` flag does 'not' mean that subsequent filenames will
-necessarily be removed: if the files still exist in your directory
-structure, the index will be updated with their new status, not
-removed. The only thing `--remove` means is that update-cache will be
-considering a removed file to be a valid thing, and if the file really
-does not exist any more, it will update the index accordingly.
-
-As a special case, you can also do `git-update-index --refresh`, which
-will refresh the "stat" information of each index to match the current
-stat information. It will 'not' update the object status itself, and
-it will only update the fields that are used to quickly test whether
-an object still matches its old backing store object.
-
-2) index -> object database
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You write your current index file to a "tree" object with the program
-
-	git-write-tree
-
-that doesn't come with any options - it will just write out the
-current index into the set of tree objects that describe that state,
-and it will return the name of the resulting top-level tree. You can
-use that tree to re-generate the index at any time by going in the
-other direction:
-
-3) object database -> index
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You read a "tree" file from the object database, and use that to
-populate (and overwrite - don't do this if your index contains any
-unsaved state that you might want to restore later!) your current
-index.  Normal operation is just
-
-		git-read-tree <sha1 of tree>
-
-and your index file will now be equivalent to the tree that you saved
-earlier. However, that is only your 'index' file: your working
-directory contents have not been modified.
-
-4) index -> working directory
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You update your working directory from the index by "checking out"
-files. This is not a very common operation, since normally you'd just
-keep your files updated, and rather than write to your working
-directory, you'd tell the index files about the changes in your
-working directory (i.e. `git-update-index`).
-
-However, if you decide to jump to a new version, or check out somebody
-else's version, or just restore a previous tree, you'd populate your
-index file with read-tree, and then you need to check out the result
-with
-
-		git-checkout-index filename
-
-or, if you want to check out all of the index, use `-a`.
-
-NOTE! git-checkout-index normally refuses to overwrite old files, so
-if you have an old version of the tree already checked out, you will
-need to use the "-f" flag ('before' the "-a" flag or the filename) to
-'force' the checkout.
-
-
-Finally, there are a few odds and ends which are not purely moving
-from one representation to the other:
-
-5) Tying it all together
-~~~~~~~~~~~~~~~~~~~~~~~~
-To commit a tree you have instantiated with "git-write-tree", you'd
-create a "commit" object that refers to that tree and the history
-behind it - most notably the "parent" commits that preceded it in
-history.
-
-Normally a "commit" has one parent: the previous state of the tree
-before a certain change was made. However, sometimes it can have two
-or more parent commits, in which case we call it a "merge", due to the
-fact that such a commit brings together ("merges") two or more
-previous states represented by other commits.
-
-In other words, while a "tree" represents a particular directory state
-of a working directory, a "commit" represents that state in "time",
-and explains how we got there.
-
-You create a commit object by giving it the tree that describes the
-state at the time of the commit, and a list of parents:
-
-	git-commit-tree <tree> -p <parent> [-p <parent2> ..]
-
-and then giving the reason for the commit on stdin (either through
-redirection from a pipe or file, or by just typing it at the tty).
-
-git-commit-tree will return the name of the object that represents
-that commit, and you should save it away for later use. Normally,
-you'd commit a new `HEAD` state, and while git doesn't care where you
-save the note about that state, in practice we tend to just write the
-result to the file pointed at by `.git/HEAD`, so that we can always see
-what the last committed state was.
-
-Here is an ASCII art by Jon Loeliger that illustrates how
-various pieces fit together.
-
-------------
-
-                     commit-tree
-                      commit obj
-                       +----+
-                       |    |
-                       |    |
-                       V    V
-                    +-----------+
-                    | Object DB |
-                    |  Backing  |
-                    |   Store   |
-                    +-----------+
-                       ^
-           write-tree  |     |
-             tree obj  |     |
-                       |     |  read-tree
-                       |     |  tree obj
-                             V
-                    +-----------+
-                    |   Index   |
-                    |  "cache"  |
-                    +-----------+
-         update-index  ^
-             blob obj  |     |
-                       |     |
-    checkout-index -u  |     |  checkout-index
-             stat      |     |  blob obj
-                             V
-                    +-----------+
-                    |  Working  |
-                    | Directory |
-                    +-----------+
-
-------------
-
-
-6) Examining the data
-~~~~~~~~~~~~~~~~~~~~~
-
-You can examine the data represented in the object database and the
-index with various helper tools. For every object, you can use
-gitlink:git-cat-file[1] to examine details about the
-object:
-
-		git-cat-file -t <objectname>
-
-shows the type of the object, and once you have the type (which is
-usually implicit in where you find the object), you can use
-
-		git-cat-file blob|tree|commit|tag <objectname>
-
-to show its contents. NOTE! Trees have binary content, and as a result
-there is a special helper for showing that content, called
-`git-ls-tree`, which turns the binary content into a more easily
-readable form.
-
-It's especially instructive to look at "commit" objects, since those
-tend to be small and fairly self-explanatory. In particular, if you
-follow the convention of having the top commit name in `.git/HEAD`,
-you can do
-
-		git-cat-file commit HEAD
-
-to see what the top commit was.
-
-7) Merging multiple trees
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Git helps you do a three-way merge, which you can expand to n-way by
-repeating the merge procedure arbitrary times until you finally
-"commit" the state.  The normal situation is that you'd only do one
-three-way merge (two parents), and commit it, but if you like to, you
-can do multiple parents in one go.
-
-To do a three-way merge, you need the two sets of "commit" objects
-that you want to merge, use those to find the closest common parent (a
-third "commit" object), and then use those commit objects to find the
-state of the directory ("tree" object) at these points.
-
-To get the "base" for the merge, you first look up the common parent
-of two commits with
-
-		git-merge-base <commit1> <commit2>
-
-which will return you the commit they are both based on.  You should
-now look up the "tree" objects of those commits, which you can easily
-do with (for example)
-
-		git-cat-file commit <commitname> | head -1
-
-since the tree object information is always the first line in a commit
-object.
-
-Once you know the three trees you are going to merge (the one
-"original" tree, aka the common case, and the two "result" trees, aka
-the branches you want to merge), you do a "merge" read into the
-index. This will complain if it has to throw away your old index contents, so you should
-make sure that you've committed those - in fact you would normally
-always do a merge against your last commit (which should thus match
-what you have in your current index anyway).
-
-To do the merge, do
-
-		git-read-tree -m -u <origtree> <yourtree> <targettree>
-
-which will do all trivial merge operations for you directly in the
-index file, and you can just write the result out with
-`git-write-tree`.
-
-Historical note.  We did not have `-u` facility when this
-section was first written, so we used to warn that
-the merge is done in the index file, not in your
-working tree, and your working tree will not match your
-index after this step.
-This is no longer true.  The above command, thanks to `-u`
-option, updates your working tree with the merge results for
-paths that have been trivially merged.
-
-
-8) Merging multiple trees, continued
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Sadly, many merges aren't trivial. If there are files that have
-been added, moved or removed, or if both branches have modified the
-same file, you will be left with an index tree that contains "merge
-entries" in it. Such an index tree can 'NOT' be written out to a tree
-object, and you will have to resolve any such merge clashes using
-other tools before you can write out the result.
-
-You can examine such index state with `git-ls-files --unmerged`
-command.  An example:
-
-------------------------------------------------
-$ git-read-tree -m $orig HEAD $target
-$ git-ls-files --unmerged
-100644 263414f423d0e4d70dae8fe53fa34614ff3e2860 1	hello.c
-100644 06fa6a24256dc7e560efa5687fa84b51f0263c3a 2	hello.c
-100644 cc44c73eb783565da5831b4d820c962954019b69 3	hello.c
-------------------------------------------------
-
-Each line of the `git-ls-files --unmerged` output begins with
-the blob mode bits, blob SHA1, 'stage number', and the
-filename.  The 'stage number' is git's way to say which tree it
-came from: stage 1 corresponds to `$orig` tree, stage 2 `HEAD`
-tree, and stage3 `$target` tree.
-
-Earlier we said that trivial merges are done inside
-`git-read-tree -m`.  For example, if the file did not change
-from `$orig` to `HEAD` nor `$target`, or if the file changed
-from `$orig` to `HEAD` and `$orig` to `$target` the same way,
-obviously the final outcome is what is in `HEAD`.  What the
-above example shows is that file `hello.c` was changed from
-`$orig` to `HEAD` and `$orig` to `$target` in a different way.
-You could resolve this by running your favorite 3-way merge
-program, e.g.  `diff3` or `merge`, on the blob objects from
-these three stages yourself, like this:
-
-------------------------------------------------
-$ git-cat-file blob 263414f... >hello.c~1
-$ git-cat-file blob 06fa6a2... >hello.c~2
-$ git-cat-file blob cc44c73... >hello.c~3
-$ merge hello.c~2 hello.c~1 hello.c~3
-------------------------------------------------
-
-This would leave the merge result in `hello.c~2` file, along
-with conflict markers if there are conflicts.  After verifying
-the merge result makes sense, you can tell git what the final
-merge result for this file is by:
-
-	mv -f hello.c~2 hello.c
-	git-update-index hello.c
-
-When a path is in unmerged state, running `git-update-index` for
-that path tells git to mark the path resolved.
-
-The above is the description of a git merge at the lowest level,
-to help you understand what conceptually happens under the hood.
-In practice, nobody, not even git itself, uses three `git-cat-file`
-for this.  There is `git-merge-index` program that extracts the
-stages to temporary files and calls a "merge" script on it:
-
-	git-merge-index git-merge-one-file hello.c
-
-and that is what higher level `git merge -s resolve` is implemented
-with.
diff --git a/Documentation/core-tutorial.txt b/Documentation/core-tutorial.txt
index 4fb6f41..4b4fd9a 100644
--- a/Documentation/core-tutorial.txt
+++ b/Documentation/core-tutorial.txt
@@ -4,34 +4,24 @@ A git core tutorial for developers
 Introduction
 ------------
 
-This is trying to be a short tutorial on setting up and using a git
-repository, mainly because being hands-on and using explicit examples is
-often the best way of explaining what is going on.
+This tutorial explains how to use the "core" git programs to set up and
+work with a git repository.
 
-In normal life, most people wouldn't use the "core" git programs
-directly, but rather script around them to make them more palatable.
-Understanding the core git stuff may help some people get those scripts
-done, though, and it may also be instructive in helping people
-understand what it is that the higher-level helper scripts are actually
-doing.
+If you just need to use git as a revision control system you may prefer
+to start with link:tutorial.html[a tutorial introduction to git] or
+link:user-manual.html[the git user manual].
+
+However, an understanding of these low-level tools can be helpful if
+you want to understand git's internals.
 
 The core git is often called "plumbing", with the prettier user
 interfaces on top of it called "porcelain". You may not want to use the
 plumbing directly very often, but it can be good to know what the
 plumbing does for when the porcelain isn't flushing.
 
-The material presented here often goes deep describing how things
-work internally.  If you are mostly interested in using git as a
-SCM, you can skip them during your first pass.
-
 [NOTE]
-And those "too deep" descriptions are often marked as Note.
-
-[NOTE]
-If you are already familiar with another version control system,
-like CVS, you may want to take a look at
-link:everyday.html[Everyday GIT in 20 commands or so] first
-before reading this.
+Deeper technical details are often marked as Notes, which you can
+skip on your first reading.
 
 
 Creating a git repository
@@ -1686,5 +1676,3 @@ merge two at a time, documenting how you resolved the conflicts,
 and the reason why you preferred changes made in one side over
 the other.  Otherwise it would make the project history harder
 to follow, not easier.
-
-[ to be continued.. cvsimports ]
diff --git a/Documentation/git.txt b/Documentation/git.txt
index 6f7db29..a7cd91a 100644
--- a/Documentation/git.txt
+++ b/Documentation/git.txt
@@ -134,9 +134,9 @@ FURTHER DOCUMENTATION
 See the references above to get started using git.  The following is
 probably more detail than necessary for a first-time user.
 
-The <<Discussion,Discussion>> section below and the
-link:core-tutorial.html[Core tutorial] both provide introductions to the
-underlying git architecture.
+The link:user-manual.html#git-concepts[git concepts chapter of the
+user-manual] and the link:core-tutorial.html[Core tutorial] both provide
+introductions to the underlying git architecture.
 
 See also the link:howto-index.html[howto] documents for some useful
 examples.
@@ -474,7 +474,56 @@ for further details.
 
 Discussion[[Discussion]]
 ------------------------
-include::core-intro.txt[]
+
+More detail on the following is available from the
+link:user-manual.html#git-concepts[git concepts chapter of the
+user-manual] and the link:core-tutorial.html[Core tutorial].
+
+A git project normally consists of a working directory with a ".git"
+subdirectory at the top level.  The .git directory contains, among other
+things, a compressed object database representing the complete history
+of the project, an "index" file which links that history to the current
+contents of the working tree, and named pointers into that history such
+as tags and branch heads.
+
+The object database contains objects of three main types: blobs, which
+hold file data; trees, which point to blobs and other trees to build up
+directory heirarchies; and commits, which each reference a single tree
+and some number of parent commits.
+
+The commit, equivalent to what other systems call a "changeset" or
+"version", represents a step in the project's history, and each parent
+represents an immediately preceding step.  Commits with more than one
+parent represent merges of independent lines of development.
+
+All objects are named by the SHA1 hash of their contents, normally
+written as a string of 40 hex digits.  Such names are globally unique.
+The entire history leading up to a commit can be vouched for by signing
+just that commit.  A fourth object type, the tag, is provided for this
+purpose.
+
+When first created, objects are stored in individual files, but for
+efficiency may later be compressed together into "pack files".
+
+Named pointers called refs mark interesting points in history.  A ref
+may contain the SHA1 name of an object or the name of another ref.  Refs
+with names beginning `ref/head/` contain the SHA1 name of the most
+recent commit (or "head") of a branch under developement.  SHA1 names of
+tags of interest are stored under `ref/tags/`.  A special ref named
+`HEAD` contains the name of the currently checked-out branch.
+
+The index file is initialized with a list of all paths and, for each
+path, a blob object and a set of attributes.  The blob object represents
+the contents of the file as of the head of the current branch.  The
+attributes (last modified time, size, etc.) are taken from the
+corresponding file in the working tree.  Subsequent changes to the
+working tree can be found by comparing these attributes.  The index may
+be updated with new content, and new commits may be created from the
+content stored in the index.
+
+The index is also capable of storing multiple entries (called "stages")
+for a given pathname.  These stages are used to hold the various
+unmerged version of a file when a merge is in progress.
 
 Authors
 -------
diff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt
index 35298e6..ecb2bf9 100644
--- a/Documentation/user-manual.txt
+++ b/Documentation/user-manual.txt
@@ -182,7 +182,7 @@ has that commit at all).  Since the object name is computed as a hash over the
 contents of the commit, you are guaranteed that the commit can never change
 without its name also changing.
 
-In fact, in <<git-internals>> we shall see that everything stored in git
+In fact, in <<git-concepts>> we shall see that everything stored in git
 history, including file data and directory contents, is stored in an object
 with a name that is a hash of its contents.
 
@@ -2708,190 +2708,202 @@ See gitlink:git-config[1] for more details on the configuration
 options mentioned above.
 
 
-[[git-internals]]
-Git internals
-=============
+[[git-concepts]]
+Git concepts
+============
 
-Git depends on two fundamental abstractions: the "object database", and
-the "current directory cache" aka "index".
+Git is built on a small number of simple but powerful ideas.  While it
+is possible to get things done without understanding them, you will find
+git much more intuitive if you do.
+
+We start with the most important, the  <<def_object_database,object
+database>> and the <<def_index,index>>.
 
 [[the-object-database]]
 The Object Database
 -------------------
 
-The object database is literally just a content-addressable collection
-of objects.  All objects are named by their content, which is
-approximated by the SHA1 hash of the object itself.  Objects may refer
-to other objects (by referencing their SHA1 hash), and so you can
-build up a hierarchy of objects.
 
-All objects have a statically determined "type" which is
-determined at object creation time, and which identifies the format of
-the object (i.e. how it is used, and how it can refer to other
-objects).  There are currently four different object types: "blob",
-"tree", "commit", and "tag".
+We already saw in <<understanding-commits>> that all commits are stored
+under a 40-digit "object name".  In fact, all the information needed to
+represent the history of a project is stored in objects with such names.
+In each case the name is calculated by taking the SHA1 hash of the
+contents of the object.  The SHA1 hash is a cryptographic hash function.
+What that means to us is that it is impossible to find two different
+objects with the same name.  This has a number of advantages; among
+others:
+
+- Git can quickly determine whether two objects are identical or not,
+  just by comparing names.
+- Since object names are computed the same way in ever repository, the
+  same content stored in two repositories will always be stored under
+  the same name.
+- Git can detect errors when it reads an object, by checking that the
+  object's name is still the SHA1 hash of its contents.
+
+(See <<object-details>> for the details of the object formatting and
+SHA1 calculation.)
+
+There are four different types of objects: "blob", "tree", "commit", and
+"tag".
+
+- A <<def_blob_object,"blob" object>> is used to store file data.
+- A <<def_tree_object,"tree" object>> is an object that ties one or more
+  "blob" objects into a directory structure. In addition, a tree object
+  can refer to other tree objects, thus creating a directory hierarchy.
+- A <<def_commit_object,"commit" object>> ties such directory hierarchies
+  together into a <<def_DAG,directed acyclic graph>> of revisions - each
+  commit contains the object name of exactly one tree designating the
+  directory hierarchy at the time of the commit. In addition, a commit
+  refers to "parent" commit objects that describe the history of how we
+  arrived at that directory hierarchy.
+- A <<def_tag_object,"tag" object>> symbolically identifies and can be
+  used to sign other objects. It contains the object name and type of
+  another object, a symbolic name (of course!) and, optionally, a
+  signature.
 
-A <<def_blob_object,"blob" object>> cannot refer to any other object,
-and is, as the name implies, a pure storage object containing some
-user data.  It is used to actually store the file data, i.e. a blob
-object is associated with some particular version of some file.
-
-A <<def_tree_object,"tree" object>> is an object that ties one or more
-"blob" objects into a directory structure. In addition, a tree object
-can refer to other tree objects, thus creating a directory hierarchy.
-
-A <<def_commit_object,"commit" object>> ties such directory hierarchies
-together into a <<def_DAG,directed acyclic graph>> of revisions - each
-"commit" is associated with exactly one tree (the directory hierarchy at
-the time of the commit). In addition, a "commit" refers to one or more
-"parent" commit objects that describe the history of how we arrived at
-that directory hierarchy.
-
-As a special case, a commit object with no parents is called the "root"
-commit, and is the point of an initial project commit.  Each project
-must have at least one root, and while you can tie several different
-root objects together into one project by creating a commit object which
-has two or more separate roots as its ultimate parents, that's probably
-just going to confuse people.  So aim for the notion of "one root object
-per project", even if git itself does not enforce that.
-
-A <<def_tag_object,"tag" object>> symbolically identifies and can be
-used to sign other objects. It contains the identifier and type of
-another object, a symbolic name (of course!) and, optionally, a
-signature.
+The object types in some more detail:
 
-Regardless of object type, all objects share the following
-characteristics: they are all deflated with zlib, and have a header
-that not only specifies their type, but also provides size information
-about the data in the object.  It's worth noting that the SHA1 hash
-that is used to name the object is the hash of the original data
-plus this header, so `sha1sum` 'file' does not match the object name
-for 'file'.
-(Historical note: in the dawn of the age of git the hash
-was the sha1 of the 'compressed' object.)
+[[commit-object]]
+Commit Object
+~~~~~~~~~~~~~
 
-As a result, the general consistency of an object can always be tested
-independently of the contents or the type of the object: all objects can
-be validated by verifying that (a) their hashes match the content of the
-file and (b) the object successfully inflates to a stream of bytes that
-forms a sequence of <ascii type without space> {plus} <space> {plus} <ascii decimal
-size> {plus} <byte\0> {plus} <binary object data>.
+The "commit" object links a physical state of a tree with a description
+of how we got there and why.  Use the --pretty=raw option to
+gitlink:git-show[1] or gitlink:git-log[1] to examine your favorite
+commit:
 
-The structured objects can further have their structure and
-connectivity to other objects verified. This is generally done with
-the `git-fsck` program, which generates a full dependency graph
-of all objects, and verifies their internal consistency (in addition
-to just verifying their superficial consistency through the hash).
+------------------------------------------------
+$ git show -s --pretty=raw 2be7fcb476
+commit 2be7fcb4764f2dbcee52635b91fedb1b3dcf7ab4
+tree fb3a8bdd0ceddd019615af4d57a53f43d8cee2bf
+parent 257a84d9d02e90447b149af58b271c19405edb6a
+author Dave Watson <dwatson@mimvista.com> 1187576872 -0400
+committer Junio C Hamano <gitster@pobox.com> 1187591163 -0700
 
-The object types in some more detail:
+    Fix misspelling of 'suppress' in docs
+
+    Signed-off-by: Junio C Hamano <gitster@pobox.com>
+------------------------------------------------
+
+As you can see, a commit is defined by:
+
+- a tree: The SHA1 name of a tree object (as defined below), representing
+  the contents of a directory at a certain point in time.
+- parent(s): The SHA1 name of some number of commits which represent the
+  immediately prevoius step(s) in the history of the project.  The
+  example above has one parent; merge commits may have more than
+  one.  A commit with no parents is called a "root" commit, and
+  represents the initial revision of a project.  Each project must have
+  at least one root.  A project can also have multiple roots, though
+  that isn't common (or necessarily a good idea).
+- an author: The name of the person responsible for this change, together
+  with its date.
+- a committer: The name of the person who actually created the commit,
+  with the date it was done.  This may be different from the author, for
+  example, if the author was someone who wrote a patch and emailed it
+  to the person who used it to create the commit.
+- a comment describing this commit.
+
+Note that a commit does not itself contain any information about what
+actually changed; all changes are calculated by comparing the contents
+of the tree referred to by this commit with the trees associated with
+its parents.  In particular, git does not attempt to record file renames
+explicitly, though it can identify cases where the existence of the same
+file data at changing paths suggests a rename.  (See, for example, the
+-M option to gitlink:git-diff[1]).
+
+A commit is usually created by gitlink:git-commit[1], which creates a
+commit whose parent is normally the current HEAD, and whose tree is
+taken from the content currently stored in the index.
+
+[[tree-object]]
+Tree Object
+~~~~~~~~~~~
+
+The ever-versatile gitlink:git-show[1] command can also be used to
+examine tree objects, but gitlink:git-ls-tree[1] will give you more
+details:
+
+------------------------------------------------
+$ git ls-tree fb3a8bdd0ce
+100644 blob 63c918c667fa005ff12ad89437f2fdc80926e21c    .gitignore
+100644 blob 5529b198e8d14decbe4ad99db3f7fb632de0439d    .mailmap
+100644 blob 6ff87c4664981e4397625791c8ea3bbb5f2279a3    COPYING
+040000 tree 2fb783e477100ce076f6bf57e4a6f026013dc745    Documentation
+100755 blob 3c0032cec592a765692234f1cba47dfdcc3a9200    GIT-VERSION-GEN
+100644 blob 289b046a443c0647624607d471289b2c7dcd470b    INSTALL
+100644 blob 4eb463797adc693dc168b926b6932ff53f17d0b1    Makefile
+100644 blob 548142c327a6790ff8821d67c2ee1eff7a656b52    README
+...
+------------------------------------------------
+
+As you can see, a tree object contains a list of entries, each with a
+mode, object type, SHA1 name, and name, sorted by name.  It represents
+the contents of a single directory tree.
+
+The object type may be a blob, representing the contents of a file, or
+another tree, representing the contents of a subdirectory.  Since trees
+and blobs, like all other objects, are named by the SHA1 hash of their
+contents, two trees have the same SHA1 name if and only if their
+contents (including, recursively, the contents of all subdirectories)
+are identical.  This allows git to quickly determine the differences
+between two related tree objects, since it can ignore any entries with
+identical object names.
+
+(Note: in the presence of submodules, trees may also have commits as
+entries.   See gitlink:git-submodule[1] and gitlink:gitmodules.txt[1]
+for partial documentation.)
+
+Note that the files all have mode 644 or 755: git actually only pays
+attention to the executable bit.
 
 [[blob-object]]
 Blob Object
------------
+~~~~~~~~~~~
 
-A "blob" object is nothing but a binary blob of data, and doesn't
-refer to anything else.  There is no signature or any other
-verification of the data, so while the object is consistent (it 'is'
-indexed by its sha1 hash, so the data itself is certainly correct), it
-has absolutely no other attributes.  No name associations, no
-permissions.  It is purely a blob of data (i.e. normally "file
-contents").
+You can use gitlink:git-show[1] to examine the contents of a blob; take,
+for example, the blob in the entry for "COPYING" from the tree above:
 
-In particular, since the blob is entirely defined by its data, if two
-files in a directory tree (or in multiple different versions of the
-repository) have the same contents, they will share the same blob
-object. The object is totally independent of its location in the
-directory tree, and renaming a file does not change the object that
-file is associated with in any way.
-
-A blob is typically created when gitlink:git-update-index[1]
-is run, and its data can be accessed by gitlink:git-cat-file[1].
+------------------------------------------------
+$ git show 6ff87c4664
 
-[[tree-object]]
-Tree Object
------------
+ Note that the only valid version of the GPL as far as this project
+ is concerned is _this_ particular version of the license (ie v2, not
+ v2.2 or v3.x or whatever), unless explicitly otherwise stated.
+...
+------------------------------------------------
 
-The next hierarchical object type is the "tree" object.  A tree object
-is a list of mode/name/blob data, sorted by name.  Alternatively, the
-mode data may specify a directory mode, in which case instead of
-naming a blob, that name is associated with another TREE object.
-
-Like the "blob" object, a tree object is uniquely determined by the
-set contents, and so two separate but identical trees will always
-share the exact same object. This is true at all levels, i.e. it's
-true for a "leaf" tree (which does not refer to any other trees, only
-blobs) as well as for a whole subdirectory.
-
-For that reason a "tree" object is just a pure data abstraction: it
-has no history, no signatures, no verification of validity, except
-that since the contents are again protected by the hash itself, we can
-trust that the tree is immutable and its contents never change.
-
-So you can trust the contents of a tree to be valid, the same way you
-can trust the contents of a blob, but you don't know where those
-contents 'came' from.
-
-Side note on trees: since a "tree" object is a sorted list of
-"filename+content", you can create a diff between two trees without
-actually having to unpack two trees.  Just ignore all common parts,
-and your diff will look right.  In other words, you can effectively
-(and efficiently) tell the difference between any two random trees by
-O(n) where "n" is the size of the difference, rather than the size of
-the tree.
-
-Side note 2 on trees: since the name of a "blob" depends entirely and
-exclusively on its contents (i.e. there are no names or permissions
-involved), you can see trivial renames or permission changes by
-noticing that the blob stayed the same.  However, renames with data
-changes need a smarter "diff" implementation.
-
-A tree is created with gitlink:git-write-tree[1] and
-its data can be accessed by gitlink:git-ls-tree[1].
-Two trees can be compared with gitlink:git-diff-tree[1].
+A "blob" object is nothing but a binary blob of data.  It doesn't refer
+to anything else or have attributes of any kind.
 
-[[commit-object]]
-Commit Object
--------------
+Since the blob is entirely defined by its data, if two files in a
+directory tree (or in multiple different versions of the repository)
+have the same contents, they will share the same blob object. The object
+is totally independent of its location in the directory tree, and
+renaming a file does not change the object that file is associated with.
 
-The "commit" object is an object that introduces the notion of
-history into the picture.  In contrast to the other objects, it
-doesn't just describe the physical state of a tree, it describes how
-we got there, and why.
-
-A "commit" is defined by the tree-object that it results in, the
-parent commits (zero, one or more) that led up to that point, and a
-comment on what happened.  Again, a commit is not trusted per se:
-the contents are well-defined and "safe" due to the cryptographically
-strong signatures at all levels, but there is no reason to believe
-that the tree is "good" or that the merge information makes sense.
-The parents do not have to actually have any relationship with the
-result, for example.
-
-Note on commits: unlike some SCM's, commits do not contain
-rename information or file mode change information.  All of that is
-implicit in the trees involved (the result tree, and the result trees
-of the parents), and describing that makes no sense in this idiotic
-file manager.
-
-A commit is created with gitlink:git-commit-tree[1] and
-its data can be accessed by gitlink:git-cat-file[1].
+Note that any tree or blob object can be examined using
+gitlink:git-show[1] with the <revision>:<path> syntax.  This can
+sometimes be useful for browsing the contents of a tree that is not
+currently checked out.
 
 [[trust]]
 Trust
------
+~~~~~
 
-An aside on the notion of "trust". Trust is really outside the scope
-of "git", but it's worth noting a few things.  First off, since
-everything is hashed with SHA1, you 'can' trust that an object is
-intact and has not been messed with by external sources.  So the name
-of an object uniquely identifies a known state - just not a state that
-you may want to trust.
+If you receive the SHA1 name of a blob from one source, and its contents
+from another (possibly untrusted) source, you can still trust that those
+contents are correct as long as the SHA1 name agrees.  This is because
+the SHA1 is designed so that it is infeasible to find different contents
+that produce the same hash.
 
-Furthermore, since the SHA1 signature of a commit refers to the
-SHA1 signatures of the tree it is associated with and the signatures
-of the parent, a single named commit specifies uniquely a whole set
-of history, with full contents.  You can't later fake any step of the
-way once you have the name of a commit.
+Similarly, you need only trust the SHA1 name of a top-level tree object
+to trust the contents of the entire directory that it refers to, and if
+you receive the SHA1 name of a commit from a trusted source, then you
+can easily verify the entire history of commits reachable through
+parents of that commit, and all of those contents of the trees referred
+to by those commits.
 
 So to introduce some real trust in the system, the only thing you need
 to do is to digitally sign just 'one' special note, which includes the
@@ -2908,103 +2920,294 @@ To assist in this, git also provides the tag object...
 
 [[tag-object]]
 Tag Object
-----------
+~~~~~~~~~~
+
+A tag object contains an object, object type, tag name, the name of the
+person ("tagger") who created the tag, and a message, which may contain
+a signature, as can be seen using the gitlink:git-cat-file[1]:
+
+------------------------------------------------
+$ git cat-file tag v1.5.0
+object 437b1b20df4b356c9342dac8d38849f24ef44f27
+type commit
+tag v1.5.0
+tagger Junio C Hamano <junkio@cox.net> 1171411200 +0000
+
+GIT 1.5.0
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.6 (GNU/Linux)
+
+iD8DBQBF0lGqwMbZpPMRm5oRAuRiAJ9ohBLd7s2kqjkKlq1qqC57SbnmzQCdG4ui
+nLE/L9aUXdWeTFPron96DLA=
+=2E+0
+-----END PGP SIGNATURE-----
+------------------------------------------------
+
+See the gitlink:git-tag[1] command to learn how to create and verify tag
+objects.  (Note that gitlink:git-tag[1] can also be used to create
+"lightweight tags", which are not tag objects at all, but just simple
+references in .git/refs/tags/).
+
+[[pack-files]]
+How git stores objects efficiently: pack files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Newly created objects are initially created in a file named after the
+object's SHA1 hash (stored in .git/objects).
+
+Unfortunately this system becomes inefficient once a project has a
+lot of objects.  Try this on an old project:
+
+------------------------------------------------
+$ git count-objects
+6930 objects, 47620 kilobytes
+------------------------------------------------
+
+The first number is the number of objects which are kept in
+individual files.  The second is the amount of space taken up by
+those "loose" objects.
+
+You can save space and make git faster by moving these loose objects in
+to a "pack file", which stores a group of objects in an efficient
+compressed format; the details of how pack files are formatted can be
+found in link:technical/pack-format.txt[technical/pack-format.txt].
+
+To put the loose objects into a pack, just run git repack:
+
+------------------------------------------------
+$ git repack
+Generating pack...
+Done counting 6020 objects.
+Deltifying 6020 objects.
+ 100% (6020/6020) done
+Writing 6020 objects.
+ 100% (6020/6020) done
+Total 6020, written 6020 (delta 4070), reused 0 (delta 0)
+Pack pack-3e54ad29d5b2e05838c75df582c65257b8d08e1c created.
+------------------------------------------------
+
+You can then run
+
+------------------------------------------------
+$ git prune
+------------------------------------------------
 
-Git provides the "tag" object to simplify creating, managing and
-exchanging symbolic and signed tokens.  The "tag" object at its
-simplest simply symbolically identifies another object by containing
-the sha1, type and symbolic name.
+to remove any of the "loose" objects that are now contained in the
+pack.  This will also remove any unreferenced objects (which may be
+created when, for example, you use "git reset" to remove a commit).
+You can verify that the loose objects are gone by looking at the
+.git/objects directory or by running
 
-However it can optionally contain additional signature information
-(which git doesn't care about as long as there's less than 8k of
-it). This can then be verified externally to git.
+------------------------------------------------
+$ git count-objects
+0 objects, 0 kilobytes
+------------------------------------------------
 
-Note that despite the tag features, "git" itself only handles content
-integrity; the trust framework (and signature provision and
-verification) has to come from outside.
+Although the object files are gone, any commands that refer to those
+objects will work exactly as they did before.
 
-A tag is created with gitlink:git-mktag[1],
-its data can be accessed by gitlink:git-cat-file[1],
-and the signature can be verified by
-gitlink:git-verify-tag[1].
+The gitlink:git-gc[1] command performs packing, pruning, and more for
+you, so is normally the only high-level command you need.
 
+[[dangling-objects]]
+Dangling objects
+~~~~~~~~~~~~~~~~
+
+The gitlink:git-fsck[1] command will sometimes complain about dangling
+objects.  They are not a problem.
+
+The most common cause of dangling objects is that you've rebased a
+branch, or you have pulled from somebody else who rebased a branch--see
+<<cleaning-up-history>>.  In that case, the old head of the original
+branch still exists, as does everything it pointed to. The branch
+pointer itself just doesn't, since you replaced it with another one.
+
+There are also other situations that cause dangling objects. For
+example, a "dangling blob" may arise because you did a "git add" of a
+file, but then, before you actually committed it and made it part of the
+bigger picture, you changed something else in that file and committed
+that *updated* thing - the old state that you added originally ends up
+not being pointed to by any commit or tree, so it's now a dangling blob
+object.
+
+Similarly, when the "recursive" merge strategy runs, and finds that
+there are criss-cross merges and thus more than one merge base (which is
+fairly unusual, but it does happen), it will generate one temporary
+midway tree (or possibly even more, if you had lots of criss-crossing
+merges and more than two merge bases) as a temporary internal merge
+base, and again, those are real objects, but the end result will not end
+up pointing to them, so they end up "dangling" in your repository.
+
+Generally, dangling objects aren't anything to worry about. They can
+even be very useful: if you screw something up, the dangling objects can
+be how you recover your old tree (say, you did a rebase, and realized
+that you really didn't want to - you can look at what dangling objects
+you have, and decide to reset your head to some old dangling state).
+
+For commits, you can just use:
+
+------------------------------------------------
+$ gitk <dangling-commit-sha-goes-here> --not --all
+------------------------------------------------
+
+This asks for all the history reachable from the given commit but not
+from any branch, tag, or other reference.  If you decide it's something
+you want, you can always create a new reference to it, e.g.,
+
+------------------------------------------------
+$ git branch recovered-branch <dangling-commit-sha-goes-here>
+------------------------------------------------
+
+For blobs and trees, you can't do the same, but you can still examine
+them.  You can just do
+
+------------------------------------------------
+$ git show <dangling-blob/tree-sha-goes-here>
+------------------------------------------------
+
+to show what the contents of the blob were (or, for a tree, basically
+what the "ls" for that directory was), and that may give you some idea
+of what the operation was that left that dangling object.
+
+Usually, dangling blobs and trees aren't very interesting. They're
+almost always the result of either being a half-way mergebase (the blob
+will often even have the conflict markers from a merge in it, if you
+have had conflicting merges that you fixed up by hand), or simply
+because you interrupted a "git fetch" with ^C or something like that,
+leaving _some_ of the new objects in the object database, but just
+dangling and useless.
+
+Anyway, once you are sure that you're not interested in any dangling
+state, you can just prune all unreachable objects:
+
+------------------------------------------------
+$ git prune
+------------------------------------------------
+
+and they'll be gone. But you should only run "git prune" on a quiescent
+repository - it's kind of like doing a filesystem fsck recovery: you
+don't want to do that while the filesystem is mounted.
+
+(The same is true of "git-fsck" itself, btw - but since
+git-fsck never actually *changes* the repository, it just reports
+on what it found, git-fsck itself is never "dangerous" to run.
+Running it while somebody is actually changing the repository can cause
+confusing and scary messages, but it won't actually do anything bad. In
+contrast, running "git prune" while somebody is actively changing the
+repository is a *BAD* idea).
 
 [[the-index]]
-The "index" aka "Current Directory Cache"
------------------------------------------
+The index
+-----------
 
-The index is a simple binary file, which contains an efficient
-representation of the contents of a virtual directory.  It
-does so by a simple array that associates a set of names, dates,
-permissions and content (aka "blob") objects together.  The cache is
-always kept ordered by name, and names are unique (with a few very
-specific rules) at any point in time, but the cache has no long-term
-meaning, and can be partially updated at any time.
-
-In particular, the index certainly does not need to be consistent with
-the current directory contents (in fact, most operations will depend on
-different ways to make the index 'not' be consistent with the directory
-hierarchy), but it has three very important attributes:
-
-'(a) it can re-generate the full state it caches (not just the
-directory structure: it contains pointers to the "blob" objects so
-that it can regenerate the data too)'
-
-As a special case, there is a clear and unambiguous one-way mapping
-from a current directory cache to a "tree object", which can be
-efficiently created from just the current directory cache without
-actually looking at any other data.  So a directory cache at any one
-time uniquely specifies one and only one "tree" object (but has
-additional data to make it easy to match up that tree object with what
-has happened in the directory)
-
-'(b) it has efficient methods for finding inconsistencies between that
-cached state ("tree object waiting to be instantiated") and the
-current state.'
-
-'(c) it can additionally efficiently represent information about merge
-conflicts between different tree objects, allowing each pathname to be
+The index is a binary file (generally kept in .git/index) containing a
+sorted list of path names, each with permissions and the SHA1 of a blob
+object; gitlink:git-ls-files[1] can show you the contents of the index:
+
+-------------------------------------------------
+$ git ls-files --stage
+100644 63c918c667fa005ff12ad89437f2fdc80926e21c 0	.gitignore
+100644 5529b198e8d14decbe4ad99db3f7fb632de0439d 0	.mailmap
+100644 6ff87c4664981e4397625791c8ea3bbb5f2279a3 0	COPYING
+100644 a37b2152bd26be2c2289e1f57a292534a51a93c7 0	Documentation/.gitignore
+100644 fbefe9a45b00a54b58d94d06eca48b03d40a50e0 0	Documentation/Makefile
+...
+100644 2511aef8d89ab52be5ec6a5e46236b4b6bcd07ea 0	xdiff/xtypes.h
+100644 2ade97b2574a9f77e7ae4002a4e07a6a38e46d07 0	xdiff/xutils.c
+100644 d5de8292e05e7c36c4b68857c1cf9855e3d2f70a 0	xdiff/xutils.h
+-------------------------------------------------
+
+Note that in older documentation you may see the index called the
+"current directory cache" or just the "cache".  It has three important
+properties:
+
+1. The index contains all the information necessary to generate a single
+(uniquely determined) tree object.
++
+For example, running gitlink:git-commit[1] generates this tree object
+from the index, stores it in the object database, and uses it as the
+tree object associated with the new commit.
+
+2. The index enables fast comparisons between the tree object it defines
+and the working tree.
++
+It does this by storing some additional data for each entry (such as
+the last modified time).  This data is not displayed above, and is not
+stored in the created tree object, but it can be used to determine
+quickly which files in the working directory differ from what was
+stored in the index, and thus save git from having to read all of the
+data from such files to look for changes.
+
+3. It can efficiently represent information about merge conflicts
+between different tree objects, allowing each pathname to be
 associated with sufficient information about the trees involved that
-you can create a three-way merge between them.'
+you can create a three-way merge between them.
++
+We saw in <<conflict-resolution>> that during a merge the index can
+store multiple versions of a single file (called "stages").  The third
+column in the gitlink:git-ls-files[1] output above is the stage
+number, and will take on values other than 0 for files with merge
+conflicts.
+
+The index is thus a sort of temporary staging area, which is filled with
+a tree which you are in the process of working on.
+
+If you blow the index away entirely, you generally haven't lost any
+information as long as you have the name of the tree that it described.
+
+[[low-level-operations]]
+Low-level git operations
+========================
+
+Many of the higher-level commands were originally implemented as shell
+scripts using a smaller core of low-level git commands.  These can still
+be useful when doing unusual things with git, or just as a way to
+understand its inner workings.
+
+[[object-manipulation]]
+Object access and manipulation
+------------------------------
 
-Those are the ONLY three things that the directory cache does.  It's a
-cache, and the normal operation is to re-generate it completely from a
-known tree object, or update/compare it with a live tree that is being
-developed.  If you blow the directory cache away entirely, you generally
-haven't lost any information as long as you have the name of the tree
-that it described.
+The gitlink:git-cat-file[1] command can show the contents of any object,
+though the higher-level gitlink:git-show[1] is usually more useful.
 
-At the same time, the index is also the staging area for creating
-new trees, and creating a new tree always involves a controlled
-modification of the index file.  In particular, the index file can
-have the representation of an intermediate tree that has not yet been
-instantiated.  So the index can be thought of as a write-back cache,
-which can contain dirty information that has not yet been written back
-to the backing store.
+The gitlink:git-commit-tree[1] command allows constructing commits with
+arbitrary parents and trees.
 
+A tree can be created with gitlink:git-write-tree[1] and its data can be
+accessed by gitlink:git-ls-tree[1].  Two trees can be compared with
+gitlink:git-diff-tree[1].
 
+A tag is created with gitlink:git-mktag[1], and the signature can be
+verified by gitlink:git-verify-tag[1], though it is normally simpler to
+use gitlink:git-tag[1] for both.
 
 [[the-workflow]]
 The Workflow
 ------------
 
+High-level operations such as gitlink:git-commit[1],
+gitlink:git-checkout[1] and git-reset[1] work by moving data between the
+working tree, the index, and the object database.  Git provides
+low-level operations which perform each of these steps individually.
+
 Generally, all "git" operations work on the index file. Some operations
 work *purely* on the index file (showing the current state of the
-index), but most operations move data to and from the index file. Either
-from the database or from the working directory. Thus there are four
-main combinations:
+index), but most operations move data between the index file and either
+the database or the working directory. Thus there are four main
+combinations:
 
 [[working-directory-to-index]]
 working directory -> index
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-You update the index with information from the working directory with
-the gitlink:git-update-index[1] command.  You
-generally update the index information by just specifying the filename
-you want to update, like so:
+The gitlink:git-update-index[1] command updates the index with
+information from the working directory.  You generally update the
+index information by just specifying the filename you want to update,
+like so:
 
 -------------------------------------------------
-$ git-update-index filename
+$ git update-index filename
 -------------------------------------------------
 
 but to avoid common mistakes with filename globbing etc, the command
@@ -3028,6 +3231,9 @@ stat information. It will 'not' update the object status itself, and
 it will only update the fields that are used to quickly test whether
 an object still matches its old backing store object.
 
+The previously introduced gitlink:git-add[1] is just a wrapper for
+gitlink:git-update-index[1].
+
 [[index-to-object-database]]
 index -> object database
 ~~~~~~~~~~~~~~~~~~~~~~~~
@@ -3035,7 +3241,7 @@ index -> object database
 You write your current index file to a "tree" object with the program
 
 -------------------------------------------------
-$ git-write-tree
+$ git write-tree
 -------------------------------------------------
 
 that doesn't come with any options - it will just write out the
@@ -3326,153 +3532,44 @@ $ git-merge-index git-merge-one-file hello.c
 
 and that is what higher level `git merge -s resolve` is implemented with.
 
-[[pack-files]]
-How git stores objects efficiently: pack files
-----------------------------------------------
+[[hacking-git]]
+Hacking git
+===========
 
-We've seen how git stores each object in a file named after the
-object's SHA1 hash.
+This chapter covers internal details of the git implementation which
+probably only git developers need to understand.
 
-Unfortunately this system becomes inefficient once a project has a
-lot of objects.  Try this on an old project:
-
-------------------------------------------------
-$ git count-objects
-6930 objects, 47620 kilobytes
-------------------------------------------------
-
-The first number is the number of objects which are kept in
-individual files.  The second is the amount of space taken up by
-those "loose" objects.
-
-You can save space and make git faster by moving these loose objects in
-to a "pack file", which stores a group of objects in an efficient
-compressed format; the details of how pack files are formatted can be
-found in link:technical/pack-format.txt[technical/pack-format.txt].
-
-To put the loose objects into a pack, just run git repack:
-
-------------------------------------------------
-$ git repack
-Generating pack...
-Done counting 6020 objects.
-Deltifying 6020 objects.
- 100% (6020/6020) done
-Writing 6020 objects.
- 100% (6020/6020) done
-Total 6020, written 6020 (delta 4070), reused 0 (delta 0)
-Pack pack-3e54ad29d5b2e05838c75df582c65257b8d08e1c created.
-------------------------------------------------
-
-You can then run
-
-------------------------------------------------
-$ git prune
-------------------------------------------------
-
-to remove any of the "loose" objects that are now contained in the
-pack.  This will also remove any unreferenced objects (which may be
-created when, for example, you use "git reset" to remove a commit).
-You can verify that the loose objects are gone by looking at the
-.git/objects directory or by running
-
-------------------------------------------------
-$ git count-objects
-0 objects, 0 kilobytes
-------------------------------------------------
-
-Although the object files are gone, any commands that refer to those
-objects will work exactly as they did before.
-
-The gitlink:git-gc[1] command performs packing, pruning, and more for
-you, so is normally the only high-level command you need.
-
-[[dangling-objects]]
-Dangling objects
-----------------
-
-The gitlink:git-fsck[1] command will sometimes complain about dangling
-objects.  They are not a problem.
-
-The most common cause of dangling objects is that you've rebased a
-branch, or you have pulled from somebody else who rebased a branch--see
-<<cleaning-up-history>>.  In that case, the old head of the original
-branch still exists, as does everything it pointed to. The branch
-pointer itself just doesn't, since you replaced it with another one.
-
-There are also other situations that cause dangling objects. For
-example, a "dangling blob" may arise because you did a "git add" of a
-file, but then, before you actually committed it and made it part of the
-bigger picture, you changed something else in that file and committed
-that *updated* thing - the old state that you added originally ends up
-not being pointed to by any commit or tree, so it's now a dangling blob
-object.
-
-Similarly, when the "recursive" merge strategy runs, and finds that
-there are criss-cross merges and thus more than one merge base (which is
-fairly unusual, but it does happen), it will generate one temporary
-midway tree (or possibly even more, if you had lots of criss-crossing
-merges and more than two merge bases) as a temporary internal merge
-base, and again, those are real objects, but the end result will not end
-up pointing to them, so they end up "dangling" in your repository.
-
-Generally, dangling objects aren't anything to worry about. They can
-even be very useful: if you screw something up, the dangling objects can
-be how you recover your old tree (say, you did a rebase, and realized
-that you really didn't want to - you can look at what dangling objects
-you have, and decide to reset your head to some old dangling state).
-
-For commits, you can just use:
-
-------------------------------------------------
-$ gitk <dangling-commit-sha-goes-here> --not --all
-------------------------------------------------
-
-This asks for all the history reachable from the given commit but not
-from any branch, tag, or other reference.  If you decide it's something
-you want, you can always create a new reference to it, e.g.,
-
-------------------------------------------------
-$ git branch recovered-branch <dangling-commit-sha-goes-here>
-------------------------------------------------
-
-For blobs and trees, you can't do the same, but you can still examine
-them.  You can just do
-
-------------------------------------------------
-$ git show <dangling-blob/tree-sha-goes-here>
-------------------------------------------------
-
-to show what the contents of the blob were (or, for a tree, basically
-what the "ls" for that directory was), and that may give you some idea
-of what the operation was that left that dangling object.
-
-Usually, dangling blobs and trees aren't very interesting. They're
-almost always the result of either being a half-way mergebase (the blob
-will often even have the conflict markers from a merge in it, if you
-have had conflicting merges that you fixed up by hand), or simply
-because you interrupted a "git fetch" with ^C or something like that,
-leaving _some_ of the new objects in the object database, but just
-dangling and useless.
+[[object-details]]
+Object storage format
+---------------------
 
-Anyway, once you are sure that you're not interested in any dangling
-state, you can just prune all unreachable objects:
+All objects have a statically determined "type" which identifies the
+format of the object (i.e. how it is used, and how it can refer to other
+objects).  There are currently four different object types: "blob",
+"tree", "commit", and "tag".
 
-------------------------------------------------
-$ git prune
-------------------------------------------------
+Regardless of object type, all objects share the following
+characteristics: they are all deflated with zlib, and have a header
+that not only specifies their type, but also provides size information
+about the data in the object.  It's worth noting that the SHA1 hash
+that is used to name the object is the hash of the original data
+plus this header, so `sha1sum` 'file' does not match the object name
+for 'file'.
+(Historical note: in the dawn of the age of git the hash
+was the sha1 of the 'compressed' object.)
 
-and they'll be gone. But you should only run "git prune" on a quiescent
-repository - it's kind of like doing a filesystem fsck recovery: you
-don't want to do that while the filesystem is mounted.
+As a result, the general consistency of an object can always be tested
+independently of the contents or the type of the object: all objects can
+be validated by verifying that (a) their hashes match the content of the
+file and (b) the object successfully inflates to a stream of bytes that
+forms a sequence of <ascii type without space> {plus} <space> {plus} <ascii decimal
+size> {plus} <byte\0> {plus} <binary object data>.
 
-(The same is true of "git-fsck" itself, btw - but since
-git-fsck never actually *changes* the repository, it just reports
-on what it found, git-fsck itself is never "dangerous" to run.
-Running it while somebody is actually changing the repository can cause
-confusing and scary messages, but it won't actually do anything bad. In
-contrast, running "git prune" while somebody is actively changing the
-repository is a *BAD* idea).
+The structured objects can further have their structure and
+connectivity to other objects verified. This is generally done with
+the `git-fsck` program, which generates a full dependency graph
+of all objects, and verifies their internal consistency (in addition
+to just verifying their superficial consistency through the hash).
 
 [[birdview-on-the-source-code]]
 A birds-eye view of Git's source code
@@ -3926,25 +4023,26 @@ Appendix B: Notes and todo list for this manual
 This is a work in progress.
 
 The basic requirements:
-	- It must be readable in order, from beginning to end, by
-	  someone intelligent with a basic grasp of the UNIX
-	  command line, but without any special knowledge of git.  If
-	  necessary, any other prerequisites should be specifically
-	  mentioned as they arise.
-	- Whenever possible, section headings should clearly describe
-	  the task they explain how to do, in language that requires
-	  no more knowledge than necessary: for example, "importing
-	  patches into a project" rather than "the git-am command"
+
+- It must be readable in order, from beginning to end, by someone
+  intelligent with a basic grasp of the UNIX command line, but without
+  any special knowledge of git.  If necessary, any other prerequisites
+  should be specifically mentioned as they arise.
+- Whenever possible, section headings should clearly describe the task
+  they explain how to do, in language that requires no more knowledge
+  than necessary: for example, "importing patches into a project" rather
+  than "the git-am command"
 
 Think about how to create a clear chapter dependency graph that will
 allow people to get to important topics without necessarily reading
 everything in between.
 
 Scan Documentation/ for other stuff left out; in particular:
-	howto's
-	some of technical/?
-	hooks
-	list of commands in gitlink:git[1]
+
+- howto's
+- some of technical/?
+- hooks
+- list of commands in gitlink:git[1]
 
 Scan email archives for other stuff left out
 
@@ -3973,3 +4071,5 @@ Write a chapter on using plumbing and writing scripts.
 Alternates, clone -reference, etc.
 
 git unpack-objects -r for recovery
+
+submodules