[PATCH 00/23] Prepare to separate out a packed_ref_store

[Michael Haggerty <mhagger@alum.mit.edu>]

This patch series is the next leg on a journey towards reading
`packed-refs` using `mmap()`, the most interesting aspect of which is
that we will often be able to avoid having to read the whole
`packed-refs` file if we only need a subset of references.

The first leg of the journey was separating out the reference cache
into a separate module [1]. That branch is already merged to master.

This patch series prepares the ground for separating out a
`packed_ref_store`, but doesn't yet take that step. (As you can see,
it's a long enough patch series already!) It's kind of a grab bag of
cleanup patches plus work to decouple the packed-refs handling code
from the rest of `files_ref_store`. Some highlights:

* Patch 07/23 adds a log message parameter to `refs_delete_refs()` and
  `delete_refs()`, for consistency with other reference-changing
  operations. Even though `files_ref_store` is incapable of storing
  reflogs for deleted references, that is no reason that the API
  shouldn't admit the possibility for future backends.

* Patch 12/23 breaks `ref_transaction_commit()` into multiple
  functions:

  * `ref_transaction_prepare()`: do pre-checks, obtain locks, etc.; do
    everything possible to make sure that the reference update will be
    successful.

  * `ref_transaction_finish()`: actually commit a prepared
    transaction.

  * `ref_transaction_abort()`: abort a prepared transaction.

  This separation will be useful for supporting a "compound" reference
  store composed of multiple reference stores that work together
  (i.e., one for loose refs and one for packed refs).

* Patch 17/23 changes `get_packed_ref_cache()` to skip `lstat()`ing
  the packed-refs file (to check its freshness) if we already have it
  locked.

* Patch 19/23 fixes the error handling in `read_packed_refs()`: if
  `fopen()` fails due to `ENOENT`, then there are no packed refs. But
  if it fails for another reason, that is a problem that should be
  reported to the user.

* Patch 22/23 (by Peff) changes ref-filter to attempt to limit the
  reference traversal to a prefix, if there is a single
  `match_as_path` pattern that starts with non-glob characters. This
  limits the number of loose references that need to be read when
  processing a command like `git for-each-ref refs/heads/`.

* Patch 23/23 makes `cache_ref_iterator` smarter about avoiding
  "priming" directories of loose references that won't be needed,
  further reducing the number of loose references that need to be read
  in some cases of iterating over references.

These changes are also available as branch `packed-ref-store-prep` in
my GitHub fork [2]. If you'd like to see a preview of the rest of the
changes (which works but is not yet polished), checkout the
`mmap-packed-refs` branch from the same place.

Michael

[1] http://public-inbox.org/git/cover.1490026594.git.mhagger@alum.mit.edu/
    http://public-inbox.org/git/cover.1490966385.git.mhagger@alum.mit.edu/
    http://public-inbox.org/git/cover.1492323985.git.mhagger@alum.mit.edu/

[2] https://github.com/mhagger/git

Jeff King (1):
  ref-filter: limit traversal to prefix

Michael Haggerty (22):
  t3600: clean up permissions test properly
  refs.h: clarify docstring for the ref_transaction_update()-related fns
  ref_iterator_begin_fn(): fix docstring
  prefix_ref_iterator: don't trim too much
  refs_ref_iterator_begin(): don't check prefixes redundantly
  refs: use `size_t` indexes when iterating over ref transaction updates
  ref_store: take `logmsg` parameter when deleting references
  lockfile: add a new method, is_lock_file_locked()
  files-backend: move `lock` member to `files_ref_store`
  files_ref_store: put the packed files lock directly in this struct
  files_transaction_cleanup(): new helper function
  ref_transaction_commit(): break into multiple functions
  ref_update_reject_duplicates(): expose function to whole refs module
  ref_update_reject_duplicates(): use `size_t` rather than `int`
  ref_update_reject_duplicates(): add a sanity check
  should_pack_ref(): new function, extracted from `files_pack_refs()`
  get_packed_ref_cache(): assume "packed-refs" won't change while locked
  read_packed_refs(): do more of the work of reading packed refs
  read_packed_refs(): report unexpected fopen() failures
  refs_ref_iterator_begin(): handle `GIT_REF_PARANOIA`
  create_ref_entry(): remove `check_name` option
  cache_ref_iterator_begin(): avoid priming unneeded directories

 builtin/fetch.c                |   2 +-
 builtin/remote.c               |   4 +-
 lockfile.h                     |   8 ++
 ref-filter.c                   |  62 ++++++++-
 refs.c                         |  83 ++++++++++--
 refs.h                         |  62 ++++++++-
 refs/files-backend.c           | 300 +++++++++++++++++++++++++----------------
 refs/iterator.c                |  14 +-
 refs/ref-cache.c               |  99 +++++++++++---
 refs/ref-cache.h               |   6 +-
 refs/refs-internal.h           |  56 ++++++--
 t/helper/test-ref-store.c      |   3 +-
 t/t1405-main-ref-store.sh      |   2 +-
 t/t1406-submodule-ref-store.sh |   2 +-
 t/t3600-rm.sh                  |   4 +-
 15 files changed, 538 insertions(+), 169 deletions(-)

-- 
2.11.0