git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Michael Haggerty <mhagger@alum.mit.edu>
To: Junio C Hamano <gitster@pobox.com>
Cc: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
	"Stefan Beller" <sbeller@google.com>, "Jeff King" <peff@peff.net>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"David Turner" <novalis@novalis.org>,
	"Brandon Williams" <bmwill@google.com>,
	git@vger.kernel.org, "Michael Haggerty" <mhagger@alum.mit.edu>
Subject: [PATCH 00/28] Create a reference backend for packed refs
Date: Thu, 15 Jun 2017 16:47:05 +0200	[thread overview]
Message-ID: <cover.1497534157.git.mhagger@alum.mit.edu> (raw)

This patch series continues the saga of picking apart the code for
handling packed references from the code for handling loose
references, all in preparation for making big changes to how the
packed-ref reading and writing works as described in [1]. As a
reminder, the final goal is to read the "packed-refs" file using mmap,
parsing it on the fly instead of storing it into an in-memory
`ref_cache`, and to read and parse only the parts of the file that are
actually needed, giving a big speedup for many operations in
repositories that have lots of refs.

In this episode, we create a `packed_ref_store` class, implementing
part of the `ref_store` API, that represents the packed references
within a repository. The `files_ref_store` now contains an instance of
`packed_ref_store` and delegates to it for the operations that have to
touch the packed refs.

After this patch series, `packed_ref_store` supports:

* Iteration
* `peel_ref`
* `pack_refs` (they're already packed, so it's a NOOP)
* `read_raw_ref`

A future patch series will add support for:

* Reference transactions (`transaction_prepare`, `transaction_finish`,
  `transaction_abort`, `initial_transaction_commit`)
* `delete_refs`

Operations that `packed_ref_store` will probably never support:

* `create_symref`
* `rename_ref` (could be supported, but is probably not useful)
* Reflog-related operations

In addition, all of the packed-refs related code has been moved to a
new module, "refs/packed-backend.{c,h}". This includes some functions
that are still called by `files_ref_store` directly to update the
packed refs.

The patch series is long, but I think relatively straightforward. In
particular, patches 2-14 are quite mechanical. Its main point is to
separate concerns, but it does bring one end-user advantage: if there
is a problem parsing the "packed-refs" file, we now report an error
and die. The old code just ignored lines that it didn't understand.

I've developed these patches on top of master plus the following
patches, which are followups to mh/packed-refs-store-prep:

* lock_packed_refs(): fix cache validity check
* for_each_bisect_ref(): don't trim refnames

The patches can also be obtained from my GitHub fork [2] as branch
"packed-ref-store".

Michael

[1] http://public-inbox.org/git/cover.1490026594.git.mhagger@alum.mit.edu/
[2] https://github.com/mhagger/git

Michael Haggerty (28):
  add_packed_ref(): teach function to overwrite existing refs
  packed_ref_store: new struct
  packed_ref_store: move `packed_refs_path` here
  packed_ref_store: move `packed_refs_lock` member here
  clear_packed_ref_cache(): take a `packed_ref_store *` parameter
  validate_packed_ref_cache(): take a `packed_ref_store *` parameter
  get_packed_ref_cache(): take a `packed_ref_store *` parameter
  get_packed_refs(): take a `packed_ref_store *` parameter
  add_packed_ref(): take a `packed_ref_store *` parameter
  lock_packed_refs(): take a `packed_ref_store *` parameter
  commit_packed_refs(): take a `packed_ref_store *` parameter
  rollback_packed_refs(): take a `packed_ref_store *` parameter
  get_packed_ref(): take a `packed_ref_store *` parameter
  repack_without_refs(): take a `packed_ref_store *` parameter
  packed_peel_ref(): new function, extracted from `files_peel_ref()`
  packed_ref_store: support iteration
  packed_read_raw_ref(): new function, replacing `resolve_packed_ref()`
  packed-backend: new module for handling packed references
  packed_ref_store: make class into a subclass of `ref_store`
  commit_packed_refs(): report errors rather than dying
  commit_packed_refs(): use a staging file separate from the lockfile
  packed_refs_lock(): function renamed from lock_packed_refs()
  packed_refs_lock(): report errors via a `struct strbuf *err`
  packed_refs_unlock(), packed_refs_is_locked(): new functions
  clear_packed_ref_cache(): don't protest if the lock is held
  commit_packed_refs(): remove call to `packed_refs_unlock()`
  repack_without_refs(): don't lock or unlock the packed refs
  read_packed_refs(): die if `packed-refs` contains bogus data

 Makefile              |   1 +
 refs.c                |  18 ++
 refs/files-backend.c  | 619 ++++-------------------------------
 refs/packed-backend.c | 868 ++++++++++++++++++++++++++++++++++++++++++++++++++
 refs/packed-backend.h |  25 ++
 refs/refs-internal.h  |  10 +
 6 files changed, 981 insertions(+), 560 deletions(-)
 create mode 100644 refs/packed-backend.c
 create mode 100644 refs/packed-backend.h

-- 
2.11.0


             reply	other threads:[~2017-06-15 14:47 UTC|newest]

Thread overview: 52+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-06-15 14:47 Michael Haggerty [this message]
2017-06-15 14:47 ` [PATCH 01/28] add_packed_ref(): teach function to overwrite existing refs Michael Haggerty
2017-06-15 14:47 ` [PATCH 02/28] packed_ref_store: new struct Michael Haggerty
2017-06-15 14:47 ` [PATCH 03/28] packed_ref_store: move `packed_refs_path` here Michael Haggerty
2017-06-16  5:30   ` Stefan Beller
2017-06-15 14:47 ` [PATCH 04/28] packed_ref_store: move `packed_refs_lock` member here Michael Haggerty
2017-06-16  5:39   ` Stefan Beller
2017-06-16  6:43     ` Michael Haggerty
2017-06-16 15:44       ` Stefan Beller
2017-06-15 14:47 ` [PATCH 05/28] clear_packed_ref_cache(): take a `packed_ref_store *` parameter Michael Haggerty
2017-06-15 14:47 ` [PATCH 06/28] validate_packed_ref_cache(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 07/28] get_packed_ref_cache(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 08/28] get_packed_refs(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 09/28] add_packed_ref(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 10/28] lock_packed_refs(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 11/28] commit_packed_refs(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 12/28] rollback_packed_refs(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 13/28] get_packed_ref(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 14/28] repack_without_refs(): " Michael Haggerty
2017-06-15 14:47 ` [PATCH 15/28] packed_peel_ref(): new function, extracted from `files_peel_ref()` Michael Haggerty
2017-06-16  5:42   ` Stefan Beller
2017-06-16  6:46     ` Michael Haggerty
2017-06-16 15:47       ` Stefan Beller
2017-06-15 14:47 ` [PATCH 16/28] packed_ref_store: support iteration Michael Haggerty
2017-06-15 14:47 ` [PATCH 17/28] packed_read_raw_ref(): new function, replacing `resolve_packed_ref()` Michael Haggerty
2017-06-15 14:47 ` [PATCH 18/28] packed-backend: new module for handling packed references Michael Haggerty
2017-06-16  5:48   ` Stefan Beller
2017-06-15 14:47 ` [PATCH 19/28] packed_ref_store: make class into a subclass of `ref_store` Michael Haggerty
2017-06-15 14:47 ` [PATCH 20/28] commit_packed_refs(): report errors rather than dying Michael Haggerty
2017-06-15 14:47 ` [PATCH 21/28] commit_packed_refs(): use a staging file separate from the lockfile Michael Haggerty
2017-07-20 23:11   ` Junio C Hamano
2017-07-20 23:30     ` Jonathan Nieder
2017-07-21  6:01       ` Junio C Hamano
2017-07-21  7:14         ` Junio C Hamano
2017-07-24 16:59         ` Jeff King
2017-07-24 17:09           ` Jonathan Nieder
2017-07-24 17:10             ` Jeff King
2017-07-24 21:43               ` Junio C Hamano
2017-07-26 23:49                 ` Michael Haggerty
2017-06-15 14:47 ` [PATCH 22/28] packed_refs_lock(): function renamed from lock_packed_refs() Michael Haggerty
2017-06-15 14:47 ` [PATCH 23/28] packed_refs_lock(): report errors via a `struct strbuf *err` Michael Haggerty
2017-06-15 14:47 ` [PATCH 24/28] packed_refs_unlock(), packed_refs_is_locked(): new functions Michael Haggerty
2017-06-15 14:47 ` [PATCH 25/28] clear_packed_ref_cache(): don't protest if the lock is held Michael Haggerty
2017-06-15 14:47 ` [PATCH 26/28] commit_packed_refs(): remove call to `packed_refs_unlock()` Michael Haggerty
2017-06-15 14:47 ` [PATCH 27/28] repack_without_refs(): don't lock or unlock the packed refs Michael Haggerty
2017-06-15 14:47 ` [PATCH 28/28] read_packed_refs(): die if `packed-refs` contains bogus data Michael Haggerty
2017-06-19 18:51 ` [PATCH 00/28] Create a reference backend for packed refs Junio C Hamano
2017-06-19 19:25   ` Junio C Hamano
2017-06-19 19:43     ` Jeff King
2017-06-19 19:53       ` Jeff King
2017-06-22  7:36         ` Michael Haggerty
2017-06-22 15:33           ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1497534157.git.mhagger@alum.mit.edu \
    --to=mhagger@alum.mit.edu \
    --cc=avarab@gmail.com \
    --cc=bmwill@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=novalis@novalis.org \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --cc=sbeller@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).