git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: Taylor Blau <me@ttaylorr.com>
To: Jonathan Nieder <jrnieder@gmail.com>
Cc: Taylor Blau <me@ttaylorr.com>,
	Christian Couder <christian.couder@gmail.com>,
	Jeff King <peff@peff.net>, git <git@vger.kernel.org>,
	Christian Couder <chriscool@tuxfamily.org>,
	Johannes Schindelin <Johannes.Schindelin@gmx.de>
Subject: Re: Git in Outreachy?
Date: Wed, 9 Sep 2020 22:19:16 -0400	[thread overview]
Message-ID: <20200910021916.GA503@nand.local> (raw)
In-Reply-To: <20200910013915.GD667601@google.com>

On Wed, Sep 09, 2020 at 06:39:15PM -0700, Jonathan Nieder wrote:
> Taylor Blau wrote:
> In other words: this isn't about changing how tests are executed ---
> it's about going from patches on list to git commits that can be
> tested.

Fair enough; it sounds like a Snowpatch-like tool might be useful for
that case.

> [...]
> >> - formats: on-disk reverse idx
> >
> > As a heads up, I think that I am going to start working on (an
> > alternative to) this myself,
>
> Neat!  I look forward to seeing what you build.  (Is this related to
> the bitmaps-using-midx-ordering work?)

You guessed it ;-). It's sort of a tricky issue: the pack-objects code
needs to know things like the position of an object in index order so
that it can look at the offset of the next object and determine its
size.

But, in the multi-pack case, things aren't so simple. It would be nice
if we could look at a bit position and say "you are at bit position N
and there are M different packs before you, so I know what position
you are within your pack", but this can't be done in general when you
have duplicate objects among packs in your MIDX.

The MIDX is only going to choose one of them, and so you don't know how
many "holes" are present before you. In my development version, I have a
bitmap extension I'm calling the "rev-cache" which acts like a mapping
between bit position and MIDX name order (which can then be used to look
up auxiliary info like which pack an object came from, its position in
the pack's index order, and so on).

But, having an on-disk mapping between objects and their position in
different orderings will be useful not only for this, but for:

  - avoiding building a reverse index in memory from scratch (which can
    occupy quite a bit of the heap in some repositories that we see),
    and

  - dropping GitHub's custom "bit-cache" extension, which maps objects
    from name order to bit position.

Anyway, obviously not related to Outreachy, and obviously many more
details to come once I have patches that are more mature, but I couldn't
help myself in the meantime.

> Thanks,
> Jonathan
>
> >> [*] https://github.com/ruscur/snowpatch

Thanks,
Taylor

  reply	other threads:[~2020-09-10  2:21 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-28  6:56 Jeff King
2020-08-31  6:55 ` Christian Couder
2020-09-03  6:00   ` Jonathan Nieder
2020-09-04 14:14     ` Philip Oakley
2020-09-07 18:49       ` Johannes Schindelin
2020-09-16 15:16         ` Philip Oakley
2020-09-16 18:43           ` Johannes Schindelin
2020-09-17 14:42             ` Philip Oakley
2020-09-09 18:26     ` Taylor Blau
2020-09-10  1:39       ` Jonathan Nieder
2020-09-10  2:19         ` Taylor Blau [this message]
2020-09-16  9:12     ` Christian Couder
2020-09-16  6:42   ` Christian Couder
2020-08-31 17:41 ` Junio C Hamano
2020-08-31 18:05 ` Emily Shaffer
2020-09-01 12:51   ` Jeff King
2020-09-03  5:41     ` Jeff King
2020-09-15 17:35       ` Jeff King
2020-09-15 17:55         ` Kaartic Sivaraam
2020-09-15 18:02           ` Jeff King
2020-09-19  8:12         ` Christian Couder
2020-09-19 15:10           ` Phillip Wood
2020-09-16  8:45     ` Christian Couder
2020-09-02  4:00 ` Johannes Schindelin
2020-09-16  9:01   ` Christian Couder
2020-09-16  9:45     ` Phillip Wood
2020-09-17  9:43     ` Christian Couder
2020-09-17 10:14       ` Phillip Wood
2020-09-18  8:37         ` Christian Couder
2020-09-17 15:34       ` Elijah Newren
2020-09-18  8:42         ` Christian Couder
2020-09-27 16:59     ` Kaartic Sivaraam
2020-09-27 21:16       ` Christian Couder
2020-10-29 10:13         ` Christian Couder
2020-09-06 18:56 ` Kaartic Sivaraam
2020-09-07 18:55   ` Johannes Schindelin
2020-09-16  9:35     ` Christian Couder
2020-09-16 20:27       ` Johannes Schindelin
2020-09-19  7:40         ` Christian Couder
2020-09-20 15:06           ` Johannes Schindelin
2020-09-20 16:31   ` Kaartic Sivaraam
2020-09-21  4:22     ` Christian Couder
2020-09-21  7:59       ` Kaartic Sivaraam
2020-09-21 20:56       ` Shourya Shukla

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200910021916.GA503@nand.local \
    --to=me@ttaylorr.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=chriscool@tuxfamily.org \
    --cc=christian.couder@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=jrnieder@gmail.com \
    --cc=peff@peff.net \
    --subject='Re: Git in Outreachy?' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.version-control.git
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.version-control.git
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git