user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: meta@public-inbox.org
Subject: Limited-history local archives
Date: Fri, 3 Jan 2020 15:15:32 -0500
Message-ID: <20200103201532.gv4rdotwuiv7ieiy@chatter.i7.local> (raw)

Hi, all:

I wonder if it would be useful to have a feature allowing someone to run 
a limited-history local copy of a larger remote archive -- for example 
if someone only wanted a 3-month copy of LKML instead of the whole 
20-year enchilada.

It's possible to accomplish this with git already [^1], e.g. you can use 
the following to grab a copy of LKML starting with December 2019:

  $ git clone --bare --shallow-since 2019-12-01 https://lore.kernel.org/lkml/git/7 lkml-since-dec.git
  $ cd lkml-since-dec.git
  $ git config --add remote.origin.fetch '+refs/heads/master:refs/heads/master'

You can now run "git fetch" as usual and perform all the normal 
operations, such as "git show {rev}:m" to get the message contents.  
Obviously, if we try to get a revision from before December 1, the 
operation fails:

  $ git show dae740ca679710fbe8b97b3e704d63e3e7883fd9:m
  fatal: Path 'm' does not exist in 'dae740ca679710fbe8b97b3e704d63e3e7883fd9'

If we enable uploadpack.allowAnySHA1InWant on the server, we can then 
fetch this object directly:

  $ git fetch --depth 1 origin dae740ca679710fbe8b97b3e704d63e3e7883fd9
  remote: Counting objects: 3, done.
  remote: Compressing objects: 100% (2/2), done.
  remote: Total 3 (delta 0), reused 3 (delta 0)
  Unpacking objects: 100% (3/3), done.
  From https://lore.kernel.org/lkml/git/7
   * branch              dae740ca679710fbe8b97b3e704d63e3e7883fd9 -> FETCH_HEAD

Now this succeeds:

  $ git show dae740ca679710fbe8b97b3e704d63e3e7883fd9:m

We can then periodically reshallow the archive (e.g. once a day) in 
order to get rid of older objects:

  $ git fetch --shallow-since 2019-12-15 --update-shallow origin master
  $ git gc --prune=now

There isn't really an RFC or anything associated with this -- I just 
wanted to share this idea as a possibly useful way of reducing local 
storage requirements while still being able to operate directly on 
public-inbox git repositories -- e.g. with a tool like l2md 
(https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/).

-K

[^1]: Theoretically, this will become even easier in the future with 
      partial-clone functionality, though I believe that's mostly 
      written to support fetching large blobs from CDNs and wouldn't be 
      as useful for very linear public-inbox repositories.

             reply index

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-03 20:15 Konstantin Ryabitsev [this message]
2020-01-03 22:02 ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200103201532.gv4rdotwuiv7ieiy@chatter.i7.local \
    --to=konstantin@linuxfoundation.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git