Re: RFC: monthly epochs for v2

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hi, all:
> 
> With public-inbox now providing manifest files, it is easy to communicate to
> mirroring services when an epoch rolls over. What do you think if we make
> these roll-overs month-based instead of size-based. So, instead of:
> 
> git/
>  0.git
>  1.git
>  2.git
> 
> it becomes
> 
> git/
>  201908.git
>  201909.git
>  201910.git
> 
> Upsides:
> 
> - if history needs to be rewritten due to GDPR edits, the impact is  limited
> to just messages in that month's archive

Epoch size should be configurable, yes.  But I'm against time
periods such as months or years being a factor for rollover.
Many inboxes (including this one) can go idle for weeks/months;
and activity can be unpredictable if there's surges.

> - if someone is only interested in a few months worth of archives, they
> don't have to clone the entire collection
> - similarly, someone using public-inbox to feed messages to their inbox
> (e.g. using the l2md tool [1]) doesn't need to waste gigabytes storing
> archives they aren't interested in

NNTP or d:YYYYMMDD..YYYYMMDD mboxrd downloads via HTTP search
are better suited for those cases.

The HTTP search will be better once it can expand threads to
fetch messages in the same thread outside of date ranges
(e.g. "mairix -t").

The client side could still import into a local v2 inbox and use
it as a cache, and configure their epoch size and expiration logic.

> - since the numbers are always auto-incrementing, this change can even  be
> done to repos currently using number-based epoch rotation, e.g.:
> 
>  git/
>    0.git
>    1.git
>    201910.git
>    201911.git
> 
> - there shouldn't be severe directory listing penalties with this, as  even
> 20 years worth of archives will only have 240 entries

That would still increase overhead for cloning + fetching as far
as installing and running extra tools.

Aside from LKML, most inboxes are pretty small and shouldn't
require more than an initial clone and then fetch via cron.

If people only want a backup via git (and not host HTTP/NNTP),
it's FAR easier for them to run ubiquitous commands such as
"git clone --mirror && git fetch" rather than
"install $TOOL which may be out-of-date-or-missing-on-your-distro"