about summary refs log tree commit homepage
path: root/script/public-inbox-init
DateCommit message (Collapse)
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-01-27init: use Import::run_die instead of system()
We already load PublicInbox::Import via PublicInbox::InboxWritable, so it's not an extra module to load. This can give us a slight speedup in tests.
2020-01-06treewide: "require" + "use" cleanup and docs
There's a bunch of leftover "require" and "use" statements we no longer need and can get rid of, along with some excessive imports via "use". IO::Handle usage isn't always obvious, so add comments describing why a package loads it. Along the same lines, document the tmpdir support as the reason we depend on File::Temp 0.19, even though every Perl 5.10.1+ user has it. While we're at it, favor "use" over "require", since it it gives us extra compile-time checking.
2019-11-16init: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub. We also need to rely on ->DESTROY instead of END{} to unlink the lock file on sub exit.
2019-10-16config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I intended for there to be a "spamrepo" (now just the ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be especially confusing, since v2 needs at least two git repositories (epoch + all.git) to function and we shouldn't confuse users by having them point to a git repository for v2. Much of our documentation already references "INBOX_DIR" for command-line arguments, so use "inboxdir" as the git-config(1)-friendly variant for that. "mainrepo" remains supported indefinitely for compatibility. Users may need to revert to old versions, or may be referring to old documentation and must not be forced to change config files to account for this change. So if you're using "mainrepo" today, I do NOT recommend changing it right away because other bugs can lurk. Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
2019-10-05init: implement locking
First, we use flock(2) to wait on parallel public-inbox-init(1) invocations while we make multiple changes using git-config(1). This flock allows -init processes to wait on each other if using reasonable POSIX filesystems. Then, we also need a git-config(1)-compatible lock to prevent user-invoked git-config(1) processes from clobbering our changes while we're holding the flock.
2019-10-05init: favor --skip-epoch instead of --skip
Since I intend to add support for --skip-artnum, disambiguating the long option name makes sense. We'll support --skip indefinitely for compatibility.
2019-09-09run update-copyrights from gnulib for 2019
2019-05-23doc: various updates to reflect current state
-index documentation avoid redundant v1 information and refers readers to apropriate v1/v2 manpages. Search::Xapian can also be optional, now, as only the PSGI search interface uses it. Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be confused for code repos which we also support. XAPIAN_FLUSH_THRESHOLD is documented for all relevant bulk commands.
2019-05-23v1writable: retire in favor of InboxWritable
In retrospect, introducing V1Writable was unnecessary and InboxWritable->importer is in a better position to abstract away differences between v1 and v2 writers. So teach InboxWritable to initialize inboxes and get rid of V1Writable.
2019-05-22init: preserve permissions for git prior to 2.1.0
"git config" did not preserve permissions of the config file it modifies prior to git 2.1.0, so workaround that.
2019-05-15admin: improve warnings and errors for missing modules
Since we lazy-load Xapian now, some errors may become more cryptic or buried. Try to improve that by making Admin show better errors.
2019-05-15lazy load Xapian and make it optional for v2
More tests work without Search::Xapian, now. Usability issues still need to be fixed
2019-05-14v1writable: new wrapper which is closer to v2writable
Import initialization is a little strange from history, but we also can't change it too much because it's technically a public API which external code may rely on... And we may need to support v1 repos indefinitely. This should make it easier to write tests for both formats.
2018-12-28init: allow --skip of old epochs for -V2 repos
This allows archivists to publish incomplete archives with newer mail while allowing "0.git" (or "1.git" and so on) epochs to be added-after-the-fact (without affecting "git clone" followers). A reindex will be necessary for Xapian and SQLite to catch up once the old epochs are added; but the reindexing code is also capable of tolerating missing epochs.
2018-12-27init: do not set publicinbox.$NAME.indexlevel by default
It is redundant to set default values in the public-inbox config file. Lets not clutter up users' screens when they view or edit the config file.
2018-07-19public-inbox-init: Initialize indexlevel
If indexlevel is specified on the command line prefer that. If indexlevel is specified in the config file prefer that. If indexlevel is not specified anywhere default to full. This should make indexlevel somewhat approachable. Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2018-04-04init: s/GIT_DIR/REPO_DIR/ in usage
2018-03-29v2writable: initializing an existing inbox is idempotent
And we do not want to start making confused repos if somebody leaves out "-V2" the second time around.
2018-03-20InboxWritable: add mbox/maildir parsing + import logic
This will make it easier to as well as supporting future Filter API users. It allows simplifying our ad-hoc import_vger_from_mbox script.
2018-03-19v2writable: allow disabling parallelization
While parallel processes improves import speed for initial imports; they are probably not necessary for daily mail imports via WatchMaildir and certainly not for public-inbox-init. Save some memory for daily use and even helps improve readability of some subroutines by showing which methods they call remotely.
2018-02-28v2/ui: get nntpd and init tests running on v2
A work-in-progress, but it appears the v2 UI pieces do will not require a lot of work to do.
2018-02-07update copyrights for 2018
Using update-copyrights from gnulib While we're at it, use the SPDX identifier for AGPL-3.0+ to ease mechanical processing.
2016-12-12init: preserve permissions of existing config file
This matches git-config(1) behavior, and implied user intent when it comes to programatically editing files.
2016-09-02init: enable pack bitmaps by default
We want to encourage users to serve repositories. So enable bitmaps by default so performance suffers less with smart HTTP.
2016-05-16declare Inbox object for reusability
From the beginning, we've avoided objects here in favor of faster startup time; but it may not be worth it since a persistent httpd/nntpd is faster and -mda isn't hit as often.
2016-05-14rename most instances of "list" to "inbox"
A public-inbox is NOT necessarily a mailing list, but it could serve as an input point for zero, one, or infinite mailing lists :D
2016-02-27move executables to script/ directory
This seems to match more closely with what is expected of Perl packages based on how blib is used. Hopefully makes the top-level source tree less cluttered and things easier-to-find.