about summary refs log tree commit homepage
DateCommit message (Collapse)
2019-06-16xcpdb: don't warn on --jobs != --reshard
It's slightly confusing since we dedicate one job to dealing with fast-import + SQLite indexing; and it's not worth complaining about when it happens.
2019-06-16Merge remote-tracking branch 'origin/newspeak' into xcpdb
* origin/newspeak: comments: replace "partition" with "shard" t/xcpdb-reshard: use 'shard' term in local variables xapcmd: favor 'shard' over 'part' in local variables search: use "shard" for local variable v2writable: use "epoch" consistently when referring to git repos adminedit: "part" => "shard" for local variables v2writable: rename local vars to match Xapian terminology v2writable: avoid "part" in internal subs and fields search*: rename {partition} => {shard} xapcmd: update comments referencing "partitions" v2: rename SearchIdxPart => SearchIdxShard inboxwritable: s/partitions/shards/ in local var tests: change messages to use "shard" instead of partition v2writable: rename {partitions} field to {shards} v2writable: count_partitions => count_shards searchidxpart: start using "shard" in user-visible places rename reference to git epochs as "partitions" admin|xapcmd: user-facing messages say "shard" v2writable: update comments regarding xcpdb --reshard doc: rename our Xapian "partitions" to "shards"
2019-06-16t/psgi_search.t: use higher-level APIs
No point in using lower-level APIs for a PSGI test.
2019-06-15searchview: add link at bottom to reverse results
I could not find a place to put the link the top without making navigation too cluttered. Putting it at the bottom of the page seems reasonable...
2019-06-15searchview: support negative offsets to reverse ordering
Taking a hint from Perl array access, we'll allow negative offsets for the 'o' parameter and to reverse the sort order.
2019-06-15Merge remote-tracking branch 'origin/ds'
* origin/ds: ds: stop caring about event flags set by epoll/poll/kqueue ds: do not distinguish between POLLHUP and POLLERR ds: remove read method, here, too nntp: use sysread to append to existing buffer ds: remove steal_socket method ds: remove {fd} field ds: reduce Errno imports and drop ->close reason ds: cleanup Errno imports and favor constant comparisons ds: simplify write buffer accounting
2019-06-15t/git-http-backend: explain purpose of test
I found myself tempted to switch to HTTP::Tiny, here, since it's distributed with Perl since 5.14, unlike Net::HTTP (which AFAIK was never a part of Perl proper). But we really want to use Net::HTTP, here, since it's lower-level and allows us to trigger server-side buffering by not reading the entity body.
2019-06-15comments: replace "partition" with "shard"
Now that the code matches Xapian terminology, ensure our comments match, too.
2019-06-14t/xcpdb-reshard: use 'shard' term in local variables
Another step in maintaining consistency with Xapian docs.
2019-06-14xapcmd: favor 'shard' over 'part' in local variables
Yet another step to keeping our naming consistent with Xapian terminology.
2019-06-14search: use "shard" for local variable
Another small step towards terminology consistency with Xapian.
2019-06-14v2writable: use "epoch" consistently when referring to git repos
Be consistent with our own terminology and use "epoch" for [0-9]+\.git repos. The term "partition" is going away entirely.
2019-06-14adminedit: "part" => "shard" for local variables
2019-06-14v2writable: rename local vars to match Xapian terminology
2019-06-14v2writable: avoid "part" in internal subs and fields
We'll be using the term "shard" from now on to be consistent with Xapian terminology.
2019-06-14search*: rename {partition} => {shard}
Another step towards keeping our internal data structures consistent with Xapian naming.
2019-06-14xapcmd: update comments referencing "partitions"
Don't confuse future readers of our code.
2019-06-14v2: rename SearchIdxPart => SearchIdxShard
Another step towards keeping our file and package names consistent with Xapian terminology.
2019-06-14inboxwritable: s/partitions/shards/ in local var
More work towards being consistent with Xapian's own terminology
2019-06-14tests: change messages to use "shard" instead of partition
Another potentially user-facing piece made consistent with Xapian terminology.
2019-06-14v2writable: rename {partitions} field to {shards}
Our internal data structure should be consistent with Xapian terminology.
2019-06-14v2writable: count_partitions => count_shards
Another step towards becoming consistent with Xapian terminology
2019-06-14searchidxpart: start using "shard" in user-visible places
We'll name our process title with "shard" instead, and update a few error messages and comments to match.
2019-06-14rename reference to git epochs as "partitions"
Try to remain consistent with our own documentation regarding v2 git "epochs", first.
2019-06-14admin|xapcmd: user-facing messages say "shard"
We're slowly getting rid of the word "partition" when it comes to remain consistent with Xapian docs.
2019-06-14v2writable: update comments regarding xcpdb --reshard
Using compact to change shard count was abandoned during the v2 development phase.
2019-06-14doc: rename our Xapian "partitions" to "shards"
For consistency with Xapian documentation (in the "master" branch).
2019-06-14v2writable: fix brainfart when capping NPROC_MAX_DEFAULT
Oops :x
2019-06-14search: require PublicInbox::Inbox ref here
No sense in supporting multiple methods of initialization for an internal class.
2019-06-14searchidx: require PublicInbox::Inbox (or InboxWritable) ref
PublicInbox::Inbox objects have minimal dependencies, so drop code to support old tests which existed before the PublicInbox::Inbox object came into existence.
2019-06-14Makefile.PL: add test scripts to syntax checks
I make syntax errors all the time :x
2019-06-14t/www_listing: favor HTTP::Tiny over Net::HTTP
More testers are likely to have HTTP::Tiny than Net::HTTP, since HTTP::Tiny is a dual-life module and distributed with Perl since Perl 5.14 (2011-05-14), whereas Net::HTTP will likely live in a separate package forever.
2019-06-14edit: fix portability of editor invocation
The eval was unnecessary, and $0 can't be "--". Tested with /bin/sh on FreeBSD 11.2
2019-06-14Merge remote-tracking branch 'origin/reshard' into next
* origin/reshard: xcpdb: support resharding v2 repos xcpdb: use destination shard as progress prefix xapcmd: preserve indexlevel based on the destination v2writable: use a smaller default for Xapian partitions
2019-06-14Merge remote-tracking branch 'origin/manifest' into next
* origin/manifest: git: ensure ->modified returns an integer www: support $INBOX/git/$EPOCH.git for v2 cloning www: wire up /$INBOX/manifest.js.gz, too wwwlisting: generate grokmirror-compatible manifest.js.gz wwwlisting: allow hiding entries from manifest
2019-06-14Merge remote-tracking branch 'origin/edit' into next
* origin/edit: edit: unlink temporary file when done v2writable: replace: kill git processes before reindexing edit: drop unwanted headers before noop check edit|purge: improve output on rewrites edit: new tool to perform edits doc: document the --prune option for -index admin: expose ->config AdminEdit: move editability checks from -purge admin: beef up resolve_inboxes to handle purge options purge: start moving common options to AdminEdit module admin: remove warning arg for unconfigured inboxes v2writable: implement ->replace call import: switch to "replace_oids" interface for purge import: extract_author_info becomes extract_commit_info v2writable: consolidate overview and indexing call
2019-06-14xcpdb: support resharding v2 repos
v2 repos are sometimes created on machines where CPU parallelization exceeds the capability of the storage devices. In that case, users may reshard the Xapian DB to any smaller, positive integer to avoid excessive overhead and contention when bottlenecked by slow storage. Resharding can also be used to increase shard count after hardware upgrades.
2019-06-14xcpdb: use destination shard as progress prefix
For M:N resharding, we'll want to display the number from the new shard number.
2019-06-14xapcmd: preserve indexlevel based on the destination
To support M:N resharding, we need to ensure we store the indexlevel in the destination shard, rather than the originating one.
2019-06-14v2writable: use a smaller default for Xapian partitions
Apparently 16 CPUs (probably HT) and SATA storage is common these days. Having excessive Xapian partitions leads to contention and excessive FD/space use. So set a smaller default but continue allowing user-specified values to bump this up.
2019-06-14git: remove cat_file sub callback interface
We weren't using it, and in retrospect, it makes no sense to use this API cat_file for giant responses which can't read quickly with minimal context-switching (or sanely fit into memory for Email::Simple/Email::MIME). For giant blobs which we don't want slurped in memory, we'll spawn a short-lived git-cat-file process like we do in ViewVCS. Otherwise, monopolizing a git-cat-file process for a giant blob is harmful to other PSGI/NNTP users. A better interface is coming which will be more suitable for for batch processing of "small" objects such as commits and email blobs.
2019-06-14nntp: filter out duplicate Message-IDs for leafnode
It's the unfortunate reality that there are some clients which reuse Message-IDs (in which we generate + use another) or set multiple Message-IDs on their own. While the v2 format addresses that, NNTP clients such as leafnode are not always prepared to deal with that case. So, ensure NNTP clients only see a single Message-ID, and show the others as 'X-Alt-Message-ID'.
2019-06-13nntp: ensure Message-ID is not folded for leafnode
Leafnode cannot handle Message-ID headers which are too long and require folding via Email::Simple::Header. Since there are already many of these messages in git with the header already folded, we need to handle the unfolding when emitting the message via NNTP. As far as we know, Leafnode is the only client software incapable of handling this case.
2019-06-13nntp: add Path: header for leafnode
Apparently leafnode just needs any junk in the Path: header. Lets not waste bandwidth and just use a single byte to keep leafnode happy. Cc: Dave Taht <dave@taht.net>
2019-06-13Makefile.PL: "dsyn" target to check syntax of changed files
We have lots of files and syntax-checking every single one of them is slow. Enable "perl -w" in the existing syntax check while we're at it.
2019-06-13doc: update dependencies for CentOS 7.x
Digest::SHA is the most notable missing package at runtime for a minimal system. Tests don't run at all without Test::Simple (or Test::More). Plack::Test is also a separate package, too... Also, the package for IO::Compress::Gzip should be IO::Compress; as perl-PerlIO-gzip is a different thing entirely which is not relevant to our needs. Test::HTTP::Server::Simple doesn't seem required at all for Plack tests. ExtUtils::MakeMaker needs to be documented as a install dependency for people installing this, too; since AFAIK public-inbox is not yet in any distros.
2019-06-13t/common.perl: fix error message for git requirements
And enable strict + warnings in the scope of t/common.perl, too.
2019-06-13searchmsg: remove unused ->get subroutine
It's obsolete and unusable since our search schema version 15; which made the Xapian document ID correspond to the NNTP article number.
2019-06-12searchidx: improve error message when Xapian fails
Make it easier to detect if a partition is corrupt.
2019-06-12edit: unlink temporary file when done
We don't need to leave temporary files lying around.