about summary refs log tree commit homepage
DateCommit message (Collapse)
2019-06-24AddTimer: avoid clock_gettime for the '0' case
We rely on immediate timers often, so we can avoid the overhead of an extra subroutine call to retrieve the monotonic time (and a sometimes-system call on some platforms).
2019-06-24ds: use and export monotonic now()
All of our internal timing code should use monotonic clocks for consistency against system clock adjustments. This can be shared by our Daemon and NNTP packages.
2019-06-24ds: get rid of more unused debug instance methods
Over a decade of using Danga::Socket and I never found the built-in debug functionality useful.
2019-06-24ds: get rid of {closed} field
Merely checking the presence of the {sock} field is enough, and having multiple sources of truth increases confusion and the likelyhood of bugs.
2019-06-23manifest: v2 epoch descriptions based on inbox->description
The default $GIT_DIR/description (provided by git.git templates) isn't very useful for v2 epochs, so use the inbox description and suffix it with the epoch number if it's otherwise unnamed. Requested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> https://public-inbox.org/meta/20190620190017.GA27175@chatter.i7.local/
2019-06-20t/httpd-corner: ensure chunk payload read doesn't overreach
It never has, AFAIK, but I'm making some changes to this code in another branch and nearly introduced a bug where it would be overreading and discarding the pipelined request.
2019-06-20t/httpd-corner.t: fix braino :x
Plack is for Perl, Rack is for Ruby; this a Perl project :x
2019-06-16ds: stop distinguishing event read and write callbacks
Having separate read/write callbacks in every class is too confusing to my easily-confused mind. Instead, give every class an "event_step" callback which is easier to wrap my head around. This will make future code to support IO::Socket::SSL-wrapped sockets easier-to-digest, since SSL_write() can require waiting on POLLIN events, and SSL_read() can require waiting on POLLOUT events.
2019-06-16t/replace.t: fix SKIP label for testing w/o Xapian
2019-06-16xcpdb: don't warn on --jobs != --reshard
It's slightly confusing since we dedicate one job to dealing with fast-import + SQLite indexing; and it's not worth complaining about when it happens.
2019-06-16Merge remote-tracking branch 'origin/newspeak' into xcpdb
* origin/newspeak: comments: replace "partition" with "shard" t/xcpdb-reshard: use 'shard' term in local variables xapcmd: favor 'shard' over 'part' in local variables search: use "shard" for local variable v2writable: use "epoch" consistently when referring to git repos adminedit: "part" => "shard" for local variables v2writable: rename local vars to match Xapian terminology v2writable: avoid "part" in internal subs and fields search*: rename {partition} => {shard} xapcmd: update comments referencing "partitions" v2: rename SearchIdxPart => SearchIdxShard inboxwritable: s/partitions/shards/ in local var tests: change messages to use "shard" instead of partition v2writable: rename {partitions} field to {shards} v2writable: count_partitions => count_shards searchidxpart: start using "shard" in user-visible places rename reference to git epochs as "partitions" admin|xapcmd: user-facing messages say "shard" v2writable: update comments regarding xcpdb --reshard doc: rename our Xapian "partitions" to "shards"
2019-06-16t/psgi_search.t: use higher-level APIs
No point in using lower-level APIs for a PSGI test.
2019-06-15searchview: add link at bottom to reverse results
I could not find a place to put the link the top without making navigation too cluttered. Putting it at the bottom of the page seems reasonable...
2019-06-15searchview: support negative offsets to reverse ordering
Taking a hint from Perl array access, we'll allow negative offsets for the 'o' parameter and to reverse the sort order.
2019-06-15Merge remote-tracking branch 'origin/ds'
* origin/ds: ds: stop caring about event flags set by epoll/poll/kqueue ds: do not distinguish between POLLHUP and POLLERR ds: remove read method, here, too nntp: use sysread to append to existing buffer ds: remove steal_socket method ds: remove {fd} field ds: reduce Errno imports and drop ->close reason ds: cleanup Errno imports and favor constant comparisons ds: simplify write buffer accounting
2019-06-15t/git-http-backend: explain purpose of test
I found myself tempted to switch to HTTP::Tiny, here, since it's distributed with Perl since 5.14, unlike Net::HTTP (which AFAIK was never a part of Perl proper). But we really want to use Net::HTTP, here, since it's lower-level and allows us to trigger server-side buffering by not reading the entity body.
2019-06-15comments: replace "partition" with "shard"
Now that the code matches Xapian terminology, ensure our comments match, too.
2019-06-14t/xcpdb-reshard: use 'shard' term in local variables
Another step in maintaining consistency with Xapian docs.
2019-06-14xapcmd: favor 'shard' over 'part' in local variables
Yet another step to keeping our naming consistent with Xapian terminology.
2019-06-14search: use "shard" for local variable
Another small step towards terminology consistency with Xapian.
2019-06-14v2writable: use "epoch" consistently when referring to git repos
Be consistent with our own terminology and use "epoch" for [0-9]+\.git repos. The term "partition" is going away entirely.
2019-06-14adminedit: "part" => "shard" for local variables
2019-06-14v2writable: rename local vars to match Xapian terminology
2019-06-14v2writable: avoid "part" in internal subs and fields
We'll be using the term "shard" from now on to be consistent with Xapian terminology.
2019-06-14search*: rename {partition} => {shard}
Another step towards keeping our internal data structures consistent with Xapian naming.
2019-06-14xapcmd: update comments referencing "partitions"
Don't confuse future readers of our code.
2019-06-14v2: rename SearchIdxPart => SearchIdxShard
Another step towards keeping our file and package names consistent with Xapian terminology.
2019-06-14inboxwritable: s/partitions/shards/ in local var
More work towards being consistent with Xapian's own terminology
2019-06-14tests: change messages to use "shard" instead of partition
Another potentially user-facing piece made consistent with Xapian terminology.
2019-06-14v2writable: rename {partitions} field to {shards}
Our internal data structure should be consistent with Xapian terminology.
2019-06-14v2writable: count_partitions => count_shards
Another step towards becoming consistent with Xapian terminology
2019-06-14searchidxpart: start using "shard" in user-visible places
We'll name our process title with "shard" instead, and update a few error messages and comments to match.
2019-06-14rename reference to git epochs as "partitions"
Try to remain consistent with our own documentation regarding v2 git "epochs", first.
2019-06-14admin|xapcmd: user-facing messages say "shard"
We're slowly getting rid of the word "partition" when it comes to remain consistent with Xapian docs.
2019-06-14v2writable: update comments regarding xcpdb --reshard
Using compact to change shard count was abandoned during the v2 development phase.
2019-06-14doc: rename our Xapian "partitions" to "shards"
For consistency with Xapian documentation (in the "master" branch).
2019-06-14v2writable: fix brainfart when capping NPROC_MAX_DEFAULT
Oops :x
2019-06-14search: require PublicInbox::Inbox ref here
No sense in supporting multiple methods of initialization for an internal class.
2019-06-14searchidx: require PublicInbox::Inbox (or InboxWritable) ref
PublicInbox::Inbox objects have minimal dependencies, so drop code to support old tests which existed before the PublicInbox::Inbox object came into existence.
2019-06-14Makefile.PL: add test scripts to syntax checks
I make syntax errors all the time :x
2019-06-14t/www_listing: favor HTTP::Tiny over Net::HTTP
More testers are likely to have HTTP::Tiny than Net::HTTP, since HTTP::Tiny is a dual-life module and distributed with Perl since Perl 5.14 (2011-05-14), whereas Net::HTTP will likely live in a separate package forever.
2019-06-14edit: fix portability of editor invocation
The eval was unnecessary, and $0 can't be "--". Tested with /bin/sh on FreeBSD 11.2
2019-06-14Merge remote-tracking branch 'origin/reshard' into next
* origin/reshard: xcpdb: support resharding v2 repos xcpdb: use destination shard as progress prefix xapcmd: preserve indexlevel based on the destination v2writable: use a smaller default for Xapian partitions
2019-06-14Merge remote-tracking branch 'origin/manifest' into next
* origin/manifest: git: ensure ->modified returns an integer www: support $INBOX/git/$EPOCH.git for v2 cloning www: wire up /$INBOX/manifest.js.gz, too wwwlisting: generate grokmirror-compatible manifest.js.gz wwwlisting: allow hiding entries from manifest
2019-06-14Merge remote-tracking branch 'origin/edit' into next
* origin/edit: edit: unlink temporary file when done v2writable: replace: kill git processes before reindexing edit: drop unwanted headers before noop check edit|purge: improve output on rewrites edit: new tool to perform edits doc: document the --prune option for -index admin: expose ->config AdminEdit: move editability checks from -purge admin: beef up resolve_inboxes to handle purge options purge: start moving common options to AdminEdit module admin: remove warning arg for unconfigured inboxes v2writable: implement ->replace call import: switch to "replace_oids" interface for purge import: extract_author_info becomes extract_commit_info v2writable: consolidate overview and indexing call
2019-06-14xcpdb: support resharding v2 repos
v2 repos are sometimes created on machines where CPU parallelization exceeds the capability of the storage devices. In that case, users may reshard the Xapian DB to any smaller, positive integer to avoid excessive overhead and contention when bottlenecked by slow storage. Resharding can also be used to increase shard count after hardware upgrades.
2019-06-14xcpdb: use destination shard as progress prefix
For M:N resharding, we'll want to display the number from the new shard number.
2019-06-14xapcmd: preserve indexlevel based on the destination
To support M:N resharding, we need to ensure we store the indexlevel in the destination shard, rather than the originating one.
2019-06-14v2writable: use a smaller default for Xapian partitions
Apparently 16 CPUs (probably HT) and SATA storage is common these days. Having excessive Xapian partitions leads to contention and excessive FD/space use. So set a smaller default but continue allowing user-specified values to bump this up.
2019-06-14git: remove cat_file sub callback interface
We weren't using it, and in retrospect, it makes no sense to use this API cat_file for giant responses which can't read quickly with minimal context-switching (or sanely fit into memory for Email::Simple/Email::MIME). For giant blobs which we don't want slurped in memory, we'll spawn a short-lived git-cat-file process like we do in ViewVCS. Otherwise, monopolizing a git-cat-file process for a giant blob is harmful to other PSGI/NNTP users. A better interface is coming which will be more suitable for for batch processing of "small" objects such as commits and email blobs.