* [PATCH 1/3] lei: safety fix for multiple WQ classes
2021-06-08 9:50 5% [PATCH 0/3] lei import: speedup repeated Maildir import Eric Wong
@ 2021-06-08 9:50 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-06-08 9:50 UTC (permalink / raw)
To: meta
For commands utilizing multiple workers, this simple change
generalizes the persistence mechanism and and prevents
lei->dclose from causing script/lei to exit if there are
still in-flight workers.
This ougth to prevent read-after-write consistency problems that
occasionally manifest in scripts (e.g. test cases) but usually
go unnoticed in normal use.
---
lib/PublicInbox/LEI.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 8adf70fa..0cf4d10b 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -580,6 +580,8 @@ sub workers_start {
$wq->wq_workers_start($ident, $jobs, $lei->oldset, { lei => $lei });
delete $lei->{pkt_op_p};
my $op_c = delete $lei->{pkt_op_c};
+ # {-lei_sock} persists script/lei process until ops->{''} EOF callback
+ $op_c->{-lei_sock} = $lei->{sock};
@$end = ();
$lei->event_step_init;
($op_c, $ops);
^ permalink raw reply related [relevance 7%]
* [PATCH 0/3] lei import: speedup repeated Maildir import
@ 2021-06-08 9:50 5% Eric Wong
2021-06-08 9:50 7% ` [PATCH 1/3] lei: safety fix for multiple WQ classes Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-06-08 9:50 UTC (permalink / raw)
To: meta
More importantly, this series starts us down the road of
generalizing the use of auxiliary WQs like {ikw} (import
keywords for IMAP) and {pmd} (parallel Maildir).
It took me a bit to figure out some consistency problems
which I reordered and split out into 1/3. 2/3 should
make future work easier, and 3/3 gives the final speedup
along the lines of what I already did with IMAP.
Eric Wong (3):
lei: safety fix for multiple WQ classes
lei: generalize auxiliary WQ handling
lei import: speed up repeated Maildir imports
MANIFEST | 1 +
lib/PublicInbox/LEI.pm | 20 ++++++++--
lib/PublicInbox/LeiBlob.pm | 2 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiExportKw.pm | 2 +-
lib/PublicInbox/LeiImport.pm | 39 +++++++++++++-------
lib/PublicInbox/LeiIndex.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 31 +++++++++++-----
lib/PublicInbox/LeiLsSearch.pm | 2 +-
lib/PublicInbox/LeiMailSync.pm | 14 +++++++
lib/PublicInbox/LeiMirror.pm | 4 +-
lib/PublicInbox/LeiP2q.pm | 4 +-
lib/PublicInbox/LeiPmdir.pm | 67 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/LeiRediff.pm | 2 +-
lib/PublicInbox/LeiRm.pm | 2 +-
lib/PublicInbox/LeiTag.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 2 +-
lib/PublicInbox/MdirReader.pm | 22 ++++++-----
lib/PublicInbox/PktOp.pm | 6 ---
t/lei-import-maildir.t | 2 +-
20 files changed, 171 insertions(+), 57 deletions(-)
create mode 100644 lib/PublicInbox/LeiPmdir.pm
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-06-08 9:50 5% [PATCH 0/3] lei import: speedup repeated Maildir import Eric Wong
2021-06-08 9:50 7% ` [PATCH 1/3] lei: safety fix for multiple WQ classes Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).