* [PATCH 03/15] config: reject newlines consistently in dir names
2023-11-30 11:40 6% [PATCH 00/15] various cindex fixes + speedups Eric Wong
@ 2023-11-30 11:40 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-30 11:40 UTC (permalink / raw)
To: meta
Explicitly drop support for "\n" in git coderepo pathnames as
we do other stuff. Gcf2 (our libgit2 helper) was always
broken with "\n" in pathnames, and I'm not sure if cgit config
files work with them, either. Dealing with newline characters
requires extra complexity that I'm not willing to deal with when
managing alternates files.
---
lib/PublicInbox/Config.pm | 32 ++++++++++++++------------------
1 file changed, 14 insertions(+), 18 deletions(-)
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 779e3140..6bebf790 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -361,12 +361,19 @@ sub parse_cgitrc {
cgit_repo_merge($self, $repo->{dir}, $repo) if $repo;
}
+sub valid_dir ($$) {
+ my $dir = get_1($_[0], $_[1]) // return;
+ index($dir, "\n") < 0 ? $dir : do {
+ warn "E: `$_[1]=$dir' must not contain `\\n'\n";
+ undef;
+ }
+}
+
# parse a code repo, only git is supported at the moment
sub fill_coderepo {
my ($self, $nick) = @_;
my $pfx = "coderepo.$nick";
- my $dir = $self->{"$pfx.dir"} // return undef; # aka "GIT_DIR"
- my $git = PublicInbox::Git->new($dir);
+ my $git = PublicInbox::Git->new(valid_dir($self, "$pfx.dir") // return);
if (defined(my $cgits = $self->{"$pfx.cgiturl"})) {
$git->{cgit_url} = $cgits = _array($cgits);
$self->{"$pfx.cgiturl"} = $cgits;
@@ -450,18 +457,15 @@ sub _fill_ibx {
my $v = $self->{"$pfx.$k"};
$ibx->{$k} = $v if defined $v;
}
- for my $k (qw(filter inboxdir newsgroup replyto httpbackendmax feedmax
+ for my $k (qw(filter newsgroup replyto httpbackendmax feedmax
indexlevel indexsequentialshard boost)) {
my $v = get_1($self, "$pfx.$k") // next;
$ibx->{$k} = $v;
}
# "mainrepo" is backwards compatibility:
- my $dir = $ibx->{inboxdir} //= $self->{"$pfx.mainrepo"} // return;
- if (index($dir, "\n") >= 0) {
- warn "E: `$dir' must not contain `\\n'\n";
- return;
- }
+ my $dir = $ibx->{inboxdir} = valid_dir($self, "$pfx.inboxdir") //
+ valid_dir($self, "$pfx.mainrepo") // return;
for my $k (qw(obfuscate)) {
my $v = $self->{"$pfx.$k"} // next;
if (defined(my $bval = git_bool($v))) {
@@ -548,12 +552,8 @@ sub _fill_ei ($$) {
my ($self, $name) = @_;
eval { require PublicInbox::ExtSearch } or return;
my $pfx = "extindex.$name";
- my $d = $self->{"$pfx.topdir"} // return;
+ my $d = valid_dir($self, "$pfx.topdir") // return;
-d $d or return;
- if (index($d, "\n") >= 0) {
- warn "E: `$d' must not contain `\\n'\n";
- return;
- }
my $es = PublicInbox::ExtSearch->new($d);
for my $k (qw(indexlevel indexsequentialshard)) {
my $v = get_1($self, "$pfx.$k") // next;
@@ -573,12 +573,8 @@ sub _fill_csrch ($$) {
return if $name ne '' && !valid_foo_name($name, 'cindex');
eval { require PublicInbox::CodeSearch } or return;
my $pfx = "cindex.$name";
- my $d = $self->{"$pfx.topdir"} // return;
+ my $d = valid_dir($self, "$pfx.topdir") // return;
-d $d or return;
- if (index($d, "\n") >= 0) {
- warn "E: `$d' must not contain `\\n'\n";
- return;
- }
my $csrch = PublicInbox::CodeSearch->new($d, $self);
for my $k (qw(localprefix)) {
my $v = $self->{"$pfx.$k"} // next;
^ permalink raw reply related [relevance 7%]
* [PATCH 00/15] various cindex fixes + speedups
@ 2023-11-30 11:40 6% Eric Wong
2023-11-30 11:40 7% ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-30 11:40 UTC (permalink / raw)
To: meta
Notable changes:
10/15 provides a huge speedup which will hopefully make
future developments faster.
12/15 probably obsoletes libgit2 for extindex "all" users.
13/15 can save some memory with many inboxes while making
configuration easier.
Eric Wong (15):
cindex: fix store_repo+repo_stored on no-op
codesearch: allow inbox count to exceed matches
config: reject newlines consistently in dir names
cindex: only create {-cidx_err} field on failures
cindex: keep batch pipe for pruning SHA-256 repos
cindex: store extensions.objectFormat with repo data
git: share unlinked pack checking code with gcf2
cindex: skip getpid guard for most OnDestroy use
spawn: drop IO layer support from redirects
cindex: speed up initial scan setup phase
inbox: expire resources more aggressively
git_async_cat: use git from "all" extindex if possible
www_listing: support publicInbox.nameIsUrl
inbox: shrink data structures for publicinbox.*.hide
codesearch: use retry_reopen for WWW
Documentation/public-inbox-config.pod | 19 +-
lib/PublicInbox/CodeSearch.pm | 54 +++--
lib/PublicInbox/CodeSearchIdx.pm | 286 ++++++++++++++++----------
lib/PublicInbox/Config.pm | 32 ++-
lib/PublicInbox/Gcf2.pm | 16 +-
lib/PublicInbox/Git.pm | 27 +--
lib/PublicInbox/GitAsyncCat.pm | 8 +-
lib/PublicInbox/Inbox.pm | 32 +--
lib/PublicInbox/MailDiff.pm | 3 +-
lib/PublicInbox/SearchIdx.pm | 5 +-
lib/PublicInbox/Spawn.pm | 32 +--
lib/PublicInbox/WwwListing.pm | 21 +-
12 files changed, 303 insertions(+), 232 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-30 11:40 6% [PATCH 00/15] various cindex fixes + speedups Eric Wong
2023-11-30 11:40 7% ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).