user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 03/15] config: reject newlines consistently in dir names
  2023-11-30 11:40  6% [PATCH 00/15] various cindex fixes + speedups Eric Wong
@ 2023-11-30 11:40  7% ` Eric Wong
  0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-30 11:40 UTC (permalink / raw)
  To: meta

Explicitly drop support for "\n" in git coderepo pathnames as
we do other stuff.  Gcf2 (our libgit2 helper) was always
broken with "\n" in pathnames, and I'm not sure if cgit config
files work with them, either.  Dealing with newline characters
requires extra complexity that I'm not willing to deal with when
managing alternates files.
---
 lib/PublicInbox/Config.pm | 32 ++++++++++++++------------------
 1 file changed, 14 insertions(+), 18 deletions(-)

diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index 779e3140..6bebf790 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -361,12 +361,19 @@ sub parse_cgitrc {
 	cgit_repo_merge($self, $repo->{dir}, $repo) if $repo;
 }
 
+sub valid_dir ($$) {
+	my $dir = get_1($_[0], $_[1]) // return;
+	index($dir, "\n") < 0 ? $dir : do {
+		warn "E: `$_[1]=$dir' must not contain `\\n'\n";
+		undef;
+	}
+}
+
 # parse a code repo, only git is supported at the moment
 sub fill_coderepo {
 	my ($self, $nick) = @_;
 	my $pfx = "coderepo.$nick";
-	my $dir = $self->{"$pfx.dir"} // return undef; # aka "GIT_DIR"
-	my $git = PublicInbox::Git->new($dir);
+	my $git = PublicInbox::Git->new(valid_dir($self, "$pfx.dir") // return);
 	if (defined(my $cgits = $self->{"$pfx.cgiturl"})) {
 		$git->{cgit_url} = $cgits = _array($cgits);
 		$self->{"$pfx.cgiturl"} = $cgits;
@@ -450,18 +457,15 @@ sub _fill_ibx {
 		my $v = $self->{"$pfx.$k"};
 		$ibx->{$k} = $v if defined $v;
 	}
-	for my $k (qw(filter inboxdir newsgroup replyto httpbackendmax feedmax
+	for my $k (qw(filter newsgroup replyto httpbackendmax feedmax
 			indexlevel indexsequentialshard boost)) {
 		my $v = get_1($self, "$pfx.$k") // next;
 		$ibx->{$k} = $v;
 	}
 
 	# "mainrepo" is backwards compatibility:
-	my $dir = $ibx->{inboxdir} //= $self->{"$pfx.mainrepo"} // return;
-	if (index($dir, "\n") >= 0) {
-		warn "E: `$dir' must not contain `\\n'\n";
-		return;
-	}
+	my $dir = $ibx->{inboxdir} = valid_dir($self, "$pfx.inboxdir") //
+				valid_dir($self, "$pfx.mainrepo") // return;
 	for my $k (qw(obfuscate)) {
 		my $v = $self->{"$pfx.$k"} // next;
 		if (defined(my $bval = git_bool($v))) {
@@ -548,12 +552,8 @@ sub _fill_ei ($$) {
 	my ($self, $name) = @_;
 	eval { require PublicInbox::ExtSearch } or return;
 	my $pfx = "extindex.$name";
-	my $d = $self->{"$pfx.topdir"} // return;
+	my $d = valid_dir($self, "$pfx.topdir") // return;
 	-d $d or return;
-	if (index($d, "\n") >= 0) {
-		warn "E: `$d' must not contain `\\n'\n";
-		return;
-	}
 	my $es = PublicInbox::ExtSearch->new($d);
 	for my $k (qw(indexlevel indexsequentialshard)) {
 		my $v = get_1($self, "$pfx.$k") // next;
@@ -573,12 +573,8 @@ sub _fill_csrch ($$) {
 	return if $name ne '' && !valid_foo_name($name, 'cindex');
 	eval { require PublicInbox::CodeSearch } or return;
 	my $pfx = "cindex.$name";
-	my $d = $self->{"$pfx.topdir"} // return;
+	my $d = valid_dir($self, "$pfx.topdir") // return;
 	-d $d or return;
-	if (index($d, "\n") >= 0) {
-		warn "E: `$d' must not contain `\\n'\n";
-		return;
-	}
 	my $csrch = PublicInbox::CodeSearch->new($d, $self);
 	for my $k (qw(localprefix)) {
 		my $v = $self->{"$pfx.$k"} // next;

^ permalink raw reply related	[relevance 7%]

* [PATCH 00/15] various cindex fixes + speedups
@ 2023-11-30 11:40  6% Eric Wong
  2023-11-30 11:40  7% ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-30 11:40 UTC (permalink / raw)
  To: meta

Notable changes:

10/15 provides a huge speedup which will hopefully make
future developments faster.

12/15 probably obsoletes libgit2 for extindex "all" users.

13/15 can save some memory with many inboxes while making
configuration easier.

Eric Wong (15):
  cindex: fix store_repo+repo_stored on no-op
  codesearch: allow inbox count to exceed matches
  config: reject newlines consistently in dir names
  cindex: only create {-cidx_err} field on failures
  cindex: keep batch pipe for pruning SHA-256 repos
  cindex: store extensions.objectFormat with repo data
  git: share unlinked pack checking code with gcf2
  cindex: skip getpid guard for most OnDestroy use
  spawn: drop IO layer support from redirects
  cindex: speed up initial scan setup phase
  inbox: expire resources more aggressively
  git_async_cat: use git from "all" extindex if possible
  www_listing: support publicInbox.nameIsUrl
  inbox: shrink data structures for publicinbox.*.hide
  codesearch: use retry_reopen for WWW

 Documentation/public-inbox-config.pod |  19 +-
 lib/PublicInbox/CodeSearch.pm         |  54 +++--
 lib/PublicInbox/CodeSearchIdx.pm      | 286 ++++++++++++++++----------
 lib/PublicInbox/Config.pm             |  32 ++-
 lib/PublicInbox/Gcf2.pm               |  16 +-
 lib/PublicInbox/Git.pm                |  27 +--
 lib/PublicInbox/GitAsyncCat.pm        |   8 +-
 lib/PublicInbox/Inbox.pm              |  32 +--
 lib/PublicInbox/MailDiff.pm           |   3 +-
 lib/PublicInbox/SearchIdx.pm          |   5 +-
 lib/PublicInbox/Spawn.pm              |  32 +--
 lib/PublicInbox/WwwListing.pm         |  21 +-
 12 files changed, 303 insertions(+), 232 deletions(-)


^ permalink raw reply	[relevance 6%]

Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-30 11:40  6% [PATCH 00/15] various cindex fixes + speedups Eric Wong
2023-11-30 11:40  7% ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).