about summary refs log tree commit homepage
path: root/t/www_listing.t
DateCommit message (Collapse)
2023-12-13tests: account for missing git-http-backend
Alpine Linux ships git-http-backend in the `git-daemon' package separately from `git', so we must test for its existence before attempting to test functionality which depends on it.
2023-09-09fix some tests when `curl' is missing
Debian-based distros allow uninstalling the `curl' CLI tool independently of libcurl* packages which git depends on.
2023-01-30use Net::SSLeay (OpenSSL) for SHA-(1|256) if installed
On my x86-64 machine, OpenSSL SHA-256 is nearly twice as fast as the Digest::SHA implementation from Perl, most likely due to an optimized assembly implementation. SHA-1 is a few percent faster, too.
2022-11-28lei_mirror: retrieve v2 description properly
2022-11-28clone: support --dry-run / -n flag
It still makes HTTP(S) requests to retrieve the manifest or scrape HTML, but doesn't make permanent changes to the FS (aside from modifying {acm}time of ${TMPDIR-/tmp}).
2022-11-28lei_mirror: set gitweb.owner from manifest
This is mainly for coderepos, but sometimes public-inboxes get shared via cgit/gitweb, too.
2022-11-28clone: support --include and --exclude with multi-clone
These will be handy when someone is interested in a subset of inboxes on a large hosting site.
2022-11-28clone: support multi-inbox clone
This is to ensure we can do `public-inbox-clone https://yhbt.net/lore' or `public-inbox-clone https://lore.kernel.org/' and clone all inboxes (and whatever else git stores).
2021-10-13t/www_listing: require opt-in for grokmirror tests
grokmirror 2.x seems to idle in several places for 5s at-a-time, causing t/www_listing.t to take longer than "make check-run" on a 4-core system when run without grokmirror. So make it optional but add some test knobs to allow tailing the log output so I can see what's going on.
2021-09-13tests: add require_cmd, require curl when needed
t/v2mirror.t and t/lei-mirror.t are now skipped when curl is missing (instead of failing in appropriate places). A bunch of which() checks are updated to use require_cmd to avoid explicitly loading Spawn.
2021-09-12import: do not write a "description" file
The default value is worthless to us and git functions fine without the file. public-inbox-init will create a useful one in the next change.
2021-08-28get rid of unnecessary bytes::length usage
The only place where we could return wide characters with -httpd was the raw $INBOX_DIR/description text, which is now converted to octets. All daemon (HTTP/NNTP/IMAP) sockets are opened in binary mode, so length() and bytes::length() are equivalent on reads. For socket writes, any non-octet data would warn about wide characters and we are strict in warnings with test_httpd. All gzipped buffers are also octets, as is PublicInbox::Eml->body, and anything from PerlIO objects ("git cat-file --batch" output, filesystems), so bytes::length was unnecessary in all those places.
2021-06-29www: fix manifest.js.gz for default publicInbox.grokManifest
ManifestJsGz->response was not invoking the new "url_filter" method properly. Furthermore, fix url_filter for returning 404 responses. Reported-by: Kyle Meyer <kyle@kyleam.com> Link: https://public-inbox.org/meta/87fsx3128a.fsf@kyleam.com/ Fixes: 520be116e8a686cb ("www_listing: start updating for pagination + search")
2021-02-21t/www_listing: require grok-pull version 2 or later
The grok-pull-based tests in www_listing are incompatible with Grokmirror v2 in two ways: the generated configuration format and the expected exit codes. Update the tests to work with v2, and skip them for earlier versions. This was tested with the latest release of Grokmirror, v2.0.7. Note that the "pull" and "fsck" sections are required even though they're empty.
2021-02-21t/www_listing: reword grok-pull skip message
Make it clear that this skip is because grok-pull isn't available at all because the next commit will add another skip for older versions of Grokmirror.
2021-02-21t/www_listing: correct the number of tests for grok-pull skip
2021-02-08tests: favor IPv6
IPv4 gets plenty of real-world coverage, and apparently there's Debian buildd hosts which lack IPv4(*). So ensure everything can work on IPv6 and not cause problems for odd setups. (*) https://bugs.debian.org/979432
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-12-19tests: more common JSON module loading
We'll probably be using JSON more in the future, so make it easier to require in tests
2020-11-24git: add manifest_entry method
We'll be using this for MiscIdx and pre-generating the necessary JSON for manifest.js.gz, so make it easier to share code for generating per-repo JSON entries for grokmirror.
2020-09-10www: manifest.js.gz generation no longer hogs event loop
It's still as slow as before with hundreds/thousands of inboxes, but at least it's fair. Future changes will allow it to be cached and memoized with persistent HTTP servers.
2020-07-06wwwlisting: use GzipFilter for HTML
The changes to GzipFilter here may be beneficial for building HTML and XML responses in other places, too.
2020-05-29treat $INBOX_DIR/description and gitweb.owner as UTF-8
gitweb does the same with $GIT_DIR/description and gitweb.owner. Allowing UTF-8 description should not cause problems when used in responses for to the NNTP "LIST NEWSGROUPS" request, either, since RFC 3977 section 7.6.6 recommends the description be UTF-8 (but does not require it). Link: https://public-inbox.org/meta/20200528151216.l7vmnmrs4ojw372g@sourcephile.fr/
2020-04-20testcommon: spawn-aware system() and qx[] workalikes
Barely noticeable on Linux, but this gives a 1-2% speedup on a FreeBSD 11.3 VM and lets us use built-in redirects rather than relying on /bin/sh.
2020-04-20import: init_bare: allow use as method, use in tests
Allowing ->init_bare to be used as a method saves some keystrokes, and we can save a little bit of time on systems with our vfork(2)-enabled spawn(). This also sets us up for future improvements where we can avoid spawning a process at all.
2020-03-21t/www_listing: avoid 'once' warnings
We reach into the WwwListing package directly to retrieve that JSON encoder/decoder object, and we can't rely on `use' since WwwListing loading may fail if Plack is missing.
2020-03-20wwwlisting: avoid lazy loading JSON module
We already lazy-load WwwListing for the CGI script, and hiding another layer of lazy-loading makes things difficult to do WWW->preload. We want long-lived processes to do all long-lived allocations up front to avoid fragmentation in the allocator, but we'll still support short-lived processes by lazy-loading individual modules in the PublicInbox::* namespace. Mixing up allocation lifetimes (e.g. doing immortal allocations while a large amount of space is taken by short-lived objects) will cause fragmentation in any allocator which favors large contiguous regions for performance reasons. This includes any malloc implementation which relies on sbrk() for the primary heap, including glibc malloc.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2019-12-26t/www_listing: test "all" HTML listing
We now have coverage for PublicInbox::WwwListing::list_all.
2019-12-26t/www_listing: quiet down stderr in -httpd
We need to init all.git for the v2 repo test to ensure `git --git-dir=v2/all.git rev-parse --git-path objects/info/alternates` doesn't warn or fail and clutter stderr. This is noticeable when setting TAIL="tail -F" in env before running this test.
2019-12-24testcommon: add require_mods method and use it
This cuts down on lines of code in individual test cases and fixes some misnamed error messages by using "$0" consistently. This will also provide us with a method of swapping out dependencies which provide equivalent functionality (e.g "Xapian" SWIG can replace "Search::Xapian" XS bindings).
2019-12-19tests: move t/common.perl to PublicInbox::TestCommon
We want to be able to use run_script with *.t files, so t/common.perl putting subs into the top-level "main" namespace won't work. Instead, make it a module which uses Exporter like other libraries.
2019-12-19t/*.t: avoid sharing "my" variables in subs
These usages of file-local global variables make the *.t files incompatible with run_script(). Instead, use anonymous subs, "our", or pass the parameter as appropriate.
2019-11-24tests: use File::Temp->newdir instead of tempdir()
We'll also introduce a tmpdir() API to give tempdirs consistent names.
2019-11-24t/common: start_script replaces spawn_listener
We can shave several hundred milliseconds off tests which spawn daemons by preloading and avoiding startup time for common modules which are already loaded in the parent process. This also gives ENV{TAIL} support to all tests which support daemons which log to stdout/stderr.
2019-11-08t/*.t: disable nntpd/httpd worker processes in most tests
And explicitly test for respawning in t/httpd-corner.t There's no need to have an extra entries in the process table for most tests we run, since that's not what we're testing.
2019-11-08t/*.t: remove IPC::Run dependency for git commands
One small step towards making tests easier-to-run. We can rely on "local $ENV{GIT_DIR}" for potentially shell-unsafe path names, and the rest of our path names are relative and don't contain characters which require escaping.
2019-10-16config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I intended for there to be a "spamrepo" (now just the ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be especially confusing, since v2 needs at least two git repositories (epoch + all.git) to function and we shouldn't confuse users by having them point to a git repository for v2. Much of our documentation already references "INBOX_DIR" for command-line arguments, so use "inboxdir" as the git-config(1)-friendly variant for that. "mainrepo" remains supported indefinitely for compatibility. Users may need to revert to old versions, or may be referring to old documentation and must not be forced to change config files to account for this change. So if you're using "mainrepo" today, I do NOT recommend changing it right away because other bugs can lurk. Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
2019-09-09tests: add tcp_connect() helper
IO::Socket::INET->new is rather verbose with the options hash, extract it into a standalone sub
2019-06-30tests: common tcp_server and unix_server helpers
IO::Socket:*->new options are verbose and we can save a bunch of code by putting this into t/common.perl, since the related spawn_listener stuff is already there.
2019-06-23manifest: v2 epoch descriptions based on inbox->description
The default $GIT_DIR/description (provided by git.git templates) isn't very useful for v2 epochs, so use the inbox description and suffix it with the epoch number if it's otherwise unnamed. Requested-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> https://public-inbox.org/meta/20190620190017.GA27175@chatter.i7.local/
2019-06-14t/www_listing: favor HTTP::Tiny over Net::HTTP
More testers are likely to have HTTP::Tiny than Net::HTTP, since HTTP::Tiny is a dual-life module and distributed with Perl since Perl 5.14 (2011-05-14), whereas Net::HTTP will likely live in a separate package forever.
2019-06-10git: ensure ->modified returns an integer
We don't want to serialize timestamps as strings to JSON. I only noticed this bug on a 32-bit system.
2019-06-09www: support $INBOX/git/$EPOCH.git for v2 cloning
And use it in manifest.js. To ease maintaining mirrors with grokmirror(1), we can accept a "git/" directory prefix before the epoch, and ".git" suffix after the epoch number. We maintain compatibility with "$INBOX/$EPOCH" cloning, of course, and it's still easier-to-type on the command-line.
2019-06-09www: wire up /$INBOX/manifest.js.gz, too
I can imagine myself just wanting to clone a single v2 inbox and all its epochs without thinking about include/exclude rules in a grokmirror config file.
2019-06-09wwwlisting: generate grokmirror-compatible manifest.js.gz
Support on-demand generation of "/manifest.js.gz" for inboxes. By default, this matches inboxes with URLs matching the given request hostname by default. This makes it easier to create full mirrors of several inboxes without needing to configure static file serving. cf. https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git