about summary refs log tree commit homepage
path: root/lib/PublicInbox/Git.pm
DateCommit message (Collapse)
2020-01-13use popen_rd for bidirectional pipes
popen_rd accepts arbitrary redirects, so we can reuse its code to setup the pipe end we want to read, saving each caller a few lines of code compared to calling pipe+spawn.
2020-01-13git: packed_bytes: use GLOB_NOSORT
File::Glob is loaded by the perl for the "glob()" op, anyways, so call bsd_glob with the GLOB_NOSORT to avoid needless sorting of the output.
2020-01-13git: modified: don't slurp `rev-parse --branches'
While v1 inboxes typically only have one branch, code repositories may have dozens or even hundreds. Slurping those into memory is a waste.
2020-01-11spawn (and thus popen_rd) die on failure
Most spawn and popen_rd callers die on failure to spawn, anyways, and some are missing checks entirely. This saves us a bunch of verbose error-checking code in callers. This also makes popen_rd more consistent, since it already dies on pipe creation failures.
2020-01-11git: remove ->commit_title method
We haven't used it in SolverGit, yet, and I'll be reworking it to work with ->cat_async, instead.
2020-01-11git: ->modified uses cat_async
While v1 inboxes are typically only a single branch, coderepos will have many branches and being able to pipeline requests to "git cat-file --batch" can help us mask seek times.
2020-01-11allow HTTP_HOST to be '0' via defined() checks
'0' is a valid value for HTTP_HOST, and maybe some folks will want to hit that as port 80 where the HTTP client won't send the ":$PORT" suffix.
2020-01-06treewide: "require" + "use" cleanup and docs
There's a bunch of leftover "require" and "use" statements we no longer need and can get rid of, along with some excessive imports via "use". IO::Handle usage isn't always obvious, so add comments describing why a package loads it. Along the same lines, document the tmpdir support as the reason we depend on File::Temp 0.19, even though every Perl 5.10.1+ user has it. While we're at it, favor "use" over "require", since it it gives us extra compile-time checking.
2019-12-30spawn: allow passing GLOB handles for redirects
We can save callers the trouble of {-hold} and {-dev_null} refs as well as the trouble of calling fileno().
2019-12-26git: allow async_cat to pass arg to callback
This allows callers to avoid allocating several KB for for every call to ->async_cat.
2019-12-12git: async batch interface
This is a transitionary interface which does NOT require an event loop. It can be plugged into in current synchronous code without major surgery. It allows HTTP/1.1 pipelining-like functionality by taking advantage of predictable and well-specified POSIX pipe semantics by stuffing multiple git cat-file requests into the --batch pipe With xt/git_async_cmp.t and GIANT_GIT_DIR=git.git, the async interface is 10-25% faster than the synchronous interface since it can keep the "git cat-file" process busier. This is expected to improve performance on systems with slower storage (but multiple cores).
2019-10-22git: remove src_blob_url
This was intended for solver, but it's unused since commit 915cd090798069a4 ("solver: switch patch application to use a callback")
2019-09-14tmpfile: give temporary files meaningful names
Although we always unlink temporary files, give them a meaningful name so that we can we can still make sense of the pre-unlink name when using lsof(8) or similar tools on Linux.
2019-09-09run update-copyrights from gnulib for 2019
2019-07-08ds: use WNOHANG with waitpid if inside event loop
While we're usually not stuck waiting on waitpid after seeing a pipe EOF or even triggering SIGPIPE in the process (e.g. git-http-backend) we're reading from, it MAY happen and we should be careful to never hang the daemon process on waitpid calls. v2: use "eq" for string comparison against 'DEFAULT'
2019-06-14Merge remote-tracking branch 'origin/manifest' into next
* origin/manifest: git: ensure ->modified returns an integer www: support $INBOX/git/$EPOCH.git for v2 cloning www: wire up /$INBOX/manifest.js.gz, too wwwlisting: generate grokmirror-compatible manifest.js.gz wwwlisting: allow hiding entries from manifest
2019-06-14git: remove cat_file sub callback interface
We weren't using it, and in retrospect, it makes no sense to use this API cat_file for giant responses which can't read quickly with minimal context-switching (or sanely fit into memory for Email::Simple/Email::MIME). For giant blobs which we don't want slurped in memory, we'll spawn a short-lived git-cat-file process like we do in ViewVCS. Otherwise, monopolizing a git-cat-file process for a giant blob is harmful to other PSGI/NNTP users. A better interface is coming which will be more suitable for for batch processing of "small" objects such as commits and email blobs.
2019-06-10git: ensure ->modified returns an integer
We don't want to serialize timestamps as strings to JSON. I only noticed this bug on a 32-bit system.
2019-06-05tighten up digit matches to ASCII for git output
While I don't expect git to suddenly start spewing non-ASCII digits in places I'd expect ASCII, this would make things easier for future hackers and reviewers.
2019-06-01git: drop the deleted err_c file
No reason to leave that (usually) empty file open after killing off "cat-file --batch-check". This wasn't an unbound leak, though, as respawning the --batch-check process would've clobbered the old err_c file.
2019-06-01git: unconditional expiry
A constant stream of traffic to either httpd/nntpd would mean git-cat-file processes never expire. Things can go bad after a full repack, as a full repack will unlink old pack indices and git-cat-file does not currently detect unlinked files. We could do something complicated by recursively stat-ing objects/pack of every git directory and alternate; but that's probably not worth the trouble compared to occasionally restarting the cat-file process. So simplify the code and let httpd/nntpd expire them periodically, since spawning a "git-cat-file --batch" process isn't too expensive. We already spawn for every request which hits git-http-backend, cgit, and git-apply. In the future, we may optionally support the Git::Raw module to avoid IPC; but we must remain careful to not leave lingering FDs open to unlinked files after repack.
2019-05-22git: workaround old git-rev-parse(1) (--git-path)
git < 2.5.0 was missing --git-path support. This means any users relying on some rare environment variables will need git 2.5.0+
2019-04-18git: calculate modified time of repository
This will be used for generating an HTML listing for v1 inboxes, at least. The logic for this follows that of grokmirror, and we may dynamically generate manifest.js.gz natively...
2019-04-04support publicinbox.cgitrc directive
We can save admins the trouble of declaring [coderepo "..."] sections in the public-inbox config by parsing the cgitrc directly. Macro expansion (e.g. $HTTP_HOST) expansion is not supported, yet; but may be in the future.
2019-04-04git: add "commit_title" method
This will be useful for extracting titles/subjects from commit objects when displaying commits.
2019-01-31inbox: perform cleanup of Git objects for coderepos
Otherwise, long-running but idle git processes may keep unlinked packs around indefinitely and waste disk space.
2019-01-30git: use "git rev-parse --git-path"
Using git worktrees was causing t/solver_git.t to fail on me.
2019-01-27solver: hold patches in temporary directory
We can avoid bumping up RLIMIT_NOFILE too much by storing patches in a temporary directory. And we can share this top-level directory with our temporary git repository. Since we no longer rely on a working-tree for git, we are free to rearrange the layout and avoid relying on the ".git" convention and relying on "git -C" for chdir. This may also ease porting public-inbox to older systems where git does not support "-C" for chdir.
2019-01-20git: support 'ambiguous' result from --batch-check
David Turner's patch to return "ambiguous" seems like a reasonable patch for future versions of git: https://public-inbox.org/git/672a6fb9e480becbfcb5df23ae37193784811b6b.camel@novalis.org/
2019-01-19git: disable abbreviations with cat-file hints
Ambiguity is not worth it for internal usage with the solver.
2019-01-19git: check saves error on disambiguation
This will be useful for disambiguating short OIDs in older emails when abbreviations were shorter. Tested against the following script with /path/to/git.git ==> t.perl <== use strict; use PublicInbox::Git; use Data::Dumper; my $dir = shift or die "Usage: $0 GIT_DIR # (of git.git)"; my $git = PublicInbox::Git->new($dir); my @res = $git->check('dead'); print Dumper({res => \@res, err=> $git->last_check_err}); @res = $git->check('5335669531d83d7d6c905bcfca9b5f8e182dc4d4'); print Dumper({res => \@res, err=> $git->last_check_err});
2019-01-19git: add git_quote
It'll be helpful for displaying progress in SolverGit output.
2019-01-19git: support multiple URL endpoints
For redundancy and centralization resistance.
2019-01-19solver: initial Perl implementation
This will lookup git blobs from associated git source code repositories. If the blobs can't be found, an attempt to "solve" them via patch application will be performed. Eventually, this may become the basis of a type-agnostic frontend similar to "git show"
2019-01-18git: git_unquote handles double-quote and backslash
We need to work with 0x22 (double-quote) and 0x5c (backslash); even if they're oddball characters in filenames which wouldn't be used by projects I'd want to work on.
2019-01-15git_unquote: perform modifications in-place
This function doesn't have a lot of callers at the moment so none of them are affected by this change. But the plan is to use this in our WWW code for things, so do it now before we call it in more places. Results from a Thinkpad X200 with a Core2Duo P8600 @ 2.4GHz: Benchmark: timing 10 iterations of cp, ip... cp: 12.868 wallclock secs (12.86 usr + 0.00 sys = 12.86 CPU) @ 0.78/s (n=10) ip: 10.9137 wallclock secs (10.91 usr + 0.00 sys = 10.91 CPU) @ 0.92/s (n=10) Note: I mainly care about unquoted performance because that's the common case for the target audience of public-inbox. Script used to get benchmark results against the Linux source tree: ==> bench_unquote.perl <== use strict; use warnings; use Benchmark ':hireswallclock'; my $nr = 50; my %GIT_ESC = ( a => "\a", b => "\b", f => "\f", n => "\n", r => "\r", t => "\t", v => "\013", ); sub git_unquote_ip ($) { return $_[0] unless ($_[0] =~ /\A"(.*)"\z/); $_[0] = $1; $_[0] =~ s/\\([abfnrtv])/$GIT_ESC{$1}/g; $_[0] =~ s/\\([0-7]{1,3})/chr(oct($1))/ge; $_[0]; } sub git_unquote_cp ($) { my ($s) = @_; return $s unless ($s =~ /\A"(.*)"\z/); $s = $1; $s =~ s/\\([abfnrtv])/$GIT_ESC{$1}/g; $s =~ s/\\([0-7]{1,3})/chr(oct($1))/ge; $s; } chomp(my @files = `git -C ~/linux ls-tree --name-only -r v4.19.13`); timethese(10, { cp => sub { for (0..$nr) { git_unquote_cp($_) for @files } }, ip => sub { for (0..$nr) { git_unquote_ip($_) for @files } }, });
2019-01-15searchidx: move git_unquote to PublicInbox::Git
We'll be using it outside of searchidx...
2019-01-05shrink low-bandwidth pipes under Linux
I've hit /proc/sys/fs/pipe-user-pages-* limits on some systems. So stop hogging resources on pipes which don't benefit from giant sizes. Some of these can use eventfd in the future to further reduce resource use.
2018-02-19git: reload alternates file on missing blob
Since we'll be adding new repositories to the `alternates' file in git, we must restart the `git cat-file --batch' process as git currently does not detect changes to the alternates file in long-running cat-file processes. Don't bother with the `--batch-check' process since we won't be using it with v2.
2018-02-19v2writable: initial cut for repo-rotation
Wrap the old Import package to enable creating new repos based on size thresholds. This is better than relying on time-based rotation as LKML traffic seems to be increasing.
2018-02-07update copyrights for 2018
Using update-copyrights from gnulib While we're at it, use the SPDX identifier for AGPL-3.0+ to ease mechanical processing.
2016-08-09searchidx: avoid holding Xapian lock in cat-file
We must ensure cat-file process is launched before Xapian grabs lock, too. Our use of "git cat-file --batch" has the same problem as "git log" did, (which was fixed in commit 3713c727cda431a0dc2865a7878c13ecf9f21851) "searchidx: release Xapian FDs before spawning git log"
2016-06-21spawn: improve error checking for fork failures
fork failures are unfortunately common when Xapian has gigabytes and gigabytes mmapped.
2016-05-21localize $/ in more places to avoid potential problems
This hopefully makes the intent of the code clearer, too. The the HTTP use of the numeric reference for getline caused problems in Git.pm, already.
2016-05-21mbox: switch generation over to pull model
This allows us to easily provide gigantic inboxes with proper backpressure handling for slow clients. It also eliminates public-inbox-httpd and Danga::Socket-specific knowledge from this class, making it easier to follow for those used to generic PSGI applications.
2016-04-27import: document API for public consumption
This is probably trivial enough to be final?
2016-04-11git: add support for qx wrapper
This lets us one-line git commands easily like ``, but without having to remember --git-dir or escape arguments.
2016-02-28reduce calls to close unless error checks are needed
We can rely on timely auto-destruction based on reference counting; reducing the chance of redundant close(2) calls which may hit the wront FD. We do care about certain close calls (e.g. writing to a buffered IO handle) if we require error-checking for write-integrity. In other cases, let things go out-of-scope so it can be freed automatically after use.
2016-02-27git: use built-in spawn implementation for vfork
This should reduce overhead of spawning git processes from our long-running httpd and nntpd servers.
2015-12-22rename 'GitCatFile' package to 'Git'
We'll be using it for more than just cat-file. Adding a `popen' API for internal use allows us to save a bunch of code in other places.