[PATCH v8 0/2] send-email: expose header information to git-send-email's sendemail-validate hook

[PATCH v8 0/2] send-email: expose header information to git-send-email's sendemail-validate hook

[Michael Strawbridge]

This patch version addresses:
- extra test for number of arguments in the header t9001 test
- removed use of envelope in the documentation
- re-arranged the validation loop and surrounding code to be closer to
  the other code not in functions
- made a note in the commit message that the moving of validation later
  can have some difference in stdout compared to before but actual email
functionality should be ok

Hopefully this looks better.

Michael Strawbridge (2):
  send-email: refactor header generation functions
  send-email: expose header information to git-send-email's
    sendemail-validate hook

 Documentation/githooks.txt |  27 +++++++--
 git-send-email.perl        | 109 ++++++++++++++++++++++---------------
 t/t9001-send-email.sh      |  27 ++++++++-
 3 files changed, 114 insertions(+), 49 deletions(-)

-- 
2.34.1

[PATCH v8 1/2] send-email: refactor header generation functions

[Michael Strawbridge]

Split process_file and send_message into easier to use functions.
Making SMTP header information widely available.

Cc: Luben Tuikov <luben.tuikov@amd.com>
Cc: Junio C Hamano <gitster@pobox.com>
Cc: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Acked-by: Luben Tuikov <luben.tuikov@amd.com>
Signed-off-by: Michael Strawbridge <michael.strawbridge@amd.com>
---
 git-send-email.perl | 49 ++++++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 18 deletions(-)

diff --git a/git-send-email.perl b/git-send-email.perl
index 5861e99a6e..42f135a266 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1495,16 +1495,7 @@ sub file_name_is_absolute {
 	return File::Spec::Functions::file_name_is_absolute($path);
 }
 
-# Prepares the email, then asks the user what to do.
-#
-# If the user chooses to send the email, it's sent and 1 is returned.
-# If the user chooses not to send the email, 0 is returned.
-# If the user decides they want to make further edits, -1 is returned and the
-# caller is expected to call send_message again after the edits are performed.
-#
-# If an error occurs sending the email, this just dies.
-
-sub send_message {
+sub gen_header {
 	my @recipients = unique_email_list(@to);
 	@cc = (grep { my $cc = extract_valid_address_or_die($_);
 		      not grep { $cc eq $_ || $_ =~ /<\Q${cc}\E>$/ } @recipients
@@ -1546,6 +1537,22 @@ sub send_message {
 	if (@xh) {
 		$header .= join("\n", @xh) . "\n";
 	}
+	my $recipients_ref = \@recipients;
+	return ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header);
+}
+
+# Prepares the email, then asks the user what to do.
+#
+# If the user chooses to send the email, it's sent and 1 is returned.
+# If the user chooses not to send the email, 0 is returned.
+# If the user decides they want to make further edits, -1 is returned and the
+# caller is expected to call send_message again after the edits are performed.
+#
+# If an error occurs sending the email, this just dies.
+
+sub send_message {
+	my ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header) = gen_header();
+	my @recipients = @$recipients_ref;
 
 	my @sendmail_parameters = ('-i', @recipients);
 	my $raw_from = $sender;
@@ -1735,11 +1742,8 @@ sub send_message {
 $references = $initial_in_reply_to || '';
 $message_num = 0;
 
-# Prepares the email, prompts the user, sends it out
-# Returns 0 if an edit was done and the function should be called again, or 1
-# otherwise.
-sub process_file {
-	my ($t) = @_;
+sub pre_process_file {
+	my ($t, $quiet) = @_;
 
 	open my $fh, "<", $t or die sprintf(__("can't open file %s"), $t);
 
@@ -1893,9 +1897,9 @@ sub process_file {
 	}
 	close $fh;
 
-	push @to, recipients_cmd("to-cmd", "to", $to_cmd, $t)
+	push @to, recipients_cmd("to-cmd", "to", $to_cmd, $t, $quiet)
 		if defined $to_cmd;
-	push @cc, recipients_cmd("cc-cmd", "cc", $cc_cmd, $t)
+	push @cc, recipients_cmd("cc-cmd", "cc", $cc_cmd, $t, $quiet)
 		if defined $cc_cmd && !$suppress_cc{'cccmd'};
 
 	if ($broken_encoding{$t} && !$has_content_type) {
@@ -1954,6 +1958,15 @@ sub process_file {
 			@initial_to = @to;
 		}
 	}
+}
+
+# Prepares the email, prompts the user, and sends it out
+# Returns 0 if an edit was done and the function should be called again, or 1
+# on the email being successfully sent out.
+sub process_file {
+	my ($t) = @_;
+
+        pre_process_file($t, $quiet);
 
 	my $message_was_sent = send_message();
 	if ($message_was_sent == -1) {
@@ -2002,7 +2015,7 @@ sub process_file {
 # Execute a command (e.g. $to_cmd) to get a list of email addresses
 # and return a results array
 sub recipients_cmd {
-	my ($prefix, $what, $cmd, $file) = @_;
+	my ($prefix, $what, $cmd, $file, $quiet) = @_;
 
 	my @addresses = ();
 	open my $fh, "-|", "$cmd \Q$file\E"
-- 
2.34.1

[PATCH v8 2/2] send-email: expose header information to git-send-email's sendemail-validate hook

[Michael Strawbridge]

To allow further flexibility in the Git hook, the SMTP header
information of the email which git-send-email intends to send, is now
passed as the 2nd argument to the sendemail-validate hook.

As an example, this can be useful for acting upon keywords in the
subject or specific email addresses.

As a consequence of needing all the header data, validation has been
moved later in the sequence to right before sending the emails instead
of at the beginning.

Cc: Luben Tuikov <luben.tuikov@amd.com>
Cc: Junio C Hamano <gitster@pobox.com>
Cc: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Acked-by: Luben Tuikov <luben.tuikov@amd.com>
Signed-off-by: Michael Strawbridge <michael.strawbridge@amd.com>
---
 Documentation/githooks.txt | 27 ++++++++++++++---
 git-send-email.perl        | 60 ++++++++++++++++++++++----------------
 t/t9001-send-email.sh      | 27 +++++++++++++++--
 3 files changed, 83 insertions(+), 31 deletions(-)

diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt
index a16e62bc8c..0decbfc92d 100644
--- a/Documentation/githooks.txt
+++ b/Documentation/githooks.txt
@@ -583,10 +583,29 @@ processed by rebase.
 sendemail-validate
 ~~~~~~~~~~~~~~~~~~
 
-This hook is invoked by linkgit:git-send-email[1].  It takes a single parameter,
-the name of the file that holds the e-mail to be sent.  Exiting with a
-non-zero status causes `git send-email` to abort before sending any
-e-mails.
+This hook is invoked by linkgit:git-send-email[1].
+
+It takes these command line arguments. They are,
+1. the name of the file which holds the contents of the email to be sent.
+2. The name of the file which holds the SMTP headers of the email.
+
+The SMTP headers are passed in the exact same way as they are passed to the
+user's Mail Transport Agent (MTA). In effect, the email given to the user's
+MTA, is the contents of $2 followed by the contents of $1.
+
+Below is an example for a few common headers. Take notice of the
+capitalization and multi-line tab structure.
+
+  From: Example <from@example.com>
+  To: to@example.com
+  Cc: cc@example.com,
+	  A <author@example.com>,
+	  One <one@example.com>,
+	  two@example.com
+  Subject: PATCH-STRING
+
+Exiting with a non-zero status causes `git send-email` to abort
+before sending any e-mails.
 
 fsmonitor-watchman
 ~~~~~~~~~~~~~~~~~~
diff --git a/git-send-email.perl b/git-send-email.perl
index 42f135a266..d123dfd33d 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -787,14 +787,6 @@ sub is_format_patch_arg {
 
 @files = handle_backup_files(@files);
 
-if ($validate) {
-	foreach my $f (@files) {
-		unless (-p $f) {
-			validate_patch($f, $target_xfer_encoding);
-		}
-	}
-}
-
 if (@files) {
 	unless ($quiet) {
 		print $_,"\n" for (@files);
@@ -1124,6 +1116,30 @@ sub expand_one_alias {
 our ($message_id, %mail, $subject, $in_reply_to, $references, $message,
 	$needs_confirm, $message_num, $ask_default);
 
+my ($message_id_stamp, $message_id_serial);
+
+$time = time - scalar $#files;
+
+if ($validate) {
+	foreach my $f (@files) {
+		unless (-p $f) {
+		        pre_process_file($f, 1);
+
+			validate_patch($f, $target_xfer_encoding);
+		}
+	}
+}
+
+$in_reply_to = $initial_in_reply_to;
+$references = $initial_in_reply_to || '';
+$message_num = 0;
+
+foreach my $t (@files) {
+	while (!process_file($t)) {
+		# user edited the file
+	}
+}
+
 sub extract_valid_address {
 	my $address = shift;
 	my $local_part_regexp = qr/[^<>"\s@]+/;
@@ -1189,7 +1205,6 @@ sub validate_address_list {
 
 # We'll setup a template for the message id, using the "from" address:
 
-my ($message_id_stamp, $message_id_serial);
 sub make_message_id {
 	my $uniq;
 	if (!defined $message_id_stamp) {
@@ -1214,10 +1229,6 @@ sub make_message_id {
 	#print "new message id = $message_id\n"; # Was useful for debugging
 }
 
-
-
-$time = time - scalar $#files;
-
 sub unquote_rfc2047 {
 	local ($_) = @_;
 	my $charset;
@@ -1738,10 +1749,6 @@ sub send_message {
 	return 1;
 }
 
-$in_reply_to = $initial_in_reply_to;
-$references = $initial_in_reply_to || '';
-$message_num = 0;
-
 sub pre_process_file {
 	my ($t, $quiet) = @_;
 
@@ -2006,12 +2013,6 @@ sub process_file {
 	return 1;
 }
 
-foreach my $t (@files) {
-	while (!process_file($t)) {
-		# user edited the file
-	}
-}
-
 # Execute a command (e.g. $to_cmd) to get a list of email addresses
 # and return a results array
 sub recipients_cmd {
@@ -2101,11 +2102,20 @@ sub validate_patch {
 			chdir($repo->wc_path() or $repo->repo_path())
 				or die("chdir: $!");
 			local $ENV{"GIT_DIR"} = $repo->repo_path();
+
+			my ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header) = gen_header();
+
+			require File::Temp;
+			my ($header_filehandle, $header_filename) = File::Temp::tempfile(
+                            ".gitsendemail.header.XXXXXX", DIR => $repo->repo_path());
+			print $header_filehandle $header;
+
 			my @cmd = ("git", "hook", "run", "--ignore-missing",
 				    $hook_name, "--");
-			my @cmd_msg = (@cmd, "<patch>");
-			my @cmd_run = (@cmd, $target);
+			my @cmd_msg = (@cmd, "<patch>", "<header>");
+			my @cmd_run = (@cmd, $target, $header_filename);
 			$hook_error = system_or_msg(\@cmd_run, undef, "@cmd_msg");
+			unlink($header_filehandle);
 			chdir($cwd_save) or die("chdir: $!");
 		}
 		if ($hook_error) {
diff --git a/t/t9001-send-email.sh b/t/t9001-send-email.sh
index 1130ef21b3..8a5c111a24 100755
--- a/t/t9001-send-email.sh
+++ b/t/t9001-send-email.sh
@@ -540,7 +540,7 @@ test_expect_success $PREREQ "--validate respects relative core.hooksPath path" '
 	test_path_is_file my-hooks.ran &&
 	cat >expect <<-EOF &&
 	fatal: longline.patch: rejected by sendemail-validate hook
-	fatal: command '"'"'git hook run --ignore-missing sendemail-validate -- <patch>'"'"' died with exit code 1
+	fatal: command '"'"'git hook run --ignore-missing sendemail-validate -- <patch> <header>'"'"' died with exit code 1
 	warning: no patches were sent
 	EOF
 	test_cmp expect actual
@@ -559,12 +559,35 @@ test_expect_success $PREREQ "--validate respects absolute core.hooksPath path" '
 	test_path_is_file my-hooks.ran &&
 	cat >expect <<-EOF &&
 	fatal: longline.patch: rejected by sendemail-validate hook
-	fatal: command '"'"'git hook run --ignore-missing sendemail-validate -- <patch>'"'"' died with exit code 1
+	fatal: command '"'"'git hook run --ignore-missing sendemail-validate -- <patch> <header>'"'"' died with exit code 1
 	warning: no patches were sent
 	EOF
 	test_cmp expect actual
 '
 
+test_expect_success $PREREQ "--validate hook supports header argument" '
+	write_script my-hooks/sendemail-validate <<-\EOF &&
+        if test "$#" -ge 2
+	then
+		grep "X-test-header: v1.0" "$2"
+	else
+		echo "No header arg passed"
+		exit 1
+	fi
+	EOF
+	test_config core.hooksPath "my-hooks" &&
+	rm -fr outdir &&
+	git format-patch \
+		--add-header="X-test-header: v1.0" \
+		-n HEAD^1 -o outdir &&
+	git send-email \
+		--dry-run \
+		--to=nobody@example.com \
+		--smtp-server="$(pwd)/fake.sendmail" \
+		--validate \
+		outdir/000?-*.patch
+'
+
 for enc in 7bit 8bit quoted-printable base64
 do
 	test_expect_success $PREREQ "--transfer-encoding=$enc produces correct header" '
-- 
2.34.1

Re: [PATCH v8 2/2] send-email: expose header information to git-send-email's sendemail-validate hook

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 18 2023, Michael Strawbridge wrote:

> To allow further flexibility in the Git hook, the SMTP header
> information of the email which git-send-email intends to send, is now
> passed as the 2nd argument to the sendemail-validate hook.
>
> As an example, this can be useful for acting upon keywords in the
> subject or specific email addresses.
>
> As a consequence of needing all the header data, validation has been
> moved later in the sequence to right before sending the emails instead
> of at the beginning.

Ah, I see. I tested this (i.e. moving it back to the previous behavior)
and you did this change because you don't have the $sender variable yet.

I tried this quickly on top, which seems to work, i.e. now we do this in
the same order as before, but we just move the $sender code earlier:
	
	diff --git a/git-send-email.perl b/git-send-email.perl
	index d123dfd33d5..7e7681116bb 100755
	--- a/git-send-email.perl
	+++ b/git-send-email.perl
	@@ -787,6 +787,28 @@ sub is_format_patch_arg {
	 
	 @files = handle_backup_files(@files);
	 
	+if (defined $sender) {
	+	$sender =~ s/^\s+|\s+$//g;
	+	($sender) = expand_aliases($sender);
	+} else {
	+	$sender = $repoauthor->() || $repocommitter->() || '';
	+}
	+
	+# $sender could be an already sanitized address
	+# (e.g. sendemail.from could be manually sanitized by user).
	+# But it's a no-op to run sanitize_address on an already sanitized address.
	+$sender = sanitize_address($sender);
	+
	+if ($validate) {
	+	foreach my $f (@files) {
	+		unless (-p $f) {
	+		        pre_process_file($f, 1);
	+
	+			validate_patch($f, $target_xfer_encoding);
	+		}
	+	}
	+}
	+
	 if (@files) {
	 	unless ($quiet) {
	 		print $_,"\n" for (@files);
	@@ -1035,18 +1057,6 @@ sub file_declares_8bit_cte {
	 	}
	 }
	 
	-if (defined $sender) {
	-	$sender =~ s/^\s+|\s+$//g;
	-	($sender) = expand_aliases($sender);
	-} else {
	-	$sender = $repoauthor->() || $repocommitter->() || '';
	-}
	-
	-# $sender could be an already sanitized address
	-# (e.g. sendemail.from could be manually sanitized by user).
	-# But it's a no-op to run sanitize_address on an already sanitized address.
	-$sender = sanitize_address($sender);
	-
	 my $to_whom = __("To whom should the emails be sent (if anyone)?");
	 my $prompting = 0;
	 if (!@initial_to && !defined $to_cmd) {
	@@ -1120,16 +1130,6 @@ sub expand_one_alias {
	 
	 $time = time - scalar $#files;
	 
	-if ($validate) {
	-	foreach my $f (@files) {
	-		unless (-p $f) {
	-		        pre_process_file($f, 1);
	-
	-			validate_patch($f, $target_xfer_encoding);
	-		}
	-	}
	-}
	-
	 $in_reply_to = $initial_in_reply_to;
	 $references = $initial_in_reply_to || '';
	 $message_num = 0;

All tests pass with that, which is less good than it sounds, because
shouldn't your tests be checking whether we have this non--quiet
print-out of the files as expected before or after the validation hook
runs?

> +
> +			my ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header) = gen_header();
> +
> +			require File::Temp;
> +			my ($header_filehandle, $header_filename) = File::Temp::tempfile(
> +                            ".gitsendemail.header.XXXXXX", DIR => $repo->repo_path());
> +			print $header_filehandle $header;
> +
>  			my @cmd = ("git", "hook", "run", "--ignore-missing",
>  				    $hook_name, "--");
> -			my @cmd_msg = (@cmd, "<patch>");
> -			my @cmd_run = (@cmd, $target);
> +			my @cmd_msg = (@cmd, "<patch>", "<header>");
> +			my @cmd_run = (@cmd, $target, $header_filename);
>  			$hook_error = system_or_msg(\@cmd_run, undef, "@cmd_msg");
> +			unlink($header_filehandle);
>  			chdir($cwd_save) or die("chdir: $!");

I'm still curious about the "stdin" question I asked in the last round.

Re: [PATCH v8 2/2] send-email: expose header information to git-send-email's sendemail-validate hook

[Michael Strawbridge]

On 2023-01-18 18:12, Ævar Arnfjörð Bjarmason wrote:
> On Wed, Jan 18 2023, Michael Strawbridge wrote:
>
>> To allow further flexibility in the Git hook, the SMTP header
>> information of the email which git-send-email intends to send, is now
>> passed as the 2nd argument to the sendemail-validate hook.
>>
>> As an example, this can be useful for acting upon keywords in the
>> subject or specific email addresses.
>>
>> As a consequence of needing all the header data, validation has been
>> moved later in the sequence to right before sending the emails instead
>> of at the beginning.
> Ah, I see. I tested this (i.e. moving it back to the previous behavior)
> and you did this change because you don't have the $sender variable yet.
>
> I tried this quickly on top, which seems to work, i.e. now we do this in
> the same order as before, but we just move the $sender code earlier:
> 	
> 	diff --git a/git-send-email.perl b/git-send-email.perl
> 	index d123dfd33d5..7e7681116bb 100755
> 	--- a/git-send-email.perl
> 	+++ b/git-send-email.perl
> 	@@ -787,6 +787,28 @@ sub is_format_patch_arg {
> 	 
> 	 @files = handle_backup_files(@files);
> 	 
> 	+if (defined $sender) {
> 	+	$sender =~ s/^\s+|\s+$//g;
> 	+	($sender) = expand_aliases($sender);
> 	+} else {
> 	+	$sender = $repoauthor->() || $repocommitter->() || '';
> 	+}
> 	+
> 	+# $sender could be an already sanitized address
> 	+# (e.g. sendemail.from could be manually sanitized by user).
> 	+# But it's a no-op to run sanitize_address on an already sanitized address.
> 	+$sender = sanitize_address($sender);
> 	+
> 	+if ($validate) {
> 	+	foreach my $f (@files) {
> 	+		unless (-p $f) {
> 	+		        pre_process_file($f, 1);
> 	+
> 	+			validate_patch($f, $target_xfer_encoding);
> 	+		}
> 	+	}
> 	+}
> 	+
> 	 if (@files) {
> 	 	unless ($quiet) {
> 	 		print $_,"\n" for (@files);
> 	@@ -1035,18 +1057,6 @@ sub file_declares_8bit_cte {
> 	 	}
> 	 }
> 	 
> 	-if (defined $sender) {
> 	-	$sender =~ s/^\s+|\s+$//g;
> 	-	($sender) = expand_aliases($sender);
> 	-} else {
> 	-	$sender = $repoauthor->() || $repocommitter->() || '';
> 	-}
> 	-
> 	-# $sender could be an already sanitized address
> 	-# (e.g. sendemail.from could be manually sanitized by user).
> 	-# But it's a no-op to run sanitize_address on an already sanitized address.
> 	-$sender = sanitize_address($sender);
> 	-
> 	 my $to_whom = __("To whom should the emails be sent (if anyone)?");
> 	 my $prompting = 0;
> 	 if (!@initial_to && !defined $to_cmd) {
> 	@@ -1120,16 +1130,6 @@ sub expand_one_alias {
> 	 
> 	 $time = time - scalar $#files;
> 	 
> 	-if ($validate) {
> 	-	foreach my $f (@files) {
> 	-		unless (-p $f) {
> 	-		        pre_process_file($f, 1);
> 	-
> 	-			validate_patch($f, $target_xfer_encoding);
> 	-		}
> 	-	}
> 	-}
> 	-
> 	 $in_reply_to = $initial_in_reply_to;
> 	 $references = $initial_in_reply_to || '';
> 	 $message_num = 0;
>
> All tests pass with that, which is less good than it sounds, because
> shouldn't your tests be checking whether we have this non--quiet
> print-out of the files as expected before or after the validation hook
> runs?

Thank you very much!  That idea to move sender earlier actually
simplifies this a lot.  I had to move a piece relating to the date field
earlier as well but after doing that I think the patch gives the same
output.

I think checking the timing of stdout in tests is not easy.  I'm a bit
unsure we would want to test such specific stdout behaviour in the
tests.  I have focused on the functional output (eg header output). 
Luckily with the above, my change will no longer introduce any stdout
differences anymore.

I will wait a little before pushing out a v9 in case there is more feedback.

>> +
>> +			my ($recipients_ref, $to, $date, $gitversion, $cc, $ccline, $header) = gen_header();
>> +
>> +			require File::Temp;
>> +			my ($header_filehandle, $header_filename) = File::Temp::tempfile(
>> +                            ".gitsendemail.header.XXXXXX", DIR => $repo->repo_path());
>> +			print $header_filehandle $header;
>> +
>>  			my @cmd = ("git", "hook", "run", "--ignore-missing",
>>  				    $hook_name, "--");
>> -			my @cmd_msg = (@cmd, "<patch>");
>> -			my @cmd_run = (@cmd, $target);
>> +			my @cmd_msg = (@cmd, "<patch>", "<header>");
>> +			my @cmd_run = (@cmd, $target, $header_filename);
>>  			$hook_error = system_or_msg(\@cmd_run, undef, "@cmd_msg");
>> +			unlink($header_filehandle);
>>  			chdir($cwd_save) or die("chdir: $!");
> I'm still curious about the "stdin" question I asked in the last round.

For the stdin question, are you referring to the git hook run question? 
I know there have been a lot of parallel threads so you may have missed
my reply
(https://public-inbox.org/git/20230117215811.78313-1-michael.strawbridge@amd.com/).

Here's what I responded with:

I was trying to follow the convention that the original hook was using.
I'm not against changing this if the out of tree patches you speak of
are going to be rolled in soon.  However, I'd prefer not to delay this
patch if these other patches are far off. Thanks.