git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com,
	"Derrick Stolee" <stolee@gmail.com>,
	"Đoàn Trần Công Danh" <congdanhqx@gmail.com>,
	"Martin Ågren" <martin.agren@gmail.com>,
	"Eric Sunshine" <sunshine@sunshineco.com>,
	"Derrick Stolee" <stolee@gmail.com>,
	"Derrick Stolee" <derrickstolee@github.com>
Subject: [PATCH v2 0/4] Maintenance IV: Platform-specific background maintenance
Date: Wed, 04 Nov 2020 20:06:04 +0000	[thread overview]
Message-ID: <pull.776.v2.git.1604520368.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.776.git.1604412196.gitgitgadget@gmail.com>

This is based on ds/maintenance-part-3.

After sitting with the background maintenance as it has been cooking, I
wanted to come back around and implement the background maintenance for
Windows. However, I noticed that there were some things bothering me with
background maintenance on my macOS machine. These are detailed in PATCH 3,
but the tl;dr is that 'cron' is not recommended by Apple and instead
'launchd' satisfies our needs.

This series implements the background scheduling so git maintenance
(start|stop) works on those platforms. I've been operating with these
schedules for a while now without the problems described in the patches.

There is a particularly annoying case about console windows popping up on
Windows, but PATCH 4 describes a plan to get around that.

Updates in V2
=============

 * This is a faster turnaround for a v2 than I would normally like, but Eric
   inspired extra documentation about how to customize background schedules.
   
   
 * New extensions to git-maintenance.txt include guidelines for inspecting
   what git maintenance start does and how to customize beyond that. This
   includes a new PATCH 2 that includes documentation for 'cron' on
   non-macOS non-Windows systems.
   
   
 * Several improvements, especially in the tests, are included.
   
   
 * While testing manually, I noticed that somehow I had incorrectly had an
   opening <dict> tag instead of a closing </dict> tag in the hourly format
   on macOS. I found that the xmllint tool can verify the XML format of a
   file, which catches the bug. This seems like a good approach since the
   test is macOS-only. Does anyone have concerns about adding this
   dependency?
   
   

Thanks, -Stolee

Derrick Stolee (4):
  maintenance: extract platform-specific scheduling
  maintenance: include 'cron' details in docs
  maintenance: use launchctl on macOS
  maintenance: use Windows scheduled tasks

 Documentation/git-maintenance.txt | 119 +++++++++
 builtin/gc.c                      | 428 ++++++++++++++++++++++++++++--
 t/t7900-maintenance.sh            |  83 +++++-
 t/test-lib.sh                     |   4 +
 4 files changed, 614 insertions(+), 20 deletions(-)


base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/776

Range-diff vs v1:

 1:  d35f1aa162 = 1:  d35f1aa162 maintenance: extract platform-specific scheduling
 -:  ---------- > 2:  709a173720 maintenance: include 'cron' details in docs
 2:  832fdf1687 ! 3:  0fafd75d10 maintenance: use launchctl on macOS
     @@ Commit message
          and 'git maintenance stop' by injecting a script that writes the
          command-line arguments into GIT_TEST_CRONTAB.
      
     +    An earlier version of this patch accidentally had an opening
     +    "<dict>" tag when it should have had a closing "</dict>" tag. This
     +    was caught during manual testing with actual 'launchctl' commands,
     +    but we do not want to update developers' tasks when running tests.
     +    It appears that macOS includes the "xmllint" tool which can verify
     +    the XML format, so call it from the macOS-specific tests to ensure
     +    the .plist files are well-formatted.
     +
     +    Helped-by: Eric Sunshine <sunshine@sunshineco.com>
          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      
     + ## Documentation/git-maintenance.txt ##
     +@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do use the full path and
     + executing the correct binaries in your schedule.
     + 
     + 
     ++BACKGROUND MAINTENANCE ON MACOS SYSTEMS
     ++---------------------------------------
     ++
     ++While macOS technically supports `cron`, using `crontab -e` requires
     ++elevated privileges and the executed process do not have a full user
     ++context. Without a full user context, Git and its credential helpers
     ++cannot access stored credentials, so some maintenance tasks are not
     ++functional.
     ++
     ++Instead, `git maintenance start` interacts with the `launchctl` tool,
     ++which is the recommended way to
     ++https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS].
     ++
     ++Scheduling maintenance through `git maintenance (start|stop)` requires
     ++some `launchctl` features available only in macOS 10.11 or later.
     ++
     ++Your user-specific scheduled tasks are stored as XML-formatted `.plist`
     ++files in `~/Library/LaunchAgents/`. You can see the currently-registered
     ++tasks using the following command:
     ++
     ++-----------------------------------------------------------------------
     ++$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git
     ++org.git-scm.git.daily.plist
     ++org.git-scm.git.hourly.plist
     ++org.git-scm.git.weekly.plist
     ++-----------------------------------------------------------------------
     ++
     ++One task is registered for each `--schedule=<frequency>` option. To
     ++inspect how the XML format describes each schedule, open one of these
     ++`.plist` files in an editor and inspect the `<array>` element following
     ++the `<key>StartCalendarInterval</key>` element.
     ++
     ++`git maintenance start` will overwrite these files and register the
     ++tasks again with `launchctl`, so any customizations should be done by
     ++creating your own `.plist` files with distinct names. Similarly, the
     ++`git maintenance stop` command will unregister the tasks with `launchctl`
     ++and delete the `.plist` files.
     ++
     ++To create more advanced customizations to your background tasks, see
     ++https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation]
     ++for more information.
     ++
     ++
     + GIT
     + ---
     + Part of the linkgit:git[1] suite
     +
       ## builtin/gc.c ##
      @@ builtin/gc.c: static int maintenance_unregister(void)
       	return run_command(&config_unset);
     @@ builtin/gc.c: static int maintenance_unregister(void)
      +		repeat = "<dict>\n"
      +			 "<key>Hour</key><integer>%d</integer>\n"
      +			 "<key>Minute</key><integer>0</integer>\n"
     -+			 "<dict>\n";
     ++			 "</dict>\n";
      +		for (i = 1; i <= 23; i++)
      +			fprintf(plist, repeat, i);
      +		break;
     @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' '
      +	for frequency in hourly daily weekly
      +	do
      +		PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" &&
     ++		xmllint "$PLIST" >/dev/null &&
      +		grep schedule=$frequency "$PLIST" &&
      +		echo "bootout gui/$UID $PLIST" >>expect &&
      +		echo "bootstrap gui/$UID $PLIST" >>expect || return 1
 3:  a9221cc4aa ! 4:  84eb44de31 maintenance: use Windows scheduled tasks
     @@ Commit message
          short term. In the long term, we can consider creating this GUI
          shim application within core Git, perhaps in contrib/.
      
     +    Helped-by: Eric Sunshine <sunshine@sunshineco.com>
          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      
     + ## Documentation/git-maintenance.txt ##
     +@@ Documentation/git-maintenance.txt: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSy
     + for more information.
     + 
     + 
     ++BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS
     ++-----------------------------------------
     ++
     ++Windows does not support `cron` and instead has its own system for
     ++scheduling background tasks. The `git maintenance start` command uses
     ++the `schtasks` command to submit tasks to this system. You can inspect
     ++all background tasks using the Task Scheduler application. The tasks
     ++added by Git have names of the form `Git Maintenance (<frequency>)`.
     ++The Task Scheduler GUI has ways to inspect these tasks, but you can also
     ++export the tasks to XML files and view the details there.
     ++
     ++Note that since Git is a console application, these background tasks
     ++create a console window visible to the current user. This can be changed
     ++manually by selecting the "Run whether user is logged in or not" option
     ++in Task Scheduler. This change requires a password input, which is why
     ++`git maintenance start` does not select it by default.
     ++
     ++If you want to customize the background tasks, please rename the tasks
     ++so future calls to `git maintenance (start|stop)` do not overwrite your
     ++custom tasks.
     ++
     ++
     + GIT
     + ---
     + Part of the linkgit:git[1] suite
     +
       ## builtin/gc.c ##
      @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd)
       	else
     @@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop ma
       '
       
      +test_expect_success MINGW 'start and stop Windows maintenance' '
     -+	echo "echo \$@ >>args" >print-args &&
     -+	chmod a+x print-args &&
     ++	write_script print-args <<-\EOF &&
     ++	echo $* >>args
     ++	EOF
      +
      +	rm -f args &&
      +	GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start &&
     -+	cat args &&
      +
      +	# start registers the repo
      +	git config --get --global maintenance.repo "$(pwd)" &&
      +
     -+	rm expect &&
      +	for frequency in hourly daily weekly
      +	do
     -+		echo "/create /tn Git Maintenance ($frequency) /f /xml .git/objects/schedule-$frequency.xml" >>expect \
     -+			|| return 1
     -+	done &&
     ++		printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \
     ++			$frequency $frequency
     ++	done >expect &&
      +	test_cmp expect args &&
      +
      +	rm -f args &&
     -+	GIT_TEST_CRONTAB="/bin/sh print-args"  git maintenance stop &&
     ++	GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop &&
      +
      +	# stop does not unregister the repo
      +	git config --get --global maintenance.repo "$(pwd)" &&
      +
      +	rm expect &&
     -+	for frequency in hourly daily weekly
     -+	do
     -+		echo "/delete /tn Git Maintenance ($frequency) /f" >>expect \
     -+			|| return 1
     -+	done &&
     ++	printf "/delete /tn Git Maintenance (%s) /f\n" \
     ++		hourly daily weekly >expect &&
      +	test_cmp expect args
      +'
      +

-- 
gitgitgadget

  parent reply	other threads:[~2020-11-04 20:06 UTC|newest]

Thread overview: 83+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-03 14:03 [PATCH 0/3] " Derrick Stolee via GitGitGadget
2020-11-03 14:03 ` [PATCH 1/3] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2020-11-03 14:03 ` [PATCH 2/3] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2020-11-03 18:45   ` Eric Sunshine
2020-11-03 21:21     ` Derrick Stolee
2020-11-03 22:27       ` Eric Sunshine
2020-11-04 13:33         ` Derrick Stolee
2020-11-04 14:17       ` Derrick Stolee
2020-11-03 14:03 ` [PATCH 3/3] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
2020-11-03 19:06   ` Eric Sunshine
2020-11-03 21:23     ` Derrick Stolee
2020-11-03 20:18 ` [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Junio C Hamano
2020-11-03 20:21 ` Junio C Hamano
2020-11-03 21:09   ` Derrick Stolee
2020-11-03 22:30     ` Junio C Hamano
2020-11-04 13:02       ` Derrick Stolee
2020-11-04 17:00         ` Junio C Hamano
2020-11-04 18:43           ` Derrick Stolee
2020-11-04 20:06 ` Derrick Stolee via GitGitGadget [this message]
2020-11-04 20:06   ` [PATCH v2 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2020-11-04 20:06   ` [PATCH v2 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget
2020-11-11  7:10     ` Eric Sunshine
2020-11-04 20:06   ` [PATCH v2 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2020-11-11  8:12     ` Eric Sunshine
2020-11-12 13:42       ` Derrick Stolee
2020-11-12 16:43         ` Eric Sunshine
2020-11-04 20:06   ` [PATCH v2 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
2020-11-11  8:59     ` Eric Sunshine
2020-11-12 13:56       ` Derrick Stolee
2020-11-13 14:00   ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget
2020-11-13 14:00     ` [PATCH v3 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2020-11-13 14:00     ` [PATCH v3 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget
2020-11-13 14:00     ` [PATCH v3 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2020-11-13 20:19       ` Eric Sunshine
2020-11-13 20:42         ` Derrick Stolee
2020-11-13 20:53           ` Eric Sunshine
2020-11-13 20:56             ` Eric Sunshine
2020-11-13 14:00     ` [PATCH v3 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
2020-11-13 20:44       ` Eric Sunshine
2020-11-13 21:32         ` Derrick Stolee
2020-11-13 21:40           ` Eric Sunshine
2020-11-16 13:13             ` Derrick Stolee
2020-11-13 20:47     ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine
2020-11-14  9:23       ` Eric Sunshine
2020-11-16 13:17         ` Derrick Stolee
2020-11-17 21:13     ` [PATCH v4 " Derrick Stolee via GitGitGadget
2020-11-17 21:13       ` [PATCH v4 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2020-11-17 21:13       ` [PATCH v4 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget
2020-11-18  0:34         ` Eric Sunshine
2020-11-18 18:30           ` Derrick Stolee
2020-11-17 21:13       ` [PATCH v4 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2020-11-18  6:45         ` Eric Sunshine
2020-11-18 18:22           ` Derrick Stolee
2020-11-17 21:13       ` [PATCH v4 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
2020-11-18  7:15         ` Eric Sunshine
2020-11-18 18:30           ` Derrick Stolee
2020-11-18 20:54             ` Eric Sunshine
2020-11-18 21:16               ` Derrick Stolee
2020-11-17 23:36       ` [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine
2020-11-24  2:20         ` Derrick Stolee
2020-11-24  2:59           ` Eric Sunshine
2020-11-17 23:54       ` Eric Sunshine
2020-11-24  4:16       ` [PATCH v5 " Derrick Stolee via GitGitGadget
2020-11-24  4:16         ` [PATCH v5 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2020-11-24  4:16         ` [PATCH v5 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget
2020-11-24  4:16         ` [PATCH v5 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2020-11-24  4:16         ` [PATCH v5 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
2020-11-27  9:08           ` Eric Sunshine
2020-12-09 19:28         ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget
2020-12-09 19:28           ` [PATCH v6 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2020-12-09 19:29           ` [PATCH v6 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget
2020-12-09 19:29           ` [PATCH v6 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2020-12-09 19:29           ` [PATCH v6 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
2020-12-10  0:32           ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Junio C Hamano
2020-12-10  0:49             ` Eric Sunshine
2020-12-10  1:04               ` Junio C Hamano
2021-01-05 12:17                 ` Derrick Stolee
2021-01-05 13:08           ` [PATCH v7 " Derrick Stolee via GitGitGadget
2021-01-05 13:08             ` [PATCH v7 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget
2021-01-05 13:08             ` [PATCH v7 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget
2021-01-05 13:08             ` [PATCH v7 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget
2021-01-10  6:34               ` Eric Sunshine
2021-01-05 13:08             ` [PATCH v7 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=pull.776.v2.git.1604520368.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=congdanhqx@gmail.com \
    --cc=derrickstolee@github.com \
    --cc=git@vger.kernel.org \
    --cc=jonathantanmy@google.com \
    --cc=jrnieder@gmail.com \
    --cc=martin.agren@gmail.com \
    --cc=sluongng@gmail.com \
    --cc=stolee@gmail.com \
    --cc=sunshine@sunshineco.com \
    --subject='Re: [PATCH v2 0/4] Maintenance IV: Platform-specific background maintenance' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).