* [PATCH 0/3] Maintenance IV: Platform-specific background maintenance @ 2020-11-03 14:03 Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 1/3] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (5 more replies) 0 siblings, 6 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 2, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 3 describes a plan to get around that. Thanks, -Stolee Derrick Stolee (3): maintenance: extract platform-specific scheduling maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks builtin/gc.c | 428 +++++++++++++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 86 ++++++++- t/test-lib.sh | 4 + 3 files changed, 498 insertions(+), 20 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v1 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v1 Pull-Request: https://github.com/gitgitgadget/git/pull/776 -- gitgitgadget ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH 1/3] maintenance: extract platform-specific scheduling 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 ` Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 2/3] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget ` (4 subsequent siblings) 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a1..c1f7d9bdc2 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,7 +1494,7 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int platform_update_schedule(int run_maintenance, int fd) { int result = 0; int in_old_region = 0; @@ -1503,11 +1503,6 @@ static int update_background_schedule(int run_maintenance) FILE *cron_list, *cron_in; const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); crontab_name = getenv("GIT_TEST_CRONTAB"); if (!crontab_name) @@ -1516,12 +1511,11 @@ static int update_background_schedule(int run_maintenance) strvec_split(&crontab_list.args, crontab_name); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); } /* Ignore exit code, as an empty crontab will return error. */ @@ -1531,7 +1525,7 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); strvec_split(&crontab_edit.args, crontab_name); @@ -1539,8 +1533,7 @@ static int update_background_schedule(int run_maintenance) crontab_edit.git_cmd = 0; if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; + return error(_("failed to run 'crontab'; your system might not support 'cron'")); } cron_in = fdopen(crontab_edit.in, "w"); @@ -1586,13 +1579,24 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; - } - fclose(cron_list); + else + fclose(cron_list); + return result; +} + +static int update_background_schedule(int run_maintenance) +{ + int result; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + result = platform_update_schedule(run_maintenance, lk.tempfile->fd); -cleanup: rollback_lock_file(&lk); return result; } -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH 2/3] maintenance: use launchctl on macOS 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 1/3] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 ` Derrick Stolee via GitGitGadget 2020-11-03 18:45 ` Eric Sunshine 2020-11-03 14:03 ` [PATCH 3/3] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget ` (3 subsequent siblings) 5 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_CRONTAB. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 209 +++++++++++++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 52 +++++++++- t/test-lib.sh | 4 + 3 files changed, 262 insertions(+), 3 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index c1f7d9bdc2..fa0ae63a80 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,214 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +#if defined(__APPLE__) + +static char *get_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *get_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_uid(void) +{ + struct strbuf output = STRBUF_INIT; + struct child_process id = CHILD_PROCESS_INIT; + + strvec_pushl(&id.args, "/usr/bin/id", "-u", NULL); + if (capture_command(&id, &output, 0)) + die(_("failed to discover user id")); + + strbuf_trim_trailing_newline(&output); + return strbuf_detach(&output, NULL); +} + +static int bootout(const char *filename) +{ + int result; + struct strvec args = STRVEC_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + + strvec_split(&args, launchctl); + strvec_push(&args, "bootout"); + strvec_pushf(&args, "gui/%s", uid); + strvec_push(&args, filename); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(uid); + return result; +} + +static int bootstrap(const char *filename) +{ + int result; + struct strvec args = STRVEC_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + + strvec_split(&args, launchctl); + strvec_push(&args, "bootstrap"); + strvec_pushf(&args, "gui/%s", uid); + strvec_push(&args, filename); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(uid); + return result; +} + +static int remove_plist(enum schedule_priority schedule) +{ + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + int result = bootout(filename); + free(filename); + free(name); + return result; +} + +static int remove_plists(void) +{ + return remove_plist(SCHEDULE_HOURLY) || + remove_plist(SCHEDULE_DAILY) || + remove_plist(SCHEDULE_WEEKLY); +} + +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = fopen(filename, "w"); + + if (!plist) + die(_("failed to open '%s'"), filename); + + preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "<dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + + /* bootout might fail if not already running, so ignore */ + bootout(filename); + if (bootstrap(filename)) + die(_("failed to bootstrap service %s"), filename); + + fclose(plist); + free(filename); + free(name); + return 0; +} + +static int add_plists(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_plist(exec_path, SCHEDULE_HOURLY) || + schedule_plist(exec_path, SCHEDULE_DAILY) || + schedule_plist(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_plists(); + else + return remove_plists(); +} +#else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1585,6 +1793,7 @@ static int platform_update_schedule(int run_maintenance, int fd) fclose(cron_list); return result; } +#endif static int update_background_schedule(int run_maintenance) { diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..f0210aa206 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,12 +389,58 @@ test_expect_success 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' + echo "#!/bin/sh\necho \$@ >>args" >print-args && + chmod a+x print-args && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + # ~/Library/LaunchAgents + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect && + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + # stop does not remove plist files, but boots them out + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect || return 1 + done && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..620ffbf3af 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1703,6 +1703,10 @@ test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' +test_lazy_prereq MACOS_MAINTENANCE ' + launchctl list +' + # Ensure that no test accidentally triggers a Git command # that runs 'crontab', affecting a user's cron schedule. # Tests that verify the cron integration must set this locally -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH 2/3] maintenance: use launchctl on macOS 2020-11-03 14:03 ` [PATCH 2/3] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-03 18:45 ` Eric Sunshine 2020-11-03 21:21 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-03 18:45 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Tue, Nov 3, 2020 at 9:05 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > maintenance: use launchctl on macOS A few comments below (not necessarily worth a re-roll)... > The launchctl command needs to be aligned with a user id in order > to initialize the command environment. This must be done using > the 'launchctl bootstrap' subcommand. This subcommand is new as > of macOS 10.11, which was released in September 2015. Before that > release the 'launchctl load' subcommand was recommended. The best > source of information on this transition I have seen is available > at [2]. It's not clear whether or not this is saying that git-maintenance will dynamically adapt to work with modern and older 'launchctl'. A glance at the actual code reveals that it knows only about modern 'bootstrap'. Perhaps this could be a bit clearer by saying that it only supports modern versions, and that support for older versions can be added later if needed. (For those of us who are stuck with 10-20 year old hardware and OS versions, 2015 isn't that long ago.) > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1491,6 +1491,214 @@ static int maintenance_unregister(void) > +static int bootout(const char *filename) > +{ > + int result; > + struct strvec args = STRVEC_INIT; > + char *uid = get_uid(); > + const char *launchctl = getenv("GIT_TEST_CRONTAB"); > + if (!launchctl) > + launchctl = "/bin/launchctl"; > + > + strvec_split(&args, launchctl); > + strvec_push(&args, "bootout"); > + strvec_pushf(&args, "gui/%s", uid); > + strvec_push(&args, filename); > + > + result = run_command_v_opt(args.v, 0); > + > + strvec_clear(&args); > + free(uid); > + return result; > +} > + > +static int bootstrap(const char *filename) > +{ > + int result; > + struct strvec args = STRVEC_INIT; > + char *uid = get_uid(); > + const char *launchctl = getenv("GIT_TEST_CRONTAB"); > + if (!launchctl) > + launchctl = "/bin/launchctl"; > + > + strvec_split(&args, launchctl); > + strvec_push(&args, "bootstrap"); > + strvec_pushf(&args, "gui/%s", uid); > + strvec_push(&args, filename); > + > + result = run_command_v_opt(args.v, 0); > + > + strvec_clear(&args); > + free(uid); > + return result; > +} The bootout() and bootstrap() functions seem to be identical except for one string literal. Code could be reduced by refactoring and passing that string literal in as an argument. > +static int remove_plist(enum schedule_priority schedule) > +{ > + const char *frequency = get_frequency(schedule); > + char *name = get_service_name(frequency); > + char *filename = get_service_filename(name); > + int result = bootout(filename); > + free(filename); > + free(name); > + return result; > +} The result of get_service_name() is only ever passed to get_service_filename(). If get_service_filename() made the call to get_service_name() itself, it would free up callers from having to remember to free(name), thus reducing the likelihood of a possible leak. > +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) > +{ > + plist = fopen(filename, "w"); > + > + if (!plist) > + die(_("failed to open '%s'"), filename); You can replace the fopen() and die() with a single call to xfopen(). > + /* bootout might fail if not already running, so ignore */ > + bootout(filename); > + if (bootstrap(filename)) > + die(_("failed to bootstrap service %s"), filename); I'm guessing that 'launchctl bootout' won't print a confusing and unexpected error message if the plist is not presently registered? > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh > @@ -389,12 +389,58 @@ test_expect_success 'stop from existing schedule' ' > +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' > + echo "#!/bin/sh\necho \$@ >>args" >print-args && > + chmod a+x print-args && Note that $@ loses its special magic if not surrounded by quotes, thus acts just like $*. So, either use "$@" or $* depending upon your requirements, but in the case of 'echo', it's just not going to matter at all, so $* is fine. To construct the script, you can do this instead, which is easier to read and handles the 'chmod' for you: write_script print-args <<-\EOF echo $* >>args EOF > + for frequency in hourly daily weekly > + do > + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && > + grep schedule=$frequency "$PLIST" && > + echo "bootout gui/$UID $PLIST" >>expect && > + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 > + done && My gut feeling is that this would be more robust if you manually determine UID in the test script the same way as the git-maintenance command itself does using '/usr/bin/id -u' rather than relying upon inheriting UID from the user's environment. > + # stop does not remove plist files, but boots them out Is this desirable? Should git-maintenance do a better job of cleaning up after itself? ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 2/3] maintenance: use launchctl on macOS 2020-11-03 18:45 ` Eric Sunshine @ 2020-11-03 21:21 ` Derrick Stolee 2020-11-03 22:27 ` Eric Sunshine 2020-11-04 14:17 ` Derrick Stolee 0 siblings, 2 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-03 21:21 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On 11/3/2020 1:45 PM, Eric Sunshine wrote: > On Tue, Nov 3, 2020 at 9:05 AM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> maintenance: use launchctl on macOS > > A few comments below (not necessarily worth a re-roll)... > >> The launchctl command needs to be aligned with a user id in order >> to initialize the command environment. This must be done using >> the 'launchctl bootstrap' subcommand. This subcommand is new as >> of macOS 10.11, which was released in September 2015. Before that >> release the 'launchctl load' subcommand was recommended. The best >> source of information on this transition I have seen is available >> at [2]. > > It's not clear whether or not this is saying that git-maintenance will > dynamically adapt to work with modern and older 'launchctl'. A glance > at the actual code reveals that it knows only about modern > 'bootstrap'. Perhaps this could be a bit clearer by saying that it > only supports modern versions, and that support for older versions can > be added later if needed. (For those of us who are stuck with 10-20 > year old hardware and OS versions, 2015 isn't that long ago.) Yes, this is a strange place to be. How far do we go back to support as many users as possible? How many users will be simultaneously stuck on an old version of macOS _and_ interested in updating to this latest version of Git? Is that worth the extra functionality to detect the the OS version and change commands? The good news is that this patch doesn't lock us in to the boot(strap|out) subcommands too much. We could add in load/unload subcommands for systems that are too old. However, I did think it was prudent to take the currently-recommended option for fear that Apple will completely _delete_ the load/unload options in an upcoming release. This makes me realize that I should update the documentation to give pointers for how to view the schedules for each platform: - Windows: Open "Task Scheduler" - macOS: 'launchctl list | grep org.git-scm.git' - Others: 'crontab -l' >> +static int bootstrap(const char *filename) >> +{ >> + int result; >> + struct strvec args = STRVEC_INIT; >> + char *uid = get_uid(); >> + const char *launchctl = getenv("GIT_TEST_CRONTAB"); >> + if (!launchctl) >> + launchctl = "/bin/launchctl"; >> + >> + strvec_split(&args, launchctl); >> + strvec_push(&args, "bootstrap"); >> + strvec_pushf(&args, "gui/%s", uid); >> + strvec_push(&args, filename); >> + >> + result = run_command_v_opt(args.v, 0); >> + >> + strvec_clear(&args); >> + free(uid); >> + return result; >> +} > > The bootout() and bootstrap() functions seem to be identical except > for one string literal. Code could be reduced by refactoring and > passing that string literal in as an argument. Good point. Or a simple boolean value for "add" or "remove". >> +static int remove_plist(enum schedule_priority schedule) >> +{ >> + const char *frequency = get_frequency(schedule); >> + char *name = get_service_name(frequency); >> + char *filename = get_service_filename(name); >> + int result = bootout(filename); >> + free(filename); >> + free(name); >> + return result; >> +} > > The result of get_service_name() is only ever passed to > get_service_filename(). If get_service_filename() made the call to > get_service_name() itself, it would free up callers from having to > remember to free(name), thus reducing the likelihood of a possible > leak. You're right. In an earlier version I thought I needed to add the name in the XML, but it turns out I did not. >> +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) >> +{ >> + plist = fopen(filename, "w"); >> + >> + if (!plist) >> + die(_("failed to open '%s'"), filename); > > You can replace the fopen() and die() with a single call to xfopen(). Thanks! I'll use that in several places and try to remember next time. >> + /* bootout might fail if not already running, so ignore */ >> + bootout(filename); >> + if (bootstrap(filename)) >> + die(_("failed to bootstrap service %s"), filename); > > I'm guessing that 'launchctl bootout' won't print a confusing and > unexpected error message if the plist is not presently registered? You're right, it does. It also returns with a non-zero exit code. Along with your later suggestion to clear the .plist files, we will want to have several conditions to not error out during a case where the task is not scheduled. >> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh >> @@ -389,12 +389,58 @@ test_expect_success 'stop from existing schedule' ' >> +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' >> + echo "#!/bin/sh\necho \$@ >>args" >print-args && >> + chmod a+x print-args && > > Note that $@ loses its special magic if not surrounded by quotes, thus > acts just like $*. So, either use "$@" or $* depending upon your > requirements, but in the case of 'echo', it's just not going to matter > at all, so $* is fine. > > To construct the script, you can do this instead, which is easier to > read and handles the 'chmod' for you: > > write_script print-args <<-\EOF > echo $* >>args > EOF TIL. thanks. >> + for frequency in hourly daily weekly >> + do >> + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && >> + grep schedule=$frequency "$PLIST" && >> + echo "bootout gui/$UID $PLIST" >>expect && >> + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 >> + done && > > My gut feeling is that this would be more robust if you manually > determine UID in the test script the same way as the git-maintenance > command itself does using '/usr/bin/id -u' rather than relying upon > inheriting UID from the user's environment. Yeah, you're right. Thanks! >> + # stop does not remove plist files, but boots them out > > Is this desirable? Should git-maintenance do a better job of cleaning > up after itself? Yes, let's clear up these .plist files. Thanks! -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 2/3] maintenance: use launchctl on macOS 2020-11-03 21:21 ` Derrick Stolee @ 2020-11-03 22:27 ` Eric Sunshine 2020-11-04 13:33 ` Derrick Stolee 2020-11-04 14:17 ` Derrick Stolee 1 sibling, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-03 22:27 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Tue, Nov 3, 2020 at 4:22 PM Derrick Stolee <stolee@gmail.com> wrote: > On 11/3/2020 1:45 PM, Eric Sunshine wrote: > > It's not clear whether or not this is saying that git-maintenance will > > dynamically adapt to work with modern and older 'launchctl'. A glance > > at the actual code reveals that it knows only about modern > > 'bootstrap'. Perhaps this could be a bit clearer by saying that it > > only supports modern versions, and that support for older versions can > > be added later if needed. (For those of us who are stuck with 10-20 > > year old hardware and OS versions, 2015 isn't that long ago.) > > Yes, this is a strange place to be. How far do we go back to support > as many users as possible? How many users will be simultaneously > stuck on an old version of macOS _and_ interested in updating to this > latest version of Git? Is that worth the extra functionality to detect > the the OS version and change commands? I don't think this patch series needs to answer these questions provided that it doesn't close the door to someone adding older-version support down the road. My review comment was more about the commit message being clearer about the choice -- supporting only recent 'launchctl' -- being made by this series. (And perhaps the documentation could mention that it requires a reasonably modern 'launchctl'.) > This makes me realize that I should update the documentation to give > pointers for how to view the schedules for each platform: > > - Windows: Open "Task Scheduler" > - macOS: 'launchctl list | grep org.git-scm.git' > - Others: 'crontab -l' Good idea. I haven't looked at the documentation, but if it doesn't already, I wonder if it should give examples of how to set these up by hand or how to customize the ones created by git-maintenance itself. I was also wondering if git-maintenance could have a mode in which it generates the template file(s) for you but doesn't actually activate/install it, instead providing instructions for activation/installation. That way, people could modify the scheduling file before actually activating it. However, this may all be outside the scope of the patch series, and could be done later if desired. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 2/3] maintenance: use launchctl on macOS 2020-11-03 22:27 ` Eric Sunshine @ 2020-11-04 13:33 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-04 13:33 UTC (permalink / raw) To: Eric Sunshine Cc: Derrick Stolee via GitGitGadget, Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On 11/3/2020 5:27 PM, Eric Sunshine wrote: > On Tue, Nov 3, 2020 at 4:22 PM Derrick Stolee <stolee@gmail.com> wrote: >> On 11/3/2020 1:45 PM, Eric Sunshine wrote: >>> It's not clear whether or not this is saying that git-maintenance will >>> dynamically adapt to work with modern and older 'launchctl'. A glance >>> at the actual code reveals that it knows only about modern >>> 'bootstrap'. Perhaps this could be a bit clearer by saying that it >>> only supports modern versions, and that support for older versions can >>> be added later if needed. (For those of us who are stuck with 10-20 >>> year old hardware and OS versions, 2015 isn't that long ago.) >> >> Yes, this is a strange place to be. How far do we go back to support >> as many users as possible? How many users will be simultaneously >> stuck on an old version of macOS _and_ interested in updating to this >> latest version of Git? Is that worth the extra functionality to detect >> the the OS version and change commands? > > I don't think this patch series needs to answer these questions > provided that it doesn't close the door to someone adding > older-version support down the road. My review comment was more about > the commit message being clearer about the choice -- supporting only > recent 'launchctl' -- being made by this series. (And perhaps the > documentation could mention that it requires a reasonably modern > 'launchctl'.) Thanks. I'll be sure to make the commit message more clear. >> This makes me realize that I should update the documentation to give >> pointers for how to view the schedules for each platform: >> >> - Windows: Open "Task Scheduler" >> - macOS: 'launchctl list | grep org.git-scm.git' >> - Others: 'crontab -l' > > Good idea. > > I haven't looked at the documentation, but if it doesn't already, I > wonder if it should give examples of how to set these up by hand or > how to customize the ones created by git-maintenance itself. I was > also wondering if git-maintenance could have a mode in which it > generates the template file(s) for you but doesn't actually > activate/install it, instead providing instructions for > activation/installation. That way, people could modify the scheduling > file before actually activating it. However, this may all be outside > the scope of the patch series, and could be done later if desired. Outside of the technical details, the biggest questions I've tried to handle with the background maintenance feature has been to balance customization with ease-of-use. My philosophy is that users fall into a few expertise buckets, and have different expectations: * Beginners will never know about background maintenance and so will never run "git maintenance start" or set the config values. * Intermediate users might discover "git maintenance start" and will appreciate that they don't need to learn cron to set up a good default schedule. * Advanced users will read the documentation and use Git config settings to customize their maintenance tasks and schedule. * Expert users might decide the task schedule available by "git maintenance start" is too restrictive, so they create their own background schedule with custom tasks. They might not even run the maintenance builtin and opt instead for 'git gc' or 'git repack' directly! My main target has been these "intermediate" users who might run the command and forget about it. However, I've also tried to keep the advanced users in mind with the advanced config options available. Your comment about documentation demonstrates a way to serve the advanced and expert users by providing a clear framework for discovering what Git is doing under the hood and how to modify or adapt that to their needs. It is also important to demonstrate how to set up schedules in a similar way without having them be overwritten by a later "git maintenance start" command. I will give this a shot in v2. Thanks. -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 2/3] maintenance: use launchctl on macOS 2020-11-03 21:21 ` Derrick Stolee 2020-11-03 22:27 ` Eric Sunshine @ 2020-11-04 14:17 ` Derrick Stolee 1 sibling, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-04 14:17 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On 11/3/2020 4:21 PM, Derrick Stolee wrote: > On 11/3/2020 1:45 PM, Eric Sunshine wrote: >> On Tue, Nov 3, 2020 at 9:05 AM Derrick Stolee via GitGitGadget >>> +static int remove_plist(enum schedule_priority schedule) >>> +{ >>> + const char *frequency = get_frequency(schedule); >>> + char *name = get_service_name(frequency); >>> + char *filename = get_service_filename(name); >>> + int result = bootout(filename); >>> + free(filename); >>> + free(name); >>> + return result; >>> +} >> >> The result of get_service_name() is only ever passed to >> get_service_filename(). If get_service_filename() made the call to >> get_service_name() itself, it would free up callers from having to >> remember to free(name), thus reducing the likelihood of a possible >> leak. > > You're right. In an earlier version I thought I needed to add the > name in the XML, but it turns out I did not. As I go through the effort to remove get_service_name() I find that actually the name is used in one place in the XML file: + "<key>Label</key><string>%s</string>\n" This "Label" should match the filename, I believe. I can still be more careful about how often this name is actually required. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH 3/3] maintenance: use Windows scheduled tasks 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 1/3] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 2/3] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 ` Derrick Stolee via GitGitGadget 2020-11-03 19:06 ` Eric Sunshine 2020-11-03 20:18 ` [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Junio C Hamano ` (2 subsequent siblings) 5 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-03 14:03 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 181 +++++++++++++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 40 ++++++++- 2 files changed, 218 insertions(+), 3 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index fa0ae63a80..24511fec2e 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1698,6 +1698,187 @@ static int platform_update_schedule(int run_maintenance, int fd) else return remove_plists(); } + +#elif defined(GIT_WINDOWS_NATIVE) + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int remove_task(enum schedule_priority schedule) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + const char *schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; + + strvec_split(&args, schtasks); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int remove_scheduled_tasks(void) +{ + return remove_task(SCHEDULE_HOURLY) || + remove_task(SCHEDULE_DAILY) || + remove_task(SCHEDULE_WEEKLY); +} + +static int schedule_task(const char *exec_path, enum schedule_priority schedule) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *xml, *schtasks; + char *xmlpath; + FILE *xmlfp; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + + xmlpath = xstrfmt("%s/schedule-%s.xml", + the_repository->objects->odb->path, + frequency); + xmlfp = fopen(xmlpath, "w"); + if (!xmlfp) + die(_("failed to open '%s'"), xmlpath); + + xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fprintf(xmlfp, xml); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml= "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(xmlfp, xml, exec_path, exec_path, frequency); + fclose(xmlfp); + + schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; + strvec_split(&args, schtasks); + strvec_pushl(&args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + unlink(xmlpath); + free(xmlpath); + free(name); + return result; +} + +static int add_scheduled_tasks(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_task(exec_path, SCHEDULE_HOURLY) || + schedule_task(exec_path, SCHEDULE_DAILY) || + schedule_task(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_scheduled_tasks(); + else + return remove_scheduled_tasks(); +} + #else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index f0210aa206..73dc0078da 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,7 +389,7 @@ test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt @@ -441,6 +441,40 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' test_cmp expect args ' +test_expect_success MINGW 'start and stop Windows maintenance' ' + echo "echo \$@ >>args" >print-args && + chmod a+x print-args && + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && + cat args && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + for frequency in hourly daily weekly + do + echo "/create /tn Git Maintenance ($frequency) /f /xml .git/objects/schedule-$frequency.xml" >>expect \ + || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + for frequency in hourly daily weekly + do + echo "/delete /tn Git Maintenance ($frequency) /f" >>expect \ + || return 1 + done && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH 3/3] maintenance: use Windows scheduled tasks 2020-11-03 14:03 ` [PATCH 3/3] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-03 19:06 ` Eric Sunshine 2020-11-03 21:23 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-03 19:06 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Tue, Nov 3, 2020 at 9:05 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > There is a deficiency in the current design. Windows has two kinds of > applications: GUI applications that start by "winmain()" and console > applications that start by "main()". Console applications are attached > to a new Console window if they are not already associated with a GUI > application. This means that every hour the scheudled task launches a > command window for the scheduled tasks. Not only is this visually > obtrusive, but it also takes focus from whatever else the user is > doing! I wonder if you could use the technique explained in [1] to prevent the console window from popping up. [1]: https://pureinfotech.com/prevent-command-window-appearing-scheduled-tasks-windows-10/ > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh > @@ -441,6 +441,40 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' > +test_expect_success MINGW 'start and stop Windows maintenance' ' > + echo "echo \$@ >>args" >print-args && > + chmod a+x print-args && Same comments as my review of [2/3] regarding $@ and write_script(). > + rm -f args && > + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && > + cat args && Is this 'cat' leftover debugging gunk? > + # start registers the repo > + git config --get --global maintenance.repo "$(pwd)" && > + > + rm expect && > + for frequency in hourly daily weekly > + do > + echo "/create /tn Git Maintenance ($frequency) /f /xml .git/objects/schedule-$frequency.xml" >>expect \ > + || return 1 > + done && Rather than using >> within the loop, it's often simpler to capture the output of the for-loop in its entirety: for frequency in hourly daily weekly do echo "/create ..." || return 1 done >expect && However, in this case 'printf' may be even simpler: printf "/create /tn ... .git/objects/schedule-%s.xml\n" \ hourly daily weekly >expect && > + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && Too many spaces before the 'git' command. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 3/3] maintenance: use Windows scheduled tasks 2020-11-03 19:06 ` Eric Sunshine @ 2020-11-03 21:23 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-03 21:23 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On 11/3/2020 2:06 PM, Eric Sunshine wrote: > On Tue, Nov 3, 2020 at 9:05 AM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> There is a deficiency in the current design. Windows has two kinds of >> applications: GUI applications that start by "winmain()" and console >> applications that start by "main()". Console applications are attached >> to a new Console window if they are not already associated with a GUI >> application. This means that every hour the scheudled task launches a >> command window for the scheduled tasks. Not only is this visually >> obtrusive, but it also takes focus from whatever else the user is >> doing! > > I wonder if you could use the technique explained in [1] to prevent > the console window from popping up. > > [1]: https://pureinfotech.com/prevent-command-window-appearing-scheduled-tasks-windows-10/ The critical part of that strategy is the "Run whether the user is logged in or not". The resulting option that triggers causes the schtasks command to require a password prompt (or a password passed as a command-line argument). I found that interaction to be too disruptive. >> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> >> --- >> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh >> @@ -441,6 +441,40 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' >> +test_expect_success MINGW 'start and stop Windows maintenance' ' >> + echo "echo \$@ >>args" >print-args && >> + chmod a+x print-args && > > Same comments as my review of [2/3] regarding $@ and write_script(). Noted! >> + rm -f args && >> + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && >> + cat args && > > Is this 'cat' leftover debugging gunk? Yes. Thanks. >> + # start registers the repo >> + git config --get --global maintenance.repo "$(pwd)" && >> + >> + rm expect && >> + for frequency in hourly daily weekly >> + do >> + echo "/create /tn Git Maintenance ($frequency) /f /xml .git/objects/schedule-$frequency.xml" >>expect \ >> + || return 1 >> + done && > > Rather than using >> within the loop, it's often simpler to capture > the output of the for-loop in its entirety: > > for frequency in hourly daily weekly > do > echo "/create ..." || return 1 > done >expect && > > However, in this case 'printf' may be even simpler: > > printf "/create /tn ... .git/objects/schedule-%s.xml\n" \ > hourly daily weekly >expect && Excellent. >> + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && > > Too many spaces before the 'git' command. Thanks! -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2020-11-03 14:03 ` [PATCH 3/3] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-03 20:18 ` Junio C Hamano 2020-11-03 20:21 ` Junio C Hamano 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget 5 siblings, 0 replies; 83+ messages in thread From: Junio C Hamano @ 2020-11-03 20:18 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: git, jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: > This is based on ds/maintenance-part-3. > > After sitting with the background maintenance as it has been cooking, I > wanted to come back around and implement the background maintenance for > Windows. However, I noticed that there were some things bothering me with > background maintenance on my macOS machine. These are detailed in PATCH 2, > but the tl;dr is that 'cron' is not recommended by Apple and instead > 'launchd' satisfies our needs. Nicely done. > This series implements the background scheduling so git maintenance > (start|stop) works on those platforms. I've been operating with these > schedules for a while now without the problems described in the patches. > > There is a particularly annoying case about console windows popping up on > Windows, but PATCH 3 describes a plan to get around that. > > Thanks, -Stolee > > Derrick Stolee (3): > maintenance: extract platform-specific scheduling > maintenance: use launchctl on macOS > maintenance: use Windows scheduled tasks > > builtin/gc.c | 428 +++++++++++++++++++++++++++++++++++++++-- > t/t7900-maintenance.sh | 86 ++++++++- > t/test-lib.sh | 4 + > 3 files changed, 498 insertions(+), 20 deletions(-) > > > base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f > Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v1 > Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v1 > Pull-Request: https://github.com/gitgitgadget/git/pull/776 ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (3 preceding siblings ...) 2020-11-03 20:18 ` [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Junio C Hamano @ 2020-11-03 20:21 ` Junio C Hamano 2020-11-03 21:09 ` Derrick Stolee 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget 5 siblings, 1 reply; 83+ messages in thread From: Junio C Hamano @ 2020-11-03 20:21 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: git, jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: > This is based on ds/maintenance-part-3. Ah, I forgot to ask those on CC list how carefully they read the part 3 of the series, as it's been left on 'seen' for some time, and I do not know if it is ready to start cooking in 'next'. Thanks. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-03 20:21 ` Junio C Hamano @ 2020-11-03 21:09 ` Derrick Stolee 2020-11-03 22:30 ` Junio C Hamano 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-03 21:09 UTC (permalink / raw) To: Junio C Hamano, Derrick Stolee via GitGitGadget Cc: git, jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee On 11/3/2020 3:21 PM, Junio C Hamano wrote: > "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: > >> This is based on ds/maintenance-part-3. > > Ah, I forgot to ask those on CC list how carefully they read the > part 3 of the series, as it's been left on 'seen' for some time, and > I do not know if it is ready to start cooking in 'next'. It has been a while since anyone has commented, and I've been running the patches locally for quite a while. I'd be very happy to see them cook in next. I wasn't quite to the place to be pushy about it. I'm hoping that parts 3 and 4 can make it in time for 2.30, so the feature is universally available for all platforms. I realize that's not entirely up to just me. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-03 21:09 ` Derrick Stolee @ 2020-11-03 22:30 ` Junio C Hamano 2020-11-04 13:02 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Junio C Hamano @ 2020-11-03 22:30 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, git, jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee Derrick Stolee <stolee@gmail.com> writes: > On 11/3/2020 3:21 PM, Junio C Hamano wrote: >> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: >> >>> This is based on ds/maintenance-part-3. >> >> Ah, I forgot to ask those on CC list how carefully they read the >> part 3 of the series, as it's been left on 'seen' for some time, and >> I do not know if it is ready to start cooking in 'next'. > > It has been a while since anyone has commented, and I've been > running the patches locally for quite a while. I'd be very > happy to see them cook in next. I wasn't quite to the place > to be pushy about it. > > I'm hoping that parts 3 and 4 can make it in time for 2.30, > so the feature is universally available for all platforms. > I realize that's not entirely up to just me. After writing this entry in the What's cooking report: * ds/maintenance-part-4 (2020-11-03) 3 commits - maintenance: use Windows scheduled tasks - maintenance: use launchctl on macOS - maintenance: extract platform-specific scheduling (this branch uses ds/maintenance-part-3.) Follow-up on the "maintenance part-3" which introduced scheduled maintenance tasks to support platforms whose native scheduling methods are not 'cron'. I was wondering if I should propose making these two parts into one, so we may be pretty much on the same page. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-03 22:30 ` Junio C Hamano @ 2020-11-04 13:02 ` Derrick Stolee 2020-11-04 17:00 ` Junio C Hamano 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-04 13:02 UTC (permalink / raw) To: Junio C Hamano Cc: Derrick Stolee via GitGitGadget, git, jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee On 11/3/2020 5:30 PM, Junio C Hamano wrote: > Derrick Stolee <stolee@gmail.com> writes: >> I'm hoping that parts 3 and 4 can make it in time for 2.30, >> so the feature is universally available for all platforms. >> I realize that's not entirely up to just me. > > After writing this entry in the What's cooking report: > > * ds/maintenance-part-4 (2020-11-03) 3 commits > - maintenance: use Windows scheduled tasks > - maintenance: use launchctl on macOS > - maintenance: extract platform-specific scheduling > (this branch uses ds/maintenance-part-3.) > > Follow-up on the "maintenance part-3" which introduced scheduled > maintenance tasks to support platforms whose native scheduling > methods are not 'cron'. This is a good summary of this series. > I was wondering if I should propose making these two parts into one, > so we may be pretty much on the same page. I'm happy to have the two topics be merged into one series, but I'd prefer to only re-roll these three patches during review. Of course, by keeping them together we have the ability to re- roll all of the patches, but part-3 has a decent length and has been stable for a while. I defer to what is easiest for you. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-04 13:02 ` Derrick Stolee @ 2020-11-04 17:00 ` Junio C Hamano 2020-11-04 18:43 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Junio C Hamano @ 2020-11-04 17:00 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, git, jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee Derrick Stolee <stolee@gmail.com> writes: > On 11/3/2020 5:30 PM, Junio C Hamano wrote: >> Derrick Stolee <stolee@gmail.com> writes: >>> I'm hoping that parts 3 and 4 can make it in time for 2.30, >>> so the feature is universally available for all platforms. >>> I realize that's not entirely up to just me. >> ... >> I was wondering if I should propose making these two parts into one, >> so we may be pretty much on the same page. > > I'm happy to have the two topics be merged into one series, but > I'd prefer to only re-roll these three patches during review. > Of course, by keeping them together we have the ability to re- > roll all of the patches, but part-3 has a decent length and has > been stable for a while. > > I defer to what is easiest for you. Two topics, one on top of the other, is fine, as long as I can remember (and you help me by reminding) to keep them together, and the bottom one is reasonably solid that I do not have to do the rebasing myself ;-) ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH 0/3] Maintenance IV: Platform-specific background maintenance 2020-11-04 17:00 ` Junio C Hamano @ 2020-11-04 18:43 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-04 18:43 UTC (permalink / raw) To: Junio C Hamano Cc: Derrick Stolee via GitGitGadget, git, jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee On 11/4/2020 12:00 PM, Junio C Hamano wrote: > Derrick Stolee <stolee@gmail.com> writes: > >> On 11/3/2020 5:30 PM, Junio C Hamano wrote: >>> Derrick Stolee <stolee@gmail.com> writes: >>>> I'm hoping that parts 3 and 4 can make it in time for 2.30, >>>> so the feature is universally available for all platforms. >>>> I realize that's not entirely up to just me. >>> ... >>> I was wondering if I should propose making these two parts into one, >>> so we may be pretty much on the same page. >> >> I'm happy to have the two topics be merged into one series, but >> I'd prefer to only re-roll these three patches during review. >> Of course, by keeping them together we have the ability to re- >> roll all of the patches, but part-3 has a decent length and has >> been stable for a while. >> >> I defer to what is easiest for you. > > Two topics, one on top of the other, is fine, as long as I can > remember (and you help me by reminding) to keep them together, > and the bottom one is reasonably solid that I do not have to do the > rebasing myself ;-) Sounds good to me. If I need to re-roll part 3, then I'll consider replacing both parts with a new series. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v2 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (4 preceding siblings ...) 2020-11-03 20:21 ` Junio C Hamano @ 2020-11-04 20:06 ` Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (4 more replies) 5 siblings, 5 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that. Updates in V2 ============= * This is a faster turnaround for a v2 than I would normally like, but Eric inspired extra documentation about how to customize background schedules. * New extensions to git-maintenance.txt include guidelines for inspecting what git maintenance start does and how to customize beyond that. This includes a new PATCH 2 that includes documentation for 'cron' on non-macOS non-Windows systems. * Several improvements, especially in the tests, are included. * While testing manually, I noticed that somehow I had incorrectly had an opening <dict> tag instead of a closing </dict> tag in the hourly format on macOS. I found that the xmllint tool can verify the XML format of a file, which catches the bug. This seems like a good approach since the test is macOS-only. Does anyone have concerns about adding this dependency? Thanks, -Stolee Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks Documentation/git-maintenance.txt | 119 +++++++++ builtin/gc.c | 428 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 83 +++++- t/test-lib.sh | 4 + 4 files changed, 614 insertions(+), 20 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v2 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v2 Pull-Request: https://github.com/gitgitgadget/git/pull/776 Range-diff vs v1: 1: d35f1aa162 = 1: d35f1aa162 maintenance: extract platform-specific scheduling -: ---------- > 2: 709a173720 maintenance: include 'cron' details in docs 2: 832fdf1687 ! 3: 0fafd75d10 maintenance: use launchctl on macOS @@ Commit message and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_CRONTAB. + An earlier version of this patch accidentally had an opening + "<dict>" tag when it should have had a closing "</dict>" tag. This + was caught during manual testing with actual 'launchctl' commands, + but we do not want to update developers' tasks when running tests. + It appears that macOS includes the "xmllint" tool which can verify + the XML format, so call it from the macOS-specific tests to ensure + the .plist files are well-formatted. + + Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> + ## Documentation/git-maintenance.txt ## +@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do use the full path and + executing the correct binaries in your schedule. + + ++BACKGROUND MAINTENANCE ON MACOS SYSTEMS ++--------------------------------------- ++ ++While macOS technically supports `cron`, using `crontab -e` requires ++elevated privileges and the executed process do not have a full user ++context. Without a full user context, Git and its credential helpers ++cannot access stored credentials, so some maintenance tasks are not ++functional. ++ ++Instead, `git maintenance start` interacts with the `launchctl` tool, ++which is the recommended way to ++https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS]. ++ ++Scheduling maintenance through `git maintenance (start|stop)` requires ++some `launchctl` features available only in macOS 10.11 or later. ++ ++Your user-specific scheduled tasks are stored as XML-formatted `.plist` ++files in `~/Library/LaunchAgents/`. You can see the currently-registered ++tasks using the following command: ++ ++----------------------------------------------------------------------- ++$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git ++org.git-scm.git.daily.plist ++org.git-scm.git.hourly.plist ++org.git-scm.git.weekly.plist ++----------------------------------------------------------------------- ++ ++One task is registered for each `--schedule=<frequency>` option. To ++inspect how the XML format describes each schedule, open one of these ++`.plist` files in an editor and inspect the `<array>` element following ++the `<key>StartCalendarInterval</key>` element. ++ ++`git maintenance start` will overwrite these files and register the ++tasks again with `launchctl`, so any customizations should be done by ++creating your own `.plist` files with distinct names. Similarly, the ++`git maintenance stop` command will unregister the tasks with `launchctl` ++and delete the `.plist` files. ++ ++To create more advanced customizations to your background tasks, see ++https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation] ++for more information. ++ ++ + GIT + --- + Part of the linkgit:git[1] suite + ## builtin/gc.c ## @@ builtin/gc.c: static int maintenance_unregister(void) return run_command(&config_unset); @@ builtin/gc.c: static int maintenance_unregister(void) + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" -+ "<dict>\n"; ++ "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && ++ xmllint "$PLIST" >/dev/null && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect && + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 3: a9221cc4aa ! 4: 84eb44de31 maintenance: use Windows scheduled tasks @@ Commit message short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. + Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> + ## Documentation/git-maintenance.txt ## +@@ Documentation/git-maintenance.txt: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSy + for more information. + + ++BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS ++----------------------------------------- ++ ++Windows does not support `cron` and instead has its own system for ++scheduling background tasks. The `git maintenance start` command uses ++the `schtasks` command to submit tasks to this system. You can inspect ++all background tasks using the Task Scheduler application. The tasks ++added by Git have names of the form `Git Maintenance (<frequency>)`. ++The Task Scheduler GUI has ways to inspect these tasks, but you can also ++export the tasks to XML files and view the details there. ++ ++Note that since Git is a console application, these background tasks ++create a console window visible to the current user. This can be changed ++manually by selecting the "Run whether user is logged in or not" option ++in Task Scheduler. This change requires a password input, which is why ++`git maintenance start` does not select it by default. ++ ++If you want to customize the background tasks, please rename the tasks ++so future calls to `git maintenance (start|stop)` do not overwrite your ++custom tasks. ++ ++ + GIT + --- + Part of the linkgit:git[1] suite + ## builtin/gc.c ## @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) else @@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop ma ' +test_expect_success MINGW 'start and stop Windows maintenance' ' -+ echo "echo \$@ >>args" >print-args && -+ chmod a+x print-args && ++ write_script print-args <<-\EOF && ++ echo $* >>args ++ EOF + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && -+ cat args && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ rm expect && + for frequency in hourly daily weekly + do -+ echo "/create /tn Git Maintenance ($frequency) /f /xml .git/objects/schedule-$frequency.xml" >>expect \ -+ || return 1 -+ done && ++ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ ++ $frequency $frequency ++ done >expect && + test_cmp expect args && + + rm -f args && -+ GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && ++ GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && -+ for frequency in hourly daily weekly -+ do -+ echo "/delete /tn Git Maintenance ($frequency) /f" >>expect \ -+ || return 1 -+ done && ++ printf "/delete /tn Git Maintenance (%s) /f\n" \ ++ hourly daily weekly >expect && + test_cmp expect args +' + -- gitgitgadget ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v2 1/4] maintenance: extract platform-specific scheduling 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 ` Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget ` (3 subsequent siblings) 4 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a1..c1f7d9bdc2 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,7 +1494,7 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int platform_update_schedule(int run_maintenance, int fd) { int result = 0; int in_old_region = 0; @@ -1503,11 +1503,6 @@ static int update_background_schedule(int run_maintenance) FILE *cron_list, *cron_in; const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); crontab_name = getenv("GIT_TEST_CRONTAB"); if (!crontab_name) @@ -1516,12 +1511,11 @@ static int update_background_schedule(int run_maintenance) strvec_split(&crontab_list.args, crontab_name); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); } /* Ignore exit code, as an empty crontab will return error. */ @@ -1531,7 +1525,7 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); strvec_split(&crontab_edit.args, crontab_name); @@ -1539,8 +1533,7 @@ static int update_background_schedule(int run_maintenance) crontab_edit.git_cmd = 0; if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; + return error(_("failed to run 'crontab'; your system might not support 'cron'")); } cron_in = fdopen(crontab_edit.in, "w"); @@ -1586,13 +1579,24 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; - } - fclose(cron_list); + else + fclose(cron_list); + return result; +} + +static int update_background_schedule(int run_maintenance) +{ + int result; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + result = platform_update_schedule(run_maintenance, lk.tempfile->fd); -cleanup: rollback_lock_file(&lk); return result; } -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v2 2/4] maintenance: include 'cron' details in docs 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 ` Derrick Stolee via GitGitGadget 2020-11-11 7:10 ` Eric Sunshine 2020-11-04 20:06 ` [PATCH v2 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget ` (2 subsequent siblings) 4 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc..4c7aac877d 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is `cron`. This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `<path>` string is loaded to specifically use the location for the +`git` executable used in the `git maintenance start` command. This allows +for multiple versions to be compatible. However, if the same user runs +`git maintenance start` with multiple Git executables, then only the +latest executable will be used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config located at `~/.gitconfig`. +The `git maintenance` process then determines which maintenance tasks +are configured to run on each repository with each `<frequency>` using +the `maintenance.<task>.schedule` config options. These values are loaded +from the global or repository config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] +for advanced scheduling techniques. Please do use the full path and +`--exec-path` techniques from the default schedule to ensure you are +executing the correct binaries in your schedule. + GIT --- -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v2 2/4] maintenance: include 'cron' details in docs 2020-11-04 20:06 ` [PATCH v2 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-11-11 7:10 ` Eric Sunshine 0 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2020-11-11 7:10 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Wed, Nov 4, 2020 at 3:06 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > Advanced and expert users may want to know how 'git maintenance start' > schedules background maintenance in order to customize their own > schedules beyond what the maintenance.* config values allow. Start a new > set of sections in git-maintenance.txt that describe how 'cron' is used > to run these tasks. > > This is particularly valuable for users who want to inspect what Git is > doing or for users who want to customize the schedule further. Having a > baseline can provide a way forward for users who have never worked with > cron schedules. A few comments below, not necessarily worth a re-roll... > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt > @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with > +The standard mechanism for scheduling background tasks on POSIX systems > +is `cron`. This tool executes commands based on a given schedule. The It is common in Git (and other Unix) documentation to refer to a Unix tool by its man page reference. So, for instance, instead of `cron`, we would say cron(8) since the `cron` man page is in section 8 of the Unix manual. > +The `<path>` string is loaded to specifically use the location for the The word "loaded" sounds odd in this context. > +`git` executable used in the `git maintenance start` command. This allows > +for multiple versions to be compatible. However, if the same user runs > +`git maintenance start` with multiple Git executables, then only the > +latest executable will be used. I had to read this paragraph four or five times to understand what it is trying to say (assuming I do understand it). Perhaps it can be rewritten more succinctly something like this: The `crontab` entry specifies the full path of the `git` executable to ensure that the `git` command run is the same one with which `git maintenance start` was issued independent of `PATH`. > +These commands use `git for-each-repo --config=maintenance.repo` to run > +`git maintenance run --schedule=<frequency>` on each repository listed in > +the multi-valued `maintenance.repo` config option. These are typically > +loaded from the user-specific global config located at `~/.gitconfig`. I wonder if mentioning `~/.gitconfig` explicitly is wise since it might also be at $XDG_CONFIG_HOME/git/config or some other location on Windows. Perhaps it would be sufficient to mention only "global Git configuration" or something. > +If the config values are insufficient to achieve your desired background > +maintenance schedule, then you can create your own schedule. If you run > +`crontab -e`, then an editor will load with your user-specific `cron` > +schedule. In that editor, you can add your own schedule lines. You could > +start by adapting the default schedule listed earlier, or you could read > +https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] For consistency with other parts of Git documentation, it might be better to reference the `crontab` man page simply as crontab(5) rather than providing an explicit URL. > +for advanced scheduling techniques. Please do use the full path and > +`--exec-path` techniques from the default schedule to ensure you are > +executing the correct binaries in your schedule. ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v2 3/4] maintenance: use launchctl on macOS 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 ` Derrick Stolee via GitGitGadget 2020-11-11 8:12 ` Eric Sunshine 2020-11-04 20:06 ` [PATCH v2 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 4 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_CRONTAB. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format, so call it from the macOS-specific tests to ensure the .plist files are well-formatted. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 43 ++++++ builtin/gc.c | 209 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 53 +++++++- t/test-lib.sh | 4 + 4 files changed, 306 insertions(+), 3 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 4c7aac877d..451ebac131 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,49 @@ for advanced scheduling techniques. Please do use the full path and executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process do not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to +https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS]. + +Scheduling maintenance through `git maintenance (start|stop)` requires +some `launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=<frequency>` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `<array>` element following +the `<key>StartCalendarInterval</key>` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation] +for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index c1f7d9bdc2..7604064a8d 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,214 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +#if defined(__APPLE__) + +static char *get_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *get_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_uid(void) +{ + struct strbuf output = STRBUF_INIT; + struct child_process id = CHILD_PROCESS_INIT; + + strvec_pushl(&id.args, "/usr/bin/id", "-u", NULL); + if (capture_command(&id, &output, 0)) + die(_("failed to discover user id")); + + strbuf_trim_trailing_newline(&output); + return strbuf_detach(&output, NULL); +} + +static int bootout(const char *filename) +{ + int result; + struct strvec args = STRVEC_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + + strvec_split(&args, launchctl); + strvec_push(&args, "bootout"); + strvec_pushf(&args, "gui/%s", uid); + strvec_push(&args, filename); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(uid); + return result; +} + +static int bootstrap(const char *filename) +{ + int result; + struct strvec args = STRVEC_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + + strvec_split(&args, launchctl); + strvec_push(&args, "bootstrap"); + strvec_pushf(&args, "gui/%s", uid); + strvec_push(&args, filename); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(uid); + return result; +} + +static int remove_plist(enum schedule_priority schedule) +{ + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + int result = bootout(filename); + free(filename); + free(name); + return result; +} + +static int remove_plists(void) +{ + return remove_plist(SCHEDULE_HOURLY) || + remove_plist(SCHEDULE_DAILY) || + remove_plist(SCHEDULE_WEEKLY); +} + +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = fopen(filename, "w"); + + if (!plist) + die(_("failed to open '%s'"), filename); + + preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + + /* bootout might fail if not already running, so ignore */ + bootout(filename); + if (bootstrap(filename)) + die(_("failed to bootstrap service %s"), filename); + + fclose(plist); + free(filename); + free(name); + return 0; +} + +static int add_plists(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_plist(exec_path, SCHEDULE_HOURLY) || + schedule_plist(exec_path, SCHEDULE_DAILY) || + schedule_plist(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_plists(); + else + return remove_plists(); +} +#else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1585,6 +1793,7 @@ static int platform_update_schedule(int run_maintenance, int fd) fclose(cron_list); return result; } +#endif static int update_background_schedule(int run_maintenance) { diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..1c43b34a93 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,12 +389,59 @@ test_expect_success 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' + echo "#!/bin/sh\necho \$@ >>args" >print-args && + chmod a+x print-args && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + # ~/Library/LaunchAgents + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + xmllint "$PLIST" >/dev/null && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect && + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + # stop does not remove plist files, but boots them out + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect || return 1 + done && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..620ffbf3af 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1703,6 +1703,10 @@ test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' +test_lazy_prereq MACOS_MAINTENANCE ' + launchctl list +' + # Ensure that no test accidentally triggers a Git command # that runs 'crontab', affecting a user's cron schedule. # Tests that verify the cron integration must set this locally -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v2 3/4] maintenance: use launchctl on macOS 2020-11-04 20:06 ` [PATCH v2 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-11 8:12 ` Eric Sunshine 2020-11-12 13:42 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-11 8:12 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Wed, Nov 4, 2020 at 3:06 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > [...] > The solution is to switch from cron to the Apple-recommended [1] > 'launchd' tool. > [...] > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt > +While macOS technically supports `cron`, using `crontab -e` requires > +elevated privileges and the executed process do not have a full user Either s/process/processes/ or s/do/does/ > +context. Without a full user context, Git and its credential helpers > +cannot access stored credentials, so some maintenance tasks are not > +functional. Nicely explained. > +Instead, `git maintenance start` interacts with the `launchctl` tool, > +which is the recommended way to > +https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS]. Nit: I worry a bit about links to Apple documentation becoming outdated. It might not hurt to omit this link altogether, or perhaps demote it to a footnote (which might allow it to be somewhat usable even when Git documentation is rendered into something other than HTML). > +Scheduling maintenance through `git maintenance (start|stop)` requires > +some `launchctl` features available only in macOS 10.11 or later. Nit: This leaves the reader wondering what modern features are needed. Would it make sense to mention that "bootstrap" is used in place of "load" in older versions of 'launchctl'? > +Your user-specific scheduled tasks are stored as XML-formatted `.plist` > +files in `~/Library/LaunchAgents/`. You can see the currently-registered > +tasks using the following command: > + > +----------------------------------------------------------------------- > +$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git Alternately (unimportant): ls ~/Library/LaunchAgents/org.git-scm.git.* although that would emit "No such file" if you don't have any registered, which might suggest: find ~/Library/LaunchAgents -name 'org.git-scm.git.*' > +To create more advanced customizations to your background tasks, see > +https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation] > +for more information. I really worry about this sort of URL becoming outdated. Would it make sense instead to just point the user at the man page, launchd.plist(5)? It's not quite the same, as it doesn't provide the range of examples as the URL you cite, but it should get the user started. > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1491,6 +1491,214 @@ static int maintenance_unregister(void) > +static int remove_plist(enum schedule_priority schedule) > +{ > + const char *frequency = get_frequency(schedule); > + char *name = get_service_name(frequency); > + char *filename = get_service_filename(name); > + int result = bootout(filename); > + free(filename); > + free(name); > + return result; > +} > > +static int remove_plists(void) > +{ > + return remove_plist(SCHEDULE_HOURLY) || > + remove_plist(SCHEDULE_DAILY) || > + remove_plist(SCHEDULE_WEEKLY); > +} The new documentation you added says that the plist files will be deleted after they are deregistered using launchctl, but I don't see anything actually deleting them. Am I missing something obvious? > +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) > +{ > + plist = fopen(filename, "w"); > + if (!plist) > + die(_("failed to open '%s'"), filename); As mentioned previously, these could be replaced with a simple xfopen(). In fact, I'm having trouble seeing changes in this re-roll which you had planned on making, such as consolidating the repeated code in bootout() and bootstrap(), and ensuring that bootout() doesn't complain if the plist files are already missing, and so forth. Did you opt to not make those changes? (Which would be fine; they were minor suggestions.) > + preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" > + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" > + "<plist version=\"1.0\">" > + "<dict>\n" > + "<key>Label</key><string>%s</string>\n" > + "<key>ProgramArguments</key>\n" > + "<array>\n" > + "<string>%s/git</string>\n" > + "<string>--exec-path=%s</string>\n" > + "<string>for-each-repo</string>\n" > + "<string>--config=maintenance.repo</string>\n" > + "<string>maintenance</string>\n" > + "<string>run</string>\n" > + "<string>--schedule=%s</string>\n" > + "</array>\n" > + "<key>StartCalendarInterval</key>\n" > + "<array>\n"; > + fprintf(plist, preamble, name, exec_path, exec_path, frequency); The Git test framework ensures that this will be written into the test directory rather than the user's actual ~/Library/LaunchAgents directory during testing. Okay. > +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' > + echo "#!/bin/sh\necho \$@ >>args" >print-args && > + chmod a+x print-args && Earlier review already mentioned write_script() and "$@". (Not necessarily worth a re-roll.) > + for frequency in hourly daily weekly > + do > + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && > + xmllint "$PLIST" >/dev/null && Do we really need to suppress xmllint's stdout? > + grep schedule=$frequency "$PLIST" && > + echo "bootout gui/$UID $PLIST" >>expect && > + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 > + done && > + test_cmp expect args && > + > + rm -f args && > + GIT_TEST_CRONTAB="./print-args" git maintenance stop && There is still an extra space between the closing quote and git command (mentioned previously). > + # stop does not unregister the repo > + git config --get --global maintenance.repo "$(pwd)" && > + > + # stop does not remove plist files, but boots them out Documentation added in this re-roll claims that the plist files do get deleted. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v2 3/4] maintenance: use launchctl on macOS 2020-11-11 8:12 ` Eric Sunshine @ 2020-11-12 13:42 ` Derrick Stolee 2020-11-12 16:43 ` Eric Sunshine 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-12 13:42 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On 11/11/2020 3:12 AM, Eric Sunshine wrote: > On Wed, Nov 4, 2020 at 3:06 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> [...] >> The solution is to switch from cron to the Apple-recommended [1] >> 'launchd' tool. >> [...] >> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> >> --- >> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt >> +While macOS technically supports `cron`, using `crontab -e` requires >> +elevated privileges and the executed process do not have a full user > > Either s/process/processes/ or s/do/does/ > >> +context. Without a full user context, Git and its credential helpers >> +cannot access stored credentials, so some maintenance tasks are not >> +functional. > > Nicely explained. > >> +Instead, `git maintenance start` interacts with the `launchctl` tool, >> +which is the recommended way to >> +https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS]. > > Nit: I worry a bit about links to Apple documentation becoming > outdated. It might not hurt to omit this link altogether, or perhaps > demote it to a footnote (which might allow it to be somewhat usable > even when Git documentation is rendered into something other than > HTML). > >> +Scheduling maintenance through `git maintenance (start|stop)` requires >> +some `launchctl` features available only in macOS 10.11 or later. > > Nit: This leaves the reader wondering what modern features are needed. > Would it make sense to mention that "bootstrap" is used in place of > "load" in older versions of 'launchctl'? > >> +Your user-specific scheduled tasks are stored as XML-formatted `.plist` >> +files in `~/Library/LaunchAgents/`. You can see the currently-registered >> +tasks using the following command: >> + >> +----------------------------------------------------------------------- >> +$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git > > Alternately (unimportant): > > ls ~/Library/LaunchAgents/org.git-scm.git.* > > although that would emit "No such file" if you don't have any > registered, which might suggest: > > find ~/Library/LaunchAgents -name 'org.git-scm.git.*' > >> +To create more advanced customizations to your background tasks, see >> +https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation] >> +for more information. > > I really worry about this sort of URL becoming outdated. Would it make > sense instead to just point the user at the man page, > launchd.plist(5)? It's not quite the same, as it doesn't provide the > range of examples as the URL you cite, but it should get the user > started. I shared similar concerns. I'll use the man page references instead. All of the information should be a short web search away after the user is given the right terminology. >> diff --git a/builtin/gc.c b/builtin/gc.c >> @@ -1491,6 +1491,214 @@ static int maintenance_unregister(void) >> +static int remove_plist(enum schedule_priority schedule) >> +{ >> + const char *frequency = get_frequency(schedule); >> + char *name = get_service_name(frequency); >> + char *filename = get_service_filename(name); >> + int result = bootout(filename); >> + free(filename); >> + free(name); >> + return result; >> +} >> >> +static int remove_plists(void) >> +{ >> + return remove_plist(SCHEDULE_HOURLY) || >> + remove_plist(SCHEDULE_DAILY) || >> + remove_plist(SCHEDULE_WEEKLY); >> +} > > The new documentation you added says that the plist files will be > deleted after they are deregistered using launchctl, but I don't see > anything actually deleting them. Am I missing something obvious? As mentioned below, this was a change that I made but somehow lost while juggling multiple copies of my branch. >> +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) >> +{ >> + plist = fopen(filename, "w"); >> + if (!plist) >> + die(_("failed to open '%s'"), filename); > > As mentioned previously, these could be replaced with a simple xfopen(). > > In fact, I'm having trouble seeing changes in this re-roll which you > had planned on making, such as consolidating the repeated code in > bootout() and bootstrap(), and ensuring that bootout() doesn't > complain if the plist files are already missing, and so forth. Did you > opt to not make those changes? (Which would be fine; they were minor > suggestions.) No, I definitely made those changes _somewhere_ but I must have gotten confused as to which of my machines had those changes. I guess that's part of the risk of testing across three platforms. Thank you for noticing, and I'll be more careful from now on. >> +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' >> + echo "#!/bin/sh\necho \$@ >>args" >print-args && >> + chmod a+x print-args && > > Earlier review already mentioned write_script() and "$@". (Not > necessarily worth a re-roll.) I'm going to go back to all of your earlier comments to make sure they are _actually_ applied in v3. >> + for frequency in hourly daily weekly >> + do >> + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && >> + xmllint "$PLIST" >/dev/null && > > Do we really need to suppress xmllint's stdout? It outputs the XML itself. Maybe there is a command to stop that from happening, but nulling stdout keeps the test log clean. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v2 3/4] maintenance: use launchctl on macOS 2020-11-12 13:42 ` Derrick Stolee @ 2020-11-12 16:43 ` Eric Sunshine 0 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2020-11-12 16:43 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Thu, Nov 12, 2020 at 8:43 AM Derrick Stolee <stolee@gmail.com> wrote: > On 11/11/2020 3:12 AM, Eric Sunshine wrote: > > On Wed, Nov 4, 2020 at 3:06 PM Derrick Stolee via GitGitGadget > > <gitgitgadget@gmail.com> wrote: > >> + xmllint "$PLIST" >/dev/null && > > > > Do we really need to suppress xmllint's stdout? > > It outputs the XML itself. Maybe there is a command to stop that from > happening, but nulling stdout keeps the test log clean. xmllint's --noout option should do the trick. ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v2 4/4] maintenance: use Windows scheduled tasks 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2020-11-04 20:06 ` [PATCH v2 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 ` Derrick Stolee via GitGitGadget 2020-11-11 8:59 ` Eric Sunshine 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 4 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-04 20:06 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 181 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 36 +++++- 3 files changed, 236 insertions(+), 3 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 451ebac131..f4f6a4091b 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -316,6 +316,28 @@ https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSy for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance (<frequency>)`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 7604064a8d..80f43a59ce 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1698,6 +1698,187 @@ static int platform_update_schedule(int run_maintenance, int fd) else return remove_plists(); } + +#elif defined(GIT_WINDOWS_NATIVE) + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int remove_task(enum schedule_priority schedule) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + const char *schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; + + strvec_split(&args, schtasks); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int remove_scheduled_tasks(void) +{ + return remove_task(SCHEDULE_HOURLY) || + remove_task(SCHEDULE_DAILY) || + remove_task(SCHEDULE_WEEKLY); +} + +static int schedule_task(const char *exec_path, enum schedule_priority schedule) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *xml, *schtasks; + char *xmlpath; + FILE *xmlfp; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + + xmlpath = xstrfmt("%s/schedule-%s.xml", + the_repository->objects->odb->path, + frequency); + xmlfp = fopen(xmlpath, "w"); + if (!xmlfp) + die(_("failed to open '%s'"), xmlpath); + + xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fprintf(xmlfp, xml); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml= "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(xmlfp, xml, exec_path, exec_path, frequency); + fclose(xmlfp); + + schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; + strvec_split(&args, schtasks); + strvec_pushl(&args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + unlink(xmlpath); + free(xmlpath); + free(name); + return result; +} + +static int add_scheduled_tasks(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_task(exec_path, SCHEDULE_HOURLY) || + schedule_task(exec_path, SCHEDULE_DAILY) || + schedule_task(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_scheduled_tasks(); + else + return remove_scheduled_tasks(); +} + #else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 1c43b34a93..e7ad130cbc 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,7 +389,7 @@ test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt @@ -442,6 +442,36 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' test_cmp expect args ' +test_expect_success MINGW 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + EOF + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + for frequency in hourly daily weekly + do + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ + $frequency $frequency + done >expect && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v2 4/4] maintenance: use Windows scheduled tasks 2020-11-04 20:06 ` [PATCH v2 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-11 8:59 ` Eric Sunshine 2020-11-12 13:56 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-11 8:59 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Derrick Stolee, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On Wed, Nov 4, 2020 at 3:06 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > Git's background maintenance uses cron by default, but this is not > available on Windows. Instead, integrate with Task Scheduler. > [...] > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1698,6 +1698,187 @@ static int platform_update_schedule(int run_maintenance, int fd) > +static int schedule_task(const char *exec_path, enum schedule_priority schedule) > +{ > + xmlpath = xstrfmt("%s/schedule-%s.xml", > + the_repository->objects->odb->path, > + frequency); Am I reading correctly that it is writing this throwaway XML file into the Git object directory? Would writing to a temporary directory make more sense? (Not worth a re-roll.) > + xmlfp = fopen(xmlpath, "w"); > + if (!xmlfp) > + die(_("failed to open '%s'"), xmlpath); Could use xfopen() as mentioned previously. > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh > @@ -442,6 +442,36 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' > +test_expect_success MINGW 'start and stop Windows maintenance' ' > + for frequency in hourly daily weekly > + do > + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ > + $frequency $frequency Nit: You lost the `|| return 1` which was present in the previous version. True, it's very unlikely that `printf` could fail, but having the `|| return 1` there makes it easier for the reader's eye to glide over the code without having to worry about whether it is handling error conditions correctly, thus reduces cognitive load. > + done >expect && Rather than a loop, you could just use: printf "/create ... (%s) /f /xml ...schedule-%s.xml\n" \ hourly hourly daily daily weekly weekly >expect && though it's subjective as to whether that is an improvement. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v2 4/4] maintenance: use Windows scheduled tasks 2020-11-11 8:59 ` Eric Sunshine @ 2020-11-12 13:56 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-12 13:56 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee On 11/11/2020 3:59 AM, Eric Sunshine wrote: > On Wed, Nov 4, 2020 at 3:06 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> Git's background maintenance uses cron by default, but this is not >> available on Windows. Instead, integrate with Task Scheduler. >> [...] >> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> >> --- >> diff --git a/builtin/gc.c b/builtin/gc.c >> @@ -1698,6 +1698,187 @@ static int platform_update_schedule(int run_maintenance, int fd) >> +static int schedule_task(const char *exec_path, enum schedule_priority schedule) >> +{ >> + xmlpath = xstrfmt("%s/schedule-%s.xml", >> + the_repository->objects->odb->path, >> + frequency); > > Am I reading correctly that it is writing this throwaway XML file into > the Git object directory? Would writing to a temporary directory make > more sense? (Not worth a re-roll.) A temp directory is a good idea. >> + xmlfp = fopen(xmlpath, "w"); >> + if (!xmlfp) >> + die(_("failed to open '%s'"), xmlpath); > > Could use xfopen() as mentioned previously. > >> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh >> @@ -442,6 +442,36 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' >> +test_expect_success MINGW 'start and stop Windows maintenance' ' >> + for frequency in hourly daily weekly >> + do >> + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ >> + $frequency $frequency > > Nit: You lost the `|| return 1` which was present in the previous > version. True, it's very unlikely that `printf` could fail, but having > the `|| return 1` there makes it easier for the reader's eye to glide > over the code without having to worry about whether it is handling > error conditions correctly, thus reduces cognitive load. > >> + done >expect && > > Rather than a loop, you could just use: > > printf "/create ... (%s) /f /xml ...schedule-%s.xml\n" \ > hourly hourly daily daily weekly weekly >expect && > > though it's subjective as to whether that is an improvement. It's sufficient, and avoids issues with deep tabbing and chaining "|| return 1"". Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget ` (3 preceding siblings ...) 2020-11-04 20:06 ` [PATCH v2 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 ` Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (5 more replies) 4 siblings, 6 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that. Updates in V3 ============= * This actually includes the feedback responses I had intended for v2. Sorry about that! * One major change is the use of a 'struct child_process' instead of just run_command_v_opt() so we can suppress error messages from the schedule helpers. We will rely on exit code and present our own error messages, as necessary. * Some doc and test fixes. Updates in V2 ============= * This is a faster turnaround for a v2 than I would normally like, but Eric inspired extra documentation about how to customize background schedules. * New extensions to git-maintenance.txt include guidelines for inspecting what git maintenance start does and how to customize beyond that. This includes a new PATCH 2 that includes documentation for 'cron' on non-macOS non-Windows systems. * Several improvements, especially in the tests, are included. * While testing manually, I noticed that somehow I had incorrectly had an opening <dict> tag instead of a closing </dict> tag in the hourly format on macOS. I found that the xmllint tool can verify the XML format of a file, which catches the bug. This seems like a good approach since the test is macOS-only. Does anyone have concerns about adding this dependency? Thanks, -Stolee cc: jrnieder@gmail.com [jrnieder@gmail.com], jonathantanmy@google.com [jonathantanmy@google.com], sluongng@gmail.com [sluongng@gmail.com]cc: Derrick Stolee stolee@gmail.com [stolee@gmail.com]cc: Đoàn Trần Công Danh congdanhqx@gmail.com [congdanhqx@gmail.com]cc: Martin Ågren martin.agren@gmail.com [martin.agren@gmail.com]cc: Eric Sunshine sunshine@sunshineco.com [sunshine@sunshineco.com]cc: Derrick Stolee stolee@gmail.com [stolee@gmail.com] Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks Documentation/git-maintenance.txt | 116 +++++++++ builtin/gc.c | 417 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 75 +++++- t/test-lib.sh | 4 + 4 files changed, 592 insertions(+), 20 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v3 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v3 Pull-Request: https://github.com/gitgitgadget/git/pull/776 Range-diff vs v2: 1: d35f1aa162 = 1: d35f1aa162 maintenance: extract platform-specific scheduling 2: 709a173720 ! 2: 0dfe53092e maintenance: include 'cron' details in docs @@ Commit message baseline can provide a way forward for users who have never worked with cron schedules. + Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## Documentation/git-maintenance.txt ## @@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems -+is `cron`. This tool executes commands based on a given schedule. The ++is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + @@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + -+The `<path>` string is loaded to specifically use the location for the -+`git` executable used in the `git maintenance start` command. This allows -+for multiple versions to be compatible. However, if the same user runs -+`git maintenance start` with multiple Git executables, then only the -+latest executable will be used. ++The `crontab` entry specifies the full path of the `git` executable to ++ensure that the executed `git` command is the same one with which ++`git maintenance start` was issued independent of `PATH`. If the same user ++runs `git maintenance start` with multiple Git executables, then only the ++latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically -+loaded from the user-specific global config located at `~/.gitconfig`. -+The `git maintenance` process then determines which maintenance tasks -+are configured to run on each repository with each `<frequency>` using -+the `maintenance.<task>.schedule` config options. These values are loaded -+from the global or repository config values. ++loaded from the user-specific global config. The `git maintenance` process ++then determines which maintenance tasks are configured to run on each ++repository with each `<frequency>` using the `maintenance.<task>.schedule` ++config options. These values are loaded from the global or repository ++config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read -+https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] -+for advanced scheduling techniques. Please do use the full path and -+`--exec-path` techniques from the default schedule to ensure you are -+executing the correct binaries in your schedule. ++the crontab(5) documentation for advanced scheduling techniques. Please ++do use the full path and `--exec-path` techniques from the default ++schedule to ensure you are executing the correct binaries in your ++schedule. + GIT 3: 0fafd75d10 ! 3: 1629bcfcf8 maintenance: use launchctl on macOS @@ Commit message of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available - at [2]. + at [2]. The current design does not preclude a future version that + detects the available fatures of 'launchctl' to use the older + commands. However, it is best to rely on the newest version since + Apple might completely remove the deprecated version on short + notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ @@ Commit message Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## Documentation/git-maintenance.txt ## -@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do use the full path and - executing the correct binaries in your schedule. +@@ Documentation/git-maintenance.txt: schedule to ensure you are executing the correct binaries in your + schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires -+elevated privileges and the executed process do not have a full user ++elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, -+which is the recommended way to -+https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS]. -+ -+Scheduling maintenance through `git maintenance (start|stop)` requires -+some `launchctl` features available only in macOS 10.11 or later. ++which is the recommended way to schedule timed jobs in macOS. Scheduling ++maintenance through `git maintenance (start|stop)` requires some ++`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- -+$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git ++$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist @@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see -+https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation] -+for more information. ++launchctl.plist(5) for more information. + + GIT @@ builtin/gc.c: static int maintenance_unregister(void) + return strbuf_detach(&output, NULL); +} + -+static int bootout(const char *filename) ++static int boot_plist(int enable, const char *filename) +{ + int result; -+ struct strvec args = STRVEC_INIT; ++ struct child_process child = CHILD_PROCESS_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + -+ strvec_split(&args, launchctl); -+ strvec_push(&args, "bootout"); -+ strvec_pushf(&args, "gui/%s", uid); -+ strvec_push(&args, filename); ++ strvec_split(&child.args, launchctl); + -+ result = run_command_v_opt(args.v, 0); ++ if (enable) ++ strvec_push(&child.args, "bootstrap"); ++ else ++ strvec_push(&child.args, "bootout"); ++ strvec_pushf(&child.args, "gui/%s", uid); ++ strvec_push(&child.args, filename); + -+ strvec_clear(&args); -+ free(uid); -+ return result; -+} ++ child.no_stderr = 1; ++ child.no_stdout = 1; + -+static int bootstrap(const char *filename) -+{ -+ int result; -+ struct strvec args = STRVEC_INIT; -+ char *uid = get_uid(); -+ const char *launchctl = getenv("GIT_TEST_CRONTAB"); -+ if (!launchctl) -+ launchctl = "/bin/launchctl"; ++ if (start_command(&child)) ++ die(_("failed to start launchctl")); + -+ strvec_split(&args, launchctl); -+ strvec_push(&args, "bootstrap"); -+ strvec_pushf(&args, "gui/%s", uid); -+ strvec_push(&args, filename); ++ result = finish_command(&child); + -+ result = run_command_v_opt(args.v, 0); -+ -+ strvec_clear(&args); + free(uid); + return result; +} @@ builtin/gc.c: static int maintenance_unregister(void) + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); -+ int result = bootout(filename); ++ int result = boot_plist(0, filename); ++ unlink(filename); + free(filename); + free(name); + return result; @@ builtin/gc.c: static int maintenance_unregister(void) + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); -+ plist = fopen(filename, "w"); -+ -+ if (!plist) -+ die(_("failed to open '%s'"), filename); ++ plist = xfopen(filename, "w"); + + preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" @@ builtin/gc.c: static int maintenance_unregister(void) + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + + /* bootout might fail if not already running, so ignore */ -+ bootout(filename); -+ if (bootstrap(filename)) ++ boot_plist(0, filename); ++ if (boot_plist(1, filename)) + die(_("failed to bootstrap service %s"), filename); + + fclose(plist); @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' ' +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' -+ echo "#!/bin/sh\necho \$@ >>args" >print-args && -+ chmod a+x print-args && ++ write_script print-args "#!/bin/sh\necho \$* >>args" && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance start && @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && -+ xmllint "$PLIST" >/dev/null && ++ xmllint --noout "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect && + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' + test_cmp expect args && + + rm -f args && -+ GIT_TEST_CRONTAB="./print-args" git maintenance stop && ++ GIT_TEST_CRONTAB="./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ # stop does not remove plist files, but boots them out -+ rm expect && -+ for frequency in hourly daily weekly -+ do -+ PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && -+ grep schedule=$frequency "$PLIST" && -+ echo "bootout gui/$UID $PLIST" >>expect || return 1 -+ done && -+ test_cmp expect args ++ printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ ++ hourly daily weekly >expect && ++ test_cmp expect args && ++ ls "$HOME/Library/LaunchAgents" >actual && ++ test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' 4: 84eb44de31 ! 4: ed7a61978f maintenance: use Windows scheduled tasks @@ Commit message Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## Documentation/git-maintenance.txt ## -@@ Documentation/git-maintenance.txt: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSy - for more information. +@@ Documentation/git-maintenance.txt: To create more advanced customizations to your background tasks, see + launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) +static int schedule_task(const char *exec_path, enum schedule_priority schedule) +{ + int result; -+ struct strvec args = STRVEC_INIT; ++ struct child_process child = CHILD_PROCESS_INIT; + const char *xml, *schtasks; -+ char *xmlpath; ++ char *xmlpath, *tempDir; + FILE *xmlfp; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + -+ xmlpath = xstrfmt("%s/schedule-%s.xml", -+ the_repository->objects->odb->path, -+ frequency); -+ xmlfp = fopen(xmlpath, "w"); -+ if (!xmlfp) -+ die(_("failed to open '%s'"), xmlpath); ++ tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); ++ xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); ++ safe_create_leading_directories(xmlpath); ++ xmlfp = xfopen(xmlpath, "w"); + + xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; -+ strvec_split(&args, schtasks); -+ strvec_pushl(&args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); ++ strvec_split(&child.args, schtasks); ++ strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + -+ result = run_command_v_opt(args.v, 0); ++ child.no_stdout = 1; ++ child.no_stderr = 1; ++ ++ if (start_command(&child)) ++ die(_("failed to start schtasks")); ++ result = finish_command(&child); + -+ strvec_clear(&args); + unlink(xmlpath); ++ rmdir(tempDir); + free(xmlpath); + free(name); + return result; @@ t/t7900-maintenance.sh: test_expect_success !MACOS_MAINTENANCE 'stop from existi GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt @@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' - test_cmp expect args + test_line_count = 0 actual ' +test_expect_success MINGW 'start and stop Windows maintenance' ' @@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop ma + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ for frequency in hourly daily weekly -+ do -+ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ -+ $frequency $frequency -+ done >expect && ++ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/temp/schedule-%s.xml\n" \ ++ hourly hourly daily daily weekly weekly >expect && + test_cmp expect args && + + rm -f args && -- gitgitgadget ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v3 1/4] maintenance: extract platform-specific scheduling 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 ` Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget ` (4 subsequent siblings) 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a1..c1f7d9bdc2 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,7 +1494,7 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int platform_update_schedule(int run_maintenance, int fd) { int result = 0; int in_old_region = 0; @@ -1503,11 +1503,6 @@ static int update_background_schedule(int run_maintenance) FILE *cron_list, *cron_in; const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); crontab_name = getenv("GIT_TEST_CRONTAB"); if (!crontab_name) @@ -1516,12 +1511,11 @@ static int update_background_schedule(int run_maintenance) strvec_split(&crontab_list.args, crontab_name); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); } /* Ignore exit code, as an empty crontab will return error. */ @@ -1531,7 +1525,7 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); strvec_split(&crontab_edit.args, crontab_name); @@ -1539,8 +1533,7 @@ static int update_background_schedule(int run_maintenance) crontab_edit.git_cmd = 0; if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; + return error(_("failed to run 'crontab'; your system might not support 'cron'")); } cron_in = fdopen(crontab_edit.in, "w"); @@ -1586,13 +1579,24 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; - } - fclose(cron_list); + else + fclose(cron_list); + return result; +} + +static int update_background_schedule(int run_maintenance) +{ + int result; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + result = platform_update_schedule(run_maintenance, lk.tempfile->fd); -cleanup: rollback_lock_file(&lk); return result; } -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v3 2/4] maintenance: include 'cron' details in docs 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 ` Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget ` (3 subsequent siblings) 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc..1aa1112418 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `crontab` entry specifies the full path of the `git` executable to +ensure that the executed `git` command is the same one with which +`git maintenance start` was issued independent of `PATH`. If the same user +runs `git maintenance start` with multiple Git executables, then only the +latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config. The `git maintenance` process +then determines which maintenance tasks are configured to run on each +repository with each `<frequency>` using the `maintenance.<task>.schedule` +config options. These values are loaded from the global or repository +config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +the crontab(5) documentation for advanced scheduling techniques. Please +do use the full path and `--exec-path` techniques from the default +schedule to ensure you are executing the correct binaries in your +schedule. + GIT --- -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v3 3/4] maintenance: use launchctl on macOS 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 ` Derrick Stolee via GitGitGadget 2020-11-13 20:19 ` Eric Sunshine 2020-11-13 14:00 ` [PATCH v3 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget ` (2 subsequent siblings) 5 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_CRONTAB. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format, so call it from the macOS-specific tests to ensure the .plist files are well-formatted. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 40 ++++++ builtin/gc.c | 195 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 48 +++++++- t/test-lib.sh | 4 + 4 files changed, 284 insertions(+), 3 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1aa1112418..5f8f63f098 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ schedule to ensure you are executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=<frequency>` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `<array>` element following +the `<key>StartCalendarInterval</key>` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index c1f7d9bdc2..da2c892f68 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,200 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +#if defined(__APPLE__) + +static char *get_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *get_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_uid(void) +{ + struct strbuf output = STRBUF_INIT; + struct child_process id = CHILD_PROCESS_INIT; + + strvec_pushl(&id.args, "/usr/bin/id", "-u", NULL); + if (capture_command(&id, &output, 0)) + die(_("failed to discover user id")); + + strbuf_trim_trailing_newline(&output); + return strbuf_detach(&output, NULL); +} + +static int boot_plist(int enable, const char *filename) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + + strvec_split(&child.args, launchctl); + + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_pushf(&child.args, "gui/%s", uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int remove_plist(enum schedule_priority schedule) +{ + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + int result = boot_plist(0, filename); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int remove_plists(void) +{ + return remove_plist(SCHEDULE_HOURLY) || + remove_plist(SCHEDULE_DAILY) || + remove_plist(SCHEDULE_WEEKLY); +} + +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + + /* bootout might fail if not already running, so ignore */ + boot_plist(0, filename); + if (boot_plist(1, filename)) + die(_("failed to bootstrap service %s"), filename); + + fclose(plist); + free(filename); + free(name); + return 0; +} + +static int add_plists(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_plist(exec_path, SCHEDULE_HOURLY) || + schedule_plist(exec_path, SCHEDULE_DAILY) || + schedule_plist(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_plists(); + else + return remove_plists(); +} +#else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1585,6 +1779,7 @@ static int platform_update_schedule(int run_maintenance, int fd) fclose(cron_list); return result; } +#endif static int update_background_schedule(int run_maintenance) { diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..29d340a828 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,12 +389,54 @@ test_expect_success 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' + write_script print-args "#!/bin/sh\necho \$* >>args" && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + # ~/Library/LaunchAgents + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + xmllint --noout "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect && + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..620ffbf3af 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1703,6 +1703,10 @@ test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' +test_lazy_prereq MACOS_MAINTENANCE ' + launchctl list +' + # Ensure that no test accidentally triggers a Git command # that runs 'crontab', affecting a user's cron schedule. # Tests that verify the cron integration must set this locally -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v3 3/4] maintenance: use launchctl on macOS 2020-11-13 14:00 ` [PATCH v3 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-13 20:19 ` Eric Sunshine 2020-11-13 20:42 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-13 20:19 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > [...] > The solution is to switch from cron to the Apple-recommended [1] > 'launchd' tool. > [...] > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1491,6 +1491,200 @@ static int maintenance_unregister(void) > +static int boot_plist(int enable, const char *filename) > +{ > + struct child_process child = CHILD_PROCESS_INIT; > + child.no_stderr = 1; > + child.no_stdout = 1; > + if (start_command(&child)) > + die(_("failed to start launchctl")); Not necessarily worth a re-roll -- in fact, it could be done atop this series to avoid holding this series up -- but this too-succinct error reporting won't help users diagnose the failure. An alternative would be to capture stdout and stderr and only print them if the command fails. Perhaps something like this: struct strbuf out = STRBUF_INIT; struct strbuf err = STRBUF_INIT; ... if (pipe_command(child, NULL, 0, &out, 0, &err, 0) { if (out.len && err.len) strbuf_addstr(&out, "; "); strbuf_addbuf(&out, &err); die(_("launchctl failed: %s"), out.buf); } By the way, won't this die() be a problem when schedule_plist() calls boot_plist() to remove the old scheduled tasks before calling it again to register the new ones? If the old ones don't exist, then it will die() unnecessarily and never register the new ones. Or am I misunderstanding? (I'm guessing that I must be misunderstanding since the test script presumably passes.) > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh > @@ -389,12 +389,54 @@ test_expect_success 'stop from existing schedule' ' > +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' > + write_script print-args "#!/bin/sh\necho \$* >>args" && write_script() takes the script body as stdin, not as an argument, and you don't need to specify /bin/sh. What you have here works by accident only because write_script() takes an optional second argument specifying the shell to use in place of the default /bin/sh. Nevertheless, it should really be written: write_script print-args <<-\EOF echo $* EOF Patch [4/4] uses write_script() correctly. > + rm -f args && > + GIT_TEST_CRONTAB="./print-args" git maintenance start && > + > + # start registers the repo > + git config --get --global maintenance.repo "$(pwd)" && > + > + # ~/Library/LaunchAgents > + ls "$HOME/Library/LaunchAgents" >actual && Not sure what the comment above the `ls` is meant to be conveying. Could be dropped but not itself worth a re-roll. > + cat >expect <<-\EOF && > + org.git-scm.git.daily.plist > + org.git-scm.git.hourly.plist > + org.git-scm.git.weekly.plist > + EOF > + test_cmp expect actual && > + > + rm expect && > + for frequency in hourly daily weekly > + do > + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && > + xmllint --noout "$PLIST" && > + grep schedule=$frequency "$PLIST" && > + echo "bootout gui/$UID $PLIST" >>expect && > + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 > + done && This is still relying upon $UID picked up from the users environment (as far as I can tell), which seems fragile. As mentioned in my first review, it probably would be more robust to compute UID manually the same way git-maintenance itself does. > + test_cmp expect args && > + > + rm -f args && > + GIT_TEST_CRONTAB="./print-args" git maintenance stop && Minor: No need for the quotes around ./print-args (though they don't hurt either, and certainly not worth re-rolling just to drop them, and it's subjective so don't drop them just for my sake). > + # stop does not unregister the repo > + git config --get --global maintenance.repo "$(pwd)" && > + > + printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ > + hourly daily weekly >expect && > + test_cmp expect args && > + ls "$HOME/Library/LaunchAgents" >actual && > + test_line_count = 0 actual > +' ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 3/4] maintenance: use launchctl on macOS 2020-11-13 20:19 ` Eric Sunshine @ 2020-11-13 20:42 ` Derrick Stolee 2020-11-13 20:53 ` Eric Sunshine 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-13 20:42 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee On 11/13/2020 3:19 PM, Eric Sunshine wrote: > On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> [...] >> The solution is to switch from cron to the Apple-recommended [1] >> 'launchd' tool. >> [...] >> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> >> --- >> diff --git a/builtin/gc.c b/builtin/gc.c >> @@ -1491,6 +1491,200 @@ static int maintenance_unregister(void) >> +static int boot_plist(int enable, const char *filename) >> +{ >> + struct child_process child = CHILD_PROCESS_INIT; >> + child.no_stderr = 1; >> + child.no_stdout = 1; >> + if (start_command(&child)) >> + die(_("failed to start launchctl")); > > Not necessarily worth a re-roll -- in fact, it could be done atop this > series to avoid holding this series up -- but this too-succinct error > reporting won't help users diagnose the failure. An alternative would > be to capture stdout and stderr and only print them if the command > fails. Perhaps something like this: > > struct strbuf out = STRBUF_INIT; > struct strbuf err = STRBUF_INIT; > ... > if (pipe_command(child, NULL, 0, &out, 0, &err, 0) { > if (out.len && err.len) > strbuf_addstr(&out, "; "); > strbuf_addbuf(&out, &err); > die(_("launchctl failed: %s"), out.buf); > } We would also want to pass a "die_on_failure" into the method, since in the 'git maintenance start' case we don't want to report a failure when 'launchctl bootout' fails before we call 'launchctl bootstrap'. > By the way, won't this die() be a problem when schedule_plist() calls > boot_plist() to remove the old scheduled tasks before calling it again > to register the new ones? If the old ones don't exist, then it will > die() unnecessarily and never register the new ones. Or am I > misunderstanding? (I'm guessing that I must be misunderstanding since > the test script presumably passes.) This die() is only if the process cannot _start_, for example due to launchctl not existing on $PATH. The result from finish_command() would be non-zero when we bootout a plist that doesn't exist. >> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh >> @@ -389,12 +389,54 @@ test_expect_success 'stop from existing schedule' ' >> +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' >> + write_script print-args "#!/bin/sh\necho \$* >>args" && > > write_script() takes the script body as stdin, not as an argument, and > you don't need to specify /bin/sh. What you have here works by > accident only because write_script() takes an optional second argument > specifying the shell to use in place of the default /bin/sh. > Nevertheless, it should really be written: > > write_script print-args <<-\EOF > echo $* > EOF > > Patch [4/4] uses write_script() correctly. Ah. Sorry for misunderstanding. That explains why it works this way on macOS but it did _not_ work that way on Windows. >> + rm -f args && >> + GIT_TEST_CRONTAB="./print-args" git maintenance start && >> + >> + # start registers the repo >> + git config --get --global maintenance.repo "$(pwd)" && >> + >> + # ~/Library/LaunchAgents >> + ls "$HOME/Library/LaunchAgents" >actual && > > Not sure what the comment above the `ls` is meant to be conveying. > Could be dropped but not itself worth a re-roll. > >> + cat >expect <<-\EOF && >> + org.git-scm.git.daily.plist >> + org.git-scm.git.hourly.plist >> + org.git-scm.git.weekly.plist >> + EOF >> + test_cmp expect actual && >> + >> + rm expect && >> + for frequency in hourly daily weekly >> + do >> + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && >> + xmllint --noout "$PLIST" && >> + grep schedule=$frequency "$PLIST" && >> + echo "bootout gui/$UID $PLIST" >>expect && >> + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 >> + done && > > This is still relying upon $UID picked up from the users environment > (as far as I can tell), which seems fragile. As mentioned in my first > review, it probably would be more robust to compute UID manually the > same way git-maintenance itself does. Sorry, I missed this comment from v1 when reapplying the changes for v3. >> + test_cmp expect args && >> + >> + rm -f args && >> + GIT_TEST_CRONTAB="./print-args" git maintenance stop && > > Minor: No need for the quotes around ./print-args (though they don't > hurt either, and certainly not worth re-rolling just to drop them, and > it's subjective so don't drop them just for my sake). Thank you for your continued attention and patience. -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 3/4] maintenance: use launchctl on macOS 2020-11-13 20:42 ` Derrick Stolee @ 2020-11-13 20:53 ` Eric Sunshine 2020-11-13 20:56 ` Eric Sunshine 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-13 20:53 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 3:42 PM Derrick Stolee <stolee@gmail.com> wrote: > On 11/13/2020 3:19 PM, Eric Sunshine wrote: > > if (pipe_command(child, NULL, 0, &out, 0, &err, 0) { > > if (out.len && err.len) > > strbuf_addstr(&out, "; "); > > strbuf_addbuf(&out, &err); > > die(_("launchctl failed: %s"), out.buf); > > } > > We would also want to pass a "die_on_failure" into the method, since > in the 'git maintenance start' case we don't want to report a failure > when 'launchctl bootout' fails before we call 'launchctl bootstrap'. Right. I started writing that we'd also need a `die_one_failure` flag but deleted the comment since I decided to wait until I got an answer... > > By the way, won't this die() be a problem when schedule_plist() calls > > boot_plist() to remove the old scheduled tasks before calling it again > > to register the new ones? If the old ones don't exist, then it will > > die() unnecessarily and never register the new ones. Or am I > > misunderstanding? (I'm guessing that I must be misunderstanding since > > the test script presumably passes.) > > This die() is only if the process cannot _start_, for example due to > launchctl not existing on $PATH. The result from finish_command() > would be non-zero when we bootout a plist that doesn't exist. ... to this question. Another thought I had was simply checking for the presence of the file and skipping `bootout` altogether if it doesn't exist. That would, I think, obviate the need for mucking with stdout/stderr oppression. > > write_script() takes the script body as stdin, not as an argument, and > > you don't need to specify /bin/sh. What you have here works by > > accident only because write_script() takes an optional second argument > > specifying the shell to use in place of the default /bin/sh. > > Nevertheless, it should really be written: > > > > write_script print-args <<-\EOF > > echo $* > > EOF > > > > Patch [4/4] uses write_script() correctly. > > Ah. Sorry for misunderstanding. That explains why it works this way > on macOS but it did _not_ work that way on Windows. Sorry on my part too. I missed the `args` redirect in my example. It should be: write_script print-args <<-\EOF echo $* >args EOF ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 3/4] maintenance: use launchctl on macOS 2020-11-13 20:53 ` Eric Sunshine @ 2020-11-13 20:56 ` Eric Sunshine 0 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2020-11-13 20:56 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 3:53 PM Eric Sunshine <sunshine@sunshineco.com> wrote: > Another thought I had was simply checking for the presence of the file > and skipping `bootout` altogether if it doesn't exist. That would, I > think, obviate the need for mucking with stdout/stderr oppression. Erm, s/oppression/suppression/. ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v3 4/4] maintenance: use Windows scheduled tasks 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2020-11-13 14:00 ` [PATCH v3 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 ` Derrick Stolee via GitGitGadget 2020-11-13 20:44 ` Eric Sunshine 2020-11-13 20:47 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget 5 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-13 14:00 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 184 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 33 +++++- 3 files changed, 236 insertions(+), 3 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 5f8f63f098..6970f2b898 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance (<frequency>)`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index da2c892f68..76a3afa20a 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1684,6 +1684,190 @@ static int platform_update_schedule(int run_maintenance, int fd) else return remove_plists(); } + +#elif defined(GIT_WINDOWS_NATIVE) + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int remove_task(enum schedule_priority schedule) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + const char *schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; + + strvec_split(&args, schtasks); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int remove_scheduled_tasks(void) +{ + return remove_task(SCHEDULE_HOURLY) || + remove_task(SCHEDULE_DAILY) || + remove_task(SCHEDULE_WEEKLY); +} + +static int schedule_task(const char *exec_path, enum schedule_priority schedule) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml, *schtasks; + char *xmlpath, *tempDir; + FILE *xmlfp; + const char *frequency = get_frequency(schedule); + char *name = get_task_name(frequency); + + tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); + xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); + safe_create_leading_directories(xmlpath); + xmlfp = xfopen(xmlpath, "w"); + + xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fprintf(xmlfp, xml); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(xmlfp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml= "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(xmlfp, xml, exec_path, exec_path, frequency); + fclose(xmlfp); + + schtasks = getenv("GIT_TEST_CRONTAB"); + if (!schtasks) + schtasks = "schtasks"; + strvec_split(&child.args, schtasks); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + unlink(xmlpath); + rmdir(tempDir); + free(xmlpath); + free(name); + return result; +} + +static int add_scheduled_tasks(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_task(exec_path, SCHEDULE_HOURLY) || + schedule_task(exec_path, SCHEDULE_DAILY) || + schedule_task(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_scheduled_tasks(); + else + return remove_scheduled_tasks(); +} + #else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 29d340a828..0dc2479117 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,7 +389,7 @@ test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE,!MINGW 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt @@ -437,6 +437,33 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success MINGW 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + EOF + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/temp/schedule-%s.xml\n" \ + hourly hourly daily daily weekly weekly >expect && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v3 4/4] maintenance: use Windows scheduled tasks 2020-11-13 14:00 ` [PATCH v3 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-13 20:44 ` Eric Sunshine 2020-11-13 21:32 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-13 20:44 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > Git's background maintenance uses cron by default, but this is not > available on Windows. Instead, integrate with Task Scheduler. > [...] > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1684,6 +1684,190 @@ static int platform_update_schedule(int run_maintenance, int fd) > +static int schedule_task(const char *exec_path, enum schedule_priority schedule) > +{ > + char *xmlpath, *tempDir; > + tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); > + xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); When I wondered aloud in my previous review whether writing these throwaway files to a temporary directory would make sense, I was thinking more along the lines of /tmp or $TEMP. More specifically, we have xmkstemp() in wrapper.c which is good for this sort of thing (or one of the other temporary-file-making functions in there). We also have a more full-featured temporary-file API in tempfile.h which would ensure that these throwaway files actually get thrown away when the command finishes. This is not necessarily worth a re-roll. > + if (start_command(&child)) > + die(_("failed to start schtasks")); > + result = finish_command(&child); > + > + unlink(xmlpath); > + rmdir(tempDir); Neither xmlpath and tempDir get cleaned up from the filesystem if the preceding die() is triggered (which may or may not make sense -- perhaps you want to keep them around if it helps with the diagnosis of the failure). The functions in tempfile.h would ensure the temporary file is cleaned up even if the program die()s, or you could manually remove the temporary file before die()ing. > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh > @@ -437,6 +437,33 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' > +test_expect_success MINGW 'start and stop Windows maintenance' ' > + write_script print-args <<-\EOF && > + echo $* >>args > + EOF Using `>>` here makes it harder to reason about the test than using `>` would, especially since `>>` seems to be unnecessary in this case. > + rm -f args && > + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && Is it a requirement on Windows to mention /bin/sh here? Specifically, I'm wondering why a simple ./print-args doesn't work. (It's especially unclear since write_script() is used heavily in the test suite and it seems to work well enough on Windows without specifying /bin/sh.) ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 4/4] maintenance: use Windows scheduled tasks 2020-11-13 20:44 ` Eric Sunshine @ 2020-11-13 21:32 ` Derrick Stolee 2020-11-13 21:40 ` Eric Sunshine 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-13 21:32 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee On 11/13/2020 3:44 PM, Eric Sunshine wrote: > On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> Git's background maintenance uses cron by default, but this is not >> available on Windows. Instead, integrate with Task Scheduler. >> [...] >> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> >> --- >> diff --git a/builtin/gc.c b/builtin/gc.c >> @@ -1684,6 +1684,190 @@ static int platform_update_schedule(int run_maintenance, int fd) >> +static int schedule_task(const char *exec_path, enum schedule_priority schedule) >> +{ >> + char *xmlpath, *tempDir; >> + tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); >> + xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); > > When I wondered aloud in my previous review whether writing these > throwaway files to a temporary directory would make sense, I was > thinking more along the lines of /tmp or $TEMP. More specifically, we > have xmkstemp() in wrapper.c which is good for this sort of thing (or > one of the other temporary-file-making functions in there). We also > have a more full-featured temporary-file API in tempfile.h which would > ensure that these throwaway files actually get thrown away when the > command finishes. > > This is not necessarily worth a re-roll. > >> + if (start_command(&child)) >> + die(_("failed to start schtasks")); >> + result = finish_command(&child); >> + >> + unlink(xmlpath); >> + rmdir(tempDir); > > Neither xmlpath and tempDir get cleaned up from the filesystem if the > preceding die() is triggered (which may or may not make sense -- > perhaps you want to keep them around if it helps with the diagnosis of > the failure). The functions in tempfile.h would ensure the temporary > file is cleaned up even if the program die()s, or you could manually > remove the temporary file before die()ing. While I do like to have access to the data when trying to resolve an issue, it's probably better to use the tempfile library. >> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh >> @@ -437,6 +437,33 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' >> +test_expect_success MINGW 'start and stop Windows maintenance' ' >> + write_script print-args <<-\EOF && >> + echo $* >>args >> + EOF > > Using `>>` here makes it harder to reason about the test than using > `>` would, especially since `>>` seems to be unnecessary in this case. Since we execute the GIT_TEST_CRONTAB executable multiple times, we need to use >> to log all three instances (and their order). Using ">args" would only capture the final call for the weekly schedule. On macOS, there are as many as six calls (three bootouts, three bootstraps). >> + rm -f args && >> + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && > > Is it a requirement on Windows to mention /bin/sh here? Specifically, > I'm wondering why a simple ./print-args doesn't work. (It's especially > unclear since write_script() is used heavily in the test suite and it > seems to work well enough on Windows without specifying /bin/sh.) I landed on this after trying several attempts to get this to work, including "$(pwd)/print-args" and I'm not sure why it doesn't work in the Windows case. It is something to do with how I am executing the subcommand from within Git. I'm pretty sure this idea of "mocking" an executable through Git is relatively new, or at least rare Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 4/4] maintenance: use Windows scheduled tasks 2020-11-13 21:32 ` Derrick Stolee @ 2020-11-13 21:40 ` Eric Sunshine 2020-11-16 13:13 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-13 21:40 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 4:32 PM Derrick Stolee <stolee@gmail.com> wrote: > On 11/13/2020 3:44 PM, Eric Sunshine wrote: > > On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget > > <gitgitgadget@gmail.com> wrote: > >> +test_expect_success MINGW 'start and stop Windows maintenance' ' > >> + write_script print-args <<-\EOF && > >> + echo $* >>args > >> + EOF > > > > Using `>>` here makes it harder to reason about the test than using > > `>` would, especially since `>>` seems to be unnecessary in this case. > > Since we execute the GIT_TEST_CRONTAB executable multiple times, we > need to use >> to log all three instances (and their order). Using ">args" > would only capture the final call for the weekly schedule. > > On macOS, there are as many as six calls (three bootouts, three bootstraps). Makes sense. Thanks. > >> + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && > > > > Is it a requirement on Windows to mention /bin/sh here? Specifically, > > I'm wondering why a simple ./print-args doesn't work. (It's especially > > unclear since write_script() is used heavily in the test suite and it > > seems to work well enough on Windows without specifying /bin/sh.) > > I landed on this after trying several attempts to get this to work, > including "$(pwd)/print-args" and I'm not sure why it doesn't work > in the Windows case. It is something to do with how I am executing > the subcommand from within Git. I'm pretty sure this idea of "mocking" > an executable through Git is relatively new, or at least rare Just for clarification... You mentioned in response to my [3/4] review that your accidentally-working write_script() only worked as expected on Mac but not on Windows. When you arrived at this solution of GIT_TEST_CRONTAB="/bin/sh ..." here, was that before or after you fixed write_script() to take the script body from stdin? ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 4/4] maintenance: use Windows scheduled tasks 2020-11-13 21:40 ` Eric Sunshine @ 2020-11-16 13:13 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-16 13:13 UTC (permalink / raw) To: Eric Sunshine Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee, Derrick Stolee On 11/13/2020 4:40 PM, Eric Sunshine wrote: > On Fri, Nov 13, 2020 at 4:32 PM Derrick Stolee <stolee@gmail.com> wrote: >> On 11/13/2020 3:44 PM, Eric Sunshine wrote: >>> On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget >>>> + GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && >>> >>> Is it a requirement on Windows to mention /bin/sh here? Specifically, >>> I'm wondering why a simple ./print-args doesn't work. (It's especially >>> unclear since write_script() is used heavily in the test suite and it >>> seems to work well enough on Windows without specifying /bin/sh.) >> >> I landed on this after trying several attempts to get this to work, >> including "$(pwd)/print-args" and I'm not sure why it doesn't work >> in the Windows case. It is something to do with how I am executing >> the subcommand from within Git. I'm pretty sure this idea of "mocking" >> an executable through Git is relatively new, or at least rare > > Just for clarification... You mentioned in response to my [3/4] review > that your accidentally-working write_script() only worked as expected > on Mac but not on Windows. When you arrived at this solution of > GIT_TEST_CRONTAB="/bin/sh ..." here, was that before or after you > fixed write_script() to take the script body from stdin? You're right. That was necessary only for the old way that I was creating the script. The correct way works with GIT_TEST_CRONTAB equal to ./print-args. -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (3 preceding siblings ...) 2020-11-13 14:00 ` [PATCH v3 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-13 20:47 ` Eric Sunshine 2020-11-14 9:23 ` Eric Sunshine 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget 5 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-13 20:47 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget; +Cc: Git List, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 9:00 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > * This actually includes the feedback responses I had intended for v2. > Sorry about that! I forgot to mention a couple things when reviewing the patches individually, so I'll point them out here... > + at [2]. The current design does not preclude a future version that > + detects the available fatures of 'launchctl' to use the older s/fatures/features/ > -+ test_cmp expect args > ++ test_line_count = 0 actual These days, we usually say: test_must_be_empty actual ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-13 20:47 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine @ 2020-11-14 9:23 ` Eric Sunshine 2020-11-16 13:17 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-14 9:23 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget; +Cc: Git List, Derrick Stolee, Derrick Stolee On Fri, Nov 13, 2020 at 03:47:15PM -0500, Eric Sunshine wrote: > I forgot to mention a couple things when reviewing the patches > individually, so I'll point them out here... In v2, you added an `xmllint` check on MacOS after discovering that gc.c was generating a malformed .plist file on that platform. That got me thinking that it would have been nice to have caught the problem earlier, if possible, even without having access to MacOS. Since none of the code added to gc.c has a hard platform dependency, it should be possible to perform all the tests on any platform rather than restricting them to specific platforms via test prerequisites. The patch below, which is built atop v3, does just that. It removes the conditional compilation directives from gc.c and the prerequisites from the test script so that all scheduler-specific code in gc.c is tested on all platform. The changes made by the patch are intended to be folded into each of your patches where appropriate (rather than existing atop your series, which, though possible, would be ugly). If you're interested in incorporating any of these improvements into v4, you can have my "Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>" in addition to the Helped-by: you already added. A few more notes... In addition to making it possible to test all platform-specific schedulers on each platform, I also made a few other changes/enhancements: * simplified UID retrieval and eliminated platform-specific dependencies (though this may need some additional tweaking on Windows, for which I did not test); also fixed the $UID issue mentioned in review * extended xmllint testing to the XML files generated for `schtasks` on Windows too; this required a small modification to the XML header boilerplate to specify the correct file encoding since `xmllint` complains when the file is UTF-8 but claims to be UTF-16; now that the test script captures the generated `schtasks` XML file for checking against `xmllint`, you have the opportunity to perform other sorts of validation checks on the XML too, such as you do in the MacOS `launchctl` test (though I did not add any additional checks) * fixed a potentially crashable `fprintf(xmlfp, xml)` by changing it to `fputs(xml, xmlfp)` since the compiler complains about the former because it can crash if `xml` contains a "%" * fixed the malformed write_script() issue for the MacOS test mentioned in review --- >8 --- From 016887b9fa4269bd4df46bea1d7849c08aba6ad6 Mon Sep 17 00:00:00 2001 From: Eric Sunshine <sunshine@sunshineco.com> Date: Sat, 14 Nov 2020 02:39:05 -0500 Subject: [PATCH] maintenance: test start/stop on all platforms from any platform Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> --- builtin/gc.c | 204 +++++++++++++++++++---------------------- t/t7900-maintenance.sh | 66 +++++++++---- t/test-lib.sh | 4 - 3 files changed, 143 insertions(+), 131 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index 76a3afa20a..955d4b3baf 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,16 +1491,28 @@ static int maintenance_unregister(void) return run_command(&config_unset); } -#if defined(__APPLE__) +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} -static char *get_service_name(const char *frequency) +static char *launchctl_service_name(const char *frequency) { struct strbuf label = STRBUF_INIT; strbuf_addf(&label, "org.git-scm.git.%s", frequency); return strbuf_detach(&label, NULL); } -static char *get_service_filename(const char *name) +static char *launchctl_service_filename(const char *name) { char *expanded; struct strbuf filename = STRBUF_INIT; @@ -1514,49 +1526,23 @@ static char *get_service_filename(const char *name) return expanded; } -static const char *get_frequency(enum schedule_priority schedule) -{ - switch (schedule) { - case SCHEDULE_HOURLY: - return "hourly"; - case SCHEDULE_DAILY: - return "daily"; - case SCHEDULE_WEEKLY: - return "weekly"; - default: - BUG("invalid schedule %d", schedule); - } -} - -static char *get_uid(void) +static char *launchctl_get_uid(void) { - struct strbuf output = STRBUF_INIT; - struct child_process id = CHILD_PROCESS_INIT; - - strvec_pushl(&id.args, "/usr/bin/id", "-u", NULL); - if (capture_command(&id, &output, 0)) - die(_("failed to discover user id")); - - strbuf_trim_trailing_newline(&output); - return strbuf_detach(&output, NULL); + return xstrfmt("gui/%d", getuid()); } -static int boot_plist(int enable, const char *filename) +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) { int result; struct child_process child = CHILD_PROCESS_INIT; - char *uid = get_uid(); - const char *launchctl = getenv("GIT_TEST_CRONTAB"); - if (!launchctl) - launchctl = "/bin/launchctl"; - - strvec_split(&child.args, launchctl); + char *uid = launchctl_get_uid(); + strvec_split(&child.args, cmd); if (enable) strvec_push(&child.args, "bootstrap"); else strvec_push(&child.args, "bootout"); - strvec_pushf(&child.args, "gui/%s", uid); + strvec_push(&child.args, uid); strvec_push(&child.args, filename); child.no_stderr = 1; @@ -1571,33 +1557,33 @@ static int boot_plist(int enable, const char *filename) return result; } -static int remove_plist(enum schedule_priority schedule) +static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) { const char *frequency = get_frequency(schedule); - char *name = get_service_name(frequency); - char *filename = get_service_filename(name); - int result = boot_plist(0, filename); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + int result = launchctl_boot_plist(0, filename, cmd); unlink(filename); free(filename); free(name); return result; } -static int remove_plists(void) +static int launchctl_remove_plists(const char *cmd) { - return remove_plist(SCHEDULE_HOURLY) || - remove_plist(SCHEDULE_DAILY) || - remove_plist(SCHEDULE_WEEKLY); + return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || + launchctl_remove_plist(SCHEDULE_DAILY, cmd) || + launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); } -static int schedule_plist(const char *exec_path, enum schedule_priority schedule) +static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) { FILE *plist; int i; const char *preamble, *repeat; const char *frequency = get_frequency(schedule); - char *name = get_service_name(frequency); - char *filename = get_service_filename(name); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); if (safe_create_leading_directories(filename)) die(_("failed to create directories for '%s'"), filename); @@ -1658,8 +1644,8 @@ static int schedule_plist(const char *exec_path, enum schedule_priority schedule fprintf(plist, "</array>\n</dict>\n</plist>\n"); /* bootout might fail if not already running, so ignore */ - boot_plist(0, filename); - if (boot_plist(1, filename)) + launchctl_boot_plist(0, filename, cmd); + if (launchctl_boot_plist(1, filename, cmd)) die(_("failed to bootstrap service %s"), filename); fclose(plist); @@ -1668,57 +1654,38 @@ static int schedule_plist(const char *exec_path, enum schedule_priority schedule return 0; } -static int add_plists(void) +static int launchctl_add_plists(const char *cmd) { const char *exec_path = git_exec_path(); - return schedule_plist(exec_path, SCHEDULE_HOURLY) || - schedule_plist(exec_path, SCHEDULE_DAILY) || - schedule_plist(exec_path, SCHEDULE_WEEKLY); + return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); } -static int platform_update_schedule(int run_maintenance, int fd) +static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) { if (run_maintenance) - return add_plists(); + return launchctl_add_plists(cmd); else - return remove_plists(); -} - -#elif defined(GIT_WINDOWS_NATIVE) - -static const char *get_frequency(enum schedule_priority schedule) -{ - switch (schedule) { - case SCHEDULE_HOURLY: - return "hourly"; - case SCHEDULE_DAILY: - return "daily"; - case SCHEDULE_WEEKLY: - return "weekly"; - default: - BUG("invalid schedule %d", schedule); - } + return launchctl_remove_plists(cmd); } -static char *get_task_name(const char *frequency) +static char *schtasks_task_name(const char *frequency) { struct strbuf label = STRBUF_INIT; strbuf_addf(&label, "Git Maintenance (%s)", frequency); return strbuf_detach(&label, NULL); } -static int remove_task(enum schedule_priority schedule) +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) { int result; struct strvec args = STRVEC_INIT; const char *frequency = get_frequency(schedule); - char *name = get_task_name(frequency); - const char *schtasks = getenv("GIT_TEST_CRONTAB"); - if (!schtasks) - schtasks = "schtasks"; + char *name = schtasks_task_name(frequency); - strvec_split(&args, schtasks); + strvec_split(&args, cmd); strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); result = run_command_v_opt(args.v, 0); @@ -1728,33 +1695,33 @@ static int remove_task(enum schedule_priority schedule) return result; } -static int remove_scheduled_tasks(void) +static int schtasks_remove_tasks(const char *cmd) { - return remove_task(SCHEDULE_HOURLY) || - remove_task(SCHEDULE_DAILY) || - remove_task(SCHEDULE_WEEKLY); + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); } -static int schedule_task(const char *exec_path, enum schedule_priority schedule) +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) { int result; struct child_process child = CHILD_PROCESS_INIT; - const char *xml, *schtasks; + const char *xml; char *xmlpath, *tempDir; FILE *xmlfp; const char *frequency = get_frequency(schedule); - char *name = get_task_name(frequency); + char *name = schtasks_task_name(frequency); tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); safe_create_leading_directories(xmlpath); xmlfp = xfopen(xmlpath, "w"); - xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n" + xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" "<Triggers>\n" "<CalendarTrigger>\n"; - fprintf(xmlfp, xml); + fputs(xml, xmlfp); switch (schedule) { case SCHEDULE_HOURLY: @@ -1831,10 +1798,7 @@ static int schedule_task(const char *exec_path, enum schedule_priority schedule) fprintf(xmlfp, xml, exec_path, exec_path, frequency); fclose(xmlfp); - schtasks = getenv("GIT_TEST_CRONTAB"); - if (!schtasks) - schtasks = "schtasks"; - strvec_split(&child.args, schtasks); + strvec_split(&child.args, cmd); strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); child.no_stdout = 1; @@ -1851,42 +1815,36 @@ static int schedule_task(const char *exec_path, enum schedule_priority schedule) return result; } -static int add_scheduled_tasks(void) +static int schtasks_schedule_tasks(const char *cmd) { const char *exec_path = git_exec_path(); - return schedule_task(exec_path, SCHEDULE_HOURLY) || - schedule_task(exec_path, SCHEDULE_DAILY) || - schedule_task(exec_path, SCHEDULE_WEEKLY); + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); } -static int platform_update_schedule(int run_maintenance, int fd) +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) { if (run_maintenance) - return add_scheduled_tasks(); + return schtasks_schedule_tasks(cmd); else - return remove_scheduled_tasks(); + return schtasks_remove_tasks(cmd); } -#else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int platform_update_schedule(int run_maintenance, int fd) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; crontab_list.out = dup(fd); @@ -1906,7 +1864,7 @@ static int platform_update_schedule(int run_maintenance, int fd) cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; @@ -1963,20 +1921,48 @@ static int platform_update_schedule(int run_maintenance, int fd) fclose(cron_list); return result; } + +#if defined(__APPLE__) +static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; +#else +static const char platform_scheduler[] = "crontab"; #endif -static int update_background_schedule(int run_maintenance) +static int update_background_schedule(int enable) { int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; struct lock_file lk; char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; + } + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) return error(_("another process is scheduling background maintenance")); - result = platform_update_schedule(run_maintenance, lk.tempfile->fd); + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 0dc2479117..e92946c10a 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -7,6 +7,19 @@ test_description='git maintenance builtin' GIT_TEST_COMMIT_GRAPH=0 GIT_TEST_MULTI_PACK_INDEX=0 +test_lazy_prereq XMLLINT ' + xmllint --version +' + +test_xmllint () { + if test_have_prereq XMLLINT + then + xmllint --noout "$@" + else + true + fi +} + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance <subcommand>" err && @@ -367,8 +380,8 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success !MACOS_MAINTENANCE,!MINGW 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && +test_expect_success 'start from empty cron table' ' + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -378,28 +391,32 @@ test_expect_success !MACOS_MAINTENANCE,!MINGW 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success !MACOS_MAINTENANCE,!MINGW 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && +test_expect_success 'stop from existing schedule' ' + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' -test_expect_success !MACOS_MAINTENANCE,!MINGW 'start preserves existing schedule' ' +test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' -test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' - write_script print-args "#!/bin/sh\necho \$* >>args" && +test_expect_success 'start and stop macOS maintenance' ' + uid=$(id -u) && + + write_script print-args <<-\EOF && + echo $* >>args + EOF rm -f args && - GIT_TEST_CRONTAB="./print-args" git maintenance start && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -417,33 +434,41 @@ test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' for frequency in hourly daily weekly do PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && - xmllint --noout "$PLIST" && + test_xmllint "$PLIST" && grep schedule=$frequency "$PLIST" && - echo "bootout gui/$UID $PLIST" >>expect && - echo "bootstrap gui/$UID $PLIST" >>expect || return 1 + echo "bootout gui/$uid $PLIST" >>expect && + echo "bootstrap gui/$uid $PLIST" >>expect || return 1 done && test_cmp expect args && rm -f args && - GIT_TEST_CRONTAB="./print-args" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && - printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + printf "bootout gui/$uid $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ hourly daily weekly >expect && test_cmp expect args && ls "$HOME/Library/LaunchAgents" >actual && test_line_count = 0 actual ' -test_expect_success MINGW 'start and stop Windows maintenance' ' +test_expect_success 'start and stop Windows maintenance' ' write_script print-args <<-\EOF && echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + test -z "$xmlfile" || cp "$xmlfile" . EOF rm -f args && - GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="schtasks:/bin/sh print-args" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -452,8 +477,13 @@ test_expect_success MINGW 'start and stop Windows maintenance' ' hourly hourly daily daily weekly weekly >expect && test_cmp expect args && + for frequency in hourly daily weekly + do + test_xmllint "schedule-$frequency.xml" + done && + rm -f args && - GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="schtasks:/bin/sh print-args" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && diff --git a/t/test-lib.sh b/t/test-lib.sh index 620ffbf3af..4a60d1ed76 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1703,10 +1703,6 @@ test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' -test_lazy_prereq MACOS_MAINTENANCE ' - launchctl list -' - # Ensure that no test accidentally triggers a Git command # that runs 'crontab', affecting a user's cron schedule. # Tests that verify the cron integration must set this locally -- 2.29.2.454.gaff20da3a2 ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-14 9:23 ` Eric Sunshine @ 2020-11-16 13:17 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-16 13:17 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget; +Cc: Git List, Derrick Stolee On 11/14/2020 4:23 AM, Eric Sunshine wrote: > On Fri, Nov 13, 2020 at 03:47:15PM -0500, Eric Sunshine wrote: >> I forgot to mention a couple things when reviewing the patches >> individually, so I'll point them out here... > > In v2, you added an `xmllint` check on MacOS after discovering that > gc.c was generating a malformed .plist file on that platform. That got > me thinking that it would have been nice to have caught the problem > earlier, if possible, even without having access to MacOS. Since none > of the code added to gc.c has a hard platform dependency, it should be > possible to perform all the tests on any platform rather than > restricting them to specific platforms via test prerequisites. The > patch below, which is built atop v3, does just that. It removes the > conditional compilation directives from gc.c and the prerequisites > from the test script so that all scheduler-specific code in gc.c is > tested on all platform. > > The changes made by the patch are intended to be folded into each of > your patches where appropriate (rather than existing atop your series, > which, though possible, would be ugly). If you're interested in > incorporating any of these improvements into v4, you can have my > "Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>" in addition > to the Helped-by: you already added. This approach is fascinating. I will tease it apart to appropriately incorporate it into my series. Thank you for your sign-off, since this elevates the patches from "Helped-by" to "Co-authored by". Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (4 preceding siblings ...) 2020-11-13 20:47 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine @ 2020-11-17 21:13 ` Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (6 more replies) 5 siblings, 7 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that. Updates in V4 ============= * Eric did an excellent job providing a patch that cleans up several parts of my series. The most impressive is his mechanism for testing the platform-specific Git logic in a way that is (mostly) platform-agnostic. * Windows doesn't have the 'id' command, so we cannot run the macOS platform test on Windows. * I noticed far too late that while my example XML files had been edited with UTF-8 encoding, Git is actually writing them as US-ASCII. Somehow xmllint and launchd are not complaining, but schtasks does complain. Unfortunately, I cannot find a way to catch this problem other than to install my tip version on all three platforms and go through the entire git maintenance start process, and double-check that the processes are running on the hour. Here is a diff from the tip of v3 + Eric's patch to the tip of v4: diff --git a/builtin/gc.c b/builtin/gc.c index 955d4b3baf..1a3725429c 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1642,13 +1642,13 @@ static int launchctl_schedule_plist(const char *exec_path, enum schedule_priorit break; } fprintf(plist, "</array>\n</dict>\n</plist>\n"); + fclose(plist); /* bootout might fail if not already running, so ignore */ launchctl_boot_plist(0, filename, cmd); if (launchctl_boot_plist(1, filename, cmd)) die(_("failed to bootstrap service %s"), filename); - fclose(plist); free(filename); free(name); return 0; @@ -1707,25 +1707,27 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority int result; struct child_process child = CHILD_PROCESS_INIT; const char *xml; - char *xmlpath, *tempDir; - FILE *xmlfp; + char *xmlpath; + struct tempfile *tfile; const char *frequency = get_frequency(schedule); char *name = schtasks_task_name(frequency); - tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); - xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); - safe_create_leading_directories(xmlpath); - xmlfp = xfopen(xmlpath, "w"); + xmlpath = xstrfmt("%s/schedule-%s.xml", + the_repository->objects->odb->path, + frequency); + tfile = create_tempfile(xmlpath); + if (!tfile || !fdopen_tempfile(tfile, "w")) + die(_("failed to create '%s'"), xmlpath); - xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" "<Triggers>\n" "<CalendarTrigger>\n"; - fputs(xml, xmlfp); + fputs(xml, tfile->fp); switch (schedule) { case SCHEDULE_HOURLY: - fprintf(xmlfp, + fprintf(tfile->fp, "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" "<Enabled>true</Enabled>\n" "<ScheduleByDay>\n" @@ -1739,7 +1741,7 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority break; case SCHEDULE_DAILY: - fprintf(xmlfp, + fprintf(tfile->fp, "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" "<Enabled>true</Enabled>\n" "<ScheduleByWeek>\n" @@ -1756,7 +1758,7 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority break; case SCHEDULE_WEEKLY: - fprintf(xmlfp, + fprintf(tfile->fp, "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" "<Enabled>true</Enabled>\n" "<ScheduleByWeek>\n" @@ -1771,7 +1773,7 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority break; } - xml= "</CalendarTrigger>\n" + xml = "</CalendarTrigger>\n" "</Triggers>\n" "<Principals>\n" "<Principal id=\"Author\">\n" @@ -1795,11 +1797,10 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority "</Exec>\n" "</Actions>\n" "</Task>\n"; - fprintf(xmlfp, xml, exec_path, exec_path, frequency); - fclose(xmlfp); - + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); strvec_split(&child.args, cmd); strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + close_tempfile_gently(tfile); child.no_stdout = 1; child.no_stderr = 1; @@ -1808,8 +1809,7 @@ static int schtasks_schedule_task(const char *exec_path, enum schedule_priority die(_("failed to start schtasks")); result = finish_command(&child); - unlink(xmlpath); - rmdir(tempDir); + delete_tempfile(&tfile); free(xmlpath); free(name); return result; @@ -1850,9 +1850,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { + if (start_command(&crontab_list)) return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - } /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1868,9 +1867,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { + if (start_command(&crontab_edit)) return error(_("failed to run 'crontab'; your system might not support 'cron'")); - } cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index e92946c10a..a26ff22541 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -408,7 +408,7 @@ test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' -test_expect_success 'start and stop macOS maintenance' ' +test_expect_success !MINGW 'start and stop macOS maintenance' ' uid=$(id -u) && write_script print-args <<-\EOF && @@ -421,7 +421,6 @@ test_expect_success 'start and stop macOS maintenance' ' # start registers the repo git config --get --global maintenance.repo "$(pwd)" && - # ~/Library/LaunchAgents ls "$HOME/Library/LaunchAgents" >actual && cat >expect <<-\EOF && org.git-scm.git.daily.plist @@ -468,12 +467,12 @@ test_expect_success 'start and stop Windows maintenance' ' EOF rm -f args && - GIT_TEST_MAINT_SCHEDULER="schtasks:/bin/sh print-args" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && - printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/temp/schedule-%s.xml\n" \ + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ hourly hourly daily daily weekly weekly >expect && test_cmp expect args && @@ -483,7 +482,7 @@ test_expect_success 'start and stop Windows maintenance' ' done && rm -f args && - GIT_TEST_MAINT_SCHEDULER="schtasks:/bin/sh print-args" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..ddbeee1f5e 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" Thanks, -Stolee cc: jrnieder@gmail.com [jrnieder@gmail.com], jonathantanmy@google.com [jonathantanmy@google.com], sluongng@gmail.com [sluongng@gmail.com]cc: Derrick Stolee stolee@gmail.com [stolee@gmail.com]cc: Đoàn Trần Công Danh congdanhqx@gmail.com [congdanhqx@gmail.com]cc: Martin Ågren martin.agren@gmail.com [martin.agren@gmail.com]cc: Eric Sunshine sunshine@sunshineco.com [sunshine@sunshineco.com]cc: Derrick Stolee stolee@gmail.com [stolee@gmail.com] Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks Documentation/git-maintenance.txt | 116 ++++++++ builtin/gc.c | 421 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 106 +++++++- t/test-lib.sh | 7 +- 4 files changed, 616 insertions(+), 34 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v4 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v4 Pull-Request: https://github.com/gitgitgadget/git/pull/776 Range-diff vs v3: 1: d35f1aa162 ! 1: 4807342b00 maintenance: extract platform-specific scheduling @@ Commit message swapped at compile time with new implementations on specialized platforms. + As we add this generality, rename GIT_TEST_CRONTAB to + GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as + "<scheduler>:<command>" so we can test platform-specific scheduling + logic even when not on the correct platform. By specifying the + <scheduler> in this string, we will be able to test all three sets of + Git logic from a Linux machine. + + Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> + Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## builtin/gc.c ## @@ builtin/gc.c: static int maintenance_unregister(void) #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) -+static int platform_update_schedule(int run_maintenance, int fd) ++static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; -@@ builtin/gc.c: static int update_background_schedule(int run_maintenance) + struct child_process crontab_list = CHILD_PROCESS_INIT; + struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; +- const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); -- + - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) -@@ builtin/gc.c: static int update_background_schedule(int run_maintenance) - strvec_split(&crontab_list.args, crontab_name); +- +- crontab_name = getenv("GIT_TEST_CRONTAB"); +- if (!crontab_name) +- crontab_name = "crontab"; +- +- strvec_split(&crontab_list.args, crontab_name); ++ strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { +- if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; +- } ++ if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - } /* Ignore exit code, as an empty crontab will return error. */ + finish_command(&crontab_list); @@ builtin/gc.c: static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. @@ builtin/gc.c: static int update_background_schedule(int run_maintenance) + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); -@@ builtin/gc.c: static int update_background_schedule(int run_maintenance) +- strvec_split(&crontab_edit.args, crontab_name); ++ strvec_split(&crontab_edit.args, cmd); + crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { +- if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; +- } ++ if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); - } cron_in = fdopen(crontab_edit.in, "w"); + if (!cron_in) { @@ builtin/gc.c: static int update_background_schedule(int run_maintenance) close(crontab_edit.in); @@ builtin/gc.c: static int update_background_schedule(int run_maintenance) + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; -- } -- fclose(cron_list); + else + fclose(cron_list); + return result; +} + -+static int update_background_schedule(int run_maintenance) ++static const char platform_scheduler[] = "crontab"; ++ ++static int update_background_schedule(int enable) +{ + int result; ++ const char *scheduler = platform_scheduler; ++ const char *cmd = scheduler; ++ char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + ++ testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); ++ if (testing) { ++ char *sep = strchr(testing, ':'); ++ if (!sep) ++ die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); ++ *sep = '\0'; ++ scheduler = testing; ++ cmd = sep + 1; + } +- fclose(cron_list); + +-cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + -+ result = platform_update_schedule(run_maintenance, lk.tempfile->fd); - --cleanup: ++ if (!strcmp(scheduler, "crontab")) ++ result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); ++ else ++ die("unknown background scheduler: %s", scheduler); ++ rollback_lock_file(&lk); ++ free(testing); return result; } + + + ## t/t7900-maintenance.sh ## +@@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' ' + ' + + test_expect_success 'start from empty cron table' ' +- GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && ++ GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && +@@ t/t7900-maintenance.sh: test_expect_success 'start from empty cron table' ' + ' + + test_expect_success 'stop from existing schedule' ' +- GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && ++ GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + # Operation is idempotent +- GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && ++ GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && + test_must_be_empty cron.txt + ' + + test_expect_success 'start preserves existing schedule' ' + echo "Important information!" >cron.txt && +- GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && ++ GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && + grep "Important information!" cron.txt + ' + + + ## t/test-lib.sh ## +@@ t/test-lib.sh: test_lazy_prereq REBASE_P ' + ' + + # Ensure that no test accidentally triggers a Git command +-# that runs 'crontab', affecting a user's cron schedule. +-# Tests that verify the cron integration must set this locally ++# that runs the actual maintenance scheduler, affecting a user's ++# system permanently. ++# Tests that verify the scheduler integration must set this locally + # to avoid errors. +-GIT_TEST_CRONTAB="exit 1" ++GIT_TEST_MAINT_SCHEDULER="none:exit 1" 2: 0dfe53092e ! 2: 99170df462 maintenance: include 'cron' details in docs @@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems -+is cron(8). This tool executes commands based on a given schedule. The ++is `cron`. This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + @@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + -+The `crontab` entry specifies the full path of the `git` executable to -+ensure that the executed `git` command is the same one with which -+`git maintenance start` was issued independent of `PATH`. If the same user -+runs `git maintenance start` with multiple Git executables, then only the -+latest executable is used. ++The `<path>` string is loaded to specifically use the location for the ++`git` executable used in the `git maintenance start` command. This allows ++for multiple versions to be compatible. However, if the same user runs ++`git maintenance start` with multiple Git executables, then only the ++latest executable will be used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically -+loaded from the user-specific global config. The `git maintenance` process -+then determines which maintenance tasks are configured to run on each -+repository with each `<frequency>` using the `maintenance.<task>.schedule` -+config options. These values are loaded from the global or repository -+config values. ++loaded from the user-specific global config located at `~/.gitconfig`. ++The `git maintenance` process then determines which maintenance tasks ++are configured to run on each repository with each `<frequency>` using ++the `maintenance.<task>.schedule` config options. These values are loaded ++from the global or repository config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read -+the crontab(5) documentation for advanced scheduling techniques. Please -+do use the full path and `--exec-path` techniques from the default -+schedule to ensure you are executing the correct binaries in your -+schedule. ++https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] ++for advanced scheduling techniques. Please do use the full path and ++`--exec-path` techniques from the default schedule to ensure you are ++executing the correct binaries in your schedule. + GIT 3: 1629bcfcf8 ! 3: ed0a0011fb maintenance: use launchctl on macOS @@ Commit message plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. + The need for a user id requires us to run 'id -u' which works on + POSIX systems but not Windows. The test therefore has a prerequisite + that we are not on Windows. The cross-platform logic still allows us to + test the macOS logic on a Linux machine. + We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the - command-line arguments into GIT_TEST_CRONTAB. + command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify - the XML format, so call it from the macOS-specific tests to ensure - the .plist files are well-formatted. + the XML format. This is useful for any system that might contain + the tool, so use it whenever it is available. - Helped-by: Eric Sunshine <sunshine@sunshineco.com> + Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> + Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## Documentation/git-maintenance.txt ## -@@ Documentation/git-maintenance.txt: schedule to ensure you are executing the correct binaries in your - schedule. +@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do use the full path and + executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS @@ builtin/gc.c: static int maintenance_unregister(void) return run_command(&config_unset); } -+#if defined(__APPLE__) ++static const char *get_frequency(enum schedule_priority schedule) ++{ ++ switch (schedule) { ++ case SCHEDULE_HOURLY: ++ return "hourly"; ++ case SCHEDULE_DAILY: ++ return "daily"; ++ case SCHEDULE_WEEKLY: ++ return "weekly"; ++ default: ++ BUG("invalid schedule %d", schedule); ++ } ++} + -+static char *get_service_name(const char *frequency) ++static char *launchctl_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + -+static char *get_service_filename(const char *name) ++static char *launchctl_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; @@ builtin/gc.c: static int maintenance_unregister(void) + return expanded; +} + -+static const char *get_frequency(enum schedule_priority schedule) ++static char *launchctl_get_uid(void) +{ -+ switch (schedule) { -+ case SCHEDULE_HOURLY: -+ return "hourly"; -+ case SCHEDULE_DAILY: -+ return "daily"; -+ case SCHEDULE_WEEKLY: -+ return "weekly"; -+ default: -+ BUG("invalid schedule %d", schedule); -+ } ++ return xstrfmt("gui/%d", getuid()); +} + -+static char *get_uid(void) -+{ -+ struct strbuf output = STRBUF_INIT; -+ struct child_process id = CHILD_PROCESS_INIT; -+ -+ strvec_pushl(&id.args, "/usr/bin/id", "-u", NULL); -+ if (capture_command(&id, &output, 0)) -+ die(_("failed to discover user id")); -+ -+ strbuf_trim_trailing_newline(&output); -+ return strbuf_detach(&output, NULL); -+} -+ -+static int boot_plist(int enable, const char *filename) ++static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; -+ char *uid = get_uid(); -+ const char *launchctl = getenv("GIT_TEST_CRONTAB"); -+ if (!launchctl) -+ launchctl = "/bin/launchctl"; -+ -+ strvec_split(&child.args, launchctl); ++ char *uid = launchctl_get_uid(); + ++ strvec_split(&child.args, cmd); + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); -+ strvec_pushf(&child.args, "gui/%s", uid); ++ strvec_push(&child.args, uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; @@ builtin/gc.c: static int maintenance_unregister(void) + return result; +} + -+static int remove_plist(enum schedule_priority schedule) ++static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) +{ + const char *frequency = get_frequency(schedule); -+ char *name = get_service_name(frequency); -+ char *filename = get_service_filename(name); -+ int result = boot_plist(0, filename); ++ char *name = launchctl_service_name(frequency); ++ char *filename = launchctl_service_filename(name); ++ int result = launchctl_boot_plist(0, filename, cmd); + unlink(filename); + free(filename); + free(name); + return result; +} + -+static int remove_plists(void) ++static int launchctl_remove_plists(const char *cmd) +{ -+ return remove_plist(SCHEDULE_HOURLY) || -+ remove_plist(SCHEDULE_DAILY) || -+ remove_plist(SCHEDULE_WEEKLY); ++ return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || ++ launchctl_remove_plist(SCHEDULE_DAILY, cmd) || ++ launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); +} + -+static int schedule_plist(const char *exec_path, enum schedule_priority schedule) ++static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); -+ char *name = get_service_name(frequency); -+ char *filename = get_service_filename(name); ++ char *name = launchctl_service_name(frequency); ++ char *filename = launchctl_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + -+ preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" ++ preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" @@ builtin/gc.c: static int maintenance_unregister(void) + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); ++ fclose(plist); + + /* bootout might fail if not already running, so ignore */ -+ boot_plist(0, filename); -+ if (boot_plist(1, filename)) ++ launchctl_boot_plist(0, filename, cmd); ++ if (launchctl_boot_plist(1, filename, cmd)) + die(_("failed to bootstrap service %s"), filename); + -+ fclose(plist); + free(filename); + free(name); + return 0; +} + -+static int add_plists(void) ++static int launchctl_add_plists(const char *cmd) +{ + const char *exec_path = git_exec_path(); + -+ return schedule_plist(exec_path, SCHEDULE_HOURLY) || -+ schedule_plist(exec_path, SCHEDULE_DAILY) || -+ schedule_plist(exec_path, SCHEDULE_WEEKLY); ++ return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || ++ launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || ++ launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); +} + -+static int platform_update_schedule(int run_maintenance, int fd) ++static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) -+ return add_plists(); ++ return launchctl_add_plists(cmd); + else -+ return remove_plists(); ++ return launchctl_remove_plists(cmd); +} -+#else ++ #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -@@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) - fclose(cron_list); +@@ builtin/gc.c: static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) return result; } + ++#if defined(__APPLE__) ++static const char platform_scheduler[] = "launchctl"; ++#else + static const char platform_scheduler[] = "crontab"; +#endif - static int update_background_schedule(int run_maintenance) + static int update_background_schedule(int enable) { +@@ builtin/gc.c: static int update_background_schedule(int enable) + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + +- if (!strcmp(scheduler, "crontab")) ++ if (!strcmp(scheduler, "launchctl")) ++ result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); ++ else if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); ## t/t7900-maintenance.sh ## -@@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' ' - test_cmp before actual - ' - --test_expect_success 'start from empty cron table' ' -+test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && - - # start registers the repo -@@ t/t7900-maintenance.sh: test_expect_success 'start from empty cron table' ' - grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt - ' - --test_expect_success 'stop from existing schedule' ' -+test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && +@@ t/t7900-maintenance.sh: test_description='git maintenance builtin' + GIT_TEST_COMMIT_GRAPH=0 + GIT_TEST_MULTI_PACK_INDEX=0 - # stop does not unregister the repo -@@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' - test_must_be_empty cron.txt - ' - --test_expect_success 'start preserves existing schedule' ' -+test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' - echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && ++test_lazy_prereq XMLLINT ' ++ xmllint --version ++' ++ ++test_xmllint () { ++ if test_have_prereq XMLLINT ++ then ++ xmllint --noout "$@" ++ else ++ true ++ fi ++} ++ + test_expect_success 'help text' ' + test_expect_code 129 git maintenance -h 2>err && + test_i18ngrep "usage: git maintenance <subcommand>" err && +@@ t/t7900-maintenance.sh: test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' -+test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' -+ write_script print-args "#!/bin/sh\necho \$* >>args" && ++test_expect_success !MINGW 'start and stop macOS maintenance' ' ++ uid=$(id -u) && ++ ++ write_script print-args <<-\EOF && ++ echo $* >>args ++ EOF + + rm -f args && -+ GIT_TEST_CRONTAB="./print-args" git maintenance start && ++ GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ # ~/Library/LaunchAgents + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && -+ xmllint --noout "$PLIST" && ++ test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && -+ echo "bootout gui/$UID $PLIST" >>expect && -+ echo "bootstrap gui/$UID $PLIST" >>expect || return 1 ++ echo "bootout gui/$uid $PLIST" >>expect && ++ echo "bootstrap gui/$uid $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && -+ GIT_TEST_CRONTAB="./print-args" git maintenance stop && ++ GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ ++ printf "bootout gui/$uid $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && @@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' ' test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && - - ## t/test-lib.sh ## -@@ t/test-lib.sh: test_lazy_prereq REBASE_P ' - test -z "$GIT_TEST_SKIP_REBASE_P" - ' - -+test_lazy_prereq MACOS_MAINTENANCE ' -+ launchctl list -+' -+ - # Ensure that no test accidentally triggers a Git command - # that runs 'crontab', affecting a user's cron schedule. - # Tests that verify the cron integration must set this locally 4: ed7a61978f ! 4: b8d86fb983 maintenance: use Windows scheduled tasks @@ Commit message logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. + Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to + specify 'schtasks' as the scheduler, we can test the Windows-specific + logic on a macOS platform. Thus, add a check that the XML file written + by Git is valid when xmllint exists on the system. + There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached @@ Commit message short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. - Helped-by: Eric Sunshine <sunshine@sunshineco.com> + Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> + Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## Documentation/git-maintenance.txt ## @@ Documentation/git-maintenance.txt: To create more advanced customizations to you Part of the linkgit:git[1] suite ## builtin/gc.c ## -@@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) - else - return remove_plists(); +@@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm + return launchctl_remove_plists(cmd); } -+ -+#elif defined(GIT_WINDOWS_NATIVE) -+ -+static const char *get_frequency(enum schedule_priority schedule) -+{ -+ switch (schedule) { -+ case SCHEDULE_HOURLY: -+ return "hourly"; -+ case SCHEDULE_DAILY: -+ return "daily"; -+ case SCHEDULE_WEEKLY: -+ return "weekly"; -+ default: -+ BUG("invalid schedule %d", schedule); -+ } -+} -+ -+static char *get_task_name(const char *frequency) + ++static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + -+static int remove_task(enum schedule_priority schedule) ++static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); -+ char *name = get_task_name(frequency); -+ const char *schtasks = getenv("GIT_TEST_CRONTAB"); -+ if (!schtasks) -+ schtasks = "schtasks"; ++ char *name = schtasks_task_name(frequency); + -+ strvec_split(&args, schtasks); ++ strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + return result; +} + -+static int remove_scheduled_tasks(void) ++static int schtasks_remove_tasks(const char *cmd) +{ -+ return remove_task(SCHEDULE_HOURLY) || -+ remove_task(SCHEDULE_DAILY) || -+ remove_task(SCHEDULE_WEEKLY); ++ return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || ++ schtasks_remove_task(SCHEDULE_DAILY, cmd) || ++ schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + -+static int schedule_task(const char *exec_path, enum schedule_priority schedule) ++static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; -+ const char *xml, *schtasks; -+ char *xmlpath, *tempDir; -+ FILE *xmlfp; ++ const char *xml; ++ char *xmlpath; ++ struct tempfile *tfile; + const char *frequency = get_frequency(schedule); -+ char *name = get_task_name(frequency); ++ char *name = schtasks_task_name(frequency); + -+ tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path); -+ xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency); -+ safe_create_leading_directories(xmlpath); -+ xmlfp = xfopen(xmlpath, "w"); ++ xmlpath = xstrfmt("%s/schedule-%s.xml", ++ the_repository->objects->odb->path, ++ frequency); ++ tfile = create_tempfile(xmlpath); ++ if (!tfile || !fdopen_tempfile(tfile, "w")) ++ die(_("failed to create '%s'"), xmlpath); + -+ xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n" ++ xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; -+ fprintf(xmlfp, xml); ++ fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: -+ fprintf(xmlfp, ++ fprintf(tfile->fp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + break; + + case SCHEDULE_DAILY: -+ fprintf(xmlfp, ++ fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + break; + + case SCHEDULE_WEEKLY: -+ fprintf(xmlfp, ++ fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + break; + } + -+ xml= "</CalendarTrigger>\n" ++ xml = "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; -+ fprintf(xmlfp, xml, exec_path, exec_path, frequency); -+ fclose(xmlfp); -+ -+ schtasks = getenv("GIT_TEST_CRONTAB"); -+ if (!schtasks) -+ schtasks = "schtasks"; -+ strvec_split(&child.args, schtasks); ++ fprintf(tfile->fp, xml, exec_path, exec_path, frequency); ++ strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); ++ close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; @@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd) + die(_("failed to start schtasks")); + result = finish_command(&child); + -+ unlink(xmlpath); -+ rmdir(tempDir); ++ delete_tempfile(&tfile); + free(xmlpath); + free(name); + return result; +} + -+static int add_scheduled_tasks(void) ++static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + -+ return schedule_task(exec_path, SCHEDULE_HOURLY) || -+ schedule_task(exec_path, SCHEDULE_DAILY) || -+ schedule_task(exec_path, SCHEDULE_WEEKLY); ++ return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || ++ schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || ++ schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + -+static int platform_update_schedule(int run_maintenance, int fd) ++static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) -+ return add_scheduled_tasks(); ++ return schtasks_schedule_tasks(cmd); + else -+ return remove_scheduled_tasks(); ++ return schtasks_remove_tasks(cmd); +} + - #else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" - - ## t/t7900-maintenance.sh ## -@@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' ' - test_cmp before actual - ' - --test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' -+test_expect_success !MACOS_MAINTENANCE,!MINGW 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && - - # start registers the repo -@@ t/t7900-maintenance.sh: test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' - grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt - ' --test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' -+test_expect_success !MACOS_MAINTENANCE,!MINGW 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && +@@ builtin/gc.c: static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) - # stop does not unregister the repo -@@ t/t7900-maintenance.sh: test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' - test_must_be_empty cron.txt - ' + #if defined(__APPLE__) + static const char platform_scheduler[] = "launchctl"; ++#elif defined(GIT_WINDOWS_NATIVE) ++static const char platform_scheduler[] = "schtasks"; + #else + static const char platform_scheduler[] = "crontab"; + #endif +@@ builtin/gc.c: static int update_background_schedule(int enable) --test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' -+test_expect_success !MACOS_MAINTENANCE,!MINGW 'start preserves existing schedule' ' - echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && - grep "Important information!" cron.txt -@@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); ++ else if (!strcmp(scheduler, "schtasks")) ++ result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + + ## t/t7900-maintenance.sh ## +@@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS maintenance' ' test_line_count = 0 actual ' -+test_expect_success MINGW 'start and stop Windows maintenance' ' ++test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args ++ while test $# -gt 0 ++ do ++ case "$1" in ++ /xml) shift; xmlfile=$1; break ;; ++ *) shift ;; ++ esac ++ done ++ test -z "$xmlfile" || cp "$xmlfile" . + EOF + + rm -f args && -+ GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance start && ++ GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/temp/schedule-%s.xml\n" \ ++ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ + hourly hourly daily daily weekly weekly >expect && + test_cmp expect args && + ++ for frequency in hourly daily weekly ++ do ++ test_xmllint "schedule-$frequency.xml" ++ done && ++ + rm -f args && -+ GIT_TEST_CRONTAB="/bin/sh print-args" git maintenance stop && ++ GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v4 1/4] maintenance: extract platform-specific scheduling 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 ` Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget ` (5 subsequent siblings) 6 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as "<scheduler>:<command>" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the <scheduler> in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 70 ++++++++++++++++++++++++++---------------- t/t7900-maintenance.sh | 8 ++--- t/test-lib.sh | 7 +++-- 3 files changed, 51 insertions(+), 34 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a1..18ae7f7138 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,35 +1494,23 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1531,17 +1519,15 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { @@ -1586,14 +1572,44 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; + else + fclose(cron_list); + return result; +} + +static const char platform_scheduler[] = "crontab"; + +static int update_background_schedule(int enable) +{ + int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; } - fclose(cron_list); -cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); + rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..eeb939168d 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -368,7 +368,7 @@ test_expect_success 'register and unregister' ' ' test_expect_success 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -379,19 +379,19 @@ test_expect_success 'start from empty cron table' ' ' test_expect_success 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..ddbeee1f5e 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v4 2/4] maintenance: include 'cron' details in docs 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 ` Derrick Stolee via GitGitGadget 2020-11-18 0:34 ` Eric Sunshine 2020-11-17 21:13 ` [PATCH v4 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget ` (4 subsequent siblings) 6 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc..4c7aac877d 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is `cron`. This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `<path>` string is loaded to specifically use the location for the +`git` executable used in the `git maintenance start` command. This allows +for multiple versions to be compatible. However, if the same user runs +`git maintenance start` with multiple Git executables, then only the +latest executable will be used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config located at `~/.gitconfig`. +The `git maintenance` process then determines which maintenance tasks +are configured to run on each repository with each `<frequency>` using +the `maintenance.<task>.schedule` config options. These values are loaded +from the global or repository config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] +for advanced scheduling techniques. Please do use the full path and +`--exec-path` techniques from the default schedule to ensure you are +executing the correct binaries in your schedule. + GIT --- -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v4 2/4] maintenance: include 'cron' details in docs 2020-11-17 21:13 ` [PATCH v4 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-11-18 0:34 ` Eric Sunshine 2020-11-18 18:30 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-18 0:34 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee, Derrick Stolee On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > Advanced and expert users may want to know how 'git maintenance start' > schedules background maintenance in order to customize their own > schedules beyond what the maintenance.* config values allow. Start a new > set of sections in git-maintenance.txt that describe how 'cron' is used > to run these tasks. > > This is particularly valuable for users who want to inspect what Git is > doing or for users who want to customize the schedule further. Having a > baseline can provide a way forward for users who have never worked with > cron schedules. > > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt > @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with > +The comments are used as a region to mark the schedule as written by Git. > +Any modifications within this region will be completely deleted by > +`git maintenance stop` or overwritten by `git maintenance start`. > + > +The `<path>` string is loaded to specifically use the location for the > +`git` executable used in the `git maintenance start` command. This allows > +for multiple versions to be compatible. However, if the same user runs > +`git maintenance start` with multiple Git executables, then only the > +latest executable will be used. It looks like this section in v4 got accidentally reverted to the wording from v2, whereas v3 had been changed to: The `crontab` entry specifies the full path of the `git` executable to ensure that the executed `git` command is the same one with which `git maintenance start` was issued independent of `PATH`. If the same user runs `git maintenance start` with multiple Git executables, then only the latest executable is used. > +These commands use `git for-each-repo --config=maintenance.repo` to run > +`git maintenance run --schedule=<frequency>` on each repository listed in > +the multi-valued `maintenance.repo` config option. These are typically > +loaded from the user-specific global config located at `~/.gitconfig`. > +The `git maintenance` process then determines which maintenance tasks > +are configured to run on each repository with each `<frequency>` using > +the `maintenance.<task>.schedule` config options. These values are loaded > +from the global or repository config values. Same problem here. This wording is from v2, whereas v3 had been changed to say generically "user-specific global config" rather than mentioning `~/.gitconfig` explicitly. > +If the config values are insufficient to achieve your desired background > +maintenance schedule, then you can create your own schedule. If you run > +`crontab -e`, then an editor will load with your user-specific `cron` > +schedule. In that editor, you can add your own schedule lines. You could > +start by adapting the default schedule listed earlier, or you could read > +https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] > +for advanced scheduling techniques. Please do use the full path and > +`--exec-path` techniques from the default schedule to ensure you are > +executing the correct binaries in your schedule. And here too. v3 had updated this to say only "crontab(5)" rather than providing an explicit URL. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 2/4] maintenance: include 'cron' details in docs 2020-11-18 0:34 ` Eric Sunshine @ 2020-11-18 18:30 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-18 18:30 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee On 11/17/2020 7:34 PM, Eric Sunshine wrote: > On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> Advanced and expert users may want to know how 'git maintenance start' >> schedules background maintenance in order to customize their own >> schedules beyond what the maintenance.* config values allow. Start a new >> set of sections in git-maintenance.txt that describe how 'cron' is used >> to run these tasks. >> >> This is particularly valuable for users who want to inspect what Git is >> doing or for users who want to customize the schedule further. Having a >> baseline can provide a way forward for users who have never worked with >> cron schedules. >> >> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> >> --- >> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt >> @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with >> +The comments are used as a region to mark the schedule as written by Git. >> +Any modifications within this region will be completely deleted by >> +`git maintenance stop` or overwritten by `git maintenance start`. >> + >> +The `<path>` string is loaded to specifically use the location for the >> +`git` executable used in the `git maintenance start` command. This allows >> +for multiple versions to be compatible. However, if the same user runs >> +`git maintenance start` with multiple Git executables, then only the >> +latest executable will be used. > > It looks like this section in v4 got accidentally reverted to the > wording from v2, whereas v3 had been changed to:> > The `crontab` entry specifies the full path of the `git` > executable to ensure that the executed `git` command is the same > one with which `git maintenance start` was issued independent of > `PATH`. If the same user runs `git maintenance start` with > multiple Git executables, then only the latest executable is used. Embarassing. Thanks for the catch. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v4 3/4] maintenance: use launchctl on macOS 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 ` Derrick Stolee via GitGitGadget 2020-11-18 6:45 ` Eric Sunshine 2020-11-17 21:13 ` [PATCH v4 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget ` (3 subsequent siblings) 6 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on POSIX systems but not Windows. The test therefore has a prerequisite that we are not on Windows. The cross-platform logic still allows us to test the macOS logic on a Linux machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 40 +++++++ builtin/gc.c | 188 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 58 +++++++++ 3 files changed, 285 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 4c7aac877d..f2d59f2bcc 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ for advanced scheduling techniques. Please do use the full path and executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=<frequency>` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `<array>` element following +the `<key>StartCalendarInterval</key>` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 18ae7f7138..782769f243 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,186 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *launchctl_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *launchctl_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static char *launchctl_get_uid(void) +{ + return xstrfmt("gui/%d", getuid()); +} + +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = launchctl_get_uid(); + + strvec_split(&child.args, cmd); + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_push(&child.args, uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) +{ + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + int result = launchctl_boot_plist(0, filename, cmd); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int launchctl_remove_plists(const char *cmd) +{ + return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || + launchctl_remove_plist(SCHEDULE_DAILY, cmd) || + launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + fclose(plist); + + /* bootout might fail if not already running, so ignore */ + launchctl_boot_plist(0, filename, cmd); + if (launchctl_boot_plist(1, filename, cmd)) + die(_("failed to bootstrap service %s"), filename); + + free(filename); + free(name); + return 0; +} + +static int launchctl_add_plists(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return launchctl_add_plists(cmd); + else + return launchctl_remove_plists(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1579,7 +1759,11 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) return result; } +#if defined(__APPLE__) +static const char platform_scheduler[] = "launchctl"; +#else static const char platform_scheduler[] = "crontab"; +#endif static int update_background_schedule(int enable) { @@ -1603,7 +1787,9 @@ static int update_background_schedule(int enable) if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) return error(_("another process is scheduling background maintenance")); - if (!strcmp(scheduler, "crontab")) + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else die("unknown background scheduler: %s", scheduler); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index eeb939168d..6d37312901 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -7,6 +7,19 @@ test_description='git maintenance builtin' GIT_TEST_COMMIT_GRAPH=0 GIT_TEST_MULTI_PACK_INDEX=0 +test_lazy_prereq XMLLINT ' + xmllint --version +' + +test_xmllint () { + if test_have_prereq XMLLINT + then + xmllint --noout "$@" + else + true + fi +} + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance <subcommand>" err && @@ -395,6 +408,51 @@ test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' +test_expect_success !MINGW 'start and stop macOS maintenance' ' + uid=$(id -u) && + + write_script print-args <<-\EOF && + echo $* >>args + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$uid $PLIST" >>expect && + echo "bootstrap gui/$uid $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/$uid $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v4 3/4] maintenance: use launchctl on macOS 2020-11-17 21:13 ` [PATCH v4 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-18 6:45 ` Eric Sunshine 2020-11-18 18:22 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-18 6:45 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee, Derrick Stolee On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) > +{ > + child.no_stderr = 1; > + child.no_stdout = 1; > + if (start_command(&child)) > + die(_("failed to start launchctl")); Did you have any thoughts on the observation I made in a followup response[1] during review of v3 in which I suggested that we might be able to avoid suppressing stderr (and stdout) here? In particular, the idea was that if, in launchctl_schedule_plist(), we do a simple existence check for the .plist file and only call launchctl_boot_plist(0,...) to `bootout` the .plist file if it exists, then we shouldn't need to muck with stderr/stdout suppression. The benefit is that if `bootout` fails for some reason, then the user would see the (hopefully) meaningful error message emitted `launchctl bootout`. The same .plist existence check could be done in launchctl_remove_plist() before trying to `bootout` the file. Anyhow, such refinement can be done later is desired, so not worth a re-roll, but I was curious about your thoughts on the issue. [1]: https://lore.kernel.org/git/CAPig+cTRJb-fn2R6rJO1hkeCc_ehVhkNufO4=LhtPQudVeonnA@mail.gmail.com/ ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 3/4] maintenance: use launchctl on macOS 2020-11-18 6:45 ` Eric Sunshine @ 2020-11-18 18:22 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-18 18:22 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee On 11/18/2020 1:45 AM, Eric Sunshine wrote: > On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) >> +{ >> + child.no_stderr = 1; >> + child.no_stdout = 1; >> + if (start_command(&child)) >> + die(_("failed to start launchctl")); > > Did you have any thoughts on the observation I made in a followup > response[1] during review of v3 in which I suggested that we might be > able to avoid suppressing stderr (and stdout) here? In particular, the > idea was that if, in launchctl_schedule_plist(), we do a simple > existence check for the .plist file and only call > launchctl_boot_plist(0,...) to `bootout` the .plist file if it exists, > then we shouldn't need to muck with stderr/stdout suppression. The > benefit is that if `bootout` fails for some reason, then the user > would see the (hopefully) meaningful error message emitted `launchctl > bootout`. > > The same .plist existence check could be done in > launchctl_remove_plist() before trying to `bootout` the file. If the file exists but isn't still registered with launchd, then the bootout command will send output "<path>.plist: Could not find specified service" and return a failure code. This output isn't helpful to users, since we still are in the desired state afterwards. > Anyhow, such refinement can be done later is desired, so not worth a > re-roll, but I was curious about your thoughts on the issue. This pattern of squashing the output even in the successful case is important for Windows where schtasks sends a line of output for each task being registered! I think it would be a reasonable extension to store the error message for logging or communicating to the user if we actually need it, but I don't believe we should be piping the output directly. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v4 4/4] maintenance: use Windows scheduled tasks 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2020-11-17 21:13 ` [PATCH v4 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 ` Derrick Stolee via GitGitGadget 2020-11-18 7:15 ` Eric Sunshine 2020-11-17 23:36 ` [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine ` (2 subsequent siblings) 6 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-17 21:13 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on a macOS platform. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 165 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 40 ++++++++ 3 files changed, 227 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index f2d59f2bcc..e1adfff6db 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance (<frequency>)`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 782769f243..f6c42f96c1 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1671,6 +1671,167 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } +static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int schtasks_remove_tasks(const char *cmd) +{ + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; + char *xmlpath; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + xmlpath = xstrfmt("%s/schedule-%s.xml", + the_repository->objects->odb->path, + frequency); + tfile = create_tempfile(xmlpath); + if (!tfile || !fdopen_tempfile(tfile, "w")) + die(_("failed to create '%s'"), xmlpath); + + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml = "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + delete_tempfile(&tfile); + free(xmlpath); + free(name); + return result; +} + +static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return schtasks_schedule_tasks(cmd); + else + return schtasks_remove_tasks(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1761,6 +1922,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) #if defined(__APPLE__) static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; #else static const char platform_scheduler[] = "crontab"; #endif @@ -1789,6 +1952,8 @@ static int update_background_schedule(int enable) if (!strcmp(scheduler, "launchctl")) result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 6d37312901..a26ff22541 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -453,6 +453,46 @@ test_expect_success !MINGW 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + test -z "$xmlfile" || cp "$xmlfile" . + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ + hourly hourly daily daily weekly weekly >expect && + test_cmp expect args && + + for frequency in hourly daily weekly + do + test_xmllint "schedule-$frequency.xml" + done && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v4 4/4] maintenance: use Windows scheduled tasks 2020-11-17 21:13 ` [PATCH v4 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-18 7:15 ` Eric Sunshine 2020-11-18 18:30 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-18 7:15 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee, Derrick Stolee On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > [...] > Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to > specify 'schtasks' as the scheduler, we can test the Windows-specific > logic on a macOS platform. Thus, add a check that the XML file written > by Git is valid when xmllint exists on the system. Nit: xmllint can be installed on Linux (and likely other platforms), as well, so it's not clear why this calls out macOS specially. More generally, it may not be important to call out xmllint at all in the commit message; it's just _one_ thing being checked by a test which is checking several other things not called out individually by the commit message. Anyhow, this is minor; not worth a re-roll. > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1671,6 +1671,167 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm > +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) > +{ > + xmlpath = xstrfmt("%s/schedule-%s.xml", > + the_repository->objects->odb->path, > + frequency); I missed this in the earlier rounds since I wasn't paying close enough attention, but placing this XML file within the object database directory (.git/objects/) feels rather odd, even if it is just a temporary file. Using the .git/ directory itself might be better, perhaps like this: struct strbuf xmlpath = STRBUF_INIT; strbuf_git_common_path(&xmlpath, the_repository, "schtasks-%s.xml", frequency); ... strbuf_release(&xmlpath); ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 4/4] maintenance: use Windows scheduled tasks 2020-11-18 7:15 ` Eric Sunshine @ 2020-11-18 18:30 ` Derrick Stolee 2020-11-18 20:54 ` Eric Sunshine 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-18 18:30 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee On 11/18/2020 2:15 AM, Eric Sunshine wrote: > On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: >> [...] >> Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to >> specify 'schtasks' as the scheduler, we can test the Windows-specific >> logic on a macOS platform. Thus, add a check that the XML file written >> by Git is valid when xmllint exists on the system. > > Nit: xmllint can be installed on Linux (and likely other platforms), > as well, so it's not clear why this calls out macOS specially. More > generally, it may not be important to call out xmllint at all in the > commit message; it's just _one_ thing being checked by a test which is > checking several other things not called out individually by the > commit message. Anyhow, this is minor; not worth a re-roll. Sorry, it should just say "other platforms" >> diff --git a/builtin/gc.c b/builtin/gc.c >> @@ -1671,6 +1671,167 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm >> +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) >> +{ >> + xmlpath = xstrfmt("%s/schedule-%s.xml", >> + the_repository->objects->odb->path, >> + frequency); > > I missed this in the earlier rounds since I wasn't paying close enough > attention, but placing this XML file within the object database > directory (.git/objects/) feels rather odd, even if it is just a > temporary file. Using the .git/ directory itself might be better, > perhaps like this: > > struct strbuf xmlpath = STRBUF_INIT; > strbuf_git_common_path(&xmlpath, the_repository, > "schtasks-%s.xml", frequency); > ... > strbuf_release(&xmlpath); It does look odd, and in this case we could use the .git directory instead. I specifically use the objects directory for the maintenance lock in 'git maintenance run' to allow maintenance to run when GIT_OBJECT_DIRECTORY points to an alternate, allowing us to maintain object databases that don't have a full .git directory around them. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 4/4] maintenance: use Windows scheduled tasks 2020-11-18 18:30 ` Derrick Stolee @ 2020-11-18 20:54 ` Eric Sunshine 2020-11-18 21:16 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-18 20:54 UTC (permalink / raw) To: Derrick Stolee Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee, Derrick Stolee On Wed, Nov 18, 2020 at 1:30 PM Derrick Stolee <stolee@gmail.com> wrote: > On 11/18/2020 2:15 AM, Eric Sunshine wrote: > > On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget > > <gitgitgadget@gmail.com> wrote: > >> + xmlpath = xstrfmt("%s/schedule-%s.xml", > >> + the_repository->objects->odb->path, > >> + frequency); > > > > I missed this in the earlier rounds since I wasn't paying close enough > > attention, but placing this XML file within the object database > > directory (.git/objects/) feels rather odd, even if it is just a > > temporary file. Using the .git/ directory itself might be better, > > perhaps like this: > > It does look odd, and in this case we could use the .git directory > instead. I specifically use the objects directory for the maintenance > lock in 'git maintenance run' to allow maintenance to run when > GIT_OBJECT_DIRECTORY points to an alternate, allowing us to maintain > object databases that don't have a full .git directory around them. I guess I'm confused. Won't a Git "common" directory exist even for such a case when GIT_OBJECT_DIRECTORY is pointing elsewhere, whether the "common" directory is .git/ or a bare repository, or whatnot? Anyhow, this brings us back to my original suggestion of creating these temporary files in a genuine temporary directory (/tmp or $TMPDIR or $TEMP) instead of arbitrarily choosing a path within the repository itself. An important reason for using a genuine temporary directory for these temporary XML files is that it makes it less confusing for those who come along later and try to understand this code; they won't have to puzzle out why it is using a repository location for a file which is clearly temporary. To make this really simple, you could use one of the x?mks_tempfile_t*() functions from tempfile.h which will automatically place the file in $TMPDIR, thus relieving this code from having to make the choice. Doing so would simplify this code even further since you would replace create_tempfile() with x?mks_tempfile_t*(), and wouldn't have to maintain (or free) `xmlpath` manually. As for the test script, the `print-args` is already picking up the pathname of the temporary file specified by the /xml option, so it should be possible to make the rest of the test work with the generated temporary filename. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 4/4] maintenance: use Windows scheduled tasks 2020-11-18 20:54 ` Eric Sunshine @ 2020-11-18 21:16 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2020-11-18 21:16 UTC (permalink / raw) To: Eric Sunshine Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee, Derrick Stolee On 11/18/2020 3:54 PM, Eric Sunshine wrote: > On Wed, Nov 18, 2020 at 1:30 PM Derrick Stolee <stolee@gmail.com> wrote: >> On 11/18/2020 2:15 AM, Eric Sunshine wrote: >>> On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget >>> <gitgitgadget@gmail.com> wrote: >>>> + xmlpath = xstrfmt("%s/schedule-%s.xml", >>>> + the_repository->objects->odb->path, >>>> + frequency); >>> >>> I missed this in the earlier rounds since I wasn't paying close enough >>> attention, but placing this XML file within the object database >>> directory (.git/objects/) feels rather odd, even if it is just a >>> temporary file. Using the .git/ directory itself might be better, >>> perhaps like this: >> >> It does look odd, and in this case we could use the .git directory >> instead. I specifically use the objects directory for the maintenance >> lock in 'git maintenance run' to allow maintenance to run when >> GIT_OBJECT_DIRECTORY points to an alternate, allowing us to maintain >> object databases that don't have a full .git directory around them. > > I guess I'm confused. Won't a Git "common" directory exist even for > such a case when GIT_OBJECT_DIRECTORY is pointing elsewhere, whether > the "common" directory is .git/ or a bare repository, or whatnot? The reason to use the object dir for the 'git maintenance run' lock is to avoid multiple enlistments pointing at a common alternate from running concurrent maintenance on the same object directory. That doesn't really apply to the temp files in this patch. > Anyhow, this brings us back to my original suggestion of creating > these temporary files in a genuine temporary directory (/tmp or > $TMPDIR or $TEMP) instead of arbitrarily choosing a path within the > repository itself. An important reason for using a genuine temporary > directory for these temporary XML files is that it makes it less > confusing for those who come along later and try to understand this > code; they won't have to puzzle out why it is using a repository > location for a file which is clearly temporary. > > To make this really simple, you could use one of the > x?mks_tempfile_t*() functions from tempfile.h which will automatically > place the file in $TMPDIR, thus relieving this code from having to > make the choice. Doing so would simplify this code even further since > you would replace create_tempfile() with x?mks_tempfile_t*(), and > wouldn't have to maintain (or free) `xmlpath` manually. > > As for the test script, the `print-args` is already picking up the > pathname of the temporary file specified by the /xml option, so it > should be possible to make the rest of the test work with the > generated temporary filename. I'll adopt your recommendations here. Thanks. -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget ` (3 preceding siblings ...) 2020-11-17 21:13 ` [PATCH v4 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-17 23:36 ` Eric Sunshine 2020-11-24 2:20 ` Derrick Stolee 2020-11-17 23:54 ` Eric Sunshine 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget 6 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-11-17 23:36 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget; +Cc: git, Derrick Stolee, Derrick Stolee On Tue, Nov 17, 2020 at 09:13:14PM +0000, Derrick Stolee via GitGitGadget wrote: > Updates in V4 > ============= > * Eric did an excellent job providing a patch that cleans up several parts > of my series. The most impressive is his mechanism for testing the > platform-specific Git logic in a way that is (mostly) platform-agnostic. > > * Windows doesn't have the 'id' command, so we cannot run the macOS > platform test on Windows. This is easy to resolve. Drop in the following patch and then replace the `$(id -u)` invocation in the test with `$(test-tool getuid)`. This way, the test should work on any platform since both launchctl_get_uid() and `test-tool` will retrieve identical values for UID. --- >8 --- From 84f623bcaec156082c0e7151f40aef18575e6f86 Mon Sep 17 00:00:00 2001 From: Eric Sunshine <sunshine@sunshineco.com> Date: Tue, 17 Nov 2020 18:30:10 -0500 Subject: [PATCH] test-helper: add `getuid` subcommand Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> --- Makefile | 1 + t/helper/test-getuid.c | 7 +++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 10 insertions(+) create mode 100644 t/helper/test-getuid.c diff --git a/Makefile b/Makefile index 790a883932..230aff5e5c 100644 --- a/Makefile +++ b/Makefile @@ -706,6 +706,7 @@ TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o +TEST_BUILTINS_OBJS += test-getuid.o TEST_BUILTINS_OBJS += test-hash-speed.o TEST_BUILTINS_OBJS += test-hash.o TEST_BUILTINS_OBJS += test-hashmap.o diff --git a/t/helper/test-getuid.c b/t/helper/test-getuid.c new file mode 100644 index 0000000000..d741302461 --- /dev/null +++ b/t/helper/test-getuid.c @@ -0,0 +1,7 @@ +#include "test-tool.h" + +int cmd__getuid(int argc, const char **argv) +{ + printf("%d\n", getuid()); + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index a0d3966b29..ab206541df 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -30,6 +30,7 @@ static struct test_cmd cmds[] = { { "example-decorate", cmd__example_decorate }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, + { "getuid", cmd__getuid }, { "hashmap", cmd__hashmap }, { "hash-speed", cmd__hash_speed }, { "index-version", cmd__index_version }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 07034d3f38..caee0a3667 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -20,6 +20,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv); int cmd__example_decorate(int argc, const char **argv); int cmd__genrandom(int argc, const char **argv); int cmd__genzeros(int argc, const char **argv); +int cmd__getuid(int argc, const char **argv); int cmd__hashmap(int argc, const char **argv); int cmd__hash_speed(int argc, const char **argv); int cmd__index_version(int argc, const char **argv); -- 2.29.2.454.gaff20da3a2 ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-17 23:36 ` [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine @ 2020-11-24 2:20 ` Derrick Stolee 2020-11-24 2:59 ` Eric Sunshine 0 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee @ 2020-11-24 2:20 UTC (permalink / raw) To: Eric Sunshine, Derrick Stolee via GitGitGadget; +Cc: git, Derrick Stolee On 11/17/2020 6:36 PM, Eric Sunshine wrote: > On Tue, Nov 17, 2020 at 09:13:14PM +0000, Derrick Stolee via GitGitGadget wrote: >> Updates in V4 >> ============= >> * Eric did an excellent job providing a patch that cleans up several parts >> of my series. The most impressive is his mechanism for testing the >> platform-specific Git logic in a way that is (mostly) platform-agnostic. >> >> * Windows doesn't have the 'id' command, so we cannot run the macOS >> platform test on Windows. > > This is easy to resolve. Drop in the following patch and then replace > the `$(id -u)` invocation in the test with `$(test-tool getuid)`. > This way, the test should work on any platform since both > launchctl_get_uid() and `test-tool` will retrieve identical values for > UID. I was giving your 'test-tool getuid' idea a try, and found that _also_ the $HOME environment variable differs from the format we expect in these subcommands: $HOME: C:\... argument in subcommand: /c/... So, there is another reason why these tests don't work on Windows. I'm of the opinion that maybe it's not worth _that_ level of cross-platform testing. Unless I'm missing something simple about a $HOME alternative here, this seems to be more work than the resulting value. Personally, I'm happy with the benefit you've already provided in allowing Linux to test all platforms. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-24 2:20 ` Derrick Stolee @ 2020-11-24 2:59 ` Eric Sunshine 0 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2020-11-24 2:59 UTC (permalink / raw) To: Derrick Stolee; +Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee On Mon, Nov 23, 2020 at 9:20 PM Derrick Stolee <stolee@gmail.com> wrote: > I was giving your 'test-tool getuid' idea a try, and found that _also_ > the $HOME environment variable differs from the format we expect in these > subcommands: > > $HOME: C:\... > argument in subcommand: /c/... Where does this problem crop up exactly? Is the test doing a literal comparison against the value in $HOME? > So, there is another reason why these tests don't work on Windows. I'm > of the opinion that maybe it's not worth _that_ level of cross-platform > testing. > > Unless I'm missing something simple about a $HOME alternative here, this > seems to be more work than the resulting value. Personally, I'm happy > with the benefit you've already provided in allowing Linux to test all > platforms. Indeed, it's not worth investing a lot of additional time into it. And it's certainly not a good reason to hold up the series. Moreover, this is the sort of thing which can be refined/handled later if someone wants to take a shot at it. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget ` (4 preceding siblings ...) 2020-11-17 23:36 ` [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine @ 2020-11-17 23:54 ` Eric Sunshine 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget 6 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2020-11-17 23:54 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget; +Cc: Git List, Derrick Stolee, Derrick Stolee On Tue, Nov 17, 2020 at 4:13 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > * I noticed far too late that while my example XML files had been edited > with UTF-8 encoding, Git is actually writing them as US-ASCII. Somehow > xmllint and launchd are not complaining, but schtasks does complain. > Unfortunately, I cannot find a way to catch this problem other than to > install my tip version on all three platforms and go through the entire > git maintenance start process, and double-check that the processes are > running on the hour. I'm having trouble understanding what problem is being described here and whether or not it has been solved by v4. I might guess that you are saying that `schtasks` insists upon seeing a UTF-8 BOM at the start of the XML file since the XML file declares itself as UTF-8, but that Git is (quite naturally) writing out the file without the UTF-8 BOM. > - xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" > + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" If the above speculation is correct, and if `schtasks` is happy with the plain text file (lacking UTF-8 BOM) declaring itself as US-ASCII, then this seems a reasonable solution. And it's easy to test that this doesn't get broken. After validating the file with `xmllint`, also grep it for US-ASCII, perhaps like this: test_xmllint "schedule-$frequency.xml" && grep "encoding=.US-ASCII." "schedule-$frequency.xml" ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v5 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget ` (5 preceding siblings ...) 2020-11-17 23:54 ` Eric Sunshine @ 2020-11-24 4:16 ` Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (4 more replies) 6 siblings, 5 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that. Updates in V5 ============= * Fixed docs from PATCH 2 to match those in v3. * Despite my best efforts, I was unable to make the macOS tests work on Windows, so they are still marked with "!MINGW". I updated the commit message to describe my problems there. * The Windows platform now uses xmks_tempfile() to create the XML files for 'schtasks'. This led to some test fallout since the pathnames are no longer predictable. Thanks, -Stolee Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks Documentation/git-maintenance.txt | 116 +++++++++ builtin/gc.c | 416 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 110 +++++++- t/test-lib.sh | 7 +- 4 files changed, 615 insertions(+), 34 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v5 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v5 Pull-Request: https://github.com/gitgitgadget/git/pull/776 Range-diff vs v4: 1: 4807342b00 = 1: 4807342b00 maintenance: extract platform-specific scheduling 2: 99170df462 ! 2: 7cc70a8fe7 maintenance: include 'cron' details in docs @@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems -+is `cron`. This tool executes commands based on a given schedule. The ++is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + @@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + -+The `<path>` string is loaded to specifically use the location for the -+`git` executable used in the `git maintenance start` command. This allows -+for multiple versions to be compatible. However, if the same user runs -+`git maintenance start` with multiple Git executables, then only the -+latest executable will be used. ++The `crontab` entry specifies the full path of the `git` executable to ++ensure that the executed `git` command is the same one with which ++`git maintenance start` was issued independent of `PATH`. If the same user ++runs `git maintenance start` with multiple Git executables, then only the ++latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically -+loaded from the user-specific global config located at `~/.gitconfig`. -+The `git maintenance` process then determines which maintenance tasks -+are configured to run on each repository with each `<frequency>` using -+the `maintenance.<task>.schedule` config options. These values are loaded -+from the global or repository config values. ++loaded from the user-specific global config. The `git maintenance` process ++then determines which maintenance tasks are configured to run on each ++repository with each `<frequency>` using the `maintenance.<task>.schedule` ++config options. These values are loaded from the global or repository ++config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read -+https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] -+for advanced scheduling techniques. Please do use the full path and -+`--exec-path` techniques from the default schedule to ensure you are -+executing the correct binaries in your schedule. ++the crontab(5) documentation for advanced scheduling techniques. Please ++do use the full path and `--exec-path` techniques from the default ++schedule to ensure you are executing the correct binaries in your ++schedule. + GIT 3: ed0a0011fb ! 3: cd015a5cbd maintenance: use launchctl on macOS @@ Commit message subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on - POSIX systems but not Windows. The test therefore has a prerequisite - that we are not on Windows. The cross-platform logic still allows us to - test the macOS logic on a Linux machine. + POSIX systems but not Windows. Further, the need for fully-qualitifed + path names including $HOME behaves differently in the Git internals and + the external test suite. The $HOME variable starts with "C:\..." instead + of the "/c/..." that is provided by Git in these subcommands. The test + therefore has a prerequisite that we are not on Windows. The cross- + platform logic still allows us to test the macOS logic on a Linux + machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the @@ Commit message Signed-off-by: Derrick Stolee <dstolee@microsoft.com> ## Documentation/git-maintenance.txt ## -@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do use the full path and - executing the correct binaries in your schedule. +@@ Documentation/git-maintenance.txt: schedule to ensure you are executing the correct binaries in your + schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS 4: b8d86fb983 ! 4: ac9a28bea3 maintenance: use Windows scheduled tasks @@ Commit message Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific - logic on a macOS platform. Thus, add a check that the XML file written + logic on other platforms. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. + Since we use a temporary file for the XML files sent to 'schtasks', we + must copy the file to a predictable filename. Use the number of lines in + the 'args' file to provide a filename for xmllint. Instead of an exact + match on the 'args' file, we 'grep' for the arguments other than the + filename. + There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; -+ char *xmlpath; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + -+ xmlpath = xstrfmt("%s/schedule-%s.xml", -+ the_repository->objects->odb->path, -+ frequency); -+ tfile = create_tempfile(xmlpath); ++ tfile = xmks_tempfile("schedule_XXXXXX"); + if (!tfile || !fdopen_tempfile(tfile, "w")) -+ die(_("failed to create '%s'"), xmlpath); ++ die(_("failed to create temp xml file")); + + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, + "</Task>\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); -+ strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); ++ strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", tfile->filename.buf, NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, + result = finish_command(&child); + + delete_tempfile(&tfile); -+ free(xmlpath); + free(name); + return result; +} @@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS mainten + *) shift ;; + esac + done -+ test -z "$xmlfile" || cp "$xmlfile" . ++ lines=$(wc -l args | awk "{print \$1;}") ++ test -z "$xmlfile" || cp "$xmlfile" "schedule-$lines.xml" + EOF + + rm -f args && @@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS mainten + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ -+ hourly hourly daily daily weekly weekly >expect && -+ test_cmp expect args && -+ + for frequency in hourly daily weekly + do -+ test_xmllint "schedule-$frequency.xml" ++ grep "/create /tn Git Maintenance ($frequency) /f /xml" args \ ++ || return 1 ++ done && ++ ++ for i in 1 2 3 ++ do ++ test_xmllint "schedule-$i.xml" && ++ grep "encoding=.US-ASCII." "schedule-$i.xml" || return 1 + done && + + rm -f args && -- gitgitgadget ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v5 1/4] maintenance: extract platform-specific scheduling 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 ` Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget ` (3 subsequent siblings) 4 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as "<scheduler>:<command>" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the <scheduler> in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 70 ++++++++++++++++++++++++++---------------- t/t7900-maintenance.sh | 8 ++--- t/test-lib.sh | 7 +++-- 3 files changed, 51 insertions(+), 34 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a1..18ae7f7138 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,35 +1494,23 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1531,17 +1519,15 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { @@ -1586,14 +1572,44 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; + else + fclose(cron_list); + return result; +} + +static const char platform_scheduler[] = "crontab"; + +static int update_background_schedule(int enable) +{ + int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; } - fclose(cron_list); -cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); + rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..eeb939168d 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -368,7 +368,7 @@ test_expect_success 'register and unregister' ' ' test_expect_success 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -379,19 +379,19 @@ test_expect_success 'start from empty cron table' ' ' test_expect_success 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..ddbeee1f5e 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v5 2/4] maintenance: include 'cron' details in docs 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 ` Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget ` (2 subsequent siblings) 4 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc..1aa1112418 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `crontab` entry specifies the full path of the `git` executable to +ensure that the executed `git` command is the same one with which +`git maintenance start` was issued independent of `PATH`. If the same user +runs `git maintenance start` with multiple Git executables, then only the +latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config. The `git maintenance` process +then determines which maintenance tasks are configured to run on each +repository with each `<frequency>` using the `maintenance.<task>.schedule` +config options. These values are loaded from the global or repository +config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +the crontab(5) documentation for advanced scheduling techniques. Please +do use the full path and `--exec-path` techniques from the default +schedule to ensure you are executing the correct binaries in your +schedule. + GIT --- -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v5 3/4] maintenance: use launchctl on macOS 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 ` Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 4 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on POSIX systems but not Windows. Further, the need for fully-qualitifed path names including $HOME behaves differently in the Git internals and the external test suite. The $HOME variable starts with "C:\..." instead of the "/c/..." that is provided by Git in these subcommands. The test therefore has a prerequisite that we are not on Windows. The cross- platform logic still allows us to test the macOS logic on a Linux machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 40 +++++++ builtin/gc.c | 188 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 58 +++++++++ 3 files changed, 285 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1aa1112418..5f8f63f098 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ schedule to ensure you are executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=<frequency>` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `<array>` element following +the `<key>StartCalendarInterval</key>` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 18ae7f7138..782769f243 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,186 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *launchctl_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *launchctl_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static char *launchctl_get_uid(void) +{ + return xstrfmt("gui/%d", getuid()); +} + +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = launchctl_get_uid(); + + strvec_split(&child.args, cmd); + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_push(&child.args, uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) +{ + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + int result = launchctl_boot_plist(0, filename, cmd); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int launchctl_remove_plists(const char *cmd) +{ + return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || + launchctl_remove_plist(SCHEDULE_DAILY, cmd) || + launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + fclose(plist); + + /* bootout might fail if not already running, so ignore */ + launchctl_boot_plist(0, filename, cmd); + if (launchctl_boot_plist(1, filename, cmd)) + die(_("failed to bootstrap service %s"), filename); + + free(filename); + free(name); + return 0; +} + +static int launchctl_add_plists(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return launchctl_add_plists(cmd); + else + return launchctl_remove_plists(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1579,7 +1759,11 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) return result; } +#if defined(__APPLE__) +static const char platform_scheduler[] = "launchctl"; +#else static const char platform_scheduler[] = "crontab"; +#endif static int update_background_schedule(int enable) { @@ -1603,7 +1787,9 @@ static int update_background_schedule(int enable) if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) return error(_("another process is scheduling background maintenance")); - if (!strcmp(scheduler, "crontab")) + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else die("unknown background scheduler: %s", scheduler); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index eeb939168d..6d37312901 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -7,6 +7,19 @@ test_description='git maintenance builtin' GIT_TEST_COMMIT_GRAPH=0 GIT_TEST_MULTI_PACK_INDEX=0 +test_lazy_prereq XMLLINT ' + xmllint --version +' + +test_xmllint () { + if test_have_prereq XMLLINT + then + xmllint --noout "$@" + else + true + fi +} + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance <subcommand>" err && @@ -395,6 +408,51 @@ test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' +test_expect_success !MINGW 'start and stop macOS maintenance' ' + uid=$(id -u) && + + write_script print-args <<-\EOF && + echo $* >>args + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$uid $PLIST" >>expect && + echo "bootstrap gui/$uid $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/$uid $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v5 4/4] maintenance: use Windows scheduled tasks 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2020-11-24 4:16 ` [PATCH v5 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 ` Derrick Stolee via GitGitGadget 2020-11-27 9:08 ` Eric Sunshine 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 4 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-11-24 4:16 UTC (permalink / raw) To: git Cc: jrnieder, jonathantanmy, sluongng, Đoàn Trần Công Danh, Martin Ågren, Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on other platforms. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. Since we use a temporary file for the XML files sent to 'schtasks', we must copy the file to a predictable filename. Use the number of lines in the 'args' file to provide a filename for xmllint. Instead of an exact match on the 'args' file, we 'grep' for the arguments other than the filename. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 160 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 44 ++++++++ 3 files changed, 226 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 5f8f63f098..6970f2b898 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance (<frequency>)`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 782769f243..43224e0dec 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1671,6 +1671,162 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } +static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int schtasks_remove_tasks(const char *cmd) +{ + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + tfile = xmks_tempfile("schedule_XXXXXX"); + if (!tfile || !fdopen_tempfile(tfile, "w")) + die(_("failed to create temp xml file")); + + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml = "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", tfile->filename.buf, NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + delete_tempfile(&tfile); + free(name); + return result; +} + +static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return schtasks_schedule_tasks(cmd); + else + return schtasks_remove_tasks(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1761,6 +1917,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) #if defined(__APPLE__) static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; #else static const char platform_scheduler[] = "crontab"; #endif @@ -1789,6 +1947,8 @@ static int update_background_schedule(int enable) if (!strcmp(scheduler, "launchctl")) result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 6d37312901..0246e4ce30 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -453,6 +453,50 @@ test_expect_success !MINGW 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + lines=$(wc -l args | awk "{print \$1;}") + test -z "$xmlfile" || cp "$xmlfile" "schedule-$lines.xml" + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + for frequency in hourly daily weekly + do + grep "/create /tn Git Maintenance ($frequency) /f /xml" args \ + || return 1 + done && + + for i in 1 2 3 + do + test_xmllint "schedule-$i.xml" && + grep "encoding=.US-ASCII." "schedule-$i.xml" || return 1 + done && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v5 4/4] maintenance: use Windows scheduled tasks 2020-11-24 4:16 ` [PATCH v5 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-11-27 9:08 ` Eric Sunshine 0 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2020-11-27 9:08 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Jonathan Nieder, Jonathan Tan, Son Luong Ngoc, Đoàn Trần Công Danh, Martin Ågren, Derrick Stolee, Derrick Stolee, Derrick Stolee On Mon, Nov 23, 2020 at 11:16 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > Git's background maintenance uses cron by default, but this is not > available on Windows. Instead, integrate with Task Scheduler. > [...] > Signed-off-by: Derrick Stolee <dstolee@microsoft.com> > --- > diff --git a/builtin/gc.c b/builtin/gc.c > @@ -1671,6 +1671,162 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm > +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) > +{ > + const char *frequency = get_frequency(schedule); > + > + tfile = xmks_tempfile("schedule_XXXXXX"); > + if (!tfile || !fdopen_tempfile(tfile, "w")) > + die(_("failed to create temp xml file")); Several comments: The "x" prefix on xmks_tempfile() means that it will die() if it can't open the tempfile, so the `!tfile` condition is pointless, thus it could be written: if (!fdopen_tempfile(tfile, "w")) The mks_tempfile_t*() functions with a trailing "t" will place the temporary file in TMPDIR, whereas xmks_tempfile() used here places it in the worktree directory, which is not as desirable. Ideally, this would be using xmks_tempfile_t(), however, that function doesn't exist yet in tempfile.h, so the best you can do (without the extra work of also adding the missing function) is to use mks_tempfile_t(). That doesn't die(), so `!tfile` would still be needed in the conditional. In earlier versions, you incorporated `frequency` into the temporary filename which was nice because it made the test easier to understand. It's not hard to do so here, as well, nor to extract a useful filename in the test (as I'll show below). For instance: struct strbuf tpath = STRBUF_INIT; strbuf_addf(&tpath, "schedule-%s-XXXXXX", frequency); tfile = mks_tempfile_t(tpath.buf); strbuf_release(&tpath); if (!tfile || !fdopen(tempfile(tfile, "w")) die(...); > + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", tfile->filename.buf, NULL); Alternately, use the getter: strvec_pushl(..., get_tempfile_path(&tfile), ...); > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh > @@ -453,6 +453,50 @@ test_expect_success !MINGW 'start and stop macOS maintenance' ' > +test_expect_success 'start and stop Windows maintenance' ' > + write_script print-args <<-\EOF && > + echo $* >>args > + while test $# -gt 0 > + do > + case "$1" in > + /xml) shift; xmlfile=$1; break ;; > + *) shift ;; > + esac > + done > + lines=$(wc -l args | awk "{print \$1;}") You're using `awk` to pluck out the line count and ignore the filename, but the same can be accomplished by feeding the file as stdin to `wc` rather than naming it as an argument, thus this is equivalent: lines=$(wc -l <args) However, this idea of constructing stable names for the files by assigning them numerically incrementing values is unnecessary and makes the test harder to understand. > + test -z "$xmlfile" || cp "$xmlfile" "schedule-$lines.xml" If you take the suggestion earlier in this review of naming the file "schedule-${frequency}-XXXXXX.xml", then you can strip it down to just "schedule-${frequency}.xml" using the expression `${xmlfile%-*}.xml`. There is no need for `$lines`. Thus, copying the file would become: test -z "$xmlfile" || cp "$xmlfile" "${xmlfile%-*}.xml" > + for i in 1 2 3 > + do Which means that you can use the more easily understood `hourly daily weekly` enumeration here rather than `1 2 3`. Having said all this, I'm not sure it's worth a re-roll. These sort of tweaks can be done atop the current series if someone wants to tackle it. ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget ` (3 preceding siblings ...) 2020-11-24 4:16 ` [PATCH v5 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-12-09 19:28 ` Derrick Stolee via GitGitGadget 2020-12-09 19:28 ` [PATCH v6 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (5 more replies) 4 siblings, 6 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-12-09 19:28 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that. Update in V6 ============ * The Windows platform uses the tempfile API a bit better, including using the frequency in the filename to make the test simpler. Thanks, -Stolee cc: jrnieder@gmail.com cc: jonathantanmy@google.com cc: sluongng@gmail.com cc: Đoàn Trần Công Danh congdanhqx@gmail.com cc: Martin Ågren martin.agren@gmail.com cc: Eric Sunshine sunshine@sunshineco.com cc: Derrick Stolee stolee@gmail.com Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks Documentation/git-maintenance.txt | 116 ++++++++ builtin/gc.c | 421 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 105 +++++++- t/test-lib.sh | 7 +- 4 files changed, 615 insertions(+), 34 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v6 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v6 Pull-Request: https://github.com/gitgitgadget/git/pull/776 Range-diff vs v5: 1: 4807342b001 = 1: 4807342b001 maintenance: extract platform-specific scheduling 2: 7cc70a8fe7b = 2: 7cc70a8fe7b maintenance: include 'cron' details in docs 3: cd015a5cbd7 = 3: cd015a5cbd7 maintenance: use launchctl on macOS 4: ac9a28bea39 ! 4: 6ad4a6b98c6 maintenance: use Windows scheduled tasks @@ Commit message by Git is valid when xmllint exists on the system. Since we use a temporary file for the XML files sent to 'schtasks', we - must copy the file to a predictable filename. Use the number of lines in - the 'args' file to provide a filename for xmllint. Instead of an exact - match on the 'args' file, we 'grep' for the arguments other than the - filename. + prefix the random characters with the frequency so it is easier to + examine the proper file during tests. Instead of an exact match on the + 'args' file, we 'grep' for the arguments other than the filename. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); ++ struct strbuf tfilename = STRBUF_INIT; + -+ tfile = xmks_tempfile("schedule_XXXXXX"); -+ if (!tfile || !fdopen_tempfile(tfile, "w")) ++ strbuf_addf(&tfilename, "schedule_%s_XXXXXX", frequency); ++ tfile = xmks_tempfile(tfilename.buf); ++ strbuf_release(&tfilename); ++ ++ if (!fdopen_tempfile(tfile, "w")) + die(_("failed to create temp xml file")); + + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, + "</Task>\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); -+ strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", tfile->filename.buf, NULL); ++ strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", ++ get_tempfile_path(tfile), NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; @@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS mainten + *) shift ;; + esac + done -+ lines=$(wc -l args | awk "{print \$1;}") -+ test -z "$xmlfile" || cp "$xmlfile" "schedule-$lines.xml" ++ test -z "$xmlfile" || cp "$xmlfile" "$xmlfile.xml" + EOF + + rm -f args && -+ GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && ++ GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" GIT_TRACE2_PERF=1 git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + for frequency in hourly daily weekly + do -+ grep "/create /tn Git Maintenance ($frequency) /f /xml" args \ -+ || return 1 -+ done && -+ -+ for i in 1 2 3 -+ do -+ test_xmllint "schedule-$i.xml" && -+ grep "encoding=.US-ASCII." "schedule-$i.xml" || return 1 ++ grep "/create /tn Git Maintenance ($frequency) /f /xml" args && ++ file=$(ls schedule_$frequency*.xml) && ++ test_xmllint "$file" && ++ grep "encoding=.US-ASCII." "$file" || return 1 + done && + + rm -f args && -- gitgitgadget ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v6 1/4] maintenance: extract platform-specific scheduling 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget @ 2020-12-09 19:28 ` Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget ` (4 subsequent siblings) 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-12-09 19:28 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as "<scheduler>:<command>" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the <scheduler> in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 70 ++++++++++++++++++++++++++---------------- t/t7900-maintenance.sh | 8 ++--- t/test-lib.sh | 7 +++-- 3 files changed, 51 insertions(+), 34 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a12..18ae7f7138a 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,35 +1494,23 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1531,17 +1519,15 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { @@ -1586,14 +1572,44 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; + else + fclose(cron_list); + return result; +} + +static const char platform_scheduler[] = "crontab"; + +static int update_background_schedule(int enable) +{ + int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; } - fclose(cron_list); -cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); + rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1a..eeb939168da 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -368,7 +368,7 @@ test_expect_success 'register and unregister' ' ' test_expect_success 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -379,19 +379,19 @@ test_expect_success 'start from empty cron table' ' ' test_expect_success 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed766..ddbeee1f5eb 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v6 2/4] maintenance: include 'cron' details in docs 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-12-09 19:28 ` [PATCH v6 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2020-12-09 19:29 ` Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget ` (3 subsequent siblings) 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-12-09 19:29 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc2..1aa11124186 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `crontab` entry specifies the full path of the `git` executable to +ensure that the executed `git` command is the same one with which +`git maintenance start` was issued independent of `PATH`. If the same user +runs `git maintenance start` with multiple Git executables, then only the +latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config. The `git maintenance` process +then determines which maintenance tasks are configured to run on each +repository with each `<frequency>` using the `maintenance.<task>.schedule` +config options. These values are loaded from the global or repository +config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +the crontab(5) documentation for advanced scheduling techniques. Please +do use the full path and `--exec-path` techniques from the default +schedule to ensure you are executing the correct binaries in your +schedule. + GIT --- -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v6 3/4] maintenance: use launchctl on macOS 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-12-09 19:28 ` [PATCH v6 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2020-12-09 19:29 ` Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget ` (2 subsequent siblings) 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-12-09 19:29 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on POSIX systems but not Windows. Further, the need for fully-qualitifed path names including $HOME behaves differently in the Git internals and the external test suite. The $HOME variable starts with "C:\..." instead of the "/c/..." that is provided by Git in these subcommands. The test therefore has a prerequisite that we are not on Windows. The cross- platform logic still allows us to test the macOS logic on a Linux machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 40 +++++++ builtin/gc.c | 188 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 58 +++++++++ 3 files changed, 285 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1aa11124186..5f8f63f0988 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ schedule to ensure you are executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=<frequency>` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `<array>` element following +the `<key>StartCalendarInterval</key>` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 18ae7f7138a..782769f2438 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,186 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *launchctl_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *launchctl_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static char *launchctl_get_uid(void) +{ + return xstrfmt("gui/%d", getuid()); +} + +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = launchctl_get_uid(); + + strvec_split(&child.args, cmd); + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_push(&child.args, uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) +{ + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + int result = launchctl_boot_plist(0, filename, cmd); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int launchctl_remove_plists(const char *cmd) +{ + return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || + launchctl_remove_plist(SCHEDULE_DAILY, cmd) || + launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + fclose(plist); + + /* bootout might fail if not already running, so ignore */ + launchctl_boot_plist(0, filename, cmd); + if (launchctl_boot_plist(1, filename, cmd)) + die(_("failed to bootstrap service %s"), filename); + + free(filename); + free(name); + return 0; +} + +static int launchctl_add_plists(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return launchctl_add_plists(cmd); + else + return launchctl_remove_plists(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1579,7 +1759,11 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) return result; } +#if defined(__APPLE__) +static const char platform_scheduler[] = "launchctl"; +#else static const char platform_scheduler[] = "crontab"; +#endif static int update_background_schedule(int enable) { @@ -1603,7 +1787,9 @@ static int update_background_schedule(int enable) if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) return error(_("another process is scheduling background maintenance")); - if (!strcmp(scheduler, "crontab")) + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else die("unknown background scheduler: %s", scheduler); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index eeb939168da..6d373129016 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -7,6 +7,19 @@ test_description='git maintenance builtin' GIT_TEST_COMMIT_GRAPH=0 GIT_TEST_MULTI_PACK_INDEX=0 +test_lazy_prereq XMLLINT ' + xmllint --version +' + +test_xmllint () { + if test_have_prereq XMLLINT + then + xmllint --noout "$@" + else + true + fi +} + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance <subcommand>" err && @@ -395,6 +408,51 @@ test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' +test_expect_success !MINGW 'start and stop macOS maintenance' ' + uid=$(id -u) && + + write_script print-args <<-\EOF && + echo $* >>args + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$uid $PLIST" >>expect && + echo "bootstrap gui/$uid $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/$uid $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v6 4/4] maintenance: use Windows scheduled tasks 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2020-12-09 19:29 ` [PATCH v6 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2020-12-09 19:29 ` Derrick Stolee via GitGitGadget 2020-12-10 0:32 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Junio C Hamano 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget 5 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2020-12-09 19:29 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on other platforms. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. Since we use a temporary file for the XML files sent to 'schtasks', we prefix the random characters with the frequency so it is easier to examine the proper file during tests. Instead of an exact match on the 'args' file, we 'grep' for the arguments other than the filename. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 165 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 39 +++++++ 3 files changed, 226 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 5f8f63f0988..6970f2b8983 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance (<frequency>)`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 782769f2438..7c989904671 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1671,6 +1671,167 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } +static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int schtasks_remove_tasks(const char *cmd) +{ + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + struct strbuf tfilename = STRBUF_INIT; + + strbuf_addf(&tfilename, "schedule_%s_XXXXXX", frequency); + tfile = xmks_tempfile(tfilename.buf); + strbuf_release(&tfilename); + + if (!fdopen_tempfile(tfile, "w")) + die(_("failed to create temp xml file")); + + xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml = "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", + get_tempfile_path(tfile), NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + delete_tempfile(&tfile); + free(name); + return result; +} + +static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return schtasks_schedule_tasks(cmd); + else + return schtasks_remove_tasks(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1761,6 +1922,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) #if defined(__APPLE__) static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; #else static const char platform_scheduler[] = "crontab"; #endif @@ -1789,6 +1952,8 @@ static int update_background_schedule(int enable) if (!strcmp(scheduler, "launchctl")) result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 6d373129016..f080c29a61d 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -453,6 +453,45 @@ test_expect_success !MINGW 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + test -z "$xmlfile" || cp "$xmlfile" "$xmlfile.xml" + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" GIT_TRACE2_PERF=1 git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + for frequency in hourly daily weekly + do + grep "/create /tn Git Maintenance ($frequency) /f /xml" args && + file=$(ls schedule_$frequency*.xml) && + test_xmllint "$file" && + grep "encoding=.US-ASCII." "$file" || return 1 + done && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (3 preceding siblings ...) 2020-12-09 19:29 ` [PATCH v6 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget @ 2020-12-10 0:32 ` Junio C Hamano 2020-12-10 0:49 ` Eric Sunshine 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget 5 siblings, 1 reply; 83+ messages in thread From: Junio C Hamano @ 2020-12-10 0:32 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget; +Cc: git, Eric Sunshine, Derrick Stolee "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: > This is based on ds/maintenance-part-3. > > After sitting with the background maintenance as it has been cooking, I > wanted to come back around and implement the background maintenance for > Windows. However, I noticed that there were some things bothering me with > background maintenance on my macOS machine. These are detailed in PATCH 3, > but the tl;dr is that 'cron' is not recommended by Apple and instead > 'launchd' satisfies our needs. > > This series implements the background scheduling so git maintenance > (start|stop) works on those platforms. I've been operating with these > schedules for a while now without the problems described in the patches. > > There is a particularly annoying case about console windows popping up on > Windows, but PATCH 4 describes a plan to get around that. > > > Update in V6 > ============ > > * The Windows platform uses the tempfile API a bit better, including using > the frequency in the filename to make the test simpler. Are two fix-up patches from Eric that have been queued near the top of ds/maintenance-part-4 still relevant? At least, the "when invoked individually" patch that added an "-f" option to two invocations of "rm" is still applicable, I would think (I didn't look at the other one). commit e3801c41e4d4cb1dd899942e04ab78310e781d07 Author: Eric Sunshine <sunshine@sunshineco.com> t7900: make macOS-specific test work on Windows Notes (amlog): Message-Id: <20201130044224.12298-3-sunshine@sunshineco.com> commit 1e5ddd79e2da18ee19b665a045d4187c5dc6234e Author: Eric Sunshine <sunshine@sunshineco.com> t7900: fix test failures when invoked individually via --run Notes (amlog): Message-Id: <20201130044224.12298-2-sunshine@sunshineco.com> ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance 2020-12-10 0:32 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Junio C Hamano @ 2020-12-10 0:49 ` Eric Sunshine 2020-12-10 1:04 ` Junio C Hamano 0 siblings, 1 reply; 83+ messages in thread From: Eric Sunshine @ 2020-12-10 0:49 UTC (permalink / raw) To: Junio C Hamano; +Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee On Wed, Dec 9, 2020 at 7:33 PM Junio C Hamano <gitster@pobox.com> wrote: > "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: > > Update in V6 > > ============ > > > > * The Windows platform uses the tempfile API a bit better, including using > > the frequency in the filename to make the test simpler. > > Are two fix-up patches from Eric that have been queued near the top > of ds/maintenance-part-4 still relevant? Both of the patches from Sunshine are still relevant atop Stolee's latest (v6), and they should apply cleanly (I would think) since v6 didn't change anything related to those patches. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance 2020-12-10 0:49 ` Eric Sunshine @ 2020-12-10 1:04 ` Junio C Hamano 2021-01-05 12:17 ` Derrick Stolee 0 siblings, 1 reply; 83+ messages in thread From: Junio C Hamano @ 2020-12-10 1:04 UTC (permalink / raw) To: Eric Sunshine; +Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee Eric Sunshine <sunshine@sunshineco.com> writes: > On Wed, Dec 9, 2020 at 7:33 PM Junio C Hamano <gitster@pobox.com> wrote: >> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: >> > Update in V6 >> > ============ >> > >> > * The Windows platform uses the tempfile API a bit better, including using >> > the frequency in the filename to make the test simpler. >> >> Are two fix-up patches from Eric that have been queued near the top >> of ds/maintenance-part-4 still relevant? > > Both of the patches from Sunshine are still relevant atop Stolee's > latest (v6), and they should apply cleanly (I would think) since v6 > didn't change anything related to those patches. Yup, I tried rebasing these two and they applied cleanly, so I'll include them in today's pushout (which I haven't finished yet). I probably would not notice if the updated 4-patch series already solved the issue in another way without causing the textual conflict with your two fix-up patches, though ;-) Thanks. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance 2020-12-10 1:04 ` Junio C Hamano @ 2021-01-05 12:17 ` Derrick Stolee 0 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee @ 2021-01-05 12:17 UTC (permalink / raw) To: Junio C Hamano, Eric Sunshine Cc: Derrick Stolee via GitGitGadget, Git List, Derrick Stolee On 12/9/2020 8:04 PM, Junio C Hamano wrote: > Eric Sunshine <sunshine@sunshineco.com> writes: > >> On Wed, Dec 9, 2020 at 7:33 PM Junio C Hamano <gitster@pobox.com> wrote: >>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes: >>>> Update in V6 >>>> ============ >>>> >>>> * The Windows platform uses the tempfile API a bit better, including using >>>> the frequency in the filename to make the test simpler. >>> >>> Are two fix-up patches from Eric that have been queued near the top >>> of ds/maintenance-part-4 still relevant? >> >> Both of the patches from Sunshine are still relevant atop Stolee's >> latest (v6), and they should apply cleanly (I would think) since v6 >> didn't change anything related to those patches. > > Yup, I tried rebasing these two and they applied cleanly, so I'll > include them in today's pushout (which I haven't finished yet). > > I probably would not notice if the updated 4-patch series already > solved the issue in another way without causing the textual conflict > with your two fix-up patches, though ;-) I noticed a subtle issue with the v6 series, so I _will_ reroll the series squashing in Eric's patches. He will remain a co-author and I'll add the Helped-by: Ævar along with the details for that patch. Thanks, -Stolee ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v7 0/4] Maintenance IV: Platform-specific background maintenance 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget ` (4 preceding siblings ...) 2020-12-10 0:32 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Junio C Hamano @ 2021-01-05 13:08 ` Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget ` (3 more replies) 5 siblings, 4 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee This is based on ds/maintenance-part-3. After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs. This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches. There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that. Update in V7 ============ * I had included an "encoding" string in the XML file for schtasks based on an example using UTF-8. The cross-platform tests then complained (in xmllint) because they wrote in ASCII instead. However, actually testing the situation on Windows (see [1]) against the real schtasks finds that it doesn't like that encoding string. I removed it entirely, and everything seems happier. * I squashed Eric's two commits making the tests better. He remains a co-author and I kept his Helped-by. I had to rearrange the commit message a bit to point out the care he took for the cross-platform tests without referring to the test doing the wrong thing. [1] https://github.com/microsoft/git/pull/304 Thanks, -Stolee cc: jrnieder@gmail.com cc: jonathantanmy@google.com cc: sluongng@gmail.com cc: Đoàn Trần Công Danh congdanhqx@gmail.com cc: Martin Ågren martin.agren@gmail.com cc: Eric Sunshine sunshine@sunshineco.com cc: Derrick Stolee stolee@gmail.com Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks Documentation/git-maintenance.txt | 116 ++++++++ builtin/gc.c | 422 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 104 +++++++- t/test-lib.sh | 7 +- 4 files changed, 615 insertions(+), 34 deletions(-) base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v7 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v7 Pull-Request: https://github.com/gitgitgadget/git/pull/776 Range-diff vs v6: 1: 4807342b001 = 1: 4807342b001 maintenance: extract platform-specific scheduling 2: 7cc70a8fe7b = 2: 7cc70a8fe7b maintenance: include 'cron' details in docs 3: cd015a5cbd7 ! 3: 3576c7aa54e maintenance: use launchctl on macOS @@ Commit message the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. + We strive to make these tests work on all platforms, but Windows caused + some headaches. In particular, the value of getuid() called by the C + code is not guaranteed to be the same as `$(id -u)` invoked by a test. + This is because `git.exe` is a native Windows program, whereas the + utility programs run by the test script mostly utilize the MSYS2 runtime, + which emulates a POSIX-like environment. Since the purpose of the test + is to check that the input to the hook is well-formed, the actual user + ID is immaterial, thus we can work around the problem by making the the + test UID-agnostic. Another subtle issue is the $HOME environment + variable being a Windows-style path instead of a Unix-style path. We can + be more flexible here instead of expecting exact path matches. + + Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> @@ t/t7900-maintenance.sh: test_expect_success 'start preserves existing schedule' grep "Important information!" cron.txt ' -+test_expect_success !MINGW 'start and stop macOS maintenance' ' -+ uid=$(id -u) && ++test_expect_success 'start and stop macOS maintenance' ' ++ # ensure $HOME can be compared against hook arguments on all platforms ++ pfx=$(cd "$HOME" && pwd) && + + write_script print-args <<-\EOF && -+ echo $* >>args ++ echo $* | sed "s:gui/[0-9][0-9]*:gui/[UID]:" >>args + EOF + + rm -f args && @@ t/t7900-maintenance.sh: test_expect_success 'start preserves existing schedule' + EOF + test_cmp expect actual && + -+ rm expect && ++ rm -f expect && + for frequency in hourly daily weekly + do -+ PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && ++ PLIST="$pfx/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && -+ echo "bootout gui/$uid $PLIST" >>expect && -+ echo "bootstrap gui/$uid $PLIST" >>expect || return 1 ++ echo "bootout gui/[UID] $PLIST" >>expect && ++ echo "bootstrap gui/[UID] $PLIST" >>expect || return 1 + done && + test_cmp expect args && + @@ t/t7900-maintenance.sh: test_expect_success 'start preserves existing schedule' + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ printf "bootout gui/$uid $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ ++ printf "bootout gui/[UID] $pfx/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && 4: 6ad4a6b98c6 ! 4: 68f5013dee3 maintenance: use Windows scheduled tasks @@ Documentation/git-maintenance.txt: To create more advanced customizations to you Part of the linkgit:git[1] suite ## builtin/gc.c ## +@@ builtin/gc.c: static int launchctl_schedule_plist(const char *exec_path, enum schedule_priorit + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + +- preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" ++ preamble = "<?xml version=\"1.0\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } @@ builtin/gc.c: static int launchctl_update_schedule(int run_maintenance, int fd, + char *name = schtasks_task_name(frequency); + struct strbuf tfilename = STRBUF_INIT; + -+ strbuf_addf(&tfilename, "schedule_%s_XXXXXX", frequency); ++ strbuf_addf(&tfilename, "%s/schedule_%s_XXXXXX", ++ get_git_common_dir(), frequency); + tfile = xmks_tempfile(tfilename.buf); + strbuf_release(&tfilename); + + if (!fdopen_tempfile(tfile, "w")) + die(_("failed to create temp xml file")); + -+ xml = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" ++ xml = "<?xml version=\"1.0\" ?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; @@ builtin/gc.c: static int update_background_schedule(int enable) else ## t/t7900-maintenance.sh ## -@@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS maintenance' ' +@@ t/t7900-maintenance.sh: test_expect_success 'start and stop macOS maintenance' ' test_line_count = 0 actual ' @@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS mainten + EOF + + rm -f args && -+ GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" GIT_TRACE2_PERF=1 git maintenance start && ++ GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && @@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS mainten + for frequency in hourly daily weekly + do + grep "/create /tn Git Maintenance ($frequency) /f /xml" args && -+ file=$(ls schedule_$frequency*.xml) && -+ test_xmllint "$file" && -+ grep "encoding=.US-ASCII." "$file" || return 1 ++ file=$(ls .git/schedule_${frequency}*.xml) && ++ test_xmllint "$file" || return 1 + done && + + rm -f args && @@ t/t7900-maintenance.sh: test_expect_success !MINGW 'start and stop macOS mainten + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + -+ rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args -- gitgitgadget ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v7 1/4] maintenance: extract platform-specific scheduling 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 ` Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget ` (2 subsequent siblings) 3 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as "<scheduler>:<command>" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the <scheduler> in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- builtin/gc.c | 70 ++++++++++++++++++++++++++---------------- t/t7900-maintenance.sh | 8 ++--- t/test-lib.sh | 7 +++-- 3 files changed, 51 insertions(+), 34 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a12..18ae7f7138a 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,35 +1494,23 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1531,17 +1519,15 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { @@ -1586,14 +1572,44 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; + else + fclose(cron_list); + return result; +} + +static const char platform_scheduler[] = "crontab"; + +static int update_background_schedule(int enable) +{ + int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; } - fclose(cron_list); -cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); + rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1a..eeb939168da 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -368,7 +368,7 @@ test_expect_success 'register and unregister' ' ' test_expect_success 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -379,19 +379,19 @@ test_expect_success 'start from empty cron table' ' ' test_expect_success 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed766..ddbeee1f5eb 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v7 2/4] maintenance: include 'cron' details in docs 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 ` Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 3 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc2..1aa11124186 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `crontab` entry specifies the full path of the `git` executable to +ensure that the executed `git` command is the same one with which +`git maintenance start` was issued independent of `PATH`. If the same user +runs `git maintenance start` with multiple Git executables, then only the +latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=<frequency>` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config. The `git maintenance` process +then determines which maintenance tasks are configured to run on each +repository with each `<frequency>` using the `maintenance.<task>.schedule` +config options. These values are loaded from the global or repository +config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +the crontab(5) documentation for advanced scheduling techniques. Please +do use the full path and `--exec-path` techniques from the default +schedule to ensure you are executing the correct binaries in your +schedule. + GIT --- -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH v7 3/4] maintenance: use launchctl on macOS 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 ` Derrick Stolee via GitGitGadget 2021-01-10 6:34 ` Eric Sunshine 2021-01-05 13:08 ` [PATCH v7 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 3 siblings, 1 reply; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of <string> tags. For example, here is my plist file for the weekly scheduled tasks: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"><dict> <key>Label</key><string>org.git-scm.git.weekly</string> <key>ProgramArguments</key> <array> <string>/usr/local/libexec/git-core/git</string> <string>--exec-path=/usr/local/libexec/git-core</string> <string>for-each-repo</string> <string>--config=maintenance.repo</string> <string>maintenance</string> <string>run</string> <string>--schedule=weekly</string> </array> <key>StartCalendarInterval</key> <array> <dict> <key>Day</key><integer>0</integer> <key>Hour</key><integer>0</integer> <key>Minute</key><integer>0</integer> </dict> </array> </dict> </plist> The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on POSIX systems but not Windows. Further, the need for fully-qualitifed path names including $HOME behaves differently in the Git internals and the external test suite. The $HOME variable starts with "C:\..." instead of the "/c/..." that is provided by Git in these subcommands. The test therefore has a prerequisite that we are not on Windows. The cross- platform logic still allows us to test the macOS logic on a Linux machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "<dict>" tag when it should have had a closing "</dict>" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. We strive to make these tests work on all platforms, but Windows caused some headaches. In particular, the value of getuid() called by the C code is not guaranteed to be the same as `$(id -u)` invoked by a test. This is because `git.exe` is a native Windows program, whereas the utility programs run by the test script mostly utilize the MSYS2 runtime, which emulates a POSIX-like environment. Since the purpose of the test is to check that the input to the hook is well-formed, the actual user ID is immaterial, thus we can work around the problem by making the the test UID-agnostic. Another subtle issue is the $HOME environment variable being a Windows-style path instead of a Unix-style path. We can be more flexible here instead of expecting exact path matches. Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 40 +++++++ builtin/gc.c | 188 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 59 ++++++++++ 3 files changed, 286 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1aa11124186..5f8f63f0988 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ schedule to ensure you are executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=<frequency>` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `<array>` element following +the `<key>StartCalendarInterval</key>` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 18ae7f7138a..782769f2438 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,186 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *launchctl_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *launchctl_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static char *launchctl_get_uid(void) +{ + return xstrfmt("gui/%d", getuid()); +} + +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = launchctl_get_uid(); + + strvec_split(&child.args, cmd); + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_push(&child.args, uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) +{ + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + int result = launchctl_boot_plist(0, filename, cmd); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int launchctl_remove_plists(const char *cmd) +{ + return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || + launchctl_remove_plist(SCHEDULE_DAILY, cmd) || + launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" + "<plist version=\"1.0\">" + "<dict>\n" + "<key>Label</key><string>%s</string>\n" + "<key>ProgramArguments</key>\n" + "<array>\n" + "<string>%s/git</string>\n" + "<string>--exec-path=%s</string>\n" + "<string>for-each-repo</string>\n" + "<string>--config=maintenance.repo</string>\n" + "<string>maintenance</string>\n" + "<string>run</string>\n" + "<string>--schedule=%s</string>\n" + "</array>\n" + "<key>StartCalendarInterval</key>\n" + "<array>\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "<dict>\n" + "<key>Hour</key><integer>%d</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "<dict>\n" + "<key>Day</key><integer>%d</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "<dict>\n" + "<key>Day</key><integer>0</integer>\n" + "<key>Hour</key><integer>0</integer>\n" + "<key>Minute</key><integer>0</integer>\n" + "</dict>\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "</array>\n</dict>\n</plist>\n"); + fclose(plist); + + /* bootout might fail if not already running, so ignore */ + launchctl_boot_plist(0, filename, cmd); + if (launchctl_boot_plist(1, filename, cmd)) + die(_("failed to bootstrap service %s"), filename); + + free(filename); + free(name); + return 0; +} + +static int launchctl_add_plists(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return launchctl_add_plists(cmd); + else + return launchctl_remove_plists(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1579,7 +1759,11 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) return result; } +#if defined(__APPLE__) +static const char platform_scheduler[] = "launchctl"; +#else static const char platform_scheduler[] = "crontab"; +#endif static int update_background_schedule(int enable) { @@ -1603,7 +1787,9 @@ static int update_background_schedule(int enable) if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) return error(_("another process is scheduling background maintenance")); - if (!strcmp(scheduler, "crontab")) + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else die("unknown background scheduler: %s", scheduler); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index eeb939168da..adf24dee72d 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -7,6 +7,19 @@ test_description='git maintenance builtin' GIT_TEST_COMMIT_GRAPH=0 GIT_TEST_MULTI_PACK_INDEX=0 +test_lazy_prereq XMLLINT ' + xmllint --version +' + +test_xmllint () { + if test_have_prereq XMLLINT + then + xmllint --noout "$@" + else + true + fi +} + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance <subcommand>" err && @@ -395,6 +408,52 @@ test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' +test_expect_success 'start and stop macOS maintenance' ' + # ensure $HOME can be compared against hook arguments on all platforms + pfx=$(cd "$HOME" && pwd) && + + write_script print-args <<-\EOF && + echo $* | sed "s:gui/[0-9][0-9]*:gui/[UID]:" >>args + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm -f expect && + for frequency in hourly daily weekly + do + PLIST="$pfx/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/[UID] $PLIST" >>expect && + echo "bootstrap gui/[UID] $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/[UID] $pfx/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH v7 3/4] maintenance: use launchctl on macOS 2021-01-05 13:08 ` [PATCH v7 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2021-01-10 6:34 ` Eric Sunshine 0 siblings, 0 replies; 83+ messages in thread From: Eric Sunshine @ 2021-01-10 6:34 UTC (permalink / raw) To: Derrick Stolee via GitGitGadget Cc: Git List, Derrick Stolee, Derrick Stolee, Derrick Stolee On Tue, Jan 5, 2021 at 8:08 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > [...] > The need for a user id requires us to run 'id -u' which works on > POSIX systems but not Windows. Further, the need for fully-qualitifed > path names including $HOME behaves differently in the Git internals and > the external test suite. The $HOME variable starts with "C:\..." instead > of the "/c/..." that is provided by Git in these subcommands. The test > therefore has a prerequisite that we are not on Windows. The cross- > platform logic still allows us to test the macOS logic on a Linux > machine. You forgot to remove the above paragraph... > We strive to make these tests work on all platforms, but Windows caused > some headaches. In particular, the value of getuid() called by the C > code is not guaranteed to be the same as `$(id -u)` invoked by a test. > This is because `git.exe` is a native Windows program, whereas the > utility programs run by the test script mostly utilize the MSYS2 runtime, > which emulates a POSIX-like environment. Since the purpose of the test > is to check that the input to the hook is well-formed, the actual user > ID is immaterial, thus we can work around the problem by making the the > test UID-agnostic. Another subtle issue is the $HOME environment > variable being a Windows-style path instead of a Unix-style path. We can > be more flexible here instead of expecting exact path matches. ...when you added this paragraph from my separate patch which you folded into this patch for v7. The two paragraphs are at odds with one another. ^ permalink raw reply [flat|nested] 83+ messages in thread
* [PATCH v7 4/4] maintenance: use Windows scheduled tasks 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget ` (2 preceding siblings ...) 2021-01-05 13:08 ` [PATCH v7 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 ` Derrick Stolee via GitGitGadget 3 siblings, 0 replies; 83+ messages in thread From: Derrick Stolee via GitGitGadget @ 2021-01-05 13:08 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Derrick Stolee, Derrick Stolee, Derrick Stolee From: Derrick Stolee <dstolee@microsoft.com> Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on other platforms. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. Since we use a temporary file for the XML files sent to 'schtasks', we prefix the random characters with the frequency so it is easier to examine the proper file during tests. Instead of an exact match on the 'args' file, we 'grep' for the arguments other than the filename. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the <Command> tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 168 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 37 +++++++ 3 files changed, 226 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 5f8f63f0988..6970f2b8983 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance (<frequency>)`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 782769f2438..fdc95d9e99f 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1589,7 +1589,7 @@ static int launchctl_schedule_plist(const char *exec_path, enum schedule_priorit die(_("failed to create directories for '%s'"), filename); plist = xfopen(filename, "w"); - preamble = "<?xml version=\"1.0\" encoding=\"US-ASCII\"?>\n" + preamble = "<?xml version=\"1.0\"?>\n" "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n" "<plist version=\"1.0\">" "<dict>\n" @@ -1671,6 +1671,168 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } +static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int schtasks_remove_tasks(const char *cmd) +{ + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + struct strbuf tfilename = STRBUF_INIT; + + strbuf_addf(&tfilename, "%s/schedule_%s_XXXXXX", + get_git_common_dir(), frequency); + tfile = xmks_tempfile(tfilename.buf); + strbuf_release(&tfilename); + + if (!fdopen_tempfile(tfile, "w")) + die(_("failed to create temp xml file")); + + xml = "<?xml version=\"1.0\" ?>\n" + "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n" + "<Triggers>\n" + "<CalendarTrigger>\n"; + fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T01:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByDay>\n" + "<DaysInterval>1</DaysInterval>\n" + "</ScheduleByDay>\n" + "<Repetition>\n" + "<Interval>PT1H</Interval>\n" + "<Duration>PT23H</Duration>\n" + "<StopAtDurationEnd>false</StopAtDurationEnd>\n" + "</Repetition>\n"); + break; + + case SCHEDULE_DAILY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Monday />\n" + "<Tuesday />\n" + "<Wednesday />\n" + "<Thursday />\n" + "<Friday />\n" + "<Saturday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(tfile->fp, + "<StartBoundary>2020-01-01T00:00:00</StartBoundary>\n" + "<Enabled>true</Enabled>\n" + "<ScheduleByWeek>\n" + "<DaysOfWeek>\n" + "<Sunday />\n" + "</DaysOfWeek>\n" + "<WeeksInterval>1</WeeksInterval>\n" + "</ScheduleByWeek>\n"); + break; + + default: + break; + } + + xml = "</CalendarTrigger>\n" + "</Triggers>\n" + "<Principals>\n" + "<Principal id=\"Author\">\n" + "<LogonType>InteractiveToken</LogonType>\n" + "<RunLevel>LeastPrivilege</RunLevel>\n" + "</Principal>\n" + "</Principals>\n" + "<Settings>\n" + "<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\n" + "<Enabled>true</Enabled>\n" + "<Hidden>true</Hidden>\n" + "<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>\n" + "<WakeToRun>false</WakeToRun>\n" + "<ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\n" + "<Priority>7</Priority>\n" + "</Settings>\n" + "<Actions Context=\"Author\">\n" + "<Exec>\n" + "<Command>\"%s\\git.exe\"</Command>\n" + "<Arguments>--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s</Arguments>\n" + "</Exec>\n" + "</Actions>\n" + "</Task>\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", + get_tempfile_path(tfile), NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + delete_tempfile(&tfile); + free(name); + return result; +} + +static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return schtasks_schedule_tasks(cmd); + else + return schtasks_remove_tasks(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1761,6 +1923,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) #if defined(__APPLE__) static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; #else static const char platform_scheduler[] = "crontab"; #endif @@ -1789,6 +1953,8 @@ static int update_background_schedule(int enable) if (!strcmp(scheduler, "launchctl")) result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index adf24dee72d..135505f6195 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -454,6 +454,43 @@ test_expect_success 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + test -z "$xmlfile" || cp "$xmlfile" "$xmlfile.xml" + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + for frequency in hourly daily weekly + do + grep "/create /tn Git Maintenance ($frequency) /f /xml" args && + file=$(ls .git/schedule_${frequency}*.xml) && + test_xmllint "$file" || return 1 + done && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && -- gitgitgadget ^ permalink raw reply related [flat|nested] 83+ messages in thread
end of thread, other threads:[~2021-01-10 6:37 UTC | newest] Thread overview: 83+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-11-03 14:03 [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 1/3] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-03 14:03 ` [PATCH 2/3] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2020-11-03 18:45 ` Eric Sunshine 2020-11-03 21:21 ` Derrick Stolee 2020-11-03 22:27 ` Eric Sunshine 2020-11-04 13:33 ` Derrick Stolee 2020-11-04 14:17 ` Derrick Stolee 2020-11-03 14:03 ` [PATCH 3/3] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-11-03 19:06 ` Eric Sunshine 2020-11-03 21:23 ` Derrick Stolee 2020-11-03 20:18 ` [PATCH 0/3] Maintenance IV: Platform-specific background maintenance Junio C Hamano 2020-11-03 20:21 ` Junio C Hamano 2020-11-03 21:09 ` Derrick Stolee 2020-11-03 22:30 ` Junio C Hamano 2020-11-04 13:02 ` Derrick Stolee 2020-11-04 17:00 ` Junio C Hamano 2020-11-04 18:43 ` Derrick Stolee 2020-11-04 20:06 ` [PATCH v2 0/4] " Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-04 20:06 ` [PATCH v2 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget 2020-11-11 7:10 ` Eric Sunshine 2020-11-04 20:06 ` [PATCH v2 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2020-11-11 8:12 ` Eric Sunshine 2020-11-12 13:42 ` Derrick Stolee 2020-11-12 16:43 ` Eric Sunshine 2020-11-04 20:06 ` [PATCH v2 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-11-11 8:59 ` Eric Sunshine 2020-11-12 13:56 ` Derrick Stolee 2020-11-13 14:00 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget 2020-11-13 14:00 ` [PATCH v3 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2020-11-13 20:19 ` Eric Sunshine 2020-11-13 20:42 ` Derrick Stolee 2020-11-13 20:53 ` Eric Sunshine 2020-11-13 20:56 ` Eric Sunshine 2020-11-13 14:00 ` [PATCH v3 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-11-13 20:44 ` Eric Sunshine 2020-11-13 21:32 ` Derrick Stolee 2020-11-13 21:40 ` Eric Sunshine 2020-11-16 13:13 ` Derrick Stolee 2020-11-13 20:47 ` [PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine 2020-11-14 9:23 ` Eric Sunshine 2020-11-16 13:17 ` Derrick Stolee 2020-11-17 21:13 ` [PATCH v4 " Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-17 21:13 ` [PATCH v4 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget 2020-11-18 0:34 ` Eric Sunshine 2020-11-18 18:30 ` Derrick Stolee 2020-11-17 21:13 ` [PATCH v4 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2020-11-18 6:45 ` Eric Sunshine 2020-11-18 18:22 ` Derrick Stolee 2020-11-17 21:13 ` [PATCH v4 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-11-18 7:15 ` Eric Sunshine 2020-11-18 18:30 ` Derrick Stolee 2020-11-18 20:54 ` Eric Sunshine 2020-11-18 21:16 ` Derrick Stolee 2020-11-17 23:36 ` [PATCH v4 0/4] Maintenance IV: Platform-specific background maintenance Eric Sunshine 2020-11-24 2:20 ` Derrick Stolee 2020-11-24 2:59 ` Eric Sunshine 2020-11-17 23:54 ` Eric Sunshine 2020-11-24 4:16 ` [PATCH v5 " Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2020-11-24 4:16 ` [PATCH v5 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-11-27 9:08 ` Eric Sunshine 2020-12-09 19:28 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Derrick Stolee via GitGitGadget 2020-12-09 19:28 ` [PATCH v6 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2020-12-09 19:29 ` [PATCH v6 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget 2020-12-10 0:32 ` [PATCH v6 0/4] Maintenance IV: Platform-specific background maintenance Junio C Hamano 2020-12-10 0:49 ` Eric Sunshine 2020-12-10 1:04 ` Junio C Hamano 2021-01-05 12:17 ` Derrick Stolee 2021-01-05 13:08 ` [PATCH v7 " Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 1/4] maintenance: extract platform-specific scheduling Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 2/4] maintenance: include 'cron' details in docs Derrick Stolee via GitGitGadget 2021-01-05 13:08 ` [PATCH v7 3/4] maintenance: use launchctl on macOS Derrick Stolee via GitGitGadget 2021-01-10 6:34 ` Eric Sunshine 2021-01-05 13:08 ` [PATCH v7 4/4] maintenance: use Windows scheduled tasks Derrick Stolee via GitGitGadget
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).