From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, Derrick Stolee <derrickstolee@github.com> Subject: [PATCH 0/7] Maintenance III: Background maintenance Date: Fri, 04 Sep 2020 15:41:59 +0000 [thread overview] Message-ID: <pull.724.git.1599234126.gitgitgadget@gmail.com> (raw) This is based on ds/maintenance-part-2 and replaces the RFC from [1]. [1] https://lore.kernel.org/git/pull.680.v3.git.1598629517.gitgitgadget@gmail.com/ This series introduces background maintenance to Git, through an integration with cron and crontab. Some preliminary work is done to allow a new --schedule option that tells the command which tasks to run based on a maintenance.<task>.schedule config option. The timing is not enforced by Git, but instead is expected to be provided as a hint from a cron schedule. The options are "hourly", "daily", and "weekly". A new for-each-repo builtin runs Git commands on every repo in a given list. Currently, the list is stored as a config setting, allowing a new maintenance.repos config list to store the repositories registered for background maintenance. Others may want to add a --file=<file> option for their own workflows, but I focused on making this as simple as possible for now. The updates to the git maintenance builtin include new register/unregister subcommands and start/stop subcommands. The register subcommand initializes the config while the start subcommand does everything register does plus update the cron table. The unregister and stop commands reverse this process. A troubleshooting guide is added to Documentation/git-maintenance.txt to advise expert users who choose to create custom cron schedules. The very last patch is entirely optional. It sets a recommended schedule based on my own experience with very large repositories. I'm open to other suggestions, but these are ones that I think work well and don't cause a "rewrite the world" scenario like running nightly 'gc' would do. I've been testing this scenario on my macOS laptop and Linux desktop. I have modified my cron task to provide logging via trace2 so I can see what's happening. A future direction here would be to add some maintenance logs to the repository so we can track what is happening and diagnose whether the maintenance strategy is working on real repos. Note: git maintenance (start|stop) only works on machines with cron by design. The proper thing to do on Windows will come later. Perhaps this command should be marked as unavailable on Windows somehow, or at least a better error than "cron may not be available on your system". I did find that that message is helpful sometimes: macOS worker agents for CI builds typically do not have cron available. Derrick Stolee (7): maintenance: optionally skip --auto process maintenance: add --schedule option and config for-each-repo: run subcommands on configured repos maintenance: add [un]register subcommands maintenance: add start/stop subcommands maintenance: recommended schedule in register/start maintenance: add troubleshooting guide to docs .gitignore | 1 + Documentation/config/maintenance.txt | 10 + Documentation/git-for-each-repo.txt | 59 ++++++ Documentation/git-maintenance.txt | 88 +++++++- Makefile | 2 + builtin.h | 1 + builtin/for-each-repo.c | 58 ++++++ builtin/gc.c | 292 ++++++++++++++++++++++++++- command-list.txt | 1 + git.c | 1 + run-command.c | 6 + t/helper/test-crontab.c | 35 ++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t0068-for-each-repo.sh | 30 +++ t/t7900-maintenance.sh | 114 ++++++++++- t/test-lib.sh | 6 + 17 files changed, 698 insertions(+), 8 deletions(-) create mode 100644 Documentation/git-for-each-repo.txt create mode 100644 builtin/for-each-repo.c create mode 100644 t/helper/test-crontab.c create mode 100755 t/t0068-for-each-repo.sh base-commit: e576ac2c7c7f6c7aa5ac08a516baeb61bf723596 Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-724%2Fderrickstolee%2Fmaintenance%2Fscheduled-v1 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-724/derrickstolee/maintenance/scheduled-v1 Pull-Request: https://github.com/gitgitgadget/git/pull/724 -- gitgitgadget
next reply other threads:[~2020-09-04 15:42 UTC|newest] Thread overview: 62+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-09-04 15:41 Derrick Stolee via GitGitGadget [this message] 2020-09-04 15:42 ` [PATCH 1/7] maintenance: optionally skip --auto process Derrick Stolee via GitGitGadget 2020-09-04 15:42 ` [PATCH 2/7] maintenance: add --schedule option and config Derrick Stolee via GitGitGadget 2020-09-08 13:07 ` Đoàn Trần Công Danh 2020-09-09 12:14 ` Derrick Stolee 2020-09-04 15:42 ` [PATCH 3/7] for-each-repo: run subcommands on configured repos Derrick Stolee via GitGitGadget 2020-09-04 15:42 ` [PATCH 4/7] maintenance: add [un]register subcommands Derrick Stolee via GitGitGadget 2020-09-04 15:42 ` [PATCH 5/7] maintenance: add start/stop subcommands Derrick Stolee via GitGitGadget 2020-09-08 6:29 ` SZEDER Gábor 2020-09-08 12:43 ` Derrick Stolee 2020-09-08 19:31 ` Junio C Hamano 2020-09-04 15:42 ` [PATCH 6/7] maintenance: recommended schedule in register/start Derrick Stolee via GitGitGadget 2020-09-04 15:42 ` [PATCH 7/7] maintenance: add troubleshooting guide to docs Derrick Stolee via GitGitGadget 2020-09-11 17:49 ` [PATCH v2 0/7] Maintenance III: Background maintenance Derrick Stolee via GitGitGadget 2020-09-11 17:49 ` [PATCH v2 1/7] maintenance: optionally skip --auto process Derrick Stolee via GitGitGadget 2020-09-11 17:49 ` [PATCH v2 2/7] maintenance: add --schedule option and config Derrick Stolee via GitGitGadget 2020-09-11 17:49 ` [PATCH v2 3/7] for-each-repo: run subcommands on configured repos Derrick Stolee via GitGitGadget 2020-09-11 17:49 ` [PATCH v2 4/7] maintenance: add [un]register subcommands Derrick Stolee via GitGitGadget 2020-09-17 14:05 ` Đoàn Trần Công Danh 2020-09-11 17:49 ` [PATCH v2 5/7] maintenance: add start/stop subcommands Derrick Stolee via GitGitGadget 2020-09-11 17:49 ` [PATCH v2 6/7] maintenance: recommended schedule in register/start Derrick Stolee via GitGitGadget 2020-09-29 19:48 ` Martin Ågren 2020-09-30 20:11 ` Derrick Stolee 2020-10-01 20:38 ` Derrick Stolee 2020-10-02 0:38 ` Đoàn Trần Công Danh 2020-10-02 1:55 ` Derrick Stolee 2020-10-05 13:16 ` Đoàn Trần Công Danh 2020-10-05 18:17 ` Derrick Stolee 2020-09-11 17:49 ` [PATCH v2 7/7] maintenance: add troubleshooting guide to docs Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 0/7] Maintenance III: Background maintenance Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 1/7] maintenance: optionally skip --auto process Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 2/7] maintenance: add --schedule option and config Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 3/7] for-each-repo: run subcommands on configured repos Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 4/7] maintenance: add [un]register subcommands Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 5/7] maintenance: add start/stop subcommands Derrick Stolee via GitGitGadget 2020-10-05 12:57 ` [PATCH v3 6/7] maintenance: use default schedule if not configured Derrick Stolee via GitGitGadget 2020-10-05 19:57 ` Martin Ågren 2020-10-08 13:32 ` Derrick Stolee 2020-10-05 12:57 ` [PATCH v3 7/7] maintenance: add troubleshooting guide to docs Derrick Stolee via GitGitGadget 2020-10-15 17:21 ` [PATCH v4 0/8] Maintenance III: Background maintenance Derrick Stolee via GitGitGadget 2020-10-15 17:21 ` [PATCH v4 1/8] maintenance: optionally skip --auto process Derrick Stolee via GitGitGadget 2020-10-15 17:21 ` [PATCH v4 2/8] maintenance: add --schedule option and config Derrick Stolee via GitGitGadget 2021-02-09 14:06 ` Ævar Arnfjörð Bjarmason 2021-02-09 16:54 ` Derrick Stolee 2021-05-10 12:16 ` Ævar Arnfjörð Bjarmason 2021-05-10 18:42 ` Junio C Hamano 2020-10-15 17:21 ` [PATCH v4 3/8] for-each-repo: run subcommands on configured repos Derrick Stolee via GitGitGadget 2021-05-03 16:10 ` Andrzej Hunt 2021-05-03 17:01 ` Eric Sunshine 2021-05-03 19:26 ` Eric Sunshine 2021-05-03 19:43 ` Derrick Stolee 2020-10-15 17:22 ` [PATCH v4 4/8] maintenance: add [un]register subcommands Derrick Stolee via GitGitGadget 2020-10-15 17:22 ` [PATCH v4 5/8] maintenance: add start/stop subcommands Derrick Stolee via GitGitGadget 2020-12-09 18:51 ` Josh Steadmon 2020-12-09 19:16 ` Josh Steadmon 2020-12-09 21:59 ` Derrick Stolee 2020-12-10 0:13 ` Junio C Hamano 2020-12-10 1:52 ` Derrick Stolee 2020-12-10 6:54 ` Junio C Hamano 2020-10-15 17:22 ` [PATCH v4 6/8] maintenance: create maintenance.strategy config Derrick Stolee via GitGitGadget 2020-10-15 17:22 ` [PATCH v4 7/8] maintenance: use 'incremental' strategy by default Derrick Stolee via GitGitGadget 2020-10-15 17:22 ` [PATCH v4 8/8] maintenance: add troubleshooting guide to docs Derrick Stolee via GitGitGadget
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://vger.kernel.org/majordomo-info.html * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=pull.724.git.1599234126.gitgitgadget@gmail.com \ --to=gitgitgadget@gmail.com \ --cc=congdanhqx@gmail.com \ --cc=derrickstolee@github.com \ --cc=git@vger.kernel.org \ --cc=jonathantanmy@google.com \ --cc=jrnieder@gmail.com \ --cc=sluongng@gmail.com \ --subject='Re: [PATCH 0/7] Maintenance III: Background maintenance' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).