From: Junio C Hamano <gitster@pobox.com>
To: Josh Steadmon <steadmon@google.com>
Cc: git@vger.kernel.org, git@jeffhostetler.com, stolee@gmail.com
Subject: Re: [PATCH v4 3/4] trace2: don't overload target directories
Date: Fri, 04 Oct 2019 09:25:44 +0900 [thread overview]
Message-ID: <xmqq1rvt9x13.fsf@gitster-ct.c.googlers.com> (raw)
In-Reply-To: <98a8440d3f0ef3cd3cdd0291051d976d4a659cc4.1570144820.git.steadmon@google.com> (Josh Steadmon's message of "Thu, 3 Oct 2019 16:32:58 -0700")
Josh Steadmon <steadmon@google.com> writes:
> trace2 can write files into a target directory. With heavy usage, this
> directory can fill up with files, causing difficulty for
> trace-processing systems.
Sorry for not mentioning this, but "don't overload" is a suboptimal
keyword for the entire topic and for this particular step for a few
reasons. For one, "overload" is an overloaded verb that gives an
incorrect impression that the problem you are dealing with is that
the target directory you specify is (mis)used for other purposes,
which is not the case. You instead refrain from creating too many
files. The other (which is probably more serious) is that it is
unclear what approach you chose to solve the "directory ends up
holding too many files". One could simply discard new traces to do
so, one could concatenate to existing files to avoid creating new
files, one could even cycle the directory (i.e. path/to/log may
become path/to/log.old.1 and path/to/log is recreated as an empty
directory when it gets a new file).
trace2: discard new traces when a target directory has too many files
or something would convey the problem you are solving (i.e. "too
many files" implying negative performance and usability impact
coming from it) and solution (i.e. "discard new traces"), if it is
the approach you have chosen.
> + /* check sentinel */
> + strbuf_addbuf(&sentinel_path, &path);
> + strbuf_addstr(&sentinel_path, OVERLOAD_SENTINEL_NAME);
> + if (!stat(sentinel_path.buf, &statbuf)) {
> + ret = 1;
> + goto cleanup;
> + }
> +
> + /* check file count */
> + dirp = opendir(path.buf);
> + while (file_count < tr2env_max_files && dirp && readdir(dirp))
> + file_count++;
> + if (dirp)
> + closedir(dirp);
So, until we accumulate too many files in the directory, every
process when it starts tracing will scan the output directory.
Hopefully the max is not set to too large a value.
next prev parent reply other threads:[~2019-10-04 0:25 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-29 22:20 [RFC PATCH] trace2: don't overload target directories Josh Steadmon
2019-07-30 13:29 ` Derrick Stolee
2019-07-30 21:52 ` Josh Steadmon
2019-07-30 16:46 ` Jeff Hostetler
2019-07-30 22:01 ` Josh Steadmon
2019-07-30 22:02 ` Josh Steadmon
2019-07-30 18:00 ` Jeff Hostetler
2019-07-30 22:08 ` Josh Steadmon
2019-08-02 22:02 ` [RFC PATCH v2 0/2] " Josh Steadmon
2019-08-02 22:02 ` [RFC PATCH v2 1/2] docs: mention trace2 target-dir mode in git-config Josh Steadmon
2019-08-02 22:02 ` [RFC PATCH v2 2/2] trace2: don't overload target directories Josh Steadmon
2019-08-05 15:34 ` Jeff Hostetler
2019-08-05 18:17 ` Josh Steadmon
2019-08-05 18:01 ` SZEDER Gábor
2019-08-05 18:09 ` Josh Steadmon
2019-09-14 0:25 ` [RFC PATCH v3 0/3] " Josh Steadmon
2019-09-14 0:25 ` [RFC PATCH v3 1/3] docs: mention trace2 target-dir mode in git-config Josh Steadmon
2019-09-14 0:25 ` [RFC PATCH v3 2/3] trace2: don't overload target directories Josh Steadmon
2019-09-14 0:26 ` [RFC PATCH v3 3/3] trace2: write overload message to sentinel files Josh Steadmon
2019-09-16 12:07 ` Derrick Stolee
2019-09-16 14:11 ` Jeff Hostetler
2019-09-16 18:20 ` Josh Steadmon
2019-09-19 18:23 ` Jeff Hostetler
2019-09-19 22:47 ` Josh Steadmon
2019-09-20 15:59 ` Jeff Hostetler
2019-09-16 18:07 ` Josh Steadmon
2019-10-03 23:32 ` [PATCH v4 0/4] trace2: don't overload target directories Josh Steadmon
2019-10-03 23:32 ` [PATCH v4 1/4] docs: mention trace2 target-dir mode in git-config Josh Steadmon
2019-10-03 23:32 ` [PATCH v4 2/4] docs: clarify trace2 version invariants Josh Steadmon
2019-10-03 23:32 ` [PATCH v4 3/4] trace2: don't overload target directories Josh Steadmon
2019-10-04 0:25 ` Junio C Hamano [this message]
2019-10-04 21:57 ` Josh Steadmon
2019-10-04 9:12 ` Johannes Schindelin
2019-10-04 22:05 ` Josh Steadmon
2019-10-03 23:32 ` [PATCH v4 4/4] trace2: write overload message to sentinel files Josh Steadmon
2019-10-04 22:08 ` [PATCH v5 0/4] trace2: discard new traces if the target directory contains too many files Josh Steadmon
2019-10-04 22:08 ` [PATCH v5 1/4] docs: mention trace2 target-dir mode in git-config Josh Steadmon
2019-10-04 22:08 ` [PATCH v5 2/4] docs: clarify trace2 version invariants Josh Steadmon
2019-10-04 22:08 ` [PATCH v5 3/4] trace2: discard new traces if target directory has too many files Josh Steadmon
2019-10-04 22:08 ` [PATCH v5 4/4] trace2: write discard message to sentinel files Josh Steadmon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqq1rvt9x13.fsf@gitster-ct.c.googlers.com \
--to=gitster@pobox.com \
--cc=git@jeffhostetler.com \
--cc=git@vger.kernel.org \
--cc=steadmon@google.com \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).