git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Jeff Hostetler <git@jeffhostetler.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Jeff King" <peff@peff.net>
Cc: git@vger.kernel.org, Derrick Stolee <stolee@gmail.com>,
	Johannes Schindelin <Johannes.Schindelin@gmx.de>
Subject: Re: Contributor Summit Topics and Logistics
Date: Wed, 30 Jan 2019 17:26:23 -0500	[thread overview]
Message-ID: <e0173444-c5c4-61d0-5a3d-e37cbfd44bad@jeffhostetler.com> (raw)
In-Reply-To: <87va253lun.fsf@evledraar.gmail.com>



On 1/30/2019 3:57 PM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Jan 22 2019, Jeff King wrote:
> 
...
> * "Structured remote logging". We had an RFC spec for turning our trace
>    format into something more structural with a way to send it to a
>    remote server. There were both implementation & privacy concernse,
>    last time at least a couple of users of git reported having in-house
>    patches for this (not ready for upstream). Where are we on this now?

I won't be attending GitMerge this year, but I can talk about
this work here.

My earlier "structured logging" and/or "telemetry" proposals
have been replaced by my Trace2 patch series now in "pu".

The Trace2 feature is designed to report trace and performance
data from within the git process to a local log file, unix
domain socket, or Windows named pipe.  Functions in the Trace2
API generate structured data and can write either structured
(JSON) or non-structured formats to disk.  (It should not be
hard to add a binary structured format too, but that is beyond
the scope of the current patch series.)

The JSON stream is suitable for post-processing by a local
process.  This can be a daemon listening to the stream or a
cron job processing the trace data after the fact.

I consider it to be the job of the post-processor (after
aggregating, filtering or whatever) to decide what to do with
the data.  This lets the the user and/or sysadmin control how
and when data is collected.  The post-processor is free to hook
into something like syslog or ETW or write to a custom DB.

Post-processing tools are not included in the patch series.


Internally within Microsoft, we have a local Windows Service
listening on a named pipe and collecting events from all
git processes for our GVFS users in the Windows OS repo.
It computes a summary record for each git command, for example
combining the argv from the "start" event with the elapsed
time from the "exit" event into a single record.  The service
then sends the aggregate records to a centralized database.

This lets us run various database queries to try to understand
pain points that our OS developers are experiencing (and that
may not show up on my machine) and help us prioritize future perf
and scaling work.

But again, this service is but one possible post-processor
and is for internal-use-only.

The Trace2 feature itself does not have any remote capability.
It just writes data locally.

Jeff



  reply	other threads:[~2019-01-30 23:05 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-22  7:50 Contributor Summit Topics and Logistics Jeff King
2019-01-22  8:26 ` Jeff King
2019-01-22  9:17   ` GSoC 2019 (was: Contributor Summit Topics and Logistics) Christian Couder
2019-01-31  2:02     ` SZEDER Gábor
2019-01-31  6:11       ` Christian Couder
2019-01-22 18:21   ` Contributor Summit Topics and Logistics Stefan Beller
2019-01-22 20:53     ` Jeff King
2019-01-22 18:23 ` Derrick Stolee
2019-01-24  8:57   ` Ævar Arnfjörð Bjarmason
2019-01-29 18:22     ` Derrick Stolee
2019-01-22 20:30 ` Elijah Newren
2019-01-30 20:57 ` Ævar Arnfjörð Bjarmason
2019-01-30 22:26   ` Jeff Hostetler [this message]
2019-01-30 22:51   ` Philip Oakley
2019-01-30 23:13     ` Christian Couder
2019-01-30 23:07 ` Jeff King
2019-02-02 12:33   ` Jakub Narebski
2019-02-04 19:30     ` Elijah Newren
2019-04-23  3:45     ` Jeff King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e0173444-c5c4-61d0-5a3d-e37cbfd44bad@jeffhostetler.com \
    --to=git@jeffhostetler.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    --cc=stolee@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).