Re: [PATCH v2 00/10] Add 'ls-files --debug-json' to dump the index in json

[Johannes Schindelin <Johannes.Schindelin@gmx.de>]

Hi Duy,

On Tue, 25 Jun 2019, Duy Nguyen wrote:

> On Tue, Jun 25, 2019 at 1:00 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > > - extension location is printed, in case you need to decode the
> > >   extension by yourself (previously only the size is printed)
> > > - all extensions are printed in the same order they appear in the file
> > >   (previously eoie and ieot are printed first because that's how we
> > >   parse)
> > > - resolve undo extension is reorganized a bit to be easier to read
> > > - tests added. Example json files are in t/t3011
> >
> > It might actually make sense to optionally disable showing extensions.
> >
> > You also forgot to mention that you explicitly disable handling
> > `<pathspec>`, which I find a bit odd, personally, as that would probably
> > come in real handy at times,
>
> No. I mentioned the land of high level languages before. Filtering in
> any Python, Ruby, Scheme, JavaScript, Java is a piece of cake and much
> more flexible than pathspec.

I heard that type of argument before. I was working on the initial Windows
port of Git, uh, of course I was working on a big refactoring of a big C++
application backed by a database. A colleague suggested that filtering
could be done much better in C++, on the desktop, than in SQL. And so they
changed the paradigm to "simplify" the SQL query, and instead dropped the
unwanted data after it had hit the RAM of the client machine.

Turns out it was a bad idea. A _really_ bad idea. Because it required
downloading 30MB of data for about several dozens computers in parallel,
at the start of every shift.

This change was reverted in one big hurry, and the colleague was tasked to
learn them some SQL.

Why am I telling you this story? Because you fall into the exact same trap
as my colleague.

In this instance, it may not be so much network bandwidth, but it is still
quite a suboptimal idea to render JSON for possibly tens of thousands of
files, then parse the same JSON on the receiving side, the spend even more
time to drop all but a dozen files.

And this is _even more_ relevant when you want to debug things.

In short: I am quite puzzled why this is even debated here. There is a
reason, a good reason, why `git ls-files` accepts pathspecs. I would not
want to ignore the lessons of history as willfully here.

> Even with shell scripts, jq could do a much better job than pathspec. If
> you filter by pathspec, good luck trying that on extensions.

You keep harping on extensions, but the reality of the matter is that they
are rarely interesting. I would even wager a bet that we will end up
excluding them from the JSON output by default.

Most of the times when I had to decode the index file manually in the
past, it was about the regular file entries.

There was *one* week in which I had to decode the untracked cache a bit,
to the point where I patched the test helper locally to help me with that.

If my experience in debugging these things is any indicator, extensions do
not matter even 10% of the non-extension data.

And that's not even taking into account the third-party software that
could definitely benefit from having this JSON format as query result.

In my work as Git for Windows maintainer, I do hear about the needs of
third-party software developers quite a bit, so I would claim that I know
a bit about what they need, why the NUL-terminated format is not a good
match, and how much a JSON-based API would help.

So while that is not your pet, it will be the most useful part of the
outcome of your work.

> It's the same reason why I will not provide a flexible way to disable
> extensions. I'm not starting a JSON API for Git. I provide an index
> file in JSON format. You do what you want with it. You have a format
> easy enough to import to native data structures of your favorite
> language.

I understand that you don't care.

Your patch series is just too good a start on something truly useful to
pass up on the opportunity.

> > especially when we offer this as a better way for 3rd-party
> > applications to interact with Git (which I think will be the use case
> > for this feature that will be _far_ more common than using it for
> > debugging).
>
> We may have conflicting goals. For me, first priority is the debug
> tool for Git developers. 3rd-party support is a stretch. I could move
> all this back to test-tool, then you can provide a 3rd-party API if
> you want. Or I'll withdraw this series and go back to my original
> plan.

You don't need JSON if you want to debug things. That would be a lot of
love lost, if debugging was your goal.

I guess I'll wait until your patch series hits `next`, and then try to
find some time to work on that feature.

Ciao,
Johannes