Apply git bundle to source tree?

Apply git bundle to source tree?

[Andreas Grünbacher]

Hi,

I'm wondering if there's a way to apply a particular head in a bundle
to a source tree, for example:

  $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
  $ cd linux-5.8
  $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1

That would allow to reconstruct either the original repository or just
the underlying source tree, so the bundle could be used as a kind of
super diff.

Thanks,
Andreas

Re: Apply git bundle to source tree?

[Taylor Blau]

Hi Andreas,

On Fri, Sep 18, 2020 at 01:13:52PM +0200, Andreas Grünbacher wrote:
> Hi,
>
> I'm wondering if there's a way to apply a particular head in a bundle
> to a source tree, for example:
>
>   $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
>   $ cd linux-5.8
>   $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1

Sort of. You can specify a refspec when fetching from the bundle to
fetch only the objects you care about, like:

  cd linux-5.8
  git fetch /path/to/bundle 'refs/tags/v5.9-rc1:refs/tags/v5.9-rc1'

(or if you prefer, "git fetch /path/to/bundle 'tag v5.9-rc1'"). Then
once you have the objects locally, you can merge it into your HEAD. You
can do all of that in one step with:

  git pull /path/to/bundle 'refs/tags/v5.9-rc1'

There's no such thing as 'git bundle apply' though, although I suspect
'git pull' is what you wanted anyway.

Thanks,
Taylor

Re: Apply git bundle to source tree?

[Andreas Grünbacher]

Hi Taylor,

Am Fr., 18. Sept. 2020 um 16:02 Uhr schrieb Taylor Blau <me@ttaylorr.com>:
>
> Hi Andreas,
>
> On Fri, Sep 18, 2020 at 01:13:52PM +0200, Andreas Grünbacher wrote:
> > Hi,
> >
> > I'm wondering if there's a way to apply a particular head in a bundle
> > to a source tree, for example:
> >
> >   $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
> >   $ cd linux-5.8
> >   $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1
>
> Sort of. You can specify a refspec when fetching from the bundle to
> fetch only the objects you care about, like:
>
>   cd linux-5.8
>   git fetch /path/to/bundle 'refs/tags/v5.9-rc1:refs/tags/v5.9-rc1'
>
> (or if you prefer, "git fetch /path/to/bundle 'tag v5.9-rc1'"). Then
> once you have the objects locally, you can merge it into your HEAD. You
> can do all of that in one step with:
>
>   git pull /path/to/bundle 'refs/tags/v5.9-rc1'
>
> There's no such thing as 'git bundle apply' though, although I suspect
> 'git pull' is what you wanted anyway.

I was actually looking for a way to apply a bundle to an actual source
tree, not a git repository. Fetching stuff from a bundle into a
repository seems to be reasonably well documented.

Thanks,
Andreas

Re: Apply git bundle to source tree?

[Taylor Blau]

On Fri, Sep 18, 2020 at 04:12:09PM +0200, Andreas Grünbacher wrote:
> I was actually looking for a way to apply a bundle to an actual source
> tree, not a git repository. Fetching stuff from a bundle into a
> repository seems to be reasonably well documented.

Unfortunately I don't think such a thing is possible, or at least if it
is, I can't think of how to do it.

Thanks,
Taylor

Re: Apply git bundle to source tree?

[Andreas Grünbacher]

Am Fr., 18. Sept. 2020 um 16:17 Uhr schrieb Taylor Blau <me@ttaylorr.com>:
> On Fri, Sep 18, 2020 at 04:12:09PM +0200, Andreas Grünbacher wrote:
> > I was actually looking for a way to apply a bundle to an actual source
> > tree, not a git repository. Fetching stuff from a bundle into a
> > repository seems to be reasonably well documented.
>
> Unfortunately I don't think such a thing is possible, or at least if it
> is, I can't think of how to do it.

Yes, maybe someone familiar with the bundle file format can tell
whether it's at least theoretically possible. Is there a way to figure
out which hashes the original files are supposed to have? Am I right
in assuming that v5.8^{tree} isn't included in a v5.8..v5.9-rc1
bundle?

Thanks,
Andreas

Re: Apply git bundle to source tree?

[Andreas Schwab]

On Sep 18 2020, Andreas Grünbacher wrote:

> Am I right in assuming that v5.8^{tree} isn't included in a
> v5.8..v5.9-rc1 bundle?

From git-bundle(1):

       As no direct connection between the repositories exists, the user must
       specify a basis for the bundle that is held by the destination
       repository: the bundle assumes that all objects in the basis are
       already in the destination repository.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Apply git bundle to source tree?

[Andreas Grünbacher]

Am Fr., 18. Sept. 2020 um 17:21 Uhr schrieb Andreas Schwab
<schwab@linux-m68k.org>:
> On Sep 18 2020, Andreas Grünbacher wrote:
> > Am I right in assuming that v5.8^{tree} isn't included in a
> > v5.8..v5.9-rc1 bundle?
>
> From git-bundle(1):
>
>        As no direct connection between the repositories exists, the user must
>        specify a basis for the bundle that is held by the destination
>        repository: the bundle assumes that all objects in the basis are
>        already in the destination repository.

It's pretty clear that the original objects are needed when trying to
reconnect the objects in the bundle to them. That wasn't my question,
though.

Thanks,
Andreas

Re: Apply git bundle to source tree?

[Junio C Hamano]

Andreas Grünbacher <andreas.gruenbacher@gmail.com> writes:

> I'm wondering if there's a way to apply a particular head in a bundle
> to a source tree, for example:
>
>   $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
>   $ cd linux-5.8
>   $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1
>
> That would allow to reconstruct either the original repository or just
> the underlying source tree, so the bundle could be used as a kind of
> super diff.

There seem to be a bit of misconception.

Do not think that a bundle is like a patch.  When you created the
bundle in the above example, you did not create a "super diff"
between v5.8 and v5.9-rc1 that you can apply to a working tree files
that correspond to v5.8 release.  That is not what you did.

What you created is an equivalent of a (shallow) repository, that
contains everything needed to get v5.9-rc1 by those who have a Git
repository that has v5.8 to fetch/pull from.  It is OK to have more,
but you MUST have v5.8 for the bundle in the example to be usable.

So assuming that your 'linux-5.8' is not just a tarball extract but
a Linux repository with v5.8 tag in it (i.e. "git log v5.8" gives
you sensible output) then the command to use is not apply but fetch,
e.g.

    $ git bundle fetch ../5.9-rc1.bundle v5.9-rc1

which will give you v5.9-rc1 tag.  What you can fetch from the bundle
can be listed by using the list-heads subcommand on the bundle.

And starting from that point, you would be able to do things like

    $ git checkout -b my-fork-of-5.9-rc1 v5.9-rc1

Now, assuming that your original question indeed came from thinking
of a bundle like a patch and not like a repository, we have a
question for you.  What in

    $ git bundle --help

gave such an incorrect impression?  The documentation must be at
fault here, and we need to clarify so that future readers of it will
not be confused into the same misconception.

Thanks.

Re: Apply git bundle to source tree?

[Andreas Schwab]

On Sep 18 2020, Andreas Grünbacher wrote:

> Am Fr., 18. Sept. 2020 um 17:21 Uhr schrieb Andreas Schwab
> <schwab@linux-m68k.org>:
>> On Sep 18 2020, Andreas Grünbacher wrote:
>> > Am I right in assuming that v5.8^{tree} isn't included in a
>> > v5.8..v5.9-rc1 bundle?
>>
>> From git-bundle(1):
>>
>>        As no direct connection between the repositories exists, the user must
>>        specify a basis for the bundle that is held by the destination
>>        repository: the bundle assumes that all objects in the basis are
>>        already in the destination repository.
>
> It's pretty clear that the original objects are needed when trying to
> reconnect the objects in the bundle to them. That wasn't my question,
> though.

Your question was whether v5.8^{tree} is included, and I think that
sentence gives the answer.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Apply git bundle to source tree?

[Andreas Grünbacher]

Am Fr., 18. Sept. 2020 um 17:41 Uhr schrieb Junio C Hamano <gitster@pobox.com>:
> Andreas Grünbacher <andreas.gruenbacher@gmail.com> writes:
> > I'm wondering if there's a way to apply a particular head in a bundle
> > to a source tree, for example:
> >
> >   $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
> >   $ cd linux-5.8
> >   $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1
> >
> > That would allow to reconstruct either the original repository or just
> > the underlying source tree, so the bundle could be used as a kind of
> > super diff.
>
> There seem to be a bit of misconception.
>
> Do not think that a bundle is like a patch.  When you created the
> bundle in the above example, you did not create a "super diff"
> between v5.8 and v5.9-rc1 that you can apply to a working tree files
> that correspond to v5.8 release.  That is not what you did.
>
> What you created is an equivalent of a (shallow) repository, that
> contains everything needed to get v5.9-rc1 by those who have a Git
> repository that has v5.8 to fetch/pull from.  It is OK to have more,
> but you MUST have v5.8 for the bundle in the example to be usable.
>
> So assuming that your 'linux-5.8' is not just a tarball extract but
> a Linux repository with v5.8 tag in it (i.e. "git log v5.8" gives
> you sensible output) then the command to use is not apply but fetch,
> e.g.
>
>     $ git bundle fetch ../5.9-rc1.bundle v5.9-rc1
>
> which will give you v5.9-rc1 tag.  What you can fetch from the bundle
> can be listed by using the list-heads subcommand on the bundle.
>
> And starting from that point, you would be able to do things like
>
>     $ git checkout -b my-fork-of-5.9-rc1 v5.9-rc1

Yes thanks, that's roughly what I could infer from the man page.

> Now, assuming that your original question indeed came from thinking
> of a bundle like a patch and not like a repository, we have a
> question for you.  What in
>
>     $ git bundle --help
>
> gave such an incorrect impression? The documentation must be at
> fault here, and we need to clarify so that future readers of it will
> not be confused into the same misconception.

I think the documentation is fine, nothing in it indicates that what
I'm looking for is supported. That's why I was asking if it's possible
to reconstruct the v5.9-rc1 source tree from the v5.8 source tree plus
a v5.8..v5.9-rc1 bundle at least theoretically.

Poking a little further, I see that the bundle depends on v5.8 and all
the ancestor commits of v5.8 that were merged into v5.9-rc1, which
only makes sense now. The bundle doesn't contain any of the objects
coming from those commits, so there's simply not enough information
there for what I was trying to achieve.

Thanks,
Andreas

Re: Apply git bundle to source tree?

[Konstantin Ryabitsev]

On Fri, 18 Sep 2020 at 07:14, Andreas Grünbacher
<andreas.gruenbacher@gmail.com> wrote:
>
> Hi,
>
> I'm wondering if there's a way to apply a particular head in a bundle
> to a source tree, for example:
>
>   $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
>   $ cd linux-5.8
>   $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1

I know this is not what you are asking, but since you used the kernel
as your example, you can use the following to achieve the result
you're looking for:
curl --header 'Accept-Encoding: gzip' -L
https://git.kernel.org/torvalds/p/v5.9-rc1/v5.8 | gunzip - | git apply

-K

Re: Apply git bundle to source tree?

[Andreas Grünbacher]

Am Fr., 18. Sept. 2020 um 22:18 Uhr schrieb Konstantin Ryabitsev
<konstantin@linuxfoundation.org>:
> On Fri, 18 Sep 2020 at 07:14, Andreas Grünbacher
> <andreas.gruenbacher@gmail.com> wrote:
> >
> > Hi,
> >
> > I'm wondering if there's a way to apply a particular head in a bundle
> > to a source tree, for example:
> >
> >   $ git bundle create v5.9-rc1.bundle v5.8..v5.9-rc1
> >   $ cd linux-5.8
> >   $ git bundle APPLY ../5.9-rc1.bundle v5.9-rc1
>
> I know this is not what you are asking, but since you used the kernel
> as your example, you can use the following to achieve the result
> you're looking for:
> curl --header 'Accept-Encoding: gzip' -L
> https://git.kernel.org/torvalds/p/v5.9-rc1/v5.8 | gunzip - | git apply

Oh, that's neat.

What I had in mind were actually distro packages: most projects
nowadays live somewhere in git repositories. When they're packaged,
this usually results in a source package with a diff on top of a
baseline release, so the commit history is lost. Friendly packagers
include the commit hashes and point users to a suitable git
repository, but that's not enforced or consistent. Including the
actual git history in packages would be much nicer (i.e., a git
bundle), but if that can't replace the patch as well, it's rather
unlikely to happen.

Thanks,
Andreas

Re: Apply git bundle to source tree?

[Thomas Guyot-Sionnest]

On Fri, 18 Sep 2020 at 17:45, Andreas Grünbacher
<andreas.gruenbacher@gmail.com> wrote:
>
> Am Fr., 18. Sept. 2020 um 22:18 Uhr schrieb Konstantin Ryabitsev
> >
> > I know this is not what you are asking, but since you used the kernel
> > as your example, you can use the following to achieve the result
> > you're looking for:
> > curl --header 'Accept-Encoding: gzip' -L
> > https://git.kernel.org/torvalds/p/v5.9-rc1/v5.8 | gunzip - | git apply
>
> Oh, that's neat.
>
> What I had in mind were actually distro packages: most projects
> nowadays live somewhere in git repositories. When they're packaged,
> this usually results in a source package with a diff on top of a
> baseline release, so the commit history is lost. Friendly packagers
> include the commit hashes and point users to a suitable git
> repository, but that's not enforced or consistent. Including the
> actual git history in packages would be much nicer (i.e., a git
> bundle), but if that can't replace the patch as well, it's rather
> unlikely to happen.

Afaik a git bundle can do all that, and it's actually a neat idea. It
will be bigger initially than a single release, especially if the
project has not been good at keeping history tidy, but bundles could
be used for both the base release *and* patches, and actually when you
need the next version old bundles could also help downloading just the
additional bits you need.

Since you mentioned it, patches that are maintained by packagers are
best maintained in git already, so the patchset could be available in
a bundle (you can have a single bundle with multiple refs; the
release, the patchset head (each parent back to release HEAD is a
patch) or even each individual patches is that suits you). If you
maintain them as separate bundles, you will always need the release
one for the rest to become useful, but in the end it's easy to convert
between bundles and patches.

FWIW I've been using bundles since the very beginning - my use case
though was backups. I figured it was an efficient way to archive an
entire bare repo into a single file and ship it offsite. I don't think
I kept the script but it's quite simple; bundle just needs the
heads/tags - without a range it goes all the way through history. For
the restore, I would list the heads from the bundle then fetch each
back into the right head/tag in an empty repo.

--
Thomas

Re: Apply git bundle to source tree?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

On 2020-09-18 at 21:45:01, Andreas Grünbacher wrote:
> What I had in mind were actually distro packages: most projects
> nowadays live somewhere in git repositories. When they're packaged,
> this usually results in a source package with a diff on top of a
> baseline release, so the commit history is lost. Friendly packagers
> include the commit hashes and point users to a suitable git
> repository, but that's not enforced or consistent. Including the
> actual git history in packages would be much nicer (i.e., a git
> bundle), but if that can't replace the patch as well, it's rather
> unlikely to happen.

Debian considered using Git as part of the 3.0 (git) format, but the
problem with that is that some upstreams include non-free or
undistributable material in their repositories, and obviously Debian
can't distribute such software in main.  Tarballs can be repacked, but
it's harder to rewrite Git history to exclude objects.

I do think the idea is cool and it would be a neat application, but
distributing the source history of an upstream project is tricky for
for packagers for practical reasons.
-- 
brian m. carlson: Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]