Git server side "pre-receive" hook to create new repositories

Git server side "pre-receive" hook to create new repositories

[Dimitri Joukoff]

Some time ago, Feb 2018, I put in a proposal at Swift.org , that would
allow the Swift Package Manager to clone packages from the Internet to a
local git server and use the local version without requiring updates to
the Swift package:
https://forums.swift.org/t/proposal-to-enable-spm-to-use-a-local-repository-mirror/10291/5
.  At the time, I also raised an issue at
https://github.com/github/platform-samples/issues/171 asking for the
feature described below.  However, today I realised that that was not
the right place to make such a request, and hence I've written this email.

The above thread contains a solution to the local mirror problem whose
only drawback was that new repositories had to be MANUALLY initialised
bare on the local server.  In all other respects, in my opinion, the
solution worked perfectly and saved me downloading huge amounts of data
each time I performed a distribution clean on the project or switched to
another project that used the same package.  The solution published at
Swift.org used a Ubuntu 16.04.2 server running Git 2.16.2, and the
client was a Macbook Pro running MacOS with its default git.

Thus, this feature request is asking that the 'pre-receive' hook
triggers when someone tries to push to a repository regardless of
whether the repository exists.  This would allow automatic creation of
new repositories and smooth the work-flow described above.  If the
semantics of the existing 'pre-receive' hook are such that it would not
be suitable for such a purpose, then an alternative way of providing the
call-back ability would be implemented.

Best regards,
Dimitri.

Re: Git server side "pre-receive" hook to create new repositories

[Jeff King]

On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:

> Thus, this feature request is asking that the 'pre-receive' hook
> triggers when someone tries to push to a repository regardless of
> whether the repository exists.  This would allow automatic creation of
> new repositories and smooth the work-flow described above.  If the
> semantics of the existing 'pre-receive' hook are such that it would not
> be suitable for such a purpose, then an alternative way of providing the
> call-back ability would be implemented.

The pre-receive hook is a bit too late for this. It runs after the
server has told the client what it has in the repo, the client decides
what to push, and the server has received the pack. So receive-pack
would have to know about this and fake having an empty repository. And
then figure out where to store the incoming packfile, since we have no
repo.

So I think it would have to be another hook that runs before the rest of
receive-pack. I.e., a system-level config option that says "if you are
asked to accept a push for a repo and it doesn't exist, run this instead
and then run as usual".

It does feel a little error-prone, though, if the client does not
positively say "I want you to create this if it doesn't exist".
Otherwise if I do "git push server:my-misspelled-repo.git", the result
is going to be rather confusing. And retro-fitting that into the
receive-pack protocol is going to be tricky.

It would be much easier to have a separate endpoint for the client to
say "please make this repo if it doesn't exist". And then just run that
before doing the push.

For an unrestricted client connecting over ssh, we already have that:
you can just run "ssh $host git init /path/to/repo". There isn't a
similar thing that can be done over HTTP, though.

-Peff

Re: Git server side "pre-receive" hook to create new repositories

[Ævar Arnfjörð Bjarmason]

On Tue, Mar 19 2019, Jeff King wrote:

> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
>
>> Thus, this feature request is asking that the 'pre-receive' hook
>> triggers when someone tries to push to a repository regardless of
>> whether the repository exists.  This would allow automatic creation of
>> new repositories and smooth the work-flow described above.  If the
>> semantics of the existing 'pre-receive' hook are such that it would not
>> be suitable for such a purpose, then an alternative way of providing the
>> call-back ability would be implemented.
>
> The pre-receive hook is a bit too late for this. It runs after the
> server has told the client what it has in the repo, the client decides
> what to push, and the server has received the pack. So receive-pack
> would have to know about this and fake having an empty repository. And
> then figure out where to store the incoming packfile, since we have no
> repo.
>
> So I think it would have to be another hook that runs before the rest of
> receive-pack. I.e., a system-level config option that says "if you are
> asked to accept a push for a repo and it doesn't exist, run this instead
> and then run as usual".
>
> It does feel a little error-prone, though, if the client does not
> positively say "I want you to create this if it doesn't exist".
> Otherwise if I do "git push server:my-misspelled-repo.git", the result
> is going to be rather confusing. And retro-fitting that into the
> receive-pack protocol is going to be tricky.
>
> It would be much easier to have a separate endpoint for the client to
> say "please make this repo if it doesn't exist". And then just run that
> before doing the push.
>
> For an unrestricted client connecting over ssh, we already have that:
> you can just run "ssh $host git init /path/to/repo". There isn't a
> similar thing that can be done over HTTP, though.

Sounds simpler to just change the user's login shell to a wrapper that
checks if the repo exists, and if not create it before proceeding.

The same with http(s). I.e. in whatever webserver that's now pointing to
git-http-backend as a script point to the same wrapper script.

Re: Git server side "pre-receive" hook to create new repositories

[Dimitri Joukoff]

Thanks for the feedback.

Whilst I understand the options that you've both proposed, the intent
was to enable the Swift Package Manager to mirror repositories in a
transparent way.  I'll look into whether these options can be
implemented inside the SPM.

Cheers,
Dimitri.


On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote:
> On Tue, Mar 19 2019, Jeff King wrote:
>
>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
>>
>>> Thus, this feature request is asking that the 'pre-receive' hook
>>> triggers when someone tries to push to a repository regardless of
>>> whether the repository exists.  This would allow automatic creation of
>>> new repositories and smooth the work-flow described above.  If the
>>> semantics of the existing 'pre-receive' hook are such that it would not
>>> be suitable for such a purpose, then an alternative way of providing the
>>> call-back ability would be implemented.
>> The pre-receive hook is a bit too late for this. It runs after the
>> server has told the client what it has in the repo, the client decides
>> what to push, and the server has received the pack. So receive-pack
>> would have to know about this and fake having an empty repository. And
>> then figure out where to store the incoming packfile, since we have no
>> repo.
>>
>> So I think it would have to be another hook that runs before the rest of
>> receive-pack. I.e., a system-level config option that says "if you are
>> asked to accept a push for a repo and it doesn't exist, run this instead
>> and then run as usual".
>>
>> It does feel a little error-prone, though, if the client does not
>> positively say "I want you to create this if it doesn't exist".
>> Otherwise if I do "git push server:my-misspelled-repo.git", the result
>> is going to be rather confusing. And retro-fitting that into the
>> receive-pack protocol is going to be tricky.
>>
>> It would be much easier to have a separate endpoint for the client to
>> say "please make this repo if it doesn't exist". And then just run that
>> before doing the push.
>>
>> For an unrestricted client connecting over ssh, we already have that:
>> you can just run "ssh $host git init /path/to/repo". There isn't a
>> similar thing that can be done over HTTP, though.
> Sounds simpler to just change the user's login shell to a wrapper that
> checks if the repo exists, and if not create it before proceeding.
>
> The same with http(s). I.e. in whatever webserver that's now pointing to
> git-http-backend as a script point to the same wrapper script.
>
>

Re: Git server side "pre-receive" hook to create new repositories

[Sitaram Chamarty]

On 19/03/2019 17.46, Dimitri Joukoff wrote:
> Thanks for the feedback.
> 
> Whilst I understand the options that you've both proposed, the intent
> was to enable the Swift Package Manager to mirror repositories in a
> transparent way.  I'll look into whether these options can be
> implemented inside the SPM.

I don't know the Swift Package Manager, but looking at the preceding
discussion reminded that gitolite allows this.

I'm not saying that someone should install gitolite just to get this
feature, but rather that *any* script that traps the incoming request
can do this.  Almost trivial if the request is coming in via ssh,
actually.  Just set the shell to a script that looks at and parses
SSH_ORIGINAL_COMMAND and do whatever is needed before passing control to
git proper.

sitaram

> 
> Cheers,
> Dimitri.
> 
> 
> On 19/3/19 7:24 pm, Ævar Arnfjörð Bjarmason wrote:
>> On Tue, Mar 19 2019, Jeff King wrote:
>>
>>> On Sat, Mar 09, 2019 at 10:46:09AM +0000, Dimitri Joukoff wrote:
>>>
>>>> Thus, this feature request is asking that the 'pre-receive' hook
>>>> triggers when someone tries to push to a repository regardless of
>>>> whether the repository exists.  This would allow automatic creation of
>>>> new repositories and smooth the work-flow described above.  If the
>>>> semantics of the existing 'pre-receive' hook are such that it would not
>>>> be suitable for such a purpose, then an alternative way of providing the
>>>> call-back ability would be implemented.
>>> The pre-receive hook is a bit too late for this. It runs after the
>>> server has told the client what it has in the repo, the client decides
>>> what to push, and the server has received the pack. So receive-pack
>>> would have to know about this and fake having an empty repository. And
>>> then figure out where to store the incoming packfile, since we have no
>>> repo.
>>>
>>> So I think it would have to be another hook that runs before the rest of
>>> receive-pack. I.e., a system-level config option that says "if you are
>>> asked to accept a push for a repo and it doesn't exist, run this instead
>>> and then run as usual".
>>>
>>> It does feel a little error-prone, though, if the client does not
>>> positively say "I want you to create this if it doesn't exist".
>>> Otherwise if I do "git push server:my-misspelled-repo.git", the result
>>> is going to be rather confusing. And retro-fitting that into the
>>> receive-pack protocol is going to be tricky.
>>>
>>> It would be much easier to have a separate endpoint for the client to
>>> say "please make this repo if it doesn't exist". And then just run that
>>> before doing the push.
>>>
>>> For an unrestricted client connecting over ssh, we already have that:
>>> you can just run "ssh $host git init /path/to/repo". There isn't a
>>> similar thing that can be done over HTTP, though.
>> Sounds simpler to just change the user's login shell to a wrapper that
>> checks if the repo exists, and if not create it before proceeding.
>>
>> The same with http(s). I.e. in whatever webserver that's now pointing to
>> git-http-backend as a script point to the same wrapper script.
>>
>>
> 
> 
> 

-- 
sitaram