auto packing with simultaneous pushes: "error: Could not stat 'objects/[…]/[…]'"

auto packing with simultaneous pushes: "error: Could not stat 'objects/[…]/[…]'"

[Ivan D Vasin]

my coworker and i just pushed some commits seconds apart from each
other.  both of our pushes triggered a server-side auto pack
(presumably as part of ``receive-pack``, as a result of the setting
``receive.autogc = true``).  each auto pack produced multiple errors
of the form ``error: Could not stat 'objects/[…]/[…]'``.  there
appears to be no overlap between the two auto packs' lists of failed
object hashes.

my suspicion is that the two auto pack processes each generated a list
of objects to pack, then proceeded to pack them, removing the
corresponding files as they went along.  as each one encountered an
entry in its list that was previously handled by the other process, it
found that it was unable to stat the corresponding file because it had
been removed by the other process.

my suggestion is that an auto pack should lock the repository,
preventing at least other auto packs (and perhaps other operations)
from running until it is finished.  alternatively, the auto pack can
skip over objects that were already packed by another process;
however, i imagine this would be much less efficient if done
correctly.

here is the console output from my push:

$ git push
X11 forwarding request failed on channel 0
Counting objects: 32, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 2.70 KiB, done.
Total 18 (delta 12), reused 0 (delta 0)
Auto packing the repository for optimum performance.
error: Could not stat 'objects/00/
c706ccf0ef06092134ccb1f9fdf0e7d39a9d5f'
error: Could not stat 'objects/02/7c2932047c9c9c7bacd6e1abdc3810866161bc'
error: Could not stat 'objects/13/b2c3f0a55e8de099553cad6901a150f19b83c2'
error: Could not stat 'objects/2a/3992fa9c096f7514a62781a3c3923a996a7073'
error: Could not stat 'objects/3d/cb6e9055b9b51911aae52f4bb0ca8d8719d645'
error: Could not stat 'objects/50/c2f6d8e0de1eba988256597cc1d6e1a387902c'
error: Could not stat 'objects/50/9f494f6b60741957a45cf51f8d097f0578c89a'
error: Could not stat 'objects/52/6597c12d730c5903d46f598c6ec438cfabfe0f'
error: Could not stat 'objects/52/a6b4ee4bebb043d21858455fd0cd33b4812a42'
error: Could not stat 'objects/53/71c6f99023efecb3a4c16daeeee63f534d63b2'
error: Could not stat 'objects/5f/bd49df3c4abdc93c7118f762ac770a0fcd6ce4'
error: Could not stat 'objects/67/900e6e598e8e1c7bde5776fd220f7daf147cde'
error: Could not stat 'objects/72/42ecd310d0286da6c3967985c6446cf105b0bd'
error: Could not stat 'objects/79/4afecb2d348dde5ad4d24cf62c8fcd4db3b63a'
error: Could not stat 'objects/7e/4c612c51c5d76f03f12c04241d3b3c5aa212e4'
error: Could not stat 'objects/84/fd1e038d4345bac1fa2b33edb73fd73813ac4f'
error: Could not stat 'objects/89/c32fdbf3400c797d01659f9ee644aa8a713574'
error: Could not stat 'objects/8b/cbe10b0d09325b49876416341d6cf4bccc9e35'
error: Could not stat 'objects/8d/6dcc95d6bea57cfafe8a08fed7947561682b68'
error: Could not stat 'objects/b9/67af0f23deb87f1708689d159ee6444503462a'
error: Could not stat 'objects/bb/9d6fafe3ba2189582ac48088ee6f732755dddb'
error: Could not stat 'objects/e0/549c20e56d6ec270ec3ffee6c93a6d6cb8f6b5'
error: Could not stat 'objects/f2/737f0d8635265a56d57c27a3ddf155da48b07a'
error: Could not stat 'objects/fc/8b2f0837aefa21d86de65f6230c86f64addb8d'
error: Could not stat 'objects/fc/ac21d190770fdee23e5e69cdba36eefd6e001a'
error: Could not stat 'objects/fd/ae521d6a3404ceabff223385d7b5b5c783f0fe'
To git@code.adverplex.com:syseng
   fb49a86..e161cf5  HEAD -> uber_release_alpha

here is the console output from my coworker's push:

$ git push
Counting objects: 181, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (154/154), done.
Writing objects: 100% (159/159), 1.81 MiB, done.
Total 159 (delta 74), reused 0 (delta 0)
Auto packing the repository for optimum performance.
error: Could not stat 'objects/06/b2aee0b4adca015d072be4ef6a655c118b6af3'
error: Could not stat 'objects/0d/66f41003c9112b8093dfee5fd56e8db25a142b'
error: Could not stat 'objects/0d/6c3e296fb7b83998677d207f9df6a69e144168'
error: Could not stat 'objects/14/64e2bd863a15e08be7aa2f33b8df5b5f695608'
error: Could not stat 'objects/14/19cad4e2a38720186eb36b3c756d05e1963877'
error: Could not stat 'objects/16/1b9e65004a9120ea250ef8ef8292d5416ed984'
error: Could not stat 'objects/23/b6bc50c16fe3740393c8a08cf57cc07a56c778'
error: Could not stat 'objects/60/1d7c562cafaf6c71c57dbbd0bab55c3698d0c7'
error: Could not stat 'objects/6c/c5c7ededba86ce291b531def840e757ce3eccc'
error: Could not stat 'objects/74/8d6bece2b4ccec41584d22152d0fd903e01908'
error: Could not stat 'objects/9a/6c21779e7c063b87bbf055e4c6b6baecc8a3e1'
error: Could not stat 'objects/9a/6519e7bb625f681ead373e9a9226f862ee1550'
error: Could not stat 'objects/9b/7027422cbce1f5f2c5929ae1313ecee2cd3537'
error: Could not stat 'objects/a4/7e0b5906c2b552158a1e5943063e983532988f'
error: Could not stat 'objects/a4/6db00ea7b7d2e618731d7b4f1e59b5f0276aff'
error: Could not stat 'objects/ab/b3a8fd9585af5a7f6dbd609e6b4ed8c4359ca6'
error: Could not stat 'objects/ad/0c98f2674807beee093cead1179d95756c1788'
error: Could not stat 'objects/bf/4fd1b9cc617d3fc832350783064418e0d7fdfd'
error: Could not stat 'objects/bf/c65d6bc705dfc5018c3d25228a4469d069f3b8'
error: Could not stat 'objects/c0/adb5b39efd31698789ee8aefcedf91913fc7f1'
error: Could not stat 'objects/ca/f4ee60f223039d2ec6185adbb4c28ab7df461e'
error: Could not stat 'objects/ca/e3b7c60b4b3a34bba4b4a5738edf0f7786a784'
error: Could not stat 'objects/cb/4a470752088b708290713b0a513e22bb0dcb15'
error: Could not stat 'objects/d3/d44ba75332e3e7e21a89eafe724e682cf124dd'
error: Could not stat 'objects/d3/8a635a420af938f680ab30c6252fdddd448a5d'
error: Could not stat 'objects/d8/f102045fc608b65cb1d75ad8de11a673ebd656'
error: Could not stat 'objects/da/97ac29d9024f589b4aec46bc0ea1f597b00802'
error: Could not stat 'objects/e5/3e9d05c4e9a3e336453af3b43fe06dfa5c6d57'
error: Could not stat 'objects/ea/b586bcc6925bcb34cdeede1562a851d5b58d90'
error: Could not stat 'objects/ec/f2cf1dd8d6045242676cd22320902c085ecbb3'
error: Could not stat 'objects/f3/819de13ed535bdd92580e2f2b01580714188e1'
To git@code.adverplex.com:syseng
   fb49a86..a39c897  master -> master

``git fsck`` is successful on both of our repos and on the bare repo
to which we pushed.

Re: auto packing with simultaneous pushes: "error: Could not stat 'objects/[…]/[…]'"

[Ivan D Vasin]

On Wed, Jan 23, 2013 at 7:28 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Ivan D Vasin <nisavid@gmail.com> writes:
>
>> my suggestion is that an auto pack should lock the repository,
>> preventing at least other auto packs (and perhaps other operations)
>> ...
>>
>> ``git fsck`` is successful on both of our repos and on the bare repo
>> to which we pushed.
>
> Successful after you pushed, or before you pushed, or both?

both

>
> I suspect both.
>
> I do not think such a lock is necessary for correctness of the
> operation, but running two auto packing sumultaneously is wasteful,
> so it would help performance.  But that would produce a larger
> problem.  What if your modified auto-packer takes a lock and then
> dies without relinquishing the lock?  The repository will never be
> repacked after such an event forever?

perhaps the lock could contain the PID of the auto pack process.  if
that PID has gone away, the lock is ignored and replaced with a new
one.

that's what comes to my mind.  of course, there could be other ways to
handle this that i'm not thinking of.

in any case, the error messages, though spurious, are alarming to the
uninformed user.  it looks like Git is saying that there is actual
data loss, where in fact there is none.  if Git doesn't prevent these
messages from appearing (via locking behavior or otherwise), then it
should at least annotate them with a message that describes their
possibly spurious nature and perhaps instructs the user to verify
everything with ``git fsck``.