Re: [PATCH] sha1_file: introduce close_one_pack() to close packs on fd pressure

[Eric Sunshine <sunshine@sunshineco.com>]

On Tue, Jul 30, 2013 at 12:05 AM, Brandon Casey <bcasey@nvidia.com> wrote:
> When the number of open packs exceeds pack_max_fds, unuse_one_window()
> is called repeatedly to attempt to release the least-recently-used
> pack windows, which, as a side-effect, will also close a pack file
> after closing its last open window.  If a pack file has been opened,
> but no windows have been allocated into it, it will never be selected
> by unuse_one_window() and hence its file descriptor will not be
> closed.  When this happens, git may exceed the number of file
> descriptors permitted by the system.
>
> This latter situation can occur in show-ref or receive-pack during ref
> advertisement.  During ref advertisement, receive-pack will iterate
> over every ref in the repository and advertise it to the client after
> ensuring that the ref exists in the local repository.  If the ref is
> located inside a pack, then the pack is opened to ensure that it
> exists, but since the object is not actually read from the pack, no
> mmap windows are allocated.  When the number of open packs exceeds
> pack_max_fds, unuse_one_window() will not able to find any windows to

s/not able/not be able/
...or...
s/not able to find/not find/

> free and will not be able to close any packs.  Once the per-process
> file descriptor limit is exceeded, receive-pack will produce a warning,
> not an error, for each pack it cannot open, and will then most likely
> fail with an error to spawn rev-list or index-pack like:
>
>    error: cannot create standard input pipe for rev-list: Too many open files
>    error: Could not run 'git rev-list'
>
> This is not likely to occur during upload-pack since upload-pack
> reads each object from the pack so that it can peel tags and
> advertise the exposed object.  So during upload-pack, mmap windows
> will be allocated for each pack that is opened and unuse_one_window()
> will eventually be able to close unused packs after freeing all of
> their windows.
>
> When we have file descriptor pressure, in contrast to memory pressure,
> we need to free all windows and close the pack file descriptor so that
> a new pack can be opened.  Let's introduce a new function
> close_one_pack() designed specifically for this purpose to search
> for and close the least-recently-used pack, where LRU is defined as
>
>    * pack with oldest mtime and no allocated mmap windows or
>    * pack with the least-recently-used windows, i.e. the pack
>      with the oldest most-recently-used window
>
> Signed-off-by: Brandon Casey <drafnel@gmail.com>