Re: [RFC/PATCH v2 00/10] Add initial experimental external ODB support

[Eric Wong <e@80x24.org>]

Christian Couder <christian.couder@gmail.com> wrote:
> Design discussion about performance
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Yeah, it is not efficient to fork/exec a command to just read or write
> one object to or from the external ODB. Batch calls and/or using a
> daemon and/or RPC should be used instead to be able to store regular
> objects in an external ODB. But for now the external ODB would be all
> about really big files, where the cost of a fork+exec should not
> matter much. If we later want to extend usage of external ODBs, yeah
> we will probably need to design other mechanisms.

I would also investigate switching run_command to use vfork+exec
or posix_spawn for performance (keeping in mind vfork
caveats documented at https://ewontfix.com/7/ )

posix_spawn in future glibc (probably 2.24) will use CLONE_VFORK
in all cases under Linux, and posix_spawn may help with
portability, too.  I think the only thing we can't support
with posix_spawn which run_command supports is chdir;
all the redirections/closing FDs should be fine.

With only 10MB malloc-ed, the following shows vfork performance
being noticeably faster than plain fork:

/* gcc -o vfork-test -Wall -O2 vfork-test.c */
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[], char *envp[])
{
	int i;
	int do_vfork = argc > 1 && !strcmp(argv[1], "vfork");
	char * const cmd[] = { "/bin/true", 0 };
	size_t n = 1024 * 1024 * 10;
	char *mem = malloc(n);

	memset(mem, 'a', n); /* make sure it's really allocated */

	for (i = 0; i < 10000; i++) {
		pid_t pid = do_vfork ? vfork() : fork();

		if (pid == 0) {
			execve(cmd[0], cmd, envp);
			write(2, "exec error\n", 11);
			_exit(1);
		}
		waitpid(pid, 0, 0);
	}
	return 0;
}