Status of the svn remote helper project (Nov, 2010)

[Jonathan Nieder <jrnieder@gmail.com>]

The svn remote helper project still has a long way to go.  In the
meantime, the svn-fe plumbing and Tomas's scripted prototype are
usable.

Here are some topics that might be roughly in their final form.  If
you would like to build on one of them, please let me know so I can
refrain from rewriting that piece of history.

A merge of these branches is available as

	git://repo.or.cz/git/jrn.git vcs-svn-pu

and individual topic branches are also available in that repository,
though for convenience they are not in the refs/heads namespace.

Thoughts and improvements welcome.

--------------------------------------------------
[Cooking]
* jn/svndiff0 (2010-11-06) 24 commits
 - vcs-svn: Allow deltas to copy from preimage
 - vcs-svn: Reject deltas that read past end of preimage
 - vcs-svn: Let deltas use data from postimage
 - vcs-svn: Reject deltas that do not consume all inline data
 - vcs-svn: Check declared number of output bytes
 - vcs-svn: Implement copyfrom_data delta instruction
 - vcs-svn: Read instructions from deltas
 - vcs-svn: Read inline data from deltas
 - vcs-svn: Read the preimage while applying deltas
 - vcs-svn: Skeleton of an svn delta parser
 - compat: helper for detecting unsigned overflow
 - vcs-svn: Learn to check for SVN\0 magic
 - vcs-svn: Learn to parse variable-length integers
 - vcs-svn: Add code to maintain a sliding view of a file
 - vcs-svn: Allow character-oriented input
 - vcs-svn: Allow input errors to be detected early
 - vcs-svn: Let callers peek ahead to find stream end
 - vcs-svn: Add binary-safe read() function
 - vcs-svn: Improve support for reading large files
 - vcs-svn: Make buffer_skip_bytes() report partial reads
 - vcs-svn: Teach line_buffer to handle multiple input files
 - vcs-svn: Collect line_buffer data in a struct
 - vcs-svn: Replace buffer_read_string() memory pool with a strbuf
 - vcs-svn: Eliminate global byte_buffer[] array

An SVN-format delta applier.  Seems okay, but it has not been heavily
exercised with real-world deltas.

* db/fast-import-cat-blob (2010-11-07) 3 commits
 - fast-import: let importers retrieve blobs
 - fast-import: clarify documentation of "feature" command
 - fast-import: stricter parsing of integer options

As David says: "it has some significant consequences".

A start for bi-directional communication with fast-import (needed by
svn-fe to avoid keeping its own database of blobs).  Seems to be in
okay shape.

* db/svn-fe-dumpfile3 (2010-11-07) 6 commits
 - vcs-svn: apply node text deltas
 - Merge branch 'jn/svndiff0' into db/svn-fe-dumpfile3
 - Merge branch 'db/fast-import-cat-blob' into db/svn-fe-dumpfile3
 - vcs-svn: Add output file param to buffer_copy_bytes()
 - vcs-svn: Find basis for deltified nodes; apply node prop deltas
 - vcs-svn: Teach dump parser about new header types
 (this branch uses jn/svndiff0 and db/fast-import-cat-blob.)

Adding support for dumpfiles with deltas (which is pretty close to
what the ra protocol sends over the wire) to svn-fe.

The tip commit could use some cleaning up.

* rr/svnfe-tests-no-perl (2010-11-07) 1 commit
 - t9010 (svn-fe): Eliminate dependency on svn perl bindings

Ejected from the jn/svndiff0 topic.  A noninvasive simplification;
what more could one ask for?

* jn/wrappers-no-libz (2010-11-06) 7 commits
 - Remove pack file handling dependency from wrapper.o
 - pack-objects: mark file-local variable static
 - wrapper: give zlib wrappers their own translation unit
 - strbuf: move strbuf_branchname to sha1_name.c
 - path helpers: move git_mkstemp* to wrapper.c
 - wrapper: move odb_* to environment.c
 - wrapper: move xmmap() to sha1_file.c

Approach seems reasonable.  More eyes on the tip commit would
be comforting.

* xx/wrappers-no-libz-svndiff0 (2010-11-07) 2 commits
 - svn-fe: stop linking to libz and libxdiff
 - Merge branch 'jn/svndiff0' into xx/wrappers-no-libz-svndiff0
 (this branch uses jn/wrappers-no-libz and jn/svndiff0.)

Example application of the jn/wrappers-no-libz topic.

--------------------------------------------------
[Not picked up yet]

* db/branch-mapper: $gmane/158375
 . contrib/svn-fe: Fast script to remap svn history

Could use a usage example (perhaps a test script).

* tc/remote-helper-usability: $gmane/157860
 . Register new packs after the remote helper is done fetching
 . Properly record history of the notes ref
 . Fix ls-remote output when displaying impure refs
 . Add git-remote-svn
 . Introduce the git fast-import-helper
 . Rename get_mode() to decode_tree_mode() and export it
 . Allow the transport fetch command to add additional refs
 . Allow more than one keepfile in the transport
 . Remote helper: accept ':<value> <name>' as a response to 'list'

The fourth-from-top seems a bit hard to review.  If it really is
necessary to introduce a separate program with a separate interface,
maybe a compile-time flag to choose between them would help?

* rr/remote-helper: http://github.com/artagnon/git
 . remote-svn: Write in fetch functionality
 . run-command: Protect the FD 3 from being grabbed
 . remote-svn: Build a pipeline for the import using svnrdump
 . run-command: Extend child_process to include a backchannel FD
 . Allow the transport fetch command to add additional refs
 . Remote helper: accept ':<value> <name>' as a response to 'list'
 . test-svn-fe: Allow for a dumpfile on stdin
 . contrib/svn-fe: Fast script to remap svn history
 . Add Tom's remote helper for reference
 . Add a stubby remote-svn remote helper
 . Add a correct svndiff applier

Work in progress, waiting on lower levels to be more functional
(in particular, svn-fe does not support incremental imports yet).

* sb/svn-fe-example: $gmane/159054

--------------------------------------------------
[Design note (vaporware)]

See $gmane/157141 for some hints about implementing incremental
imports.

$gmane means http://thread.gmane.org/gmane.comp.version-control.git