[PATCH] TODO: add item for "scraper" importers

[PATCH] TODO: add item for "scraper" importers

[Eric Wong]

The git-users mailing list is on Google Groups with obfuscated
addresses and censored archives.  We should allow users to
import them soon, as obfuscated/censored archives are better
than not having archives at all when Google decides to shut down
yet-another-service.

There's also some mangling that (most) instances of Mailman do
(e.g. cgit), but being able to follow such groups over NNTP
and use our search functionality is still useful and better
than what typical Mailman installations provide.
---
 TODO | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/TODO b/TODO
index 66e5063..ef77d1c 100644
--- a/TODO
+++ b/TODO
@@ -95,3 +95,6 @@ all need to be considered for everything we introduce)
 * code repository integration (with cgit, gitweb, etc...)
 
 * migration path to v2 without breaking v1 "git fetch" cronjobs
+
+* imperfect scraper importers for obfuscated list archives
+  (e.g. obfuscated Mailman stuff, Google Groups, etc...)
-- 
EW

Re: [PATCH] TODO: add item for "scraper" importers

[Konstantin Ryabitsev]

On Tue, 5 Feb 2019 at 16:32, Eric Wong <e@80x24.org> wrote:
> The git-users mailing list is on Google Groups with obfuscated
> addresses and censored archives.  We should allow users to
> import them soon, as obfuscated/censored archives are better
> than not having archives at all when Google decides to shut down
> yet-another-service.

I've written an importer script (in Python) for our purposes:
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/list-archive-maker.py

It doesn't do Google at this time, but it can handle Mailman and
grabbing thing from Gmane NNTP.
Perhaps it'll come in handy to identify some of the hoops I've already
had to jump through.

-K