user/dev discussion of public-inbox itself
 help / color / Atom feed
* [PATCH] TODO: add item for "scraper" importers
@ 2019-02-05 11:32 Eric Wong
  2019-02-05 18:47 ` Konstantin Ryabitsev
  0 siblings, 1 reply; 2+ messages in thread
From: Eric Wong @ 2019-02-05 11:32 UTC (permalink / raw)
  To: meta

The git-users mailing list is on Google Groups with obfuscated
addresses and censored archives.  We should allow users to
import them soon, as obfuscated/censored archives are better
than not having archives at all when Google decides to shut down
yet-another-service.

There's also some mangling that (most) instances of Mailman do
(e.g. cgit), but being able to follow such groups over NNTP
and use our search functionality is still useful and better
than what typical Mailman installations provide.
---
 TODO | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/TODO b/TODO
index 66e5063..ef77d1c 100644
--- a/TODO
+++ b/TODO
@@ -95,3 +95,6 @@ all need to be considered for everything we introduce)
 * code repository integration (with cgit, gitweb, etc...)
 
 * migration path to v2 without breaking v1 "git fetch" cronjobs
+
+* imperfect scraper importers for obfuscated list archives
+  (e.g. obfuscated Mailman stuff, Google Groups, etc...)
-- 
EW


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH] TODO: add item for "scraper" importers
  2019-02-05 11:32 [PATCH] TODO: add item for "scraper" importers Eric Wong
@ 2019-02-05 18:47 ` Konstantin Ryabitsev
  0 siblings, 0 replies; 2+ messages in thread
From: Konstantin Ryabitsev @ 2019-02-05 18:47 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

On Tue, 5 Feb 2019 at 16:32, Eric Wong <e@80x24.org> wrote:
> The git-users mailing list is on Google Groups with obfuscated
> addresses and censored archives.  We should allow users to
> import them soon, as obfuscated/censored archives are better
> than not having archives at all when Google decides to shut down
> yet-another-service.

I've written an importer script (in Python) for our purposes:
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/list-archive-maker.py

It doesn't do Google at this time, but it can handle Mailman and
grabbing thing from Gmane NNTP.
Perhaps it'll come in handy to identify some of the hoops I've already
had to jump through.

-K

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-05 11:32 [PATCH] TODO: add item for "scraper" importers Eric Wong
2019-02-05 18:47 ` Konstantin Ryabitsev

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox