user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* Re: [PATCH] TODO: add item for "scraper" importers
  2019-02-05 11:32  7% [PATCH] TODO: add item for "scraper" importers Eric Wong
@ 2019-02-05 18:47  7% ` Konstantin Ryabitsev
  0 siblings, 0 replies; 2+ results
From: Konstantin Ryabitsev @ 2019-02-05 18:47 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

On Tue, 5 Feb 2019 at 16:32, Eric Wong <e@80x24.org> wrote:
> The git-users mailing list is on Google Groups with obfuscated
> addresses and censored archives.  We should allow users to
> import them soon, as obfuscated/censored archives are better
> than not having archives at all when Google decides to shut down
> yet-another-service.

I've written an importer script (in Python) for our purposes:
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/list-archive-maker.py

It doesn't do Google at this time, but it can handle Mailman and
grabbing thing from Gmane NNTP.
Perhaps it'll come in handy to identify some of the hoops I've already
had to jump through.

-K

^ permalink raw reply	[relevance 7%]

* [PATCH] TODO: add item for "scraper" importers
@ 2019-02-05 11:32  7% Eric Wong
  2019-02-05 18:47  7% ` Konstantin Ryabitsev
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-02-05 11:32 UTC (permalink / raw)
  To: meta

The git-users mailing list is on Google Groups with obfuscated
addresses and censored archives.  We should allow users to
import them soon, as obfuscated/censored archives are better
than not having archives at all when Google decides to shut down
yet-another-service.

There's also some mangling that (most) instances of Mailman do
(e.g. cgit), but being able to follow such groups over NNTP
and use our search functionality is still useful and better
than what typical Mailman installations provide.
---
 TODO | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/TODO b/TODO
index 66e5063..ef77d1c 100644
--- a/TODO
+++ b/TODO
@@ -95,3 +95,6 @@ all need to be considered for everything we introduce)
 * code repository integration (with cgit, gitweb, etc...)
 
 * migration path to v2 without breaking v1 "git fetch" cronjobs
+
+* imperfect scraper importers for obfuscated list archives
+  (e.g. obfuscated Mailman stuff, Google Groups, etc...)
-- 
EW


^ permalink raw reply related	[relevance 7%]

Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-02-05 11:32  7% [PATCH] TODO: add item for "scraper" importers Eric Wong
2019-02-05 18:47  7% ` Konstantin Ryabitsev

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).