user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* WIP: searching all of lore
@ 2020-11-26 19:45 Eric Wong
  2020-11-28 22:34 ` Eric Wong
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Eric Wong @ 2020-11-26 19:45 UTC (permalink / raw)
  To: workflows; +Cc: meta

Requires Tor, for now:


It seems v3 onions with longer URLs are more secure, these days;
but requires newer Tor.

On Debian or RH-based systems, it's as easy as:
	<apt|yum> install tor torsocks

	# assuming w3m is installed:
	torsocks w3m http://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion/all/

Other browsers can configure SOCKS5 proxies to
or be wrapped via torsocks (which uses LD_PRELOAD)


  I don't know much about Tor security (or security in general).
  I see Tor as an alternative to paying corrupt organizations
  (ICANN) and lets me self-host without a static IP address.

  I've also had numerous ISP and power outages this year,
  probably because more neighbors are home due to the pandemic,
  so don't expect 99.999% uptime, either.  And I'm a klutz who
  always trips over cables on (relatively) good days, and I've
  only had bad days since March :<

How to replicate

	# I'm using the following to update mirror and
	# (see grokmirror docs) for more.  Old command:
	grok-pull -v -c repos.conf
	public-inbox-index --all # update per-inbox indices

	# add "-L basic" or "-L medium" to reduce space requirements
	# to either -extindex or -index commands

	# The new command, not finalized yet:
	public-inbox-extindex --all -v /path/to/ALL

The following changes in a otherwise boring ~/.public-inbox/config
(or whatever $PI_CONFIG is set to)

; not yet stable or finalized, yet:
; this section allows, "all" is a special case, currently
[extindex "all"]
	topdir = /path/to/ALL

; these are already documented in public-inbox-config(5)
	; 'all' ignores domain name matching,
	; useful for inboxes served via multiple domains
	wwwlisting = all
	grokManifest = all

	; users with larger machines may want to bump this,
	; the default is for machines w/ 256MB RAM
	; (which I still use, sometimes)
	indexBatchSize = 100m

# I'm using the following to update from
# (see grokmirror docs) for more
grok-pull -v -c repos.conf
public-inbox-index --all # update per-inbox indices
public-inbox-extindex --all -v /path/to/ALL # index [extindex "all"]

This is running commit 95cb3e48fc5c4e847cdc111c2c8c9f0b70bdea56
git clone

More changes coming (JMAP, speedups), and there's probably still
lots of stuff broken and need fixing (including my brain :<)

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2021-03-17 18:37 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-26 19:45 WIP: searching all of lore Eric Wong
2020-11-28 22:34 ` Eric Wong
2020-12-05 20:07   ` Eric Wong
2020-12-08 14:01     ` Konstantin Ryabitsev
2020-12-08 18:02       ` Eric Wong
2020-12-08 18:11         ` Konstantin Ryabitsev
2020-12-01 14:00 ` Konstantin Ryabitsev
2020-12-01 18:48   ` Eric Wong
2021-03-17  7:11 ` Eric Wong
2021-03-17 13:27   ` Konstantin Ryabitsev
2021-03-17 18:18     ` Eric Wong
2021-03-17 18:37       ` Konstantin Ryabitsev

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ \
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
 note: .onion URLs require Tor:

code repositories for project(s) associated with this inbox:

AGPL code for this site: git clone