robots.txt ignored (was: public-inbox.org VPS hopefully stable, now...)

[Eric Wong <e@80x24.org>]

Eric Wong <e@80x24.org> wrote:
> I've got a lot of orphaned sockets and OOM from the kernel the
> past few days.

No longer a problem, at least.   More rambling thoughts below...

> It's a combination of kernel TCP memory use,
> OpenSSL, zlib, glibc malloc, Perl 5, and probably other things...

Yeah...

> It looks like a lot of bot traffic trying to scrape IMAP(S),
> too :<
> 
> WolfSSL might be an option via Inline::C *shrug*
> 
> I've cut down on connections and via iptables/ip6tables
> connlimit and state modules; still not sure where they
> should be atm..

Per-IP limits don't seem effective because bots are not
reusing connections, changing IPs, and changing User-Agents.
Throttling IPv4 /24 blocks seems OK, I haven't gone to /16, yet.

IPv6 doesn't seem to be a problem at the moment.

HTTPS termination is still provided by a horribly-named Ruby
webserver (no nginx/haproxy/etc.), and some janky Ruby userspace
throttles were added last week for non-(curl|w3m|git|lynx)
User-Agents.

The chain is currently like this:

                        __/--> varnish -> public-inbox-netd
        bots -> ruby --|__
                          \--> ssh(*) -> varnish -> public-inbox-netd

I might expand the public-inbox HTTP server code to provide
reverse proxying to Varnish and get rid of ruby(**):

                        __/--> varnish -> public-inbox-netd
        bots -> perl --|__
                          \--> ssh(*) -> varnish -> public-inbox-netd

My small Varnish caches get poisoned from scanning, too.  While
Varnish is great for dealing with traffic bursts from popular sites,
it's a waste for crawlers which rarely/never repeat requests.

The "limiter" feature (see public-inbox-config(5)) could be
expanded to handle some endpoints such as /T/, /t/, /t.mbox.gz
to reduce excessive parallelism while still providing enough to
keep git processes saturated.  Basically the same thing I did
with /s/ which seems quite effective:
8d6a50ff (www: use a dedicated limiter for blob solver, 2024-03-11)
2368fb20 (viewvcs: -codeblob limiter w/ depth for solver, 2025-02-08)

OTOH, limiter currently doesn't check IP ranges.  Perhaps it could?

limiter is great for reducing memory use from zlib buffers and
would be good for limiting the the thread-skeleton structure, as
well.

> I "only" have 1GB of RAM since it's the cheapest available
> (32-bit userspace, x86_64 kernel).  Getting more RAM or CPU
> is absolutely NOT an option; optimizing data structures,
> code and tweaking knobs are the only ways to fix this.
> 
> Down with consumerism!

Unchanged :>  And I forgot to mention it's all happening on a
single core, even.

Of course, JavaScript or MITM-based services are not an option
due to accessibility.


(*) the yhbt.net/lore/ mirror is behind the SSH tunnel, I know
    I could move varnish in front of the SSH tunnel to reduce
    SSH traffic but I'm more familiar with using Ruby||Perl to
    route than VCL.

(**) deterministic DESTROY from Perl would've been so useful to
     have in the janky Ruby throttler I made