user/dev discussion of public-inbox itself
 help / color / Atom feed
* [PATCH] doc: update HACKING and TODO with a few items
@ 2019-09-21  0:06 Eric Wong
  2019-09-21 22:40 ` libgmime experiences? (was: [PATCH] doc: update HACKING and TODO ...) Eric Wong
  0 siblings, 1 reply; 2+ messages in thread
From: Eric Wong @ 2019-09-21  0:06 UTC (permalink / raw)
  To: meta

Inline::C seems alright, so we might use it more since it still
allows end users to quickly make changes.  Our performance on
rotational disks is also terrible, and could be improved...
---
 HACKING | 3 ++-
 TODO    | 8 +++-----
 2 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/HACKING b/HACKING
index c2f4825c..f7cb92e9 100644
--- a/HACKING
+++ b/HACKING
@@ -40,7 +40,8 @@ the shiny new.
 Avoid relying on compiled modules too much.  Even if it is Free,
 compiled code makes packages more expensive to audit, build,
 distribute and verify.  public-inbox itself will only be implemented
-in scripting languages (currently Perl 5).
+in scripting languages (currently Perl 5) and optional JIT-compiled C
+(via Inline::C)
 
 Performance should be reasonably good for server administrators, too,
 and we will sacrifice features to achieve predictable performance.
diff --git a/TODO b/TODO
index 9f58342f..8f78d87d 100644
--- a/TODO
+++ b/TODO
@@ -5,7 +5,7 @@ performance, ease-of-setup, installation, maintainability, etc
 all need to be considered for everything we introduce)
 
 * general performance improvements, but without relying on
-  XS or compiled code any more than we currently do.
+  XS or pre-built modules any more than we currently do.
 
 * mailmap support (same as git) for remapping expired email addresses
 
@@ -101,10 +101,8 @@ all need to be considered for everything we introduce)
 * imperfect scraper importers for obfuscated list archives
   (e.g. obfuscated Mailman stuff, Google Groups, etc...)
 
-* support hooks, since low-level git-fast-import does not run them
-  https://public-inbox.org/meta/20190405174329.GA21472@chatter.qube.local/
-  (note: may not be needed since we do grokmirror manifest.js.gz, now)
-
 * consider using HTTP::Date instead of Date::Parse, since we need the
   former is capable of parsing RFC822-ish dates, used by Plack, and
   the latter is missing from OpenBSD and maybe other distros.
+
+* improve performance and avoid head-of-line blocking on slow storage
-- 
EW


^ permalink raw reply	[flat|nested] 2+ messages in thread

* libgmime experiences? (was: [PATCH] doc: update HACKING and TODO ...)
  2019-09-21  0:06 [PATCH] doc: update HACKING and TODO with a few items Eric Wong
@ 2019-09-21 22:40 ` Eric Wong
  0 siblings, 0 replies; 2+ messages in thread
From: Eric Wong @ 2019-09-21 22:40 UTC (permalink / raw)
  To: meta

Eric Wong <e@80x24.org> wrote:
> --- a/HACKING
> +++ b/HACKING
> @@ -40,7 +40,8 @@ the shiny new.
>  Avoid relying on compiled modules too much.  Even if it is Free,
>  compiled code makes packages more expensive to audit, build,
>  distribute and verify.  public-inbox itself will only be implemented
> -in scripting languages (currently Perl 5).
> +in scripting languages (currently Perl 5) and optional JIT-compiled C
> +(via Inline::C)

I'm considering adding libgmime support after the 1.2 release
and creating an Email::MIME/msg_iter-compatible wrapper based
on it.

I've never used gmime as a user or hacker...

Pure-Perl Email::MIME will remain supported, though new versions
of Email::MIME use Email::Address::XS so there's already a
C/XS component which increases installation overhead.

My thoughts on gmime so far:

One big reason gmime looks good to me is that it offers mmap and
pipe APIs, so it hopefully deals well with big messages and
won't bloat/fragment the heap.

Another is Email::Simple::Header::_header_to_list being at the
top of some profiles, and I'm not convinced I could make it
faster.

gmime's been through several incompatible API/ABI versions, but
it seems Debian can support at least 2.6 and 3.0.  FreeBSD has
2.2, 2.4, 2.6 and 3.0, even!

gmime upstream also provides a PORTING document which should come
in handy.

The MIME::Fast wrapper for libgmime exists on CPAN, but that
hasn't been updated in years and isn't in Debian.  Both 3.0 and
2.6 APIs could be supported in parallel using Inline::C, and
more can be added in the future.

notmuch uses gmime, as does a lot of desktop software.

My experience with mainstream "desktop software" hasn't been
good, as they tends to be less stable in both user/programming
interfaces, and also more complex and difficult to debug than
server-oriented software.

However, looking at the Debian changelogs, there's only been one
CVE (CVE-2010-0409) from nearly a decade ago.  Maybe all the
bugs are fixed, or nobody really abuses it.

In any case, if I go through with this, I'll be running lots of
mail and random junk through it to ensure it doesn't break
unexpectedly.

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-21  0:06 [PATCH] doc: update HACKING and TODO with a few items Eric Wong
2019-09-21 22:40 ` libgmime experiences? (was: [PATCH] doc: update HACKING and TODO ...) Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox