about summary refs log tree commit homepage
path: root/TODO
DateCommit message (Collapse)
2023-10-17doc: fix some typos and grammar
[ew: leave install/README unchanged pending wording changes]
2023-08-28Fix some typos/grammar/errors in docs and comments
2023-02-17TODO: handle more cases of unencoded slashes
Nowadays, mutt defaults to Message-IDs with `/' in them :<
2022-09-02TODO: add item for RFC 3676 (format=flowed + DelSP)
I noticed some messages looked wonky in HTML (via w3m) compared to mutt; and neither Email::MIME, Mail::Box nor gmime seem to handle this standard... However, mflow (part of mblaze) seems to be good standalone C implementation that we can check a Perl5 implementation against.
2022-08-28TODO: add some Xapian-related stuff
Just to more clearly spell out what needs to be done on the search side.
2022-08-04TODO: remove done items, adjust/add/abandon some
public-inbox-pop3d (and -netd) gives us POP3 support, and it seems to work. Proxy support can come independently, probably after JMAP. public-inbox-netd provides the multi-protocol "super server" which allows code memory savings. Work is ongoing to further reduce memory use... Automatically updating on TLS cert and key changes on inotify/EVFILT_VNODE won't be done, since (IMHO) there's too much risk of inadvertent updates on incomplete changes. My same train-of-thought applies to auto-reloading on config file changes: an admin may save a file halfway through a multi-step change and auto-reloading can be too surprising and break things. I don't think lei+FUSE will be as portable or useful as a local IMAP server (and maybe JMAP, eventually); but r/w IMAP support would be nice.. Finally, git SHA-256 repo support will need to be taken into account.
2022-06-21search: do not index base-85 binary patches
Base-85 binary patches generated by git lead to many false positives, so skip over gibberish words which may occur in them. To avoid regressions in search results, continue to allow searching for exact size matches (via "literal $SIZE") and the phrase "GIT binary patch" for the mere presence of a binary patch.
2022-06-21search: support "patchid:" prefix (git patch-id --stable)
This allows easy searching via patch-id from a git commit. Currently, abbreviations are not supported, and it seems needless to support them since AFAIK (git) doesn't generate nor resolve abbreviated patch-ids anywhere.
2022-04-01TODO: add item for auto-detecting TLS files in daemons
I forgot to restart my -imapd and -nntpd instances on public-inbox.org after the cert expired :x
2021-09-15TODO: update some items
Some things (extindex) done, many more to do; and -fetch should set us up for v1 => v2 migrations at some point...
2021-05-30TODO: add pipelining note
Network latency is painful to deal with.
2021-03-13TODO: update tests item
Our code base is growing, and we could repeat less setup work in our tests and improve development pace.
2021-03-11doc: update 1.7 release notes, tuning, TODO
Some stuff done, some stuff still needs doing.
2021-01-31TODO: add item for dealing with base-85 binary patches
False-positives from search results are no fun
2020-09-20doc: post-1.6 updates, start 1.7
I should've dropped "PENDING" notes before the 1.6 release; they're dropped now, and a note is added to remind my future self to drop them before 1.7.
2020-09-14doc: TODO and release notes updates ahead of 1.6
Some more things have happened... And drop some items which are too expensive to support, such as automatic mirroring.
2020-06-13doc: update TODO and WIP 1.6.0 release notes
Lots of big changes coming Thanks to The Linux Foundation for sponsoring me to hack on this in 2020 :)
2020-05-10various doc updates ahead of 1.5.0
2020-02-23doc: improve wording of "inbox" vs "repository"
Since v2 inboxes contain multiple git repositories, avoid the use of the word "repository" when referring to inboxes as a whole in most places.
2020-02-08doc: mark some TODO items as done
NNTP TLS and COMPRESS support and cgit spawning from the WWW interface were implemented last year. Given the lack of syscall number stability guarantee on the OpenBSD and FreeBSD, I don't think supporting a pure-Perl kevent is feasible. Inline::C may still be an option since IO::KQueue is abandoned, though, as it is for some Linux-only syscalls and maybe some POSIX ones not covered by POSIX.pm.
2020-01-25s/news.gmane.org/news.gmane.io/
gmane still has a NNTP server, so update links to point to it. cf. https://lars.ingebrigtsen.no/2020/01/06/whatever-happened-to-news-gmane-org/
2020-01-25spelling: favor `publicly' over `publically'
While both can be correct, the former seems more common, is shorter, and is also consistent with the spelling found in the AGPL-3.0 text.
2019-12-18TODO: add UUCP address item
We should support historical archives from the old days, but I'm not sure how to best go about it, for now, given how tricky correct handling of modern email addresses is. We can deal with it if/when somebody decides to import some ancient archives...
2019-12-12Date::Parse is now optional
-mda should not be dealing with broken Date: headers nowadays, and deprioritize it in our documentation and internal checks.
2019-12-11TODO: update and add a few more items
SpamAssassin has used re2c (via sa-compile) for many years, now, and it seems to work fine, there. GMime also looks promising when combined with Inline::C since GMime can operate on mmap-ed regions. Given the inevitable demise of many .orgs when price rise; supporting a URL rewriter similar to .mailmap makes sense. And HTTP CONNECT seems like something our -httpd can support to let firewalled users read over NNTP.
2019-11-03TODO: update item for multiple Date: headers
That's the only head-scratcher of the bunch remaining, since that relies on ranges.
2019-10-23Merge branch 'regen'
* regen: v2writable: use msgmap as multi_mid queue v2writable: move git->cleanup to the correct place v2writable: reindex handles 3-headered monsters v2writable: improve "num_for" API and disambiguate v2writable: set unindexed article number
2019-10-21v2writable: reindex handles 3-headered monsters
And maybe 8-headered ones, too... I noticed --reindex failing on the linux-renesas-soc mirror due one 3-headed monster of a message having 3 sets of headers; while another normal message had a Message-ID that matched one of the 3 IDs of the 3-headed monster. We still try to do the majority of indexing backwards, but we defer indexing multi-Message-ID'd messages until the end to ensure we get all the "good" messages in before we process the multi-headered ones. Link: https://public-inbox.org/meta/20191016211415.GA6084@dcvr/
2019-10-21TODO: DHT (distributed hash table) for Message-IDs
This would encourage SPOF avoidance. It would also make it easier to do "Certificate Transparency"-style attestation of messages.
2019-10-17TODO: add a few more items around coderepos
Going where no email client/tool has gone before...
2019-10-15TODO: add an item for Python pygments
Pygments seems to be a popular highlighter and widely available, so we'll be providing support for that at some point... Link: https://public-inbox.org/meta/20190926131836.GB10467@chatter.i7.local/ Link: https://public-inbox.org/meta/874l0zt7sd.fsf@alyssa.is/
2019-10-14TODO: add item for config linter and grapher
It'll be useful for tools and users to test and perhaps visualize configs before reloading -httpd/-nntpd/-watch.
2019-10-01TODO: add item for searching based on git-patch-id(1)
I forgot about this feature when I was implementing blob-ID-based searches :x
2019-10-01TODO: update with "git cat-file" items
Millions of inboxes in an instance is probably not feasible, but dozens or even hundreds could happen and /proc/sys/fs/pipe-user-pages-soft is only 16384 on my system, with each "cat-file --batch" process using 16+1 pages worth of pipes.
2019-09-22doc: update HACKING and TODO with a few items
Inline::C seems alright, so we might use it more since it still allows end users to quickly make changes. Our performance on rotational disks is also terrible, and could be improved...
2019-07-13TODO: remove done items, add some more
It never ends...
2019-07-06nntp: support COMPRESS DEFLATE per RFC 8054
This is only tested so far with my patches to Net::NNTP at: https://rt.cpan.org/Ticket/Display.html?id=129967 Memory use in C10K situations is disappointing, but that's the nature of compression. gzip compression over HTTPS does have the advantage of not keeping zlib streams open when clients are idle, at the cost of worse compression.
2019-05-31TODO: add item for optional Cache::FastMmap
Taking one step out of setting up a performant deployment could make setup and administration easier (at the cost of installing an extra-but-common XS module). This can also be useful for the day NNTP servers see hug-of-death events.
2019-05-24TODO: more stuff: bundles, synonyms, dogfooding
git bundles could/should make self-hosting easier. Being able to configure synonym (and spelling) lists would make some searches more useful. Might as well dogfood kernel stuff, too, given the overlap and history between this project, git and the Linux kernel. Would be interesting to have *BSD folks throw their hat in the ring, too. Building/testing userspace stuff is often the most time-consuming, but necessary to ensure future compatibility.
2019-05-15lazy load Xapian and make it optional for v2
More tests work without Search::Xapian, now. Usability issues still need to be fixed
2019-05-05DS: workaround IO::Kqueue EINTR (mis-)handling
IO::Kqueue seems unmaintained, so workaround a long-standing bug where it falls over on signals: https://rt.cpan.org/Ticket/Display.html?id=116615
2019-05-05listener: use EPOLLEXCLUSIVE for listen sockets
Since our listen sockets are non-blocking and we may run multiple httpd|nntpd processes; we need a way to avoid thundering herds when there are multiple httpd|nntpd worker processes. EPOLLEXCLUSIVE was added just for that in Linux 4.5
2019-05-04bundle Danga::Socket and Sys::Syscall
These modules are unmaintained upstream at the moment, but I'll be able to help with the intended maintainer once/if CPAN ownership is transferred. OTOH, we've been waiting for that transfer for several years, now... Changes I intend to make: * EPOLLEXCLUSIVE for Linux * remove unused fields wasting memory * kqueue bugfixes e.g. https://rt.cpan.org/Ticket/Display.html?id=116615 * accept4 support And some lower priority experiments: * switch to EV_ONESHOT / EPOLLONESHOT (incompatible changes) * nginx-style buffering to tmpfile instead of string array * sendfile off tmpfile buffers * io_uring maybe?
2019-04-17TODO: add an item for hooks support
and add a note for grokmirror support/integration, too
2019-02-05TODO: add item for "scraper" importers
The git-users mailing list is on Google Groups with obfuscated addresses and censored archives. We should allow users to import them soon, as obfuscated/censored archives are better than not having archives at all when Google decides to shut down yet-another-service. There's also some mangling that (most) instances of Mailman do (e.g. cgit), but being able to follow such groups over NNTP and use our search functionality is still useful and better than what typical Mailman installations provide.
2019-02-04TODO: support migration to v2 without breaking v1 "git fetch"
I'd like to move https://public-inbox.org/git/ to v2; but cronjobs using "git fetch" to following should not break.
2019-01-31doc: remove completed TODO items
2019-01-20$INBOX/_/text/color/ and sample user-side CSS
Since we now support more CSS classes for coloring, give this feature more visibility.
2019-01-01TODO: avoid mentioning untrustworthy browser extensions
Old and new versions of Mozilla-based browsers seem to support userContent.css just fine. cf. https://www-archive.mozilla.org/unix/customizing.html#usercss http://kb.mozillazine.org/index.php?title=UserContent.css
2019-01-01TODO: support integration with cgit/gitweb/etc...
We support searching on blob identifiers for a reason :>