[RFC suggestion] Generate manpage directly with Asciidoctor

[RFC suggestion] Generate manpage directly with Asciidoctor

[Bagas Sanjaya]

Hi,

Asciidoctor has support for directly generating manpage, see [1].

We support using Asciidoctor as drop-in replacement for original
Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
to produce manpages. However, most users don't inclined to install
xmlto toolchain, partly because they had to download more than 300 MB
of data just to install xmlto and its dependencies (including dblatex
and texlive).

So completely migrating to Asciidoctor can eliminate xmlto requirement
for generating manpage.

What do you think about above?

[1]: https://docs.asciidoctor.org/asciidoctor/latest/manpage-backend/
-- 
An old man doll... just what I always wanted! - Clara

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

[Randall S. Becker]

On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
>To: Git Users <git@vger.kernel.org>
>Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
>Asciidoctor has support for directly generating manpage, see [1].
>
>We support using Asciidoctor as drop-in replacement for original Asciidoc, but
>currently we need to use xmlto together with Asciidoc(tor) to produce
>manpages. However, most users don't inclined to install xmlto toolchain, partly
>because they had to download more than 300 MB of data just to install xmlto
>and its dependencies (including dblatex and texlive).
>
>So completely migrating to Asciidoctor can eliminate xmlto requirement for
>generating manpage.
>
>What do you think about above?

Our toolchain does not support asciidoctor itself because of porting issues. I am not sure it is available everywhere.

-Randall

-- Brief whoami:
NonStop developer since approximately 211288444200000000
UNIX developer since approximately 421664400
-- In my real life, I talk too much.

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Đoàn Trần Công Danh]

On 2021-05-07 13:06:31+0700, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> Hi,
> 
> Asciidoctor has support for directly generating manpage, see [1].

While I'm impressed with this news.

> We support using Asciidoctor as drop-in replacement for original
> Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
> to produce manpages. However, most users don't inclined to install
> xmlto toolchain, partly because they had to download more than 300 MB
> of data just to install xmlto and its dependencies (including dblatex
> and texlive).

I'm pretty sure xmlto doesn't depend on dblatex and/or texlive.

I'm building Git and Git's documentation on different Linux boxes with
asciidoc, most of them don't have dblatex and/or texlive.
In fact, I only have texlive installed in my laptop.

I guess you're using Debian or one of its derived distribution?
Debian puts dblatex as xmlto's rec. [2]

I think you can use:

	apt install --no-install-recommends xmlto

to avoid those recommendations.

> So completely migrating to Asciidoctor can eliminate xmlto requirement
> for generating manpage.
> 
> What do you think about above?

Hm, I'm pretty sure Ruby (asciidoctor's language) is very hard to port.
Last I heard, it's buggy on some platforms.

I think we're better to do like this:

* If we're using asciidoc, we will use xmlto
* If we're using asciidoctor, we will generate man-pages with
  asciidoctor directly

> [1]: https://docs.asciidoctor.org/asciidoctor/latest/manpage-backend/

[2]: https://packages.debian.org/sid/xmlto

-- 
Danh

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Bagas Sanjaya]

On 07/05/21 19.27, Đoàn Trần Công Danh wrote:
> I'm pretty sure xmlto doesn't depend on dblatex and/or texlive.
> 
> I'm building Git and Git's documentation on different Linux boxes with
> asciidoc, most of them don't have dblatex and/or texlive.
> In fact, I only have texlive installed in my laptop.
> 
> I guess you're using Debian or one of its derived distribution?
> Debian puts dblatex as xmlto's rec. [2]
> 
> I think you can use:
> 
> 	apt install --no-install-recommends xmlto
> 
> to avoid those recommendations.
>

Yes, I'm using Ubuntu (derived from Debian).

> Hm, I'm pretty sure Ruby (asciidoctor's language) is very hard to port.
> Last I heard, it's buggy on some platforms.
> 
> I think we're better to do like this:
> 
> * If we're using asciidoc, we will use xmlto
> * If we're using asciidoctor, we will generate man-pages with
>    asciidoctor directly
> 

Nice for your suggestion.

But will the resulting manpage from asciidoc+xmlto and from asciidoctor
syntatically and semantically same?

-- 
An old man doll... just what I always wanted! - Clara

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

On 2021-05-07 at 06:06:31, Bagas Sanjaya wrote:
> Hi,
> 
> Asciidoctor has support for directly generating manpage, see [1].
> 
> We support using Asciidoctor as drop-in replacement for original
> Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
> to produce manpages. However, most users don't inclined to install
> xmlto toolchain, partly because they had to download more than 300 MB
> of data just to install xmlto and its dependencies (including dblatex
> and texlive).

If you're installing on Debian, you want to install with
--no-install-recommends to avoid the dependency on dblatex or fop,
libpaper-utils, and zip.  That should avoid the large downloads.

> So completely migrating to Asciidoctor can eliminate xmlto requirement
> for generating manpage.
> 
> What do you think about above?

I didn't do this because it makes things very complicated in the
Makefile and my goal was to minimize the needed changes and divergence.
I'm not opposed to someone else doing it, but I expect it will be a
bunch of work.

It also doesn't help if you're generating more than just the manual
pages, since generating the PDFs will definitely require xmlto and
either dblatex or fop.  I personally prefer fop over dblatex, but folks
have different opinions.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Fri, May 07, 2021 at 08:25:36PM +0000, brian m. carlson wrote:

> > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > for generating manpage.
> > 
> > What do you think about above?
> 
> I didn't do this because it makes things very complicated in the
> Makefile and my goal was to minimize the needed changes and divergence.
> I'm not opposed to someone else doing it, but I expect it will be a
> bunch of work.

Yeah. In the period where we support both asciidoc and asciidoctor,
using the same flow for both (even if it's suboptimal for asciidoctor)
kept the complexity down.

I _thought_ the original asciidoc was marked as deprecated /
unmaintained at some point. But it does seem to have gotten a few
releases in the last year (it looks like maybe the python 2 version was
EOL, but somebody decided to make the effort to port it to python 3?).

But I wouldn't be at all sad to just standardize on asciidoctor. I think
we're at parity in terms of the output (thanks to lots of work from you
and Martin over the past couple of years), and I've generally found it
nicer to work with.

The only downside is that it may be available in fewer places (though
I'd think that python vs ruby is not so different). IMHO it's OK to be
aggressive about the doc toolchain requirements, because the fallback is
always grabbing the preformatted roff or HTML pages that were generated
on a different system.

-Peff

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Bagas Sanjaya wrote:
> Asciidoctor has support for directly generating manpage, see [1].
> 
> We support using Asciidoctor as drop-in replacement for original
> Asciidoc, but currently we need to use xmlto together with Asciidoc(tor)
> to produce manpages. However, most users don't inclined to install
> xmlto toolchain, partly because they had to download more than 300 MB
> of data just to install xmlto and its dependencies (including dblatex
> and texlive).
> 
> So completely migrating to Asciidoctor can eliminate xmlto requirement
> for generating manpage.
> 
> What do you think about above?

Makes complete sense. If only we had some patches to go along with that
proposal.

-- 
Felipe Contreras

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Randall S. Becker wrote:
> On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
> >To: Git Users <git@vger.kernel.org>
> >Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
> >What do you think about above?
> 
> Our toolchain does not support asciidoctor itself because of porting
> issues. I am not sure it is available everywhere.

Then don't use it.

If I understand correctly the proposal affects people that build with
USE_ASCIIDOCTOR=YesPlease only.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

On 2021-05-07 at 12:02:13, Randall S. Becker wrote:
> On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
> >To: Git Users <git@vger.kernel.org>
> >Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
> >Asciidoctor has support for directly generating manpage, see [1].
> >
> >We support using Asciidoctor as drop-in replacement for original Asciidoc, but
> >currently we need to use xmlto together with Asciidoc(tor) to produce
> >manpages. However, most users don't inclined to install xmlto toolchain, partly
> >because they had to download more than 300 MB of data just to install xmlto
> >and its dependencies (including dblatex and texlive).
> >
> >So completely migrating to Asciidoctor can eliminate xmlto requirement for
> >generating manpage.
> >
> >What do you think about above?
> 
> Our toolchain does not support asciidoctor itself because of porting issues. I am not sure it is available everywhere.

I think Asciidoctor is pure Ruby, since it also uses JRuby and Opal to
run in Java and JavaScript environments.  Ruby is relatively portable to
most architectures and systems (for example, it runs on 16 Debian
architectures).

Is the problem in your case Ruby or is it Asciidoctor itself?  I'm happy
to send portability patches to Asciidoctor if necessary, but I'm not
sure I'm a good candidate to send patches to Ruby itself, especially for
an OS I don't use.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Đoàn Trần Công Danh wrote:
> On 2021-05-07 13:06:31+0700, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > for generating manpage.
> > 
> > What do you think about above?
> 
> Hm, I'm pretty sure Ruby (asciidoctor's language) is very hard to port.
> Last I heard, it's buggy on some platforms.

And I'm pretty sure it isn't.

Ruby is a mainstream language. At some point it was #8 in the TIOBE
index, above PHP and Perl. Right now it's #11.

A lot of effort has been put to make Ruby work on all platforms.

> I think we're better to do like this:
> 
> * If we're using asciidoc, we will use xmlto
> * If we're using asciidoctor, we will generate man-pages with
>   asciidoctor directly

This certainly is the way to go.

I'm not sure if Bagas suggested to make USE_ASCIIDOCTOR=YesPlease the
default, but first we would need to make sure the output without xmlto
is desirable, which we don't at the moment.

So, even if we were to consider this, *first* the option to use pure
asciidoctor needs to be in place.

Cheers.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

brian m. carlson wrote:
> On 2021-05-07 at 06:06:31, Bagas Sanjaya wrote:
> > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > for generating manpage.
> > 
> > What do you think about above?
> 
> I didn't do this because it makes things very complicated in the
> Makefile and my goal was to minimize the needed changes and divergence.
> I'm not opposed to someone else doing it, but I expect it will be a
> bunch of work.

I don't think it would be that complicated.

This patch seems to do the trick here:

diff --git a/Documentation/Makefile b/Documentation/Makefile
index c2baad0bd8..8fa4896d22 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -189,6 +189,7 @@ ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
 DBLATEX_COMMON =
 XMLTO_EXTRA += --skip-validation
 XMLTO_EXTRA += -x manpage.xsl
+TXT_TO_MAN = $(ASCIIDOC_COMMON) -b manpage
 endif
 
 SHELL_PATH ?= $(SHELL)
@@ -360,9 +361,16 @@ $(OBSOLETE_HTML): %.html : %.txto asciidoc.conf asciidoctor-extensions.rb GIT-AS
 manpage-base-url.xsl: manpage-base-url.xsl.in
 	$(QUIET_GEN)sed "s|@@MAN_BASE_URL@@|$(MAN_BASE_URL)|" $< > $@
 
+ifdef TXT_TO_MAN
+%.1 %.5 %.7 : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
+	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
+	$(TXT_TO_MAN) -o $@+ $< && \
+	mv $@+ $@
+else
 %.1 %.5 %.7 : %.xml manpage-base-url.xsl $(wildcard manpage*.xsl)
 	$(QUIET_XMLTO)$(RM) $@ && \
 	$(XMLTO) -m $(MANPAGE_XSL) $(XMLTO_EXTRA) man $<
+endif
 
 %.xml : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
 	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \

I'm sure this can be improved with some knowledge about intermediary
rules.

> It also doesn't help if you're generating more than just the manual
> pages, since generating the PDFs will definitely require xmlto and
> either dblatex or fop.  I personally prefer fop over dblatex, but folks
> have different opinions.

Huh? user-manual.pdf seems to require DBLATEX, not XMLTO.

Cheers.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

On 2021-05-07 at 23:35:21, Felipe Contreras wrote:
> I don't think it would be that complicated.
> 
> This patch seems to do the trick here:
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index c2baad0bd8..8fa4896d22 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -189,6 +189,7 @@ ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
>  DBLATEX_COMMON =
>  XMLTO_EXTRA += --skip-validation
>  XMLTO_EXTRA += -x manpage.xsl
> +TXT_TO_MAN = $(ASCIIDOC_COMMON) -b manpage
>  endif
>  
>  SHELL_PATH ?= $(SHELL)
> @@ -360,9 +361,16 @@ $(OBSOLETE_HTML): %.html : %.txto asciidoc.conf asciidoctor-extensions.rb GIT-AS
>  manpage-base-url.xsl: manpage-base-url.xsl.in
>  	$(QUIET_GEN)sed "s|@@MAN_BASE_URL@@|$(MAN_BASE_URL)|" $< > $@
>  
> +ifdef TXT_TO_MAN
> +%.1 %.5 %.7 : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
> +	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> +	$(TXT_TO_MAN) -o $@+ $< && \
> +	mv $@+ $@
> +else
>  %.1 %.5 %.7 : %.xml manpage-base-url.xsl $(wildcard manpage*.xsl)
>  	$(QUIET_XMLTO)$(RM) $@ && \
>  	$(XMLTO) -m $(MANPAGE_XSL) $(XMLTO_EXTRA) man $<
> +endif
>  
>  %.xml : %.txt asciidoc.conf asciidoctor-extensions.rb GIT-ASCIIDOCFLAGS
>  	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> 
> I'm sure this can be improved with some knowledge about intermediary
> rules.

It's not quite that simple.  It requires changes to
asciidoctor-extensions.rb and we also need to consider
NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
inconvenient, since according to the Makefile, it's impossible to write
man output that works across troff versions.

We will also want to preserve the ability to use xmlto because we
currently have doc-diff which helps us find divergences in the behavior
of AsciiDoc and Asciidoctor, many of which affect all formats, and that
will be completely useless if we use two different rendering toolchains.

So, yes, this is a good start, but it's not quite as trivial as it seems.

> > It also doesn't help if you're generating more than just the manual
> > pages, since generating the PDFs will definitely require xmlto and
> > either dblatex or fop.  I personally prefer fop over dblatex, but folks
> > have different opinions.
> 
> Huh? user-manual.pdf seems to require DBLATEX, not XMLTO.

Ah, yes, because we do it manually instead of using xmlto.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

RE: [RFC suggestion] Generate manpage directly with Asciidoctor

[Randall S. Becker]

On May 7, 2021 6:58 PM, brian m. carlson wrote:
>Subject: Re: [RFC suggestion] Generate manpage directly with Asciidoctor
>
>On 2021-05-07 at 12:02:13, Randall S. Becker wrote:
>> On May 7, 2021 2:07 AM, Bagas Sanjaya wrote:
>> >To: Git Users <git@vger.kernel.org>
>> >Subject: [RFC suggestion] Generate manpage directly with Asciidoctor
>> >Asciidoctor has support for directly generating manpage, see [1].
>> >
>> >We support using Asciidoctor as drop-in replacement for original
>> >Asciidoc, but currently we need to use xmlto together with
>> >Asciidoc(tor) to produce manpages. However, most users don't inclined
>> >to install xmlto toolchain, partly because they had to download more
>> >than 300 MB of data just to install xmlto and its dependencies (including
>dblatex and texlive).
>> >
>> >So completely migrating to Asciidoctor can eliminate xmlto
>> >requirement for generating manpage.
>> >
>> >What do you think about above?
>>
>> Our toolchain does not support asciidoctor itself because of porting issues. I
>am not sure it is available everywhere.
>
>I think Asciidoctor is pure Ruby, since it also uses JRuby and Opal to run in Java
>and JavaScript environments.  Ruby is relatively portable to most architectures
>and systems (for example, it runs on 16 Debian architectures).
>
>Is the problem in your case Ruby or is it Asciidoctor itself?  I'm happy to send
>portability patches to Asciidoctor if necessary, but I'm not sure I'm a good
>candidate to send patches to Ruby itself, especially for an OS I don't use.

In this case, Ruby is not 100% portable to all platforms. I have an old port, but the latest version is not portable because of unportable dependencies - not for a lack of a large amount of trying. So the assumption is not entirely workable. My only option is to take the pre-generated pages from the manpages repo.

-Randall

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Fri, May 07, 2021 at 11:57:32PM +0000, brian m. carlson wrote:

> It's not quite that simple.  It requires changes to
> asciidoctor-extensions.rb and we also need to consider
> NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
> inconvenient, since according to the Makefile, it's impossible to write
> man output that works across troff versions.
> 
> We will also want to preserve the ability to use xmlto because we
> currently have doc-diff which helps us find divergences in the behavior
> of AsciiDoc and Asciidoctor, many of which affect all formats, and that
> will be completely useless if we use two different rendering toolchains.

Speaking of doc-diff: with that patch applied, try "./doc-diff
--asciidoctor HEAD^ HEAD", which shows quite a few of the complications
(another one seems to be that all of the linkgit macros are broken).

-Peff

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Fri, May 07, 2021 at 11:10:50PM -0400, Jeff King wrote:

> On Fri, May 07, 2021 at 11:57:32PM +0000, brian m. carlson wrote:
> 
> > It's not quite that simple.  It requires changes to
> > asciidoctor-extensions.rb and we also need to consider
> > NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
> > inconvenient, since according to the Makefile, it's impossible to write
> > man output that works across troff versions.
> > 
> > We will also want to preserve the ability to use xmlto because we
> > currently have doc-diff which helps us find divergences in the behavior
> > of AsciiDoc and Asciidoctor, many of which affect all formats, and that
> > will be completely useless if we use two different rendering toolchains.
> 
> Speaking of doc-diff: with that patch applied, try "./doc-diff
> --asciidoctor HEAD^ HEAD", which shows quite a few of the complications
> (another one seems to be that all of the linkgit macros are broken).

Hmph. Actually, this seems to give inconsistent results. There
definitely is a problem. Try:

  cd Documentation
  make USE_ASCIIDOCTOR=1 git.1
  man -l git.1 >old
  git am .../the_patch
  make USE_ASCIIDOCTOR=1 git.1
  man -l git.1 >new
  diff -u old new

which shows it. I think the patch does not specify the dependencies to
make correctly (and doc-diff reuses the worktree to avoid work). Doing
this:

  git am .../the_patch
  ./doc-diff --asciidoctor HEAD HEAD
  ./doc-diff --asciidoctor HEAD^ HEAD

works around it and shows the extent of the problem.

(Not really a critique on the patch, which I know was meant to be a
quick illustration, but I thought it worth pointing out a doc-diff
command which actually works).

-Peff

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Bagas Sanjaya]

On 08/05/21 05.19, Jeff King wrote:

> I _thought_ the original asciidoc was marked as deprecated /
> unmaintained at some point. But it does seem to have gotten a few
> releases in the last year (it looks like maybe the python 2 version was
> EOL, but somebody decided to make the effort to port it to python 3?).

Asciidoc.py [1] now supports Python 3, however it only handles legacy
AsciiDoc syntax.

[1]: https://github.com/asciidoc-py/asciidoc-py
-- 
An old man doll... just what I always wanted! - Clara

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Bagas Sanjaya]

On 08/05/21 06.03, Felipe Contreras wrote:
  >> I think we're better to do like this:
>>
>> * If we're using asciidoc, we will use xmlto
>> * If we're using asciidoctor, we will generate man-pages with
>>    asciidoctor directly
> 
> This certainly is the way to go.
> 
> I'm not sure if Bagas suggested to make USE_ASCIIDOCTOR=YesPlease the
> default, but first we would need to make sure the output without xmlto
> is desirable, which we don't at the moment.

I'm not aiming for defaulting to AsciiDoctor, but I agree that we would like
to migrate away from xmlto by making sure the output sans it is desirable.

> So, even if we were to consider this, *first* the option to use pure
> asciidoctor needs to be in place.

Don't you know there is USE_ASCIIDOCTOR=YesPlease flag which do the job?

-- 
An old man doll... just what I always wanted! - Clara

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

On 2021-05-08 at 03:10:50, Jeff King wrote:
> On Fri, May 07, 2021 at 11:57:32PM +0000, brian m. carlson wrote:
> 
> > It's not quite that simple.  It requires changes to
> > asciidoctor-extensions.rb and we also need to consider
> > NO_MAN_BOLD_LITERAL and GNU_ROFF.  The latter especially is likely to be
> > inconvenient, since according to the Makefile, it's impossible to write
> > man output that works across troff versions.
> > 
> > We will also want to preserve the ability to use xmlto because we
> > currently have doc-diff which helps us find divergences in the behavior
> > of AsciiDoc and Asciidoctor, many of which affect all formats, and that
> > will be completely useless if we use two different rendering toolchains.
> 
> Speaking of doc-diff: with that patch applied, try "./doc-diff
> --asciidoctor HEAD^ HEAD", which shows quite a few of the complications
> (another one seems to be that all of the linkgit macros are broken).

Yes, that's because the linkgit macros are specified on a
format-specific basis by the extensions file and without updating
they won't understand how to generate output for the manpage formatter.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Martin Ågren]

On Sat, 8 May 2021 at 00:19, Jeff King <peff@peff.net> wrote:
>
> I _thought_ the original asciidoc was marked as deprecated /
> unmaintained at some point. But it does seem to have gotten a few
> releases in the last year (it looks like maybe the python 2 version was
> EOL, but somebody decided to make the effort to port it to python 3?).
>
> But I wouldn't be at all sad to just standardize on asciidoctor. I think
> we're at parity in terms of the output (thanks to lots of work from you
> and Martin over the past couple of years), and I've generally found it
> nicer to work with.

I tend to think asciidoctor even renders our manpages *better* than
asciidoc does. Not by a huge margin, but a few things here and there.
Some time around the Python 2 EOL, I was about to propose flipping the
default, but then I went to look up the asciidoc EOL schedule, and like
you, I noticed that it was a lot more alive and kicking than I thought
it was. So it's not so much "we should flip to avoid a bitrotting
dependency" as it is "asciidoctor is arguably nicer" or "it's the way
forward".

I've done some working-around in the past to try to make something look
non-broken in both of asciidoc and asciidoctor. Working with *three*
toolchains, we should probably be clear about which one it is we're
going to leave behind, and that shouldn't be the default. I'll be happy
to make a patch. I can't use the EOL argument, it seems, but I do think
it's the only thing that will allow us to reach an xmlto-less generation
of the manpages without losing our minds.

> The only downside is that it may be available in fewer places (though
> I'd think that python vs ruby is not so different). IMHO it's OK to be
> aggressive about the doc toolchain requirements, because the fallback is
> always grabbing the preformatted roff or HTML pages that were generated
> on a different system.

In general, I agree. I do think it's important that "most people
contributing to Git", whatever that means, can build the documentation
to check the part they're adding/modifying and not find it broken left
and right. They would then (quite rightly) not even bother building it.

When we looked at xmlto-less rendering around two years ago [1], we
found various asciidoctor bugs up to and around version 2.0. We would
likely need to require some >=2.0.x. The exact requirements will
probably only become clear when someone really does the work.

I think what I'm arguing for is

  1) switch the default to asciidoctor,
  2) enable optionally using it without xmlto,
  3) figure out what broke and fix it, and document which is the minimum
     asciidoctor version we're going to bother with for (2),
  4) lather, rinse, repeat (3),
  5) switch the default to not using xmlto,
  6) drop the xmlto way of generating the manpages(?).

[1] https://lore.kernel.org/git/20190317194431.GY31362@pobox.com/

Martin

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Martin Ågren]

On Sat, 8 May 2021 at 01:40, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
>
> brian m. carlson wrote:
> > On 2021-05-07 at 06:06:31, Bagas Sanjaya wrote:
> > > So completely migrating to Asciidoctor can eliminate xmlto requirement
> > > for generating manpage.
> > >
> > > What do you think about above?
> >
> > I didn't do this because it makes things very complicated in the
> > Makefile and my goal was to minimize the needed changes and divergence.
> > I'm not opposed to someone else doing it, but I expect it will be a
> > bunch of work.
>
> I don't think it would be that complicated.
>
> This patch seems to do the trick here:

This looks similar to Todd's wip [1]. Here's something
whitespace-damaged on top that improves some things, but I'm sure
there's more to do. I think in order to work on this, it would need to
be a separate switch so that we can use asciidoctor both the old-school
and the new-school way for a while.

[1] https://lore.kernel.org/git/20190317194431.GY31362@pobox.com/

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 0cfa88a92b..9f73d2d033 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -192,7 +192,9 @@ ASCIIDOC_HTML = xhtml5
 ASCIIDOC_DOCBOOK = docbook5
 ASCIIDOC_EXTRA += -acompat-mode -atabsize=8
 ASCIIDOC_EXTRA += -I. -rasciidoctor-extensions
-ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
+ASCIIDOC_EXTRA += -alitdd='\--'
+ASCIIDOC_EXTRA += -aplus='+'
+ASCIIDOC_COMMON += -a mansource="Git $(GIT_VERSION)" -a manmanual="Git Manual"
 DBLATEX_COMMON =
 XMLTO_EXTRA += --skip-validation
 XMLTO_EXTRA += -x manpage.xsl
diff --git a/Documentation/asciidoctor-extensions.rb
b/Documentation/asciidoctor-extensions.rb
index d906a00803..fb74443241 100644
--- a/Documentation/asciidoctor-extensions.rb
+++ b/Documentation/asciidoctor-extensions.rb
@@ -10,11 +10,16 @@ module Git

       def process(parent, target, attrs)
         prefix = parent.document.attr('git-relative-html-prefix')
+        s = "#{target}(#{attrs[1]})"
         if parent.document.doctype == 'book'
           "<ulink url=\"#{prefix}#{target}.html\">" \
           "#{target}(#{attrs[1]})</ulink>"
+        elsif parent.document.basebackend? 'manpage'
+          # FIXME: boldness should respect NO_MAN_BOLD_LITERAL?
+          # -- should be doable using an attribute, similar
+          # to "prefix" above
+          create_inline(parent, :quoted, s, type: :strong).convert
         elsif parent.document.basebackend? 'html'
           %(<a href="#{prefix}#{target}.html">#{target}(#{attrs[1]})</a>)
         elsif parent.document.basebackend? 'docbook'
           "<citerefentry>\n" \
             "<refentrytitle>#{target}</refentrytitle>" \

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

On Sun, May 9, 2021 at 3:26 AM Martin Ågren <martin.agren@gmail.com> wrote:

> I think what I'm arguing for is
>
>   1) switch the default to asciidoctor,
>   2) enable optionally using it without xmlto,
>   3) figure out what broke and fix it, and document which is the minimum
>      asciidoctor version we're going to bother with for (2),
>   4) lather, rinse, repeat (3),
>   5) switch the default to not using xmlto,
>   6) drop the xmlto way of generating the manpages(?).

Are there some minimal requirements to say; this documentation was
built correctly (as far as we know)? If so, maybe we can add a
checker, or perhaps add a test under t/.

Cheers.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Martin Ågren]

On Sun, 9 May 2021 at 20:46, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
>
> Are there some minimal requirements to say; this documentation was
> built correctly (as far as we know)? If so, maybe we can add a
> checker, or perhaps add a test under t/.

There's `make check-doc` which does some linting. But that's about
checking that there's a manpage at all for each builtin, that the doc
sources list sections in the right order, and such things. It doesn't
actually build the docs.

ci/test-documentation.sh runs `make check-doc` (see above), then goes on
to actually build the docs using both asciidoc and asciidoctor.  It
checks the exit status, but also that stderr is empty after filtering
out some expected, harmless output.

I think that's about the right balance. We could perhaps do something
under t/, but it probably shouldn't be to actually build the stuff. More
like, "oh, you've built the documentation -- let me sanity-check it for
you".  One way to detect that it makes sense to check it might be to see
if the Git version in the manpage footer(s) matches the version under
test. But I don't know what to check next.  (Actually, if we do find the
version number there, we have verified a fairly tricky piece of content
injection. At least "tricky" as in "we have more than one way of doing
it, because we support more than one toolchain" [1]. And if that stuff
broke, the test wouldn't notice, if we relied on detecting a version
match to even start testing...)

[1] See 7a30134358 ("asciidoctor-extensions: provide `<refmiscinfo/>`",
2019-09-16) and its parent commit.

Martin

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:

> > But I wouldn't be at all sad to just standardize on asciidoctor. I think
> > we're at parity in terms of the output (thanks to lots of work from you
> > and Martin over the past couple of years), and I've generally found it
> > nicer to work with.
> 
> I tend to think asciidoctor even renders our manpages *better* than
> asciidoc does. Not by a huge margin, but a few things here and there.

Yeah, I'd agree with that.

> Some time around the Python 2 EOL, I was about to propose flipping the
> default, but then I went to look up the asciidoc EOL schedule, and like
> you, I noticed that it was a lot more alive and kicking than I thought
> it was. So it's not so much "we should flip to avoid a bitrotting
> dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> forward".

I'm OK with those arguments, too. :)

> > The only downside is that it may be available in fewer places (though
> > I'd think that python vs ruby is not so different). IMHO it's OK to be
> > aggressive about the doc toolchain requirements, because the fallback is
> > always grabbing the preformatted roff or HTML pages that were generated
> > on a different system.
> 
> In general, I agree. I do think it's important that "most people
> contributing to Git", whatever that means, can build the documentation
> to check the part they're adding/modifying and not find it broken left
> and right. They would then (quite rightly) not even bother building it.

Agreed. But I think that is mostly the case (asciidoctor seems no harder
to acquire on most modern systems than asciidoc; there are system
packages in most cases, and decent binary-package systems for both ruby
and python if you really need it).

It does create a situation where people like Randall on NonStop might
need to do part of their dev work on another, more mainstream platform.
But I suspect that is already the case.

> When we looked at xmlto-less rendering around two years ago [1], we
> found various asciidoctor bugs up to and around version 2.0. We would
> likely need to require some >=2.0.x. The exact requirements will
> probably only become clear when someone really does the work.

That does make things a little less convenient; Debian stable, for
instance, still has 1.5.8. It's not too hard to install an updated gem,
but not quite as nice as using the system package (it also makes things
weird for building the stable Debian package itself, which would want to
rely only on other packages; but of course any proposed change to the
doc toolchain would be for new versions, and would not get backported
there anyway).

> I think what I'm arguing for is
> 
>   1) switch the default to asciidoctor,
>   2) enable optionally using it without xmlto,
>   3) figure out what broke and fix it, and document which is the minimum
>      asciidoctor version we're going to bother with for (2),
>   4) lather, rinse, repeat (3),
>   5) switch the default to not using xmlto,
>   6) drop the xmlto way of generating the manpages(?).

I'm unclear when support for python asciidoc goes away here. Is it part
of step 6 (because it does not have another way of generating them)? Or
does it live on forever as a non-default legacy system? I'd prefer not,
but as long as we are clear about the primary target and leave it up to
people interested in the legacy to do the compat fixes, that might be
OK.

-Peff

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Jeff King wrote:
> On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
> > In general, I agree. I do think it's important that "most people
> > contributing to Git", whatever that means, can build the documentation
> > to check the part they're adding/modifying and not find it broken left
> > and right. They would then (quite rightly) not even bother building it.
> 
> Agreed. But I think that is mostly the case (asciidoctor seems no harder
> to acquire on most modern systems than asciidoc; there are system
> packages in most cases, and decent binary-package systems for both ruby
> and python if you really need it).
> 
> It does create a situation where people like Randall on NonStop might
> need to do part of their dev work on another, more mainstream platform.
> But I suspect that is already the case.

Or use distributed tarballs with already built documentation.

I don't see any big issue with cross-compilation though.

> > When we looked at xmlto-less rendering around two years ago [1], we
> > found various asciidoctor bugs up to and around version 2.0. We would
> > likely need to require some >=2.0.x. The exact requirements will
> > probably only become clear when someone really does the work.
> 
> That does make things a little less convenient; Debian stable, for
> instance, still has 1.5.8.

And it has git 2.20.1, released at the end of 2018.

I've never understood developers worried about how the bleeding edge
would build in ancient platforms, when ancient platforms don't care
about the bleeding edge.

> It's not too hard to install an updated gem, but not quite as nice as
> using the system package (it also makes things weird for building the
> stable Debian package itself, which would want to rely only on other
> packages; but of course any proposed change to the doc toolchain would
> be for new versions, and would not get backported there anyway).

Anyone trying to build git master on top of Debian stable 1. probably
can live with the output of the current doc toolchain, and 2. probably
doesn't exist.

I'm not sure how much the git project gains by worrying about such
a hypothetical person.

> > I think what I'm arguing for is
> > 
> >   1) switch the default to asciidoctor,
> >   2) enable optionally using it without xmlto,
> >   3) figure out what broke and fix it, and document which is the minimum
> >      asciidoctor version we're going to bother with for (2),
> >   4) lather, rinse, repeat (3),
> >   5) switch the default to not using xmlto,
> >   6) drop the xmlto way of generating the manpages(?).
> 
> I'm unclear when support for python asciidoc goes away here. Is it part
> of step 6 (because it does not have another way of generating them)? Or
> does it live on forever as a non-default legacy system? I'd prefer not,
> but as long as we are clear about the primary target and leave it up to
> people interested in the legacy to do the compat fixes, that might be
> OK.

How about we leave the legacy system in place as an alternative, and
decide later what to do with it?

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:

> Jeff King wrote:
> > On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
> > > In general, I agree. I do think it's important that "most people
> > > contributing to Git", whatever that means, can build the documentation
> > > to check the part they're adding/modifying and not find it broken left
> > > and right. They would then (quite rightly) not even bother building it.
> > 
> > Agreed. But I think that is mostly the case (asciidoctor seems no harder
> > to acquire on most modern systems than asciidoc; there are system
> > packages in most cases, and decent binary-package systems for both ruby
> > and python if you really need it).
> > 
> > It does create a situation where people like Randall on NonStop might
> > need to do part of their dev work on another, more mainstream platform.
> > But I suspect that is already the case.
> 
> Or use distributed tarballs with already built documentation.

For users, yes. But the context above is about people who are
contributing to Git, and writing their own new documentation. Presumably
they'd like to build it to see the output.

> > That does make things a little less convenient; Debian stable, for
> > instance, still has 1.5.8.
> 
> And it has git 2.20.1, released at the end of 2018.
> 
> I've never understood developers worried about how the bleeding edge
> would build in ancient platforms, when ancient platforms don't care
> about the bleeding edge.

Again, this is about developers. Are there people contributing new
documentation to Git who are doing so on Debian stable, and would be
inconvenienced by needing to upgrade their toolchain?

> > > I think what I'm arguing for is
> > > 
> > >   1) switch the default to asciidoctor,
> > >   2) enable optionally using it without xmlto,
> > >   3) figure out what broke and fix it, and document which is the minimum
> > >      asciidoctor version we're going to bother with for (2),
> > >   4) lather, rinse, repeat (3),
> > >   5) switch the default to not using xmlto,
> > >   6) drop the xmlto way of generating the manpages(?).
> > 
> > I'm unclear when support for python asciidoc goes away here. Is it part
> > of step 6 (because it does not have another way of generating them)? Or
> > does it live on forever as a non-default legacy system? I'd prefer not,
> > but as long as we are clear about the primary target and leave it up to
> > people interested in the legacy to do the compat fixes, that might be
> > OK.
> 
> How about we leave the legacy system in place as an alternative, and
> decide later what to do with it?

That's what I was asking.

Leaving it forever does mean supporting xmlto forever, which complicates
the Makefile (and that support will bitrot if people are not actually
building it).

-Peff

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Jeff King wrote:
> On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
> > Jeff King wrote:
> > > It does create a situation where people like Randall on NonStop might
> > > need to do part of their dev work on another, more mainstream platform.
> > > But I suspect that is already the case.
> > 
> > Or use distributed tarballs with already built documentation.
> 
> For users, yes. But the context above is about people who are
> contributing to Git, and writing their own new documentation. Presumably
> they'd like to build it to see the output.

I'm a developer, I've added 261 lines and removed 212 lines to the
documentation, and I've very rarely built it. Why? Because it takes too
long.

But you were talking about Randall, who I don't know what his role is,
but my bet is that he is a packager. It's not the same thing. Randall
has contributed only one patch to the Documentation, and it was to
install pre-formatted documentation. Precisely what I thought he would be
interested in.

In fact, the bulk of his contribution was to install-doc-quick.sh, which
needs git-htmldocs and git-manpages. Two repositories I didn't even know
existed. So that's yet another option for him.

Of course Randall can say if he sees himself as a developer.

> > > That does make things a little less convenient; Debian stable, for
> > > instance, still has 1.5.8.
> > 
> > And it has git 2.20.1, released at the end of 2018.
> > 
> > I've never understood developers worried about how the bleeding edge
> > would build in ancient platforms, when ancient platforms don't care
> > about the bleeding edge.
> 
> Again, this is about developers. Are there people contributing new
> documentation to Git who are doing so on Debian stable, and would be
> inconvenienced by needing to upgrade their toolchain?

Developers don't need to create (or use) debian packages. They can
simply do `gem install asciidoctor` and be done with it. Some may even
create a docker container to install all the doc toolchain in order to
avoid polluting their main environment.

I for one would start building the documentation more if all I needed is
one dependency.

> > > I'm unclear when support for python asciidoc goes away here. Is it part
> > > of step 6 (because it does not have another way of generating them)? Or
> > > does it live on forever as a non-default legacy system? I'd prefer not,
> > > but as long as we are clear about the primary target and leave it up to
> > > people interested in the legacy to do the compat fixes, that might be
> > > OK.
> > 
> > How about we leave the legacy system in place as an alternative, and
> > decide later what to do with it?
> 
> That's what I was asking.
> 
> Leaving it forever does mean supporting xmlto forever, which complicates
> the Makefile (and that support will bitrot if people are not actually
> building it).

Indeed. If and when it's clear the xmlto part has bitrotten, and
people are happy with the asciidoc toolchain and output, then it can be
obsoleted.

That would be my vote (I don't think there will be a strong need to main
the xmlto parts).

It doesn't need to be decided today though.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jean-Noël Avila]

On 09/05/2021 at 10:20, Martin Ågren wrote :
>
> I tend to think asciidoctor even renders our manpages *better* than
> asciidoc does. Not by a huge margin, but a few things here and there.
> Some time around the Python 2 EOL, I was about to propose flipping the
> default, but then I went to look up the asciidoc EOL schedule, and like
> you, I noticed that it was a lot more alive and kicking than I thought
> it was. So it's not so much "we should flip to avoid a bitrotting
> dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> forward".

If we start to change the documentation format to "the way  forward", we
may soon end up with a format which is no longer handled by the legacy
asciidoc.py

As stated on https://github.com/asciidoc-py/asciidoc-py :

"AsciiDoc.py is a legacy processor for this syntax, handling an older
rendition of AsciiDoc. As such, this will not properly handle the
current AsciiDoc specification. It is suggested that unless you
specifically require the AsciiDoc.py toolchain, you should find a
processor that handles the modern AsciiDoc syntax."


So, as soon as the asciidoc format formal specification will gain
traction in the public, we can expect asciidoc to be abandoned for new
projects and receive minimal maintenance only for compatibility with
legacy documentation.

One argument in favor of Asciidoctor is that it's delivered "with
batteries", meaning that you can generate manpages, html and even pdf
with the same tool, without requiring secondary or even tertiary
toolchains, which should ease usage on a broader range of platforms.

FWIW, we are already using Asciidoctor for publishing the manpages to
https://git-scm.com

JN

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Ævar Arnfjörð Bjarmason]

On Tue, May 11 2021, Felipe Contreras wrote:

> Jeff King wrote:
>> On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
>> > Jeff King wrote:
>> [...]
>> > I've never understood developers worried about how the bleeding edge
>> > would build in ancient platforms, when ancient platforms don't care
>> > about the bleeding edge.
>> 
>> Again, this is about developers. Are there people contributing new
>> documentation to Git who are doing so on Debian stable, and would be
>> inconvenienced by needing to upgrade their toolchain?
>
> Developers don't need to create (or use) debian packages. They can
> simply do `gem install asciidoctor` and be done with it. Some may even
> create a docker container to install all the doc toolchain in order to
> avoid polluting their main environment.
>
> I for one would start building the documentation more if all I needed is
> one dependency.

Just because I'm developing the latest git.git revision on Debian stable
that doesn't mean that I'm keen to install the very latest openssl,
libcurl, asciidoc, C compiler, or whatever other thing we depend on.

I'm not disagreeing with bumping the dependency in this case (I haven't
looked into it). I'm just pointing out that in general there's a lot of
use-cases for e.g. building a latest git on an N year old OS.

Of course we can ask these people to just build their dependencies too,
as I noted in [1] in a past discussion. Whether we bump our required
dependencies is a trade-off between our own convenience and these sorts
of in-the-wild builds.

I'm just saying we should keep this use-case in mind, it's not an all or
nothing where you either have ancient deps + ancient git or bleeding
edge deps + bleeding edge git. A lot of people build ancient deps +
bleeding edge git.

The "just use the built doc tarballs" is only a partial solution, and
e.g. won't work for someone who's interested in building "next" or
otherwise applying local patches that have doc changes.

1. https://lore.kernel.org/git/874ltg2tvo.fsf@gmail.com/

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Martin Ågren]

On Tue, 11 May 2021 at 00:24, Jeff King <peff@peff.net> wrote:
>
> On Sun, May 09, 2021 at 10:20:37AM +0200, Martin Ågren wrote:
>
> > it was. So it's not so much "we should flip to avoid a bitrotting
> > dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> > forward".
>
> I'm OK with those arguments, too. :)

Excellent. :)

> > When we looked at xmlto-less rendering around two years ago [1], we
> > found various asciidoctor bugs up to and around version 2.0. We would
> > likely need to require some >=2.0.x. The exact requirements will
> > probably only become clear when someone really does the work.
>
> That does make things a little less convenient; Debian stable, for
> instance, still has 1.5.8. It's not too hard to install an updated gem,
> but not quite as nice as using the system package (it also makes things
> weird for building the stable Debian package itself, which would want to
> rely only on other packages; but of course any proposed change to the
> doc toolchain would be for new versions, and would not get backported
> there anyway).

Right. And 1.5.8 is perfectly fine for ascidoctor *with* xmlto, i.e., as
long as we're discussing moving away from asciidoc, not moving away from
xmlto entirely. And soon enough, Debian stable should be at 2.12. ;-)
(I realize Debian stable was just an example.)

> > I think what I'm arguing for is
> >
> >   1) switch the default to asciidoctor,
> >   2) enable optionally using it without xmlto,
> >   3) figure out what broke and fix it, and document which is the minimum
> >      asciidoctor version we're going to bother with for (2),
> >   4) lather, rinse, repeat (3),
> >   5) switch the default to not using xmlto,
> >   6) drop the xmlto way of generating the manpages(?).
>
> I'm unclear when support for python asciidoc goes away here. Is it part
> of step 6 (because it does not have another way of generating them)? Or
> does it live on forever as a non-default legacy system? I'd prefer not,
> but as long as we are clear about the primary target and leave it up to
> people interested in the legacy to do the compat fixes, that might be
> OK.

I think I meant it to happen somewhere in step 3 or 4, I just didn't
call it out. If it happens super-early in step 3, that would perhaps be
a bit unfortunate. But if fixing up the last few bits of
xmlto-vs-no-xmlto with asciidoctor involves giving up on asciidoc, then
so be it... If nothing else, we might all of a sudden find that our
asciidoc-processed docs look awful and that nobody seems to have
noticed.

Martin

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Martin Ågren]

On Tue, 11 May 2021 at 11:04, Jean-Noël Avila <avila.jn@gmail.com> wrote:
>
> On 09/05/2021 at 10:20, Martin Ågren wrote :
> >
> > I tend to think asciidoctor even renders our manpages *better* than
> > asciidoc does. Not by a huge margin, but a few things here and there.
> > Some time around the Python 2 EOL, I was about to propose flipping the
> > default, but then I went to look up the asciidoc EOL schedule, and like
> > you, I noticed that it was a lot more alive and kicking than I thought
> > it was. So it's not so much "we should flip to avoid a bitrotting
> > dependency" as it is "asciidoctor is arguably nicer" or "it's the way
> > forward".
>
> If we start to change the documentation format to "the way  forward", we
> may soon end up with a format which is no longer handled by the legacy
> asciidoc.py

We used to be in a situation where Asciidoctor looked worse and the
rendered versions were quite different. We've fixed up quite a few
discrepancies by making some change that "happens" to be a noop with one
engine but an improvement with the other. (That it just "happens" has
sometimes been my feeling anyway.) Sometimes, we've been able to improve
both by spotting a difference, so that's good.

I would also expect that with more eyes on asciidoctor-built docs
(because default) and fewer on the other, the non-default will start to
degrade.

> As stated on https://github.com/asciidoc-py/asciidoc-py :
>
> "AsciiDoc.py is a legacy processor for this syntax, handling an older
> rendition of AsciiDoc. As such, this will not properly handle the
> current AsciiDoc specification. It is suggested that unless you
> specifically require the AsciiDoc.py toolchain, you should find a
> processor that handles the modern AsciiDoc syntax."

Thanks for that quote. It's very enlightening.

> FWIW, we are already using Asciidoctor for publishing the manpages to
> https://git-scm.com

Thank you for all your work on that site!

Martin

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Ævar Arnfjörð Bjarmason wrote:
> On Tue, May 11 2021, Felipe Contreras wrote:
> 
> > Jeff King wrote:
> >> On Mon, May 10, 2021 at 11:27:54PM -0500, Felipe Contreras wrote:
> >> > Jeff King wrote:
> >> [...]
> >> > I've never understood developers worried about how the bleeding edge
> >> > would build in ancient platforms, when ancient platforms don't care
> >> > about the bleeding edge.
> >> 
> >> Again, this is about developers. Are there people contributing new
> >> documentation to Git who are doing so on Debian stable, and would be
> >> inconvenienced by needing to upgrade their toolchain?
> >
> > Developers don't need to create (or use) debian packages. They can
> > simply do `gem install asciidoctor` and be done with it. Some may even
> > create a docker container to install all the doc toolchain in order to
> > avoid polluting their main environment.
> >
> > I for one would start building the documentation more if all I needed is
> > one dependency.
> 
> Just because I'm developing the latest git.git revision on Debian stable
> that doesn't mean that I'm keen to install the very latest openssl,
> libcurl, asciidoc, C compiler, or whatever other thing we depend on.
> 
> I'm not disagreeing with bumping the dependency in this case (I haven't
> looked into it). I'm just pointing out that in general there's a lot of
> use-cases for e.g. building a latest git on an N year old OS.

I'm not disagreeing with that. The reason I mentioned it is what Jeff
said next:

> > > It's not too hard to install an updated gem, but not quite as nice
> > > as using the system package (it also makes things weird for
> > > building the stable Debian package itself, which would want to
> > > rely only on other packages; but of course any proposed change to
> > > the doc toolchain would be for new versions, and would not get
> > > backported there anyway).

Doing `gem install` solves the problem for whomever wants to build the
latest git in Debian stable.

Building Debian stable packages is something else.

> Of course we can ask these people to just build their dependencies too,
> as I noted in [1] in a past discussion. Whether we bump our required
> dependencies is a trade-off between our own convenience and these sorts
> of in-the-wild builds.
> 
> I'm just saying we should keep this use-case in mind, it's not an all or
> nothing where you either have ancient deps + ancient git or bleeding
> edge deps + bleeding edge git. A lot of people build ancient deps +
> bleeding edge git.

But what is "this"?

 1) Building the latest documentation in Debian stable
 2) Packaging the latest git in Debian stable

If it's 1) then there's no problem: `gem install` does the trick, and in
fact in their CI builds [1] they test with versions of Ruby as old as
2.3, and Debian stable ships with 2.5.

I test my own Ruby code with versions of Ruby as old as 2.0 and there's
rarely any issue. What works with Ruby 2.3 works with Ruby 2.0. Ruby is
not like python (where 3.0 is completely different from 2.0).

If it's 2) then that's where I say: who cares?

> The "just use the built doc tarballs" is only a partial solution, and
> e.g. won't work for someone who's interested in building "next" or
> otherwise applying local patches that have doc changes.

Right, but "just use the built doc tarballs" is not the suggestion for
people trying to do 1).

Cheers.

[1] https://github.com/asciidoctor/asciidoctor/actions/runs/830132862

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Tue, May 11, 2021 at 08:45:10PM +0200, Martin Ågren wrote:

> > That does make things a little less convenient; Debian stable, for
> > instance, still has 1.5.8. It's not too hard to install an updated gem,
> > but not quite as nice as using the system package (it also makes things
> > weird for building the stable Debian package itself, which would want to
> > rely only on other packages; but of course any proposed change to the
> > doc toolchain would be for new versions, and would not get backported
> > there anyway).
> 
> Right. And 1.5.8 is perfectly fine for ascidoctor *with* xmlto, i.e., as
> long as we're discussing moving away from asciidoc, not moving away from
> xmlto entirely. And soon enough, Debian stable should be at 2.12. ;-)
> (I realize Debian stable was just an example.)

Debian stable is just an example, but I also consider it a bit of a
benchmark for "reasonable". Surely there are people running RHEL6
somewhere in this world, but it's hard to care too much about them.

I think the transition you're proposing would probably take a while to
do, too. If we don't drop the python asciidoc support until close to the
end, then that buys a bit more time. Likewise, this isn't a hard limit
for OS support for users. The worst case is just making things slightly
more inconvenient for Git developers on older systems, because because
they might have to install an updated gem rather than using the system
package (you sometimes can end up in dependency hell for a gem upgrade
with versions of ruby, system libraries, etc, but I haven't found
asciidoctor particularly needy in that respect).

So I dunno. I certainly don't have a big complaint about _starting_ the
transition. If we can hold on to python asciidoc support (or even
old-asciidoctor + xmlto) for a while as a fallback, even if we know it's
slowly bitrotting, then it's possible nobody would even complain.

-Peff

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Jeff King]

On Tue, May 11, 2021 at 02:00:55PM -0500, Felipe Contreras wrote:

> > > > It's not too hard to install an updated gem, but not quite as nice
> > > > as using the system package (it also makes things weird for
> > > > building the stable Debian package itself, which would want to
> > > > rely only on other packages; but of course any proposed change to
> > > > the doc toolchain would be for new versions, and would not get
> > > > backported there anyway).
> 
> Doing `gem install` solves the problem for whomever wants to build the
> latest git in Debian stable.
> 
> Building Debian stable packages is something else.

Perhaps I wasn't clear with the "but of course" part of my statement. It
was meant to rebut the earlier half of the parenthetical. I.e., drawing
that same distinction as "we don't have to worry about packaging Debian
stable here".

-Peff

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Martin Ågren]

On Tue, 11 May 2021 at 21:07, Jeff King <peff@peff.net> wrote:
>
> So I dunno. I certainly don't have a big complaint about _starting_ the
> transition. If we can hold on to python asciidoc support (or even
> old-asciidoctor + xmlto) for a while as a fallback, even if we know it's
> slowly bitrotting, then it's possible nobody would even complain.

You made several good points; I just quoted the last few here. Thanks
for your thoughts on this.

Martin

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Jeff King wrote:
> The worst case is just making things slightly more inconvenient for
> Git developers on older systems, because because they might have to
> install an updated gem rather than using the system package (you
> sometimes can end up in dependency hell for a gem upgrade with
> versions of ruby, system libraries, etc, but I haven't found
> asciidoctor particularly needy in that respect).

The asciidoctor gem has *zero* dependencies. If you end up in a
dependency hell, it wasn't the fault of asciidoctor, and it won't be
affected.

I've been using Ruby for at least 15 years, and I've never found myself
in a dependency hell, so it would be interesting to see how anybody
could end up there. Sometimes the mix between system and user gems is
confusing, but they work fine. I wouldn't call that a "dependency hell".

And of course you can always just `rm -r $GEM_HOME` and start fresh.
Most gems install rather quickly.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

Jeff King wrote:
> On Tue, May 11, 2021 at 02:00:55PM -0500, Felipe Contreras wrote:
> 
> > > > > It's not too hard to install an updated gem, but not quite as nice
> > > > > as using the system package (it also makes things weird for
> > > > > building the stable Debian package itself, which would want to
> > > > > rely only on other packages; but of course any proposed change to
> > > > > the doc toolchain would be for new versions, and would not get
> > > > > backported there anyway).
> > 
> > Doing `gem install` solves the problem for whomever wants to build the
> > latest git in Debian stable.
> > 
> > Building Debian stable packages is something else.
> 
> Perhaps I wasn't clear with the "but of course" part of my statement. It
> was meant to rebut the earlier half of the parenthetical. I.e., drawing
> that same distinction as "we don't have to worry about packaging Debian
> stable here".

Ah, I undesrstood wrong there, it seems we are in agreement.

Semicolons in English are similar to semicolons in computer languages:
when I read ($a; $b), $a needs to stand on its own. If $b is really
important I would do ($a, but $b). But this is like splitting hairs of
splitted hairs.

I've seen too much discussion and too few patches, I'll send the mockup
I did a few days ago to see if it sticks.

Cheers.

-- 
Felipe Contreras

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2721 bytes --]

On 2021-05-11 at 04:27:54, Felipe Contreras wrote:
> I've never understood developers worried about how the bleeding edge
> would build in ancient platforms, when ancient platforms don't care
> about the bleeding edge.

Debian stable is a common environment to do development on.  I know
people who do use it for Git development, so I suspect we'll want to
continue to support it.  In fact, most of my colleagues who use a Debian
system for development use Debian stable, I'd say.

Moreover, in some cases, the distros one can use for development are
restricted due to requirements for software that runs on corporate
machines, so making things work nicely on the latest stable versions of
Debian and Ubuntu is generally kind.

Yes, people _can_ run "gem install asciidoctor", but people who are not
Ruby developers generally would prefer a distro package over installing
one-off gems, especially since getting the binaries into PATH is tricky
with gem.

> > It's not too hard to install an updated gem, but not quite as nice as
> > using the system package (it also makes things weird for building the
> > stable Debian package itself, which would want to rely only on other
> > packages; but of course any proposed change to the doc toolchain would
> > be for new versions, and would not get backported there anyway).
> 
> Anyone trying to build git master on top of Debian stable 1. probably
> can live with the output of the current doc toolchain, and 2. probably
> doesn't exist.

I believe I have just demonstrated that 2 is false above.

> > > I think what I'm arguing for is
> > > 
> > >   1) switch the default to asciidoctor,
> > >   2) enable optionally using it without xmlto,
> > >   3) figure out what broke and fix it, and document which is the minimum
> > >      asciidoctor version we're going to bother with for (2),
> > >   4) lather, rinse, repeat (3),
> > >   5) switch the default to not using xmlto,
> > >   6) drop the xmlto way of generating the manpages(?).
> > 
> > I'm unclear when support for python asciidoc goes away here. Is it part
> > of step 6 (because it does not have another way of generating them)? Or
> > does it live on forever as a non-default legacy system? I'd prefer not,
> > but as long as we are clear about the primary target and leave it up to
> > people interested in the legacy to do the compat fixes, that might be
> > OK.
> 
> How about we leave the legacy system in place as an alternative, and
> decide later what to do with it?

I think it would be fine to just leave it in place for now and let
people decide which toolchain they'd like to use.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [RFC suggestion] Generate manpage directly with Asciidoctor

[Felipe Contreras]

brian m. carlson wrote:
> On 2021-05-11 at 04:27:54, Felipe Contreras wrote:
> > I've never understood developers worried about how the bleeding edge
> > would build in ancient platforms, when ancient platforms don't care
> > about the bleeding edge.
> 
> Debian stable is a common environment to do development on.  I know
> people who do use it for Git development, so I suspect we'll want to
> continue to support it.

I didn't mean to suggest otherwise.

> Yes, people _can_ run "gem install asciidoctor", but people who are not
> Ruby developers generally would prefer a distro package over installing
> one-off gems, especially since getting the binaries into PATH is tricky
> with gem.

Yes, but there's only so much we can hold hands with our users.

If a user:

 1. Uses an acient distribution
 2. Wants to build the documentation
 3. Enables USE_ASCIIDOCTOR
 4. Doesn't know Ruby
 5. Wants to use distribution packages
 6. Is bothered by the output

I think it's valid for the project to say "you are on your own". In
fact, not really that because if they contact the mailing list we would
help them.

The only thing we could do is print a warning if they try to build with
versions of asciidoctor that we know are problematic.

That being said; it's not "tricky" to get binaries into your PATH:

  export PATH="$GEM_HOME/bin:$PATH"

And you don't need to get gem binaries into your PATH:

  export GEM_HOME=/tmp/gems
  gem install asciidoctor
  make USE_ASCIIDOCTOR=YesPlease ASCIIDOC=$GEM_HOME/bin/asciidoctor doc

Works just fine without modifying PATH.

> > > It's not too hard to install an updated gem, but not quite as nice as
> > > using the system package (it also makes things weird for building the
> > > stable Debian package itself, which would want to rely only on other
> > > packages; but of course any proposed change to the doc toolchain would
> > > be for new versions, and would not get backported there anyway).
> > 
> > Anyone trying to build git master on top of Debian stable 1. probably
> > can live with the output of the current doc toolchain, and 2. probably
> > doesn't exist.
> 
> I believe I have just demonstrated that 2 is false above.

I meant trying to build the documentation of git on git master.

> > > > I think what I'm arguing for is
> > > > 
> > > >   1) switch the default to asciidoctor,
> > > >   2) enable optionally using it without xmlto,
> > > >   3) figure out what broke and fix it, and document which is the minimum
> > > >      asciidoctor version we're going to bother with for (2),
> > > >   4) lather, rinse, repeat (3),
> > > >   5) switch the default to not using xmlto,
> > > >   6) drop the xmlto way of generating the manpages(?).
> > > 
> > > I'm unclear when support for python asciidoc goes away here. Is it part
> > > of step 6 (because it does not have another way of generating them)? Or
> > > does it live on forever as a non-default legacy system? I'd prefer not,
> > > but as long as we are clear about the primary target and leave it up to
> > > people interested in the legacy to do the compat fixes, that might be
> > > OK.
> > 
> > How about we leave the legacy system in place as an alternative, and
> > decide later what to do with it?
> 
> I think it would be fine to just leave it in place for now and let
> people decide which toolchain they'd like to use.

Agreed.

-- 
Felipe Contreras