gnulib's translation

gnulib's translation

[Akim Demaille]

Salut à tous,

Using a gnulib-po (https://www.gnu.org/software/gnulib/manual/html_node/Localization.html) makes a lot of sense: maximize the factoring of the translation effort across packages.  I was moving Bison to use this, but that's a visible regression: way too many translations are not available.  Using Bison in French used to feel right, but with that change it no longer does.

My understanding is that the last time gnulib was submitted for translation was in 2011: https://translationproject.org/domain/gnulib.html.

commit a6b16b69fe1cad695b270dd5bf3deb2850fc4dd1
Author: Paul Eggert <eggert@cs.ucla.edu>
Date:   Thu Aug 18 00:46:17 2011 -0700

    regex: port to Stratus OpenVOS

That's not enough, some form of periodicity is needed.

Also, I feel sorry for Bison's translators when submitting modules to gnulib: it's kind of throwing away their work; it would be great if there were a means to preserve these translations.  Maybe the translation project is able to fill the translations of one project based on that of another, I don't know.

Re: gnulib's translation

[Akim Demaille]

> Le 26 nov. 2018 à 07:18, Akim Demaille <akim.demaille@gmail.com> a écrit :
> 
> Using a gnulib-po (https://www.gnu.org/software/gnulib/manual/html_node/Localization.html) makes a lot of sense: maximize the factoring of the translation effort across packages.  I was moving Bison to use this, but that's a visible regression: way too many translations are not available.  Using Bison in French used to feel right, but with that change it no longer does.
> 
> My understanding is that the last time gnulib was submitted for translation was in 2011: https://translationproject.org/domain/gnulib.html.

Hi!

What is the consensus on this regard?  Is gnulib-po considered obsolete?

Re: gnulib's translation

[Bruno Haible]

Hi Akim,

> My understanding is that the last time gnulib was submitted for translation was in 2011: https://translationproject.org/domain/gnulib.html.

Yes, it would be desirable that someone regularly (maybe once every 6 months
or so) submits the gnulib .pot file to the TP.

Volunteers?

Bruno

Re: gnulib's translation

[Bruno Haible]

[CCing the TP coordinator]

Hi Benno,

Akim has just moved some code from Bison to Gnulib. He writes:

> Also, I feel sorry for Bison's translators when submitting modules to gnulib: it's kind of throwing away their work; it would be great if there were a means to preserve these translations.  Maybe the translation project is able to fill the translations of one project based on that of another, I don't know.

What is the recommended procedure, to save translator work, when a number
of messages have been moved from one domain to another domain (both domains
are managed by the TP)?

Bruno

Re: gnulib's translation

[Akim Demaille]

> Le 1 déc. 2018 à 16:57, Bruno Haible <bruno@clisp.org> a écrit :
> 
> Hi Akim,
> 
>> My understanding is that the last time gnulib was submitted for translation was in 2011: https://translationproject.org/domain/gnulib.html.
> 
> Yes, it would be desirable that someone regularly (maybe once every 6 months
> or so) submits the gnulib .pot file to the TP.
> 
> Volunteers?

I personally do no such thing for Bison.  As part of Bison's
release procedure, an announcement message is generated, with
these headers:

To: bison-patches@gnu.org
Cc: bug-bison@gnu.org, help-bison@gnu.org, Sébastien Villemot <sebastien@debian.org>, coordinator@translationproject.org
Mail-Followup-To: bug-bison@gnu.org
Subject: Bison 3.2.2 released [stable]

I assume that some robot on the translation project receives
these messages and processes them to take the latest release.
In which case, we should automate such a procedure (automating
in the sense of preparing everything, not (yet) in the sense of
a crontab.

Or maybe the translation project could work on top of git now?
It would help with the unfortunate current state in which any
new string is necessarily first release untranslated.  Translators
have no chance to catch them before (unless there is a beta
release, which is still a release).

Re: gnulib's translation

[Akim Demaille]

> Le 1 déc. 2018 à 17:36, Akim Demaille <akim.demaille@gmail.com> a écrit :
> 
> I personally do no such thing for Bison.  As part of Bison's
> release procedure, an announcement message is generated, with
> these headers:
> 
> To: bison-patches@gnu.org
> Cc: bug-bison@gnu.org, help-bison@gnu.org, Sébastien Villemot <sebastien@debian.org>, coordinator@translationproject.org
> Mail-Followup-To: bug-bison@gnu.org
> Subject: Bison 3.2.2 released [stable]

Sorry, Sébastien is not part of the standard email, I Cc'ed
him this one time because he reported the bug that 3.2.2 fixed.

Re: gnulib's translation

[Bruno Haible]

Akim Demaille wrote:
> > Yes, it would be desirable that someone regularly (maybe once every 6 months
> > or so) submits the gnulib .pot file to the TP.
> > 
> > Volunteers?
> 
> I personally do no such thing for Bison.  As part of Bison's
> release procedure, an announcement message is generated

That's what I meant: One needs to
  - prepare a tarball, that contains a POT file,
  - announce it to the TP coordinators at <coordinator@translationproject.org>
    by sending them the URL of the release tarball [1]

> maybe the translation project could work on top of git now?

This shift of work from the maintainers to the translation coordinator
would require
  1. that the translation coordinator installs whatever tooling is needed
     to build a particular package,
  2. that the translation coordinator builds each package in a separate
     VM or docker instance (for security: you don't want that the build
     procedure of one package installs a trojan horse into the environment
     of the coordinator, that could then poison the PO files of all further
     packages that get processed).

I don't know whether the translation coordinator is prepared for that?

Bruno

[1] https://translationproject.org/html/maintainers.html

Re: gnulib's translation

[Akim Demaille]

> Le 1 déc. 2018 à 20:47, Bruno Haible <bruno@clisp.org> a écrit :
> 
>> maybe the translation project could work on top of git now?
> 
> This shift of work from the maintainers to the translation coordinator
> would require
>  1. that the translation coordinator installs whatever tooling is needed
>     to build a particular package,

It is true that some of the messages might not be in the package
itself, but I would expect that it's a common case that all the
_() mark up is already available.

>  2. that the translation coordinator builds each package in a separate
>     VM or docker instance (for security: you don't want that the build
>     procedure of one package installs a trojan horse into the environment
>     of the coordinator, that could then poison the PO files of all further
>     packages that get processed).
> 
> I don't know whether the translation coordinator is prepared for that?

I understand your concern, but I was not suggesting that the
TP would run code from the repo.  I guess I'm too naive thinking
that enough information is available from a commit.

Re: gnulib's translation

[Bruno Haible]

Hi Akim,

> > This shift of work from the maintainers to the translation coordinator
> > would require
> >  1. that the translation coordinator installs whatever tooling is needed
> >     to build a particular package,
> 
> It is true that some of the messages might not be in the package
> itself, but I would expect that it's a common case that all the
> _() mark up is already available.

Yes, the POT file can usually be generated from the sources found in a git
repository. But I argue that it's not the job of the TP coordinator to
build all kinds of packages.

In other words:
  - In a version control repository you find: sources.
  - In a tarball you find: sources + generated files.
  - The POT file is a generated file.
  - Thus the interface between the package maintainers and the TP is simpler if
    the package maintainers submit a tarball to the TP.

> I was not suggesting that the
> TP would run code from the repo.  I guess I'm too naive thinking
> that enough information is available from a commit.

Ah, you were assuming that the POT file is stored in the version control system?
This is a practice that produces problems, and will be become rare after the
next gettext release.

Bruno

Re: gnulib's translation

[Benno Schulenberg]

Op 01-12-18 om 17:00 schreef Bruno Haible:
> Akim has just moved some code from Bison to Gnulib. He writes:
> 
>> Also, I feel sorry for Bison's translators when submitting modules to
>> gnulib: it's kind of throwing away their work; it would be great if there
>> were a means to preserve these translations.  Maybe the translation project
>> is able to fill the translations of one project based on that of another, I
>> don't know.
> 
> What is the recommended procedure, to save translator work, when a number of
> messages have been moved from one domain to another domain (both domains are
> managed by the TP)?

There is no procedure, as such moving of stuff across domains is a rare
affair.  But... it should not result in any Bison strings going untranslated
because Bison will now include the relevant modules from gnulib, and the
strings they contain will still be included in Bison's POT file -- they
will just be marked with "#: lib/..." instead of "#: src/...", no?

(When a new gnulib POT file gets submitted, I could do a one-time msgmerge
to "import" translated strings from Bison to gnulib.)

Benno

Re: gnulib's translation

[Benno Schulenberg]

> Akim Demaille wrote:
>> maybe the translation project could work on top of git now?

No, I'm not going to do that.  The TP is geared toward packages that
make releases.  So I need a (prerelease) tarball that contains the
corresponding POT file.

Benno

Re: gnulib's translation

[Benno Schulenberg]

Op 02-12-18 om 13:10 schreef Bruno Haible:
>   - Thus the interface between the package maintainers and the TP is simpler if
>     the package maintainers submit a tarball to the TP.

Precisely.  Furthermore, when following git, how is the translator to
know that a release is approaching and the time has come to update
his/her PO file?  We don't want to prod the translators for every
little string change.

> Ah, you were assuming that the POT file is stored in the version control system?
> This is a practice that produces problems,

What problems does this produce?  (Probably this was discussed earlier
and elsewhere?  Maybe have a URL of an archived message?)

Benno

Re: gnulib's translation

[Paul Smith]

On Sun, 2018-12-02 at 15:38 +0100, Benno Schulenberg wrote:
> > Ah, you were assuming that the POT file is stored in the version
> > control system?  This is a practice that produces problems,
> 
> What problems does this produce?  (Probably this was discussed
> earlier and elsewhere?  Maybe have a URL of an archived message?)

I'm not sure what Bruno was referring to but one issue is that POT
files contain timestamps, so source control systems consider them
"modified" every time they're rebuilt even if nothing else changes.

It's also possible that different versions of gettext could generate
slightly different POT files, so if committers have different versions
installed you get "warring commits" which change them back and forth. 
I don't know if POT files have this issue but other files generated by
autotools certainly do.

Re: gnulib's translation

[Bruno Haible]

Hi Benno,

> > Ah, you were assuming that the POT file is stored in the version control system?
> > This is a practice that produces problems,
> 
> What problems does this produce?

<BEGIN NEW DOC>

There are basically three ways to deal with generated files in the context
of a version controlled repository, such as ‘configure’ generated from
‘configure.ac’, parser.c generated from parser.y, or po/Makefile.in.in
autoinstalled by gettextize or autopoint.

“Never”
    Generated files are never committed into the repository. 
“Occasionally”
    All generated files are committed into the repository occasionally, for
    example each time a release is made. 
“Always”
    All generated files are always committed into the repository. 

Each of these three approaches has different advantages and drawbacks.

“Never”
    The advantage is less work for the maintainers. In particular, the
    handling of branches becomes much easier, both for release branches
    and “topic branches” as in Git. The drawback is that anyone who checks
    out the source not only needs tools like GNU automake, GNU autoconf,
    GNU m4 installed in his PATH, but also that he needs to perform a
    package specific pre-build step before being able to "./configure; make".
“Occasionally”
    The advantage is that anyone can check out the source, and the usual
    "./configure; make" will work. The drawbacks are: 1. The one who checks
    out the repository needs tools like GNU automake, GNU autoconf, GNU m4
    installed in his PATH; sometimes he even needs particular versions of
    them. 2. When a release is made and a commit is made on the generated
    files, the other developers get conflicts on the generated files when
    merging the local work back to the repository. Although these conflicts
    are easy to resolve, they are annoying. 3. Copying a change to a branch
    is not easy, because it involves separating the patch into a hand-made
    part, to be applied literally, and a set of commands or Makefile targets
    which will produce the other part. 4. Working with branches is time-
    consuming and error-prone.
“Always”
    The advantage is that anyone can check out the source at any moment and
    gets a working build. The drawbacks are: 1. It requires some frequent
    "push" actions by the maintainers. 2. The repository grows in size quite
    fast. 3. and 4. as for “Occasionally”. 

The “Never” approach is the predominant one nowadays, especially in projects that use branches like in Git. 

<END NEW DOC>

> (Probably this was discussed earlier
> and elsewhere?  Maybe have a URL of an archived message?)

It will be documented in the next release of the GNU gettext manual.

Bruno

Re: gnulib's translation

[Bruno Haible]

Paul Smith wrote:
> It's also possible that different versions of gettext could generate
> slightly different POT files, so if committers have different versions
> installed you get "warring commits" which change them back and forth. 
> I don't know if POT files have this issue but other files generated by
> autotools certainly do.

Yes, POT files have this issue as well. Each time there is a change to
xgettext - even if it's merely a bug fix -, different xgettext versions
will produce different POT files.

Bruno

Re: gnulib's translation

[Akim Demaille]

> Le 1 déc. 2018 à 16:57, Bruno Haible <bruno@clisp.org> a écrit :
> 
> Hi Akim,
> 
>> My understanding is that the last time gnulib was submitted for translation was in 2011: https://translationproject.org/domain/gnulib.html.
> 
> Yes, it would be desirable that someone regularly (maybe once every 6 months
> or so) submits the gnulib .pot file to the TP.
> 
> Volunteers?

I'm looking into this.

Is `./gnulib-tool --create-megatestdir --dir=/tmp/gnulib-$(date +%F)` the right starting point?

Re: gnulib's translation

[Bruno Haible]

Hi Akim,

> > Yes, it would be desirable that someone regularly (maybe once every 6 months
> > or so) submits the gnulib .pot file to the TP.
> > 
> > Volunteers?
> 
> I'm looking into this.

Thank you!!

> Is `./gnulib-tool --create-megatestdir --dir=/tmp/gnulib-$(date +%F)` the right starting point?

No, this is overkill.

Even
    ./gnulib-tool --create-testdir --without-tests --dir=...
is overkill.

It's essentially two commands:

$ version=`git describe HEAD | sed -e 's/v0.1/4.0.0/' -e 's/-/./g'`
# or similar - make sure it's increasing,
# cf. https://translationproject.org/domain/gnulib.html

$ xgettext --keyword=_ --keyword=N_ \
           --copyright-holder='Free Software Foundation, Inc.' \
           --package-name=gnulib --package-version=$version \
           --msgid-bugs-address='bug-gnulib@gnu.org' \
           -o gnulib.pot \
           `find lib -name '*.h' -o -name '*.c' | LC_ALL=C sort`

Bruno

Re: gnulib's translation

[Akim Demaille]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

Hi all,

> Le 7 déc. 2018 à 04:33, Bruno Haible <bruno@clisp.org> a écrit :
> 
> Hi Akim,
> 
>>> Yes, it would be desirable that someone regularly (maybe once every 6 months
>>> or so) submits the gnulib .pot file to the TP.
>>> 
>>> Volunteers?
>> 
>> I'm looking into this.
> 
> Thank you!!

For the records, I've had a few exchanges with Benno to tune the attached script (which is based on what Bruno offered).  Then I submitted a tarball which I hosted. Benno wants to work only with gnu.org.  Now we are stuck, because I am not allowed to upload tarballs of gnulib on alpha.gnu.org, as I'm not a maintainer.

Is someone willing to do it?

Cheers!


[-- Attachment #2: gnulib-release --]
[-- Type: application/octet-stream, Size: 837 bytes --]

#! /bin/sh

version=$(git describe HEAD | sed -e 's/v0.1/4.0.0/' -e 's/-/./g')
# or similar - make sure it's increasing,
# cf. https://translationproject.org/domain/gnulib.html

distdir=gnulib-$version
rm -rf "$distdir"
mkdir "$distdir"
xgettext --keyword=_ --keyword=N_ \
        --copyright-holder='Free Software Foundation, Inc.' \
        --package-name=gnulib --package-version=$version \
        --msgid-bugs-address='bug-gnulib@gnu.org' \
         --add-comments=TRANSLATORS: \
        -o "$distdir"/gnulib.pot \
        $(find lib -name '*.h' -o -name '*.c' | LC_ALL=C sort)
git archive --prefix="$distdir/" -o "$distdir.tar" HEAD
tar rvf "$distdir.tar" "$distdir/gnulib.pot"
gzip "$distdir.tar"
rm -rf "$distdir"

cat <<EOF
Now run:

$ ./build-aux/gnupload --to alpha.gnu.org:gnulib "$distdir.tar.gz"
$ rm "$distdir.tar.gz"
EOF

Re: gnulib's translation

[Akim Demaille]

Last update: gnulib's tarball was uploaded on alpha.gnu.org in Bison's directory, and pushed into the Translation Project (https://translationproject.org/domain/gnulib.html).  So using gnulib-po should now make more sense (and I plan to have Bison use it).

Cheers!