gh-130057: Pygettext: Support translator comments

[tomasr8]

Adds support for translator comments to pygettext.

Now that pygettext is using a parser we don't have access to the comments anymore but we can circumvent that by
first running the tokenizer and getting all comments first. The comments are then matched to their corresponding
gettext calls in the parser using line numbers.

The new CLI option is -cTAG and --add-comments=TAG which is the same option that is used by both xgettext and babel.
It is possible to specify the flag multiple times.

The usual translator comment looks like this:

# i18n: Translator comment
_('foo')

The comment can also be multi-line:

# i18n: Translator comment
# i18n: Another translator comment
_('foo')

The comment tag only needs to be on the first line:

# i18n: Translator comment
# Another translator comment
_('foo')

So far both xgettext and babel agree, but there are some discrepancies when it comes to edge cases.
For instance, babel extracts both comments from this snippet while xgettext only extracts the second one:

# i18n: comment
x = 1
# i18n: another comment
_('foo')

xgettext, on the other hand extracts this comment while babel does not:

# i18n: This comment should be ignored

_('foo')

There are more cases, but this is the gist. For pygettext, I went for an implementation which is identical to xgettext and babel in 99% cases but is simpler and more predictable for the users in the other 1%. In short, the implementation does not allow any gaps between the comments or between the first comment and the gettext call.

This means that in this snippet, only the second comment is extracted (matches xgettext):

# i18n: comment
x = 1
# i18n: another comment
_('foo')

and this comment is not extracted (matches babel):

# i18n: This comment should be ignored

_('foo')

• Issue: Pygettext: Support translator comments #130057

[tomasr8]

(Updated the branch to fix the All required checks pass workflow)

[serhiy-storchaka]

Very well. But xgettext supports also --add-comments without argument.

[serhiy-storchaka]

How does xgettext handle multiple #s?

[tomasr8]

xgettext extracts all of the following comments, while babel does not extract any:

## i18n: comment
_('foo')

# # i18n: comment
_('bar')

## #    #  i18n: comment
_('thud')

I think we can be permissive here and follow what xgettext does.

[tomasr8]

I made --add-comments optional. It extracts all comments when no tag is specified. I also added a test for it.
I also process comments only when --add-comments is specified.
To match the behaviour of xgettext, any leading # and white space is stripped from comments before checking for a comment tag.

Would you mind taking another look?

[serhiy-storchaka]

Thanks.

Could you add tests for multiple --add-comments with different tags? And test the same file without --add-comments?

[serhiy-storchaka]

Or token.string.lstrip('# \t') if you prefer.

[tomasr8]

I added both tests :) Is this what you had in mind?

[serhiy-storchaka]

I actually thought about using the same source for multiple tests: without --add-comments, --add-comments without argument, single --add-comments with tag, multiple --add-comments with different tags. But the current tests are fine too.

[tomasr8]

I actually thought about using the same source for multiple tests: without --add-comments, --add-comments without argument, single --add-comments with tag, multiple --add-comments with different tags. But the current tests are fine too.

Got it! Thanks for the clarification :)

[serhiy-storchaka]

LGTM. 👍

[serhiy-storchaka]

Thank you for your contribution @tomasr8. I did not expect that you implement this so fast.

[tomasr8]

Thanks for the review @serhiy-storchaka!