bpo-36654: add example to generate token from another file

[Windsooon]

Add example to generate token from another file.

https://bugs.python.org/issue36654

[eamanu]

This example confuse me, because the file example.py does not exist. Maybe you need to do something like hello.py

[berkerpeksag]

Using tokenize.open() would be better as it automatically detect the encoding of the file:

cpython/Lib/tokenize.py

Line 449 in 5d90954

encoding, lines = detect_encoding(buffer.readline)

[berkerpeksag]

Also, I agree with Emmanuel that using the existing hello.py as an example would make the example easier to follow.

[Windsooon]

I found the tokenize function already call detect_encoding.

[berkerpeksag]

I just realized that tokenize.open() returns a text stream, so passing it to tokenize.tokenize() wouldn't work.

We can use tokenize.generate_tokens() to demonstrate how the str API works:

import tokenize

with tokenize.open('hello.py') as f:
    token_gen = tokenize.generate_tokens(f.readline)
    for token in token_gen:
        print(token)

Then fallback to your example to show the usage of the bytes API:

import tokenize

with open('hello.py', 'rb') as f:
    token_gen = tokenize.tokenize(f.readline)
    for token in token_gen:
        print(token)

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[matrixise]

Hi @Windsooon

Could you update your PR with the last comment of @berkerpeksag and with the with statement?

Thank you

[matrixise]

Thanks for your contribution but you have to update your PR with the recommendations of @berkerpeksag

[Windsooon]

Hello, @matrixise, I would love to update the example if needed. However, Serhiy Storchaka didn't agree with it

I do not think a new example is needed. The existing example already demonstrates the use of file's readline method.

https://bugs.python.org/issue36654

[berkerpeksag]

I think Serhiy's comment was about the current form of the PR. In my last comment, I was proposing to add examples for both bytes and unicode APIs of the tokenize module.

As a core developer, even I missed that generate_token() was made public. I'm pretty sure more people would like to know that the module now has an API to pass unicode input.

[csabella]

@Windsooon, any updates? Thanks!

[Windsooon]

Sure, I will fix it in two days. Thank you for reminding.

[Windsooon]

Hi, @csabella. I just updated the PR base on @berkerpeksag 's suggestion.

[berkerpeksag]

Looks pretty good to me, thank you!

Comments have been addressed.

[bedevere-bot]

@berkerpeksag: Please replace # with GH- in the commit message next time. Thanks!

[miss-islington]

Thanks @Windsooon for the PR, and @berkerpeksag for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8.
🐍🍒⛏🤖

[bedevere-bot]

GH-18187 is a backport of this pull request to the 3.8 branch.

[bedevere-bot]

GH-18188 is a backport of this pull request to the 3.7 branch.