BUG: unicode characters when reading JSON lines #15149
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jreback
reviewed
Jan 18, 2017
| @@ -959,6 +959,25 @@ def test_read_jsonl(self): | |||
| expected = DataFrame([[1, 2], [1, 2]], columns=['a', 'b']) | |||
| assert_frame_equal(result, expected) | |||
|
|
|||
| # GH15132: ascii input of non-ascii unicode characters for Python 2 | |||
There was a problem hiding this comment.
make this a separatly named test
jreback
reviewed
Jan 18, 2017
| expected = DataFrame([[u"foo\u201d", "bar"], ["foo", "bar"]], | ||
| columns=['a', 'b']) | ||
| assert_frame_equal(result, expected) | ||
|
|
There was a problem hiding this comment.
add the same exact test for PY3
jreback
reviewed
Jan 18, 2017
| assert_frame_equal(result, expected) | ||
|
|
||
| # simulate string | ||
| json = ('{"a": "foo\xe2\x80\x9d", "b": "bar"}\n' |
There was a problem hiding this comment.
this should work in PY3 as well
There was a problem hiding this comment.
Thanks for the feedback, @jreback.
To make this work for PY3, is it OK to include # -*- coding: utf-8 -*- at the top of the pandas/io/tests/json/test_pandas.py file? If so, I can simply use the unicode character directly in the json string.
# -*- coding: utf-8 -*-
...
json = '{"a": "foo”", "b": "bar"}\n{"a": "foo", "b": "bar"}\n' # note the ” characterIf not, I would suggest using a bytes string that is utf-8 decoded if PY3:
json = b'{"a": "foo\xe2\x80\x9d", "b": "bar"}\n{"a": "foo", "b": "bar"}\n'
if compat.PY3:
# PY3 will not interpret the ascii-equivalent \xe2\x80\x9d correctly without decoding.
json = json.decode('utf-8')There was a problem hiding this comment.
yep, you can include the coding (we do it in some files already)
Fixes UnicodeDecodeError bug when reading JSON lines input with Ascii decoder, which is often the default setting in Python 2.7. Avoids issues with mixing unicode and ascii strings. xref pandas-dev#15132
rouzazari
force-pushed
the
GH_15132_json_lines_with_unicode_chars_py2
branch
from
fd5184f to
e117889
Compare
Current coverage is 85.54% (diff: 100%)@@ master #15149 diff @@
==========================================
Files 145 145
Lines 51404 51404
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 43975 43976 +1
+ Misses 7429 7428 -1
Partials 0 0
|
|
lgtm. ping on green. |
|
@jreback: all green here. |
|
thanks! |
AnkurDedania
pushed a commit
to AnkurDedania/pandas
that referenced
this pull request
Mar 21, 2017
Fixes UnicodeDecodeError bug when reading JSON lines input with Ascii decoder, which is often the default setting in Python 2.7. Avoids issues with mixing unicode and ascii strings. closes pandas-dev#15132 Author: Rouz Azari <[email protected]> Closes pandas-dev#15149 from rouzazari/GH_15132_json_lines_with_unicode_chars_py2 and squashes the following commits: e117889 [Rouz Azari] BUG: unicode characters when reading JSON lines
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes UnicodeDecodeError bug when reading JSON lines input with Ascii decoder, which is often the default setting in Python 2.7. Avoids issues with mixing unicode and ascii strings.
xref #15132
git diff upstream/master | flake8 --diff