use of iloc with heterogeneous DataFrame coerces dtype?

[jason-s]

I'm using pandas 0.11.0 and numpy 1.7.1 in Anaconda Python 1.6.0.

The iloc method seems to coerce dtype of values in some cases. For example:

x5 = pd.DataFrame(index=['A','B','C'],columns=['foo','bar','baz'])
x5.foo = np.int16([-1,-2,-3])
x5.bar = np.uint16([1,2,3])
x5.baz = np.uint32([100000,200000,300000])
print x5
print x5.dtypes

   foo  bar     baz
A   -1    1  100000
B   -2    2  200000
C   -3    3  300000
foo     int16
bar    uint16
baz    uint32
dtype: object

So far, so good. Now using iloc, it seems to cause type coercion when a single row is selected:

print x5.iloc[1:2]
   foo  bar     baz
B   -2    2  200000

print x5.iloc[1]
foo    4294967294
bar             2
baz        200000
Name: B, dtype: uint32

[jason-s]

Just upgraded to pandas 0.12.0, numpy 1.7.1, under Anaconda 1.7.0. The iloc coercion behavior is different (now appears more correct), but it still occurs:

print x5.iloc[-1]
foo        -3
bar         3
baz    300000
Name: C, dtype: int64

[jreback]

that is a correct coercion
by definition it has to pick a type that can hold the sub elements because it's a series
the DataFrame will preserve

[jason-s]

So is this the best correct way to get the values out in an iterable format that preserves their types:

lastrow = x5.tail(1).itertuples(index=False).next()
print lastrow
print [type(val) for val in lastrow]

(-3, 3, 300000)
[<type 'numpy.int16'>, <type 'numpy.uint16'>, <type 'numpy.uint32'>]

[jreback]

what are you doing with the values?

the dtypes are column based

so best to operate that way

is there a reason you actually need these dtypes like this?

[jason-s]

is there a reason you actually need these dtypes like this?

yes, they are fixed-point integers representing raw values from a data acquisition system, which are going straight into an PyTables HDF5 file. Some of the data are quantized analog samples, others are digital values. Preservation of the original format is important for auditing purposes.

They come in batches, and I'm just trying to process the last line of each batch for a UI update.

[jreback]

HDFStore will properly serialize this
but if u need a list then your approach will work