BUG: groupby().size gives casting error on 32 bit platform

[jorisvandenbossche]

In [1]: df = pd.DataFrame({"id":[1,2,3,4,5,6], "grade":['a', 'b', 'b', 'a', 'a', 'e']})

In [2]: df.groupby("grade").size()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-d8e387418f9d> in <module>()
----> 1 df.groupby("grade").size()

/home/joris/scipy/pandas/pandas/core/groupby.pyc in size(self)
    818 
    819         """
--> 820         return self.grouper.size()
    821 
    822     sum = _groupby_function('sum', 'add', np.sum)

/home/joris/scipy/pandas/pandas/core/groupby.pyc in size(self)
   1380         """
   1381         ids, _, ngroup = self.group_info
-> 1382         out = np.bincount(ids[ids != -1], minlength=ngroup)
   1383         return Series(out, index=self.result_index)
   1384 

TypeError: Cannot cast array data from dtype('int64') to dtype('int32') according to the rule 'safe'

In [4]: pd.__version__
Out[4]: '0.17.0rc1+108.g3fb802a'

[jreback]

cc @behzadnouri

[jorisvandenbossche]

Hmm, although this is maybe more an issue with my numpy installation?
As I don't really see a reason why np.bincount would fail when getting int64 values (the docs says it needs integers and not floats, but does not say anything about integer size)

[jorisvandenbossche]

Possibly an issue with bincount itself: numpy/numpy#4366

[behzadnouri]

since ids are compressed, they cannot exceed memory address space. so i think it should be safe to cast on pandas side, i.e.: ids[ids != -1].astype(np.intp).

[behzadnouri]

with copy=False arg: ids[ids != -1].astype(np.intp, copy=False)

[jreback]

I setup a vm for linux-32

these break for the same reasons as well.
https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L2580
https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L1815

[behzadnouri]

both of them u added int64 not me. xref #10988

def group_info(self): indeed requires to return int64_t values but not def nunique. Either case, the right way to do it would be to leave internal computations alone, but only change the output as in:

diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
index e72f7c6..06b1105 100644
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1808,15 +1808,17 @@ class BinGrouper(BaseGrouper):
     @cache_readonly
     def group_info(self):
         ngroups = self.ngroups
-        obs_group_ids = np.arange(ngroups, dtype='int64')
+        obs_group_ids = np.arange(ngroups)
         rep = np.diff(np.r_[0, self.bins])

         if ngroups == len(self.bins):
-            comp_ids = np.repeat(np.arange(ngroups, dtype='int64'), rep)
+            comp_ids = np.repeat(np.arange(ngroups), rep)
         else:
-            comp_ids = np.repeat(np.r_[-1, np.arange(ngroups, dtype='int64')], rep)
+            comp_ids = np.repeat(np.r_[-1, np.arange(ngroups)], rep)

-        return comp_ids, obs_group_ids, ngroups
+        return comp_ids.astype('int64', copy=False),  \
+               obs_group_ids.astype('int64', copy=False),  \
+               ngroups

     @cache_readonly
     def ngroups(self):
@@ -2565,8 +2567,8 @@ class SeriesGroupBy(GroupBy):

         # group boundries are where group ids change
         # unique observations are where sorted values change
-        idx = com._ensure_int64(np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]])
-        inc = com._ensure_int64(np.r_[1, val[1:] != val[:-1]])
+        idx = np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]]
+        inc = np.r_[1, val[1:] != val[:-1]]

         # 1st item of each group is a new unique observation
         mask = isnull(val)
@@ -2577,7 +2579,7 @@ class SeriesGroupBy(GroupBy):
             inc[mask & np.r_[False, mask[:-1]]] = 0
             inc[idx] = 1

-        out = np.add.reduceat(inc, idx)
+        out = np.add.reduceat(inc, idx).astype('int64', copy=False)
         return Series(out if ids[0] != -1 else out[1:],
                       index=self.grouper.result_index,
                       name=self.name)

[jreback]

I changed them because they didn't work on window. I will try your diff and let u know.