differences in Series.map with defaultdict with different dtypes

[dannyi96]

• closes BUG: differences in Series.map with defaultdict with different dtypes #48813
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[mroeschke]

Does this fix the defaultdict mutation issue? #47527

[dannyi96]

Does this fix the defaultdict mutation issue? #47527

Just tested this ( & followed the older threads on this issue ).
Yes, these changes does fix the defaultdict mutation issue.

Eg -
Before changes -

s = pd.Series([1, 2, np.nan])
default_map = defaultdict(lambda: "missing", {1: "a", 2: "b", np.nan: "c"})

s.map(default_map) 
# 0              a
# 1              b
# 2    missing
# dtype: object

# defaultdict is mutated
default_map 
# defaultdict(<function <lambda> at 0x7f3e792833a0>, {1: 'a', 2: 'b', nan: 'c', nan: 'missing'})

After changes -

s = pd.Series([1, 2, np.nan])
default_map = defaultdict(lambda: "missing", {1: "a", 2: "b", np.nan: "c"})

s.map(default_map) 
# 0    a
# 1    b
# 2    c
# dtype: object

# defaultdict is not mutated
default_map 
# defaultdict(<function <lambda> at 0x7fe002b773a0>, {1: 'a', 2: 'b', nan: 'c'})

[dannyi96]

this older testcase seemed incorrect & needed to be corrected with this change.
Am however unsure if the older behavior is as per intention.

[phofl]

According to the discussion we had in #47585 the old behavior is correct

[dannyi96]

ahh I see.
Do we keep it as it is now ? or is this enhancement fine - since there are some discrepancies observed (based on dtype) in map behavior as highlighted in bug description #48813
TIA

[phofl]

cc @rhshadrach can you weigh in?

[rhshadrach]

+1 on changing the behavior in this test. The 0 in question here is because np.nan does not equal itself, and NumPy often returns views so that ids are not equal either; e.g.

mapping = defaultdict(int, {1: 10, np.nan: 42})
arr = np.array([1, np.nan, 2])
print(mapping[arr[1]])

# 0

I think introducing a better lookup for NaN values makes sense, and brings this in line with the Series case:

mapping = pd.Series({1: 10, np.nan: 42})
ser = Series([1, np.nan, 2])
print(ser.map(mapping))

# 0    10.0
# 1    42.0
# 2     NaN
# dtype: float64

[mroeschke]

Yes, these changes does fix the defaultdict mutation issue.

Could you add a test to check that the default dict isn't mutated?

[mroeschke]

cc @phofl since you made a fix here recently

[dannyi96]

Yes, these changes does fix the defaultdict mutation issue.

Could you add a test to check that the default dict isn't mutated?

Updated the PR with a testcase to check defaultdict mutation.

[mroeschke]

LGTM. @phofl or @rhshadrach merge when ready

[phofl]

Thx @dannyi96