BUG: ENH: Allow callable for on_bad_lines in read_csv when engine="python" #45146

[kevinhodes]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

lst = []


def failed_line(bad_line: list[str]) -> list[str]:
    print("failed line")
    lst.append(bad_line)
    return lst


if __name__ == '__main__':
    dtypes = {
        "CV_CCT": object,
        "C_NOMBRE": object,
        "CV_TIPO": float,
        "C_TIPO": object,
        "CV_ADMINISTRATIVA": object,
        "C_ADMINISTRATIVA": object,
        "CV_ESTATUS": int,
        "C_ESTATUS": object,
        "F_FUNDACION": object,
        "INMUEBLE_CV_INMUEBLE": object,
        "INMUEBLE_CV_VIALIDAD_PRINCIPAL": int,
        "INMUEBLE_C_VIALIDAD_PRINCIPAL": object,
        "INMUEBLE_CV_VIALIDAD_DERECHA": int,
        "INMUEBLE_C_VIALIDAD_DERECHA": object,
        "INMUEBLE_CV_VIALIDAD_IZQUIERDA": int,
        "INMUEBLE_C_VIALIDAD_IZQUIERDA": object,
        "INMUEBLE_CV_VIALIDAD_POSTERIOR": int,
        "INMUEBLE_C_VIALIDAD_POSTERIOR": object,
        "INMUEBLE_N_EXTNUM": int,
        "INMUEBLE_C_EXTALF": object,
        "INMUEBLE_N_INTNUM": int,
        "INMUEBLE_C_INTALF": object,
        "INMUEBLE_CV_ENT": object,
        "INMUEBLE_C_NOM_ENT": object,
        "INMUEBLE_CV_MUN": object,
        "INMUEBLE_C_NOM_MUN": object,
        "INMUEBLE_CV_LOC": object,
        "INMUEBLE_C_NOM_LOC": object,
        "INMUEBLE_CV_ASEN": object,
        "INMUEBLE_C_NOM_ASEN": object,
        "INMUEBLE_CV_CODIGO_POSTAL": object,
        "INMUEBLE_C_DESC_UBICACION": object,
        "INMUEBLE_LATITUD": float,
        "INMUEBLE_LONGITUD": float,
        "SOSTENIMIENTO_CV_CONTROL": int,
        "SOSTENIMIENTO_C_CONTROL": object,
        "SOSTENIMIENTO_CV_SUBCONTROL": int,
        "SOSTENIMIENTO_C_SUBCONTROL": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN1": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN1": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN2": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN2": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN3": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN3": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN4": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN4": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN5": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN5": object,
        "SOSTENIMIENTO_CV_SERVICIO": object,
        "SOSTENIMIENTO_C_SERVICIO": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN1": int,
        "DEPOPERATIVA_C_DEPENDENCIAN1": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN2": int,
        "DEPOPERATIVA_C_DEPENDENCIAN2": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN3": int,
        "DEPOPERATIVA_C_DEPENDENCIAN3": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN4": int,
        "DEPOPERATIVA_C_DEPENDENCIAN4": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN5": int,
        "DEPOPERATIVA_C_DEPENDENCIAN5": object,
        "CONTACTO_CV_CARGO": int,
        "CONTACTO_C_CARGO": object,
        "CONTACTO_CV_TIPODIRECTOR": int,
        "CONTACTO_C_TIPODIRECTOR": object,
        "CONTACTO_C_ASOCIACION": object,
        "CONTACTO_C_CURP": object,
        "CONTACTO_C_RFC": object,
        "CONTACTO_C_NOMBRE": object,
        "CONTACTO_C_APELLIDO1": object,
        "CONTACTO_C_APELLIDO2": object,
        "CONTACTO_C_TELEFONO": object,
        "CONTACTO_C_CELULAR": object,
        "CONTACTO_C_EMAIL": object,
        "CONTACTO_C_EXTENSION": object,
        "CONTACTO_C_PWEB": object,
        "SERREG_CV_CCT": object,
        "JEFSEC_CV_CCT": object,
        "SUPERVISION_CV_CCT": object,
        "C_TUNO_1": object,
        "C_TUNO_2": object,
        "C_TUNO_3": object,
        "TIPONIVELSUB_CV_SERVICION1": int,
        "TIPONIVELSUB_C_SERVICION1": object,
        "TIPONIVELSUB_CV_SERVICION2": int,
        "TIPONIVELSUB_C_SERVICION2": object,
        "TIPONIVELSUB_CV_SERVICION3": int,
        "TIPONIVELSUB_C_SERVICION3": object,
        "C_SERVICIO_CAM": object,
        "CARACTERISTCA_CV_CARACTERIZAN1": int,
        "CARACTERISTCA_C_CARACTERIZAN1": object,
        "CARACTERISTCA_CV_CARACTERIZAN2": int,
        "CARACTERISTCA_C_CARACTERIZAN2": object,
        "Unnamed: 92": object
    }

    df = pd.read_csv("cct15_bad.csv", encoding='ansi', dtype=dtypes, on_bad_lines=failed_line, engine='python')

Issue Description

failed_line() never gets fired.

Expected Behavior

failed_line() to get fired.

Installed Versions

Windows 10
python 3.10
pandas 1.4.1

[cct15_bad.zip](https://github.com/pandas-dev/pandas/files/8377144/cct15_bad.zip)

[twoertwein]

Up to a certain degree, read_csv seems to absorb unexpected columns as MultiIndex:

from io import StringIO

import pandas as pd

def fun(xs):
    return None

handle = StringIO("a\na,b\nc,d,e\nf,g,h")  # multiindex
print(pd.read_csv(handle, engine="python", on_bad_lines=fun))
#     a.1
# a b
# c d   e
# f g   h

handle = StringIO("a\na,b\nc,d,e\nf,g,h,i")  # calls fun (but also uses multiindex)
print(pd.read_csv(handle, engine="python", on_bad_lines=fun))
#     a.1
# a b
# c d   e

[phofl]

Could you provide a minimal and reproducible example?

https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[phofl]

@twoertwein You could supress this through index_col=False

[kevinhodes]

cct15_bad_minimal.csv
pandas_error.py.txt

You will have to rename pandas_error.py.txt to pandas_error.py.
Lines 2,3 and 4 represent the broken csv.

[phofl]

Could you create something that we can copy and paste?

[twoertwein]

@twoertwein You could supress this through index_col=False

Still seems to infer a multiindex (I updated my above example slightly):

handle = StringIO("a\na,b\nc,d,e\nf,g,h")  # multiindex
print(pd.read_csv(handle, engine="python", on_bad_lines=fun, index_col=False))
#     a.1
# a b
# c d   e
# f g   h
handle = StringIO("a\na,b\nc,d,e\nf,g,h,i")  # calls fun (but also uses multiindex)
print(pd.read_csv(handle, engine="python", on_bad_lines=fun, index_col=False))
#     a.1
# a b
# c d   e

[phofl]

Hm this seems off. I think the c engine works as expected, but would have to look closer

[kevinhodes]

• csv

CV_CCT,C_NOMBRE,CV_TIPO,C_TIPO,CV_ADMINISTRATIVA,C_ADMINISTRATIVA,CV_ESTATUS,C_ESTATUS,F_FUNDACION,INMUEBLE_CV_INMUEBLE,INMUEBLE_CV_VIALIDAD_PRINCIPAL,INMUEBLE_C_VIALIDAD_PRINCIPAL,INMUEBLE_CV_VIALIDAD_DERECHA,INMUEBLE_C_VIALIDAD_DERECHA,INMUEBLE_CV_VIALIDAD_IZQUIERDA,INMUEBLE_C_VIALIDAD_IZQUIERDA,INMUEBLE_CV_VIALIDAD_POSTERIOR,INMUEBLE_C_VIALIDAD_POSTERIOR,INMUEBLE_N_EXTNUM,INMUEBLE_C_EXTALF,INMUEBLE_N_INTNUM,INMUEBLE_C_INTALF,INMUEBLE_CV_ENT,INMUEBLE_C_NOM_ENT,INMUEBLE_CV_MUN,INMUEBLE_C_NOM_MUN,INMUEBLE_CV_LOC,INMUEBLE_C_NOM_LOC,INMUEBLE_CV_ASEN,INMUEBLE_C_NOM_ASEN,INMUEBLE_CV_CODIGO_POSTAL,INMUEBLE_C_DESC_UBICACION,INMUEBLE_LATITUD,INMUEBLE_LONGITUD,SOSTENIMIENTO_CV_CONTROL,SOSTENIMIENTO_C_CONTROL,SOSTENIMIENTO_CV_SUBCONTROL,SOSTENIMIENTO_C_SUBCONTROL,SOSTENIMIENTO_CV_DEPENDENCIAN1,SOSTENIMIENTO_C_DEPENDENCIAN1,SOSTENIMIENTO_CV_DEPENDENCIAN2,SOSTENIMIENTO_C_DEPENDENCIAN2,SOSTENIMIENTO_CV_DEPENDENCIAN3,SOSTENIMIENTO_C_DEPENDENCIAN3,SOSTENIMIENTO_CV_DEPENDENCIAN4,SOSTENIMIENTO_C_DEPENDENCIAN4,SOSTENIMIENTO_CV_DEPENDENCIAN5,SOSTENIMIENTO_C_DEPENDENCIAN5,SOSTENIMIENTO_CV_SERVICIO,SOSTENIMIENTO_C_SERVICIO,DEPOPERATIVA_CV_DEPENDENCIAN1,DEPOPERATIVA_C_DEPENDENCIAN1,DEPOPERATIVA_CV_DEPENDENCIAN2,DEPOPERATIVA_C_DEPENDENCIAN2,DEPOPERATIVA_CV_DEPENDENCIAN3,DEPOPERATIVA_C_DEPENDENCIAN3,DEPOPERATIVA_CV_DEPENDENCIAN4,DEPOPERATIVA_C_DEPENDENCIAN4,DEPOPERATIVA_CV_DEPENDENCIAN5,DEPOPERATIVA_C_DEPENDENCIAN5,CONTACTO_CV_CARGO,CONTACTO_C_CARGO,CONTACTO_CV_TIPODIRECTOR,CONTACTO_C_TIPODIRECTOR,CONTACTO_C_ASOCIACION,CONTACTO_C_CURP,CONTACTO_C_RFC,CONTACTO_C_NOMBRE,CONTACTO_C_APELLIDO1,CONTACTO_C_APELLIDO2,CONTACTO_C_TELEFONO,CONTACTO_C_CELULAR,CONTACTO_C_EMAIL,CONTACTO_C_EXTENSION,CONTACTO_C_PWEB,SERREG_CV_CCT,JEFSEC_CV_CCT,SUPERVISION_CV_CCT,C_TUNO_1,C_TUNO_2,C_TUNO_3,TIPONIVELSUB_CV_SERVICION1,TIPONIVELSUB_C_SERVICION1,TIPONIVELSUB_CV_SERVICION2,TIPONIVELSUB_C_SERVICION2,TIPONIVELSUB_CV_SERVICION3,TIPONIVELSUB_C_SERVICION3,C_SERVICIO_CAM,CARACTERISTCA_CV_CARACTERIZAN1,CARACTERISTCA_C_CARACTERIZAN1,CARACTERISTCA_CV_CARACTERIZAN2,CARACTERISTCA_C_CARACTERIZAN2,
15MSU0041R,UNIVERSIDAD TRES CULTURAS PLANTEL ATIZAPAN,13,RECTORIA,15,M�XICO,1,ACTIVO,12/06/2017 12:00:00 a. m.,15INMDD81C,1952865,NINGUNO NINGUNO,1952867,NINGUNO NINGUNO,1952865,NINGUNO NINGUNO,1952866,NINGUNO NINGUNO,3,NA,0,NA,15,M�XICO,013,ATIZAP�N DE ZARAGOZA,0001,CIUDAD L�PEZ MATEOS,0042,EL PEDREGAL DE ATIZAPAN,59200,CARR. PROGRESO INDUSTRIAL 3 COL EXHACIENDA EL PEDREGAL
ATIZAP�N DE ZARAGOZA ESTADO DE M�XICO
UBICADA EN PLAZA PUNTO ATIZAP�N,19.562542,-99.244010,2,PRIVADO,1,PRIVADO,1,INICIATIVA PRIVADA,1,ASOCIACI�N CIVIL,0, ,0,,0,,M01,EDUCACION SUPERIOR (RECTORIA - ADMINISTRACION ESCOLAR)                                                                                                                              ,1,PRESIDENCIA DE LA REP�BLICA,2,SECRETAR�A DE EDUCACI�N P�BLICA,3,SUBSECRETAR�A DE EDUCACI�N SUPERIOR,3,DIRECCI�N GENERAL DE EDUCACI�N SUPERIOR UNIVERSITARIA,0,,1,DIRECTOR O RESPONSABLE DEL CT,1,ENCARGADO,NA,RERD761113HDFYGG00,RERD761113,JESUS ONOFRE,LANZA,LOSA,5536017500,,diego.reyes@sep.gob.mx,,,,,,DISCONTINUO,,,0,,0,,0,,,0,,0,,
15MSU0042Q,CENTRO UNIVERSITARIO DEL VALLE DE TEOTIHUACAN-CUVATE-,13,RECTORIA,15,M�XICO,1,ACTIVO,,15INME033C,1997025,CALLE PALMA,1997026,CALLE CAPUL�N,1997027,AVENIDA PIR�MIDES,1997028,CALLE ATETELCO,13,NA,0,NA,15,M�XICO,092,TEOTIHUAC�N,0001,TEOTIHUAC�N DE ARISTA,0008,PURIFICACION,55810,,19.687419,-98.863006,2,PRIVADO,1,PRIVADO,1,INICIATIVA PRIVADA,1,ASOCIACI�N CIVIL,0, ,0,,0,,M01,EDUCACION SUPERIOR (RECTORIA - ADMINISTRACION ESCOLAR)                                                                                                                              ,1,PRESIDENCIA DE LA REP�BLICA,2,SECRETAR�A DE EDUCACI�N P�BLICA,3,SUBSECRETAR�A DE EDUCACI�N SUPERIOR,3,DIRECCI�N GENERAL DE EDUCACI�N SUPERIOR UNIVERSITARIA,0,,1,DIRECTOR O RESPONSABLE DEL CT,1,ENCARGADO,NA,RERD761113HDFYGG00,RERD761113,ARMANDO,GAYTAN,JUAREZ,5536017500,,diego.reyes@sep.gob.mx,,,,,,DISCONTINUO,,,0,,0,,0,,,0,,0,,

[kevinhodes]

--code

import pandas as pd

lst = []


def failed_line(bad_line: list[str]) -> list[str]:
    print("failed line")
    lst.append(bad_line)
    return lst


if __name__ == '__main__':
    dtypes = {
        "CV_CCT": object,
        "C_NOMBRE": object,
        "CV_TIPO": float,
        "C_TIPO": object,
        "CV_ADMINISTRATIVA": object,
        "C_ADMINISTRATIVA": object,
        "CV_ESTATUS": int,
        "C_ESTATUS": object,
        "F_FUNDACION": object,
        "INMUEBLE_CV_INMUEBLE": object,
        "INMUEBLE_CV_VIALIDAD_PRINCIPAL": int,
        "INMUEBLE_C_VIALIDAD_PRINCIPAL": object,
        "INMUEBLE_CV_VIALIDAD_DERECHA": int,
        "INMUEBLE_C_VIALIDAD_DERECHA": object,
        "INMUEBLE_CV_VIALIDAD_IZQUIERDA": int,
        "INMUEBLE_C_VIALIDAD_IZQUIERDA": object,
        "INMUEBLE_CV_VIALIDAD_POSTERIOR": int,
        "INMUEBLE_C_VIALIDAD_POSTERIOR": object,
        "INMUEBLE_N_EXTNUM": int,
        "INMUEBLE_C_EXTALF": object,
        "INMUEBLE_N_INTNUM": int,
        "INMUEBLE_C_INTALF": object,
        "INMUEBLE_CV_ENT": object,
        "INMUEBLE_C_NOM_ENT": object,
        "INMUEBLE_CV_MUN": object,
        "INMUEBLE_C_NOM_MUN": object,
        "INMUEBLE_CV_LOC": object,
        "INMUEBLE_C_NOM_LOC": object,
        "INMUEBLE_CV_ASEN": object,
        "INMUEBLE_C_NOM_ASEN": object,
        "INMUEBLE_CV_CODIGO_POSTAL": object,
        "INMUEBLE_C_DESC_UBICACION": object,
        "INMUEBLE_LATITUD": float,
        "INMUEBLE_LONGITUD": float,
        "SOSTENIMIENTO_CV_CONTROL": int,
        "SOSTENIMIENTO_C_CONTROL": object,
        "SOSTENIMIENTO_CV_SUBCONTROL": int,
        "SOSTENIMIENTO_C_SUBCONTROL": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN1": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN1": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN2": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN2": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN3": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN3": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN4": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN4": object,
        "SOSTENIMIENTO_CV_DEPENDENCIAN5": int,
        "SOSTENIMIENTO_C_DEPENDENCIAN5": object,
        "SOSTENIMIENTO_CV_SERVICIO": object,
        "SOSTENIMIENTO_C_SERVICIO": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN1": int,
        "DEPOPERATIVA_C_DEPENDENCIAN1": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN2": int,
        "DEPOPERATIVA_C_DEPENDENCIAN2": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN3": int,
        "DEPOPERATIVA_C_DEPENDENCIAN3": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN4": int,
        "DEPOPERATIVA_C_DEPENDENCIAN4": object,
        "DEPOPERATIVA_CV_DEPENDENCIAN5": int,
        "DEPOPERATIVA_C_DEPENDENCIAN5": object,
        "CONTACTO_CV_CARGO": int,
        "CONTACTO_C_CARGO": object,
        "CONTACTO_CV_TIPODIRECTOR": int,
        "CONTACTO_C_TIPODIRECTOR": object,
        "CONTACTO_C_ASOCIACION": object,
        "CONTACTO_C_CURP": object,
        "CONTACTO_C_RFC": object,
        "CONTACTO_C_NOMBRE": object,
        "CONTACTO_C_APELLIDO1": object,
        "CONTACTO_C_APELLIDO2": object,
        "CONTACTO_C_TELEFONO": object,
        "CONTACTO_C_CELULAR": object,
        "CONTACTO_C_EMAIL": object,
        "CONTACTO_C_EXTENSION": object,
        "CONTACTO_C_PWEB": object,
        "SERREG_CV_CCT": object,
        "JEFSEC_CV_CCT": object,
        "SUPERVISION_CV_CCT": object,
        "C_TUNO_1": object,
        "C_TUNO_2": object,
        "C_TUNO_3": object,
        "TIPONIVELSUB_CV_SERVICION1": int,
        "TIPONIVELSUB_C_SERVICION1": object,
        "TIPONIVELSUB_CV_SERVICION2": int,
        "TIPONIVELSUB_C_SERVICION2": object,
        "TIPONIVELSUB_CV_SERVICION3": int,
        "TIPONIVELSUB_C_SERVICION3": object,
        "C_SERVICIO_CAM": object,
        "CARACTERISTCA_CV_CARACTERIZAN1": int,
        "CARACTERISTCA_C_CARACTERIZAN1": object,
        "CARACTERISTCA_CV_CARACTERIZAN2": int,
        "CARACTERISTCA_C_CARACTERIZAN2": object,
        "Unnamed: 92": object
    }

    df = pd.read_csv("InputFiles/cct15_bad_minimal.csv", encoding='ansi', dtype=dtypes, on_bad_lines=failed_line, engine='python')

[phofl]

This is not minimal, please only provide as many columns as necessary. Not all of them

[phofl]

@twoertwein Getting back to this: This in fact is the expected behavior for the python engine. Header is set to infer -> we think that the first line is the column name and the second line are the names of the index names, because the 2 names plus the column name equal 3, the length of your data rows (it assumes that the mi headers are on the second line).

If you pass a third element to the second line, it shows a warning, also if you only pass element on the second line

Summary: If elements on first and second row together equal the number of elements on the remaining rows, we think column names and multiindex names are given on 2 rows, you should be able to override this with index_col=False, which does not work right now