Skip to content

Remove ujson dependency and use python's json module #508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

rumpelsepp
Copy link

The ujson dependency ships a C-extension via a wheel which could cause unnecessary problems due to different versions of glibc in different environments, especially on NixOS:

import ujson as json
ImportError: /nix/store/9v5d40jyvmwgnq1nj8f19ji2rcc5dksd-glibc-2.37-45/lib/libc.so.6: version `GLIBC_2.38' not found (required by /nix/store/myw67gkgayf3s2mniij7zwd79lxy8v0k-gcc-12.3.0-lib/lib/libstdc++.so.6)

I propose removing this dependency, since I claim that the performance gain it did more than four years ago (added in 1c0c540) is not that much relevant any more. Here is a benchmark of several json implementations: https://github.com/TkTech/json_benchmark. The builtin json module is even faster than ujson in 50% of their benchmarks.

I claim, replacing ujson with the standard json module does not make any difference in practice; but it avoids compatability issues.

The ujson dependency ships a C-extension via a wheel which could
cause unnecessary problems due to different versions of glibc in
different environments.
rumpelsepp referenced this pull request in Fraunhofer-AISEC/gallia Jan 4, 2024
@ccordoba12
Copy link
Member

I'm sorry but I'm going to say no to this due to the following:

  • We're not interested in supporting really old Linux distros, you are. So, you can patch our pyproject.toml to remove the ujson dependency and then this package will use simply json.
  • Any gain in speed we can obtain from a more performant Json library is really important for us. According to the benchmarks you referenced, I think we could switch to orjson.

@ccordoba12 ccordoba12 closed this Jan 19, 2024
@rumpelsepp
Copy link
Author

Thanks for your answer. For the sake of correctness:

We're not interested in supporting really old Linux distros, you are

This is wrong. I did not propose supporting really old Linux distros, but very recent ones by not relying on libstdc++. Anyway, switching to orjson should also fix those problems as it relies on Rust instead. From my experience this causes fewer compatability issues with libc than C++ dependencies.

@ccordoba12
Copy link
Member

Ok, sorry for the misunderstanding. I didn't know ujson is based on C++, which is causing the problem you reported.

Could you help us then to switch to orjson instead? I'll be happy to merge that PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants