PHPC-2262: Replace ICU dep with vendored utf8proc lib #1447

jmikola · 2023-07-18T18:29:05Z

https://jira.mongodb.org/browse/PHPC-2262

PHPC-2262: Replace ICU dep with vendored utf8proc lib

This introduces a --with-mongodb-utf8proc configure flag, which accepts "bundled" (default) and "system". Windows will always use bundled sources.

CheckICU.m4 and the --with-mongodb-icu configure flag have been removed.

Bumps libmongoc to 1.25-dev. This also adds support for a new MONGOC_ENABLE_SRV config constant, which is currently set based on detection in CheckResolv.m4. We can consider exposing that as a configure option down the line in PHPC-2256.

This introduces a --with-mongodb-utf8proc configure flag, which accepts "bundled" (default) and "system". Windows will always use bundled sources. CheckICU.m4 and the --with-mongodb-icu configure flag have been removed. Bumps libmongoc to 1.25-dev. This also adds support for a new MONGOC_ENABLE_SRV config constant, which is currently set based on detection in CheckResolv.m4. We can consider exposing that as a configure option down the line in PHPC-2256.

jmikola · 2023-07-18T19:52:38Z

config.m4

@@ -318,7 +318,11 @@ if test "$PHP_MONGODB" != "no"; then

  if test "$PHP_MONGODB_SYSTEM_LIBS" = "no"; then
    PHP_MONGODB_BUNDLED_CFLAGS="$STD_CFLAGS -DBSON_COMPILATION -DMONGOC_COMPILATION"
-    PHP_MONGODB_LIBMONGOCRYPT_CFLAGS="-DKMS_MSG_STATIC -DMLIB_USER"


I relocated this within if test "$PHP_MONGODB_CLIENT_SIDE_ENCRYPTION" = "yes"; since it's only referenced within that scope (i.e. building with bundled sources and libmongocrypt is enabled).

jmikola · 2023-07-18T19:53:27Z

config.m4

+    dnl Generated with: find src/libmongoc/src/utf8proc-2.8.0 -maxdepth 1 -name '*.c' ! -name 'utf8proc_data.c' -print0 | cut -sz -d / -f 5- | sort -dz | tr '\000' ' '
+    dnl Note: utf8proc_data.c is included from utf8proc.c and should not be compiled directly


I added ! -name 'utf8proc_data.c' to the find command in case someone decides to run this manually but update-submodule-sources.php has separate code to exclude utf8proc_data.c.

jmikola · 2023-07-18T19:54:02Z

config.m4

@@ -464,6 +479,8 @@ if test "$PHP_MONGODB" != "no"; then
    fi

    if test "$PHP_MONGODB_CLIENT_SIDE_ENCRYPTION" = "yes"; then
+      PHP_MONGODB_LIBMONGOCRYPT_CFLAGS="-DKMS_MSG_STATIC -DMLIB_USER"


Relocated from above.

jmikola · 2023-07-18T19:54:56Z

config.w32

@@ -86,7 +86,7 @@ if (PHP_MONGODB != "no") {
  ADD_EXTENSION_DEP("mongodb", "openssl", false);

  var PHP_MONGODB_CFLAGS="\
-    /D BSON_COMPILATION /D MONGOC_COMPILATION \
+    /D BSON_COMPILATION /D MONGOC_COMPILATION /D UTF8PROC_STATIC \


I didn't bother adding system library detection for utf8proc on Windows, so we always build with this constant.

Sounds good - I'm not sure whether we should rely on that library being available locally for people that download our DLLs, so compiling with bundled utf8proc seems like the logical choice.

jmikola · 2023-07-18T19:55:59Z

config.w32

-    }
-
-    ADD_FLAG("CFLAGS_MONGODB", "/EHsc /D U_USING_ICU_NAMESPACE=1");
+    mongoc_opts.MONGOC_ENABLE_SRV = 1;


This is conditionally enabled if we find dnsapi.lib. I assume the behavior is the same if we don't find it, and users will just get a runtime error when attempting to connect with a mongodb+srv:// URI.

jmikola · 2023-07-18T19:56:39Z

scripts/autotools/libmongoc/CheckResolv.m4

@@ -97,6 +100,10 @@ AC_LINK_IFELSE([AC_LANG_PROGRAM([[
   ])
 ])

+AS_IF([test "$found_resolv" = "yes"],[


Note: this is set independently of whether we need to link a separate resolv library or can rely on libc.

jmikola · 2023-07-18T19:59:33Z

scripts/autotools/libmongoc/CheckUtf8Proc.m4

+PHP_ARG_WITH([mongodb-utf8proc],
+             [whether to use system or bundled utf8proc for SCRAM-SHA-256 SASLprep],
+             [AS_HELP_STRING([--with-mongodb-utf8proc=@<:@bundled/system@:>@],
+                             [MongoDB: Enable utf8proc for SCRAM-SHA-256 SASLprep [default=bundled]])],


Previously, ICU was an optional dependency. IIRC, MONGOC_ENABLE_ICU could be undefined and libmongoc would conditionally raise a runtime exception if SASLprep was needed but not possible.

Since utf8proc is vendored, libmongoc provides no mechanism to disable it. We default to bundled sources here for consistency with libmongoc (their boolean build flags default to "on").

jmikola · 2023-07-18T20:03:58Z

scripts/autotools/libmongoc/CheckUtf8Proc.m4

+  PKG_CHECK_MODULES([PHP_MONGODB_UTF8PROC],[libutf8proc],[
+    PHP_MONGODB_BUNDLED_CFLAGS="$PHP_MONGODB_BUNDLED_CFLAGS $PHP_MONGODB_UTF8PROC_CFLAGS"
+    PHP_EVAL_LIBLINE([$PHP_MONGODB_UTF8PROC_LIBS],[MONGODB_SHARED_LIBADD])
+    found_utf8proc="yes"
+  ],[
+    PHP_CHECK_LIBRARY([utf8proc],
+                      [utf8proc_NFKC],
+                      [have_utf8proc_lib="yes"],
+                      [have_utf8proc_lib="no"])
+
+    AC_CHECK_HEADER([utf8proc.h],
+                    [have_utf8proc_headers=yes],
+                    [have_utf8proc_headers=no])
+
+    if test "$have_utf8proc_lib" = "yes" -a "$have_utf8proc_headers" = "yes"; then
+      PHP_ADD_LIBRARY([utf8proc],,[MONGODB_SHARED_LIBADD])
+      found_utf8proc="yes"
+    fi


Note: I tested both of these code paths locally.

$ pkg-config libutf8proc --libs -lutf8proc

pkg-config libutf8proc --cflags -DUTF8PROC_EXPORTS

Configure only shows -lutf8proc, but I do see -DUTF8PROC_EXPORTS added to $PHP_MONGODB_BUNDLED_CFLAGS when using pkg-config. I'm not sure if it's required for the second branch. /cc @joshbsiegel

I don't think UTF8PROC_EXPORTS is relevant, as it's only referenced in the following conditional for Windows:

https://github.com/JuliaStrings/utf8proc/blob/1cb28a66ca79a0845e99433fd1056257456cef8b/utf8proc.h#L123-L137

I agree. I didn't have to do anything with UTF8PROC_EXPORTS when making changes in the C driver.

jmikola requested a review from alcaeus July 18, 2023 18:29

jmikola force-pushed the phpc-2262 branch 2 times, most recently from 9626f14 to b91b1f5 Compare July 18, 2023 19:51

jmikola force-pushed the phpc-2262 branch from b91b1f5 to db27780 Compare July 18, 2023 19:57

jmikola commented Jul 18, 2023

View reviewed changes

alcaeus approved these changes Jul 19, 2023

View reviewed changes

jmikola merged commit d2a600b into mongodb:master Jul 19, 2023

jmikola deleted the phpc-2262 branch July 19, 2023 14:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PHPC-2262: Replace ICU dep with vendored utf8proc lib #1447

PHPC-2262: Replace ICU dep with vendored utf8proc lib #1447

Uh oh!

jmikola commented Jul 18, 2023 •

edited

Loading

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

alcaeus Jul 19, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 18, 2023

Uh oh!

jmikola Jul 19, 2023

Uh oh!

joshbsiegel Jul 19, 2023

Uh oh!

Uh oh!

		dnl Generated with: find src/libmongoc/src/utf8proc-2.8.0 -maxdepth 1 -name '*.c' ! -name 'utf8proc_data.c' -print0 \| cut -sz -d / -f 5- \| sort -dz \| tr '\000' ' '
		dnl Note: utf8proc_data.c is included from utf8proc.c and should not be compiled directly

PHPC-2262: Replace ICU dep with vendored utf8proc lib #1447

PHPC-2262: Replace ICU dep with vendored utf8proc lib #1447

Uh oh!

Conversation

jmikola commented Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jmikola commented Jul 18, 2023 •

edited

Loading