[BUG] Malformed UTF-8 Characters in Query Parameters starting from v3.5.6

[lamminsalo]

Bug Report: Malformed UTF-8 Characters in Query Parameters in v3.5.6

Description
After v3.5.6, UTF-8 characters (e.g., Nordic characters like äöå) in query parameters are being malformed into � in deployed environments. This issue only occurs when navigating within the application to a cached page with UTF-8 encoded query parameters. Reverting to v3.5.5 resolves the issue.

The issue is specific to deployed environments and is not reproducible locally, making debugging and reproduction more challenging.

---

Expected Behavior
When navigating to a page with UTF-8 encoded URL parameters, the request should return:

• HTTP 304: "Not Modified" with the query parameter q=äää.

Example:

GET /search?q=äää
Response: 304 Not Modified
Query: q=äää

---

Actual Behavior
Instead of returning the cached page, navigating to a cached page results in a redirect along with malformed query parameters:

• HTTP 308: Permanent redirect with query parameters encoded incorrectly.

Malformed Example:

GET /search?q=äää
Response: 308 Permanent Redirect
To: /_next/data/[hash]/search.json?q=%C3%A4%C3%A4%C3%A4
Actual Query: q=��� (Malformed)

---

Steps to Reproduce

1. Deploy the reproduction example linked below.
2. Navigate to /search?q=äöå€ using the bottom button (direct navigation from URL works)
3. Observe the malformed query parameters after a redirect.

---

Minimal Reproduction

• Code: https://github.com/lamminsalo/Opennext-Query-Parameter-Minimal-Reproduction
• Live Environment: https://d16ppgwd8iy16k.cloudfront.net/

---

Discord Discussion
For additional context, see the ongoing thread on Discord:
https://discord.com/channels/1283128968140161065/1286094576788177059/1367919144694972569

---

Reproduction Environment

• Version: opennext v3.6.0

Manually tested that v3.5.6 is the breaking update.

---

EDIT: Added a more minimal reproduction repo and deployed it. Links and steps updated.

[sommeeeer]

As stated in the Discord thread the issue is the Location header returned here:

opennextjs-aws/packages/open-next/src/core/routing/matcher.ts

Line 343 in 9411b7a

Location: internalEvent.url,

It should be encoded, which it is not >= 3.5.6. We need to figure out what in this commit broke it.

[sommeeeer]

It seems to be this function:

opennextjs-aws/packages/open-next/src/core/routing/util.ts

Lines 126 to 137 in 9411b7a

	export function convertToQueryString(query: Record<string, string | string[]>) {
	const queryStrings: string[] = [];
	Object.entries(query).forEach(([key, value]) => {
	if (Array.isArray(value)) {
	value.forEach((entry) => queryStrings.push(`${key}=${entry}`));
	} else {
	queryStrings.push(`${key}=${value}`);
	}
	});
	return queryStrings.length > 0 ? `?${queryStrings.join("&")}` : "";
	}

In the versions prior to < 3.5.6 it was using URLSearchParams which will automatically encode the query parameters. What I think we need to do now is actually wrap encodeURIComponent() around the value and entry in that function, however im not sure if that would possibly break 533 and 537 in the opennextjs-cloudflare repo.

[conico974]

Yeah, you can't just revert to URLSearchParams without breaking those. BTW this was not only broken in cloudflare

[sommeeeer]

This PR with this commit had this line:

opennextjs-aws/packages/open-next/src/core/routing/matcher.ts

Line 208 in 77d87e7

const encodePlusQueryString = queryString.replaceAll("+", "%20");

If I add it back and also have this as convertToQueryString():

export function convertToQueryString(query: Record<string, string | string[]>) {
  const searchParams = new URLSearchParams();
  Object.entries(query).forEach(([key, value]) => {
    if (Array.isArray(value)) {
      value.forEach((entry) => searchParams.append(key, decodeURIComponent(entry)));
    } else {
      searchParams.append(key, decodeURIComponent(value));
    }
  });

  const queryString = searchParams.toString();
  return queryString.length > 0 ? `?${queryString}` : "";
}

E2E runs fine, but one unit test will fail:

FAIL  packages/tests-unit/tests/core/routing/util.test.ts > convertToQueryString > should respect existing query encoding                              
                                                                                                                                                  
Expected: "?key=value%201&key=value2+something+else&another=value3"                                                                               
Received: "?key=value+1&key=value2%2Bsomething%2Belse&another=value3" 

However, if you have a redirect with this:

const nextConfig: NextConfig = {
  redirects: async () => [
    {
      source: '/baby',
      destination: '/foo?bar=value%201&bar=value2+something+else&baz=value3',
      permanent: false,
    },
  ],
};

it will give you a Location with http://localhost:3000/foo?bar=value+1&bar=value2+something+else&baz=value3.

[conico974]

Fixed in #871