Skip to content

fix: new ISR cache handling to resolve regression in 13.4 #2165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jun 16, 2023
Merged

Conversation

orinokai
Copy link
Contributor

@orinokai orinokai commented Jun 15, 2023

Description

  • Remove previous server patching for disabling Next.js cache and middleware origin handling
  • Force on-demand revalidation on every request so that Next always serves fresh content and disables middleware, which means we can handle caching with ODBs and run middleware on the edge

Tests

Existing tests should run as before

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for netlify-plugin-nextjs-nx-monorepo-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-nx-monorepo-demo/deploys/648c6e850e1b3c00085bda0a
😎 Deploy Preview https://deploy-preview-2165--netlify-plugin-nextjs-nx-monorepo-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for netlify-plugin-nextjs-static-root-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-static-root-demo/deploys/648c6e857888230008ff1c9f
😎 Deploy Preview https://deploy-preview-2165--netlify-plugin-nextjs-static-root-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for netlify-plugin-nextjs-export-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-export-demo/deploys/648c6e8538ca6d0008233c92
😎 Deploy Preview https://deploy-preview-2165--netlify-plugin-nextjs-export-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added the type: bug code to address defects in shipped code label Jun 15, 2023
@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for netlify-plugin-nextjs-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-demo/deploys/648c6e8539e39800097867a9
😎 Deploy Preview https://deploy-preview-2165--netlify-plugin-nextjs-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for netlify-plugin-nextjs-next-auth-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/netlify-plugin-nextjs-next-auth-demo/deploys/648c6e85d4d56d0008741c86
😎 Deploy Preview https://deploy-preview-2165--netlify-plugin-nextjs-next-auth-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for nextjs-plugin-custom-routes-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/nextjs-plugin-custom-routes-demo/deploys/648c6e8549d83100088889ac
😎 Deploy Preview https://deploy-preview-2165--nextjs-plugin-custom-routes-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for next-i18next-demo ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/next-i18next-demo/deploys/648c6e8508899000088b0f76
😎 Deploy Preview https://deploy-preview-2165--next-i18next-demo.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for next-plugin-edge-middleware ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/next-plugin-edge-middleware/deploys/648c6e85b02cf5000835497d
😎 Deploy Preview https://deploy-preview-2165--next-plugin-edge-middleware.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Jun 15, 2023

Deploy Preview for next-plugin-canary ready!

Name Link
🔨 Latest commit 6565c8a
🔍 Latest deploy log https://app.netlify.com/sites/next-plugin-canary/deploys/648c6e857429b60008f7191b
😎 Deploy Preview https://deploy-preview-2165--next-plugin-canary.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@nickytonline
Copy link

Got all the tests passing after several reruns except for two which are definitely related to the changes. Looking into these valid failures.

Dynamic Routing
    ✓ renders correct page via SSR on a dynamic route (183ms)
    ✓ renders correct page via SSR on a dynamic catch-all route (168ms)
    ✓ serves correct static file on a prerendered dynamic route with fallback: false (391ms)
    ✓ renders custom 404 on a non-prerendered dynamic route with fallback: false (192ms)
    ✓ serves correct static file on a prerendered dynamic route with fallback: true (276ms)
    1) renders fallback page via ODB on a non-prerendered dynamic route with fallback: true
    ✓ serves correct static file on a prerendered dynamic route with fallback: blocking (337ms)
    ✓ renders correct page via ODB on a non-prerendered dynamic route with fallback: blocking (153ms)
    ✓ renders correct page via ODB on a prerendered dynamic route with revalidate and fallback: false (169ms)
    ✓ renders custom 404 on a non-prerendered dynamic route with revalidate and fallback: false ([141](https://github.com/netlify/next-runtime/actions/runs/5281372992/jobs/9561084169?pr=2165#step:10:142)ms)
    ✓ renders correct page via ODB on a prerendered dynamic route with revalidate and fallback: true (161ms)
    2) renders fallback page via ODB on a non-prerendered dynamic route with revalidate and fallback: true
    ✓ renders correct page via ODB on a prerendered dynamic route with revalidate and fallback: blocking (166ms)
    ✓ renders correct page via ODB on a non-prerendered dynamic route with revalidate and fallback: blocking (220ms)


  15 passing (6s)
  2 failing

  1) Dynamic Routing
       renders fallback page via ODB on a non-prerendered dynamic route with fallback: true:
     AssertionError: expected '<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width"/><meta name="next-head-count" content="2"/><link rel="preload" href="/_next/static/css/3d[149](https://github.com/netlify/next-runtime/actions/runs/5281372992/jobs/9561084169?pr=2165#step:10:150)a316ef6d56a.css" as="style"/><link rel="stylesheet" href="/_next/static/css/3d149a316ef6d56a.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-0a2cfcb64704a81c.js" defer=""></script><script src="/_next/static/chunks/main-404c5b460e8a94b0.js" defer=""></script><script src="/_next/static/chunks/pages/_app-fda6929ee941cd57.js" defer=""></script><script src="/_next/static/chunks/9097-5a56fa994ecb9bb8.js" defer=""></script><script src="/_next/static/chunks/pages/getStaticProps/withFallback/%5Bid%5D-89c020c90a9e929e.js" defer=""></script><script src="/_next/static/build-id/_buildManifest.js" defer=""></script><script src="/_next/static/build-id/_ssgManifest.js" defer=""></script></head><body><div id="__next"><div class="prose ml-14 mt-10 mb-20"><div>Loading...</div></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{},"__N_SSG":true},"page":"/getStaticProps/withFallback/[id]","query":{},"buildId":"build-id","isFallback":true,"gsp":true,"locale":"en","locales":["en","es","fr"],"defaultLocale":"en","scriptLoader":[]}</script></body></html><div data-netlify-deploy-id="648b3e21c00386000861b568" data-netlify-site-id="b7ec5732-32dc-4d00-9108-051d100b6d2e" data-vcs="github" style="position:fixed">\n  <!-- This div is automatically inserted by Netlify for all Deploy Preview URLs. -->\n  <script async src="https://netlify-cdp-loader.netlify.app/netlify.js"></script>\n</div>\n' to include 'Bitten'
      at Context.eval (webpack:///../e2e/default/dynamic-routes.cy.ts:70:26)

  2) Dynamic Routing
       renders fallback page via ODB on a non-prerendered dynamic route with revalidate and fallback: true:

      AssertionError: expected { Object (age, cache-control, ...) } to have property 'x-nf-render-mode' of 'odb ttl=60', but got 'odb'
      + expected - actual

      -'odb'
      +'odb ttl=60'
      
      at Context.eval (webpack:///../e2e/default/dynamic-routes.cy.ts:122:36)

@@ -386,82 +336,6 @@ export const getDependenciesOfFile = async (file: string) => {
return dependencies.files.map((dep) => resolve(dirname(file), dep))
}

const baseServerReplacements: Array<[string, string]> = [

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We no longer need the server patches her because we handle this in NetlifyNextServer now.

@@ -69,9 +69,6 @@ const makeHandler = ({ conf, app, pageRoot, NextServer, staticManifest = [], mod

// We don't want to write ISR files to disk in the lambda environment
conf.experimental.isrFlushToDisk = false
// This is our flag that we use when patching the source
// eslint-disable-next-line no-underscore-dangle
process.env._REVALIDATE_SSG = 'true'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was used for the server patches so is no longer required.

@@ -189,28 +187,6 @@ describe('files utility functions', () => {
})
})

describeCwdTmpDir('file patching', () => {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are obsolete as we no longer patch server files.

@@ -66,8 +66,7 @@ describe('Dynamic Routing', () => {
expect(res.status).to.eq(200)
// expect 'odb' until https://github.com/netlify/pillar-runtime/issues/438 is fixed
expect(res.headers).to.have.property('x-nf-render-mode', 'odb')
// expect 'Bitten' until the above is fixed and we can test for fallback 'Loading...' message
expect(res.body).to.contain('Bitten')
expect(res.body).to.contain('Loading...')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works now @orinokai. It was failing previously not finding Bitten which was expected pre the changes in this PR.

// eslint-disable-next-line no-underscore-dangle
if (headers['x-nf-builder-cache'] === 'revalidate' && !headers.__prerender_bypass) {
console.log(`Revalidating ${url}`)
if (!headers.__prerender_bypass) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gah, I realized last night that if we only revalidate when an ODB sends that header, then middleware will still be enabled for all other requests (because we're now relying on revalidate mode to disable middleware at the origin). This commit reverts that, so we now revalidate on every request and will have to find a different way to serve the initial static content (but perhaps that's for a separate PR, like you said).

@orinokai orinokai marked this pull request as ready for review June 16, 2023 14:16
@orinokai orinokai requested a review from a team as a code owner June 16, 2023 14:16
Copy link
Contributor

@MarcL MarcL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we just leaning on the E2E tests here or do we need any other unit/integration tests?
Assume it might be the case that we've not got the testing strategy for these.

It's great that we've removed the patching of the server.

@nickytonline nickytonline merged commit 6817cc9 into main Jun 16, 2023
@nickytonline nickytonline deleted the rs/isr-fix branch June 16, 2023 15:05
@orinokai
Copy link
Contributor Author

orinokai commented Jun 16, 2023

Are we just leaning on the E2E tests here or do we need any other unit/integration tests? Assume it might be the case that we've not got the testing strategy for these.

We are leaning on the E2E tests for now, which should be fine, but the next bit of 13.4 work will be fixing the on-demand revalidation and that includes tests for the server code in this PR. So I'll be updating those tests during that work.

@MarcL
Copy link
Contributor

MarcL commented Jun 16, 2023

That's @orinokai. Great work from you @nickytonline! 🎉

@charkour
Copy link

Thanks for working on this! I'm curious, why was patching Next Server necessary in the first place?

@charkour
Copy link

I think this resolves the regression as far back as next v13.2

@orinokai
Copy link
Contributor Author

Thanks for working on this! I'm curious, why was patching Next Server necessary in the first place?

hi @charkour, patching the Next server was previously necessary to disable the internal ISR cache so that we can handle caching on our edge CDN, but thankfully we can now achieve this by forcing on-demand revalidation, which is more reliable - and yep, 13.2 sounds right because that is when Next introduced their new caching strategy

@charkour
Copy link

charkour commented Jul 1, 2023

Roger, thanks for the explanation! That's helpful to know

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug code to address defects in shipped code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants