Error enabling offline persistence with Firebase Web SDK and SSR

[zaarheed]

I am trying to use the Google Firebase Web SDK (Firestore, Storage and Analytics) on a Next.js project hosted on Netlify with next-on-netlify.

I have some pages strictly using Firebase Firestore APIs once the page has loaded, and other pages are using Firestore APIs within the getServerSideProps() function. Still inconclusive, but it appears that the pages using Firestore within getServerSideProps are not working. There are two symptoms:

1. Pages timeout when trying to visit them through a browser, with the following returned:

{"errorMessage":"2020-10-28T23:52:17.325Z 4833e28b-d8c9-47bb-ad04-39d813a0b378 Task timed out after 10.01 seconds"}

2. If I navigate to the function logs in Netlify, where it appears to have generated a function for the SSR page, the logs are going crazy with the following line printed hundreds of times every minute:

11:51:22 PM: 2020-10-28T23:51:22.407Z	516720dc-661b-4f02-928d-6ca151031376	WARN	Error enabling offline persistence. Falling back to persistence disabled: Error: ENOENT: no such file or directory, open '/src/protos/google/firestore/v1/firestore.proto'

Is there a way around this, or am I doing something wrong?

Everything appears to be working fine on my local machine when I am developing using npm run dev - including SSR. Hence, this might be an issue related to Netlify-Next.js-Firebase.

I found this post on the Netlify community forums which suggests I am not the only one with the issue. But unlike that user, I am not using Netlify lambdas/functions nor am I used the firebase-admin SDK (server-side library). The resolution from this post is also inconclusive, citing: "Note that even though the write succeeded ... the function did not return a 200 and ‘Success!’… or a 500. It returned a 502 with a timeout error at the endpoint"

[FinnWoelm]

Hi @zaarheed,

Thanks for taking the time to post this issue and for the detailed description!

Can you try to use target: "experimental-serverless-trace" rather than target: "serverless" in your next.config.js? It should look like this:

// next.config.js

module.exports = {
  target: "experimental-serverless-trace",
};

The reason is that Firebase contains native binaries. In serverless-target mode, Next.js tries to inline all dependencies which does not work with binaries. In experimental-serverless-trace-target mode, Next.js does not inline dependencies, which allows Netlify to correctly bundle them into your functions. Longer explanation here (the Q focuses on Prisma, but it's the same issue of bundling binaries): #55 (comment)

Hopefully that will fix it! 😊 If not, we will dig deeper. We will definitely figure this out!

Cheers,
Finn

[zaarheed]

@FinnWoelm thanks for your thorough response.

I made the change in next.config.js and the Error enabling offline persistence. messages have stopped being printed in the function logs now.

However, trying to load a page that uses getServerSideProps() is still timing out with the message:

{"errorMessage":"2020-10-29T15:48:42.197Z 1ba3d3e0-dffc-4f07-b3c5-cf59f9fc5aae Task timed out after 10.01 seconds"}

Pages that do not use getServerSideProps() load and execute Firebase calls from the client fine.

FYI the function logs print the following:

3:51:24 PM: 2020-10-29T15:51:24.631Z	e1e8820d-49a0-4656-95f4-e73c2270cfd5	INFO	[request] /v/kgu0ec25kgu0ec263:51:34 PM: Duration: 10010.70 ms	Memory Usage: 110 MB	Init Duration: 548.71 ms	3:51:34 PM: 2020-10-29T15:51:34.640Z e1e8820d-49a0-4656-95f4-e73c2270cfd5 Task timed out after 10.01 seconds

So the request is working, and the function is being executed - but it's timing out. Any suggestions on how I might be able to debug this further? Running npm run dev and netlify dev (using the Netlify CLI) runs fine!

I've checked environment variables etc to weed out any silly mistakes but don't know what else I can try.

[lindsaylevine]

hey @zaarheed ! glad finn was able to try help out yesterday. have you seen this thread/comment? https://community.netlify.com/t/netlify-function-timeout-after-10-seconds-when-published/18220/5

seems like this is possibly unique to firebase <-> netlify functions. not sure. if that doesn't happen to help, i think the best next step for me and finn to be helpful would be for you to share a repo with us, but maybe finn has other ideas!

[zaarheed]

@lindsaylevine thanks for sharing that link!

Similar to the post I found on the Netlify forums, that developer is using firebase-admin within a Netlify Function/Lambda rather than firebase/firestore via the JavaScript Web SDK inside getServerSideProps().

I suppose the underlying issue might be the same. I’ll need to dig in to the source code to find an appropriate and equivalent way to close the connection - if there is one. I don’t know how these things work exactly, but it’s weird that it works on localhost but not on Netlify. Perhaps Firebase keeps an outbound connection open and Netlify interprets that as execution logic thus resulting in a timeout?

For now, I’ve solved my problem by refactoring everything to only use Firebase on the client side. Although it works, it would be nice to find a way to get Netlify/Next/Firebase to work together for SSR. I think it’s a super powerful stack esp for newcomers to React/Next.js like myself. The dev experience up until this SSR issue on Netlify has been excellent and so easy... to easy perhaps!

I haven’t had a chance to create a minimal repo to reproduce the issue just yet. But happy to invite you to my private repo if you want to deploy and try it out in the meantime.

[FinnWoelm]

Hey @zaarheed, happy that we're moving in the right direction!

Minimal Repo

I made a super minimal repo to reproduce the issue: https://github.com/FinnWoelm/next-on-netlify-firebase
Here is my page using getServerSideProps and firebase: https://github.com/FinnWoelm/next-on-netlify-firebase/blob/main/pages/test.js
You can see the "live" (and working 🎉) result here: https://next-on-netlify-firebase.netlify.app/test

I was able to reproduce your error regarding the function timeout. This seems to be a common issue when using firebase on AWS Lambda. I can't say that I fully understand what's going on, but something about an event loop that isn't empty and so AWS keeps waiting on it until the function times out, even if nothing more is happening. See: https://stackoverflow.com/a/39215697/6451879

Fixing the Issue

With a regular Netlify Function, the fix is really easy. Just add context.callbackWaitsForEmptyEventLoop = false as the first line in your lambda function. With next-on-netlify, it's a tiny bit more complicated, because we do not currently expose the context object inside Next.js pages (side note: we really should, though, I think. It would also take care of #20. We will discuss internally and see what's possible!).

In any case, what you can do in the meantime is to use your own patch! Are you familiar with patch-package?

You need to:

1. Install patch-package: npm install --save-dev patch-package
2. Copy my patch for next-aws-lambda (dependency used by next-on-netlify) to /patches: https://github.com/FinnWoelm/next-on-netlify-firebase/blob/main/patches/next-aws-lambda%2B2.5.0.patch
3. Run npx patch-package to apply the patch locally
4. Add a postinstall hook in your package.json: https://github.com/FinnWoelm/next-on-netlify-firebase/blob/82bb1c89c3048bae99533b4f5ef727d20a4bb7f7/package.json#L17 (this applies the patch when you push your app to Netlify's build pipeline)
5. You can now access the function's context object in your Next.js pages as shown below. Note that there is a context object (not what we want) and a functionContext object (which we want! Accessible via context.req.functionContext).

// See: https://github.com/FinnWoelm/next-on-netlify-firebase/blob/main/pages/test.js#L13-L23
export async function getServerSideProps(context) {
    // Get function context
  const functionContext = context?.req?.functionContext;

  // If we are currently in a Netlify function (deployed on netlify.app or
  // locally with netlify dev), do not wait for empty event loop.
  // See: https://stackoverflow.com/a/39215697/6451879
  // Skip during next dev.
  if (functionContext) {
    console.log("Setting callbackWaitsForEmptyEventLoop: false");
    functionContext.callbackWaitsForEmptyEventLoop = false;
  }

  // ... (run firebase queries, etc...)

  // return your data
  return { props: {} };
}

You can also have a look at this commit, which does the above steps.

Let me know if you run into any issues! In that case, please do feel free to share read access of your repo with us! 😊 And let us know if it ends up working, so that we can celebrate 😁

[emmbyiringiro]

Hello,

I followed the thread and try to implement the fix. unfortunately when I change target from serverless to experimental-serverless-trace build fail with following last log traces:

...

• 7:04:57 PM: (Netlify Build completed in 2m 29.1s)
• 7:04:57 PM: Caching artifacts
• 7:04:57 PM: Started saving node modules
• 7:04:57 PM: Finished saving node modules
• 7:04:57 PM: Started saving build plugins
• 7:04:57 PM: Finished saving build plugins
• 7:04:57 PM: Started saving yarn cache
• 7:04:57 PM: Finished saving yarn cache
• 7:04:57 PM: Started saving pip cache
• 7:04:57 PM: Finished saving pip cache
• 7:04:57 PM: Started saving emacs cask dependencies
• 7:04:57 PM: Finished saving emacs cask dependencies
• 7:04:57 PM: Started saving maven dependencies
• 7:04:57 PM: Finished saving maven dependencies
• 7:04:57 PM: Started saving boot dependencies
• 7:04:57 PM: Finished saving boot dependencies
• 7:04:57 PM: Started saving go dependencies
• 7:04:57 PM: Finished saving go dependencies
• 7:04:57 PM: Build script success
• 7:04:57 PM: Starting to deploy site from 'out_publish'
• 7:04:59 PM: Creating deploy tree
• 7:04:59 PM: Creating deploy upload records
• 7:04:59 PM: 3 new files to upload
• 7:04:59 PM: 3 new functions to upload
• 7:05:05 PM: Request must be smaller than 69905067 bytes for the CreateFunction operation
• 7:06:51 PM: Failed to upload file: next_index
• 7:06:51 PM: Failing build: Failed to deploy site
• 7:07:21 PM: Failed to inform the API about a failed build, please retry the build or contact support
• 7:07:21 PM: Finished processing build request in 5m20.501745492s

I don't upload any heavy file and application is lightweight. the issue happen only when next.config.js set to
module.exports = { target: "experimental-serverless-trace", };

`

[lindsaylevine]

hey @emmbyiringiro. sorry you're running into this :(. it's a known issue and we're looking at ways to address it. for now, check out this comment on a similar issue:

#68 (comment)

[lindsaylevine]

hey all! i'm going to close this issue in favor of #120, which groups several next-on-netlify issues together related to experimental-serverless-trace. this issue mentions that an improvement has been made to our internal bundling library zip-it-and-ship-it, to be released in the new year.

please feel free to re-open this issue if you think #120 does not suffice or if there's a specific piece separate from experimental-serverless-trace that still needs addressing.