Abilitty to skipping static pages generating at build time

[hlege]

Describe the feature you'd like to request

In next 13 with app directory there is no way to opt out from build time static page generation without loosing ISR. (Only dynamic render). If we want to use the same code base with different backend server (e.g configured in .env files) we force to use SSR and loose ISR.

These feature allows use case like dynamic portals with different backends servers and different data but the same pages and business logic.

Other issues is that the backend server often cannot be reach by developer/build machine.

In a big portal site this feature will also allow faster build and deploy time too, but preserve the ability to use ISR if needed.

Describe the solution you'd like

Configuration based settings:

const nextConfig = {
   ssg: false // means build time the pages wont generate static html, only runtime with ISR
};

Describe alternatives you've considered

Per paged configuration:

export const ssg = false;

or

export const preventSSG = true;

[Xevion]

Why don't you simply modify generateStaticParams to not generate any of the parameters at build time? Is there are particular use-case that makes a site-wide SSG disabling configuration necessary or abundantly better?

[hlege]

generateStaticParams only work pages with dynamic parameters (slug), if you has a page e.g "/about" with dynamic data coming from backend, then you cannot prevenet SSG, only export const revalidate = 0 works but then its become SSR only.

[dbk91]

If your app directory component is marked as a client component, using the useSearchParams hook will omit the component from the static builds and force it to fetch client-side only.

During static rendering, any component using the useSearchParams hook will skip the prerendering step and instead be rendered on the client at runtime.

But I think I am confused about your use case. Can you clarify your first statement?

In next 13 with app directory there is no way to opt out from build time static page generation without loosing ISR. (Only dynamic render). If we want to use the same code base with different backend server (e.g configured in .env files) we force to use SSR and loose ISR.

Opting out of static page generation causes you to lose ISR because there is nothing to statically generate, hence Incremental Static Regeneration. The only option when opting out of a build runtime is to fetch it dynamically server side (e.g. getServerSideProps) because you'd be choosing to not run anything at build time. I'm not sure how you would be able to skip statically generating a page while simultaneously using ISR without that being the definition of fetching data dynamically server-side.

Please let me know if I misunderstood your problem.

[hlege]

I will try to explain it via example.

Given a page named /about . This page uses data fetched from backend. The backend data will change rarely or never change at all.

My problem is that the backend server cannot be reached at build time. Also I want to use the same application with different backend server.

So right now in order to build the application I am forced to use dynamic render (SSR). The problem with SSR is that every request to the /about page will dynamically fetch the data (which "never" changes ) and render the html. Of course I can make a custom cache that store the fetched data, but still the html will render every request. Also there is no cache control headers because it's SSR, so every request will hit backend even if it is still the same result every time.

So if we generate a static html on demand (not in build time) the nextjs features like cache headers control and ISR still works without the need to create a custom server.

[Sharpiro]

If we're explicitly saying "use client" at the top of the file, can't this disable the server-side stuff. Annoying to have errors about "localStorage" not available because a client component is trying to render on the server initially for some reason.

[Sharpiro]

There should be a “use client only”

[dbk91]

@hlege You simply want to avoid data fetching during the build process due to backend resources potentially being unavailable, but you want it to fetch and render statically with ISR once deployed, correct?

If this is the case, I think you could simply avoid fetching the data within your server component by referencing a build time constant.

async function getPosts() {
  const res = await fetch(`https://.../posts`, { next: { revalidate: 60 } });
  const data = await res.json();

  return data.posts;
}

export default async function PostList() {
  let posts = [];
  if (process.env.BUILD_ENVIRONMENT !== 'local') {
    posts = await getPosts();
  }

  return posts.map((post) => <div>{post.name}</div>);
};

In the example, process.env.BUILD_ENVIRONMENT would be a user-defined build constant used to differentiate between your local environment and the deployed environment. This is of course assuming you are using a single static route and not any dynamic segments. For dynamic segments, I would recommend the solution @Xevion mentioned as it is more elegant.

[benderillo]

@dbk91

@hlege You simply want to avoid data fetching during the build process due to backend resources potentially being unavailable, but you want it to fetch and render statically with ISR once deployed, correct?

If this is the case, I think you could simply avoid fetching the data within your server component by referencing a build time constant.

async function getPosts() {
  const res = await fetch(`https://.../posts`, { next: { revalidate: 60 } });
  const data = await res.json();

  return data.posts;
}

export default async function PostList() {
  let posts = [];
  if (process.env.BUILD_ENVIRONMENT !== 'local') {
    posts = await getPosts();
  }

  return posts.map((post) => <div>{post.name}</div>);
};

In the example, process.env.BUILD_ENVIRONMENT would be a user-defined build constant used to differentiate between your local environment and the deployed environment. This is of course assuming you are using a single static route and not any dynamic segments. For dynamic segments, I would recommend the solution @Xevion mentioned as it is more elegant.

Your solution is like a hack around a limitation of the framework. And what @Xevion suggested applies only to the dynamic segments and even then it does not work (maybe a bug in nextjs13): if you have a dynamic segment page say you have a folders like this: "/posts/[slug]" and inside there is a page.tsx just like in their documentation

type Props = {
  params: {
    slug: string;
  };
};

export default async function Page({ params: { slug } }: Props) {
  const post = await getPost(slug);
  return (
    <article>
      {post.title}
    </article>
  );
}

When I try to build this with yarn build, I get the build error because getPost is called with "[slug]" as the value of the slug parameter passed into Page(). That is because I do not provide the generateStaticParams function.
Silly me assumed that this will just skip the SSG and I can have SSR.
I can make it work by exporting the export const dynamic = 'force-dynamic'; then it will force "no-cache" and I get SSR.
Like explained here: https://beta.nextjs.org/docs/api-reference/segment-config#dynamic

But what the topic starter and I actually want is the ability to have a cache control with "generate on first request" (if I understood the request correctly).

Imagine you have a page that is not generated at build time but set to SSR with a cache control that says "revalidate = false"; basically once rendered it stays cached ( as if it was SSG).

This is what ideally would be the most suitable solution for the author of this ticket.
Imagine a blogs website with a couple thousands of posts. I don't want to SSG them. Ideally I want them to be SSR on the first request with cache TTL set to like forever. So the first time someone tries to open a post, it gets rendered and cached.
Next time it is the cache all the way.
And If I want to invalidate, I can have an API route that would do manual revalidation of any page like it is supported in Nextjs 12.

[dbk91]

Your solution is like a hack around a limitation of the framework. And what @Xevion suggested applies only to the dynamic segments ...

Yes, I am aware. I called both of those points out specifically in my original comment. My intent was to provide some solution to an uncommon use-case.

Imagine a blogs website with a couple thousands of posts. I don't want to SSG them.

I don't think a blog is the best example for the author's original scenario. The author has a limited access to the resources required to generate the pages at build time. I don't think this issue was about avoiding the cost of generating content at build time. It's typically ideal to generate these at build time so that the first user to visit the page doesn't incur the cost of waiting for the page to generate using SSR. This is also what enables the effectiveness ISR — the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.

[hlege]

i am well aware that i can use build time env for that, but than everything need to be checked and its a little "ugly" and i thinks nextjs can support it out of box. Also i think a common cms/portal like webpage is use a render strategy like: render when user request a page -> then cache it until its expired or content is changed.

the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.

In nextjs 13 the streaming rendering already solve this problem with a proper page layout structure too.

[dbk91]

@hlege Totally agreed. I realize in retrospect that my eagerness to help you find a near-term solution, albeit a hack, was seemingly dismissive of your original feature request. My apologies for that. To be clear, I do agree something like this would be useful for the framework to expose — in particular the per page configuration at a minimum.

Tangentially, I found an issue specific to client components when trying to leverage the loading and error special file structure with SWR/suspense that a solution like this could fix more elegantly. SWR requires fallbackData when using suspense but that causes the application to flash content with the content of fallback data. This doesn't work well since client-side data might be specific to a user or frequently updated and will flash intermittently between the loading state and the pre-generated content. But if you don't provide content, it throws an error at build-time. The only workaround I found was to use useSearchParams to opt out at build time and then omit the fallbackData, but you have to place it before the useSWR hook, which is not obvious. It feels inelegant if you're not referencing search parameters in your component.

Of course, that example is not applicable to your issue since that's specific to client components and client-side fetching. I simply wanted to provide that to present another use-case to support this feature request.

[benderillo]

@dbk91

I don't think a blog is the best example for the author's original scenario.
No, it is not :) I added it as a generalisation of the issue. Because what the author asks will handle both.

the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.
I don't think we should be worrying about SSR performance too much these days. Look at the Remix framework that is all SSR deliberately but I doubt it has performance drawbacks compared to NextJS (they actually show in their blog that they are faster).
Rendering is moving to the Edge and that makes SSR blazingly fast :)

[dbk91]

@dbk91

I don't think a blog is the best example for the author's original scenario.
No, it is not :) I added it as a generalisation of the issue. Because what the author asks will handle both.

the worst-case performance scenario is the user views stale content as opposed to waiting for a page generated using SSR.
I don't think we should be worrying about SSR performance too much these days. Look at the Remix framework that is all SSR deliberately but I doubt it has performance drawbacks compared to NextJS (they actually show in their blog that they are faster).
Rendering is moving to the Edge and that makes SSR blazingly fast :)

There are some good points here. I think it's important to keep in mind that SSR can be negatively impacted if the API you're integrating with is slow. For example, I have some personal projects where I serve content from my Notion via their API. In my experience, it's been slower than most so I try to do everything at build time. 😅 Having said that, you're correct in that it's up to the developer to determine those scenarios and edge runtimes can help this tremendously. And as @hlege pointed out, the appDir does provide improved performance if you leverage the layout structures in a fashion where you can build the majority of the static content and then display instant loading states otherwise.

I think this feature could be an overall win for the framework.

[agusterodin]

We have a similar requirement at my organization.

Our application has whitelabeling requirements (eg: custom brand name in browser title bar). We need to deploy run multiple versions of the same exact application build multiple times but with slightly different environment variables. The application needs to be deployed via Docker to custom infrastructure (customer requirements, not ours). We want to avoid performing multiple docker builds and maintaining many redundant docker images just to change a few build-time environment variables. Our proposed solution is to use runtime environment variables.

We are able to accomplish runtime environment variables with publicRuntimeConfig. I want to use getStaticProps so we can take advantage of incremental static regeneration and not take the performance hit that SSR would cause. Unfortunately there is no means to perform something similar to fallback: 'blocking' for non-dynamic routes.

I don't even need to revalidate as we don't have any blocking data requirements. For all I care I would set revalidate to an arbitrarily high value so that revalidation never happens.

Ideally there would be a way to tell the server to generate and cache ALL pages when the built server process starts (similar to what happens with static optimization during build time).

Basically what i'm looking for is "run-time" static page optimization instead of "build-time" static page optimization. We have a very small amount of pages, so run-time static optimization slowness isn't a concern.

If this isn't currently possible, does the app directory / Suspense make this possible?

[ngwie]

I have the same problem with @agusterodin, which is variables aren’t available at build time.

At the moment I fixed it with deleting generated static file after build.

RUN yarn build

# Remove SSG file at build time
# Also removes files in .next/standalone/.next folder if next config output is 'standalone'
RUN rm ./.next/**/*.html ./.next/standalone/.next/**/*.html

I hope this helps the others. But options to skip SSG at build time is nice to have.

[mikeymop]

Such a feature is also something I've been looking for.

We're wanting to build a next.js application that uses only SSR. Reason being is that we're building a kubernetes dashboard using @kubernetes-client/javascript and all of the caching features seems to break down when not using fetch().

I've tried using getInitialProps as recommended in the docs however the TS error suggestion in the docs doesn't work when using this function.

Ideally we'd have a { disableSSG: true } in next.config.js.

[kombuchamp]

We have the same issue in our project - we cannot share some secrets in our CI and production server, so we cannot pregenerate the page with data. And after deployment regeneration is not triggered

There is a way to opt out from this optimization by configuring the route segment (see https://nextjs.org/docs/app/api-reference/file-conventions/route-segment-config) for a single page

export const dynamic = "force-dynamic";

It took me a couple of days to realize it from the docs, to be honest, but that's the only way I found to make it work. From the docs:

Good to know: The new model in the app directory favors granular caching control at the fetch request level over the binary all-or-nothing model of getServerSideProps and getStaticProps at the page-level in the pages directory. The dynamic option is a way to opt back in to the previous model as a convenience and provides a simpler migration path.

[ijjk]

Hi, as an update here in the latest versions of Next.js v13.5.3 there is now next experimental-compile command to replace next build that skips prerendering step for all paths and also a separate next experimental-generate command to prerender after the fact.

Would love any feedback on these commands to see if these alleviate the concerns in this thread!

[CaptainN]

This seems to work for me (14.0.4)! I have to ask, does this mean that single page routes are cached through ISR?

[half0wl]

Is there any way to opt into linting and typechecking with experimental-compile?

[miguelSoldado1]

the next experimental-compile stopped working once I upgraded to next@14.2.0 maybe due to the compiler optimizations... works fine in the previous 14.1.4 patch

[arekrgw]

@miguelSoldado1 look at the bottom of this discussion!

[miguelSoldado1]

@arekrgw oh damn, my bad! I was kinda triggered and didn't look that deep into the discussion... Thank you for the heads up!

[hallatore]

This works for pages with params.
Would be nice to be able to also do this for pages that doesn't take any params.

export const generateStaticParams = () => [];

[craigwardman]

+1 for this feature, the whole "call APIs at build time" thing has certainly caused a few issues for me on various projects - ranging from the "multiple targets" problem (i.e. one build artifact deployed to many environments) down to simple networking (i.e. even for a single environment the APIs/DBs etc. are behind a firewall and not accessible to the build server).

It would be good to have "runtime-only SSG"

[MicahBCode]

Is this possible in the meantime?

[MicahParks]

I would also like an easy on/off option to move network requests from build time to when the Next.js application is starting up. This would mean server rendered assets that require a network request would be built on application startup instead of build time.

I would also suggest that the web page in the error below be updated to include a note about fetch being performed at build time. It took me over an hour to find this discussion and learn that network requests were required at build time.

Error occurred prerendering page "/". Read more: https://nextjs.org/docs/messages/prerender-error

I understand network requests at build time are potentially more efficient for some use cases, but for my use cases, it is inconvenient or less efficient.

I am building my Next.js application into a Docker image. This Docker image is used in three different environments with segmented networks: development, staging, and production. I have non-Next.js custom backends.

• All three environments have different data used to create server rendered assets, but I only build my Docker images in one environment. This means the Docker image has cached the wrong data for the other two environments.
• Between the time when the Next.js Docker image was built and when it was deployed, the network request cached during build time has become irrelevant.
• Some custom backends require authentication. I do not want to provide authentication assets for these backends during build time.
• Establishing a network connection with custom backends requires configuration. I do not want to manage a separate configuration for Next.js build time, then adapt that configuration to the Docker build.
• Custom backends, especially in the development environment, do not have 100% uptime. I would like the ability to build my Next.js code when a backend is not responding.

[kelvinlouis]

We are experiencing major issues with next experimental-compile. We are using this in favor of next build because we want to opt out from building pages during build time. Our pipeline environment is completely isolated and thus breaks the build, because fetches are blocked by the firewall. Using experimental-compile skips this step and our builds run through.

Unfortunately, we realized that all our routes under pages no longer just serve the JSON to the client, after clicking on a NextLink. The pages use getServerSideProps. It now does a complete reload of the page every time the user clicks on the link. We noticed that it tries to just serve JSON to the client, but we are getting 404 by the Next.js server and after receiving a 404 it does a complete reload.

We only are experiencing this issue running next start after building with next experimental-compile. If we build with next build locally (not in the pipeline) and serve via next start the issue does not appear.

Did anyone else have similar issues? What I find weird is that it looks as if a locale is part of the help.json call, even though we do not use locales in our application.

[CaptainN]

Is it required to run next experimental-generate before next start after next experimental-compile? Ideally, if you don't run generate, then it'll generate pages in to the ISR cache at runtime.

[arekrgw]

What should be the commands order while using output mode as standalone?

Assume a multi stage docker image build. Is it possible to run next experimental-generate after copying files from standalone folder to runner docker image?

Overall, how should this process look like, or maybe it is possible to skip next experimental-generate so the static will be generated at runtime at first request?

[ReesMorris]

This isn't a direct solution, but does work for our use-case. We have a single Next.js codebase which is deployed to different customers, all with different environment variables (such as API URLs, etc).

We have a docker-entrypoint.sh file:

#!/bin/sh
ENV_PATH="/app/.env"

# Overwrite the `.env` file with variables in the current environment
printenv > $ENV_PATH

# Delete all .html and .rsc files within the .next directory
find /app/.next -type f \( -name "*.html" -o -name "*.rsc" \) -delete

# Start the Next.js application using the exec command
exec "$@"

And in our Dockerfile we simply reference it as an ENTRYPOINT:

# Copy the entry point script into the image and make it executable
COPY docker-entrypoint.sh ./docker-entrypoint.sh
RUN chmod +x ./docker-entrypoint.sh
...
ENTRYPOINT ["./docker-entrypoint.sh"]
CMD ["node", "server.js"]

Essentially, we build our application using the environment variables for one of our customers (typically a test instance with the smallest amounts of data/latency) then, just before the container runs, the docker-entrypoint.sh script will overwrite the .env file in the image (which gets generated along with the build) and delete all pre-rendered static pages that used the dummy data. This forces Next to re-compile each page on first visit, but then (as far as we can see) fall back to ISR behaviour. Since all our customer APIs return the exact same data structure, just with their own CMS' content, there's no issues.

A few notes:

• printenv will copy every environment variable over from the container
• this is designed for the standalone output, which has a different data structure
• if Next changes how pages are built/stored, this solution very likely will break.

[DarkPromise]

next experimental-compile seems to be broken in >14.2.0, prior to this, it was working with 14.1.4.

NOTE This is running in docker

The error I get in 14.2.1 now is:

[builder 4/4] RUN npm run build-experimental:
0.748
0.748 > scst@0.1.0 build-experimental
0.748 > next experimental-compile
0.748
0.878  ⚠ You are using a non-standard "NODE_ENV" value in your environment. This creates inconsistencies in the project and is strongly advised against. Read more: https://nextjs.org/docs/messages/non-standard-node-env
1.019  ⨯ Invalid project directory provided, no such directory: /app/experimental-compile

Prior to the command running, the following environment variables are injected:
ENV NEXT_TELEMETRY_DISABLED 1
ENV NODE_ENV production

Some things has changed between the version upgrades, does anyone else have this issue?

For context, this is the full build step I used:

# Rebuild the source code only when needed
FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .

ENV NEXT_TELEMETRY_DISABLED 1
ENV NODE_ENV production

# Build
# See https://github.com/vercel/next.js/discussions/46544#discussioncomment-7115457
RUN npm run build-experimental

Thanks

[arekrgw]

Confirming the issue.

For our project it's critical to work, would like to know if this will be fixed or should we migrate to something else

[arekrgw]

Okay, the scripts were refactored and now this command should be invoked differently

Now we should use the next build --experimental-build-mode compile/generate

Ref:

next.js/packages/next/src/bin/next.ts

Line 116 in c0ae6f6

'--experimental-build-mode [mode]',

next.js/docs/02-app/02-api-reference/08-next-cli.mdx

Line 189 in c0ae6f6

--experimental-build-mode [mode] Uses an experimental build mode. (choices: "compile"

[DarkPromise]

Cool, thanks for the heads up

[m-sanders]

I have just tried moving to this for our setup and ultimately reverted the change.

I was running the generate step as an npm pre-step on our start command. Our containers have memory limits of 2GB and normally remain stable around the 700-900MB mark. The generate step, eats memory and very quickly causes an OOM error and kills the pod. I understand that Vercel uses 8GB for their builders. I bumped memory limits to 8GB to match, and it worked but seems excessive. While I could rework this to a separate job, having the added generate step also slows down releases as we have to pre-render quite a lot of static routes for all of our locales.

This is a step in the right direction but it is not the ideal solution. What I would like is to make all routes behave like Page Router fallback mode on dynamic routes. Build the code and manifest but do not generate any prerenders, then rely on fallback to lazily build things at runtime. I can see that others have hacked their way to this solution by doing a build and then deleting the *.html and *.rsc output in the next directory.

EDIT: I got to the bottom of this. We do not set CPU limits on our pods but we do set memory limits. We end up with too many workers for the available memory. Setting experimental.cpus to a sensible number based on the memory available fixed this. I found that with 4 cpus and 3GB resulted in consistent crash-free generates using around 2GB. I also tried experimental.memoryBasedWorkersCount: true but that option still resulted in OOM. I felt that it was safest to explicitly set the count than rely on the implementation of memoryBasedWorkersCount.

[CaptainN]

Couple questions:

• Is it required to use the generate step? That has to be done at runtime (because it access external resources which aren't even accessible to the build environment), which requires a bunch of support files including the full source of the application and node_modules be available in that environment. It'd be stellar if we could simply skip that, and let the ISR cache be filled at runtime for all routes. It seems this is not yet possible.
• Given the above, this split between compile/generate doesn't seem applicable to standalone mode. Is that right?

[m-sanders]

Generate is a required step. It would be nice not to pre-render prior to startup as it slows the boot process.

no idea on standalone.

Be warned compile also inlines NEXT_PUBLIC_. Only server side variables can change at runtime unless you use next-runtime-env.

[tusmenko]

NEXT_PUBLIC_ is another kind of obstacle to the "build once - deploy many" place. It's sad how many next-libraries require them instead of explicit configuration - even the Next itself. Even if that is relevant primarily to React itself and the Web in general, there are good examples of avoiding this limitation in other React frameworks.

[CaptainN]

I'm aware. We have simply duplicated those values - it's not ideal, but since it doesn't hit any of the URLs defined in those variables during build time, it's fine for us. For variable output (for us there are only 2 - staging and prod), we have created multiple configs, with the differing values in there.

The only other kind of odd thing is that we are using Sentry.io - and its integration does reach out to upload sourcemaps during compile time - that's not ideal, as it's technically a "side effect" which we are supposed to avoid in Docker, but whatevs, it's fine, I guess.

[CaptainN]

I may have encountered a bug related to this? (Maybe not?) - basically, the chunk filenames are not deterministic based on BUILD_ID - is that unexpected behavior?