Skip to content

fix: use glob to select files to move #768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 8, 2021
Merged

fix: use glob to select files to move #768

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ec6bc21
fix: check default locale exists before trying to copy
ascorbic Nov 5, 2021
fe35a5d
fix: use glob to move files
ascorbic Nov 5, 2021
ad3810e
chore: add tests
ascorbic Nov 5, 2021
7ec3740
chore: exclude nft from zip
ascorbic Nov 5, 2021
f147467
chore: windows!
ascorbic Nov 5, 2021
ef0a05a
Merge branch 'main' into mk/check-locale
kodiakhq[bot] Nov 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/helpers/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ exports.configureHandlerFunctions = ({ netlifyConfig, publish, ignore = [] }) =>
`${publish}/*.json`,
`${publish}/BUILD_ID`,
`${publish}/static/chunks/webpack-middleware*.js`,
`!${publish}/server/**/*.js.nft.json`,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how many cryptocoins for this nft

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're missing a trick there

Copy link

@getTobiasNielsen getTobiasNielsen Nov 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ascorbic Do you support .nft.json files currently? I'm wondering what trick you are referencing to.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@getTobiasNielsen Oh, it's just a joke about nfts! We don't currently use the json files directly, though we do use nft during the bundle stage.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that you handle them like vercel does or are there a difference atm?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly the same. Vercel does use the JSON files if they exist, but we don't yet and instead do the tracing when bundling, like Vercel does when the JSON files aren't available.

...ignore.map((path) => `!${slash(path)}`),
)

Expand Down
47 changes: 25 additions & 22 deletions src/helpers/files.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,52 +1,55 @@
// @ts-check
const { cpus } = require('os')

const { existsSync, readJson, move, cpSync, copy, writeJson } = require('fs-extra')
const pLimit = require('p-limit')
const { join } = require('pathe')
const slash = require('slash')
const glob = require('tiny-glob')

const TEST_ROUTE = /\/\[[^/]+?](?=\/|$)/
const TEST_ROUTE = /(|\/)\[[^/]+?](\/|\.html|$)/

const isDynamicRoute = (route) => TEST_ROUTE.test(route)

exports.moveStaticPages = async ({ netlifyConfig, target, i18n, failBuild }) => {
const root = join(netlifyConfig.build.publish, target === 'server' ? 'server' : 'serverless')
const pagesManifestPath = join(root, 'pages-manifest.json')
if (!existsSync(pagesManifestPath)) {
failBuild(`Could not find pages manifest at ${pagesManifestPath}`)
}
console.log('Moving static page files to serve from CDN...')
const root = join(netlifyConfig.build.publish, target === 'server' ? 'server' : 'serverless', 'pages')

const files = []

const moveFile = async (file) => {
const source = join(root, file)
// Trim the initial "pages"
const filePath = file.slice(6)
files.push(filePath)
const dest = join(netlifyConfig.build.publish, filePath)
files.push(file)
const dest = join(netlifyConfig.build.publish, file)
await move(source, dest)
}
// Move all static files, except error documents and nft manifests
const pages = await glob('**/!(500|404|*.js.nft).{html,json}', {
cwd: root,
dot: true,
})

const pagesManifest = await readJson(pagesManifestPath)
// Arbitrary limit of 10 concurrent file moves
const limit = pLimit(10)
const promises = Object.entries(pagesManifest).map(async ([route, filePath]) => {
if (
isDynamicRoute(route) ||
!(filePath.endsWith('.html') || filePath.endsWith('.json')) ||
filePath.endsWith('/404.html') ||
filePath.endsWith('/500.html')
) {
// Limit concurrent file moves to number of cpus or 2 if there is only 1
const limit = pLimit(Math.max(2, cpus().length))
const promises = pages.map(async (rawPath) => {
const filePath = slash(rawPath)
if (isDynamicRoute(filePath)) {
return
}
return limit(moveFile, filePath)
})
await Promise.all(promises)
console.log(`Moved ${files.length} page files`)
console.log(`Moved ${files.length} files`)

// Write the manifest for use in the serverless functions
await writeJson(join(netlifyConfig.build.publish, 'static-manifest.json'), files)

if (i18n?.defaultLocale) {
// Copy the default locale into the root
await copy(join(netlifyConfig.build.publish, i18n.defaultLocale), `${netlifyConfig.build.publish}/`)
const defaultLocaleDir = join(netlifyConfig.build.publish, i18n.defaultLocale)
if (existsSync(defaultLocaleDir)) {
await copy(defaultLocaleDir, `${netlifyConfig.build.publish}/`)
}
}
}

Expand Down
2 changes: 0 additions & 2 deletions src/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

const { join, relative } = require('path')

const { copy, existsSync } = require('fs-extra')

const { ODB_FUNCTION_NAME } = require('./constants')
const { restoreCache, saveCache } = require('./helpers/cache')
const { getNextConfig, configureHandlerFunctions, generateRedirects } = require('./helpers/config')
Expand Down
56 changes: 56 additions & 0 deletions test/__snapshots__/index.js.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,62 @@ exports.resolvePages = () => {
}"
`;

exports[`onBuild() generates static files manifest 1`] = `
Array [
"en/getStaticProps/1.html",
"en/getStaticProps/1.json",
"en/getStaticProps/2.html",
"en/getStaticProps/2.json",
"en/getStaticProps/static.html",
"en/getStaticProps/static.json",
"en/getStaticProps/with-revalidate.html",
"en/getStaticProps/with-revalidate.json",
"en/getStaticProps/withFallback/3.html",
"en/getStaticProps/withFallback/3.json",
"en/getStaticProps/withFallback/4.html",
"en/getStaticProps/withFallback/4.json",
"en/getStaticProps/withFallback/my/path/1.html",
"en/getStaticProps/withFallback/my/path/1.json",
"en/getStaticProps/withFallback/my/path/2.html",
"en/getStaticProps/withFallback/my/path/2.json",
"en/getStaticProps/withFallbackBlocking/3.html",
"en/getStaticProps/withFallbackBlocking/3.json",
"en/getStaticProps/withFallbackBlocking/4.html",
"en/getStaticProps/withFallbackBlocking/4.json",
"en/getStaticProps/withRevalidate/1.html",
"en/getStaticProps/withRevalidate/1.json",
"en/getStaticProps/withRevalidate/2.html",
"en/getStaticProps/withRevalidate/2.json",
"en/getStaticProps/withRevalidate/withFallback/1.html",
"en/getStaticProps/withRevalidate/withFallback/1.json",
"en/getStaticProps/withRevalidate/withFallback/2.html",
"en/getStaticProps/withRevalidate/withFallback/2.json",
"en/image.html",
"en/middle.html",
"en/previewTest.html",
"en/previewTest.json",
"en/static.html",
"es/getStaticProps/static.html",
"es/getStaticProps/static.json",
"es/getStaticProps/with-revalidate.html",
"es/getStaticProps/with-revalidate.json",
"es/image.html",
"es/middle.html",
"es/previewTest.html",
"es/previewTest.json",
"es/static.html",
"fr/getStaticProps/static.html",
"fr/getStaticProps/static.json",
"fr/getStaticProps/with-revalidate.html",
"fr/getStaticProps/with-revalidate.json",
"fr/image.html",
"fr/middle.html",
"fr/previewTest.html",
"fr/previewTest.json",
"fr/static.html",
]
`;

exports[`onBuild() writes correct redirects to netlifyConfig 1`] = `
Array [
Object {
Expand Down
44 changes: 44 additions & 0 deletions test/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,49 @@ describe('onBuild()', () => {
expect(existsSync(path.resolve('.next/BUILD_ID'))).toBeTruthy()
})

test('generates static files manifest', async () => {
await moveNextDist()
process.env.EXPERIMENTAL_MOVE_STATIC_PAGES = 'true'
await plugin.onBuild(defaultArgs)
expect(existsSync(path.resolve('.next/static-manifest.json'))).toBeTruthy()
const data = JSON.parse(readFileSync(path.resolve('.next/static-manifest.json'), 'utf8'))
expect(data).toMatchSnapshot()
delete process.env.EXPERIMENTAL_MOVE_STATIC_PAGES
})

test('moves static files to root', async () => {
await moveNextDist()
process.env.EXPERIMENTAL_MOVE_STATIC_PAGES = 'true'
await plugin.onBuild(defaultArgs)
const data = JSON.parse(readFileSync(path.resolve('.next/static-manifest.json'), 'utf8'))

data.forEach((file) => {
expect(existsSync(path.resolve(path.join('.next', file)))).toBeTruthy()
expect(existsSync(path.resolve(path.join('.next', 'server', 'pages', file)))).toBeFalsy()
})

delete process.env.EXPERIMENTAL_MOVE_STATIC_PAGES
})

test('copies default locale files to top level', async () => {
await moveNextDist()
process.env.EXPERIMENTAL_MOVE_STATIC_PAGES = 'true'
await plugin.onBuild(defaultArgs)
const data = JSON.parse(readFileSync(path.resolve('.next/static-manifest.json'), 'utf8'))

const locale = 'en/'

data.forEach((file) => {
if (!file.startsWith(locale)) {
return
}
const trimmed = file.substring(locale.length)
expect(existsSync(path.resolve(path.join('.next', trimmed)))).toBeTruthy()
})

delete process.env.EXPERIMENTAL_MOVE_STATIC_PAGES
})

test('sets correct config', async () => {
await moveNextDist()

Expand All @@ -213,6 +256,7 @@ describe('onBuild()', () => {
'.next/*.json',
'.next/BUILD_ID',
'.next/static/chunks/webpack-middleware*.js',
'!.next/server/**/*.js.nft.json',
'!../node_modules/next/dist/compiled/@ampproject/toolbox-optimizer/**/*',
`!node_modules/next/dist/server/lib/squoosh/**/*.wasm`,
`!node_modules/next/dist/next-server/server/lib/squoosh/**/*.wasm`,
Expand Down