profile
viewpoint

Ask questionsBuild gets stuck at Generating image thumbnails / Update schema on large sites

I am querying allImageSharp in a project with 300 pages and 2000 images in gatsby-node. During gatsby build or gatsby develop, the build process freezes when generating thumbnail 20~ or so

Generating image thumbnails [==----------------------------] 25/9667 24.0 secs 0%

I also have the same query called in a static query on a page. If I remove the query from gatsby-node and use only the one in the page, the thumbnails generation correctly in around 600 seconds

Generating image thumbnails [=============================-] 9235/9667 560.0 secs 96%

This is what my query looks like:

gallery: allImageSharp {
  edges {
    node {
      fluid(
        sizes: "(orientation: portrait) 50vw, (orientation: portrait) and (min-width: 750px) 30vw, (orientation: landscape) 30vw,(orientation: landscape) and (min-width: 1170px) 18vw,(orientation: landscape) and (min-width: 1800px) 16vw"
      ) {
        aspectRatio
        base64
        presentationHeight
        presentationWidth
        sizes
        src
        srcSet
      }
    }
  }
}

Environment

System:
    OS: macOS 10.14.4
    CPU: (4) x64 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.15.2 - /usr/local/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Languages:
    Python: 2.7.15 - /usr/local/bin/python
  npmPackages:
    gatsby: ^2.4.5 => 2.4.5
    gatsby-image: ^2.0.41 => 2.0.41
    gatsby-plugin-algolia: ^0.3.0 => 0.3.0
    gatsby-plugin-intl: ^0.1.7 => 0.1.7
    gatsby-plugin-manifest: ^2.1.1 => 2.1.1
    gatsby-plugin-matomo: ^0.7.0 => 0.7.0
    gatsby-plugin-offline: ^2.1.0 => 2.1.0
    gatsby-plugin-react-helmet: ^3.0.12 => 3.0.12
    gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0
    gatsby-plugin-sass: ^2.0.11 => 2.0.11
    gatsby-plugin-sharp: ^2.0.37 => 2.0.37
    gatsby-plugin-sitemap: ^2.1.0 => 2.1.0
    gatsby-plugin-typescript: ^2.0.13 => 2.0.13
    gatsby-remark-images: ^3.0.11 => 3.0.11
    gatsby-source-filesystem: ^2.0.34 => 2.0.34
    gatsby-transformer-json: ^2.1.11 => 2.1.11
    gatsby-transformer-remark: ^2.3.12 => 2.3.12
    gatsby-transformer-sharp: ^2.1.19 => 2.1.19
  npmGlobalPackages:
    gatsby-cli: 2.5.14
gatsbyjs/gatsby

Answer questions jonniebigodes

@gvocale first of all, sorry for the "radio silence"(pardon the bad pun). But i was testing your code and trying to fully pinpoint your issue.

Moving on, in this comment i'm going to enumerate where your issue actually happens and if you don't mind some considerations.

I cloned your reproduction code and installed the dependencies and even before checking where the actual issue happens, one thing was brought to my attention. As you can see with the image below, the build is breaking in production and in development is showing that message, for based on a rough estimate around 50% of the images used in your code.

gvocale_develop_build

Left is development and right is production build.

Upon checking the images in question i can see that there's actually nothing wrong with it, other than the filename being extremely long, vscode for me complains and as i've shown you gatsby will to.

If i can offer a suggestion, shorten the filenames, i know that it might be a daunting situation, but one that might help you in the future.

Moving on, upon testing your code i was able to pinpoint it to this piece of code:

// Create tags pages
    const uniqTags = key => {
      let uniqTags = []

      const concatTags = (node) => {
        if (node[key]) {
          uniqTags = uniqTags.concat(node[key])
        }
      }

      collection.forEach(({ node }) => {
        concatTags(node)
      })

      return [...new Set(uniqTags)]
    }
    const generatePages = (tags, original, translation) => {
      tags.forEach(tag => {
        // Count how many objects for each tag
        const filtered = collection.filter(item =>
          item.node[original].includes(tag)
        )
        const numPages = Math.ceil(filtered.length / itemPerPage)
        Array.from({ length: numPages }).forEach((value, i) => {
          console.log(`createPage ${tag}`)
          createPage({
            path:
              i === 0
                ? `/${translation}/${toKebabCase(tag)}/`
                : `/${translation}/${toKebabCase(tag)}/${i + 1}`,
            component: tagTemplate,
            context: {
              limit: itemPerPage,
              skip: i * itemPerPage,
              numPages,
              currentPage: i + 1,
              tag,
              galleryAll: galleryAll,
              tagsEn: tagsEn,
              tagsIt: tagsIt,
            },
          })
        })
      })
    }
    tagsLocale.map(item => {
      const tags = uniqTags(item.original)
      generatePages(tags, item.original, item.translation)
    })

I issued gatsby develop and while starting dinner, i left the code "simmer" (pardon the bad pun), for about an hour and it was stuck there, removing that part of the code would run to completion, with the exception of the image issue. Below is a screnshot after an hour of the code running.

gvocale_develop_hangs

If i understand correctly and feel free to correct me if i'm wrong, but you're implementing a set of pages with internationalization with a set of unique tags correct?

It would be wise to go back to that piece of code and possibly approach the problem from another prespective.

Now for some considerations if you don't mind. Starting from the top, i saw that you're aliasing your graphql queries which is actually a good move on your part, but if you're already taking advantage of it, why not issue a single graphql query with all the data as it's already aliased. Transforming your createPages api call into something like the following:

exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions
  const collectionTemplate = path.resolve("src/components/Collection/index.tsx")
  const tagTemplate = path.resolve("src/components/Tag/index.tsx")
  const operaTemplate = path.resolve(`src/components/Opera/index.tsx`)
  const tagsLocale = [
    {
      original: "categoryEn",
      translation: "tags",
    },
    {
      original: "categoryIt",
      translation: "categorie",
    },
  ]
  return graphql(
    `
      {
        collection: allIndexJson {
          totalCount
          edges {
            node {
              parent {
                ... on File {
                  relativeDirectory
                }
              }
              price
              titolo
              categoryIt
              categoryEn
            }
          }
        }
        galleryAll: allImageSharp {
          edges {
            node {
              fluid(
                sizes: "(orientation: portrait) 50vw, (orientation: portrait) and (min-width: 750px) 30vw, (orientation: landscape) 30vw,(orientation: landscape) and (min-width: 1170px) 18vw,(orientation: landscape) and (min-width: 1800px) 16vw"
              ) {
                aspectRatio
                base64
                presentationHeight
                presentationWidth
                sizes
                src
                srcSet
              }
            }
          }
        }
        tagsEn: allIndexJson {
          group(field: categoryEn) {
            totalCount
            fieldValue
          }
        }
        tagsIt: allIndexJson {
          group(field: categoryIt) {
            totalCount
            fieldValue
          }
        }
      }
    `
  ).then(result => {
    if (result.errors) {
      throw result.errors
    }
    const { data } = result
    const { collection, galleryAll, tagsEn, tagsIt } = data
    const itemPerPage = 20
    
    const numPages = Math.ceil(collection.totalCount / itemPerPage)
    // Create index pages
    for (let index = 0; index < numPages; index++) {
      const nextItem = index + 1
      createPage({
        path: index === 0 ? "/" : `/${nextItem}/`,
        component: collectionTemplate,
        context: {
          limit: itemPerPage,
          skip: index * itemPerPage,
          numPages: numPages,
          currentPage: nextItem,
          galleryAll: galleryAll,
          tagsEn: tagsEn,
          tagsIt: tagsIt,
        },
      })
    }
    // Create opera pages
    const {edges}= collection
    edges.forEach(({node}) => {
      createPage({
        /* path:`/${node.parent.relativeDirectory}/`, */
        path:node.parent.relativeDirectory,
        component:operaTemplate,
        context:{
          titolo:node.titolo,
          relativeDirectory:node.parent.relativeDirectory
        }
      })
    });

    // Create tags pages

  })
}

In the code block above, you'll see that i've also made some changes to the way the pages are created, based on your reproduction, which from what i thing are a bit more streamlined and also easier to read.

I would like to point out the following lines in your code:

collection.forEach(({ node }) => {
      console.log(`createPage ${node.titolo}`)
      createPage({
        path: node.parent.relativeDirectory,
        component: operaTemplate,
        context: {
          titolo: node.titolo,
          relativeDirectory: node.parent.relativeDirectory,
        },
      })
    })

From my read on it it would seem that you're using the relative directory where the json files are housed and use it to create the paths for some pages. With that i would like to bring to your attention what is happening to those said paths/pages.

gvocale_paths

The approach is sound, but some "treatment" for the paths in question is needed. For me Gatsby would not show me the path/page in question as is. Even if / was prefixed and postfixed.

Also upon checking your templates something came to my attention.

For instance in ./src/Components/Opera/index.tsx

import React from "react"
let styles = require("./styles.module.scss")// not a good practice you can use import './styles.module.scss' directly or like mentioned here https://medium.com/@thetrevorharmon/how-to-make-a-super-fast-static-site-with-gatsby-typescript-and-sass-3742c00d4524

const Opera = ({ pageContext }) => {

  return (
    <article className={styles.opera}>
      <div className={styles.top}>
        <div className={styles.topImage} />
        {pageContext.titolo}
        <div className={styles.pdp} />
      </div>
    </article>
  )
}

export default Opera

Feel free to provide feedback, so that we can close this issue or continue to work on it till we find a solution. And sorry for the extremely long comment.

useful!

Related questions

Error: Cannot create as TypeComposer the following value: Date. hot 3
Importing Link from gatsby breaks Storybook hot 2
Loading chunks while a new release is deployed hot 2
Webpack error #98123 when running "gatsby develop" hot 2
gatsby build error #11328 A page component must export a React component for it to be valid. Please make sure this file exports a React component: /Users/tiagosanchez/Documents/Projects/personalBlog/node_modules/gatsby-plugin-offline/app-shell.js hot 2
[gatsby-telemetry] error: src/postinstall.js not found in Linux environments hot 2
IE11: Object not valid as React Child hot 2
CircleCI build Error: spawn ENOMEM hot 2
[Help] I somehow managed to butcher my site's performance hot 2
[gatsby-source-graphql] Shopify GraphQL Schema Error hot 2
UNHANDLED REJECTION Source and destination must not be the same. hot 2
gatsby-source-shopify unable to complete build hot 2
Gatsby's Use Of Polyfills and the 'Missing Resources for x' error hot 2
Autoprefixer "browsers" option is deprecated in v9, produces warnings hot 2
&#39;gatsby&#39; is not recognized as an internal or external command hot 2
Github User Rank List