Performance comparison when building a more than 5000 posts site

[iamshaynez]

Hello. I've being test a site with more than 5000 posts, and performance has been significant reduced. There a some limit static site might hit, for example Cloudflare Pages has a 20 mins building timeout limit, Github Pages has a 10 mins. Which means if building time exceed 10/20 mins, this site is no longer able to deploy.

Below performance is running on a M2 laptop, building on Pages service typically cost 2 times long of this. So in this case, I cannot deploy on Cloudflare anymore.

Performance metric on stack:

Template Metrics:

     cumulative       average       maximum      cache  percent  cached  total
       duration      duration      duration  potential   cached   count  count  template
     ----------      --------      --------  ---------  -------  ------  -----  --------
  9m12.79668134s  1.112266964s  3.238499042s          0        0       0    497  index.html
  5m0.671840143s   23.569165ms  2.444005041s         15        0       0  12757  partials/head/head.html
  4m42.188539803s   12.799988ms   967.37925ms          0        0       0  22046  partials/data/title.html
  4m15.115989748s   35.004938ms  171.338916ms          0        0       0   7288  _default/list.html
  3m44.645731085s   17.645568ms   163.16975ms         27        0       0  12731  partials/sidebar/right.html
  1m34.973947831s   12.199607ms     112.764ms        100        0       0   7785  partials/widget/search.html
  1m31.279332594s   11.725026ms  129.144333ms        100        0       0   7785  partials/widget/archives.html
  1m2.20255007s   12.513085ms   75.188791ms          0        0       0   4971  _default/single.html
  30.985878741s    2.428931ms  1.707955625s         11        0       0  12757  partials/head/opengraph/include.html
  24.650240287s    3.166376ms   67.601708ms        100        0       0   7785  partials/widget/tag-cloud.html
  19.913509236s    4.005936ms   56.086334ms          0        0       0   4971  partials/article/article.html
  16.511613045s    1.294317ms  788.548708ms         14        0       0  12757  partials/head/opengraph/provider/base.html
  16.382996122s    3.295714ms   64.799625ms          0        0       0   4971  partials/article/components/related-content.html
  13.607497853s    1.368825ms   45.204458ms          1        0       0   9941  partials/article/components/header.html
  12.748749887s     999.353µs  967.685375ms         13        0       0  12757  partials/head/opengraph/provider/twitter.html
  11.416875143s    2.662517ms  1.030093459s          0        0       0   4288  _default/rss.xml
  11.279454711s    1.135668ms   44.569375ms          0        0       0   9932  partials/article/components/details.html
  11.219589623s     879.484µs   83.698666ms        100        0       0  12757  partials/sidebar/left.html
    8.90797044s    1.144247ms   31.728417ms         48        0       0   7785  partials/pagination.html
   8.079561407s    1.037837ms   36.600334ms        100        0       0   7785  partials/widget/categories.html
   7.976819612s    1.604671ms   52.393417ms          4        0       0   4971  partials/article/components/footer.html
   6.229975688s          47µs   16.710667ms         90        0       0  132550  partials/helper/image.html
   5.800135738s    1.166794ms   14.469834ms          0        0       0   4971  partials/article/components/tags.html
   5.622004711s     135.128µs   35.004875ms         27        0       0  41605  partials/article-list/compact.html
   5.252566433s     212.156µs   55.484958ms         53        0       0  24758  partials/article-list/tile.html
   5.057541596s      46.342µs      22.172ms         54        0       0  109134  partials/helper/icon.html
   4.577554374s     921.037µs   27.891417ms         29        0       0   4970  partials/article-list/default.html
   2.457244028s     264.106µs       28.97ms          0        0       0   9304  partials/data/description.html
   1.748216512s     137.039µs    55.29375ms        100        0       0  12757  partials/footer/include.html
    1.02521832s     207.282µs   26.074083ms         57        0       0   4946  partials/widget/toc.html
    697.44313ms      54.671µs   15.837583ms        100        0       0  12757  partials/head/style.html
   369.396229ms       9.422µs   13.814459ms          0        0       0  39205  _default/_markup/render-heading.html
   234.097403ms       18.35µs    6.352542ms        100        0       0  12757  partials/head/colorScheme.html
   201.097583ms  201.097583ms  201.097583ms          0        0       0      1  _internal/_default/sitemap.xml
   187.670737ms      14.711µs   23.163458ms        100      100   12756  12757  partials/footer/components/script.html
   164.299293ms      38.316µs   14.468917ms          0        0       0   4288  _internal/alias.html
    85.428467ms      17.185µs    1.517792ms          0        0       0   4971  partials/article/components/content.html
    72.768473ms       2.512µs    3.421375ms         61      100   28967  28967  partials/data/description
    69.249313ms       2.464µs    2.200875ms         13      100   28102  28102  partials/helper/image
       53.022ms      53.022ms      53.022ms          0        0       0      1  404.html
    37.789067ms       2.329µs     1.19075ms          0      100   16225  16225  partials/data/title
    28.860221ms      57.952µs     545.375µs        100        0       0    498  partials/footer/footer.html
    25.391364ms       2.071µs    5.457583ms        100      100   12259  12259  partials/footer/footer
    21.114319ms       1.655µs    2.140959ms        100      100   12756  12757  partials/footer/components/custom-font.html
    18.503241ms       3.722µs       430.5µs        100        0       0   4971  partials/comments/include.html
     8.952369ms       1.801µs     339.958µs        100      100    4969   4969  partials/article/components/photoswipe
     6.684672ms         524ns    1.061167ms        100        0       0  12757  partials/footer/custom.html
     6.454834ms         505ns     431.458µs        100        0       0  12757  partials/head/script.html
     5.294603ms         415ns       134.5µs        100        0       0  12757  partials/head/custom.html
     3.938887ms       10.56µs     179.125µs          0        0       0    373  _default/_markup/render-link.html
     1.190292ms     396.764µs    1.119208ms         42        0       0      3  partials/helper/external.html
      238.834µs     119.417µs     196.292µs          0        0       0      2  partials/article/components/photoswipe.html
       12.501µs       1.389µs       2.834µs          1      100       9      9  partials/article/components/details


                   |  EN
-------------------+--------
  Pages            | 13549
  Paginator pages  |  3497
  Non-page files   |     0
  Static files     |    14
  Processed images |     1
  Aliases          |  4288
  Cleaned          |     0

Total in 557286 ms

Another theme's metric(while it's not a apple to apple comparison):

Template Metrics:

       cumulative       average       maximum      cache  percent  cached  total
         duration      duration      duration  potential   cached   count  count  template
       ----------      --------      --------  ---------  -------  ------  -----  --------
   1m12.21701035s    14.03362ms    253.9445ms          0        0       0   5146  _default/single.html
    38.626664453s    7.506153ms  113.057333ms          0        0       0   5146  partials/post_nav_links.html
    30.553094758s    6.162382ms  170.383375ms          0        0       0   4958  _default/list.html
    16.139888427s    1.596744ms  114.589916ms          9        0       0  10108  partials/head.html
    12.651369774s      258.46µs  127.051667ms         45        0       0  48949  partials/post_meta.html
     3.874865711s     864.346µs   99.411625ms          0        0       0   4483  _default/rss.xml
     2.647994867s      261.97µs   28.847291ms         99        5     477  10108  partials/header.html
     1.322376295s      52.138µs   11.579042ms        100        0       0  25363  partials/templates/_funcs/get-page-images.html
     1.202631148s      20.363µs   11.900875ms          8        0       0  59057  partials/author.html
     906.601142ms     176.175µs   13.063625ms          6        0       0   5146  partials/share_icons.html
     787.974858ms      78.789µs     2.57025ms         77        0       0  10001  partials/breadcrumbs.html
     669.949469ms      13.686µs    3.787292ms        100        0       0  48949  partials/cover.html
     384.567192ms      74.745µs   24.312125ms          0        0       0   5145  partials/anchored_headings.html
     190.418792ms  190.418792ms  190.418792ms          0        0       0      1  _internal/_default/sitemap.xml
     168.776232ms      37.664µs     6.01875ms          0        0       0   4481  _internal/alias.html
      161.65975ms   161.65975ms   161.65975ms          0        0       0      1  _default/index.json
     122.369792ms   61.184896ms   65.940709ms          0        0       0      2  _default/terms.html
        87.9727ms      17.095µs    1.201375ms        100        0       0   5146  partials/edit_post.html
      75.566652ms      14.684µs    5.839084ms        100        0       0   5146  partials/post_canonical.html
      52.029584ms   52.029584ms   52.029584ms          0        0       0      1  404.html
      42.247407ms       8.208µs    1.191542ms        100        0       0   5147  partials/translation_list.html
      39.623622ms        3.92µs     1.04225ms        100      100   10098  10108  partials/footer.html
       9.657041ms    9.657041ms    9.657041ms          0        0       0      1  _default/search.html
       5.510006ms         545ns      499.25µs        100        0       0  10108  partials/extend_head.html
        712.126µs      35.606µs     129.417µs          0        0       0     20  _default/_markup/render-table.html
        164.916µs     164.916µs     164.916µs        100        0       0      1  partials/home_info.html
         62.292µs      62.292µs      62.292µs        100        0       0      1  partials/social_icons.html
         42.084µs       4.208µs      38.875µs        100        0       0     10  partials/extend_footer.html
         20.625µs      20.625µs      20.625µs          0        0       0      1  robots.txt

---

I think the reason is complicated index page when doing pagination. One way to reduce the overall time cost is set pagination from default 10 to 100, however it's a workaround any, and 100 posts a page doesn't sound like reasonable...:)

I put this in a dicussion just to report what I found recently.

[loboto51]

Hello, I'm replying to your discussion because I'm in a similar case:
I tried building a website with 2000+ posts, using the github template.
The more posts I add, the more the building time seems to increase, to the point that github interrupted my last builds after 4+ hours of work.

I was wondering if you (or someone else) had found a way to reduce that building load?
The building time seems to be somehow linked to the number of posts instead of the number of modifications, there must be a part which is fully re-built at every commit instead of being modified in delta.

I wonder if there's a feature that I can disable to solve the problem?

[iamshaynez]

No, set pagination to a large enough number is the only option for now.

[CaiJimmy]

Sorry, I completely skipped this discussion 😞.

After looking at the code, I think the main cause is the line:

  {{ $pages := where .Site.RegularPages "Type" "in" .Site.Params.mainSections }}
  {{ $notHidden := where .Site.RegularPages "Params.hidden" "!=" true }}
  {{ $filtered := ($pages | intersect $notHidden) }}

This line removes hidden posts by doing a set intersection. The complexity of this operation will increase with the size of post collection.

I don't remember why this was implemented this way (which is very inefficient) instead of using the where operator. Maybe back then, Hugo did not support chaining where operators, or maybe I did not know that it was possible.

Either way, I have pushed a fix that uses where operator instead of intersection. I'll create a new release now. Please let me know if this fixes the problem.

The fix is here: #1212

[CaiJimmy]

As a side note, the hidden field will be removed in the next major release (if I have enough time to do it), since Hugo added internal support for it. Therefore, the second "where" operator won't be necessary.

[CaiJimmy]

Another potential issue is that the paginate function is called twice to display the current page number in the title. I’m not sure how significant this performance impact will be.

[loboto51]

Hello, thanks a lot for the patch, it worked very well : I did a commit + push this morning and the deployment to github pages lowered from an average of 3h to less than 30 minutes.