Build 17.6.0 timed out

[lekoder]

CI build for 17.6.0 seems to have timed out and 17.6.0 is not available on dockerhub. Please run the task again.

[kkimurak]

duplicate of #3036 and meaningless suggestion.

The build time of gitlab itself is getting longer and longer, and will almost certainly reach 1 hour (the upper limit in the free plan). If we don't review the build process itself, we will just waste time running CI.

[lekoder]

Apologies, search didn't pick that up, since the specific version is not mentioned in the issue.

Might I suggest a workaround then of building it in another environment and pushing to the dockerhub?

[kkimurak]

Might I suggest a workaround then of building it in another environment and pushing to the dockerhub?

I'm running a locally built version of 17.6.0, but if someone publishes it on dockerhub it will help people who can't build it themselves.
On the other hand, I'm just a contributor. You should ask the maintainer who has the final words (@sachilles has been the only active one in recent years)

[lekoder]

Fair enough. In the meanwhile, if anybody needs this very image built to 17.6.0 up to 17.6.2 I forked it, built them and published on the dockerhub here.

[th-2021]

I'm working on a multistage Dockerfile. Using BuildKit (docker buildx build or DOCKER_BUILDKIT=1 docker build), which allows parallel execution of stages I could save around 5 minutes.

[sachilles]

@kkimurak Unfortunately, I don't have the option of giving you the role of maintainer.

@sameersbn Do you see a possibility to either change the plan for the CI or take up kkimurak's offer. He has also maintained the project remarkably so far.

[tDo]

@sachilles @kkimurak given that @sameersbn does not seem to be very active on github anymore (At least there have been no contributions in the last year) would moving the ci/cd + docker-image to a new namespace controlled by either of you be a feasible option?

[Thomas-Ganter]

Just for curiosity’s sake: What are the requirements to build the image?
I got errors 137 after 2 hours and after increasing memory to 6GB error 134 during the build after 3.5 hours — I found no info in any "building yourself" docu (or was merely looking in the wrong places).

Any insights?

[kkimurak]

@tDo If necessary, we should do so. I would like to get permission from all maintainers (with respect) if possible, but if we can't contact them, there is nothing we can do.
Stalled project are one of my greatest fears, and the confusion that would result from moving namespaces is a minor pain compared to that.

However, I am still just a contributor (the most active - at least in terms of commits - except for the maintainer) and do not have (nor will I have) any decision-making authority.

So what I'm trying to say is that I'm asking for more human resources with administrative privileges that can respond to emergencies.

[kkimurak]

@Thomas-Ganter I don't know what the actual requirements are, but I think it's less than 6GB.

Hints for 6GB memory requirement:

Hint 1 : rake task gitlab:assets:compile is executed during build

docker-gitlab/assets/build/install.sh

Line 215 in 76dad78

exec_as_git bundle exec rake gitlab:assets:compile USE_DB=false SKIP_STORAGE_VALIDATION=true NODE_OPTIONS="--max-old-space-size=4096"

Hint 2 : It invokes system command yarn webpack (with some options based on environment variable - to detect CI)
https://gitlab.com/gitlab-org/gitlab/-/blob/f42b800081f2c4c751e8fc59df70857704c1ddee/lib/tasks/gitlab/assets.rake#L120-133

Hint 3 : node command webpack is defined in package.json and it sets --max-old-space-size to 5120
https://gitlab.com/gitlab-org/gitlab/-/blob/f42b800081f2c4c751e8fc59df70857704c1ddee/package.json#L45

Anyway, I usually do local builds on two different virtual machines (Ubuntu) with 8GB / 16GB RAM. I have run out of memory once before on an 8GB machine, but since then I have disabled most services, including the GUI, and do no other work during the build, and have not run out of memory.

[Thomas-Ganter]

@Thomas-Ganter I don't know what the actual requirements are, but I think it's less than 6GB.

Hints for 6GB memory requirement:
[…]

Thanks @kkimurak — Running the build on a dedicated 8 core / 60GB machine worked for the single-architecture build, the multi-arch build timed out after 2 hrs and 11GB consumed memory in the builder container. Bumped the pipeline timeout to 4hrs and trying again ...

[th-2021]

Running a remote build (buildx with kubernetes driver) with 4 CPU, 4 GB memory worked (in 40 minutes) Am 31.12.2024 um 11:02 schrieb Thomas Ganter:

…

@Thomas-Ganter <https://github.com/Thomas-Ganter> I don't know what the actual requirements are, but I think it's less than 6GB. Hints for 6GB memory requirement: […] Thanks @kkimurak <https://github.com/kkimurak> — Running the build on a dedicated 8 core / 60GB machine worked for the single-architecture build, the multi-arch build timed out after 2 hrs and 11GB consumed memory in the builder container. Bumped the pipeline timeout to 4hrs and trying again ... — Reply to this email directly, view it on GitHub <#3042 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVVFCFYVCKD7I4S3Y2WNHC32IJTT5AVCNFSM6AAAAABTK5PYLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRWGMYDKNRQGU>. You are receiving this because you commented.Message ID: ***@***.***>

[Thomas-Ganter]

Running a remote build (buildx with kubernetes driver) with 4 CPU, 4 GB memory worked (in 40 minutes)

Hmmm … for me a build with a 4hrs timeout just aborted.
I now have bumped the timeout to 8hrs, let's see whether this helps.

Then, I also do not know how to properly parallelize multi-architecture builds.
Maybe this is my incompetence being the main culprit in my build drama ... any hints welcome.

[th-2021]

I've no experience with multi-platform builds, but I'm working on multistage (parallelized build steps), which might help. Am 01.01.2025 um 02:09 schrieb Thomas Ganter:

…

Running a remote build (buildx with kubernetes driver) with 4 CPU, 4 GB memory worked (in 40 minutes) Hmmm … for me a build with a 4hrs timeout just aborted. I now have bumped the timeout to 8hrs, let's see whether this helps. Then, I also do not know how to properly parallelize multi-architecture builds. Maybe this is my incompetence being the main culprit in my build drama ... any hints welcome. — Reply to this email directly, view it on GitHub <#3042 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVVFCF56TOJUJLSRVUHQUBT2IM523AVCNFSM6AAAAABTK5PYLWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRWG44DIOJVGU>. You are receiving this because you commented.Message ID: ***@***.***>

[Thomas-Ganter]

I've no experience with multi-platform builds, but I'm working on multistage (parallelized build steps), which might help.

Sounds interesting.
As the bulk of the time is spent inside the "install.sh", I was also wondering whether this could be split up into separate steps (which would yield in cacheable intermediate layers) which would make it easier to restart builds …