Skip to content

Conversation

@grooverdan
Copy link
Member

Old bugs are listed as fixed, so lets test them again.

Cleans up columnstore upgrade to the two arches supported.

with the removal of columnstore and spider out of the lists they can be installed like a user normally would.

@RazvanLiviuVarzaru
Copy link
Collaborator

These are quite significant changes. Have you had a chance to test them? Otherwise, I’ll need at least half a day to test them on multiple builders.
Please also include @fauust for reviews when there are changes related to bash_lib, upgrades, and so on.

@grooverdan
Copy link
Member Author

not durable enough it seems.

$ cat Containerfile 
FROM debian:trixie
RUN apt-get update && apt-get install -y systemd wget jq procps sudo
USER root
CMD [ "/lib/systemd/systemd" ]

$ buildah bud --tag  debian_systemd:trixie .
$ podman run --name dsystemd -v ./:/scripts/:z --rm  --workdir /scripts  -d debian_systemd:trixie

$ podman exec -ti --workdir /scripts dsystemd ./deb-upgrade.sh 'https://buildbot.mariadb.org/#/builders/984/builds/48'
Sep 01 06:08:06 04083476cb99 systemd[1]: Started mariadb.service - MariaDB 11.8.3 database server.
░░ Subject: A start job for unit mariadb.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mariadb.service has finished successfully.
░░ 
░░ The job identifier is 153.
Sep 01 06:08:06 04083476cb99 /etc/mysql/debian-start[2188]: Upgrading MariaDB tables if necessary.
Sep 01 06:08:06 04083476cb99 /etc/mysql/debian-start[2208]: Checking for insecure root accounts.
Sep 01 06:08:06 04083476cb99 /etc/mysql/debian-start[2218]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting mariadb-columnstore.service - mariadb-columnstore...
░░ Subject: A start job for unit mariadb-columnstore.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mariadb-columnstore.service has begun execution.
░░ 
░░ The job identifier is 213.
Sep 01 06:08:06 04083476cb99 systemd[1]: Created slice system-mcs\x2dworkernode.slice - Slice /system/mcs-workernode.
░░ Subject: A start job for unit system-mcs\x2dworkernode.slice has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit system-mcs\x2dworkernode.slice has finished successfully.
░░ 
░░ The job identifier is 274.
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting mcs-loadbrm.service - loadbrm...
░░ Subject: A start job for unit mcs-loadbrm.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-loadbrm.service has begun execution.
░░ 
░░ The job identifier is 334.
Sep 01 06:08:06 04083476cb99 mcs-loadbrm.py[3004]: cat: /var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves_current: No such file or directory
Sep 01 06:08:06 04083476cb99 mcs-loadbrm.py[3003]: ERROR: /var/lib/columnstore/data1/systemFiles/dbrm/BRM_saves_current does not exist.
Sep 01 06:08:06 04083476cb99 mcs-loadbrm.py[3003]: INFO: brm_saves_currenty returned empty string from read_from_disk
Sep 01 06:08:06 04083476cb99 systemd[1]: mcs-loadbrm.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit mcs-loadbrm.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Sep 01 06:08:06 04083476cb99 systemd[1]: mcs-loadbrm.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit mcs-loadbrm.service has entered the 'failed' state with result 'exit-code'.
Sep 01 06:08:06 04083476cb99 systemd[1]: Failed to start mcs-loadbrm.service - loadbrm.
░░ Subject: A start job for unit mcs-loadbrm.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-loadbrm.service has finished with a failure.
░░ 
░░ The job identifier is 334 and the job result is failed.
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting [email protected] - mcs-workernode...
░░ Subject: A start job for unit [email protected] has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit [email protected] has begun execution.
░░ 
░░ The job identifier is 273.
Sep 01 06:08:06 04083476cb99 workernode[3006]: 06.781319 |0|0|0| I 30 CAL0000: WorkerNode main process has started         %%10%%
Sep 01 06:08:06 04083476cb99 systemd[1]: Started [email protected] - mcs-workernode.
░░ Subject: A start job for unit [email protected] has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit [email protected] has finished successfully.
░░ 
░░ The job identifier is 273.
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting mcs-controllernode.service - mcs-controllernode...
░░ Subject: A start job for unit mcs-controllernode.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-controllernode.service has begun execution.
░░ 
░░ The job identifier is 335.
Sep 01 06:08:06 04083476cb99 IDBFile[3017]: 06.813985 |0|0|0| D 35 CAL0002: Failed to open file: /var/lib/columnstore/data1/systemFiles/dbrm/tablelocks, exception: unable to open file: /var/lib/columnstore/data1/systemFiles/dbrm/tablelocks, exception: No such file or directory
Sep 01 06:08:06 04083476cb99 controllernode[3017]: 06.814034 |0|0|0| D 29 CAL0000: TableLockServer::load(): either this is the first cluster start or could not open the save file/var/lib/columnstore/data1/systemFiles/dbrm/tablelocks         %%10%%
Sep 01 06:08:06 04083476cb99 controllernode[3017]: 06.814245 |0|0|0| D 29 CAL0000: DBRM Controller: Connected to DBRM_Worker1         %%10%%
Sep 01 06:08:06 04083476cb99 controllernode[3015]: 06.814663 |0|0|0| I 29 CAL0000: ControllerNode main process has started         %%10%%
Sep 01 06:08:06 04083476cb99 systemd[1]: Started mcs-controllernode.service - mcs-controllernode.
░░ Subject: A start job for unit mcs-controllernode.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-controllernode.service has finished successfully.
░░ 
░░ The job identifier is 335.
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting mcs-primproc.service - mcs-primproc...
░░ Subject: A start job for unit mcs-primproc.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-primproc.service has begun execution.
░░ 
░░ The job identifier is 395.
Sep 01 06:08:06 04083476cb99 env[3019]: Please install jemalloc to avoid ColumnStore performance degradation and unexpected service interruptions.
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory cGroupDefined (from getConfig) 0
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemoryFromProc  reading /proc/meminfo
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory Total mem available (bytes) 33409175552 (GIB) 31
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory cGroupDefined (from getConfig) 0
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemoryFromProc  reading /proc/meminfo
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory Total mem available (bytes) 33409175552 (GIB) 31
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory cGroupDefined (from getConfig) 0
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemoryFromProc  reading /proc/meminfo
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory Total mem available (bytes) 33409175552 (GIB) 31
Sep 01 06:08:06 04083476cb99 env[3032]: getFreeMemory : returned from getFreeMemoryFromProc 23151595520 (GIB) 21
Sep 01 06:08:06 04083476cb99 env[3032]: Starting PrimitiveServer: st = 1, sq = 10, pw = 128, pq = 10240, nb = 815653, nt = 32, nc = 1, ra = 512, db = 128, mb = 512, rd = 0, tr = 0, ss = 67108864, bp = 32
Sep 01 06:08:06 04083476cb99 env[3023]: PrimProc main process has started
Sep 01 06:08:06 04083476cb99 env[3032]: FairThreadPool started 32 thread/-s.
Sep 01 06:08:06 04083476cb99 env[3032]: started 5 high, 0 med, 0 low.
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory cGroupDefined (from getConfig) 0
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemoryFromProc  reading /proc/meminfo
Sep 01 06:08:06 04083476cb99 env[3032]: getTotalMemory Total mem available (bytes) 33409175552 (GIB) 31
Sep 01 06:08:06 04083476cb99 env[3032]: Starting ExeMgr: st = 50, qs = 20, mx = 95, cf = /etc/columnstore/Columnstore.xml
Sep 01 06:08:06 04083476cb99 systemd[1]: Started mcs-primproc.service - mcs-primproc.
░░ Subject: A start job for unit mcs-primproc.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-primproc.service has finished successfully.
░░ 
░░ The job identifier is 395.
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting mcs-writeengineserver.service - WriteEngineServer...
░░ Subject: A start job for unit mcs-writeengineserver.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-writeengineserver.service has begun execution.
░░ 
░░ The job identifier is 455.
Sep 01 06:08:06 04083476cb99 env[3121]: WriteEngineServer is ready
Sep 01 06:08:06 04083476cb99 writeengineserver[3115]: 06.945340 |0|0|0| I 32 CAL0001: WriteEngine main process has started
Sep 01 06:08:06 04083476cb99 systemd[1]: Started mcs-writeengineserver.service - WriteEngineServer.
░░ Subject: A start job for unit mcs-writeengineserver.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-writeengineserver.service has finished successfully.
░░ 
░░ The job identifier is 455.
Sep 01 06:08:06 04083476cb99 systemd[1]: Starting mcs-dmlproc.service - mcs-dmlproc...
░░ Subject: A start job for unit mcs-dmlproc.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-dmlproc.service has begun execution.
░░ 
░░ The job identifier is 515.
Sep 01 06:08:06 04083476cb99 DMLProc[3127]: 06.977129 |0|0|0| I 20 CAL0002: DMLProc starts rollbackAll.
Sep 01 06:08:06 04083476cb99 DMLProc[3127]: 06.982370 |0|0|0| I 20 CAL0002: DMLProc will rollback 0 tables.
Sep 01 06:08:06 04083476cb99 DMLProc[3127]: 06.989677 |0|0|0| I 20 CAL0002: DMLProc finished rollbackAll.
Sep 01 06:08:06 04083476cb99 DMLProc[3127]: DMLProc is ready...
Sep 01 06:08:06 04083476cb99 DMLProc[3124]: ddlpackageproc[3124]: 06.993397 |0|0|0| C 23 CAL0008: DMLProc main process has started
Sep 01 06:08:06 04083476cb99 ddlpackageproc[3124]: 06.993397 |0|0|0| C 23 CAL0008: DMLProc main process has started
Sep 01 06:08:06 04083476cb99 systemd[1]: Started mcs-dmlproc.service - mcs-dmlproc.
░░ Subject: A start job for unit mcs-dmlproc.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-dmlproc.service has finished successfully.
░░ 
░░ The job identifier is 515.
Sep 01 06:08:07 04083476cb99 systemd[1]: Starting mcs-ddlproc.service - mcs-ddlproc...
░░ Subject: A start job for unit mcs-ddlproc.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-ddlproc.service has begun execution.
░░ 
░░ The job identifier is 575.
Sep 01 06:08:07 04083476cb99 DDLProc[3146]: DDLProc is ready...
Sep 01 06:08:07 04083476cb99 DDLProc[3144]: ddlpackageproc[3144]: 07.035620 |0|0|0| C 23 CAL0008: DDLProc main process has started
Sep 01 06:08:07 04083476cb99 ddlpackageproc[3144]: 07.035620 |0|0|0| C 23 CAL0008: DDLProc main process has started
Sep 01 06:08:07 04083476cb99 systemd[1]: Started mcs-ddlproc.service - mcs-ddlproc.
░░ Subject: A start job for unit mcs-ddlproc.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mcs-ddlproc.service has finished successfully.
░░ 
░░ The job identifier is 575.
Sep 01 06:08:07 04083476cb99 su[3151]: (to mysql) root on none
Sep 01 06:08:07 04083476cb99 su[3151]: pam_env(su:session): Unable to open env file: /etc/default/locale
Sep 01 06:08:07 04083476cb99 su[3151]: pam_unix(su:session): session opened for user mysql(uid=100) by root(uid=0)
Sep 01 06:08:07 04083476cb99 su[3151]: pam_unix(su:session): session closed for user mysql
Sep 01 06:08:07 04083476cb99 systemd[1]: Finished mariadb-columnstore.service - mariadb-columnstore.
░░ Subject: A start job for unit mariadb-columnstore.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit mariadb-columnstore.service has finished successfully.
░░ 
░░ The job identifier is 213.
Sep 01 06:08:07 04083476cb99 systemd[1]: Stopping mariadb.service - MariaDB 11.8.3 database server...
░░ Subject: A stop job for unit mariadb.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit mariadb.service has begun execution.
░░ 
░░ The job identifier is 635.
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] InnoDB: FTS optimize thread exiting.
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] InnoDB: Starting shutdown...
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] InnoDB: Buffer pool(s) dump completed at 250901  6:08:07
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] InnoDB: Shutdown completed; log sequence number 58772; transaction id 33
Sep 01 06:08:07 04083476cb99 mariadbd[2172]: 2025-09-01  6:08:07 0 [Note] /usr/sbin/mariadbd: Shutdown complete
Sep 01 06:08:07 04083476cb99 systemd[1]: mariadb.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit mariadb.service has successfully entered the 'dead' state.
Sep 01 06:08:07 04083476cb99 systemd[1]: Stopped mariadb.service - MariaDB 11.8.3 database server.
░░ Subject: A stop job for unit mariadb.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit mariadb.service has finished.
░░ 
░░ The job identifier is 635 and the job result is done.
Sep 01 06:08:07 04083476cb99 systemd[1]: mariadb.service: Consumed 971ms CPU time, 102.9M memory peak.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit mariadb.service completed and consumed the indicated resources.
Sep 01 06:08:09 04083476cb99 systemd[1]: systemd-binfmt.service - Set Up Additional Binary Formats was skipped because of an unmet condition check (ConditionPathIsMountPoint=/proc/sys/fs/binfmt_misc).
░░ Subject: A start job for unit systemd-binfmt.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit systemd-binfmt.service has finished successfully.
░░ 
░░ The job identifier is 636.
Sep 01 06:08:09 04083476cb99 systemd[1]: Reload requested from client PID 3187 ('systemctl') (unit init.scope)...
Sep 01 06:08:09 04083476cb99 systemd[1]: Reloading...
Sep 01 06:08:09 04083476cb99 systemd[1]: Reloading finished in 53 ms.
Sep 01 06:08:09 04083476cb99 sudo[141]: pam_unix(sudo:session): session closed for user root
Sep 01 06:08:09 04083476cb99 sudo[3215]:     root : TTY=pts/0 ; PWD=/scripts ; USER=root ; COMMAND=/usr/bin/systemctl status mariadb --no-pager
Sep 01 06:08:09 04083476cb99 sudo[3215]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
Sep 01 06:08:09 04083476cb99 sudo[3215]: pam_unix(sudo:session): session closed for user root
Sep 01 06:08:09 04083476cb99 sudo[3219]:     root : TTY=pts/0 ; PWD=/scripts ; USER=root ; COMMAND=/usr/bin/journalctl -xe --no-pager
Sep 01 06:08:09 04083476cb99 sudo[3219]: pam_unix(sudo:session): session opened for user root(uid=0) by root(uid=0)
+ get_columnstore_logs
+ [[ columnstore == \c\o\l\u\m\n\s\t\o\r\e ]]
+ bb_log_info 'storing Columnstore logs in columnstore_logs'
+ set +x
INFO: storing Columnstore logs in columnstore_logs
+ set +ex
./bash_lib.sh: line 493: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 494: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 493: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 494: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 493: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 494: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 493: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 494: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 493: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 494: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 493: /home/buildbot/columnstore_logs: No such file or directory
./bash_lib.sh: line 494: /home/buildbot/columnstore_logs: No such file or directory
ERROR: mariadb service didn't start properly after installation
+ exit 1
(base) 

@grooverdan grooverdan marked this pull request as draft September 8, 2025 03:36
BUILDERS_UPGRADE.append(builder_name_autobake + "-major-upgrade")
BUILDERS_UPGRADE.append(builder_name_autobake + "-distro-upgrade")

if arch in ["amd64", "aarch64"]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I my understanding correct that columnstore packages are not built for ppc64le as per server configuration?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

package_list=mariadb-server
;;
columnstore)
get_packages_file_mirror
Copy link
Collaborator

@RazvanLiviuVarzaru RazvanLiviuVarzaru Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test is useful, if it exists with an error.
Imagine on the server development side or when configuring the buildbot docker environment,
a mistake is made and columnstore packages are no longer generated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An error is good. This would have failed with cannot install columnstore at a later step, like any other failure to generate package, that is in the package list below.

grep 'spider' | awk '{print $2}' |
sort -u | xargs)
fi
if grep -si tokudb Packages; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to link the MDEV / commit that removed TokuDB.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MDEV-19780 Remove the TokuDB storage engine (for 10.6). there was a disable for an earlier version.

all)
get_packages_file_mirror
package_list=$(grep "^Package:" Packages |
grep -vE 'galera' |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to link MDEV / commit that changed this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the package mariadb-galera-server was in 10.0 and 10.1 and 10.2 had the galera "patch" as part of mariadb-server.

I've yet to find a specific MDEV but given it was always going to be EOL with 10.1 I suspect there isn't one.

@RazvanLiviuVarzaru
Copy link
Collaborator

RazvanLiviuVarzaru commented Sep 9, 2025

@grooverdan some minor comments, I will schedule a full run on buildbot.dev.mariadb.org with the changes
presented in this patch so we get a clear picture if the MDEV's are solved.

@grooverdan
Copy link
Member Author

@grooverdan some minor comments, I will schedule a full run on buildbot.dev.mariadb.org with the changes presented in this patch so we get a clear picture if the MDEV's are solved.

Thanks. Would provide a good basis for the columnstore bug if it still exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants