Cherry-pick commits from pxf-archive to cloudberry-pxf

[tuhaihe]

Cloudberry Database and PXF version

What happened

We need to catch up the codebase of pxf-archive.

What you think should happen instead

No response

How to reproduce

No response

Operating System

No response

Anything else

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.

[MisterRaindrop]

hello @tuhaihe

I tried to cherry-pick the latest commits from pxf-archive. However, I found that the base has changed, and each cherry-pick requires resolving a large number of conflicts, which seem to be adaptations for GP7, GP6, or Cloudberry. So I was wondering if we could first bring over the latest commits from pxf-archive based on the common commit, and then gradually adapt our later modifications accordingly.

I’ve already found the common commit with the ID 08a30ac.

This approach, while involving some effort, seems feasible.

[tuhaihe]

Hi @MisterRaindrop sounds good!

I can help set up a work branch using the most recent archived code from greenplum-db/pxf-archive. That way, the developers can easily submit pull requests to bring over the improvements from the main branch to this work branch. Once we've given it a good test drive, we can go ahead and make this work branch the new main branch! How does that sound?

I will let more PXF developers know this plan and have their feedback on this.

[MisterRaindrop]

@tuhaihe GLTM, thanks

[MisterRaindrop]

@my-ship-it hi max, what do you think? if it's ok, we can prepare a base branch

[my-ship-it]

@my-ship-it hi max, what do you think? if it's ok, we can prepare a base branch

I think you're right. We can directly re-adapt Cloudberry based on the latest code, which will significantly reduce the workload.

[my-ship-it]

@my-ship-it hi max, what do you think? if it's ok, we can prepare a base branch

I think you're right. We can directly re-adapt Cloudberry based on the latest code, which will significantly reduce the workload.

And in addition, we'd better set up CI first, which is good to all parties collaborating to merge code and ensuring code quality

[MisterRaindrop]

Sounds good!

When we get the latest code, we should prioritize getting the CI (Continuous Integration) working. With the guarantee of CI, we can reduce issues during the merging process.

hi @ostinru what do you think?

[ostinru]

When we are talkint about CI... I can see following CI stages:

• FAST CIRCUIT: let github actions run java unit tests & golang unit tests
• HEAVY CIRCUIT: run pxf-autmation - I have first steps on making this work for open-gpdb, however there still long way to finish it.

1. I would start from very basic CI(java code compiles & unit tests passed): Add fast-CI workflows to github actions #6
2. Then, cherry-pick commits
3. Then, add golang to CI: https://github.com/open-gpdb/pxf/blob/main/.github/workflows/pxf-cli-on-pull-requests.yml As far as I remember it should work since pxf 6.7 or something like this.

Obviously, automation as e2e tests is the best option for CI. However it is not easy to run since S3 files in vmware/broadcom bucket expired, and now we should configure hadoop cluster on our own. I managed to run smoke tests, but it still far from something that is ready to be merged.

[MisterRaindrop]

hi max, @ostinru already answer the CI tasks have been listed.

At this stage
step 1: we should add CI based on the latest code.
step 2: Once CI is added we need to merge the commits from the main branch.
step 3: add golang to ci...

Are there any supplements to the entire process? @my-ship-it

[my-ship-it]

hi max, @ostinru already answer the CI tasks have been listed.

At this stage step 1: we should add CI based on the latest code. step 2: Once CI is added we need to merge the commits from the main branch. step 3: add golang to ci...

Are there any supplements to the entire process? @my-ship-it

Thanks guys. The plan looks good to me. 👍

[MisterRaindrop]

max will help prepare the latest code for PXF.

Once everything is ready, we can proceed with Step 1. Regarding Step 1, is there a significant amount of work involved? If so, could you list the subtasks that can be broken down? Can we involve more community members to help us complete it together?

If we are to list the subtask, I suggest opening a separate issue to track it.

what do you think? @ostinru

[ostinru]

Once everything is ready, we can proceed with Step 1. Regarding Step 1, is there a significant amount of work involved? If so, could you list the subtasks that can be broken down? Can we involve more community members to help us complete it together?

It depends on definition 'CI based on the latest code'. I would estimate:

• java compiles - 10 minutes to run tests and merge PR
• golang compiles - 15 minutes to copy github action file, test and merge (after merging pxf 6.7.x patches)
• C/C++ compiles - 2-5 days to create proper devops scripts
• automation works - something like 5-15 working days

[my-ship-it]

Hi guys, a branch with the latest pxf-archive code has been created:
https://github.com/apache/cloudberry-pxf/tree/merge-with-upstream

[MisterRaindrop]

@ostinru
Is there a task that needs to be broken down? If so, please help create a new issue to track it. The broken-down tasks can allow interested community members to get involved.

[MisterRaindrop]

As far as I know, there are some people interested in this. However, it's uncertain how strong the support will be.