Skip to content

Commit aa2180b

Browse files
autophagyautophagy
committed
[FLINK-38231][python] Standardise use of uv for PyFlink building, testing & linting
1 parent 250ab88 commit aa2180b

File tree

14 files changed

+152
-366
lines changed

14 files changed

+152
-366
lines changed

‎docs/content/docs/flinkDev/building.md

Lines changed: 40 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,15 @@ The `fast` and `skip-webui-build` profiles have a significant impact on the buil
7979
# the version printed here must be 3.9, 3.10, 3.11 or 3.12
8080
```
8181

82+
3. Uv for building, installing and managing Python environments (optional)
83+
84+
The tool [uv](https://docs.astral.sh/uv/) is a Python package and project manager that can make building and installing PyFlink from source much easier.
85+
To verify that it is installed:
86+
87+
```shell
88+
$ uv --version
89+
```
90+
8291
3. Build PyFlink with Cython extension support (optional)
8392

8493
To build PyFlink with Cython extension support, you’ll need a C compiler. It's a little different on how to install the C compiler on different operating systems:
@@ -87,21 +96,49 @@ The `fast` and `skip-webui-build` profiles have a significant impact on the buil
8796
8897
* **Mac OS X** To install GCC on Mac OS X, you need to download and install "[Command Line Tools for Xcode](https://developer.apple.com/downloads/index.action)", which is available in Apple’s developer page.
8998
90-
You also need to install the dependencies with following command:
99+
You also need to install the dependencies with following commands. Either with `uv`:
100+
101+
```shell
102+
$ uv pip install --group flink-python/pyproject.toml:dev
103+
```
104+
105+
or with `python` and `pip`:
91106
92107
```shell
93108
$ python -m pip install --group flink-python/pyproject.toml:dev
94109
```
95110
96111
#### Installation
97112
98-
Then go to the root directory of flink source code and run this command to build the sdist package and wheel package of `apache-flink` and `apache-flink-libraries`:
113+
##### With `uv`
114+
115+
To build and install the `apache-flink` and `apache-flink-libraries` packages with `uv`, go to the root directory of flink source code and run the following:
116+
117+
```bash
118+
cd flink-python; uv build; uv build apache-flink-libraries --sdist;
119+
```
120+
121+
The sdist package of `apache-flink-libraries` will be found under `./flink-python/apache-flink-libraries/dist/`. It can be installed as follows:
122+
123+
```bash
124+
uv pip install apache-flink-libraries/dist/*.tar.gz
125+
```
126+
127+
The sdist and wheel packages of `apache-flink` will be found under `./flink-python/dist/`. Either of them could be used for installation, such as:
128+
129+
```bash
130+
uv pip install dist/*.whl
131+
```
132+
133+
##### With `python` and `pip`
134+
135+
To build and install the `apache-flink` and `apache-flink-libraries` packages with `python` and `pip`, go to the root directory of flink source code and run the following:
99136
100137
```bash
101138
cd flink-python; python setup.py sdist bdist_wheel; cd apache-flink-libraries; python setup.py sdist; cd ..;
102139
```
103140
104-
The sdist package of `apache-flink-libraries` will be found under `./flink-python/apache-flink-libraries/dist/`. It could be installed as following:
141+
The sdist package of `apache-flink-libraries` will be found under `./flink-python/apache-flink-libraries/dist/`. It can be installed as follows:
105142
106143
```bash
107144
python -m pip install apache-flink-libraries/dist/*.tar.gz

‎docs/static/downloads/setup-pyflink-virtual-env.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ echo "Detected OS: ${sys_os}"
2020
sys_machine=$(uname -m)
2121
echo "Detected machine: ${sys_machine}"
2222

23-
wget "https://github.com/astral-sh/uv/releases/download/0.5.23/uv-installer.sh" -O "uv-installer.sh"
23+
wget "https://github.com/astral-sh/uv/releases/download/0.8.9/uv-installer.sh" -O "uv-installer.sh"
2424
chmod +x uv-installer.sh
2525

2626
UV_UNMANAGED_INSTALL="uv-bin" ./uv-installer.sh
@@ -34,10 +34,10 @@ source venv/bin/activate ""
3434
# install PyFlink dependency
3535
if [[ $1 = "" ]]; then
3636
# install the latest version of pyflink
37-
pip install apache-flink
37+
uv pip install apache-flink
3838
else
3939
# install the specified version of pyflink
40-
pip install "apache-flink==$1"
40+
uv pip install "apache-flink==$1"
4141
fi
4242

4343
# deactivate the python virtual environment

‎flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,9 @@ cd "${FLINK_PYTHON_DIR}"
5454

5555
if [[ -d "dist" ]]; then rm -Rf dist; fi
5656

57-
pushd apache-flink-libraries
57+
uv build apache-flink-libraries --sdist
5858

59-
python setup.py sdist
60-
61-
popd
62-
63-
python setup.py sdist
59+
uv build
6460

6561
cd dev
6662

‎flink-end-to-end-tests/test-scripts/test_pyflink.sh

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -97,19 +97,7 @@ cd "${FLINK_PYTHON_DIR}"
9797

9898
rm -rf dist
9999

100-
uv pip install --group dev
101-
102-
python setup.py sdist
103-
104-
pushd apache-flink-libraries
105-
106-
python setup.py sdist
107-
108-
uv pip install dist/*
109-
110-
popd
111-
112-
uv pip install dist/*
100+
uv pip install --group dev .
113101

114102
cd dev
115103

‎flink-python/README.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,5 @@ Then you can activate your environment and run tests, for example:
6767

6868
```shell
6969
source pyflink_39/bin/activate
70-
uv pip install --group dev
7170
./dev/lint-python.sh
7271
```

‎flink-python/apache-flink-libraries/setup.cfg renamed to ‎flink-python/apache-flink-libraries/pyproject.toml

Lines changed: 25 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,30 @@
1515
# See the License for the specific language governing permissions and
1616
# limitations under the License.
1717
################################################################################
18+
[build-system]
19+
# Minimum requirements for the build system to execute.
20+
requires = [
21+
"setuptools>=75.3",
22+
"wheel",
23+
]
1824

19-
[bdist_wheel]
20-
universal = 1
25+
[project]
26+
name = "apache-flink-libraries"
27+
description = "Apache Flink Libraries"
28+
readme = "README.md"
29+
license = "Apache-2.0"
30+
authors = [
31+
{name = "Apache Software Foundation", email = "[email protected]"}
32+
]
33+
requires-python = ">=3.9"
34+
classifiers = [
35+
"Development Status :: 5 - Production/Stable",
36+
"Programming Language :: Python :: 3.9",
37+
"Programming Language :: Python :: 3.10",
38+
"Programming Language :: Python :: 3.11",
39+
"Programming Language :: Python :: 3.12",
40+
]
41+
dynamic = ["version", "dependencies"]
2142

22-
[metadata]
23-
description-file = README.md
43+
[project.urls]
44+
homepage = "https://flink.apache.org"

‎flink-python/apache-flink-libraries/setup.py

Lines changed: 0 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -215,27 +215,11 @@ def find_file_path(pattern):
215215
PACKAGE_DATA['pyflink.licenses'] = ['*']
216216

217217
setup(
218-
name='apache-flink-libraries',
219218
version=VERSION,
220219
packages=PACKAGES,
221220
include_package_data=True,
222221
package_dir=PACKAGE_DIR,
223222
package_data=PACKAGE_DATA,
224-
url='https://flink.apache.org',
225-
license='https://www.apache.org/licenses/LICENSE-2.0',
226-
author='Apache Software Foundation',
227-
author_email='[email protected]',
228-
python_requires='>=3.9',
229-
description='Apache Flink Libraries',
230-
long_description=long_description,
231-
long_description_content_type='text/markdown',
232-
classifiers=[
233-
'Development Status :: 5 - Production/Stable',
234-
'License :: OSI Approved :: Apache Software License',
235-
'Programming Language :: Python :: 3.9',
236-
'Programming Language :: Python :: 3.10',
237-
'Programming Language :: Python :: 3.11',
238-
'Programming Language :: Python :: 3.12'],
239223
)
240224
finally:
241225
if in_flink_source:

‎flink-python/dev/build-wheels.sh

Lines changed: 23 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -15,41 +15,35 @@
1515
# limitations under the License.
1616
set -e -x
1717

18-
## 1. install python env
19-
dev/lint-python.sh -s py_env
20-
21-
PY_ENV_DIR=`pwd`/dev/.uv/envs
22-
py_env=("3.9" "3.10" "3.11" "3.12")
23-
## 2. install dependency
24-
for ((i=0;i<${#py_env[@]};i++)) do
25-
source `pwd`/dev/.uv/envs/${py_env[i]}/bin/activate
26-
echo "Installing dependencies for environment: ${py_env[i]}"
27-
uv pip install --group dev
28-
deactivate
29-
done
18+
## 1. Setup virtual environment with uv
19+
dev/lint-python.sh -s basic
20+
21+
python_versions=("3.9" "3.10" "3.11" "3.12")
22+
23+
if [[ "$(uname)" != "Darwin" ]]; then
24+
# force the linker to use the older glibc version in Linux
25+
export CFLAGS="-I. -include dev/glibc_version_fix.h"
26+
fi
27+
28+
source "$(pwd)"/dev/.uv/bin/activate
3029

31-
## 3. build wheels
32-
for ((i=0;i<${#py_env[@]};i++)) do
33-
echo "Building wheel for environment: ${py_env[i]}"
34-
if [[ "$(uname)" != "Darwin" ]]; then
35-
# force the linker to use the older glibc version in Linux
36-
export CFLAGS="-I. -include dev/glibc_version_fix.h"
37-
fi
38-
${PY_ENV_DIR}/${py_env[i]}/bin/python setup.py bdist_wheel
30+
## 2. Build wheels for each Python version
31+
for ((i=0;i<${#python_versions[@]};i++)) do
32+
echo "Building wheel for Python: ${python_versions[i]}"
33+
uv build --wheel --python "${python_versions[i]}"
3934
done
4035

41-
## 4. convert linux_x86_64 wheel to manylinux1 wheel in Linux
36+
## 3. Convert linux_x86_64 wheel to manylinux1 wheel in Linux
4237
if [[ "$(uname)" != "Darwin" ]]; then
4338
echo "Converting linux_x86_64 wheel to manylinux1"
44-
source `pwd`/dev/.uv/bin/activate
45-
# 4.1 install patchelf and auditwheel
46-
uv pip install patchelf==0.17.2.1 auditwheel==3.2.0
47-
# 4.2 convert Linux wheel
4839
for wheel_file in dist/*.whl; do
49-
auditwheel repair ${wheel_file} -w dist
50-
rm -f ${wheel_file}
40+
uv run --group auditwheel auditwheel repair "${wheel_file}" -w dist
41+
rm -f "${wheel_file}"
5142
done
52-
deactivate
5343
fi
54-
## see the result
44+
45+
deactivate
46+
47+
## 4. Output the result
48+
echo "Build finished - created the following wheels:"
5549
ls -al dist/

‎flink-python/dev/install_command.sh

Lines changed: 0 additions & 49 deletions
This file was deleted.

0 commit comments

Comments
 (0)