Tekton Orchestrated Benchmarking POC

[kalantar]

See tekton_poc/README.md

[sriumcp]

Does this skip the model download if it is already present in the path? I believe this is the entire idea of using the PVC to reuse models locally ... right?

[jgchn]

I believe hf download detects whether or not the model has been downloaded. Furthermore mkdir -p accepts folders that already exist, so I think this is good.

[kalantar]

hf download is intelligent and will skip the work. In the current implementation, the PVC is created locally and used only once. Each parallel experiment is running in a different namespace. In principle, we can define a PVC to an existing PV that has only once copy of the model. If we do this, we probably want to add a task before the parallelism to download the model.

[sriumcp]

Is this chart needed?

Can we simply use the inference-perf container as part of the StepAction, and configure it so that the step actually directly sends the Inference request?

This approach eliminates the chart, along with the need for a separate harness pod (the Tekton step will run as part of some pod anyway).

[kalantar]

The step has now been updated to use the llm-d-benchmark image directly. It isn't as clean as it might be to use the inference-perf image directly. However, it should also be possible to modify it to be easier to use.

[sriumcp]

This section needs to be completed.

Let us start by implementing HTTP(S)-based results upload to an S3 compatible storage.

See example here: https://www.ibm.com/docs/en/storage-scale/5.2.1?topic=storage-connectivity-cloud-object

[kalantar]

This has been done; the results folder is tarred and uploaded to and s3 compatible bucket.

[jgchn]

I believe hf download detects whether or not the model has been downloaded. Furthermore mkdir -p accepts folders that already exist, so I think this is good.

[jgchn]

I am guessing the user would be creating these values yaml files for the experiment pipeline?

[kalantar]

Today, the pipeline takes the location (url) as input. It can take a stringified values file as well. When sweeping through values, the values file ise overridden using --set.

An alternative is to use a (yaml) description of the desired environment to generate the values files. This seems to assume we can express it more simply than the values files to today. It is not clear to me that this is the case.

[jgchn]

Curious what the "unrolled" output looks like here.

[jgchn]

Suggested change

	export $NAMESPACE=your_namespace
	export NAMESPACE=your_namespace

[jgchn]

This appears as one-liner after rendering. May need to un-indent.

[jgchn]

Suggested change

	```shell
	kubectl apply -f pipeline/stepactions.yaml
	```shell
	cd tekton-poc
	kubectl apply -f pipeline/stepactions.yaml

[jgchn]

Tested and looks good! I was able to execute three experiments in parallel asynchronously, and saw the results uploaded to IBM COS. Lets get this merged!

[jgchn]

Suggested change

	This proof of concept currently implements a variation of the inference-scheduling [scenairo](https://github.com/llm-d/llm-d-benchmark/blob/main/scenarios/guides/inference-scheduling.sh)/[experiment](https://github.com/llm-d/llm-d-benchmark/blob/main/experiments/inference-scheduling.yaml).
	This proof of concept currently implements a variation of the inference-scheduling [scenairo](https://github.com/llm-d/llm-d-benchmark/blob/main/scenarios/guides/inference-scheduling.sh)/[experiment](https://github.com/llm-d/llm-d-benchmark/blob/main/experiments/inference-scheduling.yaml).
	To change the Inference Scheduling configs for the experiment, update `tekton-poc/examples/inference-scheduling/gaie-values.yaml`, then `git push` to your fork, and supply the new URL to `inference-scheduling/` for the `experimentBaseUrl` value in the [pipeline run yaml](./pipeline/pipelinerun-matrix.yaml#L46).