Skip to content

Commit d562c34

Browse files
docs: complete README
1 parent 58bb900 commit d562c34

File tree

3 files changed

+56
-4
lines changed

3 files changed

+56
-4
lines changed

README.md

Lines changed: 56 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -107,8 +107,61 @@ docker run --rm -v "./evaluation-workdir/":"/home/user/evaluation-workdir" mpatc
107107

108108
#### Expected outcome
109109

110-
TODO TODO TODO
110+
> [!NOTE]
111+
> If you executed the evaluation in a custom directory, all mentioned files will be located there.
112+
113+
114+
The verification should begin with output that looks similar to the following screenshot:
115+
```shell
116+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.PatcherEvaluationMainKt.main()
117+
INFO: Starting experiment initialization.
118+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.execution.EvalUtilsKt.createOrLoadSamples()
119+
INFO: Loading dataset for C with 1 usable repositories
120+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.execution.EvalUtilsKt.createOrLoadSamples()
121+
...
122+
INFO: Loading dataset for TypeScript with 1 usable repositories
123+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.execution.EvalUtilsKt.createOrLoadSamples()
124+
INFO: Done.
125+
126+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.PatcherEvaluationMainKt.main()
127+
INFO: Processing 5 repos in parallel
128+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.PatcherEvaluationMainKt.main()
129+
INFO: Already considered 0 repos.
130+
2025-08-21 14:36:30 [main] org.variantsync.evaluation.PatcherEvaluationMainKt.main()
131+
INFO: Already processed a total of 0 evaluation runs.
132+
133+
2025-08-21 14:36:35 [main] org.variantsync.evaluation.PatcherEvaluationMainKt.main()
134+
INFO: Considering a total of 85 cherry-picks for repetition 1
135+
2025-08-21 14:36:35 [pool-1-thread-3] org.variantsync.evaluation.execution.EvalUtilsKt.cloneGitHubRepo()
136+
INFO: cloning https://github.com/tensorflow/serving.git into /home/user/evaluation-workdir/REPOS/tensorflow_serving
137+
...
138+
```
139+
The output shows that the dataset used for verification contains one repository for each project language. The projects are cloned into the `evaluation-workdir`.
140+
Once a project has been cloned, the patchers are evaluated on the cherry picks (i.e., patches) that have been found for that repository.
111141

142+
The verification should complete with the following output:
143+
```shell
144+
Latexmk: All targets (metrics-verification.pdf) are up-to-date
145+
146+
++++++++++++++++++++++++++++++++++++
147+
Analysis done
148+
++++++++++++++++++++++++++++++++++++
149+
150+
The result table can be found under evaluation-workdir/metrics-verification.pdf
151+
```
152+
153+
After all repositories have been considered, the result analysis is executed.
154+
The raw results can be found in the `evaluation-workdir/results` directory.
155+
156+
157+
158+
In addition, the script generates a PDF file with a result table similar to the one presented in our paper.
159+
This table can be found under `evaluation-workdir/metrics-verification.pdf`.
160+
It should look similar to this:
161+
![](misc/verification-results.png)
162+
163+
> [!NOTE]
164+
> The verification results shown are based on only a tiny portion of our dataset and are therefore not representative.
112165
113166
# Starting the reproduction
114167
Once you have verified the correct installation, you can start the reproduction similar to how you started the verification.
@@ -125,7 +178,8 @@ On other machines, you may start a Docker container from the Docker image with t
125178
docker run --rm -v "./evaluation-workdir/":"/home/user/evaluation-workdir" mpatch-reproduction reproduction
126179
```
127180

128-
181+
> [!NOTE]
182+
> The results of the reproduction will be stored in the same manner as the results of the verification.
129183
130184
> [!NOTE]
131185
> Our evaluation processes large amounts of data.

docker/run-simulation.sh

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,11 @@ start() {
66

77
if [ "$1" == 'reproduction' ]; then
88
echo "Running full reproduction of evaluation on the entire patch dataset."
9-
ls -l
109
java -jar -Dtinylog.configuration=/home/user/tinylog.properties cherries.jar config-reproduction.properties
1110

1211
analysis $1
1312
elif [ "$1" == 'verification' ]; then
1413
echo "Verifying the evaluation setup on a tiny subset of the patch dataset."
15-
ls -l
1614
java -jar -Dtinylog.configuration=/home/user/tinylog.properties cherries.jar config-verification.properties
1715

1816
analysis $1

misc/verification-results.png

528 KB
Loading

0 commit comments

Comments
 (0)