[Contributions Welcome] Improving Our Fine-Tuning Pipeline

[sergiopaniego]

Call for Contributions: Improving Our Fine-Tuning Pipeline

Hi all!

We've already generated some detection results (see below 👇) from our current fine-tuning pipeline, and we're opening a contributions call to help us take the project further! We believe the approach is promising and want to build on top of it to unlock even more exciting outcomes.

Contribution Ideas

These are some of the improvements we're considering, but feel free to suggest others!

• Add LoRA [Add] LoRA support #10
• Add QLoRA Add QLoRA support file #13
• Add data augmentation Added data augmentation via Albumentations and W&B #11
• Add logging via Weights & Biases Added data augmentation via Albumentations and W&B #11
• Improve training procedure (e.g., save best checkpoint, use validation data) Improved Training Loop: Add Validation and Best Checkpoint Saving #23
• Update dataset to COCO with location labels
  → Goal: build generic scripts with a --dataset flag generic script for Dataset #17
• Create an evaluation script to compute mAP, IoU, Recall/Precision FEAT: Basic and Advanced Evaluation: WIP #14
• Test other VLMs without native object detection
  → Again, generalize using a --model flag
• Add location tokens to tokenizer and train the embedding Add location tokens to training #34
• Improve overall documentation

We’d love your help! If you’re interested in contributing or have new ideas, feel free to open an issue or PR.

[JagjeetChauhan]

I would like to work on this. Please assign me this task.

[Vidit-Ostwal]

@sergiopaniego #10, and #11 links are broken.

[Moksh-10]

I am interested to work on this, can you assign me the task?

[Korn-Monit]

I am interested too

[ariG23498]

Hey folks! It is great to see you all.

Please feel free to take up any task and start a PR (tagging this issue).

[Vidit-Ostwal]

I am taking up the Add QLoRA.

Hey folks! It is great to see you all.

Please feel free to take up any task and start a PR (tagging this issue).

[PT-10]

I'd like to work on adding the evaluation script, taking it up.

[bhaskar9221]

Hello @sergiopaniego and @ariG23498 , I am taking up the task to work on Improve training procedure (e.g., save best checkpoint, use validation data) first.
Thank you

[prasannakotyal]

Hi @sergiopaniego and @ariG23498,

I've been looking at the contribution ideas and have been working on addressing a couple of them:

• Improve training procedure (e.g., save best checkpoint, use validation data)
• Create an evaluation script to compute mAP, IoU, Recall/Precision

I've implemented the logic for both of these points. The training script (train.py) now includes validation, checkpoint saving (including saving the best model based on validation loss), and enhanced logging. I've also created a new script (evaluate.py) that calculates mAP, IoU, Precision, and Recall.

However, I'm running into some challenges with full end-to-end verification on my local setup. I have a laptop with an RTX 3050 (4GB VRAM), and even after optimizing the batch size (set it to 1), I'm still encountering memory errors that prevent the training script from completing an epoch.

While I haven't been able to run the entire training-to-evaluation pipeline myself, I've done a thorough code review of the changes.

Would anyone with access to hardware with more VRAM or system RAM be able to help test these changes? Or are there any further optimization suggestions you might have that could help run this on lower-spec machines?

I'm happy to open a draft PR with the code for review, but I wanted to check in here first regarding the testing difficulty.

Thanks!

[AnimeshMaheshwari22]

Hi I would like to contribute! Taking up Add location tokens to tokenizer and train the embedding

[FazeelUsmani]

Hi, I'm interested to work on this. Please assign me

[Arush04]

Hi @sergiopaniego @ariG23498 Please review:

1. QLoRA
   PR

2. Update dataset to COCO with location labels
   → Goal: build generic scripts with a --dataset flag
   PR

[AnimeshMaheshwari22]

Hi @sergiopaniego @ariG23498 create a PR for Add location tokens to tokenizer and train the embedding

Please do review the same. Thanks

[rbavery]

is improving the data mix within scope? I work a lot with overhead aerial, satellite, and drone imagery. There are many open labels datasets out there that could be paired with this imagery. If in scope, down to suggest.