Updates beaker-py version to >2. #1021

finbarrtimbers · 2025-09-18T17:34:09Z

Fixes #677.

Experiments:

Debug run: Beaker
Tool run: Beaker
Multi-node run: Beaker
Finetune run: Beaker

hamishivi

Hmm running this I get

Traceback (most recent call last):
  File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 924, in <module>
    main()
  File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 915, in main
    tasks=[make_task_spec(args, full_command, i, beaker_secrets, whoami, args.resumable) for i, full_command in enumerate(full_commands)],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 915, in <listcomp>
    tasks=[make_task_spec(args, full_command, i, beaker_secrets, whoami, args.resumable) for i, full_command in enumerate(full_commands)],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 852, in make_task_spec
    spec = beaker.TaskSpec(
           ^^^^^^^^^^^^^^^
AttributeError: module 'beaker' has no attribute 'TaskSpec'

finbarrtimbers · 2025-09-20T15:11:43Z

Oh, that's interesting. Can you share the script? I don't know why my runs didn't catch that.

…

On Fri, Sep 19, 2025 at 14:21, Hamish Ivison ***@***.***> wrote: ***@***.**** commented on this pull request. Hmm running this I get Traceback (most recent call last): File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 924, in <module> main() File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 915, in main tasks=[make_task_spec(args, full_command, i, beaker_secrets, whoami, args.resumable) for i, full_command in enumerate(full_commands)], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 915, in <listcomp> tasks=[make_task_spec(args, full_command, i, beaker_secrets, whoami, args.resumable) for i, full_command in enumerate(full_commands)], ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/weka/oe-adapt-default/hamishi/pr_review/open-instruct/mason.py", line 852, in make_task_spec spec = beaker.TaskSpec( ^^^^^^^^^^^^^^^ AttributeError: module 'beaker' has no attribute 'TaskSpec' — Reply to this email directly, view it on GitHub <#1021 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAYN6ROF7C2FQHRBZNIMYNL3TRQVFAVCNFSM6AAAAACG4OR37KVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTENBWHE4TGMRXGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

hamishivi · 2025-09-21T01:26:52Z

@finbarrtimbers running a slightly edited version of scripts/train/tulu3/finetune_8b.sh:

python mason.py     --cluster ai2/jupiter-cirrascale-2     --workspace ai2/tulu-3-dev     --priority high     --image nathanl/open_instruct_auto --pure_docker_mode     --preemptible     --num_nodes 8     --budget ai2/oe-adapt     --gpus 8 -- accelerate launch     --mixed_precision bf16     --num_processes 8     --use_deepspeed     --deepspeed_config_file configs/ds_configs/stage3_no_offloading_accelerate.conf     --deepspeed_multinode_launcher standard     open_instruct/finetune.py     --exp_name tulu3_8b_sft     --model_name_or_path meta-llama/Llama-3.1-8B     --model_revision main     --tokenizer_name meta-llama/Llama-3.1-8B     --tokenizer_revision main     --use_slow_tokenizer     --dataset_mixer_list allenai/tulu-3-sft-mixture 512     --max_seq_length 4096     --per_device_train_batch_size 1     --gradient_accumulation_steps 2     --learning_rate 5e-06     --lr_scheduler_type linear     --warmup_ratio 0.03     --weight_decay 0.0     --num_train_epochs 2     --reduce_loss sum     --use_flash_attn     --gradient_checkpointing     --report_to wandb     --with_tracking     --logging_steps 1     --seed 8

(edited just to reduce the dataset size)

- Changed beaker_client.workspace.secrets() to beaker_client.secret.list() - Changed beaker_client.account.whoami() to beaker_client.user.get() - Changed beaker.ExperimentSpec to beaker.BeakerExperimentSpec - Changed beaker.Constraints to beaker.BeakerConstraints - Changed beaker.RetrySpec to beaker.BeakerRetrySpec

- Changed all beaker.EnvVar to beaker.BeakerEnvVar - Changed beaker.DataMount to beaker.BeakerDataMount - Changed beaker.DataSource to beaker.BeakerDataSource - Changed beaker.TaskResources to beaker.BeakerTaskResources - Changed beaker.ImageSource to beaker.BeakerImageSource - Changed beaker.ResultSpec to beaker.BeakerResultSpec - Changed beaker.TaskContext to beaker.BeakerTaskContext - Changed beaker.Priority to beaker.BeakerPriority

The experiment.create() method now returns a BeakerWorkload object, which has an experiment field containing the ID.

- Fixed exception names: ConfigurationError → BeakerConfigurationError, ExperimentNotFound → BeakerExperimentNotFound - Updated to use workload.get() and experiment.get_spec() instead of experiment.get() - Changed description update to use workload.update() instead of experiment.set_description() - Updated test mocks to match the new API structure

hamishivi

LGTM apart from the local debug issue! some other nits.

hamishivi · 2025-10-09T19:18:06Z

scripts/train/debug/finetune.sh

+
+echo "Using Beaker image: $BEAKER_IMAGE"
+
+uv run python mason.py \


Can we make using beaker optional for this script? I think for debug scripts its useful to run both in interactive and in beaker.

I'm down; how do I make it optional?

mason.py

open_instruct/utils.py

* Updates beaker-py version. * Fixed secrets ref. * Fixed whoami command * Fixed experimentspec error * Fixed constraints reference. * Update mason.py for beaker-py v2 API changes - Changed beaker_client.workspace.secrets() to beaker_client.secret.list() - Changed beaker_client.account.whoami() to beaker_client.user.get() - Changed beaker.ExperimentSpec to beaker.BeakerExperimentSpec - Changed beaker.Constraints to beaker.BeakerConstraints - Changed beaker.RetrySpec to beaker.BeakerRetrySpec * Fix beaker.TaskSpec to beaker.BeakerTaskSpec for v2 API * Fix remaining beaker v2 API changes - Changed all beaker.EnvVar to beaker.BeakerEnvVar - Changed beaker.DataMount to beaker.BeakerDataMount - Changed beaker.DataSource to beaker.BeakerDataSource - Changed beaker.TaskResources to beaker.BeakerTaskResources - Changed beaker.ImageSource to beaker.BeakerImageSource - Changed beaker.ResultSpec to beaker.BeakerResultSpec - Changed beaker.TaskContext to beaker.BeakerTaskContext - Changed beaker.Priority to beaker.BeakerPriority * Fix BeakerPriority to BeakerJobPriority for v2 API * Fix BeakerJobPriority enum access to use string key * Fix experiment ID access for beaker v2 API The experiment.create() method now returns a BeakerWorkload object, which has an experiment field containing the ID. * Fix beaker v2 API compatibility in utils.py and test_utils.py - Fixed exception names: ConfigurationError → BeakerConfigurationError, ExperimentNotFound → BeakerExperimentNotFound - Updated to use workload.get() and experiment.get_spec() instead of experiment.get() - Changed description update to use workload.update() instead of experiment.set_description() - Updated test mocks to match the new API structure * Updated finetune script * Now, finetune_8b.sh uses uv. * Updated finetune_8b.sh to work with build_image_and_launch.sh. * Updated code * added chat template to script. * changed priority * updated priority * Updated scripts/train/debug/finetune.sh. * added non-resumable flag. * Set description for debug finetune script. * Actually set chat template name. * Updated env var name * updated typo * Updated code * Updated error handling for interactive * Updated script * Refactored finetune.sh

finbarrtimbers marked this pull request as ready for review September 19, 2025 17:13

finbarrtimbers requested a review from hamishivi September 19, 2025 17:14

hamishivi reviewed Sep 19, 2025

View reviewed changes

finbarrtimbers added 16 commits October 1, 2025 11:47

Updates beaker-py version.

cb94af5

Fixed secrets ref.

5c8795d

Fixed whoami command

169fe8a

Fixed experimentspec error

662661b

Fixed constraints reference.

ea655b9

Fix beaker.TaskSpec to beaker.BeakerTaskSpec for v2 API

26c8c2a

Fix BeakerPriority to BeakerJobPriority for v2 API

7f8e4db

Fix BeakerJobPriority enum access to use string key

5aee8fc

Fix experiment ID access for beaker v2 API

1e46770

The experiment.create() method now returns a BeakerWorkload object, which has an experiment field containing the ID.

Updated finetune script

c1a6768

Now, finetune_8b.sh uses uv.

7e4cfb1

Updated finetune_8b.sh to work with build_image_and_launch.sh.

50423b2

Updated code

cdd2391

finbarrtimbers force-pushed the update-beaker-py branch from 0d9911f to cdd2391 Compare October 1, 2025 17:47

finbarrtimbers added 8 commits October 1, 2025 12:17

added chat template to script.

abf1b45

changed priority

64fa692

updated priority

f20e13e

Updated scripts/train/debug/finetune.sh.

5bcda40

added non-resumable flag.

8bdd718

Set description for debug finetune script.

892f5a9

Actually set chat template name.

92add3e

Merge branch 'main' into update-beaker-py

1277e4e

finbarrtimbers added 2 commits October 1, 2025 13:20

Updated env var name

8371060

updated typo

c291c04

finbarrtimbers requested a review from hamishivi October 1, 2025 19:38

finbarrtimbers added 3 commits October 7, 2025 06:55

Merge branch 'main' into update-beaker-py

3f15ee7

Updated code

252b7a5

Merge branch 'main' into update-beaker-py

2ea42f9

hamishivi reviewed Oct 10, 2025

View reviewed changes

Updated error handling for interactive

5bef970

This comment was marked as outdated.

Sign in to view

finbarrtimbers added 2 commits October 10, 2025 14:11

Updated script

fd565de

Refactored finetune.sh

faeacc3

hamishivi approved these changes Oct 10, 2025

View reviewed changes

Merge branch 'main' into update-beaker-py

f45806b

finbarrtimbers requested a review from hamishivi October 10, 2025 20:16

finbarrtimbers added this pull request to the merge queue Oct 10, 2025

Merged via the queue into main with commit 50cd847 Oct 10, 2025
4 checks passed

finbarrtimbers deleted the update-beaker-py branch October 10, 2025 20:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Updates beaker-py version to >2. #1021

Updates beaker-py version to >2. #1021

finbarrtimbers commented Sep 18, 2025 •

edited

Loading

Uh oh!

hamishivi left a comment

Uh oh!

finbarrtimbers commented Sep 20, 2025 via email

Uh oh!

hamishivi commented Sep 21, 2025

Uh oh!

hamishivi left a comment

Uh oh!

hamishivi Oct 9, 2025

Uh oh!

finbarrtimbers Oct 10, 2025

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		echo "Using Beaker image: $BEAKER_IMAGE"

		uv run python mason.py \

Updates beaker-py version to >2. #1021

Updates beaker-py version to >2. #1021

Conversation

finbarrtimbers commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hamishivi left a comment

Choose a reason for hiding this comment

Uh oh!

finbarrtimbers commented Sep 20, 2025 via email

Uh oh!

hamishivi commented Sep 21, 2025

Uh oh!

hamishivi left a comment

Choose a reason for hiding this comment

Uh oh!

hamishivi Oct 9, 2025

Choose a reason for hiding this comment

Uh oh!

finbarrtimbers Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

finbarrtimbers commented Sep 18, 2025 •

edited

Loading