Skip to content

Commit 8097631

Browse files
Update byte_pair_tokenizer.py to close merges file properly (keras-team#1440)
1 parent 70c57cc commit 8097631

File tree

1 file changed

+2
-1
lines changed

1 file changed

+2
-1
lines changed

keras_nlp/tokenizers/byte_pair_tokenizer.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,8 @@ def set_vocabulary_and_merges(self, vocabulary, merges):
350350
f"`type(vocabulary)={type(vocabulary)}`."
351351
)
352352
if isinstance(merges, str):
353-
self.merges = [bp.rstrip() for bp in open(merges, encoding="utf-8")]
353+
with open(merges, encoding="utf-8") as f:
354+
self.merges = [bp.rstrip() for bp in f]
354355
elif isinstance(merges, Iterable):
355356
self.merges = list(merges)
356357
else:

0 commit comments

Comments
 (0)