Added DLRM notebook and generated python script #2131
Conversation
There was a problem hiding this comment.
Thanks for the PR! Left some comments, let me know what you think :).
Here are some notes for posterity (we don't need to implement these now in this PR):
- We should, in the future, also think about the specific case of how how DLRM implements distributed training. The embeddings are sharded across devices, whereas the MLP layers are DDP'd.
- The paper mentions that they use
torch.nn.EmbeddingBaginstead oftorch.nn.Embedding. Apparently,torch.nn.EmbeddingBagis more efficient thantorch.nn.Embeddingfollowed by some aggregation op. We should think about this at some later point (maybe,keras_rs.embeddings.DistributedEmbeddingsolves this). Anyway, we can worry about this later!
| self.embedding_layers = [] | ||
| for feature_name, vocabulary in vocabularies.items(): | ||
| self.embedding_layers.append( | ||
| keras.layers.Embedding( | ||
| input_dim=len(vocabulary) + 1, | ||
| output_dim=embedding_dim, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
One of the highlights of DLRM is that it can process both categorical and dense features. We should use some dense features present in the MovieLens dataset. Do we have any such features?
There was a problem hiding this comment.
The movielens 100k-ratings dataset mostly contains categorical features, However it also has user age feature but its bucketized, which transforms it to be a categorical feature.
There was a problem hiding this comment.
Hmmm, doesn't it have raw_user_age as a feature? The reason I'm insisting on this is because the two towers for dense and categorical features is a salient part of DLRM. And do you think we can use normalised timestamp as a feature? Will that help/does that make sense?
There was a problem hiding this comment.
Sure! like you mentioned, two towers for dense and categorical features is a salient part of DLRM. I'll have another look at the code and see if any dense features can be used. Timestamp looks like a suitable dense feature. I'll modify the code and try using that
There was a problem hiding this comment.
And we can remove user_bucketised_age and use raw_user_age, maybe
Updated docstrings to match the context according to code
|
@kharshith-k - let me know when this is ready for another round of review. Thanks! |
|
.DS_Store
Outdated
examples/.DS_Store
Outdated
There was a problem hiding this comment.
Please remove this file. You can also add these files to .gitignore, BTW, to avoid these getting added by accident.
examples/keras_rs/.DS_Store
Outdated
There was a problem hiding this comment.
Please remove this file. You can also add these files to .gitignore, BTW, to avoid these getting added by accident.
examples/keras_rs/img/.DS_Store
Outdated
examples/keras_rs/ipynb/dlrm.ipynb
Outdated
There was a problem hiding this comment.
There are two copies of this notebook. Let's delete one of them
examples/keras_rs/ipynb/DLRM.ipynb
examples/keras_rs/ipynb/dlrm.ipynb
| self.embedding_layers = [] | ||
| for feature_name, vocabulary in vocabularies.items(): | ||
| self.embedding_layers.append( | ||
| keras.layers.Embedding( | ||
| input_dim=len(vocabulary) + 1, | ||
| output_dim=embedding_dim, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Hmmm, doesn't it have raw_user_age as a feature? The reason I'm insisting on this is because the two towers for dense and categorical features is a salient part of DLRM. And do you think we can use normalised timestamp as a feature? Will that help/does that make sense?
Added DLRM.ipynb and generated dlrm.py