Repository navigation
PassTrough Embeddings #132
Description
in some tasks we need deep contextual embedding from LLM
Ways to include:
- PCA over embeddings on preprocessing stage (unsupervised, linear)
- LDA over embeddings on preprocessing stage (supervised, linear)
- UMAP over embeddings on preprocessing stage (unsupervised, non-linear)
- VectorNormalization over embeddings on preprocessing stage (unsupervised, linear)
-> np.linalg.norm(embd) and sum over it (in practice we can generate two features out of it, unit vector and magnitude) - Raw include
-> x = concat([self.embeddings(num_features, cat_features), emb_features])
-> x = self.embeddings(num_features, cat_features) + emb_features
Activity
AnFreTh commentedon Sep 28, 2024
Are you looking to extract the embeddings after a model is trained?
Or are you suggesting to include extracted embeddings from a LLM and using these as features? If the later, it is more a feature engineering step and can easily be done by adding the embeddings to the data.
EpicUsaMan commentedon Sep 28, 2024
Yes, adding the embeddings
The problem is that it's computation intensive to have embeddings over embeddings
Raw include it's what I'm looking for, but other methods can be used as well to make it less computation intensive
AnFreTh commentedon Sep 28, 2024
To clarify, you want to do something like this:
Where the embeddings_train are pre-computed embeddings of shape N x T x d, with N=length of X_train, T= sequence length of the pre-computed embeddings (could also be one if already pooled) and d = embedding dimension.
Then you want to simply concatenate these pre-trained embeddings onto the embeddings that are generated for the tabular data such that the overall embedding shape is
N x T+J x d, where J=number of features of X_train
EpicUsaMan commentedon Sep 28, 2024
Yes, exactly
Also, probably I missed something in documentation, but there is any way to make some specific feature categorical?
I've few numeric columns, which are better to be trained as numbers (not finite number of elements) and some are better to be trained as cat_features, but my finite-number features has less unique entries than my categorical :D (very task specific case)
AnFreTh commentedon Sep 29, 2024
I am not quite sure whether this would a useful addition. It's a very specific usecase where other model structures - a task specific head on the embeddings and additional features, without subsequent Mamba/Attention - seem more sensible.
I will turn this into a discussion. If more people are interested in this we might include it.