How Do Multiple Embedding Layers Work in a Popular Atom Encoder?
There Is This Popular Atomencoder Snippet That Is Suggested on Various Websites. I Used It Many Year's Ago with Success, but I Never Understood How It Works...
There is this popular AtomEncoder snippet that is suggested on various websites. I used it many years ago with success, but I never understood how it works.
class AtomEncoder(torch.nn.Module):
def __init__(self, hidden_channels):
super(AtomEncoder, self).__init__()
self.embeddings = torch.nn.ModuleList()
for i in range(9):
self.embeddings.append(Embedding(100, hidden_channels))
def reset_parameters(self):
for embedding in self.embeddings:
embedding.reset_parameters()
def forward(self, x):
if x.dim() == 1:
x = x.unsqueeze(1)
out = 0
for i in range(x.size(1)):
out += self.embeddings[i](x[:, i])
return out
With hidden_channels equal to, say, 12, this is what it creates:
(embeddings): AtomEncoder(
(embeddings): ModuleList(
(0-8): 9 x Embedding(100, 12)
)
)
Does that mean that I get 9 embedding layers?
Where do they go? All 9 between each linear layer, or one between 9 layers?
Given the lack of explanation, the number 9 itself sounds arbitrary to me. Why not just one layer?
I have searched the documentations and the internet thoroughly for answers, queried Stack Overflow about "AtomEncoder" embedding, as well as watched videos explaining embedding in details.
If I were to make a guess, it would probably be that all 9 layers come all together between every pair of linear layers, but I don't actually know it for sure. It's never explicitly stated anywhere. Could someone please help me understand this?
Sincerely thanks,