llama : deci : support ffn-free with attention

[CISC]

Fix partially overlooked change request, sorry for premature merge.

cc/ @ymcki

[ymcki]

I have yet seen any deci layer that is ffn free yet have attention. Whether that will make a viable model is also unknown. I think it is better to make this change when there is actually a real model with such a layer.

[CISC]

I have yet seen any deci layer that is ffn free yet have attention. Whether that will make a viable model is also unknown. I think it is better to make this change when there is actually a real model with such a layer.

While probably true I think the issue is that if this model were to exist it would cause the code to proceed to build_ffn for a perfectly preventable reason. :)

[ymcki]

While studying the architectures of the major models, I have yet to see a layer that has attention but no feed forward, I think it shouldn't hurt to make this change. However, someone needs to do the testing to make sure it doesn't break things. As I don't have the resource to run the 253B model, let's see if someone can help you on this.

[CISC]

Should be less likely to break anything than before, and certainly not 253B, we'll deal with any issues that may crop up as/if they do.