thank U

Master piece. Thank you so much 💗

very cool stuff. Any idea how this compares to Feature Pyramid Networks, which are typically used to enrich the high-res early convolutional layers? I would imagine that the FPN works well if the thing of interest is "compact". I.e. can be captured well by a quadratic crop, whereas the attention would even work for non-compact things. Examples would be donuts with large holes and little dough, or long sticks, etc.

@LukeDitria

25 күн бұрын

I believe Feature Pyramid Networks were primarily for object detection, and are a way of bringing fine grain information from earlier layers deeper into the network with big residual connections, they sill rely on multiple conv layers to combine spatial information. What we're trying to do here is mix spatial information early in the network. With attention the model can also choose how exactly to do that.

Good video! Do you think this experiment of adding the attention head so early on can extrapolate well to graph neural networks?

@LukeDitria

24 күн бұрын

Hi thanks for your comment! Yes, Graph Attention Networks do what you are describing!

I'm guessing that adding self-attention in deeper layers would have lesser of an impact due to each value having greater receprive field? If not, then why not to add at the end, where it would be less expensive? Without the fact that we could incorporate it in every conv block if we had infinite compute

@LukeDitria

26 күн бұрын

Thanks for your comment! Yes you are correct, in terms of combining features spatially it won't have as much of an impact if the features already have a large receptive field. The idea is to try to add it as early as possible, and yes you could add it multiple times throughout your network, though you would probably stop once your feature map is around 4x4 etc...

@unknown-otter

26 күн бұрын

Thanks for the clarification! Great video