Contrastive Learning for Unpaired Image-to-Image Translation
Ғылым және технология
Contrastive learning has provided a huge boost in self-supervised representation learning. This paper shows that this can even improve other self-supervised learning algorithms, like generative models in the GAN framework. I am really excited about how image to image translation networks can be used for domain similarity analysis. Thanks for watching! Please Subscribe!
Paper Links:
Contrastive Unpaired Translation (contains code, video, website, and paper link): github.com/taesungp/contrasti...
Contrastive Predictive Coding: arxiv.org/pdf/1905.09272.pdf
SinGAN: arxiv.org/pdf/1905.01164.pdf
EfficientDet: arxiv.org/pdf/1911.09070.pdf
Feature Pyramid Networks for Object Detection: arxiv.org/pdf/1612.03144.pdf
Don't Stop Pretraining: arxiv.org/pdf/2004.10964.pdf
CycleGAN: arxiv.org/pdf/1703.10593.pdf
SimCLR: arxiv.org/pdf/2002.05709.pdf
MoCo: arxiv.org/pdf/1911.05722.pdf
On the Measure of Intelligence: arxiv.org/pdf/1911.01547.pdf
Chapters
0:00 Beginning
1:37 Image-to-Image Translation
2:42 Example with Robots! (AVID)
3:26 High-level overview of algorithm
4:23 How Image Patches are Compared
6:53 PatchNCE Loss
7:52 MLP Projection Head
8:48 PatchNCE Loss (Equation)
10:52 External Negative test from Dataset rather than the Same Image
11:54 Final Objective
13:48 Ablation Takeaways
14:37 Results
15:38 Application to Domain Similarity
17:50 Interest in Domain Similarity Metrics
Пікірлер: 13
Great improvements over cyclegan! It's so much faster to train and occupies less gpu memory so I was able to train on larger images (than cyclegan). Thanks for thr detailed explanation, great work as always 👍
@connorshorten6311
3 жыл бұрын
Thank you so much! That's really awesome to hear, thanks for sharing your experience!
Sweet! Special props for explaining patches. In the PacthNCE formulation, the addition of the representations from the other spatial locations has a great impact on the contrastive learning task here I think.
@connorshorten6311
3 жыл бұрын
Thank you! Yeah, it reminds me of early style transfer algorithms with the gram matrix the way they compare features in intermediate layers.
@sayakpaul3152
3 жыл бұрын
Damn right!
Great video @Henry AI Labs Can you make a video on implementation of Contrastive Learning for Unpaired Image-to-Image Translation?
Let's consider an image of a synthetic hand, we want to translate it to a real hand, different patches in the source image are very similar, how do you eliminate the false positive patches in this case?
great video
@connorshorten6311
3 жыл бұрын
Thanks for watching!
Won't making the embeddings of patches at similar locations same make the feature extractor learn style invariant or content features. Will this not make the discriminator bad ?
@connorshorten6311
3 жыл бұрын
There are two separate losses here, the adversarial loss discriminator is separate from this. This loss is derived from a projection head after re-passing x and y' through the generator's encoder layers.
@NM-jq3sv
3 жыл бұрын
@@connorshorten6311 ah got it