Bias Variance

در مقالات اخیر بحثی مطرح شده است که زمینه مناسبی برای فعالیت در حوزه یادگیری عمیق برای نوشتن پایان نامه و مقاله است. Vision Transformer بیان می کند که می توانیم از transformer ها که در NLP استفاده می شوند در حوزه computer vision استفاده کنیم. مقاله Searching for Efficient Multi-Stage Vision Transformers که اخیرا نگاشته شده است مربوط به این حوزه است. کد این مقاله متن باز است که می توانید در اینجا مشاهده کنید.

Vision Transformer (ViT) demonstrates that Transformer for natural language processing can be applied to computer vision tasks and result in comparable performance to convolutional neural networks (CNN), which have been studied and adopted in computer vision for years. This naturally raises the question of how the performance of ViT can be advanced with design techniques of CNN. To this end, we propose to incorporate two techniques and present ViT-ResNAS, an efficient multi-stage ViT architecture designed with neural architecture search (NAS). First, we propose residual spatial reduction to decrease sequence lengths for deeper layers and utilize a multi-stage architecture. When reducing lengths, we add skip connections to improve performance and stabilize training deeper networks. Second, we propose weight-sharing NAS with multi architectural sampling. We enlarge a network and utilize its sub-networks to define a search space. A super-network covering all sub-networks is then trained for fast evaluation of their performance. To efficiently train the super-network, we propose to sample and train multiple sub-networks with one forward-backward pass. After that, evolutionary search is performed to discover high-performance network architectures. Experiments on ImageNet demonstrate that ViT-ResNAS achieves better accuracy-MACs and accuracy-throughput trade-offs than the original DeiT and other strong baselines of ViT.

لینک مقاله


#معرفی_مقاله #یادگیری_عمیق #transformer #پیشنهاد_موضوع

🌴 سایت | 🌺 کانال | 🌳 پشتیبانی

GitHub

GitHub - yilunliao/vit-search: Code for "Searching for Efficient Multi-Stage Vision Transformers"

Code for "Searching for Efficient Multi-Stage Vision Transformers" - GitHub - yilunliao/vit-search: Code for "Searching for Efficient Multi-Stage Vision Transformers"

www.tgoop.com/biasvariance_ir/37

107 viewsBias Variance, edited  Sep 8, 2021 at 22:41

در مقالات اخیر بحثی مطرح شده است که زمینه مناسبی برای فعالیت در حوزه یادگیری عمیق برای نوشتن پایان نامه و مقاله است. Vision Transformer بیان می کند که می توانیم از transformer ها که در NLP استفاده می شوند در حوزه computer vision استفاده کنیم. مقاله Searching for Efficient Multi-Stage Vision Transformers که اخیرا نگاشته شده است مربوط به این حوزه است. کد این مقاله متن باز است که می توانید در اینجا مشاهده کنید.

Vision Transformer (ViT) demonstrates that Transformer for natural language processing can be applied to computer vision tasks and result in comparable performance to convolutional neural networks (CNN), which have been studied and adopted in computer vision for years. This naturally raises the question of how the performance of ViT can be advanced with design techniques of CNN. To this end, we propose to incorporate two techniques and present ViT-ResNAS, an efficient multi-stage ViT architecture designed with neural architecture search (NAS). First, we propose residual spatial reduction to decrease sequence lengths for deeper layers and utilize a multi-stage architecture. When reducing lengths, we add skip connections to improve performance and stabilize training deeper networks. Second, we propose weight-sharing NAS with multi architectural sampling. We enlarge a network and utilize its sub-networks to define a search space. A super-network covering all sub-networks is then trained for fast evaluation of their performance. To efficiently train the super-network, we propose to sample and train multiple sub-networks with one forward-backward pass. After that, evolutionary search is performed to discover high-performance network architectures. Experiments on ImageNet demonstrate that ViT-ResNAS achieves better accuracy-MACs and accuracy-throughput trade-offs than the original DeiT and other strong baselines of ViT.

لینک مقاله


#معرفی_مقاله #یادگیری_عمیق #transformer #پیشنهاد_موضوع

🌴 سایت | 🌺 کانال | 🌳 پشتیبانی

BY Bias Variance