Evaluating the Adversarial Robustness of Adaptive Test-time Defenses

Adaptive defenses that use test-time optimization promise to improve robustness to adversarial examples. We categorize such adaptive test-time defenses and explain their potential benefits and drawbacks. In the process, we evaluate some of the latest proposed adaptive defenses in a thorough case study. Unfortunately, none significantly improve upon static defenses when evaluated appropriately. Some even weaken the underlying static defense while simultaneously increasing inference cost. While these results are disappointing, we still believe that adaptive test-time defenses are a promising avenue of research and, as such, we provide recommendations on evaluating such defenses. We go beyond existing evaluation checklists by providing concrete steps that are specific to adaptive defenses.

Authors' notes