It specifically says "duel" - one on one or any formal combat he automatically loses, either due to quirks of fate, or better skill from the opponent or whatnot. It sounds like informal fights he can win, as he can win fights against unintelligent opponents but as soon as the foe is able to...