You're not defining magic damage as a different form to affect reality, fam. The same as any other? See, that makes no sense when explaining why a weak flame harms a metal monster.
They are not arbitraty numbers, tho, It's a part of reality given name and measurement within a logical system...