Fungi of the genus Alternaria are ubiquitous plant pathogens and saprophytes which are able to grow under varying temperature and moisture conditions as well as on a large range of substrates. A spectrum of structurally diverse secondary metabolites with toxic potential has been identified, but occurrence and relative proportion of the different metabolites in complex mixtures depend on strain, substrate, and growth conditions. This review compiles the available knowledge on hazard identification and characterization of Alternaria toxins. Alternariol (AOH), its monomethylether AME and the perylene quinones altertoxin I (ATX-I), ATX-II, ATX-III, alterperylenol (ALP), and stemphyltoxin III (STTX-III) showed in vitro genotoxic and mutagenic properties. Of all identified Alternaria toxins, the epoxide-bearing analogs ATX-II, ATX-III, and STTX-III show the highest cytotoxic, genotoxic, and mutagenic potential in vitro. Under hormone-sensitive conditions, AOH and AME act as moderate xenoestrogens, but in silico modeling predicts further Alternaria toxins as potential estrogenic factors. Recent studies indicate also an immunosuppressive role of AOH and ATX-II; however, no data are available for the majority of Alternaria toxins. Overall, hazard characterization of Alternaria toxins focused, so far, primarily on the commercially available dibenzo-α-pyrones AOH and AME and tenuazonic acid (TeA). Limited data sets are available for altersetin (ALS), altenuene (ALT), and tentoxin (TEN). The occurrence and toxicological relevance of perylene quinone-based Alternaria toxins still remain to be fully elucidated. We identified data gaps on hazard identification and characterization crucial to improve risk assessment of Alternaria mycotoxins for consumers and occupationally exposed workers.