Over the past decade, deep learning (DL) has achieved a big performance leap in company with the booming of big data. While it has been successfully applied to some industrial domain-specific tasks (e.g., face recognition, speech recognition), its quality and reliability raise great concern especially in the context of safety- and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently resolved, on which a DL software makes incorrect decisions. Such defects can occur through either intentional manipulation of adversarial attack or physical-world noise perceived by input sensors, potentially hindering the industry deployment. The intrinsic uncertainty nature of deep learning decision could be a fundamental reason for its incorrect behavior. Although many testing, adversarial attack and defense techniques have been proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty. In this paper, we conduct a large-scale study towards bridging this gap. We first investigate the capability of multiple uncertainty metrics on differentiating natural benign examples (BEs) and AEs. Then, we identify and categorize the uncertainty patterns of AEs and BEs, and find that while natural BEs and AEs generated by existing methods do follow common uncertainty patterns, some other uncertainty patterns are largely missed. Based on this, we propose an automated testing technique to generate multiple types of uncommon AEs and BEs. Our further evaluation reveals that the uncommon data generated by our methods is hard to be defensed by the state-of-the-art defense techniques with the average defense success rate reduced by 35%. Our results call for attention to generate more diverse data for evaluating and designing quality and reliable assurance solutions for DL software.