Abstract: This paper presents a systematic literature review on the disruptive health and biomedical informatics technologies such as artificial intelligence (AI), that have accelerated medical operations from patient-centered medical experience data management to simplified medical procedures in this information era. As these technologies get integrated into traditional approaches, they raise critical technological and medical concerns, entailing transparency and interpretability of these AI models. Using Preferred Reporting Items for Systematic Reviews and Meta-Analyse (PRISMA) and 1,837 articles published between 2014 and 2024 from eight popular academic databases: PubMed, ACM Library, Springer, Scopus, IEEE Xplore, ScienceDirect, Google Scholar, and Web of Science based on the relevance of the AI method to healthcare and biomedicine. We included 148 articles in our review. The studied studies demonstrated that most medical people still find it complex to effectively explain the reasoning behind the decisions AI models make during biomedical experiments, leading to limited trust, biased decision-making, and unknown patient data safety. The main challenges are AI model intricacy, initial regulatory acceptance, usability, fairness, and predisposition. Achieving a balance between AI explainability, fairness, and performance is fundamental to fostering ethical and responsible AI deployment in healthcare, ensuring improved patient outcomes and regulatory compliance. Limited studies are focusing on improving AI openness, trust, and interpretability. Model-agnostic strategies and explainable AI frameworks’ full implementation are inspected and ar. The availability of diverse and high-quality open datasets is indispensable for improving AI model interpretability; however, issues related to data privacy, standardization, and accessibility remain key concerns.