Session 8 & 9: Interdisciplinary

Supercharging Computer Vision with Foundation Model Fine-Tuning

Shubing Wang and Andy Liaw

Shubing earned a Bachelor of Science in Mathematics from the University of Science and Technology of China and a Master of Arts in Computational Mathematics from the University of Texas at Austin. He completed his Ph.D. in Statistics at the University of Wisconsin–Madison, focusing on Weighted Fourier Representation and Multiple Testing in MR Brain Image Analysis. Shubing has worked at Merck since 2006, starting as a summer intern and later joining the Biometric Research department at BARDS. He supports preclinical studies and research, applying machine learning, high-dimensional, and longitudinal data analysis, including early-stage development of Keytruda. His recent research includes deep computer vision, self-supervised learning, foundation model fine-tuning, and Generative AI, developing Retrieval-Augmented Generation (RAG) applications for AI-driven projects at MRL.

Abstract: Hailed as the GPT-4 of computer vision, the Segment Anything Model (SAM) is a foundational model that exhibits robust zero-shot performance across a variety of computer vision tasks, including segmentation and object detection, applicable to diverse imaging platforms. Despite SAM's superior out-of-the-box adaptation compared to previous methods, there remains significant potential for improvement, particularly in challenging scenarios such as cryo-electron microscopy (cryoEM) image analysis. By leveraging smaller, noisier annotated datasets from earlier approaches, fine-tuning SAM in different contexts—such as region of interest (ROI) detection, particle segmentation, and classification in cryoEM image analysis—has significantly enhanced performance in these tasks. However, the large size of fine-tuned SAM models often results in slow training and inference, requiring substantial computational resources. To address these challenges, we demonstrate that fine-tuned SAM can significantly assist in fine-tuning smaller foundational models, such as YOLO, to improve efficiency through knowledge distillation. This presentation will demonstrate how fine-tuning foundation models can transform cryo-electron microscopy (cryoEM) from a specialized technique into a fundamental platform for the development of various vaccines and adjuvants.

Predicting the Unpredictable: How AI Predicts Therapy/Drug Responses and Survival Rates

Xueyan Mei

Dr. Xueyan Mei is an Instructor in the BioMedical Engineering and Imaging Institute (BMEII), the Department of Diagnostic, Molecular and Interventional Radiology, the Windreich Department of Artificial Intelligence and Human Health, and the Department of Emergency Medicine at the Icahn School of Medicine at Mount Sinai. Dr. Mei obtained her Ph.D. in Biological Science and completed her postdoctoral training at BMEII under the supervision of Dr. Zahi A. Fayad. Dr. Mei has numerous publications in leading international journals and conferences. Dr. Mei is the recipient of the prestigious Eric and Wendy Schmidt AI in Human Health Fellow award from the Eric Schmidt Foundation as of June 2024. Dr. Mei is actively working in designing innovative methods for medical image analysis, developing multi-modal AI and ML models for diagnosis, and creating vision-language models that integrate biomedical images with medical notes. Her work also extends to large language models adept at managing electronic health records and AI bots designed to streamline physician workflows.

Abstract: The integration of Artificial Intelligence (AI) in medical imaging and patient data analysis represents a transformative development in healthcare diagnostics and prognostics. This presentation explores the role of multi-modal AI in predicting clinical outcomes of interstitial lung disease and multiple myeloma. Additionally, I will discuss the creation of pseudo patients through AI, a novel approach that simulates patient responses to therapies for rare diseases. This session highlights AI's potential to further improve personalized medicine by integrating diverse data sources and sophisticated simulations.

Algorithmic Classification of Psychiatric Disorder Related Spontaneous Communication Using Large Language Model Embeddings

Ryan Shewcraft

Ryan is a Senior Manager on the Data Science team at Bristol Myers Squibb, where he supports the neuroscience therapeutic area. Prior to this role, he was a data scientist at Sema4 (now GeneDx) working  on building predictive models from real-world data. He earned a PhD in neuroscience and was a post-doctoral fellow at NYU. He completed a bachelor's degree in mathematics-physics and philosophy from Brown University.

Abstract: Traditional structured clinical interviews for assessing psychiatric disorders are often subjective, infrequent, and limited in capturing the variability and complexity of symptoms. We present a novel approach that leverages large language models to analyze features from patients' spontaneous communication. This method detects subtle language variations that may be indicative of psychiatric health changes, offering a low-burden, high-frequency, and patient-centric alternative. By requiring only natural conversation, it provides an objective and scalable solution for monitoring psychiatric disorders. This innovation has the potential to enhance drug efficacy insights and streamline psychiatric drug development in clinical trials.

AI in Drug Discovery: Ensuring Patent Protection Through Human Inventorship

Jameson Ma

Jameson Ma is a seasoned patent attorney with over 15 years of experience specializing in intellectual property strategy, patent prosecution, and patent counseling. He has worked extensively with pharmaceutical, medical device, and AI-driven technology companies, advising on complex patentability issues, particularly in AI-assisted drug discovery. His expertise spans industries such as consumer electronics, athletic apparel, automotive, and digital health, allowing him to provide strategic IP solutions tailored to innovation-driven companies.

Abstract: The integration of artificial intelligence (AI) into drug discovery is accelerating the identification of novel therapeutics, optimizing molecular design, and streamlining early-stage research. However, the increasing role of AI in these processes raises significant legal considerations regarding patent eligibility, particularly concerning the requirement that only natural persons can be recognized as inventors. Under current U.S. patent law, AI systems cannot be listed as inventors, and the extent of human contribution is a key factor in determining patentability. Failure to demonstrate sufficient human involvement in AI-assisted discoveries may compromise a company’s ability to secure and enforce valuable pharmaceutical patents. This session will provide an overview of the evolving intellectual property landscape for AI-driven drug discovery, with a focus on legal precedents and best practices for ensuring compliance. Attendees will gain practical insights into (1) defining human inventorship in AI-assisted research by understanding the U.S. Patent and Trademark Office’s criteria for determining whether a human’s contribution is legally sufficient, (2) documenting human contributions to AI-generated discoveries to strengthen patent eligibility, and (3) enhancing collaboration between scientific and legal teams to effectively structure disclosures and support robust patent applications.

Advancing Clinical Data Science: A Comprehensive Training Blueprint for Future Professionals

Richard Ittenbach

Richard Ittenbach, PhD, PSTAT is a Professor of Pediatrics, in the Division of Biostatistics and Epidemiology at Cincinnati Children’s Hospital and the University of Cincinnati College of Medicine. He is a professionally certified statistician through the American Statistical Association (PSTAT) and has been actively involved in biostatistics and clinical data management at the local and national levels for more than 20 years, with a particular interest clinical trials, data safety monitoring boards, and training the next generation of clinical data scientists. Dr. Ittenbach remains active in the American Statistical Association, Society for Clinical Data Management, and American Association for the Advancement of Science. Dr. Ittenbach has served as a consultant to academe and industry, and has been a data management content area expert for the U.S. Department of Labor. Areas of research interest include mixed methods study designs as well as the development and refinement of pediatric measures for children with chronic and life-threatening illnesses. His articles have appeared in Academic Medicine, American Journal of Bioethics, Clinical and Translational Science, Medical Decision Making, and Journal of the Society for Clinical Data Management: Informatics and Data Science in Clinical Research.

Abstract: The transition from clinical data management to clinical data science marks a pivotal shift in the biomedical research landscape. The purpose of this presentation is to highlight the need and provide a rationale for a new training model, one that builds upon the training programs currently used in industry today. This new graduate degree program in Clinical Data Science features a series of four core courses in clinical data science, five research courses representing the foundational areas of biostatistics, biomedical informatics, regulatory science, and clinical medicine and operations, and a full-time, field-based practicum experience. Embedded within the formal coursework is an implicit emphasis on pervasive (soft) skills needed for professional development and practice. The curriculum was built using key tenets of educational theory and practice and solidified using a two-tiered quality assurance process that maps instructional content to the core competencies of influential professional societies, and a review and feedback process that extends from local professionals to an international advisory panel. This presentation provides a comprehensive blueprint for the education and training of clinical data scientists, addressing the complexities of today’s research while preparing students for scientific practice in academic, industry, and government research settings.

Causal AI for business-critical decision making in Pharmaceutical Manufacturing

Juan Carlos

Dr. Juan Carlos Araque is a Principal Scientist in Process Science and Modeling (MSAT) at Johnson & Johnson Innovative Medicine. With extensive experience in mathematical process modelling, he previously worked at NASA, leading projects in computational materials discovery. He has held research and teaching positions at Cornell University, The University of Iowa, and Benedictine College, specializing in computational modelling and molecular simulations.

Abstract: Commercial manufacturers in the pharmaceutical industry face substantial challenges in ensuring scalability, maintaining product quality, complying with stringent regulatory requirements, and managing high production costs. Enhancing process understanding is critical to addressing these obstacles. Causal AI is emerging as a transformative approach compared to traditional machine learning methodologies, enabling the extraction of deeper insights from observational data by identifying the causal relationships underlying the data-generating process. This presentation will explore the application of Causal AI methodologies in pharmaceutical manufacturing, covering topics from causal discovery to guiding critical business decisions. We will demonstrate how Causal AI seamlessly integrates essential domain knowledge with advanced algorithms to provide a comprehensive understanding of key process drivers in manufacturing workflows. Additionally, we will discuss how the combination of manufacturing data with causal inference can substantially improve decision-making capabilities. Our aim is to highlight the effectiveness of deployed Causal AI solutions that are accessible to both technical experts and decision-makers. Finally, we will present a forward-looking perspective on the role of Causal AI in transforming decision-making processes, including improving resource allocation, streamlining operations, and optimizing scalability across manufacturing systems.