ML4H 2024 invites submissions describing innovative research that lies in the broad purview of Machine Learning for Health. Authors are invited to submit work on relevant problems in a variety of health-related disciplines including healthcare, biomedicine, and public health. This year, ML4H 2024 will accept submissions for two distinct tracks: the Proceedings track, for formal archival publications, and the non-archival Findings track.
In response to the growing ML4H community, ML4H has transitioned into a standalone symposium rather than a NeurIPS-affiliated workshop. This event represents a continuation of prior ML4H workshops/symposiums (2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023) and will continue to be co-located with NeurIPS in December. ML4H 2024 will feature:
Aug 1st: Submission site opensSep 6th AoE Sep 11th AoE: Paper submission deadline
Oct 11th : Reviews available; Author response period starts
Oct 18th : Author response period ends
Nov 1st: Decisions released
Nov 8th: Early Career Travel Grant Application
Nov 15th [tentative]: Camera-ready deadline
Dec 15-16th: In-person event
Submission Site: https://openreview.net/group?id=ML4H/2024/Symposium
ML4H 2024 LaTeX templates:
The paper submission deadline is September 6th September 11th 11:59 PM AoE. For this deadline, the title, author list, paper content, track, area, subject area, and data modality should be submitted. Edits to these metadata after the deadline will not be considered in the reviewing process. All submissions will be managed through the OpenReview system. Submissions must be formatted using the ML4H 2024 LaTeX templates with proper anonymization. Gross violations of formatting guidelines, malformed, non-blinded, non-health-related, or grossly insufficient works may be desk rejected by the organizing committee without undergoing additional review.
As part of the submission, authors will indicate whether they would like the submission to be in the Materials and Methods, Applications and Practice, or Impact and Society area. Authors are also required to fill out a submission form that will be visible to reviewers to help them assess the work.
Data and Code: We encourage anonymized code and data submissions (if it can be made available with appropriate approval and guidelines) as supplemental material during review. If you are not sharing code, you must explicitly state this in the submission checklist. If your paper is accepted, we encourage public sharing of your code and/or data for the camera-ready version of the paper.
Ethics Board Approval: If your research requires IRB (or equivalent) approval or has been evaluated by your IRB as Not Human Subject Research, then for the camera-ready version of the paper, you must provide relevant information. At the time of submission for review, to preserve anonymity, it suffices to include a statement that relevant ethics approval information will be provided if the paper is accepted. If your research does not require IRB approval, please explicitly state this to be the case and provide a justification in the submission checklist.
ML4H 2024 will feature two main symposium submission tracks.
Submissions to the main tracks will undergo double-blind peer review. Accepted submissions to all tracks will be featured at the event’s poster session. Accepted works for all tracks will be chosen based on their technical merit and contribution to the event. More details on how to write an excellent ML4H full paper or findings paper can be found here. The salient differences between these tracks are described below.
Excellent ML4H Proceedings papers should be compelling, cohesive works with a high degree of technical sophistication as well as clear and high-impact relevance to health. Accepted proceedings papers will be published in the Proceedings for Machine Learning Research (PMLR). Full proceedings papers can be up to 8 pages at submission (excluding references and appendices). If your submission is accepted, you will be allowed 1 additional content page for the camera-ready version.
Dual submission policy: Papers that are submitted to the ML4H proceedings track cannot be already published or under review in any other archival venue. Similarly, papers published to the ML4H proceedings may not be published again later at any other venue.
An excellent findings paper is one that leads to insights at the event through interaction with other attendees. This can be through presenting new ideas/ways of thinking, leading to insightful discussion and feedback, dissemination of new valuable resources, or enabling new opportunities for collaborations. We also especially solicit “non-traditional research artifacts” as submissions to the findings track, such as papers highlighting novel datasets, insightful negative results, exciting preliminary results that warrant rapid dissemination, reproducibility studies, and opinion pieces or critiques.
Findings papers can be up to 4 pages at submission (excluding references and appendices), though additional information not critical for understanding the work can be included in an appendix without penalty (reviewers will review the work based predominantly on the main text). Findings papers will not appear in the ML4H proceedings, but upon acceptance, we invite (but do not require) authors to submit their findings (no page limit) to the ML4H arxiv.org index.
Authors of accepted findings papers (non-archival submissions) retain full copyright of their work, and acceptance of such a submission to ML4H 2024 does not preclude publication of the same material in another archival venue (e.g. journal or conference). Furthermore, findings submissions that are under review or have been recently published in a conference or a journal are allowed; if this is the case, authors should clearly state any overlapping published or submitted work at the time of submission (in the confidential comments), and must ensure that they are not violating any other venue’s dual submission policies.
Proceedings submissions that are not accepted will automatically be considered for Findings. Findings track submissions cannot be considered for the Proceedings track. Decisions for both tracks will be released simultaneously.
Submitted papers should describe innovative machine learning research focused on relevant problems in health-related disciplines. Topics of interest include but are not limited to the following.
Note. these areas are non-exhaustive. Past works have spanned data integration, temporal models, deep learning, semi-supervised learning, reinforcement learning, transfer learning, few/zero shot learning, learning from missing or biased data, learning from non-stationary data, causality, model biases, model evaluation, model criticism, model interpretability, model deployment, human-computer interaction, privacy/security, and many more topics.
Advances in machine learning are critical for a better understanding of health. This area seeks technical contributions in modeling, inference, and estimation in health-focused or health-inspired settings. We welcome submissions that develop novel methods and algorithms, introduce relevant machine learning tasks, identify challenges with prevalent approaches, or learn from multiple sources of data (e.g. non-clinical and clinical data).
Our focus on health is broadly construed, including clinical healthcare, public health, and population health. While submissions should be primarily motivated by problems relevant to health, the contributions themselves are not required to be directly applied to real health data. For example, authors may use synthetic datasets to demonstrate properties of their proposed algorithms.
We welcome submissions from many perspectives, including but not limited to supervised learning, unsupervised learning, reinforcement learning, causal inference, representation learning, survival analysis, domain adaptation or generalization, interpretability, robustness, and algorithmic fairness. All kinds of health-relevant data types are in scope, including tabular health records, time series, text, images, videos, knowledge graphs, and more. We welcome all kinds of methodologies, from deep learning to probabilistic modeling to rigorous theory and beyond.
Khanna, S., Michael, D., Zitnik, M., Rajpurkar, P. “Learning Generalized Medical Image Representations Through Image-Graph Contrastive Pretraining.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Mao, H., Liu, H., Dou, J. X., Benos, P. V. “Towards Cross-Modal Causal Structure and Representation Learning.” Proceedings of the 2nd Machine Learning for Health Symposium (ML4H), 2022.
Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C., Reis, E. P., Rajpurkar, P. “Med-Flamingo: A Multimodal Medical Few-Shot Learner.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Wang, K. A., Fox, E. B. “Interpretable Mechanistic Representations for Meal-Level Glycemic Control in the Wild.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Tang, S., Dunnmon, J. A., Qu, L., Saab, K. K., Baykaner, T., Lee-Messer, C., Rubin, D. L. “Modeling Multivariate Biosignals With Graph Neural Networks and Structured State Space Models.” Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2023.
Shalit, U., Johansson, F.D., Sontag, D. “Estimating individual treatment effect: generalization bounds and algorithms.” Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
The goal of this area is to highlight works applying robust methods, models, or practices to identify, characterize, audit, evaluate, or benchmark ML approaches to healthcare problems. Additionally, we welcome unique deployments and datasets used to empirically evaluate these systems are necessary and important to advancing practice. Whereas the goal of Area 1 is to select papers that show significant algorithmic novelty, submit your work here if the contribution is describing an emerging or established innovative application of ML in healthcare. Areas of interest include but are not limited to:
Introducing a new method is not prohibited by any means for this area, but the focus should be on the extent of how the proposed ideas contribute to addressing a practical limitation (e.g., robustness, computational scalability, improved performance). We encourage submissions in both more traditional clinical areas (e.g., electronic health records (EHR), medical image analysis), as well as in emerging fields (e.g., remote and telehealth medicine, integration of omics).
Goel, A., Gueta, A., Gilon, O., Liu, C., Erell, S., Nguyen, L. H., Hao, X., et al. “LLMs Accelerate Annotation for Medical Information Extraction.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Gröger, F., Lionetti, S., Gottfrois, P., Gonzalez-Jimenez, A., Groh, M., Daneshjou, R., Labelling Consortium, Navarini, A. A., Pouly, M. “Towards Reliable Dermatology Evaluation Benchmarks.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Kocielnik, R., Wong, E. Y., Chu, T. N., Lin, L., Huang, D.-A., Wang, J., Anandkumar, A., Hung, A. J. “Deep Multimodal Fusion for Surgical Feedback Classification.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Nguyen, D., Chen, C., He, H., Tan, C. “Pragmatic Radiology Report Generation.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Kinyanjui, N.M., Johansson, F.D. ADCB: “An Alzheimer’s disease simulator for benchmarking observational estimators of causal effects”. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2022.
Zhou, H., Chen, Y., Lipton, Z. “Evaluating Model Performance in Medical Datasets Over Time”. Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2023.
Algorithms do not exist in a vacuum: instead, they often explicitly aim for important social outcomes. This area considers issues at the intersection of algorithms and the societies they seek to impact, specifically for health. Submissions could include methodological contributions such as algorithmic development and performance evaluation for policy and public health applications, large-scale or challenging data collection, combining clinical and non-clinical data, as well as detecting and measuring bias. Submissions could also include impact-oriented research such as determining how algorithmic systems for health may introduce, exacerbate, or reduce inequities and inequalities, discrimination, and unjust outcomes, as well as evaluating the economic implications of these systems. We invite submissions tackling the responsible design of AI applications for healthcare and public health. System design for the implementation of such applications at scale is also welcome, which often requires balancing various tradeoffs in decision-making. Submissions related to understanding barriers to the deployment and adoption of algorithmic systems for societal-level health applications are also of interest. In addressing these problems, insights from social sciences, law, clinical medicine, and the humanities can be crucial.
Afzal, M. M., Khan, M. O., Mirza, S. “Towards Equitable Kidney Tumor Segmentation: Bias Evaluation and Mitigation.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Cheng, J. J., Huling, J. D., Chen, G. “Meta-Analysis of Individualized Treatment Rules via Sign-Coherency.” Proceedings of the 2nd Machine Learning for Health Symposium (ML4H), 2022.
Gupta, M., Gallamoza, B., Cutrona, N., Dhakal, P., Poulain, R., Beheshti, R. “An Extensive Data Processing Pipeline for MIMIC-IV.” Proceedings of the 2nd Machine Learning for Health Symposium (ML4H), 2022.
Khan, M. O., Afzal, M. M., Mirza, S., Fang, Y. “How Fair Are Medical Imaging Foundation Models?.” Proceedings of the 3rd Machine Learning for Health Symposium (ML4H), 2023.
Lopez-Martinez, D., Yakubovich, A., Seneviratne, M., Lelkes, A. D., Tyagi, A., Kemp, J., Steinberg, E., et al. “Instability in Clinical Risk Stratification Models Using Deep Learning.” Proceedings of the 2nd Machine Learning for Health Symposium (ML4H), 2022.
Merrill, M., Safranchik, E., Kolbeinsson, A., Gade, P., Ramirez, E., Schmidt, L., Foschini, L., Althoff, T. “Homekit2020: A Benchmark for Time Series Classification on a Large Mobile Sensing Datset with Laboratory Tested Ground Truth of Influenza Infections.” Proceedings of the Conference on Health, Inference, and Learning (CHIL), 2023.
Initial reviews will be released on October 11th. From October 11th to October 18th, 11:59 PM AoE, authors can submit responses to the reviews. Author responses may address any aspect of the reviews, including by adding specific types of new experimental results as requested by the reviewers, e.g. missing baselines. No conceptual changes to the original formulation are allowed beyond clarifications. After the author response period, the reviewers and meta-reviewer will discuss and reach a final decision for the papers. We reserve the right to solicit additional reviews after the author response period in the rare case that there are not sufficient high quality reviews to make a final decision.
During the reviewer discussion period, reviewers and meta-reviewers will discuss the paper, their reviews, and the author response. This process aims at seeking a consensus between reviewers and meta-reviewers. We ask reviewers to change their initially submitted review scores and recommendations during the discussion period, if applicable, and state this in the discussion along with justification. Discussions will take place within OpenReview by using the comment function in each respective submission and should remain double-blind, i.e. comments may not de-anonymize the authors or reviewers.
In general, these discussions will be between reviewers and meta-reviewers only. However, when further clarifications from the authors are necessary, reviewers may reach out to authors through OpenReview comments. It is only in response to such direct questions that authors should add comments beyond their author response, and said comments should be limited to directly answering the asked question. The reviewer discussion period formally ends on October 25 11:59 PM AoE , but discussions may be finalized earlier.
To promote community interaction, at least one presenting author of accepted works must register for the event. Registration details are forthcoming.
Please direct questions to: info@ml4h.cc and follow us on Twitter at @symposiumml4h.
