On May 27th and 28th, the PATH consortium convened in Utrecht for an intensive two-day Screenathon, organized and hosted by FwdFaster AI, which is an industry-partner of PATH-IHI Project.
This large-scale collaborative crowdsourcing event’s main objective was to screen an immense volume of academic and policy documents (69,073 records) in order to build the PATH Knowledge Warehouse and to contribute to a comprehensive Landscape Analysis.
To efficiently handle the massive volume of literature, the event utilized ASReview, an open-source, AI-aided machine learning framework that uses active learning to prioritize and rank documents for screening.
Reviewers were trained using a specific decision tree based on three main inclusion criteria:
- Topic: The document must address the integration, deployment, or governance of AI tools in hospital clinical workflows.
- Analytical Focus: The text must assess barriers/challenges, gaps/shortcomings, or enablers/opportunities for AI deployment.
- Dimension: The content must cover at least one of six key dimensions: Legal, Ethical, Technical, Social, Economic, or Care Delivery Impact
Confronted with the massive database of around 69,000 records, the team kicked off Day 1 at UtrechtInc with a clear objective: systematically evaluate the literature to extract the vital evidence needed to propel the initiative forward. A Screenathon is a synchronized effort to align human judgment at scale. Through intensive calibration and real-time consensus-building, the domain experts aligned on complex definitions, capping off a productive day with a refreshing afternoon tour of the Botanical Gardens Utrecht.
On Day 2, the momentum continued with the team having gathered at the Social Impact Factory to deep-dive into the remaining literature.
The crowdsourcing effort turned out to be a major success, with participants following the live leaderboards as they screened a total of 4,042 records. Thanks to their hard work, 1,822 relevant papers were successfully identified to form the basis of the PATH Knowledge Warehouse. This impressive achievement was wrapped up with a closing ceremony to award well-deserved prizes to our top contributors.
Next Steps
- Further Screening to identify all relevant records for the Knowledge Warehouse
- NLF (Noisy Label Filter) Quality Check: Scheduled for the following week to review and catch human screening errors (false inclusions/exclusions).
- Data Extraction: Detailed breakdown of the selected papers (June).
- Statistical Synthesis: Analyzing data trends (July).
- Reporting: Writing the final Landscape Analysis Report (July).
