Patient-level pathology prediction requires integrating sparse diagnostic evidence across multiple whole-slide images, but conventional MIL methods are designed around single-slide supervision. AGE-MIL constructs a patient-level anchor from slide representations to guide evidence retrieval from large patch pools, then models risk through evidence-aware representation learning and progressive accumulation. Across six clinical prediction tasks from two independent prostate cancer cohorts, it consistently outperforms eight state-of-the-art MIL methods.