Few-shot pathology anomaly detection depends on subtle region-level abnormalities, yet generic vision-language adaptation often fails because semantic prompts are not grounded in fine-grained visual evidence. HAAF tackles this granularity mismatch with a hierarchical adaptation and alignment strategy centered on cross-level scaled alignment, where visual context first refines text prompts and the adapted prompts then guide anomaly-focused visual encoding. A dual-branch inference design further improves stability, and experiments on four benchmarks show strong gains over existing few-shot baselines.