ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation