A scheme is proposed to deterministically generate four-dimensional entangled states for two distant atoms trapped in two separate cavities via adiabatic passage by one step. An optimal expression of time-dependent Rabi frequencies is selected through numerical simulation. Numerical simulation also shows that the excited probabilities of the atoms, the cavity modes and the fiber modes are very low during adiabatic passage, and the scheme is insensitive to the fluctuation of experimental parameters. The four-dimensional entanglement may be generated with high fidelity with current experimental technology.