We study the relaxation of the center of mass or dipole oscillations in the system of interacting fermions confined spatially. With the confinement frequency ${\ensuremath{\omega}}_{\ensuremath{\perp}}$ fixed, the particles were considered to freely move along one (quasi-1D) or two (quasi-2D) spatial dimensions. We have focused on the regime of rare collisions, such that the inelastic collision rate $1/{\ensuremath{\tau}}_{\mathrm{in}}\ensuremath{\ll}{\ensuremath{\omega}}_{\ensuremath{\perp}}$. The dipole oscillation relaxation rate $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}$ is obtained at three different levels: by direct perturbation theory, solving the integral Bethe-Salpeter equation, and applying the memory function formalism. As long as anharmonicity is weak, $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}\ensuremath{\ll}1/{\ensuremath{\tau}}_{\mathrm{in}}$, the three methods are shown to give identical results. In the quasi-2D case, $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}\ensuremath{\ne}0$ at zero temperature. In quasi-1D system, $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}\ensuremath{\propto}{T}^{3}$ if the Fermi energy
EF lies below the critical value ${E}_{F}<3{\ensuremath{\omega}}_{\ensuremath{\perp}}/4$. Otherwise, unless the system is close to integrability, the rate $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}$ has a temperature dependence similar to that in quasi-2D. In all cases, the relaxation results from the excitation of particle-hole pairs propagating along unconfined directions resulting in the relationship $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}\ensuremath{\propto}1/{\ensuremath{\tau}}_{\mathrm{in}}$, with the inelastic rate $1/{\ensuremath{\tau}}_{\mathrm{in}}\ensuremath{\ne}0$ as the phase-space opens up at finite energy of excitation ${\ensuremath{\omega}}_{\ensuremath{\perp}}$. While $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}\ensuremath{\propto}{\ensuremath{\tau}}_{\mathrm{in}}$ in the hydrodynamic regime, ${\ensuremath{\omega}}_{\ensuremath{\perp}}\ensuremath{\ll}1/{\ensuremath{\tau}}_{\mathrm{in}}$, in the regime of rare collisions, ${\ensuremath{\omega}}_{\ensuremath{\perp}}\ensuremath{\gg}1/{\ensuremath{\tau}}_{\mathrm{in}}$, we obtain the opposite trend $1/{\ensuremath{\tau}}_{\ensuremath{\perp}}\ensuremath{\propto}1/{\ensuremath{\tau}}_{\mathrm{in}}$.