We analyze optical conductivity of a clean two-dimensional electron system in a Fermi liquid regime near a
T=0 Ising-nematic quantum critical point (QCP) and extrapolate the results to a QCP. We employ direct perturbation theory up to the two-loop order to elucidate how the Fermi surface's geometry (convex vs concave) and fermionic dispersion (parabolic vs nonparabolic) affect the scaling of the optical conductivity $\ensuremath{\sigma}(\ensuremath{\omega})$ with frequency $\ensuremath{\omega}$ and correlation length $\ensuremath{\xi}$. We find that for a convex Fermi surface the leading terms in the optical conductivity cancel out, leaving a subleading contribution $\ensuremath{\sigma}(\ensuremath{\omega})\ensuremath{\propto}{\ensuremath{\omega}}^{2}{\ensuremath{\xi}}^{4}\mathcal{L}$, where
L=const for a parabolic dispersion and $\mathcal{L}\ensuremath{\propto}ln\ensuremath{\omega}{\ensuremath{\xi}}^{3}$ in a generic case. For a concave Fermi surface, the leading terms do not cancel, and $\ensuremath{\sigma}(\ensuremath{\omega})\ensuremath{\propto}{\ensuremath{\xi}}^{2}$. We extrapolate these results to a QCP and obtain $\ensuremath{\sigma}(\ensuremath{\omega})\ensuremath{\propto}{\ensuremath{\omega}}^{2/3}$ for a convex Fermi surface and $\ensuremath{\sigma}(\ensuremath{\omega})\ensuremath{\propto}1/{\ensuremath{\omega}}^{2/3}$ for a concave Fermi surface.