Ultracold trapped atomic systems are ideal model systems to investigate physical phenomena in the quantum regime.In this work we studied instabilities in spin dynamics of a sample of nondegenerate trapped 87 Rb gas.This is the first experimental investigation of the so-called Castaing instability in such systems.The Castaing instability is an instability in the spin dynamics of spin-polarized systems as a result of introducing sharp spin gradients in their spin profile.We used an optical technique via the ac Stark effect to initialize arbitrary spin profiles in a sample of evaporatively cooled nondegenerate 87 Rb gas.The experimental results manifest evidence for the presence of the Castaing instability in both transverse and longitudinal spin channels.The results agree reasonably well with theoretical studies and results of a numerical solution to the spin transport equation.Some imperfection in the longitudinal spin gradient due to the spin-state preparation technique makes the signature of the instability less clear in the transverse channel.We propose another preparation technique to overcome this shortcoming in future experiments.First, I would like to thank my supervisor, Dr. Jeffrey McGuirk.This thesis would not be possible without his help and guidance.I have been fortunate to work in his group and benefit from his incredible knowledge, enthusiasm and patience.He is always supportive and willing to give advice, yet knows when to let you experience research and learn science on your own.I am grateful for having two great physicists