Gravitational wave observations offer unique opportunities to probe gravity in the strong and dynamical regime, which was difficult to access previously. We here review two theory-agnostic ways to carry out tests of general relativity with gravitational waves, namely (i) parameterized waveform tests and (ii) consistency tests between the inspiral and merger-ringdown portions. For each method, we explain the formalism, followed by results from existing events, and finally we discuss future prospects with upgraded detectors, including the possibility of using multi-band gravitational-wave observations with ground-based and space-borne interferometers. We show that such future observations have the potential to improve upon current bounds on theories beyond general relativity by many orders of magnitude. We conclude by listing several open questions that remain to be addressed.