HESS J0632+057 is one of only two unidentified very-high-energy gamma-ray sources which appear to be point-like within experimental resolution. It is possibly associated with the massive Be star MWC 148 and has been suggested to resemble known TeV binary systems like LS I +61 303 or LS 5039. HESS J0632+057 was observed by VERITAS for 31 hr in 2006, 2008, and 2009. During these observations, no significant signal in gamma rays with energies above 1 TeV was detected from the direction of HESS J0632+057. A flux upper limit corresponding to 1.1% of the flux of the Crab Nebula has been derived from the VERITAS data. The nondetection by VERITAS excludes with a probability of 99.993% that HESS J0632+057 is a steady gamma-ray emitter. Contemporaneous X-ray observations with the Swift X-Ray Telescope reveal a factor of 1.8 ± 0.4 higher flux in the 1–10 keV range than earlier X-ray observations of HESS J0632+057. The variability in the gamma-ray and X-ray fluxes supports interpretation of the object as a gamma-ray emitting binary.