The Conservation Effects Assessment Project was established to quantify water quality benefits of conservation practices supported by the U.S. Department of Agriculture (USDA). In 2004, watershed assessment studies were begun in fourteen agricultural watersheds with varying cropping systems, landscapes, climate, and water quality concerns. This paper reviews USDA Agricultural Research Service 'Benchmark' watershed studies and the challenge of identifying water quality benefits in watersheds. Study goals included modeling and field research to assess practices, and evaluation of practice placement in watersheds. Not all goals were met within five years but important lessons were learned. While practices improved water quality, problems persisted in larger watersheds. This dissociation between practice-focused and watershed-scale assessments occurred because: (1) Conservation practices were not targeted at critical sources/pathways of contaminants; (2) Sediment in streams originated more from channel and bank erosion than from soil erosion; (3) Timing lags, historical legacies, and shifting climate combined to mask effects of practice implementation; and (4) Water quality management strategies addressed single contaminants with little regard for trade-offs among contaminants. These lessons could help improve conservation strategies and set water quality goals with realistic timelines. Continued research on agricultural water quality could better integrate modeling and monitoring capabilities, and address ecosystem services.