Yeah, that’s probably true. Calling a one-person operation an agency is a stretch, so, no, I don’t use that term. Perhaps I should. 
More than just the semantic difference, though, is the clientele. Organizations using an ad agency are usually prepared to spend money and understand what an agency does.
Organizations hiring independent contract designers are typically smaller, operate with some naive assumptions, and mostly need one-off things done. I think crowdsourcing has contributed to this change in perception that didn’t exist 30–40 years ago at the beginning of my career.