The answer is that there were a lot of silent-era films where the butler did it. There were quite a few between 1915-1927. This is where the trope was more widely dispersed.

The deeper answer is that the wealthy are always suspicious and a little afraid of their hired help, especially ones who know such intimate details about one’s life as a butler would. Thus, the trope appears almost out of thin air.