Just for back ground purposes.
"I thought the use of the tiller plus my day of help tilling up the garden would mean more than this."
It might be worth your while to do a little accounting. Find out what it would have cost to rent a tiller like his. Add up any hours of his labor at minimum wage. Then figure out how much time you and your dh have spent weeding, watering, etc, and how much $ you might have spent on water, soil amendment, seeds or starts, etc and add up your side. At the point that your labor and expenses plus the 3 bags of produce are greater than his contribution tell him the gig is up and show him the math if necessary. This guy sounds like he will try to guilt you into letting him do and have whatever he wants, and subtlety does not seem to get through.
I don't consider it petty, and I know how these things get more irritating the longer they go on. Its your home and you are entitled to privacy and control over when and if you get company. You don't have to "receive" visitors if you don't feel like it. I would keep the doors locked at all times and don't answer when he knocks if you don't