Can I Prevent Somebody Off My Property? See Below.

I live in a small town in Maine. I am well versed in our local ordinances. Here is the situation:

Setting: densely settled older houses on a hill. My uphill neighbor is the problem. The neighbor's property is flat and then steeply drops about 10 ft in elevation to my property. The slope is covered with trees and brush and is almost too steep to climb. There is about 3 ft of flat lawn at the bottom of the steep slope that is our property, that is, the property line essentially is at the bottom of the steep slope. Beyond the three feet of lawn is our driveway, and beyond that is our house right up against the driveway.

Problem: My neighbor is wacko about trees being cut on the slope that he owns. It's a long story where he has been irrational and unreasonable to the point of putting up two video cameras looking right on our house, allegedly to keep us from cutting any trees down on the slope. My research shows that the cameras are legal and that actually is not my question. Unless it is a slam dunk I have no interest in addressing the cameras legality.

Question: The neighbor has threatened to build a 6-ft tall fence on the property line. Anything under or at 6 ft is not defined as a "spite" fence in Maine (spite fences can be taken to court for removal), and the ordinance allows a fence to be right on the property line, so from that information the threatened fence would be legal.

However, from my description above, I think it would be impossible to build that fence if the contractors could only work on their side of the property line, basically on that steep slope. Common sense would say that they would be standing on my property for the entire fence installation.

Can I prohibit them from stepping on my property to build that fence? I would assume so.

Feel free to ask questions. Thanks.