He can’t block you because of right to passage. And He can’t take your land legally unless you give in.
But I have an unusual opinion,
Chill out. He used to be your family friend right? The way you have explained it, it seems, he must be hoping a good business out of this land and that is why he is forcing you to bail out. Partner up with him for now in this venture, draw it strong legally on paper and sneak in your right to road on paper. If the business works, you earn, if it doesn’t you would still be owner of the land and be able to fight legal battles.
Anyways, what was your thought process while buying this land? Genuinely curious, you bought it as an investment or bought it to actually use it yourself and live there?