I have experienced almost exactly the same thing, and although sometimes, for various reasons, including timing, that one great love of your love doest quite work out, you never ever forget them.
The man in question for me is also over 6 ft and heavily set, was in the army in his younger days. Time, life experiences have taken their toll, as they do on all of us, but he still (for me) has an overwhelming sense of gentle powerful masculinity which exudes out of him. Although we had to go our separate ways, I still miss him dreadfully at times.
Once you experience a man like that, it is very difficult not to compare them to someone else, although one must in order to get on with one's life. You just keep a part of them in your heart, and take it to your grave.