As for the question of what ninja actually did wear, ask yourself this : what does the average spy wear ? The answer, of course, is whatever it takes to blend in and get the job done, whether that means shorts and a T-shirt, military fatigues, or a suit and tie. The same was true of the ninja. Hiding in plain sight was their entire modus operandi. Most dressed like farmers, both because it worked well as camouflage in a nation of farmers, and because most of them really were farmers. A real ninja would laugh at the portrayal of an intruder tiptoeing across a rooftop wearing black pajamas in broad daylight.
Picture this : the year is 1597.
Your name is Arima Kihei. You have no home, no possessions to speak of save for the clothing on your back and the pair of swords strapped to your waist. Your days are spent focused on a solitary purpose : seeking out martial artists – the more renowed the better – and killing them.
You aren't a stalker or a serial killer. You are a shugyosha : a swordsman on a pilgrimage to test your skills in the only way that truly matters.