Allergies possibly. It's hereditary. Doesn't sound like yeast; that would be more inside the ear, smelly and very yucky looking

so if you can stick a q-tip in there and it comes out not looking so bad, it's not yeast. The raw skin you mentioned could just be that his ears are inflamed from whatever he might be allergic to (or it could be from him scratching

)
Honestly, I just took Caney to the vet 2 days ago, POSITIVE she had a yeast infection -- nope, just allergies

But I guess it helped to ease my mind that nothing was actually infected. Itchy ears was one of her symptoms (and VERY sensitive to touch). She's on prednisone now to control the itching, and I'm working on doing everything I can to keep the place clean and free of possible allergens and irritants. Ugh.
You might try giving him benadryl for a few days. Antihistamines aren't always enough to combat allergies (at least not in Caney's particular case, since they'd gotten pretty bad) but you might give it a try to see if that helps. If so, there's your answer

But you have to give it consistently for at least a few days, maybe longer (something I didn't know) for it to actually start "fixing" his overall itchiness.
Fish oil pills (or omega 3 pills), 2 or 3 daily, will also benefit his skin and coat overall.