Thanks for the suggestions! I've been keeping doors closed with garbages in them (babies room, bathrooms) and keeping the kitchen cupboard shut tight (although with some effort he can open it to get to the garbage, but at least I hear him trying and can stop him before he gets there. He is never left to roam free when we aren't home with him (we have much bigger issues than garbage when that is the case!) so I just have been trying to be more diligent to catch him and redirect him if I see him with his face shoved in a garbage can. Thankfully he hasn't gotten any more diapers since I posted this! That is definitely the biggest gross factor! He has snuck the odd kleenex though and paraded around the house with it in his mouth.
The funny thing is, our garbage in our back entrance that usually has the food waste in it is the one that he doesn't generally bother. It is covered but he knows how to open the lid. I have been careful to not put anything too "stinky" in there though or any meat leftovers that would draw the dogs to it. The bathroom garbages are the most commonly dug through ones. He likes those snotty kleenexes and waxy q-tips! (eww)