4 days may not seem that much to you - but it is quite significant when you consider that your puppy will only be 52 days old when you bring him home at just over 7 weeks. That is 8% of his lifespan. More importantly, when you consider that most of his learning about "how to be a dog" - social behviours, etc takes place
after about 6 weeks of age, depriving him of 4 days actually means depriving him of nearly 30% of his learning opportunities/time. 4 days isn't as minor as you think
Having the opportunity to spend 4 whole days with him isn't that minor an advantage either. But if you did go ahead with that, it would be wise to keep in mind what he's lost out on and to make very sure that he gets plenty of time with other dogs and puppies of about his own age from the outset. If you can't do that, then you might consider that the balance of benefit is greater if he's allowed to remain with mother/littermates for those extra days.