Your brake shoes are probably only making contact with the drums in the middle; when you push the pedal harder, the shoes flex until the lining is making full contact, then the pedal is hard.
I adjust my drums by backing off the star wheel until they're just locked, then going 15 (or so) clicks tighter on all wheels. There will still be some drag; until the shoes are fully seated you'll have to keep an eye on the adjustment, and whether the brakes are too hot.