I did HKDL on a very quiet day earlier this year.
We skipped a few things, such as Autopia and one of the shows, because I wanted to do things like Mystic Manor and the Big Grizzly Mine Cars multiple times. It was not a problem doing everything in one day, but it definitely felt like a full day park.
As Garet said, if you're a Disney fan you may really enjoy how quiet the park is. Much nicer than the wall to wall crowds one often sees at Disney. With only four days in HK however, I'd probably stick to the one at Disney.
I only spent a few hours at the Peak. I combined it with a walking tour of the city. I didn't do the touristy things like the Wax Museum at the top.
Are your kids into toys? I spent a lot of time exploring Hong Kong's toy malls. We're talking multiple stories filled with dozens of stores, selling things like Transformers.