The "official" growing time for herbs is about 4x20 = 80 mins. However, I've had some patches that have take 2-3 hours to grow. What gives? Thanks.

Because the growth times given are sort of the minimum

Other factors occur:

1) Logging in or log resets current grow cycle, so you slow them down that way

2) As in nature sometimes plants just don't grow. Plants randomly repeat growth cycles sometimes to mimic this.

3) Getting diseased and being cured will reset the growth cycle also (though this doesnt apply so much to herbs as it does to trees etc since no-ione really check herbs to cure em)