It would be a good idea.

Where the duct goes through unheated areas, insulating the duct will prevent condensation inside the duct -- a damper on the outside won't help here, because the moist air being vented out will condense on the inside of a cold, uninsulated pipe. In heated areas, insulation is unnecessary.

Placing a damper on the outside vent may not be necessary for preventing air infiltration since there's an inside damper, but it IS necessary to prevent insects, birds, and vermin from entering the duct and building their nests in that nice, warm spot.

P.S. -- you said that there's no tub/shower, but the heated air in your house will contain enough moisture that condensation in a cold duct could be a problem.