There is now a product which claims to remove silicone residue at the big-box stores. I've never tried it and know nothing about it- I just remember seeing it as I rushed through there one day.

The OP may also be trying to caulk too large an area, in which case it will take more than a few days for the caulk to cure completely all the way through. Most caulking is meant to be applied thinly so it can stretch and in areas that do not exceed 1/4 to 3/8 inch in width. Caulking doesn't have to be pretty to be effective so tool it as good as you can quickly and forget about it if it begins to stick in globs or balls to your finger or tool- by then it is too well cured to be tooled. As always the cleaner the surface the better it will adhere..