It sounds like you've got a cultured product - a man made material that mimics natural stone in appearance. Whether or not your pan can withstand mineral spirits would depend on the color, age, and brand of pan, however, mineral spirits won't touch the silicone, so using it would be an excersize in futility. To be quite honest, I've not found a product that can remove silicone residue, the only way is to use a sharp single edged razor blade and carefully sc****/cut the residue as best you can. Another method I've used on exterior siding is to use a white rubber dog chew and rub it on the silicone residue like an eraser. Get one of the solid chews and you can carve it to fit into nooks and crannies. Colored chew toys will leave scuff marks that the white toys don't.

Ultimately, it's going to take time and lots of elbow grease to remove the silicone to any acceptable level.