The best way to remove silicone sealant is to soften it with long-term exposure to mineral spirits to attack its bond and then scrub the remaining residue with a brush, although a razor knife might be necessary for the most stubborn areas. No solvents dissolve silicone sealant, so there are no shortcuts.
It might be tempting to get sandpaper out to scrape away the residue. However, even with the coarsest grit, silicone sealant just makes the sandpaper gummy, taking away its abrasive properties. If you are dealing with a finished surface, try denatured alcohol instead of mineral spirits to preserve the finish. Test any surface with the mineral spirits in an unobtrusive spot to see if any discoloration takes place.