To be honest, I pick these up so cheap I don't even bother trying to sharpen them anymore. I just toss a few in my sketch kit whenever I go out for the night, and replace them when worn.

But prior to that, I too always kept either a small piece of sandpaper or a emery board at hand for this exact purpose.