As with many questions "always yes" or "always no" is the always wrong answer.
Usually, it's best to color up for the reasons mentioned earlier in the thread. You don't want to be the person who makes their job harder. However, if coloring up gets you high denomination chips which will be difficult to cash, you might consider skipping the color-up. Typically, this doesn't become a factor unless the color-up involves chip denominations of $1000 or more.
A situation where you would skip a color-up is if you suspect a backoff is in the works. In these situations, your goal is to get the hell out as quickly as possible. If you color up even a small amount, they can delay you for as long as they need/want in order to assemble the crew who executes the backoff. You're already going to be memorable, so there is little additional harm with leaving them stuck with an empty chip tray. Your primary goal is to leave before they have the opportunity to start a conversation where they formally tell you "no more blackjack".