We all know by now that refactoring is important. The problem is- when do you refactor?
In my opinion, the time to refactor is when the thought first crosses your mind.
Let's say you're implementing a new feature or fixing a bug. You think 'hmm, I seem to be duplicating code' or 'looks like we could reorganize a bit of this'. Instead of pushing the thought 'this could be refactored later' to the back of your head where it will stay there for a very long time and come back to haunt you once your code is so messy that even you don't know how it got that way, the answer is to refactor immediately. Do what you can to improve your code. If you don't, then you and everybody else just pass the buck on and say "X will do that someday", and soon you are in programming hell. Moreover, if you refactor regularly, you most often need to just refactor small areas at a time. It's much better then saying we'll refactor this entire application- more often than not, throwing it out and restarting would be more appealing.
Of course there are reasons why people are frightened of refactoring. Acknowledging the need to refactor does not mean saying to the world "I have bad code". You do not have to have glaring bugs to think of refactoring. Think of it as general housekeeping. Just because the hotel housekeeping staff cleans your room every morning does not mean you left it looking like a pig sty the night before.
Another important reason is the fear of breaking code. Which is why to refactor confidently, you need to be backed by a solid suite of test cases. If you don't, then this is a very real risk you run.
Refactoring is easier said than done. You rarely find that you've been told to spend the next week refactoring. There are always more pressing tangible deliverables. However, it is extremely important to realize that the intangible ones will actually affect your tangible deliverables in the near future. Just as we need to consciously budget time to write automated tests, refactoring does not happen magically either. A small bit of effort regularly will go a long way to contributing to better manageable, extensible and flexible code.