A custom Control is something you provide when you create base functionality of a new UI component. There are various pros and cons for this, but for example, if you want custom selection behaviour of an ItemsControl, you could best do it by subclassing Selector or MultiSelector (the wpftoolkit DataGrid does this). Also, if you want an object which would contain a new DependencyProperty, you will in most cases derive from Control.
The wpf principle contained here is the "lookless" control paradigm, or "be sure to expect someone templating your Control, or at least make it behave nicely in your own template scenario". Custom Controls are usually created with reusability in mind, often as parts of framework dlls.
A ControlTemplate is essentially a description of a replacement visual tree, and can be set either explicitly on FrameworkElements, or as a part of a Style. This is the option you should aim at when your goal is primarily to make an application and be done with it. You can do almost anything with a ControlTemplate visually, if you are able to get the bindings and triggers (and the possible containing Style itself) right. All this can be declared as a resource an reused, to give your application a common "theme".
A UserControl is a self-contained composite control, with parts individually editable in the designer, and is best used if you need to see your components and manage them in the designer. A ControlTemplate, on the other hand, will not expose its components for manipulation in the designer (although it will be visible). You usually create a UserControl for a Customer details page, or a Product display browser, or any case where you don't want to create a full-blown Control, but want detailed view with full designer support.
A special case here is if you use the MVVM pattern. Many great MVVM implementations use UserControls as Views, and ControlTemplates and Styles as resources used by those views. MVVM practice also minimizes the need for a custom Control, and has many other benefits.