I've done heavy duty exercise in all conditions for many years (and not just cycling). I did once spend a solid 5 days in the pouring rain the BWCA once and I have cycled at temps down to minus 23F and been comfortable. Brevets are somewhat challenging sometimes because you have limited space and a long time so things can change quickly and unexpectedly for the worse; that somewhat hosed me on my first one when temps dropped 10 degrees lower than we expected. I added a rack to my bike after that one so that I could carry extra clothes and stash others.

But on my last brevet of the season, an October 200K in Rochester Minnesota, I was toasty warm and never had any issues with either the cold or the rain, both of which were ever present. Since I am lucky enough to actually have a thermometer on my bike (a feature I highly recommend), I can say that the day started out at 42 degrees once we were away from the Rochester heat island. It rained lightly, but continuously until about 2:00 pm at which point the showers started to come and go. It rained harder in Rushford, about halfway; I expected that since valleys hold clouds. It warmed up to about 46 and stayed there for most of the afternoon with brief surges. The wind picked up considerably at about 3:30. I had my clothing perfect for the weather (having honed it with year round commuting). Here is everything I had on and my comments:

1. Wool cycling socks (high ankle) with plastic bags that I added at Stewartville. I had regular cycling shoes on which are highly vented so I added wind blockers. For me, the wind covers are great, but if I zip them up, they really constrict blood flow to my feet so I like to leave them half zipped. So the plastic bags extend up into the tights and make sure the rain water flows into the shoes, but doesn't get directly on my feet. It then drains through all those air holes. Happily, wool insulates when wet so the sweat doesn't make my feet cold. Also, I tend to completely loosen the buckles on my shoes so that I have lots of wiggle space. The wind blockers ensure that this isn't a problem.
2. I had a pair of specialized equinox gloves that I bought at my LBS. These had a lot of neoprene in them and worked well. These were wind-proof and breathable and were warm even when soaking wet. I have also cycled with my 2mm scuba gloves (which are made to insulate using the water as the insulation).
3. I had regular cycling shorts on with bib tights over the front. I bought these a few years back for winter riding. They are neoprene on the front and regular lycra on the back. The venting of the back ensures that they draw any sweat away, but the front blocks everything.
4. I had a long-sleeved cycling jersey that had some wind blocking abilities on top. I added a thin vest for more wind blocking and a waterproof, breathable jacket on top. I got the jacket from REI and it's great but I notice that Novarra (the REI brand) has cheap zippers, so I am careful with them. Pit zips are a really nice feature so that you can open them and vent/dry when it isn't raining.
5. I had a cycling beanie on my head and a lycra neck wrapper (also from LBS). I brought my balaclava, but didn't need it. I eventually put a shower cap over my helmet. My helmet is so well vented that as long as the cap doesn't wrap my head, it still vents. In really pouring rain, keeping a ton of water off your head is a good idea. In light rain or scattered showers, it's optional.

In the years of commuting (pouring rain, thunderstorms, snow, sleet, etc). I have several cardinal rules for dressing:

1. If you are not cold for the first 3-5 minutes of the ride, start taking things off. You are overdressed. Overdressing is the number #1 cause of being cold while riding. Another way to think of it is that if you are standing around waiting for the ride to start, you should be quite cold. Have an extra jacket for this period and throw it in the car right before you leave.
2. It's more important to block the wind than to block the rain. You generally don't need much in the way of insulation until you are at or near 32. Your body generates its own heat, your clothing needs to ensure the wind doesn't blow it away. Along with this is the importance of not wearing anything totally impervious to water from the inside out. Breathable fabrics ensure that you move the sweat away from your skin. Modern base layer fabrics work this way - they keep the side facing your body dry while the outside saturates with water and slowly evaporates. If you wear a rubber suit, the base layer can't get rid of the sweat. Then you have the problem that any time you stop, you instantly get shivering cold.
3. Leave anything with cotton at home. In mountaineering, the saying is "cotton kills". Cotton gets wet and sucks the heat out of you. It also dries poorly.
4. Layers don't work unless they are thin (makes them easier to pack too). I only wear true fleece (and it is only about 5mm thick) when it is under 20 degrees. Above this temp, it's generally too thick. Thin wool is in vogue, but there are all kinds of fall jerseys that block wind. Always plan that the temps will be at least 10 degrees less than the forecast.
5. Protect your core first. If your core is warm, your extremities take care of themselves. Wear a wind vest, even if it is under your jacket.
6. EAT - your body uses lots of calories to keep you warm. The amount you eat at 40 degrees and raining should be greater than what a summer ride is on the same course. I looked at my records and found that I burned about 150 calories per hour more on this brevet than on the 400K in late August riding at about the same pace. Who doesn't need an excuse to eat peanut butter cups? And don't forget to drink - in the cold rain, you don't get as thirsty, but you still have to drink to avoid dehydration (more trips to the restroom though since you are not going to be sweating so much so avoid drinking too much).

Everyone should read about the symptoms and stages of hypothermia which are on the net somewhere. Shivers are your body's last defense so if you are shivering and suddenly stop shivering, it's a really bad thing (unless you have come into a warm area or done something else to warm yourself up). Time to knock on someone's door and beg for mercy.