I've been reading lots of yak-yak in the blogosphere about how there needs to be a primary challenge to Obama. OK, let's talk about that in some real terms. For shits and giggles, we'll toss in 3rd party/independent challenges, too. This gives us two different modes of challenge, one internal to a party and another intended to cut across institutional coalitions.
There are a few models of what an internal challenge can look like. The classic is a moderately powerful insider taking a run at an incumbent who is clearly failing or is perceived to be weak. In recent times, that gives us Teddy Kennedy's challenge to Carter, Bill Bradley's challenge to Gore, who, as Vice-President, was the default choice for the nomination on the Democratic side, and Ronald Reagan's 1976 challenge to Ford and Pat Buchanan's 1992 challenge to Bush I for the Republicans.
In the Democratic cases, what we have is an older, displaced faction (Northeast Democrat) trying to oust a newer, up-and-coming faction (Southern post-LBJ Democrat). These are dynastic battles, if you will, of the older power center losing ground to the newer. Carter really was weak, mostly due to circumstances out of his control, but also since he had no Congressional coalition partners who could really back him up. Mondale as VP was his biggest asset. Carter also had come to power through an accident of history - Watergate. Without that political catastrophe for the Republicans, it is unlikely the Dems would have won in 1976. The challenge from Kennedy turned Carter into a loser in the eyes of party regulars. Kennedy never had a chance since his only reason for running was his name, but that alone was worth enough to mortally wound Carter.
Gore was not so much weak as targeted by the media and by the same Northeast faction as backed Kennedy, grudgingly supported Fritz Mondale (Kennedy's absence in 1984 is not an accident), had won the nomination with Dukakis, and had backed Tsongas in 1992, only to lose to a different Southerner. Absent a challenge from that faction, he would not have had to run a primary campaign. Bill Bradley, who would go on to be one of the founders of the Unity party movement, had marginal successes in some Northeastern states (New Hampshire, Delaware, Connecticut, Rhode Island, Maine, Vermont), but never broke 50%. What he did was provide the media opportunities to attack Gore from the start of the election season on and to encourage party disaffection among the Stevensonians. Gore made a probably fatal mistake of selecting Lieberman as VP in an attempt to move to a more "centrist" position. The shifting political stances in combination with unrelenting mockery from the media did weaken Gore in the general. He then had to face a 3rd party challenge.
A slightly different internal challenge is when the challenger is not from an older faction trying to hold on to power, but from a new faction aiming for a takeover. This is the situation of 1976, when Reagan challenged Ford and contributed to Ford losing the general election. I have written about this contest before, and I do think it is closer to where the Democratic Party is now than the previous two examples. The key in this challenge is that Reagan was not running as a personal power trip (or, not merely that). He was the leader of a compact but growing faction that had figured out how to capitalize on cultural resentment for mass appeal plus complete openness to the money elite for cash flow. He was very successful in the primaries, winning an electoral map that looks very much like the recent midterms, and very nearly won the nomination. He may have cost himself the nomination by trying to appeal to moderates with a unsubtle selection of a moderate as a running mate - it didn't convince moderates and pissed off conservatives. Reagan's lesson from that was never moderate one inch further than necessity demanded. His challenge to Ford probably did not directly cost Ford the general election (Ford was running under Nixon's ghost, after all), but he stirred up party dissatisfaction and was set up as the heir apparent for 1980, coming away with influence among the power brokers and a riled up voting base.
Buchanan was not a viable challenger to Bush in 1992, but that was not his purpose. It was to discipline Bush and keep him positioned to the hard right, and to that degree, he succeeded. However, with the 3rd party general election challenge by Perot, the move to the right may have cost Bush centrist Republicans.
The next model of challenge is when a sitting president drops out of the race after a successful challenge. There are two modern examples - Harry Truman challenged by Sen. Estes Kefauver of Tennessee and LBJ challenged first by Sen. Eugene McCarthy of Minnesota and then by Sen. Robert Kennedy of New York.
Truman went into the primary season, which bears very little resemblance to the current primaries, with rock bottom poll numbers and was challenged by the populist Kefauver. Truman had more Democratic support, but Kefauver was favored by independents. Kefauver upset Truman in New Hampshire, and Truman withdrew from the race not long afterwards. There was a pile-on of entrants after Truman withdrew, including people like Hubert Humphrey and Averell Harriman. Truman engaged in a number of back-room deal to try to get other challengers to Kefauver, including, Adlai Stevenson, who said he wasn't interested and wanted to run for Governor of Illinois. At the convention in Chicago, Stevenson gave a well received speech and was convinced to stand for nomination. After three intensely contested rounds, he won the nomination, never having campaigned. This kind of overnight back-room deal is not going to happen again.
In 1968, McCarthy was mostly doing a protest campaign, trying to influence party support of the Vietnam war. He made a crucial decision to pour almost all of his campaign resources into New Hampshire, and came in a close second to Johnson, though not as close as Bradley came to Gore (42%/49% vs. 46%/50%). Kennedy jumped into the race four days later. By the end of the month, Johnson had withdrawn from the contest. Like Truman, LBJ was not content to let the challengers have the field, and privately supported Humphrey, his VP, who went on to win the nomination. The primary system was very different than we have today, closer to the situation in 1952 with Stevenson's first nomination. Humphrey did not campaign, focusing on winning delegates outside of the few primaries, and was handed the nomination at the convention. The primary contest shook loose the old New Deal coalition. The fragmentation of the party after the convention remains evident in the voting habits of the party to this day - a split between the working class and ideologues, or more specifically, between the populists and the progressives. In some ways, it was the opposite of the 1976 Republican convention, where a leader and a direction came out of the convention to dominate in the following electoral cycles, much like FDR in 1932. The Democrats lost the Dixiecrats, who bolted first to Wallace and then to Nixon and Reagan. This may be the only unequivocal good to come out of the 1968 clashes.
Now we get to 3rd party challenges. In recent times, this would include George Wallace, John Anderson, Ross Perot and Ralph Nader.
Ironically, of all these candidates, the most successful was Wallace, the most execrable. He ran a pure race-based campaign and won electoral votes as a result. It is the bogey man of 1968 that haunts Democrats, even though all of his electoral votes would not have made the difference for Humphrey. His campaign marked the modern use of pure resentment as the motivating force for a national campaign, and probably drew more from Nixon than from Humphrey.
With Anderson in 1980, we see the last hurrah of Republican style progressives. He can also be seen as a proto-Unity candidate, having chosen a Democrat as running mate. It is unlikely he drew too many votes from Carter, who was disliked by his own party, and simply attracted a nebbishy middle.
Perot's two runs involve a lot of maybes. The conventional wisdom is that he drew support away from Bush, allowing Clinton to squeak by in 1992, though exit polls show he at most drew form each equally and, given Bush's plummeting poll numbers, probably dug more deeply into Clinton's support. In the 1996 run, exit polls showed that if Perot had not run, his supporters were more likely to support Clinton. Thus, in both situations, Perot inflicted greater losses on the Democratic side than the Republican side. Unlike defections to Anderson or Wallace, these defections materially harmed the Democratic candidate.
1996 was also when Ralph Nader entered the scene. His impact was negligible, and probably attracted voters who might otherwise have left the ballot blank or written in "Mickey Mouse". In 2000, voting for Nader was very much a protest against Gore, fed in no small part by the media. Nader probably siphoned off votes that would have gone to Gore, most crucially in Florida. Nader votes in New Hampshire, Missouri, Ohio, Nevada and Tennessee were also direct losses to Gore. However, had it not been for the intraparty assault on Gore by Bradley in the primaries and the incessant drumbeat of spite and vitriol from the so-called liberal media, it is unlikely that the sliver of votes Nader received - barely 1/3 the amount of John Anderson in 1980 - would have made a difference. Nader was merely an insult added to injury. Thanks, Ralph.
So, what conclusions can we draw from this? A primary challenge is going to be more damaging to a sitting president than a 3rd party challenge because it represents a schism within the base. An internal challenge is an attempt to rearrange the power structure within a party in a big way. In no case of a primary challenge did a challenger manage to go on to win the nomination, though a few were able to win in primaries. A challenger is usually a spoiler, not a contender, and is more beloved of the media and party insiders than of the base. The chief outlier here is Reagan, the only clear example of a rising power who went on to capture his party. The Democratic examples have two insurgents (Kefauver & McCarthy) who upset the dominant faction, but both were suppressed by party insiders at the conventions. With Kennedy and Bradley, it was the party insiders opposing a rival faction and willing to see the party lose to ensure destabilizing the other faction's power base. Party positions are patronage jobs, after all.
A 3rd party candidate decided an event only when the fractions between the major candidates were so close that missing less than a tenth of a percent in an elector rich state like Florida flipped an election. The degree of difference caused by Nader is so extreme, it has to be considered an outlier. More typical would be the other 3rd party runs where percentages were high enough, particularly with Perot's first run, that it could have wreaked havoc in a tight race. In the end, the weight of the electoral system dampens the damage that can be inflicted by a 3rd party candidate.
There's the landscape for a primary challenge. Anyone who would try this had best be ready to give up all hope of a political future, as he or she would be mounting a campaign against the dominant faction. There simply isn't anything in the Democratic party analogous to the Movement Conservatives behind Reagan. Hillary is probably the only Democrat who has that kind of support and loyalty from her coalition, but is the last person in the party to deliberately undermine a sitting president that way. No other candidate has the right combination of name recognition, an established base, and/or a defined cause.
Someone could act as a spoiler, much like Kennedy in 1980 or Bradley in 2000, but those two came in with media support and a dedicated base. Obama is unpopular in the party, but not mortally so (contra Carter), and he is still the beloved of the media for intraparty battles (contra Gore), so is unlikely to be as damaged by a primary challenge as Carter or Gore. If there is another big economic shock and/or if unemployment stays as is or worsens, then the cumulative bad news may embolden challengers from his own faction and make the press distance itself from a loser.
Mostly, the danger to Obama will be in the general, where wider dissatisfaction, resentment voting and a less fawning press will put him in a condition closer to Humphrey's or Gore's in the general. In this case, a 3rd party challenge from the left could siphon away votes in a tough challenge. That person would need to be able to capture the public's imagination the way Perot did in 1992.
The trouble with the Democratic Party, an organization that is the worst in politics except for all the others, is that it has not discovered a way to recombine the populist and progressive modes of its liberalism in a way that matches the force of the New Deal coalition. Until it purged itself of the Dixiecrats, it could not do this. Since doing so, the progressive faction has not cared to do this.
It needs to build an analogue of the Movement Conservatives, and that cannot be done without the majority of the white working class once again agreeing that it's material interests are best served by this party.
Update - I have made small edits throughout the post to fix typos, correct a couple of dates, and clarify some points. Also, in answer to some private notes, I am not saying there shouldn't be a primary challenge. I'm saying that people are not thinking through what a primary challenge might accomplish, based on what we can observe from prior challenges and what we know about intraparty factionalism. This post is to clarify objectives, not advocate a specific action.