The handful of protesters standing across the street did little to dampen the enthusiasm of the hundreds of guests gathered in late July for a Planned Parenthood fundraiser. In the backyard of the waterfront home in Maple Bluff the sun was shining, the lake was sparkling and the white wine was flowing. First up on the agenda was the presentation of the family-planning group’s first lifetime achievement award. The recipient: Gov. Jim Doyle.

“Wisconsin women have never had a greater champion in the governor’s office than Jim Doyle,” proclaimed Teri Huyck, the president of Planned Parenthood Advocates of Wisconsin. The national president of Planned Parenthood, Cecile Richards, visiting from New York, chimed in. “He has been the most incredible leader of any governor in the nation,” she said.

In his remarks, Doyle recalled the many times he frustrated Republican legislators by vetoing bills to block access to birth control and other reproductive health services. He also noted the major victories under his watch,  including a long-fought-for bill that requires hospitals to dispense emergency contraception to rape victims.

Doyle, 65, was looser than usual; more personable, even funny — a reflection, some say, of how deeply he cares about these issues. “He is truly passionate about them,” says Chris Taylor, political director of Planned Parenthood of Wisconsin.

And those who care deeply about these and other social welfare issues are, in turn, passionate about him. But as Doyle prepares to step down after eight years as governor, he is not, in general, a well-liked guy. Admittedly, few incumbents in this economy are riding high, especially Democrats. But Doyle, who is not seeking re-election, is leaving under a cloud. His approval ratings are the lowest they’ve ever been, and it’s hard to find lawmakers even in his own party who have anything nice to say about him.

“I don’t think you could put a bowling team together with his allies in the Legislature,” says one Democratic lawmaker, who, like most of the other two dozen Capitol insiders and observers interviewed, requested that their critical comments be off-the-record.

Even a longtime Republican critic of the governor was reluctant to be quoted, concerned Doyle could use some of his then $2 million war chest to feed the campaign of his state Senate opponent.

Doyle’s poor relations with state lawmakers on both sides of the aisle are an open secret. Legislators and their aides have complained bitterly for years that Doyle did not work collaboratively, involve them in agenda setting or adhere to any of the niceties of political life. When he brought biking great Lance Armstrong to Milwaukee to drum up support for the state’s proposed smoking ban, for instance, he didn’t invite the Democratic assemblyman who lived just a few blocks from the press conference site.

Doyle’s defenders counter that the governor was unimpressed with the talent he saw in the Legislature and instead hitched his star to the institutional power of his office, with its partial veto and absence of term limits.

No one disputes that Doyle has been a strong governor. “He knew how to use the reins of power, and he fully occupied the office,” says one longtime Democratic legislator. But he did not share power well, according to critics, and seems to have alienated pretty much everyone except those in his tight inner circle.

It’s always tricky to try to assess a political legacy in the short term. The Watergate scandal that led to former President Richard Nixon’s resignation overshadowed for years the accomplishments of his administration, which included the 1970 Clean Air Act and creation of Supplemental Security Income, a program for disabled and older low-income persons. When it comes to Doyle, it is especially challenging because people seem to have a hard time separating the man from his deeds.

While his close friends say he is warm and caring, his public persona is anything but. Irrefutably smart and politically savvy, he is also described as “snarky,” “self-centered,” “a sore winner” and “out for Jim Doyle and Jim Doyle only.” He expects absolute loyalty and, if crossed, will hold a grudge — forever. Case in point: Lt. Gov. Barbara Lawton, who never got back in the governor’s good graces after running against former state Sen. Kevin Shibilski, Doyle’s hand-picked candidate for lieutenant governor.

As one former executive staffer put it: “The governor is not a nice person. He’s not a nice human being. If that were a qualifier to be governor, he would not be governor.”

•    •    •    •

When Doyle first ran for governor in 2002, only a handful of lawmakers backed him in the Democratic primary when he faced Tom Barrett, then a member of the U.S. House of Representatives, and Dane County Executive Kathleen Falk. Partly this was because lawmakers knew Barrett, a former state rep himself. And partly it was because lawmakers knew Doyle, the attorney general at the time.

“He seemed to have no respect for the Legislature,” says one former lawmaker who served on the Joint Finance Committee, the budget-writing panel. The lawmaker recalls that Doyle seemed put out by having to make routine appearances required of all agency heads during budget season to argue for funding or to answer questions. He also recalls that Doyle’s top aide, Susan Goodwin, had the same attitude. “And to this day, they’re both like that.”

Goodwin, who has served as Doyle’s chief of staff since he took office in 2003, remains perhaps his closest confidante. She first worked for Doyle as a policy analyst in the Department of Justice, becoming his executive assistant in his third term as attorney general. She managed his 2002 campaign for governor and has run the executive office with an iron fist ever since. Goodwin, on a couple of occasions, has said she sees her role as Doyle’s “enforcer.”

Observers say Goodwin and Doyle are “two peas in a pod” and the reason the administration is so guarded and secretive in its dealings with lawmakers and the press.

Despite numerous requests, Doyle declined to be interviewed for this article. Spokesman Adam Collins says the governor and his administration are not yet in “reflection mode,” as there is still work to be done before Doyle leaves office in January. Collins says the governor will eventually be doing “these types of interviews” but not until after the November elections. Even Roberta Gassman, secretary of the Department of Workforce Development, and one of Doyle’s biggest fans and longest-serving Cabinet members, did not respond to an interview request.

That Doyle and his inner circle refused to be interviewed is really no surprise. Access to him and his administration has been tightly controlled throughout his two terms in office. Longtime civil servants found themselves under a virtual gag order, directed to refer all press queries to agency spokespeople.

Lawmakers and their aides, many of whom had been dealing with the same department staffers for years, were also suddenly cut off, forced to funnel communications through administrative executive assistants and legislative liaisons — individuals usually appointed by Cabinet heads but often viewed as Doyle’s people.

“Doyle made staffers go through legislative liaisons — people more loyal to Doyle than people with expertise,” says one Democratic aide who also worked under governors Tommy Thompson and Scott McCallum. “When Doyle took over, it became clear the legislative liaisons were there to control what information went between the agency and the Legislature.”

“He ‘corporatized’ the Capitol,” agrees one former Republican legislative aide.

One veteran Democratic lawmaker says this not only made it difficult for legislators to get complete information, but demoralized state workers and stifled their creativity. “He imposed too strong a hand on the civil servants in the state,” he says. “Civil servants answered to Jim Doyle.”

It was no different for Doyle’s own staffers. The governor’s office was a top-down affair, with Doyle the de facto CEO, says one former staffer. Aides did not talk to Doyle unless they had an appointment or he asked to talk to them. No one presented a legislative proposal to Doyle that wasn’t in his best interest. “You just didn’t bring back things that weren’t going to benefit him,” says the former executive staffer.

Absolute obedience and loyalty were required. The best thing you could be was a hard worker. If you got on the wrong side of Goodwin, you were done.

It was a very difficult, “unhealthy” place to work, says this aide. You got yelled at for mistakes, but never heard about a job well done: “There are no hugs in Doyle land.”

•    •    •    •

When Doyle beat McCallum to become Wisconsin’s 44th governor, he was elected with just 45 percent of the vote, by no means a mandate. Nevertheless, he did little to build new relationships or mend fences. “I had encouraged the new governor to get out, come to our office, say ‘Hi,’ talk to staff,” says Democratic lawmaker Marlin Schneider, who has served in the Assembly since 1970. “He never did that.”

Schneider says there was always a lot of stress in the Assembly Democratic caucus, where strategy is discussed, because many lawmakers felt the Doyle administration did not keep them in the loop. It got so bad once, he says, that members kicked Doyle’s representative out of the caucus meeting.

David Walsh, who has known Doyle since both were students at West High School, says that his longtime friend is confident in his own judgment and doesn’t need a lot of people around him. “The principles to him are more important than the relationships,” says Walsh, a local attorney who  was appointed to the University of Wisconsin Board of Regents by Doyle in 2003.

Walsh says Doyle is a “tactician and a practical person” and comes to early judgments about who he’ll need in the long run. “It’s not personal,” says Walsh, who attended Harvard Law School with Doyle. “It’s about achieving the goals he seeks.”

And in some ways, Doyle did not need to ingratiate himself with legislators, especially members of his own party, and especially during his first term. In 2003 he was dealing not only with a $3.2 billion budget deficit, which put the kibosh on any ambitious policy-making, but a Republican-controlled Legislature that was passing bills on wedge issue after wedge issue. Doyle spent lots of time fending off such measures, including a concealed weapons bill and a voter ID bill, which he vetoed twice and three times, respectively.

Scot Ross, executive director of the advocacy group One Wisconsin Now, lauds Doyle for refusing to sign a law that would, in his view, deny legitimate voters the right to vote.

“One of Gov. Doyle’s most important legacies is that three times he vetoed voter ID laws, which are a shameful piece of legislation out of the era of Jim Crow,” says Ross, praising Doyle for standing up to “legislative cowardice” and “pandering.”

“The governor said ‘We’re not going to deprive people the right to vote on my watch.’”

During his first term, Doyle did try to woo the state’s biggest business lobby, Wisconsin Manufacturers & Commerce (WMC), with his “Grow Wisconsin” initiative. Framing it as a plan to “create a business-friendly regulatory climate while protecting the environment,” Doyle supported and signed a tax cut for multi-state corporations that meant they’d be taxed solely on sales, rather than a three-part formula based on sales, number of employees and property values.

And he made it easier for Wisconsin companies to receive federal air pollution control permits from the Department of Natural Resources. Critics say these efforts were to appease WMC in the hopes of bringing in dollars for Doyle’s re-election campaign and to curry support for administrative initiatives. Supporters acknowledge Doyle eventually realized WMC would never be a friend, but the damage was done in the eyes of the governor’s progressive supporters.

Rep. Spencer Black, D-Madison, who is stepping down this year after 26 years in the state Assembly, says the weakening of Wisconsin’s environmental regulations in Doyle’s first term was unfortunate. But he gives Doyle generally high marks for his environmental record, noting the governor’s 10-year extension in 2007 of the Stewardship Fund, which is used to purchase land for conservation and public use; his signing of the Great Lakes Compact; and his support for energy conservation, renewable energy and phosphorous regulations.

“He has quite a few significant achievements that will carry on to the future in the environmental area,” says Black.

•    •    •    •

Former Wisconsin Gov. Tommy Thompson would have been a tough act to follow under any scenario. Though former Lt. Gov. Scott McCallum served briefly as governor after Thompson stepped down in 2001 to serve as U.S. secretary of Health and Human Services, it was the back-slapping Thompson who made an indelible mark on the state and is still greeted at appearances with cries of “Tommy, Tommy, Tommy!”

It’s safe to say there have been few, if any, times Jim Doyle has been serenaded with “Jimmy, Jimmy, Jimmy!” at a public appearance. Not only does the guarded Doyle fail to exude the kind of personal warmth that elicits such devotion, he governed during starkly different economic times.

Thompson took advantage of the prosperity that reigned during much of the 1990s, doubling state spending and tax collections during his 14 years in office. Doyle, on the other hand, was never able to “play Santa Claus,” as one observer put it, since he faced massive budget deficits throughout his tenure as well as a crippling recession in his final term.

He also tied his own hands with a campaign pledge not to raise taxes. Keeping his promise likely helped secure a second term, but by refusing to raise income or sales taxes, or to consider an overhaul of the tax system, Doyle had to get creative in coming up with new revenue to patch successive budget deficits.

Some thought Doyle got a little too creative. In each of his budgets, he transferred money from the transportation fund for a total over the last seven years of $428.5 million, according to the Legislative Fiscal Bureau. The move angered road builders and local governments in particular.

Doyle also transferred $200 million from the Wisconsin Injured Patients and Families Compensation Fund to the state’s general fund in the 2007-2009 budget. The fund is intended to cover money damages for victims of medical malpractice, including past and future medical expenses, and Doyle’s maneuver was challenged in court. The Wisconsin Supreme Court ruled in 2010 that transferring the money was illegal and ordered the state to repay it.

He also increased the tax on cigarettes, imposed a new hospital tax that leveraged federal dollars for Medicaid and negotiated compacts with Indian tribes to bring in new revenue.

Doyle defended his budget strategies as a way to protect his top policy priorities, including education, which he often said was key to the state’s future. Republicans charged that his loyalty was payback to the powerful teachers union, but it’s hard to doubt his genuine dedication. His wife, Jessica, was a public school teacher for 26 years before he became governor and his mother, Ruth Bachhuber Doyle, served as president of the Madison School Board; the school district’s administration building, in fact, is named for her.

Doyle created the Wisconsin Covenant Program to make a college education affordable for all Wisconsin students, invested heavily in the UW-Madison campus and boosted biomedical research, including stem cell research.

Doyle never wavered in his support for stem cell research, even though it infuriates abortion opponents who equate the destruction of embryos with the destruction of human life. In fact, when running for re-election in 2006, Doyle often talked about how stem cell research could help people like his mother, who suffered from Parkinson’s disease.

On these hot-button social issues, Doyle never flinched. He signed into law a bill providing domestic partner benefits to state workers and, in the middle of his re-election campaign, spoke out against a proposed constitutional amendment to ban gay marriage, which was eventually approved by 59 percent of voters.

On women’s health issues he held the line, fighting back repeated attempts to allow health care professionals to refuse to provide services they found morally objectionable and to prevent the University of Wisconsin System health services from dispensing birth control.

Doyle made sure, said Planned Parenthood’s Huyck, “that everyone has access to basic health care, and that women and families have the opportunity — the right — to make the most personal, private health care decisions without intrusion by the government.”

•    •    •    •

Jim Doyle and his three sisters grew up on Madison’s west side. When Doyle and his wife, Jessica, returned to Madison after serving in the Peace Corps in Tunisia and working on a Navajo reservation in Arizona, they too raised their two sons on the near west side.

Doyle’s parents, Ruth and Jim Doyle Sr., are credited with helping rebuild the Democratic Party in Wisconsin after World War II. Ruth, elected to the state Assembly in 1948, was the fourth generation of the Bachhuber family to serve in the state Legislature.

Jim Doyle Sr., a graduate of Columbia Law School, clerked for a United States Supreme Court justice and worked for President Franklin Delano Roosevelt before moving to Madison. He ran unsuccessfully for governor in 1954 and was appointed by President Lyndon Johnson to the federal bench in 1965.

Jim Doyle Jr. met John F. Kennedy in Madison in 1960 when Kennedy was running for president.

According to newspaper accounts, Doyle, then 14, was smitten with Kennedy though his father continued to try to draft Adlai Stevenson, considered more liberal than Kennedy, as the Democratic nominee.

Despite Doyle’s progressive pedigree, he charted a centrist route while in office, targeting goals he could most easily advance through his executive powers.

With health care, for instance, Doyle declined to sign on to a universal health care plan pushed by majority Democrats in the state Senate but opposed by Republicans, who controlled the state Assembly. Even when Democrats gained control of the state Assembly in 2009, Doyle did not jump on board to support “Healthy Wisconsin.”

Avoiding a battle in the Legislature and mindful that health care reform was playing out on the national stage, Doyle instead worked behind the scenes in Washington to secure needed federal funds to build on the state’s BadgerCare program, which, in its various incarnations, now covers nearly 770,000 uninsured and low-income families as well as single adults.

Some wish Doyle had taken the bolder approach, but Sen. Jon Erpenbach, D-Middleton, one of the authors of Healthy Wisconsin, is not one of them. “I don’t think it was a lost opportunity given what we were able to do with BadgerCare programs,” he says.

“One thing I give the governor a lot of credit for is he made sure health care continued to be a high priority,” he adds. “No other governor has done that and it’s something he should be very proud about.”

One area where progressives are unforgiving is Doyle’s failure to act on campaign finance reform. “That clearly was a broken promise,” says Mike McCabe, executive director of the watchdog group Wisconsin Democracy Campaign. “He said it would be at the top of his agenda and he never took it on.”

McCabe says the longer Doyle stayed in the governor’s office, the more enamored he became of the advantages of incumbency. He notes that Doyle raised and spent $10 million during his last gubernatorial campaign, making him the “most prolific” fundraiser in Wisconsin political history. “He fell in love with the money game and he was really good at it.”

Doyle’s ethics came under fire in 2005, amid accusations that a state procurement agent steered a $750,000 travel contract to a Milwaukee-based travel agency connected to the governor’s re-election campaign; two of the company’s owners had donated $10,000 each to the governor’s war chest, the most allowed under law, at around the same time. The allegations came in the midst of the governor’s re-election race against Republican challenger Mark Green.

Doyle always maintained his innocence and was never charged with any crime. But the state employee, Georgia Thompson, was indicted, convicted and spent four months in prison as a result of what many saw as a politically motivated witch hunt orchestrated by Republicans with the help of one of their own, U.S. Attorney Steven Biskupic.

Doyle distanced himself from Thompson. It was only after she was cleared and freed by the 7th Circuit Court of Appeals in Chicago that the governor came to her defense, stating that Thompson was “an innocent woman who was used as a political football.” Doyle did make good on his offer to reinstate her as a state employee.

Doyle’s failure to defend Thompson when it seemed to many she was being unfairly targeted was damaging to his public image: He came off as self-interested and unwilling to protect his own people.

The travel contract case led to another ethics scandal for the governor, this time involving Dennis Troha, a large contributor to Doyle’s re-election campaign and at the time the principal developer of a Menominee casino in Kenosha. Troha was charged in March 2007 with directing $100,000 from the trucking company he controlled to his children, who donated the money to political campaigns, including Doyle’s in 2002 and 2006. An associate was also later charged with making similar contributions. Prosecutors said the money was aimed at currying support for gaming and relaxed trucking restrictions.

Campaign records further showed that Troha and 12 family members had donated $200,000 since Doyle’s first run for governor. State law caps an individual’s donations to a gubernatorial candidate at $10,000 per campaign. Troha was eventually convicted and sentenced to six months of probation for money laundering.

As McCabe sees it, Doyle’s failure to enact campaign finance reform underscores the corrupting influence of money in state politics.

“Part of his legacy will be the sorry state of our politics and the fact that so many people in our state are so incredibly disillusioned with politics and politicians and so many people have become convinced they don’t have a voice in state government.”

•    •    •    •

Doyle’s legacy will include some historic firsts: He appointed the state’s first black Wisconsin Court of Appeals judge, Paul Higginbotham; the first black justice to sit on the Wisconsin Supreme Court, Louis Butler; and the state’s first black district attorney, Ismael Ozanne, in Dane County.

Isthmus reported that Doyle chose Ozanne, at the time the deputy secretary of the state Department of Corrections, even though his office received more than 70 letters of support for another applicant, Tim Verhoff, who was highly recommended by outgoing DA Brian Blanchard, among others. Ozanne got two letters of support, including one from Amy Smith, who was deputy secretary of the state Department of Corrections until Doyle tapped her for Dane County Circuit judge in 2009.

Smith is one of three Dane County judges appointed by Doyle who worked under him in the Attorney General’s Office. Doyle chose her even though his own selection committee had not recommended her as a finalist. The other two AG alums tapped for Dane County Circuit Court are William Hanrahan and Juan Colas.

Doyle also got a bad rap for aligning with specific constituencies and ignoring others. “He was a strong governor for the issues he wanted,” says Rick Stadelman, executive director of the Wisconsin Towns Association. “We weren’t on that agenda. We had to fight for everything we could.”

Stadelman says state aid did not keep pace with the needs of towns and counties and the transfers out of the transportation fund did not help matters. A prevailing wage law pushed by Doyle increased labor costs by up to 20 percent on local government construction projects, he adds. “That was done for the unions.”

UW-Madison political science professor Barry Burden says Doyle governed in extraordinarily difficult times and had to make hard choices.

“It’s just a tough time to be governor,” he says. “So many states are facing giant structural deficits. When you’re not taking in as much money as you’re spending, you either have to raise taxes or cut things. There’s no solution to that that’s palatable to most folks.”

Burden notes that Doyle helped bring in billions in stimulus money and got high-speed rail in gear. And he gives him props for longevity, noting that most Wisconsin governors in recent decades, other than Thompson, served one term or less.

Speaking of Thompson, former Milwaukee Mayor John Norquist says that Doyle dealt admirably with the “incredible” financial mess left by the three-and-a-half-term governor.

“Without a lot of whining or complaining, he did the best he could to clean it up,” says Norquist, a Democrat who does not automatically toe the party line. “He certainly is leaving the state from a financial standpoint — even with this recession — in better condition than he found it.”

One lobbyist and longtime Capitol observer acknowledges that Doyle’s moves to plug massive budget deficits and ride out the recession were not “elegant” or “innovative” and didn’t address basic structural problems. But Wisconsin, unlike other states, saw no widespread teacher layoffs or cuts to Medical Assistance during his tenure:  “Whatever else you think about him, Jim Doyle has kept it together.”