System Failure: “Technology Mayor” Ed Lee’s City “Stuck in 1999”

Written by Chris Roberts. Posted in News, Politics

Tagged: , , , , ,

Published on April 04, 2012 with 4 Comments

The City That Knows How to Praise Innovation But Slow to Replace Ancient Software, Organize Data, Reform Municipal IT

By Chris Roberts

April 2, 2012

Technology is never far from Mayor Ed Lee’s mind or lips. With good reason: piles of cash from Silicon Valley luminaries representing Google, Facebook, and others compelled celebrities like MC Hammer and Brian Wilson to dance for the heretofore obscure former City Administrator and propelled Lee, the onetime interim mayor, to election to a full term as San Francisco’s chief executive last November.

Following his lavishly appointed and well-attended victory party at the Palace Hotel, Lee was quick to signal his appreciation for his friends in the technology sector by gifting Twitter, Inc. a generous tax break to persuade the estimated $8 billion company to not leave the city that “gets it done.”

Board President David Chiu.

Popular tech buzzwords like “entrepreneurship” and “innovation” now spice up Lee’s speech, the latter of which the mayor chose as the subject for his inaugural column for the San Francisco Examiner.

“Innovation is the driver for job creation in San Francisco,” wrote Lee, the mayor of the self-declared “innovation capital of the world,” where 1,500 tech firms employ 30,000 people, just the tip of a technology peninsula that spans from North Point to San Jose.

But there’s a problem for Lee and his image as an innovative tech mayor, one that he inherited from former Mayor Gavin Newsom – the city’s antiquated information technology systems.

“We’re in 2012,” said Board of Supervisors President David Chiu, who sits on the city’s Committee on Information Technology (COIT), a confederation of department heads and bureaucrats tasked with overhauling the city’s IT infrastructure and systems, “but our city’s information technology is stuck in 1999.”

All Gloss and Glamour

There’s no doubt San Francisco’s government has embraced technology. “Twitter Prince” Newsom was one of the social media Web site’s very first celebrities, using the-then startup to announce his aborted bid for California governor (so total was Newsom’s devotion to Twitter that he showed off the company’s Folsom Street HQ to Russian President Dmitry Medvedev during the foreign leader’s 2010 visit to San Francisco).

Newsom insisted city departments become active on social media. Citizens can tweet to DPW when their driveways are blocked or when trash is not picked up, and can similarly publicly query 311 via Twitter – and Newsom used his clout to pass some of the nation’s first open data legislation (though, years on, city departments are asked, not required, to share data with the public).

Chief Innovation Officer Jay Nath.

Under Lee, the city created what it says is the nation’s first “chief innovation officer” position, filled by Jay Nath (a position since copied by other cities), and made much of partnerships with the private sector, intended to get apps created and mine the private sector for ideas meant to “make city government better, more responsive, efficient and connected.”

It is no small wonder Lee looked to the private sector to upgrade San Francisco city government. If San Francisco is a broadband-equipped new Macbook, its city government is trying to get by with a 486 processor-equipped computer using dialup. That is, if they had 486s or dialup modems twenty-eight years ago, which is when some of the software currently in use by city departments was developed and installed.

“Gavin Newsom used to say that he was the Facebook and the Twitter mayor. Ed Lee likes to say he’s the innovation mayor,” said a high-placed city source, speaking candidly on condition of anonymity. “The fact of the matter is, the focus is on the glossy and glitzy aspect of what tech could be. Meanwhile, our IT infrastructure is a mess, and nobody’s focused on it.”

The Mess

The city’s technology infrastructure has long lagged behind the private sector in efficiency, consolidation, and even basic equipment. The San Francisco Police Department made for headline-friendly outrage when it announced that all officers had finally gained access to email – in the summer of 2011. Other reports have detailed the 15-year odyssey of JUSTIS, the still-not-complete technology overhaul of the city’s criminal justice departments, a failure that has been blamed for at least one death.

In 2012, San Francisco city employees use human resources and ledger software developed in the mid-1980s, which the public cannot access, to ascertain line-item budget information; a mishmash of data centers, servers, and computers stuffed in closets house data from payroll information to health records to business and building permits (part of which has only in the last month been backed up to the cloud, a basic first step in Silicon Valley firms), and multiple servers house city workers’ email accounts, all of which were identified as in need of upgrades in city Budget Analyst and Board of Supervisors reports five years ago.

The city also overpays for basic software, with multiple departments paying licensing fees to the same software companies. If the city’s software licenses were consolidated, it is estimated the city could save $50 million – a good chunk of its current $165 million budget deficit.

This makes conducting simple city business more challenging and more expensive, but it also makes city government less open.

One of Lee’s challengers in the November 2011 election, venture capitalist Joanna Rees, tried to make issue of bloated city salaries and pension benefits on the campaign trail. She requested the Mayor’s Office supply line-item budget information, detailing employees’ salaries down to the dollar (as would be available from any of the technology startups Lee’s government swoons over) but she was told no such information was available, not with the way the city’s antiquated human resources system – called FAMIS – stores data.

No One In Charge

Acting Chief Information Officer Jon Walton.

Lee shares at least part of the blame for the city’s state of IT affairs: the city’s Chief Information Officer reports to the City Administrator, Lee’s old job. Former Chief Information Officer Chris Vein oversaw the release of report after report from the City Controller to the Budget Analyst detailing municipal IT’s shortcomings before he left for a job in government in Washington, DC. But in a way, neither Lee nor Vein, nor Acting Chief Information Officer Jon Walton are to blame – because no one person is in charge.

This is common among cities, according to Walton. “Large cities are faced with the common challenge of balancing centralized IT services with decentralized department IT services,” he told Fog City Journal. This means Walton must often rely on departments’ individual goodwill to implement reforms or practices recommended, and accountability for deadlines missed or budgets blown is hard, if not impossible to track. Walton chairs a committee called COIT (Committee on Information Technology), which last year issued the city’s first unified citywide IT plan.

In the Five Year Technology plan released last year, COIT identified ten possible tech upgrades, including a citywide security plan, a citywide dark fiber network, and replacing the telephone system with VOIP (voice over internet protocol) technology. Some of the upgrades mentioned, like a citywide enterprise software agreement – i.e., one license for word processing and data programs instead of multiple licenses – were highlighted in the 2007 report, and have yet to be implemented (translation from tech-speak: five years on, and the city’s still wasting money on multiple licenses for software).

Three major technology upgrades – merging the city’s email platforms, replacing the decades-old HR software, and consolidating and safeguarding the city’s data – are underway. Each has been tasked to a different city department: the Department of Technology, the Controller, and chief innovation officer Nath, respectively.


It’s hard in 2012 to get excited over something as basic as email, yet it’s email consolidation that will be one of the city’s biggest IT accomplishments this year.

Years ago, multiple city reports addressed the need for a consolidated email platform, with some city users on LotusNotes, and others on Microsoft Exchange. Today, “I still cannot easily e-mail a city employee,” said Chiu, who named seven different city email servers.

This is at last changing. The city’s email consolidation project is “well underway,” Walton wrote in an email, with about 1,000 city users on a consolidated Microsoft Exchange platform. That leaves about 20,000 users left to go, however, according to the most recent information provided by Walton in response to an email request. Some city departments have yet to be scheduled for a “go-live date,” and no users were able to import their contacts, attachments, or calendaring, according to a city report.


Close to one-third of San Francisico’s $6.8 billion budget goes to city employees. The software used to manage the city’s 22,000 employees is 15 years old, a spring chicken compared to the software used to watch over the city payroll, which is 28 years old, according to the Controller’s Office, which is tasked with the $45 million consolidation/replacement of the ancient software, an undertaking dubbed “E-Merge.”

Deputy Controller Monique Zmuda.

Replacing the vintage computer systems has thus far taken four years, according to Deputy Controller Monique Zmuda, in part because of software customization by city programmers. The new system will manage “all human resources functions, payroll, benefits administration, and time reporting,” Zmuda said.

Still to be done is the replacement of FAMIS, the city’s financial accounting system that is entering its 25th year of service. The cost of this project is unknown as well as the total scope, in part because “many departments utilize their own small sub-systems to compliment the city’s central accounting system,” Zmuda wrote.

Solving that lack of centralization and putting the city’s data under fewer roofs has been tasked to Nath.

There’s not much room for public visibility of city data, yet. The Controller plans to make available to the public more budgetary data, but to do so in a way that’s “understandable and readable” is tricky, Zmuda said. “The trick is to modify the old system to enable the data to be reformatted and re-sorted.”

Efforts to put the city’s budgeting data in a publicly-available portal were initially dismissed because it would be “difficult,” according to a city source. Difficult, and more under the purview of the city’s data initiative, which is asking – politely – for city departments to make data available to the public on a Web site.


Lee and Nath announced in early March that the city had moved some of its data from local servers and hard drives to the cloud. The city’s open data is also the envy of the rest of the world, Nath told Fog City Journal. With “200 datasets” online, and over “60 apps” developed, “we’ve been an international leader on open data,” Nath wrote via e-mail.

Yet private companies would still find much lacking in the way the city houses its data, in a mélange of data centers, data closets and stand-alone servers. A corporation or another comparable organization should have “two, at most maybe three” data centers, according to a high-placed city source. “Instead, we have three dozen.”

San Francisco is budgeted to spend $7 million over the next two years to shore up this situation, with “900+” data locations consolidated into two servers at 200 Paul Avenue, a privately-owned data center in the city’s Bayview District, according to Walton, with another city data center located at the airport. The Bayview data center is scheduled to be built by November, with city departments to move their IT servers to the data center in about a year from now, according to documents provided by Walton.

The city’s data is still susceptible to a major disaster. In the event of a major earthquake, for example, city workers would be unlikely to have access to email – and there are no plans to consolidate the city’s data into a user-friendly “data warehouse” akin to what was done at the federal level with Data.Gov, according to Nath. “It’s proved expensive [for other cities],” he said, “and doesn’t really bring us to where we need to be.”

What is available on are datasets in the form of spreadsheets listing all licensed San Francisco businesses, for example. While it’s perhaps not likely a Microsoft Excel file would excite a Facebook executive at an innovation meeting, this is a big step for “transparency” in local government, Nath said. “Just because it’s not the most cutting edge approach, doesn’t diminish the inherent value of the information.”

“There is no simple way to just push all data to the public – this takes time,” said Nath, who told a reporter, “If you have ideas on strengthening legislation, I’m open to hearing it.”


Calls to centralize the city’s IT systems were made as long ago as 2001. “Currently there is no formal, or dotted line reporting relationship between the City CIO and the various department CIO’s or technology staff,” the Department of Technology wrote in 2007. This remains true today.

No one person or department is in charge of the city’s necessary IT upgrades – many city departments have their own CIOs and own IT staff. This arrangement not only duplicates roles and personnel, it makes citywide reform very hard to implement. The city’s IT is run like a “confederation,” according to one insider, and ostensible leaders like Walton must depend on department’s good will in order to ensure their cooperation.

The city has a Department of Technology, but it has no sway over many of the city’s IT workers, decisions, and infrastructure. That’s left to individual city departments’ information officers and IT staff. No one person has control over the one main command-making body, COIT, which meets once a month for about an hour and a half, according to records.

COIT’s main function is to provide “policy and guidance,” and most city tech projects do go to COIT for approval, said Ron Vinson, who heads up the Department of Technology. Often COIT provides little more than a rubber stamp, said one former city employee.

Information technology does not appear to be a priority, at least from a budgeting perspective. Of the city’s $6.8 billion budget, a little more than $40 million was dedicated to new IT projects, such as upgrading the city’s 911 dispatch center to the latest software ($1.2 million alone).

Meanwhile, the city’s overall IT budget has shrunk. The city spent $173 million on information technology between 2006-2007, according to a 2007 Budget Analyst report.  In the current fiscal year, the city is spending $160 million, according to Walton, whose own position as Chief Information Officer means he chairs COIT, but does not hold sway over the Department of Technology. Walton is also still “acting” Chief Information Officer (newly-minted City Administrator Naomi Kelly is tasked with making a permanent hire; her office did not respond to telephone calls seeking comment).

Insiders have before called for COIT’s dissolution, to get the city to move more quickly and efficiently, to get San Francisco to “take a forward-thinking approach… responsive to the ever changing industry of information technology.” In other words, “innovation.”

Those words are lifted directly from the 2007 Budget Analyst’s IT report which suggested reforms still waiting to be implemented. In the meantime, without a true structural shakeup or reform, it’s proven difficult and slow to innovate a city IT bureaucracy the size of San Francisco’s.

“At the end of the day, someone at the top needs to make sure these projects get done, no excuses or this bureaucratic stuff,” one insider said. “People are going to help, and if they can’t, they ought to be fired or be demoted.” That, at least by today’s standards, would be innovative.

Chris Roberts

Just trying to make a little trouble.

More Posts


Comments for System Failure: “Technology Mayor” Ed Lee’s City “Stuck in 1999” are now closed.

  1. Someone just pointed out this article. It looks like you got your information spoon-fed from DTIS (actually DT). It seems like this is a long reply, but I am just given a quick overview.
    – Chris Vein is still on the City’s HR system and payroll, it may be DTIS. He’s not gone yet.
    – Jay Nath – His resume is underwhelming. Basically a low level programmer that somehow has wormed up in the hierarchy. Unfortunately, the most qualified of the bunch you mentioned.
    – Jon Walton is acting CIO because he does not live in the City. All city department heads must live in SF.
    Under the Sunshine ordinance, all information (unless specifically excuded) is avaialble to the public with a request. By the way, he got canned at his last job before he started with the City. Something about making commitment to customers that were never fulfilled.
    – The Mess – Actually JUSTIS has been operational for 2 years. It is a very large, very well thought-out, state of the art project. Walton bungled the management of this project so badly he was brought back under the Mayor’s office. The major obstacle is the deparments (courts, police, City attorney, etc…) get to say when they actually get on JUSTIS. You should contact the mayor’s office and interview the team. They really have a very good plan and can explain why it’s a waiting game to complete.
    – Email has been a fiasco so far, very small departments have been brought in. The project actually is to take the mail out of the City and put it into Microsoft’s cloud. Unfortunately, no one has seem to considered that the same issues regarding security with LA’s move to gmail are the same as this Microsoft move, that is the Police cannot use it. The FBI expressly forbids cloud usage for agencies that use their system.
    – E-merge has not been in process for 4 years, it’s overdue by 4 years. The orginal replacement project started sometime in 2002. The budget for this has kept doubling. Another slow-rolling disaster with immense overruns. Shelly Thompson was running this. Failed in Florida just to get one small part running and somehow got the job to implement the new Peoplesoft system.
    – As far as FAMIS goes, it’s a fairly complicated public accounting system that was already dated technology when it was replaced/upgraded in the late 90’s. Jay is way over his head on this one.
    – Data – around 5 years ago DTIS pushed the decentralization of data. They forced the departments to take their applications, people and hardware in once was a centralized center (contact retirement, assessor, property tax, etc….). This “consolidation” is not. It is an attempt by DTIS to justify it’s existance. Why would anyone be so stupid as to put a critical data center on a airport on landfill?
    You got the data centers mixed up, Paul street is already built, the airport site is new construction.
    The question you are not asking is why is the city building this at all? Why not go to the “cloud” like the Federal government does?
    – Data – This is complete mis-information. The data is available, however the entire top layer is full of politcal appointees that have no understanding, or deliberately lie about the availability. FAMIS data (accounting), ADPICS (procurement), HR (employment, at least the number and type), payroll, assesor, parking ticket, etc… is all accessible. It’s alot easier to lie, get more funding and expand than it is to provide information.
    DTIS itself has a wealth of information they won’t give up. Very good example, all phone information is in two system, Telus and Teleweb (If I have the spelling correct). All phone records of all calls made is incredibly easy to access.
    Again, Jay Nath may be glib, but he really has no undetstanding of the systems or information the City has.
    – Leadership. Let us also add expertise. The top layer of DTIS has been fully politized. There are a number of extremely dedicated employees at DTIS that are incredibly knowledgeable about how the systems work. However they are not running the show and have not input into any changes. There were some very effective leaders, one I can think of in particular who was highly qulified and instrumental in information consolidation at DTIS who was “downsized two years ago..

    Also, reading the comments:
    – whether information is spread on multiple sites or a single site, it doesn’t matter. It’s understanding how to extract the information. That’s how all phone apps run, taking data from multiple sources without concern of location. The consolidation efforts will cost the city more than it spends on the separate sites now.
    – Ron Vinson was working as a spokeman  for the Housing Authority when he got moved to DTIS 10 (?) years ago. Willie Brown made Chris Vein put Ron in as a high level IT manager without any qualifications, making twice the amount he made before. Eventually they created the PR position he holds now.
    Another tidbit. Cisco exerts an amazing undue influence in the City. John Chambers has personally put pressure on DTIS and the mayor. This has warped decsions in order to purchase only Cisco products (very, very expensive) and services instead of trying out alternatives or bidding out. See the Airport consolidation and what equipment they will be using.

    I can comment in more detail if you like. 

  2. Hi Parker —

    Ron Vinson is DTIS’s director of media. Jon Walton is the acting Chief Information Officer, chair of COIT, and head of DTIS. Jay Nath is the Chief Innovation Officer and works in the Mayor’s Office. That is an error, thanks for pointing it out, but the main idea that the city’s CIO isn’t directly in charge of the city’s IT is unchanged.

    Everything else in the article is cited, with either information from city Web sites or from city employees. Is this data inaccurate? If so, please edify us and provide the correct information.

    Currently, the city’s data is spread throughout the city and not located in any city data centers. Think that’s the issue. There *will* be two data centers — at SFO and in the Bayview — as reflected in the article.

    And thanks for the career advice.

  3. What a mess! This article is a mess; riddled with inaccuracies and mixes up the jobs of Walton, Vinson, and Nath. I offer you a quiz: Where is the city building a data center? Bayview or the Airport? Wrong, the airport. Does Walton have “sway” over the dept. of technology. Yes, he is the head of that dept. Is Nath in charge of “consolidating and safeguarding the city’s data”? No, he has nothing to do with the project; he’s busy making Lee look good.

    Stick to writing about Marijuana. Technology is not your strong suit.

  4. Funny that you mention Ron Vinson. He was strategically emplaced at the DTIS more than a decade ago to ensure that Willie Brown’s voice would be heard, both in directing contracts to the “right” recipients as well as helping ensure that executive power was conserved via keeping info systems ancient, inoperable and closed.

    The SFPD, for instance, is as willing to share their data on citation and enforcement patterns as it is willing to do its job to keep us safe on the streets according to objective evidence of what is endangering us. The SFPD keeps much of its data on index cards and is notoriously slow and resistant to responding to sunshine requests by attorneys seeking background for lawsuits.

    As far as keeping up with evolving standards, the private sector can do this because it tends to have an ample flow of cash for those purposes and there are competitive advantages and tax incentives for spending what it takes. The technology evolution cycle is so short and the time required for the City to process adapting to change so long, that the City might be able to adapt once per five cycles of change in the real world on a good day.

    I do not know that there is a good solution to this in the public sector and there is no advantage to the public sector outsourcing this to the private sector unless there is some supply of capital infusion that I’m not seeing. We’ve seen how the public sector has gotten bit before by trying to play the innovative financial markets, elsewhere on rate swaps and locally on the Breda car sale/lease/tax deals.

    In short, DIT plays the same role with public data as the Ethics Commission plays with public integrity, responding to the needs of entrenched executive power to maintain the game of ‘keep away’ of the public sector from voters, citizens, taxpayers and residents. Vinson should be let go before his pension kicks in and replaced an apolitical professional.