Key Take-Aways of Colocation Research for Silicon Valley Power

Last fall EMI, under subcontract to Summit Blue Consulting (now a part of Navigant Consulting), performed a process evaluation for Silicon Valley Power (SVP) in Santa Clara California.  The process evaluation focused on identifying barriers to colocation data centers (or “colos”) participating in SVP’s energy efficiency programs.  Colos provide data center infrastructure to support other companies’ IT equipment – basically data center space for rent.  This space can be leased in units as small as half a rack and as big as thousands of square feet.  The latter situation, where whole rooms or even whole data centers are leased to one tenant, are known as “wholesale” colocation facilities.

Santa Clara has a dense concentration of data centers and colocation facilities.  This is largely due to their location in the heart of Silicon Valley, but also is due to their relatively low cost of power and high reliability of their power delivery. As a result of this high concentration, data centers are a main focus of the SVP energy efficiency programs, and in 2007-2008, roughly 60% of SVP’s energy savings came from data center related projects.

EMI conducted this research by performing: online research and a document review of appropriate reports, and in-depth interviews with colo managers from inside and outside SVP’s service territory and other industry experts.  Here are some of the main take-aways from EMI’s research into participation of colocation facilities in SVP’s programs.

• Aggravated Barriers – Since providing reliable data center space is a colo’s main business, some of the barriers to energy efficiency for typical data centers are aggravated in the colocation facilities’ case.  These include an extreme focus on reliability, with little interest in energy efficiency.  It also creates an extreme case of the split incentive, because the people paying the power bill and the IT purchasers work for completely different companies, so there is little motivation to invest in more efficient equipment.
• Pricing Models of Colos Affect Investment in Efficiency– Different colocation companies have different methods for splitting up charges for power, cooling and space.  While data centers become more constrained by power and cooling (and less constrained by space), some colocation facilities are moving away from space-based charges and more towards charging directly for power and cooling, which helps create more of an incentive for their customers to save energy.
• Difficulty in Reaching the Colocation Facilities’ Customers –SVP, like other utilities, is restricted to only offer incentives to their customers of record.  This is necessary as it allows SVP to be able to recoup the incentive amounts if the measure does not stay in place for the contracted five-year period, and therefore the measure does not deliver the full five years of savings SVP claims for the measure.  As a result, SVP cannot give incentives to the colocation facilities’ customers, as these customers do not pay SVP for their power, but instead the cost of power is bundled with their colo charges.  This is a major barrier for many utilities to getting colocation customers to participate in virtualization incentives, for example.
• Lack of Expertise for Completing Calculations – Some facilities indicated that they did not have the engineering expertise on staff to complete the necessary calculations to receive incentives, as operating colos are basically “a couple IT guys with a sales department.” SVP offers support to fill out the applications, but some potential participants were not aware of this, so this was an area where better communication of the program offerings could help increase participation from companies that need this support.

All in all, the evaluation found that SVP’s focus on data centers has been very successful and that they are undertaking a lot of efforts to help overcome these barriers such as: emphasizing new construction where the barriers and inertia to energy efficiency are not as great, and offering technical support where it is needed.  Other opportunities lie in collaborating with other utilities to identify new opportunities (e.g., for prescriptive measures which simplify the application process), and investigating new ways to get to colo customers.  Although there are many barriers in place for colocation facilities, this is a large data center market, it is growing rapidly, and it is worth having progressive utilities like SVP continue to push to develop programs and processes to overcome these barriers.

The full report (available here) offers more detail on the colocation market and barriers to their participation in energy efficiency programs.

How big can data centers be? How about 19 football fields?

Today I realized that my understanding of how large data centers can get was significantly understated.  This realization came as I reviewed Data Center Knowledge’s special report on the world’s largest data centers.  I have previously used #5, Microsoft’s Chicago data center, as an example of one of the largest, but was shocked to realize that the largest is almost 60% bigger.  Another interesting result is that seven of the top ten are colocation facilities.  This is significant because it is often difficult to get colocation facilities to engage in energy efficiency programs, especially after they’re operational. The other three – and the only corporate data centers in the top 10 – are all Microsoft facilities. A number of the facilities (including the largest) are also buildings converted to data centers from other uses.  Since these are not purpose-built data centers, my guess would be that they are probably not ideally designed in terms of efficiency.

I’m also disappointed to see that relatively few have energy use or even power capacity listed.  In an environment where power is starting to dominate as the primary constraint on data center growth, wouldn’t it make sense to track a list of the largest data centers in terms of energy use?

Here are some other highlights from the report:

#10. SuperNAP (407,000 SF) – Number ten is notable mostly for its power consumption.  At 250 MW capacity it boasts densities of up to 1,500 w/SF, made possible through advanced cooling using “a high-density T-SCIF (Thermal Separate Compartment in Facility) containment system to fully separate the hot and cold aisles.”

#7. i/o Data Centers Phoenix ONE (538,000 SF) – This one just seems to keep popping up, with “enormous rooftop array of solar panels that will eventually generate as much as 4.5 megawatts of power for the data center, and a thermal storage system that will allow i/o Data Centers to run chillers for its cooling systems at night when power rates are lower.”

#6. Microsoft’s Dublin Data Center (550,000 SF) – This one operates 100% of the time on outside air through the use of economizers and “Microsoft says it can run its server rooms at temperatures of up to 95 degrees F (35 degrees Celsius),” which should give it an efficiency advantage.

#5. Microsoft Chicago Data Center (700,000 SF) – A large portion of this data center consists of double-stacked 40-foot shipping containers that are each filled with up to 2,000 servers.  Containers make the system highly scalable and efficient.

#1. Digital Realty Trust Lakeside Technology Center (1.1 M SF, 100+ MW of power) – In Chicago, this data center used to house the printing presses for the Yellow Book and the Sears Catalog. It was converted to telecom use in 1999 and is now 2^nd largest power customer for Commonwealth Edison.

Some people might wonder why a bohemoth such as Google doesn’t show up on this list? Well, it seems that Google likes to focus on many data centers together on a campus, while Microsoft tends to go big, and the report only looks at individual buildings not campuses.

So how big are these?  Let’s put it in perspective:

1.1  million square feet is equivalent to just over 19 football fields

250 MW is equivalent to the average power use of about 200,000 American homes

These numbers really speak to the massive amount of computing needed in modern society, but this is actually not where the majority of energy use from data centers come from.  According to the US EPA’s 2007 report to Congress, only 38% of data center energy use in the US comes from “enterprise-class” data centers of greater than 5,000 SF.  The remaining 62% is used in the smaller data centers, which means that these smaller data centers offer the largest overall chance for energy savings in this industry.

Thoughts from PG&E’s Former Data Center Efficiency Program Manager

I’m getting caught up on some news from over the holidays (happy new year by the way!) and came across this two part interview with Mark Bramfitt, the former program manager for PG&E’s High Tech program which includes data center efficiency projects (found here: Part 1, Part 2).  Mark has been very vocal over the years in spreading information on PG&E’s ground breaking programs in this area and has been a great ambassador to the high tech and utility industries alike on data center utility incentives.

EMI performed the process evaluation of the PG&E’s High Tech Program (which can be found here) and I got to know Mark initially through his support for my work with the EPA on the ENERGY STAR Computer Server specifications.

Scouring through the two interviews I found a number of interesting points from Mark.

On Barriers to Program Adoption:

In my discussions with utilities across the U.S., this is probably the single biggest barrier to program adoption – they can’t find firms who can do the calculations, or resources to appropriately evaluate and review them.

What has slowed us down, I think, is that the IT industry and IT managers had essentially no experience with utility efficiency programs three years ago. It simply has taken us far longer than we anticipated to get the utility partnership message out there to the IT community.

These two quotes emphasize the fact that there’s a gap of knowledge (and talent) between the utility industry and the high tech companies that equip and run data centers.  On the utility front – there is a gap in knowledge about the IT industry.  The fast pace of technological innovation and quick growth in this industry presents challenges in finding or developing the expertise to implement effective programs (including performing the necessary calculations and analysis).  On the high tech company front – there is a gap in knowledge about how to identify and leverage these new programs and efficiently perform the analysis and calculations to receive the incentives.  My conversations with industry members on both sides highlight these frustrations, and the ultimate success of data center efficiency programs will hinge on closing these gaps in the coming years.

New Opportunities

On the retrofit side, we’re seeing interest in air flow management measures as the hot spot, perhaps because customers are getting the message that the returns are great, and it is an easy way to extend the life and capacity of existing facilities.

Metering and monitoring systems lead people to make simple changes, and can directly measure energy savings in support of utility incentive programs. We also like that some systems are moving beyond just measurement into control of facility and IT equipment, and to the extent that they can do so, we can provide incentive funding to support implementation.

There is a lot of room for potential growth from the basic programs currently offered by utilities.  Mark points out one of the areas of low hanging fruit is with air flow management.  This can include simple efforts like blanking panels or more advanced efforts like switching to hot/cold aisle containment.  The challenges here are in confirming the energy savings, which is where the expertise mentioned above is needed.

Metering and monitoring is always a hot topic of conversation in this industry.  Future and current efforts for quantifying energy efficiency gains rely on access to quality data, so the implementation of better measurement and monitoring would be a big a boost to future energy efficiency projects. The challenge, again is quantifying concrete savings from these measures.

These two points bring up what I see as a main challenge to the utility industry in simply and reliably quantifying the energy savings from these measures so they can use incentives to drive these right behaviors.  It’s a significant challenge and will take some creative thinking. Another of Mark’s points emphasizes the potential outcome of these challenges:

That being said, utilities in California are under tremendous pressure to deliver energy efficiency as cost effectively as possible, so some of the industry leadership activities undertaken by PG&E may have to be de-emphasized, and we may not be able to afford to develop new programs and services if they won’t deliver savings.

To get over this hurdle, the industry needs to think creatively on how to efficiently (i.e. cost effectively) justify incentives and programs that help drive the right behavior.  There’s a large potential for energy savings in this industry, and utilities should have a role in driving these behaviors, we just have to continue to push to find effective models for doing this.

Future Growth of the Industry

PG&E is not seeing the level of new data center construction that we had in ’07 and ’08, but the collocation community tells me demand is exceeding supply by 3-to-1. They just can’t get financing to build new facilities.

This last point emphasizes that despite the credit crunch, demand remains high.  Once some capital is freed up we should continue to see rapid growth of the industry and increased opportunities to effect change through intelligent incentives.