Post · bonfire.cafe

There's a lot of stuff going around about datacenters, so I decided to do a quick tour yesterday of some of the datacenters in the Salt Lake Valley. Some are indeed quite large, but there are a bunch of smaller ones too - and they are not always where you think!

All of these are publicly known, and you can find them (and ones in your own area) at https://www.datacentermap.com/ .

Let's start with a datacenter that I go by all the time! It's across the street from my grocery store in downtown #SLC. It's listed as a colocation facility; datacenters are famously secretive about who their tenants are, but we can guess that it probably hosts servers belonging to nearby businesses, especially ones that want their storage, etc. nearby, but don't want to have to maintain a secure, cooled room. Given the number of banks that have headquarters nearby, I'd bet at least some of them are customers.

This is a fairly little guy, with apparently 16k square feet of floorspace and 1.6MW of power.

A scene from downtown Salt Lake City, with a 3-story brick building in the foreground. The windows on the building are blacked out, and there are cooling units on the roof, as well as what appear to be backup power generators. In a grass strip near the building, there are many covers for accessing fiber-optic cable. A 10-story parking garage is in the background, and various cars, trees, etc. can be seen.

#slc

Rob Ricci

@ricci@discuss.systems · 19 hours ago

Next, an even smaller datacenter, that just about anyone in #SLC has seen! This is XMission, a local Internet Service Provider that's been running since 1993, so one of The Ancients in Internet time. It's on a very busy part of 4th South, and if you've been by at night, you've seen the big LED display on the front of the building that they put various animations on.

One of the things that I *think* is probably in this building is SLIX: https://slix.net/traffic/ - this is an Internet Exchange Point (IXP), where various carriers meet up to exchange traffic without it having to travel long distances. These are often run as a sort of community infrastructure - it's in the best interests of all networks involved to connect to each other so that they can do their jobs more efficiently.

SLIX is fairly small (according to their own data they carry ~200Gbps, with some spikes up to 1Tbps). There are about 40 networks that meet there: https://slix.net/participants/ . Funny story, when I first got Google Fiber at my house, I was getting routed through California to get to the University of Utah campus just a few miles away. I pinged a guy I know who pinged a guy he knows who ... learned that some of the participants in SLIX didn't have their routes set up right. A config change later, and not only me, but basically everyone on any commercial ISP in the Salt Lake Valley had much more direct routes to campus!

An aerial shot of a street in downtown Salt Lake City. In the center of the frame is a building with tan front and black windows, with large cooling units on the roof. It's surrounded by other low (2-3) story buildings, including the one - directly to the left - that houses XMission. Trees line the street, and the TRAX light rail line can be seen in the foreground.

SLIX

Salt Lake Internet eXchange

SLIX

Salt Lake Internet eXchange

Rob Ricci

@ricci@discuss.systems · 19 hours ago

This one is larger, physically (22.5 sq ft), than the first datacenter we looked it, but claims less power: 490kw. That's not a ton of power - my Chevy Bolt can draw 150kw from its batteries at max acceleration, and there are much bigger and sportier EVs that can draw something almost up in the range of this datacenter! (though only for brief periods of course! this datacenter probably draws a substantial fraction that much 24/7) Why is there so much less power for this datacenter?

Well, one of the key factors of datacenters is how power-dense they are: how much power they are designed to deliver to each rack, and how much heat they are capable of moving out.

Compute - especially GPU compute for AI - is incredibly power-dense and incredibly hot. So we can guess this datacenter is probably not for compute. If I had to guess, this is probably mainly intended as a "carrier hotel" - it's probably focused on having telecoms companies as tenants. I base this both on the lower power density, and where it is: it's near the Utah State Fairpark, which is in turn relatively close to train tracks heading both east-west and north-south. A lot of long-distance fiber in the US follows both the rail and Interstate road networks, because it's relatively straightforward to run fiber alongside transportation links. Salt Lake City lies on the west side of some of the few passes through the Rockies, so it has a ton of fiber, following I-80, the Union Pacific, etc. This is a good place for carrier hotels.

How is a carrier hotel different from an IXP? At an IXP, the carrier is just pulling in some fiber, maybe one or two routers or something. But they have a lot more equipment that they need than that - they have servers of various kinds too, plus the bigger backbone routers that fan out in many directions, etc. Mobile carriers have a fair amount of wired topology to deal with. That's the kind of stuff they put in carrier hotels, and this is a good spot for them.

I picked this shot because, in the background, you can see the Gadsby Power Plant, one of the main sources for power in #SLC. That's a natural gas plant that generates about 300MW. Put a pin in that number, we'll come back to it later.

In the foreground, a two-story white building houses a datacenter. There are some coolers on the roof, but not a lot. In the distance is the Gadsen Power Station; the most prominent factor being three smokestacks. The actual gas turbines are behind the stacks.

Rob Ricci

@ricci@discuss.systems · 19 hours ago

Now we're getting a bit bigger, and also more residential. This one sits on the edge of a residential neighborhood, on 200 E, in Milcreek. This is a 36k sq ft, 1.9 MW facility. What's in there? I don't know, as mentioned above, datacenters don't tend to tell you who their tenants are. There's probably some reasonable computing power in there, but it's probably not dense enough to be very GPU-heavy.

The sounds of the HVAC systems were quite noticeable at this one. Any time you are dealing with electricity, you are also dealing with heat. In a datacenter, the power drawn by the compute and network equipment gets turned into heat, and you need to get rid of it. Of course, you want to spend as little electricity getting rid of heat as you can. Datacenters call this "Power Usage Effectiveness", commonly called PUE. A PUE of 1.5 means that for every KW that goes to computers, .5 KW goes to other stuff - mostly cooling, but also heat losses, lighting, etc. A 1.5 PUE is pretty good, supposedly some of the biggest datacenters have PUE of around 1.1 .

This actually highlights one way in which having a fairly large-scale datacenter is efficient: putting all the computers in one place does enable you to use cooling systems that get rid of more heat for less power. Of course, how many computers you have, where your power is coming from, what mechanisms you use to cool them, etc. matters too! Again, we'll get back to that later.

By the way, my guess would be that only the building in the front is a datacenter - the building in the back has too many truck bays and not enough cooling. It's probably a small warehouse of some sort.

An aerial shot of a long white two-story building. It's in a mostly residential neighborhood, with houses and trees adjacent. Behind it sits a similarly sized buildings that has several trucks parked at bays.

Rob Ricci

@ricci@discuss.systems · 19 hours ago

This is the biggest datacenter I visited - campus, actually. This facility is in West Jordan, near the South Valley Regional Airport. It's big enough that I have to post several pictures to get you a real sense of the size (but it's not the biggest datacenter in Utah.)

What you're looking at here is three buildings that, together, have a power capacity that's reported (depending on the source) to be around 160 MW (put a pin in that number too.)

Two of these buildings are multi-tenant (the ones with the flat white roofs) like the others we've seen.

That third one in the back, with all of the cooling towers on top, has supposedly been built for a single hyperscaler, and is supposedly something like an 80-100MW building. Which hyperscaler? That information is not public. That's a whole lot of cooling on the roof (which is reported to be water-free), so my money would be on this being an AI data center.

In these pictures, you can see more electrical infrastructure. Bringing that much power into one place takes a lot of wires.

3 media

A wide shot of a datacenter campus; three large buildings can be seen. The one on the left has a flat roof and is surrounded by cooling and generation equipment. The one near the center looks similar. The one to the right is twice as tall as the others, and has its equipment mounted on the roof. Power lines, transformers, etc can be seen on the right and in the foreground. In the background, the Wasatch Mountains still have some snow.

A different angel of the same campus, showing more of the surroundings; we can see a small canal to the right, and various tree-lined neighborhoods in the background.

The tallest of the buildings on the campus - the one with cooling units mounted on the roof - dominates the picture. In the foreground, is a standard suburban neighborhood, with detached single family houses and trees.

1+ more replies