What is a data center? And Why Are They So Important??

Many people may have heard the term “data center” but most don’t really know what that means.

Nonetheless, these data centers are incredibly important to our everyday way of life.

What is a data center?

Data centers are these vast, huge, sometimes-sprawling buildings dedicated to holding the back-end computer systems, machinery and associated components needed to help run what we consumers know of as “the internet”. In a way these data centers are “the cloud” that we think of when we say our info is “stored out there…in the cloud”, and these data centers can use as much electricity as a small town.

What is colocation?

A colocation data center is aka a multi-tenant data center. The most commonly referenced or thought of kind of data center. A facility operated by the data center provider where companies, end-users, or tenants can leaser power and space.

Why would companies such as Apple, Microsoft and others outsource to a data center?

All the benefits of a server without having to physically host the server. All the benefits of an infrastructure without the upfront costs of physical equipment to build-out the infrastructure. Data centers offer the solutions and in scalable ways for companies. Individual consumers started flocking to “cloud” services to host them and help enable the rise of individual journalism and social media. The consumer no longer had to worry about the high barrier of upfront costs associated with the back-end stack infrastructure, outsourcing all that to focus on their main front-end projects, be it journalism or whatever. In that same way, big tech companies are finding they no longer want to or wish to deal with a physical infrastructure if they don’t have to. Data centers have shifted from being just “server farms”, to now being able to offer fully-enabled, architected and engineered hybrid digital infrastructures for companies.

Data centers offer base rates of power, space, cooling and physical security. That is the base minimum and then other services are usually offered on top and can be added in an à la carte fashion. Other than power space and cooling, managed services like firewall management may be added.

In addition to providing the physical infrastructure, data centers can be certified to meet rigorous environmental and energy management standards, another headache companies will be glad to not be hassled with. For example, some data centers tout being compliant with industry standards such as NIST 800-53/FISMA, HIPAA, PCI-DSS, and ISO 27001.

Some data center companies deal with the relevant regulatory, legislative and political hurdles that are always omnipresent yet extremely varying from different geographic regions. Data centers offer this so that their enterprise clients don’t have to bear that burden.

In addition, some high-end data centers have the expert staff of engineers, technicians and customer care specialists to offer technical and logistical support 24 hours a day, 7 days a week all working to further contribute to uptimes of 99.9999% (Six 9’s!).

Not to Mention That Network Effect! The benefits and appeal of having so many networks operating out of one data center and pooling resources is mutually beneficial all around. Take this quote from Morningstar’s website (touting Digital Realty Trust, a well known data center company) nicely summing up the power of the network effect:

Data centers that have many network providers connecting to the buildings are extremely valuable to tenants, which need to connect with networks to reach the rest of the world, and are extremely costly to replicate. Data centers that currently have dozens or hundreds of networks connecting to them are likely to remain their regions’ leaders because of the unlikelihood of having so many network providers building into new locations when an alternative already exists. These network-dense data centers are most likely to draw the cloud providers’ on-ramps, and the cloud providers then draw their customers to the same properties.

We believe the best attribute a data center can have is network density. In our view, high network density provides a huge competitive advantage because it is extremely costly and difficult to replicate. It also leads to a network effect moat source, as other network service providers and cloud providers typically need to connect with many other networks, so they are drawn to locations where many partners are present. Cloud providers in turn draw their customers, which use cloud on ramps to connect with them.[1]

via Morningstar

What is an Internet exchange point (IX or IXP)? An Internet exchange point is:

“the physical infrastructure through which Internet service providers (ISPs) and content delivery networks (CDNs) exchange Internet traffic among their networks (autonomous systems) and peer together. Typically, IXPs occupy standalone buildings with its own switches. IXPs reduce the portion of an ISP’s traffic that must be delivered via their upstream transit providers, thereby reducing the average per-bit delivery cost of their service. Furthermore, the increased number of paths available through the IXP improves routing efficiency and fault-tolerance. In addition, IXPs exhibit the characteristics of what economists call the network effect.”

via Wikipedia

What is interconnection? Why is it helpful? Data center interconnection is a model in which the assets within a multi-tenant data center are directly connected, meaning a hard-wired connection ( usually over fiber) and in a peer-to-peer fashion. An interconnect agreement is “a business contract between telecommunications organizations for the purpose of interconnecting their networks and exchanging telecommunications traffic. Interconnect agreements are found both in the public switched telephone network and the Internet.”[2].

The main advantages of direct interconnection are lower cost, lower latency (less lag) and greater bandwidth.

In the public switched telephone network, an interconnect agreement invariably involves settlement fees based on call source and destination, connection times and duration, when these fees do not cancel out between operators.

On the Internet, where the concept of a “call” is generally hard to define, settlement-free peering and Internet transit are common forms of interconnection. A contract for interconnection within the Internet is usually called a peering agreement.

Interconnect agreements are typically complex contractual agreements involving payment schemes and schedules, coordination of routing policies, acceptable use policies, traffic balancing requirements, technical standards, coordination of network operations, dispute resolution, etc. Legal and regulatory requirements are often an issue. For example, network operators may be forced by law to interconnect with their competitors. In the United States, the Telecommunications Act of 1996 mandated methods of interconnection and the compensation models for doing so.

via Wikipedia

Basically, interconnection is a process whereby telecom operators can handle calls and data for other (possibly even competitor) operators. This allows the end-users (the customers!) of both operators to communicate with each other, despite them having different networks. This is obviously a mutual benefit for all involved, including the underlying industry that all parties have an interest in seeing succeed.

What is Peering? And What is De-peering?

Anyone who was around during the early 2000s will recall the wireless wars and things like peak/off-peak, daytime/nighttime/weekend minutes, and the kicker of “in-network” calls where you could speak at a lower (or free) rate to contacts who had the same mobile network provider. Ah, memories!!

These “peering agreements” are deals betweens ISPs to create a direct link to route each other’s data packets rather than both paying a separate third-party network service provider for transport.

Interconnections are helpful for the purpose of exchanging traffic between the users of each network.

The pure definition of peering is settlement-free, also known as “bill-and-keep,” or “sender keeps all,” meaning that neither party pays the other in association with the exchange of traffic; instead, each derives and retains revenue from its own customers.

via Wikipedia

Public peering is accomplished across a Layer 2 (that’s the Data Link level layer) access technology, generally called a shared fabric [3].

At these locations, multiple carriers interconnect with one or more other carriers across a single physical port. Historically, public peering locations were known as network access points (NAPs). Today they are most often called exchange points or Internet exchanges (“IXP”). Many of the largest exchange points in the world can have hundreds of participants, and some span multiple buildings and colocation facilities across a city.  Since public peering allows networks interested in peering to interconnect with many other networks through a single port, it is often considered to offer “less capacity” than private peering, but to a larger number of networks.

via Wikipedia

By definition, peering is the voluntary and free exchange of traffic between two networks, for mutual benefit. If one or both networks believes that there is no longer a mutual benefit, they may decide to cease peering: this is known as depeering. Some of the reasons why one network may wish to depeer another include:

A desire that the other network pay settlement, either in exchange for continued peering or for transit services.

A belief that the other network is “profiting unduly” from the no-settlement interconnection.

Concern over traffic ratios, which is related to the fair sharing of cost for the interconnection. A desire to peer with the upstream transit provider of the peered network.

Abuse of the interconnection by the other party, such as pointing default or utilizing the peer for transit.

Instability of the peered network, repeated routing leaks, lack of response to network abuse issues, etc.

The inability or unwillingness of the peered network to provision additional capacity for peering.

The belief that the peered network is unduly peering with one’s customers.

Various external political factors (including personal conflicts between individuals at each network). [4]

I think of it this way…A peer is another person on my level. We’ve all had peers growing up in school and in the workplace. Well, as it turns out, large corporate companies have peers too! And in the telecommunications world a certain clique/posse/gang/group of companies called “Tier 1” providers have full access to all avenues of these internet’s either by way of their own networks or by “peering” with other providers they deem to be on their level.

When both companies agree and respect each others status, then things are all good. But. If one player (Player #1) decides that the other player (Player #2) isn’t holding up their end of the deal, or that they aren’t bringing as much to the table, then Player #1 may decide to “depeer” with Player #2.

This can have bad consequences for the innocent bystanders (the customers of both companies who are the ones that end up getting screwed).

What is a ‘fabric’ or a ‘data center fabric’?

My initial thought is that the name “Fabric” is an attempt to convey the meaning of being woven into the infrastructure and that pieces are stitched on and off (or in and out) as needed in an almost quilt-like nature. The underlying fabric is the base material that holds everything together. Just like how cotton is “The Fabric of Our Lives®” (seriously, check the domain name), so this “fabric” offers this “Fabric of Our Internets®”, if you’ll allow.

My inclination is that the word ‘fabric’ here can be used in multiple ways such as the generic version of the term and the trademarked version.

I did find a rather informative PDF about Data Center Fabric Fundamentals[5] while researching and this seems to be a great read. They offer the following definition:

A data center fabric is a system of switches and servers and the interconnections between them that can be represented as a fabric.  Because of the tightly woven connections between nodes (all devices in a fabric are referred to as nodes), data center fabrics are often perceived as complex, but actually it is the very tightness of the weave that makes the technology inherently elegant.

A data center fabric allows for a flattened architecture in which any server node can connect to any other server node, and any switch node can connect to any server node (server refers to both compute and storage).

The flattened architecture of fabrics is key to their agility. Data center fabric architectures typically use only one or two tiers of switches as opposed to data centers that implement multi-tier data center network architectures.

In almost all data center fabric architectures traffic can be transmitted between server nodes by traversing a set number of switches, which results in extreme efficiency and low latency (a few microseconds of latency at each hop can become seconds of latency per transaction).

The data center fabric architecture model unites holistically all data center resources from processor cores to memory—servers, storage, the network, and peripherals. Its architecture is designed to handle the increase in east-west traffic while maintaining north-south traffic connectivity to end users.

via Juniper Networks

Biblio:

[1] https://www.morningstar.com/articles/982137/digital-realty-is-our-new-favorite-data-center

[2] https://en.wikipedia.org/wiki/Interconnect_agreement

[3] https://en.wikipedia.org/wiki/Peering

[4] https://wikimili.com/en/Peering

[5] https://www.juniper.net/documentation/en_US/learn-about/LA_DCFabricsFundamentals.pdf