Welcome!

Related Topics: SDN Journal, Java IoT, Linux Containers, Containers Expo Blog

SDN Journal: Blog Post

There Is Cost Per Port and Then There Is Cost Per Port…

I let you do the math on the cost of that infrastructure. And the installation. And the maintenance

Earlier this week, Ethan Banks wrote a very nice article about Mellanox’s dual spine and leaf network in support of a large amount 10GbE access ports. After describing the scaled up network design, he reviews 8 observations about the design, not to point out good or bad, but merely to point out specific points to consider. Fully coincidental (Ethan lives close to us, but I am pretty sure he cannot peek through our windows) we had gone through a similar exercise this week, documenting the choices and limitations of spine and leaf networks. And as always, the conclusions are not ones of right or wrong, more of awareness of choices and consequences.

The Mellanox design Ethan describes employes an extra spine layer, we have seen and heard the same from Arista and others, some calling it a spine-spine or similar. Nitpicking perhaps, but adding a spine layer to a spine and leaf network and still calling it a spine and leaf network is like adding a docking station, screen and keyboard to a phone and still calling it a phone. It’s a computer that can make calls. And a spine and leaf network with an extra spine is a fat tree.

Ethan points out in his first few points that the sheer amount of cables and optics is astonishing. Let me try and put some numbers around that statement. If you build an approximate 3:1 oversubscribed spine and leaf network out of generic switches, I would probably use a Trident2 based ToR switch with 48 SFP+/10GbE ports and 6 QSFP/40GbE ports. The 48 ports should serve most rack deployments with single homed servers, only really dense or heavily multi homed servers would need more ports and I will use 4 of the QSFP ports to connect to my spine. That leaves 2 QSFPs on each leaf to be used for extra access ports, at the cost of some oversubscription. Or if I wanted I could use these as extra spine connections, lowering my oversubscription to 2:1. As a spine, and staying away from chassis based systems, I would pick a 32xQSFP/40GbE switch.

The largest spine and leaf I could build out of this combination is one that contains 16 spine switches and 96 leaf switches. In a spine and leaf I need to connect each leaf to each spine and I have the equivalent of 16x10GbE to use out of each leaf, which I can connect to at best 16 spines. With each spine receiving 1x10GbE from each leaf, I can build out to 96 leafs to fill out my 32xQSFP or 96x10GbE equivalent spine switch capacity. 96 Leaf switches give me 4608 10GbE access ports at 3:1 oversubscription, 5376 at a slightly worse oversubscription if I use the extra QSFP ports on my leaf switches. To support these 5376 access ports, I need 3584 10GbE fabric ports: 16 each from 96 leaf switches plus 32×4 from each spine. That means 1792 switch interconnect cables. And 3584 10GbE short reach optics, because the vast majority of connections between spines and leafs is likely to be at a distance that DAC cables cannot cover (not considering the fact that most high density 10GbE switches are designed without PHYs, limiting the use of passive cables to usually 5m or less).

I let you do the math on the cost of that infrastructure. And the installation. And the maintenance. And the sheer complexity of running a 10GbE from each leaf to each spine. If you want to reduce some of this complexity, you can switch to using 40GbE/QSFPs between the leafs and spines, but by doing so you have reduced the maximum size of the network. Each leaf will now contribute 1 QSFP worth of interconnect to each spine, to a 32xQSFP spine can only support 32 leaf switches, or 1536 10GbE access ports (1792 if you use the 2 extra leaf QSFP ports). And you probably noticed that I left no room on the spine switches to actually leave this spine and leaf network to the rest of the network infrastructure. Taking a few QSFPs for that will reduce the size of the network more.

Even in the spine, spine and leaf network (or spine, leaf and ToR in Mellanox terminology) the amount of cabling between the ToR (someone explain why that is not the leaf of the network?) and the aggregation leaf may be reduced, the cabling between their aggregation leaf and spine still follows the same model as above.

We often focus on the cost of switches and the derived cost per port. Of course the cost per port is important, but don’t fool yourself by taking the cost of a switch and dividing it by the number of ports it supports. It makes for great quotes in press releases, but the actual cost for that port is way higher the moment you add in the overhead required to connect that port to the rest of the network. And for spine and leaf networks, the cost of the rest of the network has a very large portion of cables and optics. Even for reasonably priced optics, that piece of the infrastructure is likely going to cost more than the cost of the spine switches together. Even when you create one of these multispined animals…

[Today's fun fact: The human spine contains 120 muscles and approximately 220 individual ligaments.]

The post There is cost per port and then there is cost per port… appeared first on Plexxi.

Read the original blog entry...

More Stories By Marten Terpstra

Marten Terpstra is a Product Management Director at Plexxi Inc. Marten has extensive knowledge of the architecture, design, deployment and management of enterprise and carrier networks.

Latest Stories
"We focus on SAP workloads because they are among the most powerful but somewhat challenging workloads out there to take into public cloud," explained Swen Conrad, CEO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are still a relatively small software house and we are focusing on certain industries like FinTech, med tech, energy and utilities. We help our customers with their digital transformation," noted Piotr Stawinski, Founder and CEO of EARP Integration, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"I think DevOps is now a rambunctious teenager – it’s starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We're here to tell the world about our cloud-scale infrastructure that we have at Juniper combined with the world-class security that we put into the cloud," explained Lisa Guess, VP of Systems Engineering at Juniper Networks, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"With Digital Experience Monitoring what used to be a simple visit to a web page has exploded into app on phones, data from social media feeds, competitive benchmarking - these are all components that are only available because of some type of digital asset," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
As enterprise cloud becomes the norm, businesses and government programs must address compounded regulatory compliance related to data privacy and information protection. The most recent, Controlled Unclassified Information and the EU’s GDPR have board level implications and companies still struggle with demonstrating due diligence. Developers and DevOps leaders, as part of the pre-planning process and the associated supply chain, could benefit from updating their code libraries and design by in...
"Peak 10 is a hybrid infrastructure provider across the nation. We are in the thick of things when it comes to hybrid IT," explained Michael Fuhrman, Chief Technology Officer at Peak 10, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
"We are an IT services solution provider and we sell software to support those solutions. Our focus and key areas are around security, enterprise monitoring, and continuous delivery optimization," noted John Balsavage, President of A&I Solutions, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We were founded in 2003 and the way we were founded was about good backup and good disaster recovery for our clients, and for the last 20 years we've been pretty consistent with that," noted Marc Malafronte, Territory Manager at StorageCraft, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.