> > I would not be surprised if just making your own NICs and switches is cheaper once you hit a certain deployment size.
> Ideas you’ll never hear at Google or meta
You'd be surprised. Google has a very strong tradition of "not-invented here" which extends to some of our production networking gear as well.
To be fair, at the time, some of this was justified because the available devices on the market couldn't support our use cases back then.
Per section 3.2 of the 2013 B4 paper [0]:
Even so, the main reason we chose to build our own hardware
was that no existing platform could support an SDN deployment,
i.e., one that could export low-level control over switch forwarding
behavior. Any extra costs from using custom switch hardware are
more than repaid by the efficiency gains available from supporting
novel services such as centralized TE.
You couldn't have picked a better/worse duo in tech to be wrong on such an assertion.
Adding to sibling comment about Google, Meta[1] built 2 large-scale production training clusters for science: one with Infiniband, the other one with a custom RDMA over RoCE fabric.
> Custom designing much of our own hardware, software, and network fabrics allows us to optimize the end-to-end experience for our AI researchers while ensuring our data centers operate efficiently.
> With this in mind, we built one cluster with a remote direct memory access (RDMA) over converged Ethernet (RoCE) network fabric solution based on the Arista 7800 with Wedge400 and Minipack2 OCP rack switches.
Google, Meta and Netflix are among the most obsessive on optimizing their infrastructure - it's bold to assume they haven't looked at their COTS network gear and thought "hmmm..."
