Harvester’s Disruption of the HCI Space

Harvester is an open source hyper-converged infrastructure (HCI) solution designed for bare metal servers that uses other open source technologies such as Kubernetes, KubeVirt and Longhorn. It’s a flexible, affordable solution that provides virtualization and storage capabilities that are fully integrated into your cloud infrastructure without requiring specific knowledge of these technologies. Harvester also places Virtual Machine (VM) workloads close to the edge of the cloud, near the Internet of Things (IoT).

This software is still quite new, but it’s already making gains in the HCI market. This space has traditionally been dominated by a few major players, resulting in a high price for HCI. Many enterprises are particularly attracted to Harvester’s low cost, which is primarily because it’s open source software. Harvester also has a feature set that will allow it to compete with established players like Microsoft and VMWare. However, organizations looking to make this switch also need to consider the challenges of adopting this latest HCI solution.

HCI

HCI is a software-defined IT infrastructure that virtualizes the components of a traditional infrastructure defined by hardware. It consists of virtualized computing, storage and networking at a minimum, although these solutions also include other software-defined elements. An HCI solution usually runs on any commercial off-the-shelf (COTS) server.

The greatest single advantage of HCI over hardware-defined infrastructure is that building the virtualization on top of the operating system (OS) allows administrators to manage many systems as a single, shared resource. They no longer need to think of their infrastructure in terms of VMs running on individual servers, so they can add and remove new hardware as needed. Administrators can thus provision resources whenever they need to add capacity, making the process of running two servers just as easy as running 100.

Many people who are new to HCI confuse it with converged infrastructure (CI), but the technologies are distinct. With CI, the hypervisor only virtualizes some components such as a storage area network, whereas all components are virtualized in the case of HCI. When a hypervisor defines all of an infrastructure’s components with software, administrators can share all resources across all instances of the HCI.

Managing VMs at Scale

VMs have forever changed the way software engineers deploy and manage infrastructure. They have become so common that almost no one deploys production code to a physical server anymore. VMs host the OS of your choice in a secure, isolated environment while sharing the underlying server’s resources. This approach allows the infrastructure to allocate resources more efficiently, thus reducing the cost of hardware.

Hypervisors perform comprehensive management of VMs running on a single server. Administrators can use simple consoles or command-line interfaces (CLIs) to easily perform common VM actions such as creating new VMs, cloning them, backing them up, starting and stopping them. However, system architects often deploy VMs across multiple servers due to the flexibility and power they provide.

Managing VMs becomes more challenging as you increase the scale of your operations, which may require administrators to first gain access to a server before they can interact with a hypervisor running on that server. Managing VMs at scale can also involve moving VMs from one physical server to another, requiring a careful orchestration of shutting the VM down, backing it up, restoring it to the new server and booting the VM. Tasks that are routine for one server become slightly more time-consuming for two servers, and can easily be overwhelming for 100 servers.

Administrators clearly need a highly efficient method of managing their VMs as the number of VMs increases. Containerization systems like Kubernetes provide administrators with the ability to perform routine VM tasks across multiple servers from a single centralized console. They also help administrators consolidate virtualized resources, which simplifies their job by increasing the efficiency of resource allocation.

Harvester

Harvester uses Kubernetes and other container orchestration software as its foundation for managing VMs. Kubernetes is currently the de facto standard in this space, as it performs many VM functions such as the following:

• Application programming interfaces (APIs)

• Authentication

• Authorization

• Availability

• CLIs

• Declarative state

• Fault tolerance

• Flexible resource definitions

• Node management

• Software development kits (SDKs)

All of these Kubernetes features have been tested over time with many large-scale clusters. Kubernetes also orchestrates other resources besides containers through the use of custom operators and custom resource definitions (CRDs). These features allow Kubernetes to provision any type of resource, which Harvester leverages by extending Kubernetes with other container solutions like KubeVirt and Longhorn. As a result, Harvester is able to manage both physical servers and VMs.

Partnerships

Rancher Harvester's initial release in December 2021 follows a partnership between Rancher and NetApp in November of that year. This deal allows NetApp to embed Kubernetes into Rancher Harvester. This partnership also allows Rancher to partner with other HCI and storage vendors to offer a completely cloud-native technology to its customers. For example, Harvester is now one of many initiatives of SUSE, an open-source software developer. Harvester delivers some capabilities of commercial HCI solutions, but it doesn't attempt to compete with them due to its much smaller feature set. However, analysts do expect Harvester to broaden the market interest in HCI by IT administrators.

Other platforms such as Platform9 can also integrate with containerization projects, providing scalable management of the VMs as containers. However, this approach requires substantial knowledge about the specific project. While Kubernetes is the industry standard, detailed knowledge of its operation still isn’t common among VM administrators. Harvester doesn't require its users to have such knowledge, making it easier to install and operate. As a result, Harvester is a particularly important project for VM management.

Architecture

Harvester isn't the first project that has built VM management on top of Kubernetes, as Rancher's own RancherVM solution did so before Harvester. However, these other projects haven’t been commercially successful, largely because they all require users to have detailed knowledge of container platforms, including pods and PVCS. Harvester eliminates this requirement by hiding the underlying Kubernetes platform from end users. Instead, Harvester presents users with more familiar concepts such as disk volumes, ISO images, Network Interface Controllers (NICs) and PMs. This approach allows Harvester to leverage Kubernetes while still providing administrators with a traditional view of their infrastructure.

The chart below illustrates Harvester's architecture in greater detail:

Fig. 1: Harvester HCI Architecture

In the chart above, K3OS is a Linux distribution that minimizes OS maintenance in a Kubernetes cluster. KubeVirt is a VM management add-on for Kubernetes, while Longhorn is a lightweight, distributed block storage system for Kubernetes. Harvester’s architecture allows users to create VMs that they can attach to either a virtual LAN (VLAN) and management network, or just the network. H3O3 runs on top of the nodes, which can be bare metal, Longhorn VMs and Kubevirt VMs. However, this architecture could change when Harvester enters the General Availability phase of its development.

Features

Software developers have made many attempts to manage VMS through container platforms, including KubeVirt, Virtlet and Rancher's own RancherVM. There is some demand for these solutions, primarily for users who want to run legacy software alongside containers. However, none of these solutions have approached the popularity of leading virtualization solutions such as Nutanix and vSphere.

Harvester is specifically designed for users who don't know anything about Kubernetes but want to benefit from its capabilities. It runs on top of many established cloud-native solutions and uses proven open-source software (OSS) components to build virtualization kernels that are hidden from the user, rather than visible proprietary kernels. This design helps future-proof Harvester as the HCI industry shifts towards software that uses containers, multiple clouds and edge devices.

Harvester v0.1.0 has the following features:

Networking Management

Several methods of managing networks are available in Harvester. Each VM in Harvester has a management NIC by default, which is provided by Kubernetes’ overlay networking feature. Users can also add their own NICs to VMs, as Harvester supports VLAN. However, VLAN can’t use the physical NIC and Kube-vip simultaneously when Kube-vip gets the virtual IP address (VIP) from DHCP server. In addition, Multus provides multi-network functionality for Harvester.

Raw Block Device Support

Harvester improved its support of raw block devices in version 0.2.0. For example, it removed a dependency on Minio, which is a high-performance object storage solution. Harvester also uses Longhorn to manage the VM image with Backing Image, which stores the image in Longhorn. This block storage system uses the storage space on the node to provide highly available (HA) storage for VMs inside the cluster. This approach also reduces the time needed to import the image and boot the VM from that image.

PXE/iPXE Boot Support

Harvester provides Preboot eXecution Environment (PXE) boot support as of version 0.2.0, which allows administrators to automatically provision nodes on a bare metal server with an OS. It also supports iPXE, which is an open-source implementation of PXE. iPXE is a fork of gPXE that was created in 2010.

VM Backup Support and Live Migration

VM live migration is another key feature introduced in Harvester v0.2.0, which allows administrators to migrate a VM from one node to another. Harvester administrators can also backup VM's to a target outside their cluster, which requires them to add a backup target as an NFS server or S3-compatible endpoint.

Installation from ISO

You can download Harvester and ISO format from its release page on get up, allowing you to install it directly on a physical server. For this type of installation, you can add the node to a new or existing cluster. The installer will prompt you for the information that it needs to automatically create a new cluster, if needed.

VM Lifecycle Management

Harvester uses KubeVirt to perform the VM operations such as creation, deletion and updates. KubeVirt can also inject SSH keys and initialize clouds in Harvester. Additional features for managing VM life cycles include consoles that allow users to access the VM.

Rancher Integration

Rancher is an open-source platform that manages multiple clusters. Harvester is fully integrated into Rancher of v0.2.0 by default when using its HCI installation mode. This integration also creates Kubernetes clusters on top of physical modes.

Install as a Helm Chart

You install Harvester as a Helm chart on top of an existing Kubernetes cluster, typically for development purposes. However, these nodes must be able to support KVM through hardware virtualization or nested virtualization. Hardware virtualization options for Harvester include AMD-V and Intel VT-x.

Image Management

Harvester has a built-in image repository provided by MinIO. This feature allows users to easily download and manage images for VMs within the cluster.

Use Case

The idea of using Kubernetes to manage VMs may not seem particularly useful to HCI users at first. They will typically consider it easier to simply containerize workloads or use a hypervisor such as KVM or Xen to orchestrate the VM natively. These approaches often make sense except in the case of edge computing. Harvester provides a solution for the technically challenging task of running Windows legacy applications and containerized micro-servers on the same host. In this scenario, architects can greatly benefit by mapping out an edge stack and installing Harvester.

The next step is to apply updates to the host's Kubernetes and OS installation using a solution like VM Fleet. This collection of scripts deploys VMs that perform I/O operations for the purpose of testing the VM's underlying storage system. The architect can then use a tool like Terraform to complete the Harvester solution. Terraform is an open-source tool that allows end-users to define infrastructure through its own HashiCorp Configuration Language (HCL) or JSON.

Edge computing often lacks resources such as a cloud platform or even hardware. These limitations benefit from running Linux containers alongside Windows VM's to provide the flexibility this operating environment requires. It also provides administrators with the control and simplicity they need to manage an entire deployment through the Kubernetes API. This use case can use Harvester in app mode to maximize an edge node's usefulness by running both containers and VMs at the host OS level.

A typical stack for this purpose could include a server-oriented Linux distribution, Ubuntu and SUSE Linux Enterprise Server (SLES). An administrator can also customize the system-update operator for other clinics operating systems. This example will also use K3s as the Kubernetes distribution which is designed for unattended operating environments with severe resource constraints. This distribution is ideal for this purpose because it has a very small footprint, primarily due to the elimination of code related to cloud computing.

Harvester has two modes of operation, including HCI and a Helm application. The HCI mode allows Harvester to attach to another cluster as a hosting node, whereas a Helm application can be deployed into an existing cluster that administrators can install through a Helm chart and CRDs, providing it with great operational flexibility. A rancher cluster can orchestrate these PMs through Its Continuous Delivery feature, which is powered by VM Fleet. Harvester can also use features that are installed with the Helm chart by default, including libvirt, kubevirt, multus and minIO.

Once Harvester has been installed, the administrator can add a Windows image and deploy a VM through a CRD. Later, the administrator can also execute the scripts needed to provide the minimum viable product (MVP) for that deployment. Multi-layered VMs may also require additional CPU functionality, which can place some restrictions on the testing environment.

This example uses a stack composed of VM Fleet, Harvester and K3s running on top of a Linux distribution, which provides a solid solution for an edge use case. This solution should be capable of scaling up to millions of edge devices with many development teams. This solution certainly isn't the only one that you could use, as these components can be exchanged individually with other open-source components. However, it is one that works with the current version of Harvester.

Implementation

An architect can implement an HA control plane in Harvester by using Kube-VIP, providing a fixed VIP for both the control plane and all Kubernetes services. Kube-VIP is a self-contained HA option for all operating environments, but it requires the architect to add the VIP with a TLS certification before configuring Harvester, allowing users to access the API servers. Harvester is a K3s cluster, so the value of the VIP should be set to the --tls-san in parameter. Kube-vip will then set the VIP as a floating IP address to the specified NIC. An architect should also consider the floating IP address when configuring the VLAN network.

Kube-vip supports two VIP failover mechanisms, including Address Resolution Protocol (ARP) and Border Gateway Protocol (BGP). ARP is a communication protocol that identifies a link-layer address associated with a particular Internet-layer address, which is usually an IPv4 address. The primary drawback of ARP is that the failover can be slow, primarily due to single-node bottlenecking. BGP is the industry-standard for exchanging reachability and routing information between systems. The primary disadvantage of BGP is that its configuration depends on external routers.

Load Balancing

Load balancing In Harvester requires two components, including Load Balancer and Entry. Load Balancer forwards traffic to the Kubernetes load-balancing service, while Entry accesses the services behind the load balancer. Kube-vip is an inappropriate choice for implementing both components, which may involve leveraging Kubernetes or using a reverse proxy.

Kubernetes provides built-in load balancing for pods in Harvester. Each VM in Harvester is a pod, making it a simple process to balance loads for PMs. Each load balancing service has a node port that serves as the target port for the service, allowing each service to become a load balancer for a cluster. The primary advantage of this approach to load balancing in Harvester is that it's easy to configure, since it doesn't require modification of the source code. However, it also requires a health check and has poor extensibility, as it won’t support QoS or TLS.

A reverse proxy like Traefik can also implement load-balancing in Harvester. The main advantage of this method is that it offers better forwarding performance than Kube-vip. It also supports protocols for both layer 4 and layer 7, health checks, dynamic reloads and TLS. The primary disadvantages of this load-balancing method are that it requires custom development and maintenance for the reverse proxy configurations and LB controller.

Networking

Harvester supports two kinds of networks, including management networks and VLAN networks. Both networks use KubeVirt to implement VM networking via binding mechanisms. The current options include bridge, masquerade and slip, but only bridge and masquerade are used in Harvester networking. VLAN networks use a bridge-binding mechanism, while management networks use masquerade binding. Both mechanisms use a tap as the VM's NIC and a bridge to forward data packages.

Some differences also exist between the two methods. For example, the bridge-binding mechanism doesn't use the bridge's IP address. Furthermore, the IP address of the interface inside the VM can be any value. However, both the bridge and the interface in the VM use a fixed IP address in the masquerade-binding mechanism. As a result, every bridge in each VM has the same IP address for this mechanism.

The two binding mechanisms also use different methods of forwarding packages to the VM interface. A bridge-binding mechanism uses a pod interface like container network interface (CDI) to generate the packages, which then forwards the pods to the bridge via the Layer 2 network. In this model, the bridge controls the pod interface. With masquerade binding, the pod interface uses Network Address Translation (NAT) and IP tables to forward packages to the VM interface.

Masquerade binding is usually the better choice for a management network due to several issues with the bridge binding. For example, delegating IPv4 addresses in bridge mode prevents the configuration of the pod's IP address, which can cause incompatibility with third-party solutions that use this IP address. This issue applies specifically to Istio, which extends to Kubernetes to create a programmable network with the Envoy service proxy. Istio works with both traditional workloads and Kubernetes to provide capabilities such as telemetry and universal traffic management, which is especially useful for complex KubeVirt deployments. Furthermore, bridge-binding doesn't allow live migration, nor does it allow some CNI plug-ins to use a custom MAC address for VM instances. The KubeVirt documentation provides more detail on binding mechanisms.

Management Network

A management network is the same as a Kubernetes network, except that it adds VM networking. The Harvester cluster in a management network uses Flannel as the default CNI, allowing it to directly access the VM with the pod address. However, this VM is only accessible within the cluster unless you use other methods to externally expose it, since the cluster maintains the address.

VLAN Network

The key points of a VLAN network include the creation of a bridge for each node, allowing the VLAN filter to divide the network into multiple VLANs. This process typically uses veth to create a pair of virtual Ethernet devices connecting the pods with their bridge. It also requires the physical NIC to serve as the bridge's slave, allowing the network to transfer traffic externally. The NIC's switch should be set to trunk mode to prevent it from dropping packages with a PVID that’s different from its VLANID. Supporting DHCP also requires a cbtables rule to ensure the slave NIC receives DHCP messages instead of the bridge itself.

VMWare

VMware is a virtualization and cloud-computing firm founded in 1998 and was the first company to commercially succeed in virtualizing the x86 architecture. Its products include vSAN and vSphere, which provide an HCI solution for standard x86 servers on VMWare’s Hardware Compatibility List (HCL). VMWare has led the HCI market since 2018, with Nutanix in a fairly distant second place. Together, the two companies account for the great majority of the HCI market. For example, International Data Corporation (IDC) reports that VMWare and Nutanix dominate the HCI market in terms of revenue.

While the revenue of the two firms was relatively even until 2018, VMWare’s revenue has increased while Nutanix’s revenue has remained steady or even decreased. For example, the revenue of both companies was $500 million in Q2 2018. For Q3 2020, VMWare had a revenue of $821 million, which was a 4.9 percent increase for the past year. Nutanix’s revenue dropped to $512 million in Q3 2020, an annual decrease of 6.7 percent.

Several possible explanations exist for Nutanix’s drop in the HCI market. For one thing, the company is transitioning to a subscription-based business model. In addition, Rajiv Ramaswami, former executive at VMWare, has recently taken the position of CEO at Nutanix, so the firm is still adjusting to this change.

The table below describes the changes in HCI market share by company during 2020:

Fig. 2: HCI Revenue by Company for Q3 2020

The following table shows the sales of the four leading HCI firms, including WMWare, Nutanix, HPE and Cisco:

Fig. 3: HCI Sales by Company for 2017 to 2020

Note the difference in sales between the four companies during this time period. VMWare has taken market share from Nutanix, while Cisco and HPE have generally remained steady. The revenue of other HCI competitors like Nimble and SimpliVity is too small to show up on the above chart. Another point to note in the above chart is that Cisco could be gaining market share, although it’s still too early to be sure. This firm’s HyperFlex HCI-branded systems generated $121.1 million in Q3 2020, resulting in an 11.1 percent increase for the previous 12 months and a gain in market share from 5.1 to 5.4 percent.

Harvester vs. VMWare

The great difference in price is one of the primary reasons that many organizations are considering Harvester as an alternative to VMWare. This difference became particularly large a few years ago when VMWare changed its licensing model, largely as a result of its superior market position over its competitors. This change, dubbed the “VMWare tax,” generally requires VMWare users to pay for all of VMWare’s capabilities, whether or not they actually use them. Smaller companies are most strongly affected by the VMWare tax because they typically use only a small portion of VMWare’s feature set. Harvester’s open source nature has the potential to disrupt VMWare’s leadership in the HCI market by eliminating the need for users to pay the VMWare tax.

The demand for a cheaper alternative to VMWare is extremely high, so many companies have attempted to compete in the HCI space. The most common strategies include adding features to an existing container platform or developing a new solution with more features. The most successful attempts include KubeVirt and Rancher VM, but they haven’t come close to achieving VMWare’s feature set.

The primary challenge with these solutions is they require users to possess a fair degree of proficiency with container platforms like Kubernetes. However, Kubernetes still isn’t a common skill set among VM administrators, even though it’s the industry standard. Previously, organizations that were trying to transition away from WMWare had to acquire or develop this expertise for this strategy to succeed. Harvester’s greatest advantage against VMWare is that it’s built on top of cutting-edge technologies like Kubernetes. As a result, Harvester is easy to install and use because users don’t need to know anything about Kubernetes.

There are reasons for organizations to stay with VMWare, despite its significantly higher cost. For one thing, WMWare doesn’t require an OS to control the management components, nor does it require security patches to control layer components. In addition, Amazon Web Services (AWS) are available to VMWare users.

VMWare also has other disadvantages besides its cost, providing additional incentive to find an alternative HCI solution. For example, VMWare solutions don’t support many types of hardware, including some popular brands. These solutions also use complex device drivers, which can cause slow initialization times. Furthermore, VMWare trial software doesn’t include all functionality, making it difficult to evaluate these solutions.

Pricing

Companies interested in Rancher Harvester are most often motivated by a desire to reduce their long-term costs of cloud service, especially licensing fees. Harvester has a Lower Total Cost of Ownership (TCO) compared to traditional solutions like Azure Stack HCI and VMWare. Harvester is completely open source, so it’s free of the licensing fees that are a major part of the TCO for other solutions. In addition, Harvester is built on top of existing technology like Linux and kernel- based VMs, so it doesn’t need to provide all the needed functionality itself.

Rancher offers customized pricing, so standard prices aren’t available for Harvester. Fill out this form to get detailed pricing information from one of Rancher’s regional sales teams.

Summary

Individual servers expose their resources to users, which HCI abstracts. This capability provides administrators with a seamless interface, allowing them to manage VMs from a single location. These tasks typically include provisioning, moving, cloning and backing up VMs, which can otherwise become unmanageable for large virtualized operations. Scalability is thus a key requirement for HCI solutions, especially enterprises that are growing quickly.

Harvester is an HCI solution that leverages other open-source container projects like Kubernetes, KubeVirt and Longhorn, although it’s specifically designed not to expose these tools to the end user. The purpose of this design is to give administrators with a view of their infrastructure that they’re already familiar with, while still providing them with the most powerful VM capabilities available.

Harvester is positioned to gain market share from VMWare, which currently dominates the HCI market. However, many users are dissatisfied with VMWare’s pricing structure, especially those that only use a few of the features for these solutions. Harvester has a much lower cost because it’s open source, as are the technologies it’s built on top of. In addition, Harvester’s ability to leverage existing technologies without exposing those technologies to users is a major benefit for VMWare users looking for alternative HCI solutions.

About Baytech

Baytech is passionate about the technology we use to build custom business applications, especially enterprise solutions that optimize business processes. We’ve been delivering software solutions in a variety of technologies since 1997. Our success is due to the skill and efficiency of our senior staff, which includes software engineers, project managers, and DevOps experts. All of our engineers are onshore, salaried staff members.

We focus on the quality, usability, and scalability of our software and don’t believe in mitigating cost at the risk of quality. We manage project costs by implementing an efficient development process that’s completely transparent and uses the latest standards and practices to build software right the first time. Contact us today to learn more about how we can help your business. Find us online at https://www.baytechconsulting.com/contact.