Building Reliable NX-OS LAB on Your Personal PC - Part 2
Welcome to the second part of "Building Reliable NX-OS LAB on Your Personal PC". As we said before, we will cover in this post building multiple NX-OS Machines along with the connectivity between them.
We have stopped on getting the first NX-OS VM up and running, now you will need to power it off as we are going to clone it to create the other NX-OS machines.
There is two types of cloning in VirtualBox, Full cloning and Linked cloning. The difference between them is that the full cloning create new files for the VM which duplicates the space as the new VM will be separate VM with its own Files. On the other hand, linked cloning will take a snapshot of the original machine and will write the new variations in a snapshot file for the original VM, and a snapshot file for the cloned VM . This type saves space significantly as only snapshots are created.
In our Lab, we will be using linked cloning. Please follow the below steps to create a second NX-OS Machine from the Original NX-OS Machine with linked cloning.
Right click the Original NX-OS VM, then click clone.
Click the "Expert Mode" box (it will be at the bottom of the window).
In the Name Field, type the name of the second NX-OS Machine.
Check "Linked Cloning" checkbox.
Check " Reinitialize the MAC address of all network cards" then click clone.
Repeat the previous steps to create another NX-OS VMs as per your desire and clone them all from the original VM.
For each VM, we will need to change its serial path to be able to console them all. Please follow the below steps to change serial path for the second VM:
Right Click the NX-OS VM, then click setting
Click the Serial Ports icon:
Choose the Serial Ports tab/Port 1 tab:
Check the Enable Serial Port checkbox.
Ensure that Port Mode is Host Pipe.
Uncheck Connect to existing pipe/socket
Ensure Path/Address is \\.\pipe\COM2
Start the VM
Open PuTTY, then apply the below configuration:
Open "Basic options for your PuTTY session.
Select "Serial" for "Connection type"
Specify "\\.\pipe\COM2" for Serial line path.
Select "9600" for "Speed".
Now, for the third VM, the path should be "\\.\pipe\COM3" and on the putty you need to make it "\\.\pipe\COM3". Repeat the above steps for each VM to change its COM serial path and make sure that each VM has a unique path.
Example: If you have total of four NX-OS Machines, configure the serial paths for them as below:
For NX-OS VM1, use this serial path \\.\pipe\COM1
For NX-OS VM2, use this serial path \\.\pipe\COM2
For NX-OS VM3, use this serial path \\.\pipe\COM3
For NX-OS VM4, use this serial path \\.\pipe\COM4
After you get login to all VMs, configure the IP address of the management port and make sure to have IP address in the same range of the adapter you have Bridged-to. Try to SSH to the management port from your PC/Laptop to each NX-OS machine. Now you can configure a default route in the management vrf towards the subnet gateway and you can SSH remotely to your Nexus Machines without the need to console locally from your PC/Laptop.
Example: If you have total of four NX-OS Machines and you have bridged Adapter 1 of them all to your Laptop wireless adapter which has the IP address of 192.168.1.1/24 and GW of 192.168.1.254, you can configure Management Ports IP addresses as below:
For NX-OS VM1, mgmt0 to have IP address of 192.168.1.11/24.
For NX-OS VM2, mgmt0 to have IP address of 192.168.1.12/24.
For NX-OS VM3, mgmt0 to have IP address of 192.168.1.13/24.
For NX-OS VM4, mgmt0 to have IP address of 192.168.1.14/24.
If you need to access them all from outside the 192.168.1.0/24 subnet, configure a defualt route on each of them in the management VRF towards 192.168.1.254.
Now the last step to finalize building your Lab, is to connect the NX-OS Machines together according to a topology of your desire. Before we begin we need to understand first the mapping between the NX-OS VM Adapters and the NX-OS interfaces.
The VirtualBox support up to 8 Network adapters per VM, which will be enough to build a lab even with 6 NX-OS machines. (Adapters can be increased to 36 when changing chipset from PIIX3 to ICH9, but this is not recommended for VMs that's already built with PIIX3 and may not boot or to be unstable).
The corresponding ethernet interfaces of NX-OS to VirtualBox Adapters are described below:
VM Adapter 1 >>>> interface mgmt0
VM Adapter 2 >>>> interface ethernet 1/1
VM Adapter 3 >>>> interface ethernet 1/2
VM Adapter 4 >>>> interface ethernet 1/3
VM Adapter 5 >>>> interface ethernet 1/4
VM Adapter 6 >>>> interface ethernet 1/5
VM Adapter 7 >>>> interface ethernet 1/6
VM Adapter 8 >>>> interface ethernet 1/7
For example, any changes in VirtualBoX NX-OS VM adpater 3 will affect interface ethernet 1/2 in the NX-OS.
We have already finalized the configuration of VM adpater 1 on all NX-OS machines which corresponds to interface mgmt0 of the NX-OS interfaces. All other interfaces (from adapter 2 to adapter 8) will be configured with same mode but with different linking. To configure the adapters, VirtualBox GUI only gives you access to configure Adpater 1,2,3 and 4. to configure the remaining, you have to configure them using VBoxManage commands from the CLI.
In order to use the VirtualBox CLI, you have to install the extension pack (already stated in the requirements in the previous post - Part 1). To install the extension pack and get access to VBoxManage commands, please follow the below steps:
In the VirtualBox Manager GUI, click file (you will find it in the upper left side of the window).
Click Preference.
Click on the extensions icon:
Click on add new package icon (you will find it on the right side)
Choose the extension pack you have downloaded and make sure it's the same version of your VirtualBox Manager, then click open.
Click Ok
Open cmd of your windows, then type: cd c:\Program Files\Oracle\VirtualBox
Now you can use the VBoxManage commands
You can configure adapter 2,3 and 4 from GUI, then the remaining from CLI. But it's preferred to do all the configuration from CLI as it's almost identical.
For any ethernet interface that will be connected between NX-OS machines, you need to configure the corresponding VM adpater with mode internal network and to assign it to the relative internal network. To do this, you need to execute three commands in CLI as below:
VBoxManage modifyvm VMNAME --nic<N> intnet
Where VMNAME is the name of the NX-OS VM, N is the adapter number
This command create adapter (N) with mode internal_network
VBoxManage modifyvm VMNAME --nicpromisc<N> allow-all
Where VMNAME is the name of the NX-OS VM, N is the adapter number
This command enable Promiscous mode with allow-all for the adapter
VBoxManage modifyvm VMNAME --intnet<N> NETWORKNAME
Where VMNAME is the name of the NX-OS VM, N is the adapter number
NETWORKNAME is the connection name between this port and the opposite port on another NX-OS Machine, you can choose it to be anything but both ports should have the same network name)
Example; We need to link interface ethernet 1/3 on NX-OS VM called N9K-1 with interface ethernet 1/5 on NX-OS VM called N9K-2. Below will be the commands:
VBoxManage modifyvm N9K-1 --nic4 intnet VBoxManage modifyvm N9K-1 --nicpromisc4 allow-all VBoxManage modifyvm N9K-1 --intnet4 N9K-1_N9K-2
VBoxManage modifyvm N9K-2 --nic6 intnet VBoxManage modifyvm N9K-2 --nicpromisc6 allow-all VBoxManage modifyvm N9K-2 --intnet6 N9K-1_N9K-2
Note that we configured nic4 on N9K-1 which corresponds to interface ethernet 1/3 and configured nic6 on N9K-2 which corresponds to interface ethernet 1/5. And both interfaces are connected to internal network called N9K-1_N9K-2.
Repeat the above step for each link in your topology.
Last thing to note is that for routed ports on any nexus, it may choose same MAC address as the opposite port of peer switch of that link, if so make sure to change the mac address of the opposite port with mac-address XXXX.XXXX.XXXX. command under the routed interface.
Hope this topic was fruitful to you and see you in further posts ISA.. Keep Tuned !