pfSense Firewall build using Intel NUC 11th Gen Pro

Hardware Selection: 

I was planning to migrate my existing pfSense firewall which is currently running on top of Pentium Gold G5400 + Gigabyte iTX motherboard which I have built 2 years back. It is running fine without any issue, however the power utilization of this build is higher if you run it 24x7x365. The latest Intel NUC Pro 11th Generation comes with lot of IO options and less power utilization. This generation comes in i3, i5 and i7 models. For running the pfSense I wanted to go with i3 version which should have more than sufficient horsepower to run. 

You can see the complete specification of the 11th Gen i3 Pro model from here. Since this post is about pfSense build not detailing about Intel NUC and it's specifications, I will directly jump into the parts that will be of our interest. It comes with 1 x 2.5 Gbps Onboard Ethernet port, and an add-on option to install additional Ethernet interface of 1 x 1 Gbps internally which you can see in the picture shown below. The Ethernet port in the top got 2.5 Gbps and the add-on shown in the bottom got 1 Gbps connectivity. 

The additional module is currently not available in my country, so I had to use some alternative approach until the add-on is available here. We can overcome the 2nd Ethernet option with USB based Ethernet adapter, continue reading the blog. 

Note: Buying the pfSense physical appliance from NETGATE (model  2100 MAX ) would be really a good and cost effective option, however the cost of this appliance in my country is 3 times more than the actual cost its available in United States. 

This new build will reduce the power bill in longer run. I know the ROI of NUC Unit, Memory, Nvme and USB3 to RJ45 will not match the power bill unless we are running the NUC for another 10 years without any upgrade, however that's just an excuse to test the latest NUC Pro model. 

When I checked the interweb anyone built similar specifications using 11th Gen I was not able to find much details. So thought of taking the plunge and ordered the NUC unit from Amazon. Some of the parts I have used are chosen based on availability and to reduce cost of this build. This system will be running 24 x 7 providing internet connectivity for me to work from home and to attend online classes for my kids. Let's quickly jump into the actual build without wasting much time. 

Below given are the part I have used for this build. 

1. NUC11TNHi3 x 1

2. Crucial RAM 16GB DDR4 - 2666 Mhz SODIMM 1.2 V (CT16G4SFRA266) x 1.

3. Crucial P2 250GB M.2 2280 Nvme Internal SSD x 1.

4. UGreen USB 3.0 to RJ45 Gigabit Ethernet LAN Adapter x 1.

5. Sandisk Cruzer Blade 16GB x 1 (This can be reused for other purpose later)

This model of NUC can support RAM up to 3200 Mhz, However I went with 2666 Mhz speed to reduce the cost of overall build. The Nvme and USB3 to RJ45 I have mentioned were purchased earlier this year so I could get them at cheaper price compared to their prices now. 

Note: As I mentioned earlier to get additional Ethernet port, I have used UGreen - USB Adapter (AX88179 Chipset). This adapter is natively supported by pfSense. I am using it for more than 20 months without any issue. 

pfSense Installation:

While I am building this setup, the available pfSense free version is 2.5.2. You can click this link to download the "USB Memstick installer". Detailed Bootable USB media creation steps are available from pfSense. I followed the pfSense instructions, used 7-Zip, diskpart and Rufus to create the USB Installer. 

Opened the NUC, installed the Nvme disk, DDR4 memory in respective slots. After that I have connected the NUC with Power plug, USB 3.0 to RJ45 Adapter connected to my ISP provided cable, 2.5 Gbps port to to Ubiquity AC Pro Access point, Sandisk Cruzer Blade with pfSense installer image, Keyboard and Mouse (optional). The mouse is required for doing some tweaks in the BIOS. If you are comfortable with Keyboard navigation we can skip the mouse. 

Powered on the NUC, Pressed F10 to select the Boot Device, Selected "UEFI: Sandisk, Partition 1 (SanDisk)" option and pressed enter.  I was greeted with an error message as given below. 

• The solution is very much simple. Reboot the system, Get into the BIOS by pressing F2, Go to Boot -> Secure Boot -> Select the Secure Boot option as "Disabled". Leave the Boot Mode as "Custom". If you have mouse plugged in you can use it for selecting the menus and values, else Keyboard is your friend. 

• Press F10 to save the changes made, Confirm by Ok
• When the system is rebooting press F10 to get the Boot Drive Selection Menu

• The System will start with usual FreeBSD text based installation, let the devices detected and the installer wizard kick starts

• Read the Copyright and distribution notice, Press "Accept"

• In the Welcome Screen, Select "Install" option, Press Ok.

• Leave the Keymap selection as it is, Press "Select"

• Leave default option in Partitioning as "Guided Root-on-ZFS", Press "Ok"

• In the ZFS configuration option proceeded with Installation as given below, Continue by pressing "Select"

• Leave the Device Type as "Stripe - No Redundancy", Click Ok

• If there are more than one drive and the drive is with data, be cautious in the following step. Installer may format and delete the valuable data if wrong disk is selected. 
• In my case the disk is a brand new, so I have selected "nvd0" and Pressed "Ok"

• In the warning, This will be last chance to stop the installer formatting the selected drive, I have continued by pressing "Yes"

• Installer will continue and will get the OS installed in the selected drive. Be patient, you don't have to grab any drink :) because the installation will be done quickly. 

• The installation got completed. I didn't have anything to modify. So I have selected "No" 

• That's all. The installation got completed successfully. I was prompted to restart the system, I selected restart and the system was getting rebooted. This time I have removed the USB Installer, continued to boot the system from Nvme. pfSense has detected the onboard Ethernet Controller i225-LM and AX88179 Chipset based UGreen Ethernet without any tweak or modification. So the configuration was smooth. 

Stay tuned for Part 2 of this series to continue with the configuration of firewall policies and WiFi Configurations. 

Home Lab - Part 2

Design Decisions:

Why Intel i5-9400 is chosen?

While buying the above-mentioned parts from IT store near to me, I had to be very specific about the components that I was looking for. In general, the system with i5/i7 processor, 32 GB or more RAM configurations are used by "Gamers" and not an average desktop user. They were trying to sell the i5-9400F along with separate Graphic card. Though the i5-9400F costs cheaper than 9400, the overall cost will be higher after adding the price of dedicated graphics card (+ Tax). Also make sure you are not buying one of those 9x00K series which will not benefit the virtualization. The “K” will help you to overclock while building “Gaming”machines and not Whiteboxes. With my previous Whitebox build experiences, Intel i5 should be more than sufficient for running a decent home lab. If you have enough budget, go for i7 Processor without the K.

Why B365M-D3H Motherboard is chosen?

Since we have locked down to the processor, now we have to look for a compatible motherboard that can support. While selecting Motherboards with Intel 9th Generation chipsets, below given points will give us some idea about the specification and cost. 

Brand	Chipset	Purpose	Cost
Intel	B310	Entry level desktop purpose with limited hardware features	Low
Intel	B360/B365	Mid Level desktop purpose with moderate hardware features	High
Intel	H370	High end desktop purpose with good hardware features	Higher
Intel	Z370	Higher end desktop purpose. Supports over-clocking. Used for graphical editing, gaming, etc. Not much benefit in homelab.	Highest

Though the B360/365 chipset is mid-level and the price is higher than B310 chipsets, the D3H version of motherboard from Gigabyte is one of the killer combinations for running ESXi in homelab environments within budget. I am highlighting some of the useful specification of this motherboard which will benefit the home lab.

• Supports 8^th and 9^th Generation Intel Core processors
• Supports 4 x 16 GB of Non-ECC Unbuffered DDR4 Memory up to 2666 MHz
• Ultra-Fast M.2 which can be used for local M.2 NVMe disks
• Intel GBE LAN Supported by VMware ESXi bundled drivers (No need for any .VIBs)
• Supports Intel Optane Memory
• Supports all type of Graphics output – 1 x VGA, 1 x HDMI (1.4), 1 x DVI, 1 x Display port
• 1 x PCI slot (Addional Intel Pro/M1000 NIC can be installed)
• 1 x PCI Exp x 1 slot . This can support TP-Link TG3468 or Intel CT Desktop NICs
• 1 x PCI Exp x 16 slot (runs as x4).  This can support TP-Link TG3468 or Intel CT Desktop NICs
• 1 x PCI Exp x 16 slot (runs as x16).  This can support TP-Link TG3468 or Intel CT Desktop NICs
• 6 x SATA 6Gb/s – These ports can be used for local storage using SSDs/HDDs
• 1 x CPU and 3 x System FAN headers to keep the box cool enough
• Micro ATX Form factor. Can fit in decent sized system cabinets/case. 

Other Hardware Components:

There are other hardware components required like Memory, SSDs, NVMe, NICs, Desktop Case/Cabinet, etc., to make the build complete. 

We have lot of options in the market based on brand/cost/color/specification/configuration which are personal preferences. I will not discuss about them and will leave them to individuals.  Before purchasing, make sure about the compatibility of the motherboard, VMware hypervisor you are going to use.

I will be discussing about the ESXi installation and VM builds in future blogs. Stay tuned... 

Home Lab - Part 1

Background: 

For learning VMware Virtualization and VMware VCP certification studies, I had built a whitebox about 8 years back with below given configurations:

• Intel processor i7-3770 
• Gigabyte B75M-D3H Motherboard 



Previous whitebox Components

• G-Skill 4 x 8 GB DDR3 = 32 GB Memory 

This CPU is still powerful enough to run the latest ESXi version 6.7,  however the memory supported by this CPU and the motherboard is 32 GB as maximum. Virtualization is memory hungry when you run multiple VMs also the latest appliances of VMware, Windows Server VMs, Linux Server VMs, Firewalls, etc.,  require more memory than their previous versions. I was thinking to upgrade my lab with latest hardware configuration as much as possible.

Points to Ponder:

While building a home lab we have to keep below given points in mind. 

1. Budget to procure and run the hardware/upgrades
2. Location from where the lab will be operated from
3. Hardware components and their upgrade possibilities/maximums
4. Warranty of the hardware components used
5. Size of the servers/desktop enclosures

There are IT enthusiasts who are running their lab in older/newer server hardware which are enterprise grade. If we are planning to take this direction, there are few challenges that need to be considered. 

• The heat these servers produce while running (location)
• The tiny fans which rotates at very high speed @ 10K RPMs, generate loud noise (location)
• High cost that we have to pay for replacing any faulty hardware during the run (Warranty)
• Identifying suitable hardware for any component upgrade (Budget)
• Replacement/upgrade may be difficulty after certain period (Upgrade possibilities)
• Replacement/upgrade may be costlier now than when they were available (Budget)
• A garage/basement/dedicated room with sufficient air circulation to run the Lab (Location)
• We also need server rack to install the enterprise grade heavy hardware (Size)
• High Electricity bills while running the lab (Cost)

The advantage we will have after overcoming above challenges would be
(this will be based on the old server model/hardware we choose)

• Knowledge about enterprise grade hardware, driver/firmware upgrade processes
• Operating the server with one or more Intel Xeon Processors 
• 128 GB or more memory with ECC support using DDR3/DDR4
• Enterprise grade hard drives/SSDs which have higher MTF than consumer grade
• Additional network interface ports that can be used in virtualization
• Remote management using iLO/IPMI which reduce the need for KVMs. 

Since I don't have dedicated/air conditioned rooms to keep the server grade hardware, I always prefer running my lab on Desktop based hardware. The Lab devices will be placed in a custom rack in the study room where me and my kids will be using it for academic purpose. Also not I am very much bothered about the system form factor to keep it small and good looking. The cost of those systems will be higher and the upgrade options will be limited. 

Requirement Freeze:

Before deciding the final bill of material, I would like to freeze the hardware configuration required to run my lab. Since we require more memory to run multiple VMs simultaneously,  the processor and motherboard should be supporting maximum memory configurable in desktop chipset available while we are building the whitebox systems. At the time of writing this blog, 64 GB is the maximum possible memory using Intel 8th/9th Generation desktop processors + compatible motherboards.

I will be running this whitebox installed with VMware ESXi as baremetal hypervisor, add-on graphic cards will be overkill and will increase the total cost. A desktop processor with integrated graphics will be more than sufficient to run.

We don't require multiple USB ports, Sound cards, different type of display connectivity options like HDMI, Display port, DVI  Ports, etc., to run baremetal hypervisor. However in a desktop segment of motherboards you don't have much choice. These desktop motherboards are customized for running Windows/Linux Single User operating system, not for lab purpose. They may be bundled with additional components that are required for host desktop users in mind,. We may not be using those components anytime in our whitebox system but we have to live with them.

I was going through hardware manufacturer portals, blogs, YouTube videos. etc., before finalizing the hardware components. After spending few month reviewing the specifications, I have decided to go with below given list of items.

Bill Of Material: 

1. CPU: Intel I5 - 9400 - Intel 9th Generation Processor with Integrated Graphics x 1
2. Motherboard: Gigabyte B365M-D3H  Intel 8/9th Generation Compatible Motherboard x 1
3. Memory: 16GB DDR4 2400 Mhz Corsair Vengeance LPX x 2
4. Power Supply: Corsair VS450 80 Plus x 1
5. Computer Case: Cooler Master Elite 344 USB3 - Silver x 1
6. USB Stick: HP v215b - 8 GB (USB 2.0) x 1
7. Intel Pro/1000 GT - PCI Gigabit Network Adapter Single Port x 1 
8. Intel CT Desktop - PCI-E(x4) Gigabit Network Adapter Single Port x 1

I will be explaining why I have selected above mentioned components in my next blog.