Network Trivia Buzzer

Basics:

iOS/Android clients support 1 or 2 player “buzzer” operation by setting unique player names on each device.

The MQTT server (Android Only) runs on a large table/chromecast/fireTV stick and manages a list of players (auto generated as the network connection is established) and then displays the top 3 people who have buzzed in, as well as the time in milliseconds between 1st/2nd, and 2nd/3rd.

The software is provided free and is as-is where-is.  Have fun!

 

Mellanox 10GB IBM OEM Card Woes

So I bought some 10GB Mellanox cards on eBay recently in order to add some cheap 10GB capabilities to my test cluster.  Here is my story so I can hopefully save you some headaches.

So my two machines, ClusterF1 and ClusterF2 booted up with the cards fine, however one of the two ports showed as could not start in device manager and would not work.  A quick google suggested using the proper driver/firmware update would remedy the problem.  Now the cards I bought were OEM IBM cards, so I ended up having to goto their site to get the driver/firmware update.  All  is right in the world right?  WRONG….

So, after downloading and running the installer on ClusterF1 (I was RDP’d in) I lost my network connection, thinking that it was just doing something with the networking stack, I mistakenly started it on ClusterF2 as well.  Later, using the iDrac console I discovered that the installer had completely F’d up my 2012R2 fresh install on both machines.  I was getting blue screens of death and it would not boot back up 😦

In digging into things, I seemed like my best bet was going to be to get some non-OEM firmware installed on the card… so here we go again.  I now had my 3rd cluster member to play with while re-installing the OS on the other two.

The cards I bought were labeled as:

IBM Mellanox 10Gb ConnectX2 EN Dual Port PCI-E Adapter 81Y1541

Here is the fix:

  1. Download the Mellanox firmware tools here: http://www.mellanox.com/page/management_tools
  2. After installing, goto the folder where the tools installed and run:
    1. MST Statusmststatus
    2. Get the PCI Device ID of the card and use it to run this command:
    3. flint -d <pci device ID> queryflintquery
    4. Note the PSID (mine was IBMxxxxxxxxxxxxx as I had an oem card and most importantly look for what kind of card it is, mine was a MT26448
    5. Dig through the non-oem Mellanox firmware on their site and find the proper firmware for your card, download it, extract it and place it in the firmware tools folder (firmware for my card was fw-ConnectX2-rel-2_9_1200-MNPH29C-XTR_A2-A5-FlexBoot-3.3.400.bin)
    6. Run this command: flint -d <PCI Device ID> –allow_psid_change -i
      firmwarefile.bin burn
    7. In my case it asked me if I wanted to overwrite the boot rom, I said yes, and then it warned I was changing the PSID, I confirmed that as well.
    8. It takes a reboot for the card to load the new firmware.  I did this, then installed the 4.8 driver (in their archive section) for Server 2012R2 and all is now right with the world.  In IO tests with Jumbo Frames enabled I’m hitting 9.55Gb/s with some cheap Amazon DAC cables from host to host.
  3. Hopefully this helps someone else, good luck!

Making a HP Procurve 5406ZL quieter for homelab use

Before and after video with sound

So the 5406 is a great switch that can be had cheap on eBay if you keep an eye out.  My current unit came with 2 blades and 2 power supplies for $200 shipped.  The only problem is that now that my basement is finished and used, the noise which was ok for a datacenter, was too much for a homelab.

I solved this problem by ordering 6 Noctua fans.

4-NF-R8 Noctua’s for the power supplies (2 each)

PSFans

2-NF-P12 Noctua’s for the fan card

Fancard

WARNING – taking apart the high voltage powersupply is bad, mmmmkay?  You could get shocked, killed, deaded, hurteded or blowned up (according to my youngest).

The power supplies are the hardest part of the project, requiring about 16 tiny screws to be removed.  But once inside, remove the 8 exterior screws holding the existing fans, unplug the fan harness, and remove the fans.  I cut and re soldered the stock connector onto my two fans, and re-assembled.  No problems.  The power supply fans are 2 pin, you’ll only connect black/red just like the factory wires.

The fan card was much easier.  Use a screw driver to pry the 8 quick release pins out, swap in the new fans, solder the wires (I taped and shrink-tubed them) and re-attach using the Noctua low noise mounts.  The Noctua’s I used for replacement here run at 1300rpm, the stock fans ran at 2300rpm.  The HP does flash a fan error, however it runs cool and quiet.  You can check by running “show system temp” at the command line.  It will throw errors into the event log.  If someone finds a quiet fan that runs at 2300rpms, let me know!

Put everything back together and viola, your HP is whisper quiet.

 

 

HP ZL Compute Blade on the Cheap

So I recently was lucky enough to come into a cheap HP ProCurve 5406ZL setup.  I’m talking sub $200 cheap, with 4 blades, and dual power supplies.  This gave me 96 POE gig ports, plus power redundancy plus two empty slots I could fill at some other time. (note there is also the HP5412ZL which is 2x the size of the 5406 but is often cheaper due to it’s increased size, the power supplies and modules are interchangeable).

As I was searching for a deal on one, I noticed the  HP Advanced Service module w/Vmware.   At $4k+ retail and $1500+ on eBay it was out of reach.

HP Advanced Services v2 zl Module with HDD

The idea however, of having a redundant hypervisor or Pfsense blade with TWO 10GB uplinks built in got me thinking….

As I researched it more, I found HP also had a wireless controller option, that could be had as cheap as $50 (or less) on eBay and was extremely plentiful (at the time I wrote this it was, now they are $150+ and not as common dirt cheap…).  It also appeared to have a large heatsink and DIMM slots + a SATA hardive option and a CF card slot.  Seemed PC like to me…  The part # may either be: msm765zl or J9154A

On a whim I ordered one to see what I could do with it….

The unit had no front serial/VGA connection and I’d never seen/used one before.  It arrived minus hard drive and a wiped CF card.  So I threw a USB drive in the front with VMWARE 5.5 on it, a kickstart file to auto install, and prayed for the best.

I followed the instructions from here: http://www.virten.net/2014/12/unattended-esxi-installations-from-an-usb-flash-drive/

Note that I needed a longer password than their example or it failed!

I waited about an hour, then reset the module by reseating it.  I checked and saw a new DHCP lease, so tried it an VIOLA!  I was in VMWARE!

Turns out the module has a Core 2 Duo T7500 processor at 2.2Ghz, and 4GB of ram.  I then added a normal SATA laptop hard drive, restarted the module and formatted that drive as VMFS5. (the auto install will install to the CF card by default).  You also get two Intel 82598EB 10GB AT CX4 network connections to the 5406ZL.

vmwareshot1

vmwareshot2

Then I hit a problem, the VWMARE windows management tool was timing out and slow to gain access, I couldn’t figure it out – I had a 10GB link why so slow….  I checked the switch and all looked ok, until I ran a sho run.  The running config had added 2 lines, each line was rate limiting the interface ports on the wireless controller (now vmware box) to 100kbps (I assume a safegaurd for new installs).  I simply issued a No rate-limit in command on both interfaces to undo what it auto-did, in my case interfaces E1 and E2 and all was good.

The bummer is that these things are limited to 4GB of ram, so pretty limited for VMWARE use.  However perefect for pfSense, or running Pfsense virtualized but redundant.  I went ahead and created a Trunk (team/port group depending on switches you are use to) that created a LACP group of the two 10GB interfaces then used VLAN’s and VMWARE vswitches to divy up the different networks I needed.  pfSense does install without a hitch and detects the 10GB interfaces just fine.

I decided to try and figure out as much of these little controllers as I could so I then started studying the circuit board and I found a 10pin connector labeled “VGA”.  After probing I was able to identify the ground pins (easy!) and 3 pins that had 70-75ohms of resistance to ground (my R/G/B signals).  All I had left was vsync and hsync.  I broke a VGA connector out to a bread board and on my first try had the pinout right (graphic below).  Ok not perfect, the image is fuzzy if anyone knows why let me know!  It was good enough to see what was going on.  I went to get into the BIOS and was presented with a “PASSWORD:” prompt – DOH!  But with some luck and the Hiren’s boot CD, I got the password, “PCMFG”.  That let me check out the bios settings, I ended up not changing anything but was interesting to poke around.  Here is a pinout for the header labeled VGA:

VGA Pinout

For now I plan to install pfsense bare-metal on the hardware and forgo VMWARE.  ESXi itself takes 1gb of ram idle with no VM’s installed/running and since I don’t own licenses for home use I can’t benefit from the cool failover features.

At this point you are probably asking – why the hell does this guy have a 5406ZL at his house?

I have the entire house networked.  It’s no Taj, but I have network up and downstairs in about every room.  I work from home and support the network of the company I work for, including hosting some “DR” servers at my home.  I need 24/7 VPN from the corporate office to my home for replication, but also want to isolate that traffic from my normal “home” network.  I have PBXinaFlash running a few PoE IP phones (1 has registrations to both corporate and home phone).  I use the IP phones built in intercom function to communicate from my office to upstairs and vice-versa.  I also have roughly 8 IP cameras for home security, two AP’s to cover the house, and a ton of network devices for other “stuff”.  I had purchased a 24Port managed POE switch and amazingly it cost me more than the 5406!  I got 4x the ports, redundant power, plus now a redundant pfsense install with 20GB of throughput.

Oh – and the managed switch w/GBICs requirement came after losing a TON of network equipment to lightning last year.  See this post on how and why I went this route…

 

Lightning-proofing the home LAN

So I work from home and the home network is important for a number of reasons (TV runs through it being the most lol).

Last summer while I was traveling I got a call from the wife that internet was down and the TV’s didn’t work.  About an hour later my neighbor texted a picture of a hole blown in the ground at his house by lightning.  A direct hit to the cable wire, it severed it!  No equipment in his house was damaged, but as I would find not true at my place!

Displaying IMG950676.jpgIMG950676

When I got home I found that though I had an enterprise grade APC battery backup/surge protection device lightning had found it’s way through both the cable modem AND my HD Homerun Prime.  The damage:

  1. Cable modem dead
  2. HD Homerun prime dead
  3. Asus wireless router dead
  4. 2 TP-link gig switches dead
  5. Media center PC on-board NIC dead (PC ok)
  6. Swaan security DVR on-bard NIC dead (unit worked at console though).

In all $700ish in equipment lost or broken.

I ordered replacements for all of it and got to thinking that I’d realllly like for that to not happen again.  So I did what any other (in)sane person would do and I optically isolated both the cable modem and HDHomerun Prime from my LAN.

My initial setup involved 4- TrendNET Media Converters w/2 multimode patch cables:

DCF 1.0

 

A rough diagram looked like this:

 

FiberConvertors

So now my network was optically isolated.  Lightning could take out the cable modem and/or the HD HomeRun, but not anything past it….  It cost me roughly $220 for the 4x converters + fiber cables….

One odd problem I had/have is that I can’t connect the HDHomerun directly to the media converter or the TV pixelates.  I have to put a dumb switch between it and the converter, still looking at that one….

At some point later I added a LOT of IP cameras (PoE) and IP Phones (also PoE) and began looking at managed switches.  I picked up a used HP ProCurve (lifetime warranty!) 24 port POE switch on ebay as well as two mini-gbic’s for fiber….

I bought a HP Procurve 2520G-24-POE:

procurve-2520g-24-port

You’ll see that that last 4 ports look odd, thats because they are shared minigbic ports.  I put two multimode transceivers in those ports.

I created an isolated VLAN for the internet on port 20 and port 24.  The cable modem network jack goes to a Media converter changing it to fiber, that plugs into port 24 on the HP, from there port 20 is connected to the WAN port on my Asus router and the switch allows the fiber to to convert back to copper while being on an isolated VLAN it keeps the internet/LAN traffic separate. The config for this is below: (you can do this in the GUI or SSH to the switch).

vlan 200
name “INET”
untagged 20,24
no ip address
exit

 

I did the same for the HDHomeRun Prime with the exception of leaving it on the default LAN VLAN. (Port 23 take the fiber from the converter to the switch, from there the device is talking to the LAN as usual.)

I later upgraded to a 5406zl – see my other posts about this….