Strange network troubles with NFS and VMware

Last week i add a new NFS storage to our environment and from that moment on i got some strange situations. The storage is intended for our lab but production continues to grow so with that in mind i wanted to add the storage to all ESX hosts and not only the host that is dedicated to Labmanager. The environment looks like this:

 

simple network diagram 

I have 4 host all have two network cards dedicated for connection to the storage and already 1 NFS storage box is connected. So i thought this is done in a blink of a eye. I have a power shell scrip and copied this one en rebuild is for the second NFS storage. and i pushed the button. Naturally i already prepared the storage system. when I came back with my cup of tea i saw that only ESX host number 3 did connect with the NFS storage box. I play with the storage box and ESX3 for a will before i have added to the environment so i went back to the storage box and checked if i had gave all the Hosts the appropriate permissions  everything was as it should be.

Ok next step vmkping that must work because i had already one NFS storage connected to all Hosts. I could vmkping tot all hosts from all host from except of ESX3 this host could not ping to each network card of all host and not to NFS2. Always the second network card (vmk2) did not respond.  And i could not ping from the NFS server to ESX3. Oke what is going on here? Because i played with ESX3 i decided to reinstall this host to be sure that is is the same as the other three. But this had the same outcome. Oke i also played with the the NFS server so start over again by the manual ( it is the same system as NFS1)  to be sure that it is correctly installed and configured. But it had the same outcome. I saw the the NFS storage had extra network card  and when i wanted to disabled the network cards that are not used i accidentally disabled the card  of the storage lan. Oeps I enable and what do i see. I had connection to ESX 1,2 and 4 but not to ESX3! Now i am getting slightly mad! I rebooted the storage system and i had connection to ESX3 but not to ESX1,2 and 4. Oke disable and enable the network card and…. Yes i had connection to ESX1,2, and 4 but not with ESX3. I had check everything i could think of network configuration and routing table (route –n and route –C) on the NFS storage and ESX host. I thought it would not be the switch…. No it cant be.  Oke to be sure and yes configuration was oke but when i switch the network cable in to a other port al 4 ESX host had connection! i build a network team on the NFS host and run the script again an yes all host have the new Datastores connected.

This was a long journey it has been a will since i had a broken switch port it also took a will to figure this one out.