Most network monitoring relies in the individual network devices themselves telling you that they are healthy or unhealthy via syslog messages, SNMP data, etc. In a Facebook scale network we just can’t trust the network devices to accurately report their health in all the possible failure cases that may exist. In addition to the standard network monitoring tools, we also actively probe our network with test traffic to ensure it’s behaving exactly as we expect. We can now find the network devices that don’t even know they are dropping packets even when they exist several layers deep inside the network.