I hate situations like the one I faced this afternoon… I’ve a daemon, written in Perl, running on a Linux box for months now. When I say for months, it means: Without restarting the process or the host except regular upgrades. But, for maintenance reasons, I had to reboot the box and… my Perl daemon suddenly stopped working!
First reflex, checking out the logs… Nothing! (Thanks Murphy) I hate this even more! Checking the daemon configuration files, nothing suspicious. Network connectivity was available. I tested the same code on a similar system, it worked! The problem was definitively on my server. Let’s execute the code in debug mode. This revealed that the problem was caused by the IO::Socket::NET Perl module. It did not resolve host names to IP addresses anymore. I wrote a quick Perl script, same problem. Conclusion: it was not the daemon, but really the Perl module.
What did change? My host is monitored by OSSEC. Let’s have a look at the changes performed since the last days.
- No patch was installed
- No new packages/software were installed
- My resolv.conf changed!
Indeed, a few weeks ago I switched more home services to IPv6 and enabled the resolution of AAAA records by adding the option “inet6” in resolv.conf. I did a rollback to the previous version of this file and bingo: problem solved! As the Perl daemon was running for a while and just resolved hostname during its initialization phase, the problem never occurred before. It seems that my Perl IO::Socket::NET module does not supports this option.
Some conclusions:
- Log errors properly! Even if the code is broken, they’re methods to catch errors. There is nothing more annoying than code ending without information
- Monitor changes on your hosts! In the example above OSSEC was very useful to track the changes performed during a specific period of time.
- Dear Developers, IPv6 is also a challenge for you! My little story is a good example of how a software could be affected after switching to an IPv6 environment. I don’t blame you, just be careful! Even if your application does not directly access network resources, you have to make they “IPv6 Ready”. It’s like new car plates or telephone numbers formats. Applications must be able to handle IP addresses not only as “x.x.x.x”.
- Changes performed at a time “X” may cause problems at a time “X+Y” (Y being quite long).
Are your applications “IPv6 Ready“? Sure?
I told it before and I will keep telling everybody ..
Everything is a ufcking dns problem.
And IPv6 will only make it “worse” 🙂