More than two weeks ago I blogged about my server being down. After multiple emails, phone calls, and even a fax trying to reach the support team, the server is still dead. But at least I know (a little bit) more now.
I managed to get someone from support on the phone and he fixed the system at least so far that I could ssh to it again. I was able to pull a complete backup of the system, including a database dump.
That means that Unmaintained Free Software and all other sites hosted on the server will eventually return, no data will be lost.
After I created the backup, I wanted to reinstall the whole system and then install the backup to restore all services. As it turned out, the (automatic) reboot- and reinstall-script they use is obviously broken, I cannot reach the server anymore after I initated the reinstall. This is probably something more serious, as other people seem to be affected, too.
I have not the slightest idea what the hell happened on the server. There was something really, really strange going on. An example:
# ls -l /usr/bin/traceroute
-rw-rw---- 1 mysql mysql 310872 Jun 21 03:21 traceroute
Why the hell is
traceroute not executable and belongs to user/group
mysql? There are several other anomalies there:
/usr/share/doc/apt is not a directory as it is supposed to be, but a Perl script.
/usr/bin/id is a directory. Multiple system tools (awk, sed, ...) are not executable and partly directories with strange stuff in them. What gives?
One possible explanation is that the server was hacked and some rootkit wrecked havoc on the server. After a quick glance at the logs, I couldn't find any hints for a successful breakin, though. Another possibility is that the hard drive simply died and/or the filesystem was (heavily) corrupted. I don't know...
Has anybody ever seen something like this? Please enlighten me what could have happened...
It features a good historical introduction of the Free Software movement in general and discusses topics such as geeks, nerds, hacking etc. The main part is about the social aspects, though, especially motivation and commitment in the Free Software community.
The results of four interviews with famous Free Software developers are a major part of the thesis. There developers are:
I'm through half of the thesis now and really recommend reading it, as it gives some interesting insights into the social aspects of Free Software development.
(via Harald Welte's blog)
I have just uploaded fw_laptop, the firewall script I use to secure my laptop and/or desktop machines (but not my servers), in the Security section of my homepage. The script is GPL'ed and work in progress. I'm happy to receive feedback and/or corrections and suggestions regarding the script.
I will publish other similar scripts for different purposes when time permits.
Linux 18.104.22.168 has been released, which fixes two locally exploitable security issues. Another similar bug has been fixed recently by 22.214.171.124.
Every Linux box with local users should be upgraded ASAP. My laptop is already compiling away happily (although I'm the only one who has an account on it of course, but you never know)...