Using libcurl: follow redirect and get header only

Let’s look at a sample code first(I use the WWW::Curl perl module here, the idea is the same for other language).

use warnings;
use strict;

use WWW::Curl::Easy;

my $url = '';
my $resp_body;

#Get file length via HTTP HEAD  request
my $length;
my $curl = WWW::Curl::Easy->new();
$curl->setopt(CURLOPT_URL, $url);
#follow redirect
#inlcude header in response
$curl->setopt(CURLOPT_HEADER, 1);
#do not include body in response
$curl->setopt(CURLOPT_NOBODY, 1);
my $retcode = $curl->perform();
if($retcode == 0){
        print "header:$resp_body\n";
        print "*" x 80,"\n";
        $length = $curl->getinfo(CURLINFO_CONTENT_LENGTH_DOWNLOAD);
        if($length == -1 ){
                print "content length not available\n";
        else {
                print "length: $length\n";
        print "error happened:$retcode " . $curl->strerror($retcode) ." | "
                . $curl->errbuf ."\n";
        exit 1;

There’s nothing complicated, the points are:

  • CURLOPT_FOLLOWLOCATION: set to 1 to follow HTTP redirect response
  • CURLOPT_HEADER: set to 1 to include HTTP Header in response
  • CURLOPT_NOBODY: set to 1 to exclude body content from response, so that we get header only
  • CURLOPT_WRITEDATA: you get HTTP headers here, more about this later

You may think that we should use CURLOPT_WRITEHEADER to collect the HTTP headers, but it doesn’t work,we get only the first response header in this way, not the final headers after several redirection.

check for more about libcurl

Posted in Perl, Programming | Tagged , , | Leave a comment

add utf8 subject support to mailwatch 1.0.5

In mailwatch v1.0.5, if the subject of a message is utf8 encoded, it can’t display correctly on the message list page. All the non-ascii characters will be replaced with question mark. For example. the subject “我爱Linux” will appears as “??Linux”.

We can patch mailwatch to support this:

1. patch

---    2011-08-19 14:15:45.565769650 +0800
+++        2011-08-19 15:08:47.343311859 +0800
@@ -284,7 +284,7 @@
    $msg{from_domain} = $message->{fromdomain};
    $msg{to} = join(",", @{$message->{to}});
    $msg{to_domain} = $todomain;
-   $msg{subject} = $message->{subject};
+   $msg{subject} = $message->{utf8subject};
    $msg{clientip} = $clientip;
    $msg{archiveplaces} = join(",", @{$message->{archiveplaces}});
    $msg{isspam} = $message->{isspam};

2. patch mailscanner/detail.php

--- detail.php.bak      2011-08-19 14:54:51.515734927 +0800
+++ detail.php  2011-08-19 15:04:17.003311710 +0800
@@ -147,7 +147,7 @@
    $output .= "\n";
    $row[$f] = $output;
-  if ($fieldn == "To:" || $fieldn == "Subject:") {
+  if ($fieldn == "To:") {
    $row[$f] = htmlentities($row[$f]);
   if ($fieldn == "To:") {
@@ -155,7 +155,7 @@
   if ($fieldn == "Subject:") {
    $row[$f] = decode_header($row[$f]);
-   //$row[$f] = htmlentities($row[$f]);
+   $row[$f] = htmlentities($row[$f],ENT_COMPAT,'utf-8');
   if ($fieldn == "Spam Report:") {
    $row[$f] = format_spam_report($row[$f]);
Posted in Email, System Administration | Tagged | Leave a comment

Apache options directive trap

When configuring the Apache HTTP server, we use the Options directive to control which server features are available in a particular directory. We may, for example, write a config like this:

<Directory /usr/share/nagios>
    Options +ExecCGI Indexes FollowSymLinks

With this configuration, we hope to enable CGI if it has not been enabled, and set auto index and follow symlink option. However, this is completely wrong and will not work. when you access a cgi in this directory, you will get a 403 Forbidden page.
Continue reading

Posted in System Administration, Web | Tagged , | Leave a comment

Build the Linux kernel for other host

Building the linux kernel is time and resource consuming, especially when you do this in a VM guest.

fortunately, We can choose a fast Linux machine to take the burden of building, then install the kernel to the target Linux host

This simple article assumes that you know how to config and compile your custom kernel. so I will focus on building the kernel for other host.

1. Download the linux kernel from extract, make xxxconfig, then make, assume that the directory you build your kernel is called KERNEL_BUILD_DIR
Continue reading

Posted in System Administration | Tagged , , | Leave a comment

Install from source or use rpm, this is a question

For RHEL and its derived distribution, installing software packages using rpm or yum is sometimes convenient, painless and time saving. However, because distribution vendors build rpms independent of software provider, there may be time delay, and even inconsistency between rpms and the original software package.

Here’s what I come across today. I installed ProFTPD from the epel repo on two servers. One is CentOS 6.0 and the other is CentOS 5.6.

#on CentOS 6.0
[curu@el6 ~]$ rpm -q proftpd
#on CentOS 5.6
[curu@el5 ~]# rpm -q proftpd

Seems they are of the same version. However, When I copy the proftpd.conf from el6 to el5, I get this error

proftpd -t
Checking syntax of configuration file
 - Fatal: VRootAlias: source path 'etc/security/pam_env.conf' is not an absolute path on line 11 of '/etc/proftpd.conf'

And here’s line 11 of my /etc/proftpd.conf

VRootAlias                      /etc/security/pam_env.conf etc/security/pam_env.conf

According to the mod_vroot document on proftpd site

Syntax: VRootAlias src-path dst-path

It’s obvious that my config is right, the src-path here is /etc/security/pam_env.conf, definitely absolute! But what’s wrong? Ask the source!

In order to find the cause of this inconsistency, I downloaded the source rpms of these two rpms, and finally find why. In fact, in the RPMs from EPEL, proftpd-1.3.3e-1.el5 comes with mod-vroot-0.8.5, whereas proftpd-1.3.3e-1.el6 comes with mod-vroot-0.9.2. The bad thing is, the directive syntax of VRootAlias has changed between these two version of mod_vroot.

for mod-vroot-0.8.5

Syntax: VRootAlias dst-path src-path

Damn! the change is completely nonsense and incompatible.

So, what can when draw from this story? Install software packages from prebuilt rpm is not always simple and good. If we need more configure option and consistent behavior, build from source.

Posted in System Administration | Tagged , | Leave a comment

Shorewall: allow communication between VPN clients

If you setup Linux as a PPTP or L2TP/IPSec VPN server, every client connection will have a corresponding pppx interface on the server. You may have a shorewall interface config like this:


#zone interface boradcast options
l2tp    ppp+    -       

Assume that client A connected, get IP for ppp0, client B get IP for ppp1. When client A try to ping client B. you may get the following shorewall log:

Jul  8 00:19:20 vpngateway kernel: Shorewall:FORWARD:REJECT:IN=ppp0 OUT=ppp1
 SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=311

In this case, if you really think that the connected VPN clients should be able to communicate with each other, you may add the routeback option to shorewall-interface config file, like this:

#zone interface boradcast options
l2tp    ppp+    -       routeback

Reference: shorewall-interfaces man page

Posted in System Administration, VPN | Tagged , , , | Leave a comment

Fedora 15 下安装 Openfetion 2.2

本文简要描述在Fedora 15系统下编译安装Openfetion 2.2的办法


1. 安装所需软件开发包

sudo yum install cmake libnotify-devel gstreamer-devel \
	NetworkManager-develgtk2-devel openssl-devel libXScrnSaver-devel

Continue reading

Posted in Linux Desktop | Tagged , , | 1 Comment

Create new KVM guest from template

Most of us don’t like to install guest OS repeatedly, instead, we often install one guest OS, then do some setup and customization. After that, we make a backup of the disk image(We use it as a template). If we want to install the same OS later, we only need to copy that template disk image and create a new guest config file.

When create a new guest config file, we need to give it a unique name, a unique uuid, and a unique MAC address. Also, we need to change it’s disk file path. We can do this kind of changes manually, hower, according to the rule “automate all that can be automated”, we can use a script to do this.
Continue reading

Posted in Programming, Python, System Administration, Virtualization | Tagged | 3 Comments

A convenient Bash function to backup configuration file

For sysadmins, it’s a good practice to backup configuration files before changing them. We normally do it via cp. eg:

cp named.conf named.conf.bak
cp named.conf named.conf.orig

I personally like to use the date time as the file name suffix, so that I can tell from the file name when I did that backup. Also, with this method, multiple version of the config file can be kept.

Because I use this method frequently, I wrote a Bash function to do this:

        for f in "$@"
                do cp -a "$f" "$f".$(date +%Y%m%d%H%M)

Add it to my ~/.bash_profile . And run

source ~/.bash_profile

Then every time I need to backup a file, I just need to do this

b /etc/mail/
#this will create a backup file named like /etc/mail/
cd /var/named/chroot/etc/
b named.conf #relative file name also works
b /etc/postfix # you can also backup a whole directory

In fact, its usage is not limited to configuration file, you can use it to backup whatever you like.

Posted in Programming, System Administration | Tagged , | Leave a comment

Install fedora 15 from usb

This assumes that the capacity your USB drive is no less than 4GB.

  1. Install grub4dos to the USB.
    Use grubinst to install grub4dos MBR to the USB, then copy the grldr file in grub4dos package to the root directory of the USB drive. See DoIT’s post for instructions.
  2. Put Installation media to the USB
    Download the fedora DVD iso image, extract vmlinuz and initrd.img in the isolinux directory to the root of the USB drive, and then put the iso itself to the root of the USB drive
  3. Boot from the USB
    Boot from the USB, you will be dropped into a grub shell, type the following command to start the install.

    root (hd0,0)
    kernel /vmlinuz askmethod
    initrd /initrd.img

    The will start the fedora installer, when it comes to the install method selection, select Local drive, and select your usb drive partition to continue.

Discussion: Why not simply use UNetbootin? 1st, Fedora 15 is not yet listed on the homepage, maybe it is not supported yet. 2nd, the most important reason, UNetbootin will copy the contents of the ISO image file by file to USB, which is quite slow. If drop the iso image itself to USB, it will be much faster.

Reference: Official Fedora 15 Installation Guide

Posted in Linux Desktop | Tagged , | Leave a comment