SimpleDB: python MySQLdb wrapper with auto connect

You may get the simpledb.py module source on google code.

Usage:

Create SimpleDB object and connect to database

Note: you must have MySQLdb module installed first.

import simpledb
db = simpledb.SimpleDB(host='localhost', user='test_user', password='test_password', db_name='test')

Select one row from db as a list

>>> db.get_one('SELECT * FROM users')
(1L, u'bill gates')

Select one row from db as dict

>>> db.get_one_dict('SELECT * FROM users')
{'user_name': u'bill gates', 'id': 1L}

Select all row from db as a list of rows in list

>>> db.get_all('SELECT * FROM users')
((1L, u'bill gates'), (2L, u'linus torvalds'))

Select all row from db as a list of rows in dict

>>> db.get_all_dict('SELECT * FROM users')
({'user_name': u'bill gates', 'id': 1L}, {'user_name': u'linus torvalds', 'id': 2L})

execute update/insert statement which does not return result set, commit changes to database

>>> db.execute_sql('INSERT INTO users(user_name) VALUES(%s)', 'Dennis Ritchie')
>>> db.get_all('SELECT * FROM users')
((1L, u'bill gates'), (2L, u'linus torvalds'), (3L, u'Dennis Ritchie'))

Note: if you don’t want to commit SQL, eg. you want to use transaction, you may use the execute_sql_no_commit function:

db.exuecte_sql_no_commit('START TRANSACTION')
db.exuecte_sql_no_commit('xxx...')
db.exuecte_sql_no_commit('yyy...')
db.commit()

Note, the auto re-connection is performed behind the scenes: If a DB operation failed, it will try to reconnect to the DB and then execute the same operation once again.

Posted in Programming, Python | Tagged | Leave a comment

A python exception retry decorator

We usually call some API functions which may throw exception because of unexpected condition. These unexpected condition include but not limited to service too busy, network packet loss, etc. and it would be OK if we retry again.

So we need a decorator for this purpose, then we can easily make the API calls more stable.

def retry(tries, delay=1, backoff=2):
    """
    A retry decorator with exponential backoff,
    Retries a function or method if Exception occurred

    Args:
        tries: number of times to retry, set to 0 to disable retry
        delay: initial delay in seconds(can be float, eg 0.01 as 10ms),
            if the first run failed, it would sleep 'delay' second and try again
        backoff: must be greater than 1, 
            further failure would sleep delay *= backoff second 
    """
    import time
    import math

    if backoff <= 1:
        raise ValueError("backoff must be greater than 1")

    tries = math.floor(tries)
    if tries < 0:
        raise ValueError("tries must be 0 or greater")

    if delay <= 0:
        raise ValueError("delay must be greater than 0")

    def decorator(func):
        def wrapper(*args, **kwargs):
            _tries, _delay = tries, delay
            _tries += 1 #ensure we call func at least once
            while _tries > 0:
                try:
                    ret = func(*args, **kwargs)
                    return ret
                except Exception as e:
                    _tries -= 1
                    #retried enough and still fail? raise orignal exception
                    if _tries == 0: raise
                    time.sleep(_delay)
                    #wait longer after each failure
                    _delay *= backoff
        return wrapper

    return decorator

Usage example:

@retry(3, delay=0.1)
def give_me_five():
    n = random.randint(1, 10)
    print "got #: %s" % n
    if n != 5:
        raise ValueError("Please give me five")
    return n

n = give_me_five()
print "main got:%s " % n
Posted in Programming, Python | Tagged , , , | Leave a comment

some fabric trap and tricks

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks. Here’s some trap and tricks I met and got from my usual use.

trap 1: operation error  cause execution abort
fabric operations like run or put, would cause task execute to fail when error occured.
trick:
run fab with -w option, or turn on warn_only temporarily with code like this:

with settings(warn_only = True):
    run('some commmand')

trap 2. low level network error cause execution abort
Low level network error like name resolve error, connection timeout, connection refuesed, would normally cause execution abort.
trick:
catch NetworkError exception like this:

from fabric.exceptions import NetworkError
#...
@task
def some_task():
    try:
        whatever()
    except NetworkError as e:
        print e

trap 3. too many connection/thread cause execution abort
Fabric will keep a connection cache for performance, but when you are running task on thousands of hosts, Fabric may run out of resource(especially reach thread limit) with error message like thread.error: can’t start new thread
trick:
turn on eagerly_disconnect setting to close connection resource as soon as possible

env.eagerly_disconnect = True

Posted in Programming, Python, System Administration | Tagged , | Leave a comment

how to build v8 static library

You can generate shared v8 library by following the wiki Building With GYP, but how about static library?

Note: use of static library is discouraged, because it wastes disk space and memory, and is hard to upgrade. Continue reading if you really want this.

In fact, the “.a” files of the default build is not actually ar static library, it’s “thin archive” format which contains reference to other object files  instread of packing object files together in one file. That’s to say, if you copy only the .a files to other people, they will not be able to compile because the referenced object files are missing.

In order to get real static libraries that are easy to distribute, we need to change two gyp files, add the “standalone_static_library” = 1 option,  (the changes made here are based on v8-3.24.5 revision: 18377).

Changes to tools/gyp/v8.gyp:

--- tools/gyp/v8.gyp    (revision 18377)
+++ tools/gyp/v8.gyp    (working copy)
@@ -107,6 +107,7 @@
     {
       'target_name': 'v8_snapshot',
       'type': 'static_library',
+      'standalone_static_library': 1,
       'conditions': [
         ['want_separate_host_toolset==1', {
           'toolsets': ['host', 'target'],
@@ -179,6 +180,7 @@
     {
       'target_name': 'v8_nosnapshot.<(v8_target_arch)',
       'type': 'static_library',
+      'standalone_static_library': 1,
       'dependencies': [
         'v8_base.<(v8_target_arch)',
       ],
@@ -236,6 +238,7 @@
     {
       'target_name': 'v8_base.<(v8_target_arch)',
       'type': 'static_library',
+      'standalone_static_library': 1,
       'variables': {
         'optimize': 'max',
       },

Changes made to third_party/icu/icu.gyp

--- icu.gyp     (revision 239289)
+++ icu.gyp     (working copy)
@@ -53,6 +53,7 @@
         {
           'target_name': 'icudata',
           'type': 'static_library',
+          'standalone_static_library': 1,
           'defines': [
             'U_HIDE_DATA_SYMBOL',
           ],
@@ -123,6 +124,11 @@
         {
           'target_name': 'icui18n',
           'type': '<(component)',
+          'conditions': [
+            [ 'componen!="shared_library"', {
+                'standalone_static_library': 1,
+            }],
+          ],
           'sources': [
             'source/i18n/anytrans.cpp',
             'source/i18n/astro.cpp',
@@ -366,6 +372,11 @@
         {
           'target_name': 'icuuc',
           'type': '<(component)',
+          'conditions': [
+            [ 'component!="shared_library"', {
+                'standalone_static_library': 1,
+            }],
+          ],
           'sources': [
             'source/common/bmpset.cpp',
             'source/common/brkeng.cpp',

make clean and make again, you will get the real static library, the top two large files are:
libv8_base.x64.a(name may differ depends on your build) of 15MB size and libicudata.a of 9M size.

Note: earlier version of gyp didn’t have support for building standalone_static_library, you may need to update it to the latest version first:

cd build/gyp
svn up
cd -
Posted in Programming | Tagged , , | 1 Comment

out-of-date document hurts

The headache of building google V8 Javascript Engine

We have a project which would make use of the google V8 engine to parse and execute javascript.

I followed the downloading and building V8 doc and successfully built v8 library. However, I failed to follow “Getting Started” guide to compile the hello world example.

g++ -Iinclude -pthread -o hello_world hello_world.cc lib64/libicudata.a lib64/libv8_base.x64.a lib64/libv8_nosnapshot.x64.a lib64/libv8_snapshot.a
hello_world.cc: In function 'int main(int, char**)':
hello_world.cc:23:27: error: 'New' is not a member of 'v8::String'
hello_world.cc:32:22: error: 'class v8::Persistent' has no member named 'Dispose'
hello_world.cc:35:3: error: 'AsciiValue' is not a member of 'v8::String'
hello_world.cc:35:22: error: expected ';' before 'ascii'
hello_world.cc:36:19: error: 'ascii' was not declared in this scope
make: *** [hello_world] Error 1

I am sure that this has nothing to do with library, it may be caused by prototype mismatch. But why failed? I trusted the docs, it should work!

After several hours of experiment and googling, I gave up. tried to checkout an older version instead(here I checked out version 3.19.18.9 from svn tag http://v8.googlecode.com/svn/tags/3.19.18.9). Compile the example ode again, everything works like magic!

Note, when build the lastest version(currently 3.24.5), you may also encounter error like this:

make[1]: Entering directory `/home/curu/src/v8/out'
  CXX(target) /home/curu/src/v8/out/x64.release/obj.target/v8_base.x64/src/hydrogen.o
cc1plus: warnings being treated as errors
/usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/stl_algo.h: In member function 'void v8::internal::HOptimizedGraphBuilder::HandlePolymorphicCallNamed(v8::internal::Call*, v8::internal::HValue*, v8::internal::SmallMapList*, v8::internal::Handle)':
/usr/lib/gcc/x86_64-redhat-linux/4.4.6/../../../../include/c++/4.4.6/bits/stl_algo.h:2178: error: array subscript is above array bounds
make[1]: *** [/home/curu/src/v8/out/x64.release/obj.target/v8_base.x64/src/hydrogen.o] Error 1
make[1]: Leaving directory `/home/curu/src/v8/out'
make: *** [x64.release] Error 2


We can see that the error information is related to gcc itself, update GCC to 4.6 should solve this prolem.

The out-of-date document hurt me a lot. Great software should come with great and up-to-date document, espically for Open Source Software.

Posted in Programming | Tagged , | 3 Comments

Nginx userdir with php

you come here with this problem, here’s the solution:

location ~* ^/~(.+?)(/.*\.php)$
{
	alias /home/$1/public_html$2;
	fastcgi_pass  127.0.0.1:9001;
	include fastcgi_params;
	fastcgi_param SCRIPT_FILENAME $request_filename;
}

location ~ ^/~(.+?)(/.*)?$ {
	alias /home/$1/public_html$2;
	index  index.html index.htm index.php;
	autoindex on;
}

Assumption:

  • user’s public html / php files are located at /home/username/public_html directory
  • php is running as php-fpm which listen on port 9001

There’s some solution on the Internet which tried to use nested location

location ~ ^/~(.+?)(/.*)?$ {
	alias /home/$1/public_html$2;
	index  index.html index.htm index.php;
	autoindex on;

	location ~ .+\.php$ {
		fastcgi_pass  127.0.0.1:9001;
		include fastcgi_params;
		fastcgi_param SCRIPT_FILENAME $request_filename;
	}
}

This won’t work, because nginx refer variables by name, and in the nested location, $request_filename(which is a result of alias) refers regex capture $1 and $2, both value are empty. As a result, $request_filename will always be /home//public_html, definitely wrong.

Posted in Web | Tagged , , | Leave a comment

nginx – try files on multiple named location or server

Let’s start from the simplest case: How to serve files locally first, and proxy the request to other backend/internal server if not found?

listen 192.168.1.10:80;
...
location /static/ {
    try_files $uri @static_svr1;
}

location @static_svr1{
    proxy_pass http://192.168.1.11$uri;
}

That’s simple! What if we want to try files on multiple backend server?
Say, if files do not locally exist, try static_svr1, if not found there, try static_svr2, if still not found, try static_svr3, and return 404 if finally no luck.

We know that there can be only one named location for the try_files directive. so it can’t help in this case.

Fortunately, from the nginx documentation:

...
In this example, the directive try_files

try_files $uri $uri/ /index.php?q=$uri&$args;
Is basically the same as this:

location / {
  error_page     404 = @drupal;
  log_not_found  off;
}

location @drupal {
  rewrite ^ /index.php?q=$uri last; # for drupal 6
}

From the example, it may be possible to use error_page directive chain to implement our setup:

location /static/ {
	try_files $uri @static_svr1;
}
location @static_svr1{
	proxy_pass http://192.168.1.11$uri;
	error_page 404 = @static_svr2;
}

location @static_svr2{
	proxy_pass http://192.168.1.12$uri;
	error_page 404 = @static_svr3;
}

location @static_svr3{
	proxy_pass http://192.168.1.13$uri;
}

However, this doesn’t work properly. let’s see what would happen with the above config.
Assuming that file /static/0.html is on static_svr0, …, /static/3.html is on static_svr3. Here’s the result:

URL Accessed status code
http://192.168.1.10/static/0.html 200
http://192.168.1.10/static/1.html 200
http://192.168.1.10/static/2.html 404
http://192.168.1.10/static/3.html 404

Only static_svr1 works as expected. Also, the access_log of static_svr2 shows that 192.168.1.10 never tried to access it. Why? Because there’s no 404 error_page processing for proxy_pass unless proxy_intercept_errors is on. let’s add it and test again:

location /static/ {
	try_files $uri @static_svr1;
}
location @static_svr1{
	proxy_pass http://192.168.1.11$uri;
	proxy_intercept_errors on;
	error_page 404 = @static_svr2;
}

location @static_svr2{
	proxy_pass http://192.168.1.12$uri;
	proxy_intercept_errors on;
	error_page 404 = @static_svr3;
}

location @static_svr3{
	proxy_pass http://192.168.1.13$uri;
}
URL Accessed status code
http://192.168.1.10/static/0.html 200
http://192.168.1.10/static/1.html 200
http://192.168.1.10/static/2.html 200
http://192.168.1.10/static/3.html 404

static_svr2 now works! But why not static_svr3?  The log  also suggests that it’s never accessed? Maybe the error_page directive doesn’t work for static_svr3, search nginx documentation with keyword error_page, I find the very recursive_error_pages directive, it mentioned the word “chain”, that’s just what we want.  Adjust the config again:

location /static/ {
	try_files $uri @static_svr1;
}
location @static_svr1{
	proxy_pass http://192.168.1.11$uri;
	proxy_intercept_errors on;
	recursive_error_pages on;
	error_page 404 = @static_svr2;
}

location @static_svr2{
	proxy_pass http://192.168.1.12$uri;
	proxy_intercept_errors on;
	recursive_error_pages on;
	error_page 404 = @static_svr3;
}

location @static_svr3{
	proxy_pass http://192.168.1.13$uri;
}

Access the test URLs again, now all return status code 200, perfect!

Posted in System Administration, Web | Tagged , , , , | 2 Comments

LinuxMint – Why package manager failed to start

I haven’t used my Linux Mint maya for about half year.  Today I logged in,  did a system update with the following command:

sudo apt-get update
sudo apt-get dist-upgrade
sudo reboot

Then when I logged in again and tried to remove the old kernels with the Package Manager, it just died after prompting me for the password, without saying a word. That’s quite rude. I tried other application that may require root privilege like Login Window and Update Manager, they won’t start too!

So, what’s the problem? Why they all didn’t work? I need to get more diagnostic message.

First, find out what command will be executed when we click the  Package Manager menu

Package Manager
Continue reading

Posted in Linux Desktop | Tagged , , , , , | 2 Comments

who attach shared memory

Want to know which shared memory segment is attached by which processes ?

Want to know the shm segments a process attached on ?

Try who_attach_shm.pl , just download and run.(update, lsof is no longer required)

[curu@linuxplayer.org]$ ./who_attach_shm.pl
##################################################################
shm attach process list, group by shm key
##################################################################

0x2d5feab4:    /home/curu/mem_dumper /home/curu/playd
0x4e47fc6c:    /home/curu/playd
0x77da6cfe:    /home/curu/mem_dumper /home/curu/playd /home/curu/scand

##################################################################
process shm usage
##################################################################
/home/curu/mem_dumper [2]:    0x2d5feab4 0x77da6cfe
/home/curu/playd [3]:    0x2d5feab4 0x4e47fc6c 0x77da6cfe
/home/curu/scand [1]:    0x77da6cfe
Posted in Perl, System Administration | Tagged , , , | Leave a comment

Change shared memory ownership and permission

We know that we can use ipcs to view shared memory segment information,we can use ipcrm to delete a segment.

But wait. Assume that you first run a program as root user,  which creates a shared memory segment of 40GB size. Then you realize that for security resaon, we should use as less privilege as possible, so you decide to run it as the nobody user. However, because the segment was created with perm 0600, the nobody user can’t access it. You definitely don’t want to dump the whole memory and recreate a new segment and recover.  It would be nice if we can simply change the ownership and permission of the  segment.

Continue reading

Posted in Programming, System Administration | Tagged , | Leave a comment