Database performance in Web applications

by Stéphane

29 10 2008

It’s more efficient to connect a Web application with an Unix Domain
Socket than TCP/IP one (reduced overhead) so I’ll explain the required
configuration with the following pairs:
1 - TurboGears/SQLAlchemy
2 - Django/PostgreSQL
3 - Django/MySQL

1 - TurboGears/SA

SQLObject is dead, isn’t it? So With SQLalchemy, the syntax is:

sqlalchemy.dburi="postgres:///dbname?user=mydbuser&password=XXXXXX" ([1])
http://docs.turbogears.org/1.0/DatabasePostgres

2 - Django/PostgreSQL

You just need to define DATABASE_ENGINE = ‘postgresql_psycopg2′ and DATABASE_NAME. Leave DATABASE_HOST setting empty to use UDS.

3 - Django/MySQL

Create a database in UTF8, either with default-character-set = utf8
under [mysqld] section in the my.cnf file or with an explicit ‘create
database bla charset=utf8;’

In settings.py:


DATABASE_HOST = '/var/run/mysqld/mysqld.sock'
DATABASE_OPTIONS = {
'read_default_file': '/etc/mysql/my.cnf',
'init_command': 'SET storage_engine=INNODB'
}

A - Note about PostgreSQL

When the user isn’t the same one who runs the process, you must edit the PostgreSQL configuration (/etc/postgresql/8.3/main/pg_hba.conf):
# "local" is for Unix domain socket connections only
local   user     database      md5
local   all      all           ident sameuser

You must create an user with a encrypted password (encrypted by default).

$ CREATEUSER username
$ psql
postgres=# ALTER USER username WITH ENCRYPTED PASSWORD 'my_password';

If you want to be sure, remove the lines with ‘host’ to deny nonlocal connections.



Vertimus 1.0.2

by Stéphane

22 06 2008

Bugfixes and new translations.

To upgrade from previous version, run the following scripts:

  1. sql/upgrade-sql-vtm-1.0.1-to-1.0.2.sql
  2. sql/upgrade-sql-vtm-1.0.1-to-1.0.2.php

* MAJOR FIX to get the last user who has locked the module
* New $url_hosted_link and $url_hosted_name variables to set in
localconfig.inc.php
* FIX Sending of emails to the committers on ‘Ready to commit’
* New Brazilian Portuguese translation by Leonardo Ferreira Fontenelle
* New Czech translation by Lucas Lommer (#237486)
* FIX #237472 reported by Lucas Lommer
VARCHAR > 255 needs MySQL 5.0.3 or later



Saving power with Linux and reducing the boot time

by Stéphane

12 06 2008

There are a great web site and some tools about this subject (thanks to Intel and their great developers)
http://www.lesswatts.org

Services

The first step is to remove some useless services or rarely used, I chosen the following ones :

apt-get remove bluz-utils hplip* scim* tracker*

update-rc.d -f pcmcia remove, for mysql, apache2, postgresql-8.3 (it’s easy to launch when necessary)

You can really check the impact of your changes with bootchart and view the result with eog /var/log/bootchart/*.png (my laptop starts in 27 seconds).

In gnome-session-properties, I also unchecked Applet Tracker, Evolution Alarm Notifier, Bluetooth Manager (don’t forget to save in the third tab).

Settings

You can measure the number of wakeups per second with the wonderful PowerTop ($ sudo powertop), on my computer the main guilty was the proprietary nvidia driver (around 60 fps like the refresh rate of my screen), I added the following line in my xorg.conf to resolve that issue (~ 2 wps after) :

Option         “OnDemandVBlankInterrupts” “true”

I also blacklisted some modules in /etc/modprobe.d/blacklist

blacklist pcmcia
blacklist yenta_socket
blacklist rsrc_nonstatic

With no Wifi and no USB mouse connected, with my GNOME Desktop, a gnome-terminal and Emacs, I have only 20 wakeups per second (not bad :).

I wrote a little script to test some more aggressive settings if they works fine I will add them to sysctl.conf:

# By setting this to ‘1′, under light load scenarios, the process
# load is distributed such that all the cores in a processor package
# are busy before distributing the process load to other processor
# packages.
echo 1 > /sys/devices/system/cpu/sched_mc_power_saving

# From 500 by default
echo 1500 > /proc/sys/vm/dirty_writeback_centisecs

# SATA
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy

# Sound
echo 1 > /sys/module/snd_hda_intel/parameters/power_save

I don’t use the laptop_mode or hdparm because I don’t want to stress my hard drive with too many spin up and down. It’s not necessary to insert the ‘noatime’ option in fstab with Ubuntu Hardy Heron because the new ‘relatime’ option is already activated.



libmodbus 2.0.0 “Slaves to Our Machines” is out!

by Stéphane

18 05 2008

libmodbus is library to send/receive data to/from ModBus devices which respect the protocol established by Modicon.

This new release contains major enhancements like the slave component.
A special thanks to Todd Denniston for this release.

The Ubuntu users can obtain the packages of this project here:
https://launchpad.net/~sra/+archive

libmodbus 2.0.0 (2008-05-18)
============================
- Slave API
https://blueprints.launchpad.net/libmodbus/+spec/slave-api
- No more glib dependency
https://blueprints.launchpad.net/libmodbus/+spec/glib-dependency
- Unit testing and many test programs
- Waf build support
https://blueprints.launchpad.net/libmodbus/+spec/waf-support
- MacOS X support by Matthew Butch
https://blueprints.launchpad.net/libmodbus/+spec/macosx-support
- No more glib dependency
https://blueprints.launchpad.net/libmodbus/+spec/glib-dependency
- Unit testing (unit-test-slave and unit-test-master)
- Port number is now defined at initialisation by Dirk Reusch
- Better memory management (uint8_t *data and packing of
modbus_param_t)
- Better error management
- Declare many static functions and const arrays
- Enhance an integer division
- The GNU licences LGPL and GPL are in version 3
- Debian and RPM packages (#224496)
- Many cleanups
- Fix #159443 reported by Stefan Bisanz
Index of incoming data in force multiple coils function
- Fix #161989 reported by Konstantinos Togias
Serial device paths more than 10 chars long (eg. /dev/ttyUSB0) don’t
fit to modbus_param_t -> device char[11] var.
- Fix #188189 reported by Chris Hellyar
Compute_response_size() no entry for read_input_status()
- Fix #191039 reported by Todd Denniston
modbus.h is not installed at prefix.
- Fix #211460 reported by Todd Denniston
With TCP, automatic reconnect on error may not be desired.
- Fix #224485 reported by Todd Denniston
libmodbus does not link with c++ code.
- Fix #224496 reported by Todd Denniston
It is easier to install on rpm based systems with a spec file.



Reduce open calls on Ubuntu

by Stéphane

13 05 2008

You’ve installed a fresh Ubuntu and you’re not English. Just try:

$ strace gnome-terminal

and you’ll see this long list of calls only to open gtk20.mo (12 calls on Hardy Heron):

open("/usr/share/locale/fr_FR.UTF-8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/fr_FR.utf8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/fr_FR/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/fr.UTF-8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/fr.utf8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/fr/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/fr_FR.UTF-8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/fr_FR.utf8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/fr_FR/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/fr.UTF-8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/fr.utf8/LC_MESSAGES/gtk20.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/fr/LC_MESSAGES/gtk20.mo", O_RDONLY) = 3

There are many solutions to workaround this performance problem, either you use my little script reduce-mo-open-calls.sh which detect your current locale and create only some symlinks, or you can try someting more adventurous (I don’t know the result after a upgrade of your langpack), move /usr/share/locale-langpack/fr/LC_MESSAGES/* to /usr/share/locale/fr/LC_MESSAGES/ and create the following symlinks after deleting of /usr/share/locale-langpack/fr:

ln -s /usr/share/locale/fr /usr/share/locale-langpack/fr

then from /usr/share/locale/, /usr/share/locale-langpack and /usr/lib/locale:

ln -sfvT fr fr_FR.UTF-8
ln -sfvT fr fr_FR.utf8
ln -sfvT fr fr.UTF-8
ln -sfvT fr fr.utf8
ln -sfvT fr fr_FR

If someone knows a better solution to configure the paths order, I’m interested! I’m intend to post a bug report if I don’t see any strange systems calls in the next weeks.

This hack removes 89 open calls at the launch of gnome-terminal (the result is better with bigger applications), it’s pretty nice, isn’t it?



Localisation française de la glibc

by Stéphane

8 04 2008

Il me semble très important de disposer d’outils de développement puissants pour développer des applications traduites (p. ex. « File » devient « Fichier ») et localisées (monnaie, date, poids et mesures, nombre, etc) [1].
La traduction est appellée i18n (contraction du mot anglais internationalization) et la localisation, L10n (localization), sur cette dernière partie, il y a du nouveau dans la glibc 2.8 :

http://sourceware.org/bugzilla/show_bug.cgi?id=6040

Pour résumé, les changements sont les suivants :

  1. nouvelles abréviations des mois (vous ne devriez plus confondre juin et juillet) ;
  2. utilisation d’un point final dans les abréviations ;
  3. strftime et l’argument “%x”, affichent dorénavant 08/04/2008 et non 08.04.2008 ;
  4. printf(”%’d”, 20000) affiche « 20 000 ».

Merci à l’équipe de traduc.org pour leur aide précieuse et plus particulièrement au travail formidable de Jean-Philippe Guérard.

[1] Par contre, je préfère utiliser les versions non traduites de gcc, du bash ou de la glibc.



Vertimus 1.0

by Stéphane

22 03 2008

GNOME translation teams,
Vertimus is the perfect tool to follow each translation, translate, proofread and enhance the quality of contributions.

https://launchpad.net/vertimus

===================================
Overview of changes in Vertimus 1.0
===================================
tab: vertimus-1-0 (2008-03-21)

* XHTML and CSS pass the W3C validator
* New tool to download all PO files of a release
* Show informations about other modules with the same name in the
module page (Javascript)
* Move old files in a backup directory
* Add some maintenance scripts
* Display the authorized extensions in module page
* Add a search tool
* Reduce require_once calls
* Add informations about APC installation
* FIX Backup of files
* FIX No error message when the extension file isn’t valid
* FIX Web site title translation
* FIX call to date_default_timezone_set
* Fix #189903 - RSS feed error

New and last release wrote in PHP! Take it while it’s hot! The next version will be in Python to integrate features from Damned-lies and Transifex.



Waf build system

by Stéphane

17 03 2008

I integrate the Waf build system to libmodbus. Waf is a really nice project I hope Thomas Nagy will have time to update his documentation. The code is small and really easy to browse but the most impressive difference is the execution time:

time (./autogen.sh && ./configure && make && sudo make install)
=> 21.576s (with AM_DISABLE_STATIC)

time (./waf configure && ./waf build && sudo ./waf install)
=> 2.779s

The output is nice (compact and colorful), you can also use Waf in daemon mode (works with inotify) and each time a file of your project is modified, Waf build the new component. The size of the configure script is ~700Kb and if you choose to embed Waf it’s only 100Kb. Last but not least, your build system doesn’t check for Fortran anymore :).

Thanks to Ali Sabil for his demonstration of Waf at FOSDEM!



Performance of a PHP application with APC

by Stéphane

9 03 2008

To evaluate the performance of Vertimus with a opcode cache like APC, I used xdebug and Kcachegrind.
The results are really interesting, without APC, the index page has a total time cost of 191 032, the Zend Framework requires to use many classes, PHP is not really fast to parse and execute this code :

Vertimus without APC

and with APC, the total time cost is only 123 904:
Vertimus with APC

The CPU load is reduced by 36% but you need a bit of memory to store cache data (30 Mo by default). The results has been obtained with APC 3.0.16 and the following configuration:

;;;;;;;;;;;;;;;;;;
; APC ;
;;;;;;;;;;;;;;;;;;
extension=apc.so
apc.enabled=1
apc.shm_segments=1
apc.shm_size=30
apc.ttl=7200
apc.user_ttl=7200



Cairo-Clock is now really faster!

by Stéphane

3 03 2008

capture-cairo-clock-par-macslow.pngCairo-Clock is CPU intensive and consume around 13 % of CPU on my Core 2 Duo (on one CPU), the workload is spread on Xorg (10 %), compiz.real (2 %) and cairo-clock (1 %) with a smoothing value defined at 30 (default). The second hand drawing is disabled (default).

I wrote some patches to reduce the CPU consumption and now, Cairo-Clock doesn’t appear in ‘top’ list, woooh! The solution was simple, it useless to refresh the clock at 30 Hz if the second hand is not drawn. I also use common code to manage timeout and g_timeadd_seconds reduce the number of interrupts when many applications use this function.

I prefer bzr to git but I published my patches on github so MacSlow can take my patches here: http://github.com/stephane/cairo-clock/tree/master

PS: I couldn’t resist to translate in French (and fix Cairo-Clock for translation).