Database performance in Web applications

by Stéphane

29 10 2008

It’s more efficient to connect a Web application with an Unix Domain
Socket than TCP/IP one (reduced overhead) so I’ll explain the required
configuration with the following pairs:
1 - TurboGears/SQLAlchemy
2 - Django/PostgreSQL
3 - Django/MySQL

1 - TurboGears/SA

SQLObject is dead, isn’t it? So With SQLalchemy, the syntax is:

sqlalchemy.dburi="postgres:///dbname?user=mydbuser&password=XXXXXX" ([1])
http://docs.turbogears.org/1.0/DatabasePostgres

2 - Django/PostgreSQL

You just need to define DATABASE_ENGINE = ‘postgresql_psycopg2′ and DATABASE_NAME. Leave DATABASE_HOST setting empty to use UDS.

3 - Django/MySQL

Create a database in UTF8, either with default-character-set = utf8
under [mysqld] section in the my.cnf file or with an explicit ‘create
database bla charset=utf8;’

In settings.py:


DATABASE_HOST = '/var/run/mysqld/mysqld.sock'
DATABASE_OPTIONS = {
'read_default_file': '/etc/mysql/my.cnf',
'init_command': 'SET storage_engine=INNODB'
}

A - Note about PostgreSQL

When the user isn’t the same one who runs the process, you must edit the PostgreSQL configuration (/etc/postgresql/8.3/main/pg_hba.conf):
# "local" is for Unix domain socket connections only
local   user     database      md5
local   all      all           ident sameuser

You must create an user with a encrypted password (encrypted by default).

$ CREATEUSER username
$ psql
postgres=# ALTER USER username WITH ENCRYPTED PASSWORD 'my_password';

If you want to be sure, remove the lines with ‘host’ to deny nonlocal connections.



Publication et démos de zgeo.wfs

by ebr

27 10 2008

Makina Corpus a publié aujourd’hui zgeo.wfs, une implémentation d’un serveur WFS pour Plone 3.

Le site suivant propose des démonstrations élémentaires de zgeo.wfs.



Autocomplete Ajax search with Dojo and Zend Framework

by regilero

26 10 2008
autocomplete-ajax-search-with-dojo-and-zend-framework

search filter 2With the new Zend Framework 1.6 we’ve these nice Dojo widgets.

New things lacks documentations most of times. So if you want to build something really usefull like theses nice autocomplete search combobox this example could save you a lot of time.
We assume you have dojo already installed and activated on your views, and that acl verifications are done elsewhere, on your Controller plugins for example.
search filter

First let’s see HTML code (in your view):
<script type="text/javascript">
    dojo.require("dojo.parser");
    dojo.require("dojox.data.QueryReadStore");
    dojo.require("dijit.form.ComboBox");
    dojo.require("dijit.form.FilteringSelect");
    dojo.require("custom.FindAutoCompleteReadStore");
    dojo.require("dijit.form.Form");
    dojo.require("dijit.form.Button");
</script>
<form id="Find_Form" action="/module/foo/edit" method="get" dojoType="dijit.form.Form">
<div dojoType="custom.FindAutoCompleteReadStore" jsId="NameStore" url="/module/foo/find/format/json" requestMethod="get"></div>
<label for="id" class="optional">Recherchez un nom:</label>
<span class="formelement"><select name="id" id="FindByName" hasDownArrow="" store="NameStore" size="25" tabindex="99" autocomplete="1" dojoType="dijit.form.FilteringSelect" pageSize="10" ></select></span>
<span class="actionbuttons"><input id="Find_go" name="Find_go" value="Go:" type="submit" label="go:"dojoType="dijit.form.Button" /></span>
</form>

As we can see you’ll need an additional custom js:
custom.FindAutoCompleteReadStore
This is a really simple js to write, create your custom directory in the same level as dojo or dijit directory and create FindAutoCompleteReadStore.js like that:
dojo.provide("custom.FindAutoCompleteReadStore");
dojo.require("dojox.data.QueryReadStore");
dojo.declare("custom.FindAutoCompleteReadStore", dojox.data.QueryReadStore, {
    fetch:function (request) {
        request.serverQuery = { Find:request.query.name };
        // cal superclass fecth
        return this.inherited("fetch", arguments);
    }
});

Now you’ll need to serve the requested Ajax query (requested by the Dojo store linked with our FilteringSelect or Combobox) : /module/foo/find/format/json
This is the method ‘findAction’ in the Controller ‘foo’ on module ‘module’.
But first let’s see the preDispatch function of this controller where we handle the format/json instruction to switch in Ajax mode:

public function preDispatch()
{
    $contextSwitch = $this->_helper->getHelper('contextSwitch');
    $contextSwitch->setAutoJsonSerialization( true );
    $contextSwitch->addActionContext('find', 'json');
    $contextSwitch->initContext();
}

So now let’s write the find function:

public function findAction()
{
    // handle filtering of recieved data
    $replacer = new Zend_Filter_pregReplace('/\*/','%');
    // emulate alpha+num filter with some more characters enabled
    //**** http://www.regular-expressions.info/unicode.html ****
    // \p{N} --> numeric chars of any language
    // \s -> withespace
    //\x0027 : APOSTROPHE

    //\x002C : COMMA
    //\x0025% : % in UTF-8 and not in utf-8
    //\x002D : HYPHEN / MINUS
    //\x005F : UNDERSCORE
    //\. DOT
    $mylimit = new Zend_Filter_pregReplace('/[^\p{L}\p{N}\s\x0027\x002C\x002D\x005F\x0025%\.]/u’,”);
    $filters = array(
        ’*’     => ‘StringTrim’
        ,’Find’ => array(
            ’StripNewlines’
            ,$replacer
            ,$mylimit
            ,’StripTags’
        )
        ,’start’ => ‘Int’
        ,’count’ => ‘Int’
    );
    $validators =array();
    $input = new Zend_Filter_Input($filters, $validators, $_GET);
    $find = $input->getUnescaped(’Find’);
    if (empty($find)) $find = ‘%’;
    $start = intval($input->getUnescaped(’start’));
    if (empty($start)) $start = 0;
    $count = intval($input->getUnescaped(’count’));
    if (empty($count)) $count = 3;
    // get the model, here you should adjust with the way you work
    // then make your query with limits
    $this->_modeltable = new My_Zend_Db_Table_Foo($this->db)
    $fieldid = ‘my_id_field’;
    $fieldident = ‘my_name_field’;
    $select = $this->_modeltable->select();
    $db = $this->_modeltable->getAdapter();
    $select->where($db->quoteinto($db->quoteIdentifier($fieldident).’ LIKE ?’, $find));
    $select->limit($count, $start);
    $rows= $this->_modeltable->fetchAll($select);
    $rowsarray = $rows->ToArray();
    $finalarray=array();
    foreach ($rowsarray as $row)
    {
        $key = $row[$fieldid];
        $finalarray[$key] = $row[$fieldident];
    }
//Zend_Debug::dump($finalarray);
//die(__METHOD__);
    $this->_helper->autoCompleteDojo($finalarray);
}

And it should be sufficient, pffiuu.
But… there’s one remaining problem after that. We put the search autocomplete inside a form and we wanted the ‘go’ button to send a request to something like that:

/module/foo/edit/id/1245 OR /module/foo/edit?id=1245

But we’ll have something like:

/module/foo/edit?id=THE NAME

too bad…

To get it done I had to change one thing in Zend Framework library on the Zend/Controller/Action/Helper/AutoCompleteDojo.php Helper:

62 public function prepareAutoCompletion($data, $keepLayouts = false)
63 {
64 $items = array();
65 foreach ($data as $key => $value) {
66 $items[] = array(’label’ => $value, ‘name’ => $value, ‘key’ => $key);
67 }
68 $final = array(
69 ‘identifier’ => ‘key’,
70 ‘items’ => $items,
71 );
72 return $this->encodeJson($final, $keepLayouts);
73 }

Line 66 ‘key’ is added on the item and line 69 ‘identifier’ is set to ‘key’ and not ‘name’. ‘identifier’ is used by the Dojo Filtering Select to decide which field will be used for the form, for more info see dojo book page and search ‘abbreviation’. There’s also a bug talking about that for Zend Framework, to get other solutions or info on the way it will be fixed later look here



Pylons, xmlrpc and doctest

by kiorky

21 10 2008

I m actually developping some application around XMLRPC protocol at work.

We are using Pylons for the framework part, and i played this afternoon at setting up some testing environnement for doing doctests.

This test is a proof of concept, it 's code extracted from our internal application, it's just a starter for you. The whole is working with some tweaks.


controllers/mycontroller.py, a simple controller doing simple stuff

class MyController(XMLRPCController):                                                                                                                                      
    """controller."""                                                                                                                                                   

    def index(self):
        return '\_o<'

 

lib/base.py, Please add the XMLRPCController import

lib/base.py:from pylons.controllers import WSGIController, XMLRPCController

 

Then, we are setted up to continue with tests

First of all, the doctest boilerplate:

tests/test_doctest_files.py

import doctest                                                                   
from doctest import DocFileSuite                                                      

from myproject.tests import setUp, tearDown
flags = (doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE | doctest.REPORT_ONLY_FIRST_FAILURE)

def test_suite():
    return DocFileSuite(
        "test.txt",     
        setUp = setUp,  
        tearDown = tearDown,
        optionflags = flags 
    )

 

setUp and tearDown will have a central place as they are intialising the application.

As we can't use paste.fixture.TestApp objects with XMLRPC because it does not bind everywhere, the idea is

  • launch the server somewhere in a thread
  • use it later, as usual throught xmlrplib.
  • We will even declare it as a global to ease the doctests writings.
  • We also add a wrapper to url_for to return the host to bind to.
tests/__init__.py
                                                                                                                                                                                                                                                                            
import os                                                                                                                                                                                                                                                                        
import sys                                                                                                                                                                                                                                                                       
import re                                                                                                                                                                                                                                                                        
import threading                                                                                                                                                                                                                                                                 
from ConfigParser import ConfigParser                                                                                                                                                                                                                                            
from unittest import TestCase                                                                                                                                                                                                                                                    

import paste.fixture
import paste.script.appinstall
from paste.deploy import loadapp
from paste.httpserver import serve
from routes.util import url_for

here_dir = os.path.dirname(os.path.abspath(__file__))
conf_dir = os.path.dirname(os.path.dirname(here_dir))
test_file = os.path.join(here_dir, 'test.ini')

cmd = paste.script.appinstall.SetupCommand('setup-app')

cmd.run([test_file])

def setUp(test, *args, **kwargs):
    print "\t-----------------------------------------------------------------"
    print "\t---    Setting up database test environment, please stand by. ---"
    print "\t-----------------------------------------------------------------"
    config = ConfigParser()
    config.read(
        os.path.join(os.path.dirname(sys.argv[0]), '..', 'etc', 'config.ini')
    )

    infos = ConfigParser()
    infos.read(test_file)
    sinfos = infos._sections['server:main']
    wsgiapp = loadapp('config:test.ini', relative_to = here_dir)
    server = test.globs['server'] = serve(wsgiapp,
                                 sinfos['host'],
                                 sinfos['port'],
                                 socket_timeout=1,
                                 start_loop=False,
                                )
    t = threading.Thread(target=server.serve_forever)
    t.setDaemon(True)
    t.start()
    test.globs['app'] = paste.fixture.TestApp(wsgiapp)
    def url_for_wrapper(*args, **kwargs):
        lkwargs = {'protocol': 'http' ,'host':  "%s:%s" % (server.server_name, server.server_port)}
        lkwargs.update(kwargs)
        return url_for(*args, **lkwargs)
    test.globs['url_for'] = url_for_wrapper
    test.globs['url_for_orig'] = url_for

def tearDown(test):
    test.globs['server'].server_close()

class TestController(TestCase):
    def __init__(self, *args, **kwargs):
        wsgiapp = loadapp('config:test.ini', relative_to = here_dir)
        self.app = paste.fixture.TestApp(wsgiapp)
        TestCase.__init__(self, *args, **kwargs)


 

And finally, letz play with our doctest

tests/text.txt

>>> create_url = url_for(controller='mycontroller')

>>> import xmlrpclib
>>> s = xmlrpclib.Server(create_url)
>>> s.index()
'\\_o<'




[OFF] Et si j’étais un prof de philo ?

by pounard

19 10 2008

.. je pense que je donnerais comme sujet de dissertation des citations chinoises !

Et oui, en balladant sur le net, j'en ai repéré une ou deux que je trouve tout à fait juste, mais libre à chacun d'avoir son opinion sur le sujet.

Petite note, elles sont toutes de Cofucius (étonnant tiens!).

en lire plus


JMeter, improving performance of a Plone web site

by toutpt

13 10 2008

Last week i have made a rush to improve performance of a Plone based web site. For performance testing i have used JMeter, because i have seen Using open source tools for performance testing

JMeter is really nice to use. Just launch it’s proxy, plug your browser on it, and do your test. Next you save it as xml and you can edit the test. So you can login (it support cookies) you can create content (with an once logic controller) consult content, and stress your server.

What i have learn from this about Plone is:

  • Do not use brains or any object in templates, or you will not beeing able to cache your logic code in ramcache. Use dict that contains every strings ready to be displayed in the templates.
  • How to use the ram cache
  • i can store acl_users in ramcache, and i have been surprised to see the difference. On 5 tabs hitted, i have hit the cache 278 times …
  • Archetypes is damly slow (about one second to set some attributes of an object in a btree and reindexIt)
  • CMFPlone.utils.createObjectByType do a reindexObject
  • Do not add any index to the portal_catalog, use the buinding done by archetype_tool to be able to use other index. I m adding about one catalog tool per custom content type.
  • A query on the portal_catalog can take one second if you have for example a list of 100 paths (query['path'] = ['/first/path', '/second/path'] and more than 100 000 entries.

I have learn many other things during the last week, but now i m using stress tests during the dev

      


Drupal 5 modules release !

by pounard

6 10 2008

Working on a wide drupal site project, I had to make some custom modules, feeting to my needs. After some monthes, I finaly have stable releases for most of them, so I intend to release them to the community.

en lire plus



Minitage, projects and profils

by kiorky

26 09 2008

A little but wonderfull tool in the python world is paster.

It permits you to generate with some code logics a full set of files.

You can for example create a buildout, apache configurations files and etc.

The documentation for profils are in the official minitage documentation : http://www.minitage.org/doc/rst/

 

What is interesting there is that i use it into minitage to generate stuff inside.

There are 2 main sort of templates:

  • Profils: Something relative to a project, a database, a ldap instance (not finnished atm.), a varnish instance... Profils are applied onto an existing project !
  • Projects: A project and a minibuild referencing to it, a zope project, a turbogears project ...

 

To get those templates in a minitage instance:

source minitage/bin/activate
easy_install -U minitage.paste
paster create --list-templates

 After you can for example create a plone31 projet:

paster create -t minitage.plone31zeo myproject

 And after that, because plone is slow, put inside a varnish instance

paster create -t minitage.profils.varnish myproject

 Because you need svn 1.5 but it is not packaged on your debian potatoe, just reference it in the dependencies part of your project minibuild:

dependencies=openssl-0.9 libxml2-2.6 libxslt-1.1 zlib-1.2  py-libxml2-2.6 py-libxslt-1.1 python-2.4 pilwotk-1.1.6.4 libiconv-1.12 subversion-1.5
install_method=buildout
src_uri=http://hg.foo.net/myproject
src_type=hg
category=zope
homepage=http://foo.net
description= a plone 3.1 buildout for myproject

And install the minitage.env template:

paster create -t minitage.profils.env myproject

 This profil will create for you a small bash script that will change your environnemnent according to 'myproject' needs. PATH, LD_LIBRARY_PATH and so will be feeded with all minitage dependencies. That will enable you to use svn-1.5 when you source it (like source bin/activate)

source zope/myproject/sys/share/minitage/minitage.env
which svn
/minitage/dependencies/subversion-1.5/parts/part/bin/svn

Another interresting integrations are the zope3 and tubogears projects.

  • Zope3 template realisation allowed me to make some fixes in the minitage.recipe:scripts recipe. Now, this one generates for you great launchers scripts which include ALL their respective dependencies.
  • Something particular with tubogears is that i must have done some overhead over buildout.

You can see the full documentation there : http://www.minitage.org/doc/rst/tg_project.html

 



Nouvelle version de Plomino (1.2 RC1)

by ebr

8 09 2008

L’apport majeur de la version 1.2 est la mise en adéquation de la sécurité Plomino avec la sécurité Plone.

Jusqu’ici seules les permissions Plone posées sur la base Plomino elle-même étaient prises en compte. Dorénavant, on peut également poser des permissions Plone sur les objets de la base (documents, vues, masques).

L’intérêt principal est qu’on peut donc utiliser les workflows Plone standards sur les documents Plomino.

Ce release comporte également d’autres améliorations, plus d’information ici.



Forcing HTTP/1.0 Apache response when PHP is there…

by regilero

29 08 2008
forcing-http10-apache-response-when-php-is-there

Recently I had to force an HTTP/1.0 response with Apache because of a bad Java Parser.

This parser/browser was asking for HTTP/1.1 responses but did’nt understood chunked content encoding. And so giving me a nice Sax exception “content not allowed in prolog”. So, well, I won’t fix this #$*%! code. Better trying to talk to this special User Agent in HTTP/1.0, he might handle it in a better way. Here’s what a chunked content looks like. See the 306c hexa lenght code before the body of the response?

HTTP/1.1 207 Multi-Status
(... lot of headers, but no lenght one ...)
Content-Type: text/xml; charset="utf-8"

306c
<?xml version="1.0" encoding="utf-8"?>
(... here the content ...)

So I have the user agent of this Java HTTP Client, let’s call it ‘NoobieJavaParser’.

I simply wrote in my apache virtualhost config file:

BrowserMatch "^NoobieJavaParser" nokeepalive force-response-1.0 downgrade-1.0

And it should be sufficient. In fact it’s not because of a very old PHP bug (saw first bug report in 2004).

PHP is building is $_SERVER variable by reading Apache env, and PHP doesn’t want any dot in this parsed content. The ‘downgrade-1.0‘ env name seem malicious for PHP. So the env setting looks like that in PHP:

echo $_SERVER['downgrade-1_0'];
-> 1

See the dots is now a ‘_’. It should not hurt anyone, except PHP changed this env name in Apache as well. So when apache is sending the response, he does not care anymore about this downgrade-1_0 settings.

If you want Apache to have the real behaviour, i.e: sending HTTP/1.0 responses for this ‘NoobieJavaParser’ User Agent you must re-set the env of Apache in PHP, with something like that:

if ($_SERVER['downgrade-1_0']){
        apache_setenv(’downgrade-1.0′,’true’);
}
if ($_SERVER['force-response-1_0']){
        apache_setenv(’force-response-1.0′,’true’);
}

Ugly, but it’s PHP’s fault. and no more chunked content after that. And the nice thing is that apache_setenv is not changing $_SERVER, so PHP still does not have this malicious dot.