morespeed


  • Since I am facing performance problem due to the load of visitors and a badly configured server, I decide today to document my findings in this new series of articles.

    As soon as PHP run as FastCGI and no more inside an Apache module (mod_php4.so is not multi threaded), You should be able to switch the default MPM setting of Apache from MPM prefork to MPM worker.

    So, what's the difference between prefork and worker?

    Quoting from the Apache MPM Prefork page: http://httpd.apache.org/docs/2.0/mod/prefork.html
    MPM Prefork implements a non-threaded, pre-forking web server that handles requests in a manner similar to Apache 1.3.

    And for the Apache MPM Worker says: http://httpd.apache.org/docs/2.0/mod/worker.html
    MPM Worker implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with less system resources than a process-based server.


    Server: Strato (www.strato.de)
    Operating system: SuSE / openSuSE
    Requirements: root access and basic Unix knowledge

    1 Edit the file  
    # vi /etc/sysconfig/apache2

    and change the key:

    APACHE_MPM=" "


    to

    APACHE_MPM="worker"


    2 You can now tune also  default parameters in file /etc/apache2/server-tuning.conf
    Here are my settings, these are still the default setting of Apache

    vi/etc/apache2/server-tuning.conf

    # worker MPM

    <IfModule worker.c>
        ServerLimit 16
        # initial number of server processes to start
        StartServers         2
        # minimum number of worker threads which are kept spare
        MinSpareThreads     25
        # maximum number of worker threads which are kept spare
        MaxSpareThreads     75
        # maximum number of simultaneous client connections
        MaxClients       150
        # constant number of worker threads in each server process
        ThreadsPerChild     25
        # maximum number of requests a server process serves
        MaxRequestsPerChild  6000
    </IfModule>




    3 start
    # apache2-reconfigure-mpm
    this recreate the file  /etc/apache2/sysconfig.d/loadmodule.conf
    and restart apache automatically. Test Your site to ensure everything still work as expected.


  • This module controls the setting of the Expires HTTP header and the max-age directive of the Cache-Control HTTP header in server responses. The expiration date can set to be relative to either the time the source file was last modified, or to the time of the client access.

    These HTTP headers are an instruction to the client about the document's validity and persistence. If cached, the document may be fetched from the cache rather than from the source until this time has passed. After that, the cache copy is considered "expired" and invalid, and a new copy must be obtained from the source.

    How to activate mod_expires.so
    # vi /etc/apache2/conf.d/mod_expires.conf

    config is rough right now, but it is better than nothing,put the following in the file

    LoadModule evasive20_module     /usr/lib/apache2/mod_expires.so
    <IfModule mod_expires.c>
     ExpiresActive On

     ExpiresDefault "access plus 1 month"
     ExpiresByType text/html "access plus 1 month 15 days 2 hours"
     ExpiresByType image/gif "modification plus 1 month"
     ExpiresByType image/png "modification plus 1 month"
     ExpiresByType image/jpg "modification plus 1 month"
     ExpiresByType text/css "access plus 1 month 15 days 2 hours"
     ExpiresByType text/javascript "access plus 1 month 15 days 2 hours"
    </IfModule>

    You can also set the expire header by type, but it wa snot working in my case, that is why I use the ExpiresDefault directive

  • joomla_cms

    Since my server is still suffering, Ive decide today to take some actions.

    user: changes can be done on shared hosting with limited user rights.
    root. changes require a full access to the server (root access with secure shell ssh)

    I have currently per months, 160.000 visitors and 2 Millions Hits.or per day 8000 visitors and 24000 pages view.
    Server has only 1GB RAM.

    UPDATE: I found one/THE reason why my host is slowing down...SPAMMERS!

    click read more

    user: Joomla! settings
    • I switch enable gzip compression OFF. Since this is meant to reduce the usage of bandwidth, not the load on my  server. The server has to encode all the files in order to send them, which only puts additional load on your page.
    • I switch Joomla!! statistics off, as AWSTATS is providing a much better job.
    user: MySQL maintenance

    I Optimize (repair, refresh statistics) MySQL tables through MySQL admin, but It can be done through PLESK
    user: Tune Joomla!! cache

    I Increase Joomla! cache lifetime from 900seconds to 24 hours as it better reflect the way I update my site (daily)
    user: keep pages small

    • Reduce size of banner, using GIMP so they are using a web palette, most of them were shrink from 40kb to 7kb
    • I remove all un-needed space from the main templates file (index.php), an action which will help first users visitors only.
    • Attention, it is a never ending task...
    user: Hunting software bugs
    By switching the site to debug mode, I notice some nasty queries (select count(*) from), all created by my statistics module ( Content Statistics on right side), In fact Joomla!! module do not inherit from Joomla!! cache automatically. I fix value in this module since I do not want to program cache support in it right now.
    root One morecache

    I decide to install a PHP accelerator: PHPA from  http://www.php-accelerator.co.uk/
    " The ionCube PHP Accelerator is an easily installed PHP Zend engine extension that provides a PHP cache, and is capable of delivering a substantial acceleration of PHP scripts without requiring any script changes, loss of dynamic content, or other application compromises."

    Install is straightforward: just copy library into /usr/local/lib/php_accelerator_1.3.3r2.so
    and add these lines to /etc/php.ini

    ; PHP Accelerator extension
    zend_extension="/usr/local/lib/php_accelerator_1.3.3r2.so"
    phpa = on
    phpa.c0_size = 64
    phpa.cache_dir = /tmp
    phpa.c0_logging = on


    ;The shm_stats_check_period is the minimum interval between checks of the
    ;cache for expired scripts. The first server request after the interval has
    ;elapsed will trigger a scan of the cache for expired scripts, and remove
    ;any entries that it finds.
    phpa.c0_stats_check_period = 5m

    ;The shm_ttl value is the value used to set the
    ;time-to-expiry value when a script is accessed. Put another way, the shm_ttl
    ;value is the period after which an unaccessed script expires.
    phpa.c0_ttl = 12h

    ;phpa.ignore_files = ""
    ;phpa.ignore_dirs = ""

    I use that tool: HTTP Viewer to check if my page now contains a headerX-Accelerated-By:·PHPA/1.3.3r2

    Reduce surface of attack: I found my components that were not used by Joomla! (very old code and uused components . So go through all directories with FTp/SCP and remove any un-needed code....


    Review table data directly in the database...
    This is how I found 27 000 spams in my gallery (Zoom gallery)
    solution:
    • I remove all entries
    • Disallow comments operations (in Zoom gallery admin panel),
    but spammers were still able to insert comments, so I edit the file components/com_zoom/lib/image.class.php
    //add because of spammers
    header("HTTP/1.0 403 Forbidden");
    //$database->setQuery("INSERT INTO __zoom_comments (imgid,cmtname,cmtcontent,cmtdate) VALUES ('".mysql_escape_str

    Note: I  recommend You to use also mod_evasive and mod_security  (root access needed), see aprevious article on my site



    Some links, where I borrow some ideas:


    http://www.primakoala.com/tutorials/guides/speeding_up_joomla.html
    http://forum.joomla.org/index.php/topic,50278.0.html
    http://forum.joomla.org/index.php/topic,54175.0.html
  • apache-poi-logo The Apache POI Project's mission is to create and maintain Java APIs for manipulating various file formats based upon the Office Open XML standards (OOXML) and Microsoft's OLE 2 Compound Document format (OLE2). In short, you can read and write MS Excel files using Java. In addition, you can read and write MS Word and MS PowerPoint files using Java. Apache POI is your Java Excel solution (for Excel 97-2008). We have a complete API for porting other OOXML and OLE2 formats and welcome others to participate.

    Switch Off logging

    From the documentation at http://poi.apache.org/utils/logging.html

    Logging in POI is used only as a debugging mechanism, not a normal runtime logging system. Logging is ONLY for autopsie type debugging, and should NEVER be enabled on a production system. Enabling logging will reduce performance by at least a factor of 100. If you are not developing POI or trying to debug why POI isn't reading a file correctly, then DO NOT enable logging. You've been warned.

    In order to effectively disable the logging functionality in Apache POI you must use an alternative logger. This is accomplished by providing a property to the POILogFactory to override the default logger. You can add one of these –D to your JVM settings

    -Dorg.apache.poi.util.POILogger=org.apache.poi.util.NullLogger
    -Dorg.apache.poi.util.POILogger=org.apache.commons.logging.impl.NoOpLog

    I found Apache POI to slightly better perform with the NoOpLog of apache common!

    Recompile poi with more adapted settings

    You can create a custom build of Apache POI 3.8 and alter the following properties to better match the size of the excel files you are generating or reading:

  • org.apache.poi.hssf.usermodel.HSSFRowINITIAL_CAPACITY=5;
  • org.apache.poi.hssf.usermodel.HSSFSheetINITIAL_CAPACITY= 20;&160;&160;&160; // used for compile-time optimization.&160; This is the initial size for the collection of rows.&160; It is currently set to 20.&160; If you generate larger sheets you may benefit by setting this to a higher number and recompiling a custom edition of HSSFSheet.
  • org.apache.poi.hssf.usermodel.HSSFWorkbookINITIAL_CAPACITY=3;&160; // used for compile-time performance/memory optimization.&160; This determines the&160; initial capacity for the sheet collection.&160; Its currently set to 3.Changing it in this release will decrease performance since you're never allowed to have more or less than three sheets!&160;&160;&160;&160;&160;
  • INITIAL_CAPACITY">http://poi.apache.org/apidocs/org/apache/poi/hssf/usermodel/HSSFWorkbook.htmlINITIAL_CAPACITY

    Don’t use xlsx, prefer xls!

    This will only work if you do not reach xls limitations which may avoid you to go to that extreme solution. XLS is not compressed (XLSX is xml based and compressed) and your workbook may double size in memory as a result!

    For example, data beyond 256 (IV) columns by 65,536 rows will not be saved in xls! In Excel 2010 and Excel 2007, the worksheet size is 16,384 columns by 1,048,576 rows, but the worksheet size of Excel 97-2003 is only 256 columns by 65,536 rows. Data in cells outside of this column and row limit is lost in Excel 97-2003. But there is a lot more limitations listed at office.com

    The biggest side effect was that my excel file went from 354kb to 967kb, but the speed increase was quite interesting: more than 44% less evaluation time.

    Small localized optimization

    I don’t think these bring a lot of speed, JIT should optimize this bad piece of code for us but it is always worth trying Speeding up org.apache.poi.hssf.usermodel.HSSFRow.compareTo() and http://affy.blogspot.ch/2004/04/poi-optimization-speeding-_108265938673224937.html

  • apache

    The acronym LAMP refers to a solution stack of software, usually free and open source software, used to run dynamic Web sites or servers. It stand for:

    • Linux, for the operating system;
    • Apache, the Web server;
    • MySQL, the database management system (or database server);
    • Perl, Python, and PHP, the programming languages.

    &160;ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving.
    Apache-utils package contains utility programs for webservers and some add-on programs useful for any webserver. These include:

    • ab (Apache benchmark tool)
    • Logresolve (Resolve IP addresses to hostname in logfiles)
    • htpasswd (Manipulate basic authentication files)
    • htdigest (Manipulate digest authentication files)
    • dbmmanage (Manipulate basic authentication files in DBM format, using perl)
    • htdbm (Manipulate basic authentication files in DBM format, using APR)
    • rotatelogs (Periodically stop writing to a logfile and open a new one)
    • split-logfile (Split a single log including multiple vhosts)
    • checkgid (Checks whether the caller can setgid to the specified group)
    • check_forensic (Extract mod_log_forensic output from apache log files)

    This package Apache-Utils can be install through apt or YaST depending if you are using a Debian base distro or OpenSuse

    Prerequistes

    • Define realistic objectives, do not create too much virtual clients if you do not have usually that kind of user traffic..
    • For example an objective could be: number of users served, or percentage of the requests served within a certain time
    • This tool ab do not simulate realistic user behavior, it just hit a page without being able to simulate a complex workflow (like login, navigate and do things users usually do)
    • Try to monitor at the same time the CPU/Memory consumed in order not to make false assumption on apache settings (use top d 1)

    Attention

    It is an iterative process!

    1. Benchmark,
    2. Change settings and
    3. Restart benchmark.

    It is very important to only change a setting a time in order to better identify what is really bringing something! By changing only one settings at a time, you can:

    • Better see the influence on CPU, memory (you must look also at resources, a server swapping to disk is never good)
    • There is not so much universal settings bringing a speed kick (except DNSlookup off, keep alive small), some settings are depending on your Linux kernel version, CPU class, disk speed, network latency

    Other components

    mysql While tuning apache, you will see that most of the time is used in PHP/MySQL, for MySQL recommend to run at the same time tuning-primer.sh, read more here


    Usage

    ab [ -A auth-username:password ] [ -c concurrency ] [ -C cookie-name=value ] [ -d ] [ -e csv-file ] [ -g gnuplot-file ] [ -h ] [ -H custom-header ] [ -i ] [ -k ] [ -n requests ] [ -p POST-file ] [ -P proxy-auth-username:password ] [ -q ] [ -s ] [ -S ] [ -t timelimit ] [ -T content-type ] [ -v verbosity] [ -V ] [ -w ] [ -x <table>-attributes ] [ -X proxy[:port] ] [ -y <tr>-attributes ] [ -z <td>-attributes ] [http://]hostname[:port]/path

    Options

    -A auth-username:password
    Supply BASIC Authentication credentials to the server. The username and password are separated by a single : and sent on the wire base64 encoded. The string is sent regardless of whether the server needs it (i.e., has sent an 401 authentication needed).
    -c concurrency
    Number of multiple requests to perform at a time. Default is one request at a time.
    -C cookie-name=value
    Add a Cookie: line to the request. The argument is typically in the form of a name=value pair. This field is repeatable.
    -d
    Do not display the "percentage served within XX [ms] table". (legacy support).
    -e csv-file
    Write a Comma separated value (CSV) file which contains for each percentage (from 1% to 100%) the time (in milliseconds) it took to serve that percentage of the requests. This is usually more useful than the 'gnuplot' file; as the results are already 'binned'.
    -g gnuplot-file
    Write all measured values out as a 'gnuplot' or TSV (Tab separate values) file. This file can easily be imported into packages like Gnuplot, IDL, Mathematica, Igor or even Excel. The labels are on the first line of the file.
    -h
    Display usage information.
    -H custom-header
    Append extra headers to the request. The argument is typically in the form of a valid header line, containing a colon-separated field-value pair (i.e., "Accept-Encoding: zip/zop;8bit").
    -i
    Do HEAD requests instead of GET.
    -k
    Enable the HTTP KeepAlive feature, i.e., perform multiple requests within one HTTP session. Default is no KeepAlive.
    -n requests
    Number of requests to perform for the benchmarking session. The default is to just perform a single request which usually leads to non-representative benchmarking results.
    -p POST-file
    File containing data to POST.
    -P proxy-auth-username:password
    Supply BASIC Authentication credentials to a proxy en-route. The username and password are separated by a single : and sent on the wire base64 encoded. The string is sent regardless of whether the proxy needs it (i.e., has sent an 407 proxy authentication needed).
    -q
    When processing more than 150 requests, ab outputs a progress count on stderr every 10% or 100 requests or so. The -q flag will suppress these messages.
    -s
    When compiled in (ab -h will show you) use the SSL protected https rather than the http protocol. This feature is experimental and very rudimentary. You probably do not want to use it.
    -S
    Do not display the median and standard deviation values, nor display the warning/error messages when the average and median are more than one or two times the standard deviation apart. And default to the min/avg/max values. (legacy support).
    -t timelimit
    Maximum number of seconds to spend for benchmarking. This implies a -n 50000 internally. Use this to benchmark the server within a fixed total amount of time. Per default there is no timelimit.
    -T content-type
    Content-type header to use for POST data.
    -v verbosity
    Set verbosity level - 4 and above prints information on headers, 3 and above prints response codes (404, 200, etc.), 2 and above prints warnings and info.
    -V
    Display version number and exit.
    -w
    Print out results in HTML tables. Default table is two columns wide, with a white background.
    -x <table>-attributes
    String to use as attributes for <table>. Attributes are inserted <table here >.
    -X proxy[:port]
    Use a proxy server for the requests.
    -y <tr>-attributes
    String to use as attributes for <tr>.
    -z <td>-attributes
    String to use as attributes for <td>.


    Some real examples

    time /usr/sbin/ab2 -n 500 -c 30 http://www.waltercedric.com
    This will make 500 requests on them and hammering localhost for 30 seconds

    After tuning Before tuning
    Benchmarking www.waltercedric.comCompleted 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Finished 500 requests
    Server Software:&160;&160;&160;&160;&160;&160;&160; NOYB
    Server Hostname:&160;&160;&160;&160;&160;&160;&160; www.waltercedric.com
    Server Port:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 80
    Document Path:&160;&160;&160;&160;&160;&160;&160;&160;&160; /index.php
    Document Length:&160;&160;&160;&160;&160;&160;&160; 45532 bytes
    Concurrency Level:&160;&160;&160;&160;&160; 30
    Time taken for tests:&160;&160; 38.576375 seconds
    Complete requests:&160;&160;&160;&160;&160; 500
    Failed requests:&160;&160;&160;&160;&160;&160;&160; 19&160;
    &160;&160; (Connect: 0, Length: 19, Exceptions: 0)
    Write errors:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    Total transferred:&160;&160;&160;&160;&160; 23000106 bytes
    HTML transferred:&160;&160;&160;&160;&160;&160; 22762106 bytes
    Requests per second:&160;&160;&160; 12.96 [#/sec] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 2314.582 [ms] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 77.153 [ms] (mean, across all concurrent requests)
    Transfer rate:&160;&160;&160;&160;&160;&160;&160;&160;&160; 582.25 [Kbytes/sec] received
    Connection Times (ms)
    &160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; min&160; mean[+/-sd] median&160;&160; max
    Connect:&160;&160;&160;&160;&160;&160;&160; 0&160;&160;&160; 8&160; 36.9&160;&160;&160;&160;&160; 0&160;&160;&160;&160; 207
    Processing:&160;&160; 394 2239 345.3&160;&160; 2237&160;&160;&160; 6223
    Waiting:&160;&160;&160;&160;&160; 379 2197 340.9&160;&160; 2190&160;&160;&160; 6173
    Total:&160;&160;&160;&160;&160;&160;&160; 397 2247 344.2&160;&160; 2239&160;&160;&160; 6223
    Percentage of the requests served within a certain time (ms)
    &160; 50%&160;&160; 2239
    &160; 66%&160;&160; 2294
    &160; 75%&160;&160; 2327
    &160; 80%&160;&160; 2357
    &160; 90%&160;&160; 2457
    &160; 95%&160;&160; 2560
    &160; 98%&160;&160; 2973
    &160; 99%&160;&160; 3341
    100%&160;&160; 6223 (longest request)
    real&160;&160;&160; 0m38.617s
    user&160;&160;&160; 0m0.024s
    sys&160;&160;&160;&160; 0m0.240s

    Benchmarking www.waltercedric.com
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Finished 500 requests

    Server Software:&160;&160;&160;&160;&160;&160;&160; NOYB
    Server Hostname:&160;&160;&160;&160;&160;&160;&160; www.waltercedric.com
    Server Port:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 80

    Document Path:&160;&160;&160;&160;&160;&160;&160;&160;&160; /index.php
    Document Length:&160;&160;&160;&160;&160;&160;&160; 45532bytes

    Concurrency Level:&160;&160;&160;&160;&160; 30
    Time taken for tests:&160;&160; 108.897481 seconds
    Complete requests:&160;&160;&160;&160;&160; 500
    Failed requests:&160;&160;&160;&160;&160;&160;&160; 19
    &160;&160; (Connect: 0, Length: 19, Exceptions: 0)
    Write errors:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    Total transferred:&160;&160;&160;&160;&160; 23000106bytes
    HTML transferred:&160;&160;&160;&160; 23000106bytes
    Requests per second:&160;&160;&160; 4.59 [#/sec] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 6533.849 [ms] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 217.795 [ms] (mean, across all concurrent requests)
    Transfer rate:&160;&160;&160;&160;&160;&160;&160;&160;&160; 178.41 [Kbytes/sec] received

    Connection Times (ms)
    &160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; min&160; mean[+/-sd] median&160;&160; max
    Connect:&160;&160;&160;&160;&160;&160;&160; 0&160; 114 478.9&160;&160;&160;&160;&160; 0&160;&160;&160; 2276
    Processing:&160;&160; 336 6186 1665.2&160;&160; 6108&160;&160; 16189
    Waiting:&160;&160;&160; -5148 5982 1982.8&160;&160; 6066&160;&160; 16009
    Total:&160;&160;&160;&160;&160;&160;&160; 391 6301 1580.2&160;&160; 6120&160;&160; 17093

    Percentage of the requests served within a certain time (ms)
    &160; 50%&160;&160; 6120
    &160; 66%&160;&160; 6453
    &160; 75%&160;&160; 6778
    &160; 80%&160;&160; 7046
    &160; 90%&160;&160; 7861
    &160; 95%&160;&160; 8516
    &160; 98%&160; 10110
    &160; 99%&160; 12418
    100%&160; 17093 (longest request)

    real&160;&160;&160; 1m48.905s
    user&160;&160;&160; 0m0.024s
    sys&160;&160;&160;&160; 0m0.152s

    &160;

    time /usr/sbin/ab2 -kc 10 -t 30 http://www.waltercedric.com
    This will open 10 connections, using Keep-Alive on them and hammering localhost for 30 seconds


    Same tests but without mod_security

    • Mod_security is a module for Apache which act like a software firewall
    • Depending on the number of rules, can greatly affect through output speed


    time /usr/sbin/ab2 -kc 10 -t 30 http://www.waltercedric.com
    This will open 10 connections, using Keep-Alive on them and hammering localhost for 30 seconds

    real&160;&160;&160; 0m39.040s
    user&160;&160;&160; 0m0.020s
    sys&160;&160;&160;&160; 0m0.208s

    Nearly one second more with mod_security gotroot rules, worth the added security!

    If you want to know more options and how to use apache ab check the apache ab/ab2 man page click here for this man page

    How to optimize Apache/Joomla/PHP

    I forward You to some of my previous articles:

    And more ideas here Secure, Safe, Fast Linux Hosting

  • Nginx-logo

    nginx (pronounced “engine-x”) is an open source Web server and a reverse proxy server for HTTP, SMTP, POP3 and IMAP protocols, with a strong focus on high concurrency, performance and low memory usage. It is licensed under a BSD-like license and it runs on Unix, Linux, BSD variants, Mac OS X, Solaris, AIX and Microsoft Windows [WikiPedia]

    These are my reusable settings for any Joomla hosting, these are the most secure, and fastest settings to the best of my knowledge.

    Configuration files are provided using Gist&160; and are CONSTANTLY updated for added security and speed. Gist is a simple way to share snippets and pastes with others. All gists are git repositories, so they are automatically versioned, forkable and usable as a git repository. I recommend you to starred them to stay up to date.

    Joomla.conf for nginx

    Create a new directory nginx/conf to be able to place reusable nginx settings:

    mkdir -p /etc/nginx/conf

    vi /etc/nginx/conf/joomla.conf

    Edit or create joomla.conf, you can find the latest joomla.conf documented version in one of my Gist at https://gist.github.com/1620307

    Adding a new Joomla Site to nginx

    Create required directory anywhere on your disk, here is an example with a domain www.example.com

    mkdir -p /var/www/vhosts/example.com/httpdocs
    mkdir -p /var/www/vhosts/example.com/logs

    Set the right permission to the user and group you have defined in nginx.conf

    chown -fR www-data:www-data /var/www/vhosts/example.com/httpdocs

    Copy the nginx template and adapt to your liking

    cp /etc/nginx/sites-available/default /etc/nginx/sites-available/example
    vi /etc/nginx/sites-available/example

    Edit or create example, you can find the latest file example documented version in one of my Gist at https://gist.github.com/1620307

    this file include Joomla.conf to avoid duplicating nginx settings

    Activate the new domain

    ln -s /etc/nginx/sites-available/example /etc/nginx/sites-enabled/example
    service nginx restart
  • css.sprite.generator

    Here is an useful tool for improving performance of your website: The CSS sprite generator takes a number of source images and combines them into a single "sprite" image and supplies CSS to map to each image within the sprite. The technique this tool employs is often used to improve the performance of a web site by reducing the number of HTTP requests that the client's browser needs to make.

    CSS sprites are a way to reduce the number of HTTP requests made for image resources referenced by your site. Images are combined into one larger image at defined X and Y coordinates. Having assigned this generated image to relevant page elements the background-position CSS property can then be used to shift the visible area to the required component image.

    This technique can be very effective for improving site performance, particularly in situations where many small images, such as menu icons, are used. The Yahoo! home page, for example, employs the technique for exactly this.

    I’ve installed a copy of this tool at:

    http://css-sprite-generator.waltercedric.com

    And took time to:

    1. Improve the overall internal security of the tools, by using some technique of Joomla! framework, so most type of attacks (XSS, parameter tampering, path browsing, avoid direct execution of files) should be eliminated,
    2. The tool now extract the ZIP file without being forced to use the dangerous shell_exec php command (I switch it off on my host anyway), the tool support&160; if they are available in that order: ZipArchive(), zip_open() and shell_exec()
    3. Make the code a bit more portable (avoiding .. inclusion for files, use DIRECTORY_SEPARATOR)

    And I did audit the code using Acunetix Web security Scanner, the result can be seen below:

    css.sprite.generator.acunetix&160;

    The source code for CSS Sprite Generator is available for download and is covered by a BSD license. Want to run a local copy, get the source code at https://launchpad.net/css-sprite-generator

    My version will be available soon, link to appear later this week.

    Note about PHP and Zip support

    To install zip PECL package run the following as root:

    # pear installhttp://pecl.php.net/get/zip

    and create a new file at

    # vi /etc/php5/conf.d/zip.ini

    with following content

    extension=zip.so

    Restart apache

  • apache_maven

    We had serious performance problems with MAVEN in our environment. It seems to be a recurrent problem
    for MAVEN... anyway I did came through the following changes...the 2.0.9.db1 Maven2 patch make really
    Maven fly!

    General settings to speed up Maven:

    • More memory for Maven process, change the launcher of eclipse to set MAVEN_OPTS like this:
      -DMAVEN_OPTS="-Xms64m –Xmx128m"
    • Use the latest version of Maven, but be careful of regressions! the latest as for today is 2.0.9
    • There is a patch available for Maven 2.0.9, which speed up build by 40%. It is just simply day and
      night! try it, you'll love it! Basically Don Brown alter MAVEN2 2.0.9 to

    General settings to speed up Eclipse:

    1. Use javaw.exe to start eclipse and not java.exe (more for console base program with a lot of feedback),
      while javaw.exe is more for graphical environment.
    2. Aggressive JIT and double core processors should use:  
       -XX:-UseParallelGC -XX:+AggressiveOpts -XX:-UseConcMarkSweepGC -XX:+UseFastAccessorMethods
    3. Give more memory, MORE MEMORY for eclipse, on a 4GB machine, these are my settings: 
      -Xms768m -Xmx1024m -XX:MaxPermSize=256m
    4. Reduce the number of warning reported by eclipse per compilation unit (class), default is 100, reduce it to 10.
      It help nobody to see a workspace slowing down because of too many warning logging.
      Remove the warnings instead ;-)
    5. SVN console with subversive is too verbose as default, go to eclipse preferences - Team – SVN - Console.
      Logging SVN errors should be enough.
    6. Use a Defragmenter! NTFS fragment fast with so many small files in workspace, every 2 week is a good practice.
    7. I am using Java 1.6u10 (BETA!) and have experience no crash till now,
      being on the edge can be costly in time through. Maven forking should benefit from the reduce java kernel
      size and bootstrap time
  • I will try to keep an history of all my previous machine on this page...yes it is just a nerd/geek behavior...

    geek-inside

    In 15 years...

    • Cost in euro are in green, are similar in amount, you just get more size, disk, speed for your buck
    • 1208 times more IPS and my PC is still sometimes hanging (In order Vista, XP, Linux I look at you) :-(
    • 81 times more disk space!
    • Power consumption 360 watts today :-(
    • PC are still slower than what I expect...

     

    Instructions per second (IPS) is a measure of a computer's processor speed. Many reported IPS
    values have represented "peak" execution rates on artificial instruction sequences with few branches,
    whereas realistic workloads consist of a mix of instructions and even applications, some of which take
    longer to execute than others. The performance of the memory hierarchy also greatly affects processor
    performance, an issue barely considered in MIPS calculations. Because of these problems, researchers
    created standardized tests such as SPECint to (maybe) measure the real effective performance in
    commonly used applications, and raw IPS has fallen into disuse. [WikiPedia]

  • Impact of Zend Optimizer on PHP Performance

    The Zend Optimizer FAQ answers the question "Why use the Zend Optimizer?" with this statement: "The standard Zend run-time compiler used by PHP is indeed extremely fast, generating code that is usually 2 to 10 times faster. But an application that uses the Zend Optimizer typically executes another 40% to 100% faster."

    Read the results of the load test HERE.

  • A compendium of the fastest things the world has to offer, and a celebration of the technological breakthroughs that feed the rush
    Both man and machine are approaching the future at an ever-accelerating clip. Almost every year, our vehicles break speed records. This past fall, the X-43A scramjet-powered aircraft reached a speed of nearly Mach 10, beating a record of Mach 6.8 set only six months before. Today’s fastest supercomputer, IBM’s Blue Gene, is about 450,000 times as speedy as the ruling machine of 30 years ago and twice as fleet as the fastest machine of just one year ago. More at Popular Science

  • mysql.logo I found this interesting tool (beside tuning-primer.sh) while trying to optimizing my server setting for Joomla!

    mysqlreport

    mysqlreport makes an easy-to-read report of important MySQL status values. Unlike SHOW STATUS which simply dumps over 100 values to screen in one long list, mysqlreport interprets, formats, and then nicely presents the values in report readable by humans. Numerous example reports are available at the mysqlreport web page.
    The benefit of mysqlreport is that it allows you to very quickly see a wide array of performance indicators for your MySQL server which would otherwise need to be calculated by hand from all the various SHOW STATUS values. For example, the Index Read Ratio is an important value but it is not present in SHOW STATUS; it is an inferred value (the ratio of Key_reads to Key_read_requests).

    Grab it at http://hackmysql.com/mysqlreport

    How to run it (more options), it require PERL to run.

    # ./mysqlreport --user xxxxx--password xxxxxx| more

    Running mysqlreport against my host, gave the following results:

    • very good table lock 0.02%
    • very good read ratio 99.93%
    • good query cache, but could be reduce to 40Mb to avoid wasting memory resource

    if you need something more professional and can afford it, you can try Mysql Enterprise (free for 30 days, enough to tune any small server bottleneck)

    &160;

    &160;

    MySQL 5.0.41-log&160;&160;&160;&160;&160;&160;&160;&160; uptime 4 10:56:4&160;&160;&160;&160;&160;&160; Fri Jan&160; 2 22:45:47 2009

    __ Key _________________________________________________________________
    Buffer used&160;&160;&160;&160; 2.44M of&160;&160; 5.00M&160; %Used:&160; 48.75
    &160; Current&160;&160;&160;&160;&160;&160; 2.97M&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; %Usage:&160; 59.38
    Write hit&160;&160;&160;&160;&160; 47.41%
    Read hit&160;&160;&160;&160;&160;&160; 99.93%

    __ Questions ___________________________________________________________
    Total&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 4.07M&160;&160;&160; 10.6/s
    &160; QC Hits&160;&160;&160;&160;&160;&160; 1.93M&160;&160;&160;&160; 5.0/s&160; %Total:&160; 47.35
    &160; DMS&160;&160;&160;&160;&160;&160;&160;&160; 973.13k&160;&160;&160;&160; 2.5/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 23.89
    &160; Com_&160;&160;&160;&160;&160;&160;&160; 936.64k&160;&160;&160;&160; 2.4/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 23.00
    &160; COM_QUIT&160;&160;&160; 249.35k&160;&160;&160;&160; 0.6/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 6.12
    &160; -Unknown&160;&160;&160;&160; 14.78k&160;&160;&160;&160; 0.0/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.36
    Slow 5 s&160;&160;&160;&160;&160; 123.77k&160;&160;&160;&160; 0.3/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 3.04&160; %DMS:&160; 12.72&160; Log:&160; ON
    DMS&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 973.13k&160;&160;&160;&160; 2.5/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 23.89
    &160; SELECT&160;&160;&160;&160;&160; 589.04k&160;&160;&160;&160; 1.5/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 14.46&160;&160;&160;&160;&160;&160;&160;&160; 60.53
    &160; UPDATE&160;&160;&160;&160;&160; 135.53k&160;&160;&160;&160; 0.4/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 3.33&160;&160;&160;&160;&160;&160;&160;&160; 13.93
    &160; INSERT&160;&160;&160;&160;&160; 125.80k&160;&160;&160;&160; 0.3/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 3.09&160;&160;&160;&160;&160;&160;&160;&160; 12.93
    &160; DELETE&160;&160;&160;&160;&160; 119.91k&160;&160;&160;&160; 0.3/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 2.94&160;&160;&160;&160;&160;&160;&160;&160; 12.32
    &160; REPLACE&160;&160;&160;&160;&160;&160; 2.85k&160;&160;&160;&160; 0.0/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.07&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.29
    Com_&160;&160;&160;&160;&160;&160;&160;&160;&160; 936.64k&160;&160;&160;&160; 2.4/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 23.00
    &160; set_option&160; 411.63k&160;&160;&160;&160; 1.1/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 10.11
    &160; change_db&160;&160; 230.65k&160;&160;&160;&160; 0.6/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 5.66
    &160; show_tables&160; 68.89k&160;&160;&160;&160; 0.2/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1.69

    __ SELECT and Sort _____________________________________________________
    Scan&160;&160;&160;&160;&160;&160;&160;&160;&160; 205.15k&160;&160;&160;&160; 0.5/s %SELECT:&160; 34.83
    Range&160;&160;&160;&160;&160;&160;&160;&160;&160; 27.27k&160;&160;&160;&160; 0.1/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 4.63
    Full join&160;&160;&160;&160;&160; 13.73k&160;&160;&160;&160; 0.0/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 2.33
    Range check&160;&160;&160;&160;&160;&160;&160;&160; 8&160;&160;&160;&160; 0.0/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.00
    Full rng join&160;&160; 4.46k&160;&160;&160;&160; 0.0/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.76
    Sort scan&160;&160;&160;&160;&160; 76.29k&160;&160;&160;&160; 0.2/s
    Sort range&160;&160;&160; 110.20k&160;&160;&160;&160; 0.3/s
    Sort mrg pass&160;&160;&160;&160;&160;&160; 0&160;&160;&160;&160;&160;&160; 0/s

    __ Query Cache _________________________________________________________
    Memory usage&160;&160; 25.86M of&160; 70.00M&160; %Used:&160; 36.94
    Block Fragmnt&160; 16.52%

    Hits&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1.93M&160;&160;&160;&160; 5.0/s
    Inserts&160;&160;&160;&160;&160;&160; 533.75k&160;&160;&160;&160; 1.4/s
    Insrt:Prune&160;&160; 13.29:1&160;&160;&160;&160; 1.3/s
    Hit:Insert&160;&160;&160;&160; 3.61:1

    __ Table Locks _________________________________________________________
    Waited&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 269&160;&160;&160;&160; 0.0/s&160; %Total:&160;&160; 0.02
    Immediate&160;&160;&160;&160;&160;&160; 1.71M&160;&160;&160;&160; 4.5/s

    __ Tables ______________________________________________________________
    Open&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1482 of 2000&160;&160;&160; %Cache:&160; 74.10
    Opened&160;&160;&160;&160;&160;&160;&160;&160; 44.50k&160;&160;&160;&160; 0.1/s

    __ Connections _________________________________________________________
    Max used&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 16 of&160;&160; 25&160;&160;&160;&160;&160; %Max:&160; 64.00
    Total&160;&160;&160;&160;&160;&160;&160;&160; 250.45k&160;&160;&160;&160; 0.7/s

    __ Created Temp ________________________________________________________
    Disk table&160;&160;&160;&160; 65.75k&160;&160;&160;&160; 0.2/s
    Table&160;&160;&160;&160;&160;&160;&160;&160; 198.32k&160;&160;&160;&160; 0.5/s&160;&160;&160; Size: 120.0M
    File&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 5&160;&160;&160;&160; 0.0/s

    __ Threads _____________________________________________________________
    Running&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1 of&160;&160;&160; 3
    Cached&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 13 of&160;&160; 32&160;&160;&160;&160;&160; %Hit:&160; 99.99
    Created&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 16&160;&160;&160;&160; 0.0/s
    Slow&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0&160;&160;&160;&160;&160;&160; 0/s

    __ Aborted _____________________________________________________________
    Clients&160;&160;&160;&160;&160;&160;&160;&160; 2.20k&160;&160;&160;&160; 0.0/s
    Connects&160;&160;&160;&160;&160;&160;&160; 3.41k&160;&160;&160;&160; 0.0/s

    __ Bytes _______________________________________________________________
    Sent&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1.48G&160;&160;&160; 3.8k/s
    Received&160;&160;&160;&160;&160; 757.33M&160;&160;&160; 2.0k/s

    __ InnoDB Buffer Pool __________________________________________________
    Usage&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 7.98M of&160;&160; 8.00M&160; %Used:&160; 99.80
    Read hit&160;&160;&160;&160;&160;&160; 99.80%

    Pages
    &160; Free&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; %Total:&160;&160; 0.20
    &160; Data&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 510&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 99.61 %Drty:&160;&160; 0.00
    &160; Misc&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.20
    &160; Latched&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.00
    Reads&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 1.03M&160;&160;&160;&160; 2.7/s
    &160; From file&160;&160;&160;&160; 2.10k&160;&160;&160;&160; 0.0/s&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0.20
    &160; Ahead Rnd&160;&160;&160;&160;&160;&160;&160; 79&160;&160;&160;&160; 0.0/s
    &160; Ahead Sql&160;&160;&160;&160;&160;&160;&160;&160; 6&160;&160;&160;&160; 0.0/s
    Writes&160;&160;&160;&160;&160;&160;&160;&160; 45.01k&160;&160;&160;&160; 0.1/s
    Flushes&160;&160;&160;&160;&160;&160;&160; 12.42k&160;&160;&160;&160; 0.0/s
    Wait Free&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0&160;&160;&160;&160;&160;&160; 0/s

    __ InnoDB Lock _________________________________________________________
    Waits&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0&160;&160;&160;&160;&160;&160; 0/s
    Current&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    Time acquiring
    &160; Total&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0 ms
    &160; Average&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0 ms
    &160; Max&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0 ms

    __ InnoDB Data, Pages, Rows ____________________________________________
    Data
    &160; Reads&160;&160;&160;&160;&160;&160;&160;&160; 2.30k&160;&160;&160;&160; 0.0/s
    &160; Writes&160;&160;&160;&160;&160;&160; 23.18k&160;&160;&160;&160; 0.1/s
    &160; fsync&160;&160;&160;&160;&160;&160;&160; 14.15k&160;&160;&160;&160; 0.0/s
    &160; Pending
    &160;&160;&160; Reads&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    &160;&160;&160; Writes&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    &160;&160;&160; fsync&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0

    Pages
    &160; Created&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 5&160;&160;&160;&160; 0.0/s
    &160; Read&160;&160;&160;&160;&160;&160;&160;&160;&160; 2.60k&160;&160;&160;&160; 0.0/s
    &160; Written&160;&160;&160;&160;&160; 12.42k&160;&160;&160;&160; 0.0/s

    Rows
    &160; Deleted&160;&160;&160;&160;&160;&160;&160;&160; 843&160;&160;&160;&160; 0.0/s
    &160; Inserted&160;&160;&160;&160;&160; 2.07k&160;&160;&160;&160; 0.0/s
    &160; Read&160;&160;&160;&160;&160;&160;&160; 107.49k&160;&160;&160;&160; 0.3/s
    &160; Updated&160;&160;&160;&160;&160;&160; 2.83k&160;&160;&160;&160; 0.0/s

  • google_logospeedapache_logo

    mod_pagespeed is the latest addition to our family of products, performing on-the-fly optimization in the Apache™ 2 HTTP Server. It provides webmasters and web developers that manage their Apache 2 configuration with technology that applies Page Speed performance best practices automatically. Webmasters can get the benefits of the optimizations without having to manually going back and fixing the performance issues that Page Speed detects.

    This solution is valid if you do not want to carefully optimize your site, and it come for now at huge initial CPU costs till the cache is filled up. The load on my server has proven to be unacceptable (for me) and I was forced to switch it off. But it may work on your server depending on the page size, number of visitors. Authors are working hard and communicating a lot on the official mailing list to reduce that load and improve the code in the coming weeks

    If you trust me, you can download the library module libmod_pagespeed.so for Apache 2.2

    from http://drivers.waltercedric.com/

    Check before installing the MD5 keys of this file

    # md5sum -b mod_pagespeed.so
    187995e3623a222ec5b54c331ee7ffaa *mod_pagespeed.so

    If it matches, drop it into your Apache library folder /usr/lib64/apache2/

    Build Apache module mod_pagespeed yourself

    On my OpenSuSE system, I was forced to install first

    zypper in gcc-c++

    Then read  http://code.google.com/p/modpagespeed/wiki/HowToBuild

    you should get after a successful build

    • ./out/Release/libmod_pagespeed.so
    • ./out/Release/install/common/pagespeed.conf

    Install the module library

    cp ./out/Release/libmod_pagespeed.so  /usr/lib64/apache2/mod_pagespeed.so

    Configuration

    After installing mod_pagespeed, it can be customized and configured by editing the Apache configuration file

    /etc/apache2/conf.d/pagespeed.conf

    so you can run

    # cp ./out/Release/install/common/pagespeed.conf /etc/apache2/conf.d/pagespeed.conf

    Create some directories

    # mkdir /var/mod_pagespeed/
    # mkdir /var/mod_pagespeed/cache/
    # mkdir /var/mod_pagespeed/files

    Has to be writtable by apache

    chown -R wwwrun:www /var/mod_pagespeed/

    and finally restart apache

     apache2ctl restart

    There is a lot of available settings that are all well explained in this page http://code.google.com/speed/page-speed/docs/using_mod.html

    Here is my /etc/apache2/conf.d/pagespeed.conf stripped of all comments as an example:

    LoadModule pagespeed_module /usr/lib64/apache2/mod_pagespeed.so
    
        SetOutputFilter MOD_PAGESPEED_OUTPUT_FILTER
        ModPagespeed on
    
        ModPagespeedDomain www.waltercedric.com
        ModPagespeedUrlPrefix                "http://www.waltercedric.com/mod_pagespeed/"
        ModPagespeedFileCachePath "/var/mod_pagespeed/cache/"
        ModPagespeedGeneratedFilePrefix "/var/mod_pagespeed/files/"
        ModPagespeedRewriteLevel CoreFilters
        ModPagespeedEnableFilters collapse_whitespace
        ModPagespeedEnableFilters extend_cache
        ModPagespeedEnableFilters combine_css
        ModPagespeedEnableFilters rewrite_css
        ModPagespeedEnableFilters rewrite_javascript
        ModPagespeedEnableFilters rewrite_images
        ModPagespeedEnableFilters remove_comments

    # This page lets you view statistics about the mod_pagespeed module. Order allow,deny # You may insert other "Allow from" lines to add hosts you want to # allow to look at generated statistics. Another possibility is # to comment out the "Order" and "Allow" options from the config # file, to allow any client that can reach your server to examine # statistics. This might be appropriate in an experimental setup or # if the Apache server is protected by a reverse proxy that will # filter URLs in some fashion. Allow from localhost SetHandler mod_pagespeed_statistics

    Troubleshooting

    You may be forced to remove mod_deflate

    You can disable mod_pagespeed by adding in the url  ?mod_page_speed=0

    Some mod_pagespeed filters

    Links

  • joomla_cms

    I did optimize a bit my Joomla! homepage in the last few days. This has been achieved with

    • Enabling Joomla module caching in all 3rd party module where it was missing/ not implemented at all,
    • Starting to offload some assets (JavaScript) to faster hosting,

    Click read more to apply the same for your internet site.

    Use Joomla Module caching

    Not all 3rd party Joomla modules are using caching. This means that in worst case, some Joomla! modules may create way too much SQL queries. A way to reduce the load is to activate module caching. You’ll have to go through all 3rd party modules and check that they have in their administrator panel a setting to enable/disable cache.

    jooomla.enable.caching

    You’ll see that 90% of all modules (except official Joomla! modules which are able to deal with caching) are NOT supporting caching. We will change that now:

    For every module without cache, open the xml file at /modules/mod_xxxxxxx/mod_xxxxxxx.xml and add between <params> .. </params>

    <param name="cache" type="radio" default="0" label="Enable Cache" 
           description="Select whether to cache the content of this module">
     <option value="0">No</option>
     <option value="1">Yes</option>
    </param>

    Note that if <params> .. </params> do not exist, just add it like below

    <params>
     <param name="cache" type="radio" default="0" label="Enable Cache" 
           description="Select whether to cache the content of this module">
      <option value="0">No</option>
      <option value="1">Yes</option>
     </param>
    </params>

    Visit or reload the admin panel of that module and set the Enable Cache to Yes.Click Save/Apply at least once.

    Now the output of this module will be saved in /cache and only refresh when global Joomla cache timeout (900 seconds as default). Consider also contacting the author of the module so he can patch his code.

    Offload assets

    Offloading assets (JavaScript, static images, static files) can bring tremendous speed gains, at the cost of resolving more DNS name. Using this technique will help your Apache concentrate on php instead of streaming static data.

    Offload JavaScript

    When you look at Joomla! frontend source code, you will see that the JavaScript library mootols.js is 74kb big. Google is offering to host all major AJAX  libraries free of charge at http://code.google.com/apis/ajaxlibs/documentation/ so why not profiting of their datacenter speed/bandwidth/response time?

    Now the dirty part, You can’t tell Joomla! not to include the mootols.js from /media/system/js/mootools.js at rendering time. We will have to patch the code of Joomla!

    open /libraries/joomla/html/html/behavior.php and search for

    if ($debug || $konkcheck) {
      JHTML::script('mootools-uncompressed.js', 'media/system/js/', false);
    } else {
     //JHTML::script('mootools.js', 'media/system/js/', false); // old Joomla code
     JHTML::script('mootools-yui-compressed.js', 'http://ajax.googleapis.com/ajax/libs/mootools/1.11/', false);
    }

    Joomla use mootools.js in version 1.11, don’t use the latest version (1.2.3) as most Joomla! plugin wont work (but your mileage may vary).

    To be continued

  • According to this MySQL page here, you can win 30 to 50% more performances using MySQL jdbc named pipe!

    Named pipes only work when connecting to a MySQL server on the same physical machine as the one the JDBC driver is being used on.
    In simple performance tests, it appears that named pipe access is between 30%-50% fasterthan the standard TCP/IP access.


    As default, when you install mySQL on windows using the installer, TCP IP is the default option. The question remains why nobody seems
    to use named pipe, or has write any articles on internet about it. Lorenz and me were trying nearly 2 hours to make this damn things work.
    We googled on internet, finding nothing, e try and try until we finally succeed. That is the reason why I write this article now.

    We were testing successfuly MySQL 5.0 using named pipe, and what an increase in speed! 50% in the case of this big project
    First it may be a good idea to use the latest JConnector driver from MySQL

    MySQL Connector/J is a native Java driver that converts JDBC (Java Database Connectivity) calls into the network protocol used by
    the MySQL database. It lets developers working with the Java programming language easily build programs and applets that interact
    with MySQL and connect all corporate data, even in a heterogeneous environment. MySQL Connector/J is a Type IV JDBC driver and
    has a complete JDBC feature set that supports the capabilities of MySQL.

    Download Binaries & Source

    Just put the file mysql-connector-java-5.0.7-bin.jar in your classpath. You'll have to sligthly alter the my.ini file as following:

    [mysqld]
    skip-networking
    enable-named-pipe
    socket=mysql.sock


    Add these 3 keys in [mysqld]section of my.ini and restart MySQL. The first key switch the TCP-IP port off, so you wont be able
    now to connect to port 3306. The 2 others remaining just activate MySQL named pipe.

    Verify that everything has work correctly by firing MySQL Query Browser and connect to your database.
    (see details of connection below)
    mysql query browser with named pipe

     For the most intrepid of You of if wou want to add named pipe capabilities to previous saved connections, just can also use the menu "tools" - "manage connections" and under the tab "advanced parameters", just add these 2 new keys

    NAMED_PIPE    value  Yes
    SOCKET_PATH   value   mysql.sock

    We are nearly finished, all we have to do is to alter now the JDBC url, and this is where we fight against Windowstm till we find the path to the mysql.sock file handle.
    For the unpatient of you, I deliver here the solution:

    jdbc:mysql:///database
    ?socketFactory=com.mysql.jdbc.NamedPipeSocketFactory
    &namedPipePath=\\\\.\\Pipe\\mysql.sock


    database being the database/schema name

    socketFactory=com.mysql.jdbc.NamedPipeSocketFactory  JConnector also supports access to MySQL via named pipes on Windows NT/2000/XP using
    the
    NamedPipeSocketFactory as a plugin-socket factory via thesocketFactory property. If you don't use anamedPipePath property, the default
    of '\\.\pipe\MySQL' will be used. If you use the
    NamedPipeSocketFactory, the hostname and port number values in the JDBC url will be ignored.
    You can enable this feature using:   socketFactory=com.mysql.jdbc.NamedPipeSocketFactory

    namedPipePath=\\\\.\\Pipe\\mysql.sock The path to the file socket. Notice how strange the path is looking like under Windowstm (escaping  \ in java is normal).
    Under linux we would have write /var/log/mysql.sock and forget everything. In Windowstm  You really have no chance to find it until  you use
    FileMon(a SysInternals tool)

    FileMonmonitors and displays file system activity on a system in real-time. Its advanced capabilities make it a powerful tool for exploring the way
    Windows works, seeing how applications use the files and DLLs, or tracking down problems in system or application file configurations. Filemon's
    timestamping feature will show you precisely when every open, read, write or delete, happens, and its status column tells you the outcome.
    FileMon
    is so easy to use that you'll be an expert within minutes. It begins monitoring when you start it, and its output window can be saved to a file for off-line
    viewing. It has full search capability, and if you find that you're getting information overload, simply set up one or more filters.
    Download it HERE

    You can use FileMon to filter file by name, search for mysql* and you'll see that strange url. Note the documentation give some advice about this url
    (more or less)  the default of '\\.\pipe\MySQL', would have work if we have name the file MySQL and not mysql.sock

    But wait there is more to learn, MySQL is supporting a wide range of parameters when you open the connection. Just read this page, a lot of settings may also speed your application even more.
  • I publish here some of my server settings in the hope that it will also help others...

    Server Setup

    AMD64 1 Gb RAM, Linux OSS 10.0, 7 Joomla instances (one being waltercedric.com with 250'000 unique visitors per months), 1 simple machine forums, 3 gallery2 install

    All MySQL tables are myISAM (table locking instead of row loacking in innodb, myIsam make sense as ther is more read then insert) 

    # vi  in /etc/my.cnf

    Below the diff command between a standard MySQL install

    # diff my.cnf my-beforeOptimizations.cnf
    30c30
    < key_buffer = 50M
    ---
    > key_buffer = 16M
    32,33c32,33
    < table_cache = 1500
    < sort_buffer_size = 4M
    ---
    > table_cache = 64
    > sort_buffer_size = 512K
    35,46c35,37
    < read_buffer_size = 4M
    < read_rnd_buffer_size = 1024K
    < myisam_sort_buffer_size = 64M
    < join_buffer_size = 4M
    < thread_cache_size = 128
    < wait_timeout = 14400
    < connect_timeout = 10
    < max_connect_errors = 10
    < query_cache_limit = 2M
    < query_cache_size = 128M
    < query_cache_type = 1
    < thread_concurrency=4
    ---
    > read_buffer_size = 256K
    > read_rnd_buffer_size = 512K
    > myisam_sort_buffer_size = 8M
    163,166c154,157
    < key_buffer = 64M
    < sort_buffer = 64M
    < read_buffer = 16M
    < write_buffer = 16M
    ---
    > key_buffer = 20M
    > sort_buffer_size = 20M
    > read_buffer = 2M
    > write_buffer = 2M

  • Digging into mySQL settings can be time consuming, like with any other component. Most of the time, and just by changing a few settings, you can expect a performance increase. The problem is to change what, and  to which value. This is where tuning-primer.sh help you:

    Get this script, http://forge.mysql.com/projects/view.php?id=44 upload it, unzip it, and install it in your /etc folder. Then run it from the command line by entering ./path-to-file/tuning-primer.sh

    MySQL Server must run a few days or weeks, or it wont be be safe to follow these recommendations.

    To find out more information on how each of these runtime variables effects performance visit:
    http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html

    Here is an example of  tuning-primer.sh output

    SLOW QUERIES
    Current long_query_time = 5 sec.
    You have 2856 out of 4725688 that take longer than 5 sec. to complete
    The slow query log is enabled.
    Your long_query_time seems to be fine

    WORKER THREADS

    Current thread_cache_size = 128
    Current threads_cached = 55
    Current threads_per_sec = 0
    Historic threads_per_sec = 0
    Your thread_cache_size is fine

    MAX CONNECTIONS

    Current max_connections = 100
    Current threads_connected = 15
    Historic max_used_connections = 55
    The number of used connections is 55% of the configured maximum.
    Your max_connections variable seems to be fine.

    MEMORY USAGE
    Max Memory Ever Allocated : 305 M
    Configured Max Per-thread Buffers : 1017 M
    Configured Max Global Buffers : 143 M
    Configured Max Memory Limit : 1 G
    Total System Memory : 2.99 G
    Max memory limit seem to be within acceptable norms

    KEY BUFFER
    Current MyISAM index space = 4 M
    Current key_buffer_size = 5 M
    Key cache miss rate is 1 : 3740
    Key buffer fill ratio = 35.00 %
    Your key_buffer_size seems to be too high.
    Perhaps you can use these resources elsewhere


    QUERY CACHE
    Query cache is enabled
    Current query_cache_size = 128 M
    Current query_cache_used = 27 M
    Current query_cach_limit = 2 M
    Current Query cache fill ratio = 21.13 %
    Your query_cache_size seems to be too high.
    Perhaps you can use these resources elsewhere
    MySQL won't cache query results that are larger than query_cache_limit in size


    SORT OPERATIONS
    Current sort_buffer_size = 4 M
    Current record/read_rnd_buffer_size = 1020 K
    Sort buffer seems to be fine

    JOINS
    Current join_buffer_size = 1.00 M
    You have had 7065 queries where a join could not use an index properly
    You should enable "log-queries-not-using-indexes"
    Then look for non indexed joins in the slow query log.
    If you are unable to optimize your queries you may want to increase your
    join_buffer_size to accommodate larger joins in one pass.


    Note! This script will still suggest raising the join_buffer_size when
    ANY joins not using indexes are found.

     


  • Since I am facing performance problem due to the load of visitors and a badly configured server, I decide today to document my findings in this new series of articles.

    FastCGI is an extention to CGI that increases speed. Instead of creating a new process every time a page is reloaded, FastCGI will use the same process, queuing up requests. This increases startup speed, at the expense of forcing users to wait for the previous requests to finish. FastCGI also allows programs to get the web server to do certain operations, like reading in a file, before the request is handed over. [WikiPedia]

    FastCGI provides a way to keep PHP frequently-run script files loaded on Your server so that script are already preloaded and initialized the next time they are used. This can make some scripts run up to five times more quickly (in most cases it's as fast as mod_php, without the drawbacks). Moreover You can run different version of PHP (PHP4 and PHP5) in the same server...

    Server: Strato (www.strato.de)
    Operating system: SuSE / openSuSE
    Requirements: root access and basic Unix knowledge

     

    1 Use yast2 to install mod_fastcgi on the system, this will create files (*.so) in   /usr/lib/apache2-worker/ and  /usr/lib/apache2-prefork/
    2 Edit the file  
    # vi /etc/sysconfig/apache2

    and add fastcgi, suexec
    remove php4/php5 if any

    APACHE_MODULES="fastcgi cache mem_cache deflate headers access actions alias auth auth_dbm autoindex cgi dir env expires include log_config mime negotiation setenvif ssl   userdir php4 php5 perl     rewrite suexec"
    3 create a new file in /etc/apache2/conf.d/mod_fastcgi.conf

    vi /etc/apache2/conf.d/mod_
    fastcgi.conf

    and add these lines, at the end of file, adapt all path according to your system. Keep in mind, I have a SuSE system


    <IfModule mod_fastcgi.c>
        ScriptAlias /fcgi-bin/ /srv/www/fcgi-bin/
        FastCgiIpcDir /var/lib/apache2/fastcgi
        # FastCGI applications can be static, dynamic, or external.
        # Dynamic FastCGI applications can be put in this directory.
        # Use the FastCgiConfig directive in your httpd.conf.local
        # to change the defaults for them.
        # http://www.fastcgi.com/mod_fastcgi/docs/
        # mod_fastcgi.htmlFastCgiConfig

        <Directory "/srv/www/fcgi-bin">
            AllowOverride None
            Options +ExecCGI -Includes
            SetHandler fastcgi-script
            Order allow,deny
            Allow from all
        </Directory>
        AddHandler fastcgi-script fcg fcgi fpl

        # Static or external FastCGI applications are started when apache
        # is started. They are configured with the
        # FastCgiServer and FastCgiExternalServer directives:
        # http://www.fastcgi.com/mod_fastcgi/docs/
        # mod_fastcgi.htmlFastCgiServer

        # http://www.fastcgi.com/mod_fastcgi/docs/
        # mod_fastcgi.htmlFastCgiExternalServer

        # which can be put into httpd.conf.local.
        #
        # Examples:
        FastCgiServer /srv/www/fcgi-bin/echo -processes 5
        FastCgiServer /srv/www/htdocs/some/path/echo.fcgi
        # see http://www.FastCGI.com/docs/faq.htmltypical_httpd.conf
        # for more  information...


        Alias /php5-fcgi/ /srv/www/cgi-bin/
        FastCgiServer /srv/www/cgi-bin/php5  -processes 5
        AddType application/x-httpd-fastphp5 .php5
        Action application/x-httpd-fastphp5 /php5-fcgi/php5

        Alias /php4-fcgi/ /srv/www/cgi-bin/
        FastCgiServer /srv/www/cgi-bin/php  -processes 5
        AddType application/x-httpd-fastphp4 .php
        Action application/x-httpd-fastphp4 /php5-fcgi/php

    </IfModule>

    4 start
    # apache2-reconfigure-mpm
    this recreate the file  /etc/apache2/sysconfig.d/loadmodule.conf
    and restart apache automatically.

    References:


  • Since I am facing performance problem due to the load of visitors and a badly configured server, I decide today to document my findings in this new series of articles.

    mod_cache
    implements an RFC 2616 compliant HTTP content cache that can be used to cache either local or proxied content. mod_cache requires the services of one or more storage management modules.

    Server: Strato (www.strato.de)
    Operating system: SuSE / openSuSE
    Requirements: root access and basic Unix knowledge



    1 Use yast2 to install mod_cache on the system, this will create files (*.so) in   /usr/lib/apache2-worker/ and  /usr/lib/apache2-prefork/
    2 Edit the file  
    # vi /etc/sysconfig/apache2

    and add cache and mem_cache
    APACHE_MODULES="cache mem_cache deflate headers access actions alias auth auth_dbm autoindex cgi dir env expires include log_config mime negotiation setenvif ssl   userdir php4 php5 perl     rewrite suexec"
    3 create a new file in /etc/apache2/conf.d/mod_cache.conf

    vi /etc/apache2/conf.d/mod_cache.conf

    and add these lines, at the end of file, adapt all path according to your system.

     <IfModule mod_cache.c>
    LoadModule disk_cache_module modules/mod_disk_cache.so
    <IfModule mod_disk_cache.c>
     CacheRoot /tmp/cacheroot
     CacheSize 256
     CacheEnable disk /
     CacheDirLevels 5
     CacheDirLength 3
    </IfModule>

    <IfModule mod_mem_cache.c>
     CacheEnable mem /
     MCacheSize 4096
     MCacheMaxObjectCount 100
     MCacheMinObjectSize 1
     MCacheMaxObjectSize 2048
     </IfModule>
    </IfModule>

    4 start
    # apache2-reconfigure-mpm
    this recreate the file  /etc/apache2/sysconfig.d/loadmodule.conf
    and restart apache automatically.

    References:


  • Since I am facing performance problem due to the load of visitors and a badly configured server, I decide today to document my findings in this new series of articles.

    The mod_deflate module provides the DEFLATE output filter that allows output from your server to be compressed before being sent to the client over the network.

    Server: Strato (www.strato.de)
    Operating system: SuSE / openSuSE
    Requirements: root access and basic Unix knowledge

    Why installing mod_deflate? is PHP gzip not enough?
    PHP Gzip only compress PHP output, not unical file, like CSS, Javascipt, fixed HTML, other ressources.

    1 Use yast2 to install mod_deflate on the system, this will create files (*.so) in   /usr/lib/apache2-worker/ and  /usr/lib/apache2-prefork/
    2 Edit the file  
    # vi /etc/sysconfig/apache2

    and add deflate and headers(optional A)
    APACHE_MODULES="deflate headers access actions alias auth auth_dbm autoindex cgi dir env expires include log_config mime negotiation setenvif ssl   userdir php4 php5 perl     rewrite suexec"
    3 create a new file in /etc/apache2/conf.d/mod_deflate.conf

    vi /etc/apache2/conf.d/mod_deflate.conf


    and add these lines, at the end of file, adapt all path according to your system.

    <IfModule mod_deflate.c>
      # place filter 'DEFLATE' on all outgoing content
      SetOutputFilter DEFLATE
      AddOutputFilterByType DEFLATE text/html text/plain text/xml

      # exclude uncompressible content via file type
      SetEnvIfNoCase Request_URI \.(?:gif|jpeg|jpg|png|rar|zip)$ no-gzip dont-vary
      SetEnvIfNoCase Request_URI \.(?:mpg|avi|mpeg|wmv|mp3|wma|ogg)$ no-gzip dont-vary
      <IfModule mod_headers.c>
        #
    Aproperly handle requests coming from behind proxies
        Header append Vary User-Agent
      </IfModule>

      # deflate.log, log compression ratio on each request
      DeflateFilterNote Input instream
      DeflateFilterNote Output outstream
      DeflateFilterNote Ratio ratio
      LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate
      CustomLog /var/log/apache2/deflate.log deflate

      # Properly handle old browsers that do not support compression
      BrowserMatch ^Mozilla/4 gzip-only-text/html
      BrowserMatch ^Mozilla/4\.0[678] no-gzip
      BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
    </IfModule>
     
    4 start
    # apache2-reconfigure-mpm
    this recreate the file  /etc/apache2/sysconfig.d/loadmodule.conf
    and restart apache automatically.
    verification: View HTTP Request and Response Header with http://web-sniffer.net/

    search for Content-Encoding: gzip

    References:
  • YSlow analyzes web pages and tells you why they're slow based on the rules for high performance web sites. YSlow is a Firefox add-on integrated with the popular Firebug web development tool. YSlowgives you:

    • Performance report card
    • HTTP/HTML summary
    • List of components in the page
    • Tools including JSLint

    Download HERE
    Visit also the homepage of the team behind this tool: Yahoo!'s Exceptional Performance Team

    You should not take too much time to guess that I am currently optimizing my homepage :-)
    Be careful, running this tool more than 5 times make my Firefox crash ;-)

    Some statistics:
    Some useful links
    Be careful! a lot of host get pretty low metering (just test your favorite internet site), but are still very fast-> this tools do not tell You their infrastructure: memcache server, multiple db server, multiple file server for static content and so on.