AJAXify phpMyAdmin Interface - GSOC 2011 - First week

This is the fist week of coding time period in my gsoc 2011 project with phpMyAdmin(PMA). On 23rd of May the official coding time started. Since there were some free time with my in the community building period I started my works on before hand. It made me easy to be familiar with the code base and style which is used in PMA.

My first task is give the ajax behavior to the "Create Table" option in the navigation panel of the database. Initially when we click on the "Create Table" option it directs the content frame to a new page. So I wanted to give a dialog box which loads the create table form and then submit it using ajax.

First I went through a similar feature in the "database->structure" and "database->operation" by using the "create_table_form_minimal" form. At that case I got some knowledge on how we can use dialog boxes for this kind of scenario. Then I tried to find out of them what are the code segments which I can reuse for my task. There were three actions which used for "Create Table" option in "function.js". Those are


("#create_table_form_minimal").live('submit') - triggers when the initial form has submit.
("#create_table_form input[name=do_save_data]").live('click') - triggers when the save button clicks in the form.
("#create_table_form.ajax input[name=submit_num_fields]").live('click') - triggers when the add new column option clicks.


Out of those I realized that the second and third actions are common to both cases. I had to add a new action instead of the first one.

Then I planed how to structure my new option. After discussing with Marc I decided to add my code the "navigation.js" so it is logical and easy to use in the future. Then I developed the (#newtable).live('click') action in "naviation.js" file and after discuss with Marc I put the common logic to the "PMA_createTableDialog()" function in the "fucntion.js" file.
Finally Marc requested a modification on changing the "Add 'x' column(s) Go" option to the top of the create table form and I did that. Now the "Create Table" option in the navigation panel works fine and I think the modification added some more usability on that.
I have successfully finished my first task of my project. Next I 'm willing to work with adding the ajax behavior to the "database->privileges" option.

AJAXify phpMyAdmin Interface - Gsoc 2011

This time my proposal on "AJAXify phpMyAdmin Interface" got accepted for Google Summer of Competition with phpMyAdmin (PMA). Marc Delisle, Michal Čihař and Ninad Pundalik who is the last year Gsoc contributor helped me a lot when I improve my patches and the proposal. Marc Delisle act as my mentor for my project in this summer.

Ajaxifing the PMA is a project idea which continues from the last year gsoc session which was done by Ninad Pundalik. Last year he has improved some important features by giving the ajax behaviour to them in the PMA. Since ajax is a trendy behaviour among web domain and since it improves the usability of the system, moving in to ajax is a huge improvement step on PMA. This time I’m willing to improve the usability of some identified important features in PMA by applying ajax behavior using jquery library. After improving those features, they will be able to provide a quick and consistent flow of operations to the PMA users.

Next series of blog post will discuss more about my project in this summer.

Optical Character recognition with Tesseract

Since I was very busy with my final year stuff I could not write a blog post for months. But today I got some time to introduce an amazing tool for you, which will help you when dealing with the Optical Character Recognition(OCR) domain. Lets get a basic knowledge on the OCR domain, before moving to Tesseract.

What is OCR?

"Translating the handwritten, typewritten or printed texts in a scanned image to a machine encoded text."

This OCR concept is very important and widely used in converting important documents which are in image formats to text format. The first major use of OCR was in processing petroleum credit card sales drafts. This application provides recognition of the purchaser from
the imprinted credit card account number and the introduction of a transaction. Most of the modern scanners provides the feature of OCR enabling for the images scanned using it which will directly output the text content of the scanned image. This concept helps to save a huge time which will waste on rewriting the image contents. There are lots of pattern recognition and image processing activities involved with this field and this is an very interesting and research area in the current computing word.

What is Tesseract?

"The Tesseract OCR engine was one of the top 3 engines in the 1995 UNLV Accuracy test and it is probably one of the most accurate open source OCR engines available. The source code will read a binary, grey or color image and output text. A tiff reader is built in that will read uncompressed TIFF images. Tesseract is licensed under the apace license and any one can use it for free. The core developer on the project is Ray Smith "

Sometimes back I involved with a project to modifying an OCR module for Sahana Disaster Management System. So I searched for an OCR engine for integrate it with the SahanaOCR system. Tesseract was the best solution I found. It has an immense power on recognizing printed text and also it is very easy to localize for your own mother languages.

How to install Tesseract on your PC ?.

Tessearct provides a set of binary executable for Windows and installers for Linux systems. The detailed installation guild is available here.

The Windows users can directly download the latest exe file here. The installation guide for the Linux users also provided at the general installation guide . Then you have to download the trained data folder which is available at the download page. Currently there are trained data sets for many languages such as English, Chines, Spanish etc. You can download any of those tessdata files and extract it into your working directory.

Then there you can go into the directory where the tesseract.exe is located and use the following command to recognize the images and finally to output the recognized characters in an text file.

tesseract image outputbasename [-l lang] [configs]

In this the image file is name of your image using for OCR and the output_base_name is the name of the text file which will contains the recognized output. The images should be in TIFF format and you can use the Libtiff library for handling compressed images with Tesseract. Other configurations are optional and they facilitate for more advance features.

Training Tesseract.

There is another amazing feature provided by Tesseract which is customizing it for your local languages. For that initially we have to train Tessearct for the new language. Here is the training guide for the Tessearct.

For training it, you have to have a proper training set of characters. Then after you select a proper training set you can follow the training procedure which is described at the guide. After following all the steps it will provide you a lang.trainddata file which contains all the newly trained data.

Tessbase API.

Now you have a good idea on how to use Tesseract for OCR. But thats all for using Tesseract on its own. What if we need to integrate it with our own application. So for that Tesseract provides a highly user friendly API which is named as Tessbase API. So you can use tessbaseAPI class to call the internal functions provided by Tesseract.

Here is the general forum where the common discussions carried out on Tesseract project and you can find more details on the project by subscribing to that.

My project on SahanaOCR Gsoc 2010

This is my first post after about 8 months from my last post. Since I was very busy on the training period and the Gsoc competition I was unable to write a post o my blog. How ever again I managed to find some time to write a blogpost fortunately. So I decided to continue the blogpost on SahanaOCR project from where I have stopped at my last post.

So as I mentioned earlier since I have worked with the SahanaOCR system before I came up with the idea of applying for the Gsoc competition for the SahanaOCR project. So I have wrote a proposal on the project with discussing the major implementations and developments that I'm going to do in the project in the session. Here is my project proposal, So you may have a better idea on how to write acceptable project proposal for the Gsoc competition.

My project extension plan was to improve the most necessary features of the current project. The proposal propose 5 main ideas for the existinng Sahana OCR project

1. Make the system platform independent

2. Integrate the Tesseract code to the project

3. Adding the feature of differentiating the different forms by a bar-code embedded to the forms automatically handles those forms.

4. Read the labels of the data fields and identify the type of data field so can give restrictions to the Tesseract to identify the numbers or letters separately.

5. Automatically send the data to the databases of the relevant modules

So after completing these steps the Sahana OCR project will be able to complete the process of extracting data from forms with a great accuracy.

After being active on the Sahana mailing list and exchanged lot of ideas on the project with earlier mentors and contributors they have accepted my project proposal for the Gsoc 2010 competition. that was a day which gave me the same happiness as I passed my Advance Level exam. My mentor was Mr. Gihan Chamara who was graduated from my department last year. During the whole project he was the great strength for me and also Mr chamindra, Mr Jo and Mr Hayesha guided me to make this task success. I have started my project implementations on the 24^th May and it gave me lot of memorable experiences to my life during the whole project time which ran up to 10^th September.

It taught me how to plane a project, how to overcome the challenges we met during that, how to behave withing the public communities, how to get helps form others as well as some more things which were new to me.

Here is the wiki page that I have maintained during the project and you can get the more information on the technical details and how the project ran by referring that.

Here is the link for my project report which I have submitted at the end of the project period and it contains all the details of the current status of the project.

Here is my project branch which I have submitted my code in the Launchpad.

Even though the Gsoc competition and over my effort on the SahanaOCR has not over. So my wish is to continue on working with it and be with it until it became a applicable standalone product.

Sahana Ocr

I have explained about the Sahana project I my previous blog post. So today I'm going to explain about the Sahana OCR project which is an sub module in the main Sahana project. You can refer my research paper about the extension I did for this project.

Sahana Ocr project was done to automate the data entering process to the system and it was developed on .net framework using Visual C++. It was developed by many volunteers as well as gsoc contributers as Omega and Gihan in past two years. I have selected this project as my level 3 project and did some improvements to the existed system to improve the character recognition of the module.

After a disaster situation we have to collect the data about the victims and manage them using Sahana as I mentioned in my earlier post. So we can download the form template by the Sahana server, print the forms in to hard copies, distribute the forms to the victims, ask them to fill the forms and collect the forms back. Then we have to enter those data manually to the Sahana system. But in the case of a large disaster the manually data entering process is difficult and very tome consuming. So Sahana Ocr project is developed to automate that data entering process by using optical character recognition (OCR) technology.

This is a sample of the forms distributed to the victims and you can refer the wiki abou this module to get a better understanding about the forms and the xforms. sahana_xform

So in this we have to provide the scanned image of the filled form and a xml file called xform which have the coordinates of the data fields in the template if the form to the Sahana Ocr module. Then the module sent the two files to the Form_processor which is the manager class of the system. First Form_processor do the rotation compensation to the image. The system uses five back squares at the form to identify the right side of the form. Then it validate the form image with the xform whether the correct xform coordinates are in the form image. If it is not the system gives an error.

After validating the image the image was sent to the the image processor and the image processor finds the Input areas of the form then data fields within them and finally letter boxes according to the xforms coordinates. The data fields, input areas and letter boxes are explained in the above image.

The image processor was developed using the OpenCv libraries and it worked greatly in this project. You can find more abou the OpenCv libraries by referring here.

Then those letter boxes are sent to the Ocr class which is capable to recognize the character in that letter box. The initial Ocr class was developed using Fann neural network libraries and the accuracy of the recognitions was very bad because of the lack of training of the neural network. That’s the point that I contributed to this project. After discussing with the previous mentors and the developers of this project I have used Tesseract Ocr engine which was an Google product to replace the Fann neural network.

You can get information about the Tesserat from here.

I did some testing with the Tesseract for the hand written characters and check whether it is suitable for our module. The initial accuracy of the recognition with the Fann Library was below 30% and when it was tested with the Tesseract it was increased up to more than 80% and the worst case with was 80% when I tested for different data sets. I realized that it is a very powerful tool and selected it for the project. Then I have customized the Tesseract to make compatible with the expectations and changed the Form-processor in order to make it compatible with the Tesseract.

Finally the recognized letters are consolidated as strings and assign them to the appropriate labels such as Name, Address and Date of Birth. Then I have developed a GUI for the module which gives a graphical view for this process. Then the results are display at the GUIs result tab window. Then it allows the users to save the data in a file by submitting it.

So it was the greatest project that I was involved up to now because it gave me a lots of knowledge and it was an great opportunity made contacts with some interesting personalities involved in that project. I have an idea to work on this project more and extend this project to make the accuracy at least up to 97% and make the rest of the project as uploading the data in to the server directly kind of things. These some more ideas with me to work further on this project.

· Train the Tesseract more for hand written characters

· Increasing the accuracy by reading the labels. That means currently the OCR only reads the letters at the letter boxes and output the result. But in that case there are conflicts when recognizing the “0” and ”O” which the first one is zero and the second one is letter O. So when we read the label we can give a logic that the letter boxes under the Name label cannot have the zero number and also the letter boxes under the date of birth cannot have the letter O with it. So in that way using label reading we can further improve the accuracy.

· Making the data embedded with the Url given to the each form. That may make easy to transfer data from one place to another.

So please try on this project and try to make some innovative ideas to serve the people in the world in a situation that they are in need.

You can refer some more articles about this project from here.

How link another folder to your www web folder.

In last post I have describe the way to link your public_html folder to the www folder. To I will describe a simple way to link a folder to your /var/www to make easy your web developing stuff on linux.
1.To do that first you have to make a folder in your favorite place using mkdir command.
mkdir /home/abc/myFolder

2.Then go to your /var/www/ folder.
cd /var/www

3.Then link your folder to the www folder using ln command.
sudo ln -s /home/abc/myFolder

This will make an linking file in the www folder and you can see it in your www folder in the same name as you used for your linked folder

Now you can use your myFolder folder for run files in the web server by the address
/http://localhost/myFolder/

So use this method to make easy of your life in using servers for advance developments.

What is public HTML

If you are working with any kind of web server on linux operating system you might have suffered enough from the warning massages saying “This is write disabled file, please make the file write enable”, “Cannot move the file in to www folder” so on. So is it a headache for you. Here is an great and simple solution for your problem.

In linux most of the situations you have to copy your files such as html and php, in to the /var/www/ folder to run them on your favorite web servers such as apache web server. But this folder is protected and you need some permission to copy, remove or change files in the www folder. So when we are involve in a web development using large number of files moving here and there it is difficult to work in this kind of environment.

So now you can use another folder that is away from www folder which is not a protected one as your www folder. It will help you to make changes in the folder as you wish and you can get the same functions as in the /var/www folder. Following are the steps of making a public_html folder and make it enable for access by your user-name as http://localhost/~ABC.

1. make a folder in your favorite place in your folder structure. But make sure it is not a protected folder. As an example make a folder in the user folder by using the following command in command prompt:

sudo mkdir /home/ABC/public_html/

2. Go to the folder of your web server. As an example if you are using apache2 web server you can use following command

cd /etc/apache2/

3. Then link your public_html folder to the web server.

sudo cp -r mods-available/userdir.* mods-enabled/

4. Then restart your web server

sudo /etc/init.d/apache2 restart

Now you are ready to use your public_html folder as your web servers working folder.

You can access the folder by typing the following address on the address bar.

http://localhost/~ABC

The most important thing is if you are using your computer connected to a network and if others are access to your computer via network you can use this folder as the access folder to the out side, so the others can access to that folder with out permission enabled as well as you can keep your protected data away from this folder so others can't access those. Other wise if you enabled permission in the www folder the others may access to some other important data in your computer. The others can access to this folder by the address of your Internet address. As an example

http://www.your_net_address.com/~ABC

So please try this and feel the difference.