JS-ctypes

With the unavailability of XulRunner in Ubuntu Oneiric and the possible removal of it from other distro, we decided to switch to js-ctypes components. The main reason is that our code won't be tied to a XR version (which are released every 6 weeks) as components are. It also has some side-advantages

JS-ctypes
ctypes is a Foreign Function Interface which is available in several languages. It allows one to call C functions (of a library) from Javascript or Python or other language supporting it. It doesn't require specific C code. It just calls arbitrary functions.

Porting
Porting Kiwix components to js-ctypes requires a lot of work so we decided to incorporate other changes. Porting a component consist of the following:


 * 1) Rewrite the C++ component into a C++ library with no mozilla dependency.
 * 2) Write a C++ program testing the C++ API.
 * 3) Write a C Wrapper around the C++ API.
 * 4) Write a C program testing the C Wrapper/API.
 * 5) Write a JS module interfacing with js-ctypes and the C wrapper.
 * 6) Fix existing JS code (gui.js, etc) to use the JS module.

Note: Because of the number of layers, it is important to respect the convention on naming which differs from layer to layer. Examples:


 * component: zimAccessor
 * component path: src/components/zimAccessor
 * library name: zimAccessor
 * library path: src/zimAccessor
 * C++ library files: zimAccessor.cpp zimAccessor.h
 * C Wrapper files: ZimAccessorWrapper.cpp ZimAccessorWrapper.h
 * Tester path: src/zimTester/
 * C++ tester file: zimTester.cpp
 * C Wrapper tester file: zimCTester.c

Rewrite the C++ component into a C++ library
The main rule here is to remove Mozilla dependency. It is very important to remove it completely otherwise it's a waste of time. Advantages of removing MOZ dep:
 * We build a shared/static lib which is not a component.
 * We don't need the mozilla stack to build it (ease Windows, and other system setup like arm)
 * We don't need an every-release recompile of our code
 * We don't even need to release anything to have it work with newer XR.
 * We can use it to build a Kiwix UI with webkit (for example on Android). We'll code all component code and just need a new UI.
 * We can use the same code for kiwix-serve
 * We could use it to create a Kiwix server instead of a kiwix-http-server (allow one to administer the server library using a Web UI)
 * We can then split kiwix code with components code and have a kiwix-libs package.

Writing this library is quite easy. It's mostly a copy-paste of the component code then cleaning.

Note: The goal is to replace the component but until all components are ported, we'll have a duplication of the code in the tree. We need to port-back every addition to the comonent to the new library until it replaces the component completely.


 * C++ Component code example
 * C++ library example
 * C++ library header example

Rules to embrace/respect:
 * 1) nsAString is replaced with string
 * 2) nsACString is replaced with char *
 * 3) nsIURI is replaced with string and code adapted consequently.
 * 4) No more retVal nor NS_OK. Methods uses return type (bool mostly)
 * 5) Keep case on method names.
 * 6) don't add features while you port to keep it understandable and revert-able.

The component uses no header file (it is generated from IDL at compile time and is really verbose) so you also need to write it manually. Refer to example.

Write a C++ program testing the C++ API
Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.

The idea here is just to make sure that all methods are available and working as expected.


 * C++ library header file
 * C++ tester source

I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.

Write a C Wrapper around the C++ API
js-ctypes (and ctypes) only works with C code and not with C++. Because our code base is in C++ (for good reasons), we need to add a C layer to interface with the C++ library. This is done by creating a C++ library (exported as C) containing only functions which would each call the C++ API. We'll add a couple functions to create/destroy the instance of the C++ class and each function will accept a pointer to the instance as parameter.


 * C Wrapper example
 * C wrapper header example

Rules:
 * 1) Create a void pointer type to match the class pointer.
 * 2) Create functions to crete/destroy the class instance.
 * 3) All functions are prefixed with library name (and keep case)
 * 4) All functions have previously created type as first argument
 * 5) expect C types as input (char *)
 * 6) convert char * to string inside your function
 * 7) convert outputed string to char *
 * 8) return the actual data. Most of the time, we only have one return variable. Use that as return type for the wrapper. It's easier and cleaner from the JS perspective.
 * 9) if you need to return multiple values, use a struct example

Header example ""

Source example ""

Write a C program testing the C Wrapper/API
Instead of writing a program to test the features of the API, we should write unit tests but I have no prior experience with unit testing with C/C++ so I choose the loose way. Feel free to improve that.

The idea here is just to make sure that all methods are available and working as expected.

Note: This testing is more different than the previous C++ one as the API has changed while switching C in order to simplify it (make it look more like JS).


 * C++ library header file
 * C++ tester source

I'm not very proud of it, a lot of stuff are hard coded and while it gets a ZIM as argument, it expects the swahili zim to test everything.

Write a JS module interfacing with js-ctypes and the C wrapper
Creating a JS module is not required but it's cleaner and convenient: ctypes calls requires the pointer as first argument and the return value of the calls is specific to ctypes. Our module will use js-ctypes abstract the C librar so that the whole js-ctypes is hidden.


 * JS module example

Documentation:
 * js-ctypes intro
 * MDN page
 * js-ctypes reference
 * API on the wiki
 * a tutotial
 * example with struct

Rules:
 * 1) Open library and declare functions in register method.
 * 2) you need to declare every function of the C API.
 * 3) char * returned by API are available via .readString
 * 4) int and bool returned are available via contents (raw).
 * 5) declare method gets name, abi (we use default so that it's multiplatform), return type (you need to choose this), then parameter (first one is our pointer to the class).
 * 6) retrieving values from a struct example
 * 7) Change case on method names to differentiate with original API.
 * 8) abstract the API by providing direct value output to your methods.
 * 9) export your jsm as libXXX

Beware that wrong declaration can cause your jsm not to load (error message is misleading: File Not Found). Comment your declarations out and add them back one by one to find the problem. I chose arbitrary types which worked for my simple tests and some might be wrong. Think for your self and adjust.

I chose:
 * int16_t (16b integer) for boolean returned
 * ctypes.int32_t.ptr (pointer to 32b integer) for instance pointer
 * ctypes.char.ptr pointer to char for all strings.

Fix existing JS code to use the JS module
This part hasn't been done yet. Only local tests. Due to the way the current code mixes the various components it is very difficult to change that code until all components are converted. Once all are converted, we can rewrite part of that code and make it a lot simpler & cleaner than it currently is.

Tips

 * To test js-ctypes and your jsm, you need to put your statically built CWrapper .so lib in xulrunner folder. We'll search later if there's a way to place it elsewhere and have it discovered. Anyway, js-ctypes.open can take a full path as parameter.
 * jsm are compiled and cache at registration. You need to remove cache (or whole profile) after changes.

unicode
Absolutely zero work as been done regarding unicode and string handlings. It just work on Linux but I'm 100% sure it will fail badly on windows or with very exotic paths. Because of the pain it is to manage string properly, it has been agreed that we would wait until the components are ported then we'll find a way to fix the strings.

I believe we might want then to duplicate code from nsString (mozilla) or libICU.

makefiles
library, wrapper and tester makefiles are hardcoded right now. Well need to move them to autotools.

Replace XR with FF
This is the target. Once we have those compo working, we need to investigate using FF as a XR engine. This is theoretically possible but I haven't found example and initial tests showed difficulties with main.xul.

Compatibility with XR 1.9.2
js-ctypes have been introduced with XR 1.9.2 but the ABI was incomplete or different. Once we have everything working we'll need to dig that and check if we need it.

Progression

 * zimAccessor: complete
 * contentManager: jsm not finished
 * xapianAccessor: not started
 * zimXapianIndexer: not started

In the JSM we open libXX.so. We'll need to replace that with appropriate extension for each platform. There's an xpcom component which guess the extension based on platform but can't remember the name.