Back to blogging: What to expect

Standard

Hello Readers,

I started this blog as a way to share my experiments and experiences while learning web development and computer science in general. In the first 2 years (between Apr’08 and Aug’10) I wrote as many as 100 blog posts. Quite a frenzy. Ever since, I only managed to write 5-6 posts in the following 4 years, about nearly 45 drafts which may now never get published. Good thing is that, I am back to blogging, which means a lot to share.

Briefly, here is what (or what not) to expect in the future posts:

  1. PHP – In past, PHP has dominated the content on this blog. Mostly web demos, some quick hacks or some JAXL library examples. However, I am no longer working actively with PHP since ’10 and probably never saw it after ’12. Expect zero PHP.

  2. JAXL – No more PHP essentially means no more JAXL posts. In fact, I recently moved JAXL repository to it’s own Github organization where other collaborators can maintain, improve and work on it without requiring my active involvement. This organization also contains other repositories that I managed to open source from my startup Jaxl.

  3. XMPP – Unfortunately, I am no longer in touch with progress on XMPP specifications. Specs has evolved a lot, to an extent that some developers have reported mod_message_carbon no longer works as expected with new Ejabberd server version (also, Message Carbon extension XEP-0280 has itself been deprecated). However, XMPP will always be my preferred choice whenever I need entire suite of user-to-user, group messaging, presence, contacts management, Jingle / SIP integration and other features baked into XMPP XEPs. For my everyday messaging needs, new technologies like ZeroMQ, AMQP (RabbitMQ), MQTT or even Redis PubSub are more suitable.

  4. Java – After some journey I am now finally working full-time with Java. I still hate it but trying to adapt, learn and love it for at least what it’s worth for.

  5. Python – Thanks to my stint with Appurify, I had a chance to work full-time with Python. I even managed to work on some interesting open source Python projects. Even though now it’s no longer my primary language, Python is always fun specially when one is in a hurry of getting things done.

  6. Golang / Erlang – I met Golang a year back. I met Erlang while hacking Ejabberd, Riak etc for my startup Jaxl and immediately fell in love with it. Nowadays, I am in love with Golang. It’s simple and precise, has similar message passing semantics (buffered channels) as found in Erlang (mail boxes). I highly recommend digging into these languages and getting comfortable with message passing programming paradigm. They will change how you approach and think about your application structure. Expect lots of Golang and some Erlang.

  7. Docker – Who is not into docker these days? If that’s not the case with you, leave this post right now and head over to docker user guide. That’s how important I find this piece of beauty (technology). Expect a lot about docker in my future posts.

  8. Startups – A lot of startup fun has kept me busy since ’10, some experiences and learnings are worth sharing.

  9. Android – I have been working full-time with mobiles (both Android and iOS) since ’12. Not much of application making but a lot of hacking with Adb protocol and libimobiledevice.

  10. System designing – Luckily, I happened to experience a lot of end-to-end system and network designing. This domain is of great interest once you start to have fun with Racks, Subnet, Routes, Switches, Firewalls, DNS, Multi-cast and entire suite of technology under this umbrella.

Will end this post with some interesting images from the past.

Fsck'd iPhone screen

Swollen iPhone screen due to high device temperature

Rackframe

Setting up Racks

How to perform X-FACEBOOK-PLATFORM and Google Talk X-OAUTH2 XMPP authentication with PHP Jaxl library

Standard

Ever since Jaxl library first introduced support for X-FACEBOOK-PLATFORM XMPP authentication mechanism, it has changed significantly. Also, Google Talk now supports OAuth 2.0 Authorization, an XMPP extension to allow users to log in using OAuth 2.0 credentials.

Both these mechanisms are a big win for XMPP developers, since real-time conversation experience can now be provided to their application users without asking them for their passwords. In this blog post, I will demonstrate how to perform X-FACEBOOK-PLATFORM and X-OAUTH2 XMPP authentication mechanism using Jaxl v3.x PHP Library.

X-FACEBOOK-PLATFORM XMPP Authentication
Here is a quick guide on how to perform X-FACEBOOK-PLATFORM XMPP authentication using xfacebook_platform_client.php which comes bundled with Jaxl v3.x examples:

  • Visit Facebook Developer Apps page and register your application
  • Once registered, visit access token tool to get required parameters to perform X-FACEBOOK-PLATFORM authentication Facebook Access Token Tool
  • Click on the debug button next to User Token and make sure xmpp_login is one of the extended permissions (scope)
  • Enter downloaded Jaxl library folder and run from command line as follows:

    $ php examples/xfacebook_platform_client.php fb_user_id_or_username fb_app_key fb_access_token

You can now take the source code of xfacebook_platform_client.php and customize it for your application needs.

Google Talk X-OAUTH2 XMPP Authentication
Here is a quick guide on how to perform Google Talk X-OAUTH2 XMPP authentication using xoauth2_gtalk_client.php which comes bundled with Jaxl v3.x examples:

  • Visit Google OAuth Playground and input https://www.googleapis.com/auth/googletalk as the required scope. Press “Authorize API” and then “Allow Access” button on the redirected page
  • In step 2, simply press “Exchange authorize code for tokens” and copy the access token
  • Enter downloaded Jaxl library folder and run from command line as follows:

    $ php examples/xoauth2_gtalk_client.php username@gmail.com access_token

You can now take the source code of xoauth2_gtalk_client.php and customize it for your application needs.

Wasn’t that simple :)

JAXLXml – Strophe style XML Builder : Working with Jaxl – A Networking Library in PHP – Part 2

Standard

Prior to Jaxl v3.x, the most ugliest piece of code inside Jaxl library was handling of XML packets. If you are working with XMPP protocol which is all about sending and receiving XML packets, it can become a nightmare if you don’t have a proper XML manipulation library in your toolkit. For Jaxl v3.x, first thing I decided to write was JAXLXml class, which is a custom XML packet implementation with no external dependencies and is an extension over the ideas from Strophe.Builder class written by Jack Moffitt.

JAXLXml is generic enough to find a place inside any PHP application that requires easy and elegant XML packet creation. In this blog post, I will give an exhaustive overview of how to create XML packets using JAXLXml class.

JAXLXml Constructor
Depending upon the need, there are several different ways of initializing a JAXLXml object:

  • $xml_obj = new JAXLXml($name, $ns, $attrs, $text);
  • $xml_obj = new JAXLXml($name, $ns, $attrs);
  • $xml_obj = new JAXLXml($name, $ns, $text);
  • $xml_obj = new JAXLXml($name, $attrs, $text);
  • $xml_obj = new JAXLXml($name, $attrs);
  • $xml_obj = new JAXLXml($name, $ns);
  • $xml_obj = new JAXLXml($name);

where:

  • $name – the XML node name
  • $ns – the XML namespace
  • $attrs – Key-Value (KV) pair of XML attributes
  • $text – XML content

Here are a few examples for each constructor style shown above:

$ ./jaxlctl shell
jaxl 1> $name = 'node-name';
jaxl 2> $ns = 'my:xml:ns';
jaxl 3> $attrs = array('k1'=>'v1', 'k2'=>'v2');
jaxl 4> $text = 'this is an example';
jaxl 5>
jaxl 5> $xml_obj = new JAXLXml($name, $ns, $attrs, $text);
jaxl 6> echo $xml_obj->to_string();
<node-name xmlns="my:xml:ns" k1="v1" k2="v2">this is an example</node-name>
jaxl 7> 
jaxl 7> $xml_obj = new JAXLXml($name, $ns, $attrs);
jaxl 8> echo $xml_obj->to_string();
<node-name xmlns="my:xml:ns" k1="v1" k2="v2"></node-name>
jaxl 9> 
jaxl 9> $xml_obj = new JAXLXml($name, $ns, $text); 
jaxl 10> echo $xml_obj->to_string();
<node-name xmlns="my:xml:ns">this is an example</node-name>
jaxl 11> 
jaxl 11> $xml_obj = new JAXLXml($name, $attrs, $text);
jaxl 12> echo $xml_obj->to_string();
<node-name k1="v1" k2="v2">this is an example</node-name>
jaxl 13>
jaxl 13> $xml_obj = new JAXLXml($name, $attrs);
jaxl 14> echo $xml_obj->to_string();
<node-name k1="v1" k2="v2"></node-name>
jaxl 15>
jaxl 15> $xml_obj = new JAXLXml($name, $ns);
jaxl 16> echo $xml_obj->to_string();
<node-name xmlns="my:xml:ns"></node-name>
jaxl 17> 
jaxl 17> $xml_obj = new JAXLXml($name);
jaxl 18> echo $xml_obj->to_string();
<node-name></node-name>
jaxl 19>

JAXLXml will sanitize attributes and text values as shown below:

$ ./jaxlctl shell
jaxl 1> $xml_obj = new JAXLXml('msg', array(), '<a href=""></a>');
jaxl 2> echo $xml_obj->to_string();
<msg>&lt;a href=&quot;&quot;&gt;&lt;/a&gt;</msg>
jaxl 3>
jaxl 3> $xml_obj = new JAXLXml('msg', array('a'=>'some < tag data >'), '<a href="javascript:void(0);"></a>');
jaxl 4> echo $xml_obj->to_string();
<msg a="some &lt; tag data &gt;">&lt;a href=&quot;javascript:void(0);&quot;&gt;&lt;/a&gt;</msg>
jaxl 5>
jaxl 5> quit
$

Manipulating Attributes, Child Nodes and Content
Below is an exhaustive list of methods available over initialized JAXLXml object $xml_obj for manipulating attributes, child nodes and content:

  • c($name, $ns=null, $attrs=array(), $text=null) : Append a child node at current rover and update the rover to point at newly added child node. Rover is nothing but a pointer indicating the level in the XML tree where this and other methods will perform. When an JAXLXml instance is initialized, rover points to the top level node.
  • cnode($node) : Append a child node given by $node (a JAXLXml object) at current rover and update the rover to point at newly added child node.
  • t($text, $append=FALSE) : Update text of the node pointed by current rover
  • top() : Move rover back to the top in the XML tree
  • up() : Move rover one step up the XML tree
  • attrs($attrs) : Merge new attributes specified as KV pair $attrs with existing attributes at the current rover.
  • match_attrs($attrs) : Accepts a KV pair of attributes $attrs, return bool if all keys exist and have same value as specified in the passed KV pair.
  • exists($name, $ns=null, $attrs=array()) : Checks if a child with $name exist. If found, return matching child as JAXLXml object otherwise false. If multiple children exist with same name, this function will return on first matching child
  • update($name, $ns=null, $attrs=array(), $text=null) : Update $ns, $attrs and $text (all at once) of an existing child node $name
  • to_string($parent_ns=null) : Return string representation of JAXLXml object

Method Chaining
The best thing one will find while working with JAXLXml class is that all the above methods are chain-able i.e. Any complex XML structure can be built with a single line of code.

Here is an example building a fairly nested XML structure in a single line of code:

$ ./jaxlctl shell
jaxl 1> $xml_obj = new JAXLXml('message', array('to'=>'1@a.z', 'from'=>'2@b.c'));
jaxl 2> $xml_obj->c('body')->attrs(array('xml:lang'=>'en'))->t('Hello World!')->up()
....... ->c('thread')->t('id-1234')->up()
....... ->c('nested-stuff')
....... ->c('nest')->t('nest1')->up()
....... ->c('nest')->t('nest2')->up()
....... ->c('nest')->t('nest3')->up()->up()
....... ->c('c')->attrs(array('hash'=>'84jsdmnskd'));
jaxl 3> echo $xml_obj->to_string();
'<message to="1@a.z" from="2@b.c"><body xml:lang="en">hello</body><thread>1234</thread><nested><nest>nest1</nest><nest>nest2</nest><nest>nest3</nest></nested><c hash="84jsdmnskd"></c></message>'
jaxl 4>
jaxl 4> quit
$

Working with Jaxl – A Networking Library in PHP – Part 1 – An Introduction, Philosophy and History

Standard

Development of Jaxl library started way back in December’07 while I was working on a self-initiated project called Gtalkbots. The project is now dead, if you are interested in knowing more about it go through Gtalkbots BlogSpot. Jaxl v1.x was first released in Jan’09 and about a year later in Aug’10 Jaxl v2.x was released. First two versions were released as JAbber XMPP Library for writing clients and external server components.

While working on my startup Jaxl – A Platform As A Service (PAAS) for developing real-time applications, I started experiencing v2.x limitations when my external server side components were unable to process XMPP packets at the speed they were sent by ejabberd server. I started restructuring and refactoring the library which gave birth to Jaxl v3.x. Since v3.x was initially being used for developing the entire infrastructure, it shaped up as a networking library in PHP with stable support for XMPP protocol. However, later I had to rewrite several infrastructure components in Erlang Programming Language due to several issues that PHP as a language couldn’t solve (after all PHP wasn’t made for such tasks). Finally in April’12, Jaxl v3.x was open sourced.

Jaxl v3.x is an asynchronous, non-blocking, event based networking library in PHP for writing custom TCP/IP client and server implementations. From previous versions, Jaxl library inherits a full blown stable support for XMPP protocol stack. In v3.0, support for HTTP protocol stack was also introduced. At the heart of every protocol stack sits a Core stack. It contains all the building blocks for everything that we aim to do with Jaxl library. Both XMPP and HTTP protocol stacks are written on top of the Core stack. Infact the source code of these protocol implementations knows nothing about the standard (inbuilt) PHP socket and stream methods.

Philosophy
Jaxl is designed to work asynchronously in a non-blocking fashion and provides an event based callback API. Now what does all that mean?

By non-blocking and asynchronous it means, when a library function like:
$jaxl->send($stanza); is called, it will return immediately i.e. this function call will NOT block any further execution of your application script until $stanza has actually been sent over the connected TCP socket. Infact, when this function is called, passed $stanza object is put into an output buffer queue, which will be flushed as and when underlying TCP socket is available for writes. Similarly, most of the available methods (wherever required and possible) inside Jaxl library are non-blocking and asynchronous in nature.

By event based callback API it means, application code will need to register/add callbacks over necessary events as they occur inside Jaxl instance lifecycle. A list of available event callbacks with some explanation can be found here. For example, most of the XMPP applications will usually register a callback over on_auth_success event. As and when this event occurs inside Jaxl instance lifecycle, registered function will be callback’d with necessary parameters (if any).

Related Links

  • Read library documentation
  • Download the latest and greatest source from GitHub.
  • Have any Question? Want to discuss? Need Help? Use Google Group/Forum.
  • Found something missing or a bug in the source code? Kindly report an issue.
  • Fixed a bug? Want to submit a patch? Want to improve documentation? Checkout source code and contribute to the library

XMPP Application Examples

HTTP Application Examples

Stay Tuned
In coming weeks, under this series of blog posts titled “Working with Jaxl – A Networking Library in PHP”, I will cover following major topics with sample code:

  • Explanation of each Core stack class and how to use them
  • Design of each XMPP and HTTP stack class
  • XMPP over HTTP
  • XMPP File Transfer and Multimedia Sessions
  • Understanding and Using External Jabber Components
  • Asynchronous Job/Task Queues
  • Developing Concurrent and Parallel Systems

If you have any specific topic that you would like me to be cover, kindly let me know via your comments here.

Announcing Jaxl v3.x – asynchronous, non-blocking I/O, event based PHP client/server library

Standard

Jaxl v3.x is a successor of v2.x (and is NOT backward compatible), carrying a lot of code from v2.x while throwing away the ugly parts. A lot of components have been re-written keeping in mind the feedback from the developer community over the last 4 years. Also Jaxl shares a few philosophies from my experience with erlang and python languages.

Jaxl is an asynchronous, non-blocking I/O, event based PHP library for writing custom TCP/IP client and server implementations. From it’s previous versions, library inherits a full blown stable support for XMPP protocol stack. In v3.0, support for HTTP protocol stack was also added.

At the heart of every protocol stack sits a Core stack. It contains all the building blocks for everything that we aim to do with Jaxl library. Both XMPP and HTTP protocol stacks are written on top of the Core stack. Infact the source code of protocol implementations knows nothing about the standard (inbuilt) PHP socket and stream methods.

Source code on GitHub

Examples

Documentation

Group and Mailing List

Create a bug/issue

Read why v3.x was written and what traffic it has served in the past.

How to Write a Spelling Corrector in Erlang (ESpell)

Standard

Erlang is a beautiful programming language from Ericsson which i first came across while cutomizing authentication flow of ejabberd about 2 years back. Ever since then I have been using erlang for all my application backend needs including custom http server, custom bosh conn. manager, xmpp components and clients, … Recently i have even started churning my application html pages via erlang using erlydtl (an Erlang implementation of the Django Template Language).

Years ago, i gave a successful shot at implementing Peter Norvig’s Spell Corrector in PHP. Last weekend i attempted the same “Spell Corrector” algorithm in about 45 lines of Erlang code.

ESpell:

Complete code file with comments and explaination can be found here:
https://github.com/abhinavsingh/espell

-module(espell).
-define(alphabet, "abcdefghijklmnopqrstuvwxyz").
-export([start/0, correct/1]).

%%
%% API Functions
%%

%% @doc start spell checker
start() -> train(words()).

%% @doc returns most probable correct candidate with score
correct(Word) ->
	lists:foldl(
		fun(Candidate, {Correction, Score}) -> 
			case ets:lookup(?MODULE, list_to_binary(Candidate)) of
				[{_, Counter}] when Counter > Score -> {Candidate, Counter};
				_ -> {Correction, Score}
			end
		end, {Word, 0}, get_candidates(Word)).

%%
%% Local Functions
%%

words() ->
	{ok, Bin} = file:read_file("../priv/big.txt"),
	{ok, Words} = regexp:split(binary_to_list(Bin), "[^a-zA-Z]"),
	lists:map(fun(X) -> string:to_lower(X) end, Words).

train(Features) ->
	io:fwrite("training initial word list...~n"),
	ets:new(?MODULE, [set, named_table]),
	lists:foreach(fun(X) ->
		case ets:insert_new(?MODULE, {list_to_binary(X), 1}) of
			false -> ets:update_counter(?MODULE, list_to_binary(X), 1);
			true -> true
		end
	end, Features),
	io:fwrite("training complete...~n"),
	ok.

edits1(Word) ->
	Splits = lists:foldl(fun(I, Acc) -> Acc ++ [{string:substr(Word, 1, I), string:substr(Word, I+1, string:len(Word)-I)}] end, [{"", Word}], lists:seq(1, string:len(Word))),
	Deletes = [A ++ string:substr(B, 2) || {A,B} <- Splits, B =/= []],
	Transposes = [A ++ string:substr(B, 2, 1) ++ string:substr(B, 1, 1) ++ string:substr(B, 3) || {A,B} <- Splits, string:len(B) > 1],
	Replaces = [A ++ binary_to_list(<>) ++ string:substr(B, 2) || {A,B} <- Splits, B =/= [], C <- ?alphabet],
	Inserts = [A ++ binary_to_list(<>) ++ B || {A,B} <- Splits, C <- ?alphabet],
	lists:usort(Deletes ++ Transposes ++ Replaces ++ Inserts).

%%edits2(Word) -> lists:usort([E2 || E1 <- edits1(Word), E2 <- edits1(E1)]).
known_edits2(Word) -> lists:usort([E2 || E1 <- edits1(Word), E2 <- edits1(E1), ets:member(?MODULE, list_to_binary(E2))]).
known(Words) -> lists:usort([Word || Word <- Words, ets:member(?MODULE, list_to_binary(Word))]).

get_candidates(Word) ->
	C1 = known([Word]),
	if 
		length(C1) > 0 -> C1;
		true -> C2 = known(edits1(Word)),
			if
				length(C2) > 0 -> C2;
				true ->	C3 = known_edits2(Word),
					if length(C3) > 0 -> C3; 
					true -> [Word] end
			end
	end.

Try It Out:
espell provides 2 simple function for all it’s working:

  • start() : start espell which initiates reading initial data and training phase
  • correct(Word) : this accepts 1 parameter, which is the word you want to correct. It returns a 2-tuple, where 1st element is the correct word and 2nd element is a score (which right now simply means number of times correct word was seen in training data set)
$ cd espell
$ erlc -o ebin/ src/espell.erl
$ erl -pa ebin/
Erlang R14B03 (erts-5.8.4) [source] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> espell:start().
training initial word list...
training complete...
ok
2> espell:correct("speling").
{"spelling",4}
5> espell:correct("somthing").
{"something",683}

This code makes extensive use of list comprehensions in erlang, which is hugely responsible for cutting down espell code to just 45 lines of erlang.

JAXL library – List of available hooks for various XMPP events

Standard

Jaxl 2.x provides an event mechanism using which developers can register callbacks for various xmpp events inside their application code. This blog post will demonstrate how to register callbacks for required xmpp events and go through a list of all available hooks. Finally, we will discuss parameters that are passed to called back methods by Jaxl core.

Registering callback on XMPP events
Applications can register callback for various XMPP events. Jaxl core will then callback application methods (with 2 parameters) every time associated XMPP event occurs. Shown below are some sample examples for registering callbacks.

When application callback’d method is a function:

function postAuth($payload, $jaxl) {

}
$jaxl->addPlugin('jaxl_post_auth', 'postAuth');

When application callback’d method is a public static method of a class:

class MyXMPPApp {
    public static function postAuth($payload, $jaxl) {

    }
}
$jaxl->addPlugin('jaxl_post_auth', array('MyXMPPApp', 'postAuth'));

When application callback’d method is a public method inside a class:

class MyXMPPApp {
    function postAuth($payload, $jaxl) {

    }
}
$MyXMPPApp = new MyXMPPApp();
$jaxl->addPlugin('jaxl_post_auth', array($MyXMPPApp, 'postAuth'));

In all the above examples jaxl_post_auth is one of the available hook for registering callbacks.

List of available hooks
Below is a complete list of available hooks in order of their occurrence within a Jaxl instance life cycle:

Hooks for events related to instance connection and authentication steps in various modes:

  • jaxl_post_connect
  • jaxl_get_auth_mech
  • jaxl_get_facebook_key
  • jaxl_post_auth_failure
  • jaxl_post_auth
  • jaxl_post_handshake
  • jaxl_pre_shutdown
  • jaxl_post_disconnect
  • jaxl_get_empty_body

Hooks for events related to XMPP stream and stanza’s:

  • jaxl_get_stream_error
  • jaxl_get_presence
  • jaxl_get_message
  • jaxl_get_iq_get
  • jaxl_get_iq_set
  • jaxl_get_iq_error
  • jaxl_send_message
  • jaxl_send_presence

Hooks for events related to reading/writing of XMPP packets and internal packet routing:

  • jaxl_get_xml
  • jaxl_send_xml
  • jaxl_send_body
  • jaxl_pre_handler
  • jaxl_post_handler

TO-DO: Update when every hook is called inside your application life cycle and list of parameters passed for each callback. As of now you can var_dump($payload); inside your callback method.