Customizing Redis pubsub for message persistence – Part 2

Redis Logo

In the last post we saw how Redis can easily be modified to persist the last published message on PubSub channels. Without subscribing to the PubSub channel we were able to get the last published message from Redis db. In this post, I will take that idea one step ahead and add native capabilities within Redis to persist all the unprocessed messages published on PubSub channel in channel specific lists. We’ll also preserve our capability to send the last published message to clients upon subscription.

But why are we doing this?

Popular open source application that provide support for Redis are based out of it’s list class of API. For example, let’s look at Celery which is a distributed task queue written in Python. Start redis-cli MONITOR on a terminal and then start celery in another window as follows:

$ pip install celery
$ celery worker --queues=testing --broker=redis:// --without-gossip --without-mingle --without-heartbeat

You’ll find celery polling Redis periodically as indicated by log lines like "BRPOP" "testing" "testing\x06\x163" "testing\x06\x166" "testing\x06\x169" "1". Celery uses the Redis PubSub mechanism (that we disabled in the above command) only for internal features. Sentry, a popular exception logging and aggregation library, internally depends upon Celery. There is an open pull request that claims to add Redis pubsub based support to Celery. In the world of Ruby, background processing frameworks like Requeue and Sidekiq depend upon Redis list class of API’s.

However, with no native support for persistence of PubSub messages in Redis, it’s not difficult to understand why adopting to Redis PubSub can be tricky for some. Currently, Redis simply drops the message if no subscribers are found. Hence, question really is whether your application is tolerant to loss of published messages (for example, dropped messages while you were upgrading your application) ?

To solve persistence problem with Redis pubsub, the usual approach is to start multiple application instances. Some instances can continue to serve while others get deployed. However still, your active instances might be experiencing a network partition and unable to receive published messages. After all, primary goal is to guarantee processing of every message received by Redis irrespective of whether we are using list or pubsub based backend. A native support to solve persistence problems with Redis PubSub is clearly desirable.

Persisting dropped Redis PubSub messages in a list

In the last post we added a single line of code to persist the last published message on channels in a separate Redis key. We’ll update implementation to push every received message at the end of a channel specific list. Replace `setKey(c->db, c->argv[1], c->argv[2]);` line that we added the last time with following code:

// Persist messages in list only if no receivers were found
if (receivers == 0) {
    int j, pushed = 0, where = REDIS_TAIL;

    // Fetch list key from the database
    robj *lobj = lookupKeyWrite(c->db,c->argv[1]);

    // For every published message on the channel
    for (j = 2; j < c->argc; j++) {
        c->argv[j] = tryObjectEncoding(c->argv[j]);

        // Ensure we have our quicklist initialized
        if (!lobj) {
            lobj = createQuicklistObject();
            quicklistSetOptions(lobj->ptr, server.list_max_ziplist_size,
                                server.list_compress_depth);
            dbAdd(c->db,c->argv[1],lobj);
        }

        // Push message at the tail of the list
        listTypePush(lobj,c->argv[j],where);
        pushed++;
    }

    // Signal key watchers and internal event subscribers
    if (pushed) {
        char *event = (where == REDIS_HEAD) ? "lpush" : "rpush";
        signalModifiedKey(c->db,c->argv[1]);
        notifyKeyspaceEvent(REDIS_NOTIFY_LIST,event,c->argv[1],c->db->id);
    }
    server.dirty += pushed;
}

I have added some comments in the code for clarity. This code is merely is a rip off of src/t_list.c:pushGenericCommand function. We don’t want to send replies to the client that are usually sent after an RPUSH command. Frankly, we can further refactor pushGenericCommand function to turn the above code into a single function call.

make test and start ./src/redis-server. Using ./src/redis-cli try:

$ ./src/redis-cli 
127.0.0.1:6379> publish persistent-channel this
(integer) 0
127.0.0.1:6379> publish persistent-channel is
(integer) 0
127.0.0.1:6379> publish persistent-channel gonna
(integer) 0
127.0.0.1:6379> publish persistent-channel be
(integer) 0
127.0.0.1:6379> publish persistent-channel awesome
(integer) 0
127.0.0.1:6379> lrange persistent-channel 0 -1
1) "this"
2) "is"
3) "gonna"
4) "be"
5) "awesome"
127.0.0.1:6379>

Voila! Since we published a few messages with no active subscriber, they all got persisted in a list named after the channel itself. Now incoming subscribers can process pending messages by fetching them from the list which otherwise would have been dropped.

Delivering unprocessed messages to subscribers upon subscription

Instead of depending upon clients to poll for channel list length, server can deliver unprocessed messages to subscribers upon successful subscription. Since this can get overwhelming for subscribers if there are several pending messages waiting in the list, we may not want to do this at all and leave it up to the clients. Let’s preserve our feature from the last post i.e. to deliver the last published message to clients upon subscription.

Here we don’t want to remove the last published message from our persistent list. We simply wish to send it to the incoming subscribers. For example, imagine cases like user status, location and mood being published on separated channels. Here is a method that will give back the last element from the list without removing it (equivalent to LRANGE key -1 -1):

robj *llast(redisClient *c, robj *key) {
    listTypeEntry entry;
    robj *o, *value = NULL;
    long llen;

    // Fetch list object from db
    o = lookupKeyRead(c->db, key);

    // Ensure key contains a list
    if(o != NULL && o->encoding == REDIS_ENCODING_QUICKLIST) {
        // Get list iterator for "lrange key -1 -1" use case
        llen = listTypeLength(o);
        listTypeIterator *iter = listTypeInitIterator(o, llen-1, REDIS_TAIL);

        // Fetch last item, prepare value
        listTypeNext(iter, &entry);
        quicklistEntry *qe = &entry.entry;
        if (qe->value) {
            value = createStringObject((char *)qe->value,
                                       qe->sz);
        } else {
            value = createStringObjectFromLongLong(qe->longval);
        }
        listTypeReleaseIterator(iter);
    }
    return value;
}

We now need to replace our modifications within subscription handling methods from the last post. Note, callee must call decrRefCount on the returned robj to free up created string object after delivery. Kindly check this github commit for all the changes since the last post. You can also checkout my Redis fork and directly play with the modified code.

Conclusion

We saw how easy it is to modify Redis for fun and profit. By adding native persistence capabilities, we offload our task of ensuring processing of every message received by Redis cluster. After all, none of the magical client side logics will ever be as reliable as native support. By the way, Redis 3.0.0 was released this week with native support for clustering, checkout while it’s hot.

Leave your comments and feedback.

Customizing Redis pubsub for message persistence

Redis Logo

Redis comes packed with a simple yet powerful PubSub API.  It provides low latency and scales well.  A message published on a channel is received by subscriber(s) at the other end.  However, if no active subscriber is found the message is simply lost.  This drawback puts Redis out of the probables list for several use cases where message persistence of unprocessed published messages is desired.  It’s also probably a reason why several open source projects that support Redis as a broker are based upon it’s list push / pop API.  In this post I will demonstrate how to modify Redis PubSub API to support message persistence, opening possibilities for several interesting use cases.

Last Published Message

Ability to fetch the last published message on a particular channel without subscribing to the channel opens doors for several interesting use cases.  src/pubsub.c:publishCommand is where Redis handles publish command.  Let’s add a line of code to persist the most recently published message on a channel:

void publishCommand(redisClient *c) {
    ....

    /* Persist last published message in channel specific key */
    setKey(c->db, c->argv[1], c->argv[2]);

    addReplyLongLong(....);
}

Above, we added a call to src/db.c:setKey function that sets the value of key c->argv[1] (channel name) to c->argv[2] (published message).

Run make from the project root directory and start ./src/redis-server. Now we can do something like:

127.0.0.1:6379> publish channel1 c1m1
(integer) 0
127.0.0.1:6379> get channel1
"c1m1"
127.0.0.1:6379> publish channel1 c1m2
(integer) 0
127.0.0.1:6379> get channel1
"c1m2"

Voila. We published a message with no subscriber. However, an incoming user can still be served with the last published message on the channel by fetching the value of key channel1 without explicitly subscribing to the channel.

Let’s take this idea one step ahead. XMPP Publish-Subscribe (XEP-0060) defines a specification for receiving the last published item. It says,

When a subscription request is successfully processed, the service MAY send the last published item to the new subscriber.

Let’s add this idea to Redis PubSub mechanism. src/pubsub.c:subscribeCommand function is where Redis processes channel subscription requests. Add the following lines of code at the end of this function.

void subscribeCommand(redisClient *c) {
    ....

    /* Send last received message on the subscribed channel(s) */
    robj *o;
    for (j = 1; j < c->argc; j++) {
    	o = lookupKeyRead(c->db, c->argv[j]);
    	if(o != NULL) {
		addReply(c,shared.mbulkhdr[3]);
		addReply(c,shared.messagebulk);
		addReplyBulk(c,c->argv[j]);
		addReplyBulk(c,o);
    	}
    }
}

Here, post subscription, we fetch and send the last published message for all channels that the client just subscribed to. make and restart ./src/redis-server. Now on a new ./src/redis-cli terminal subscribe to channel1:

127.0.0.1:6379> subscribe channel1
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "channel1"
3) (integer) 1
1) "message"
2) "channel1"
3) "c1m2"

Voila! Now Redis server will send the last published message upon subscription. But what about PSUBSCRIBE use case?

src/pubsub.c:psubscribeCommand handles pattern based channel subscription logic. Add following lines of code at the end of this function:

void psubscribeCommand(redisClient *c) {
    ....

    /* Send last received message on the channel(s) matching subscribed patterns */
    for (j = 1; j < c->argc; j++) {
    	robj *pat = c->argv[j];
    	dictIterator *di = dictGetIterator(server.pubsub_channels);
    	dictEntry *de;
    	while((de = dictNext(di)) != NULL) {
		robj *cobj = dictGetKey(de);
		sds channel = cobj->ptr;
		if (stringmatchlen((char*)pat->ptr,
				sdslen(pat->ptr),
				(char*)channel,
				sdslen(channel), 0)) {
			robj *o = lookupKeyRead(c->db, cobj);
			if(o != NULL) {
	                addReply(c,shared.mbulkhdr[4]);
	                addReply(c,shared.pmessagebulk);
	                addReplyBulk(c,pat);
	                addReplyBulk(c,cobj);
	                addReplyBulk(c,o);
			}
		}
	}
	dictReleaseIterator(di);
    }
}

Above, for every subscribed pattern, we iterate over active server.pubsub_channels and check if the active channel matches the subscription pattern. On match we fetch and send the last published message on the channel to the client.

With previous redis-cli subscribe terminal running, open a new terminal and try:

127.0.0.1:6379> psubscribe channel*
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "channel*"
3) (integer) 1
1) "pmessage"
2) "channel*"
3) "channel1"
4) "c1m2"

Next

You can checkout my Redis fork and commits under pubsub-persistence branch. Enhancements described above can also be found on this github commit.

Currently it is unclear what Antirez (Sanfilippo Salvatore) plans to do further with PubSub in Redis. It stands on a solid base and recent efforts are rightly put behind Redis cluster. However, I see some interesting enhancements that can be made to Redis PubSub mainline. In the next post I will take the current idea one step ahead and add persistence support for all or only unprocessed published messages in a Redis list (possibly with a cap or expiration on persisted messages).

Back to blogging: What to expect

Hello Readers,

I started this blog as a way to share my experiments and experiences while learning web development and computer science in general. In the first 2 years (between Apr’08 and Aug’10) I wrote as many as 100 blog posts. Quite a frenzy. Ever since, I only managed to write 5-6 posts in the following 4 years, about nearly 45 drafts which may now never get published. Good thing is that, I am back to blogging, which means a lot to share.

Briefly, here is what (or what not) to expect in the future posts:

  1. PHP – In past, PHP has dominated the content on this blog. Mostly web demos, some quick hacks or some JAXL library examples. However, I am no longer working actively with PHP since ’10 and probably never saw it after ’12. Expect zero PHP.

  2. JAXL – No more PHP essentially means no more JAXL posts. In fact, I recently moved JAXL repository to it’s own Github organization where other collaborators can maintain, improve and work on it without requiring my active involvement. This organization also contains other repositories that I managed to open source from my startup Jaxl.

  3. XMPP – Unfortunately, I am no longer in touch with progress on XMPP specifications. Specs has evolved a lot, to an extent that some developers have reported mod_message_carbon no longer works as expected with new Ejabberd server version (also, Message Carbon extension XEP-0280 has itself been deprecated). However, XMPP will always be my preferred choice whenever I need entire suite of user-to-user, group messaging, presence, contacts management, Jingle / SIP integration and other features baked into XMPP XEPs. For my everyday messaging needs, new technologies like ZeroMQ, AMQP (RabbitMQ), MQTT or even Redis PubSub are more suitable.

  4. Java – After some journey I am now finally working full-time with Java. I still hate it but trying to adapt, learn and love it for at least what it’s worth for.

  5. Python – Thanks to my stint with Appurify, I had a chance to work full-time with Python. I even managed to work on some interesting open source Python projects. Even though now it’s no longer my primary language, Python is always fun specially when one is in a hurry of getting things done.

  6. Golang / Erlang – I met Golang a year back. I met Erlang while hacking Ejabberd, Riak etc for my startup Jaxl and immediately fell in love with it. Nowadays, I am in love with Golang. It’s simple and precise, has similar message passing semantics (buffered channels) as found in Erlang (mail boxes). I highly recommend digging into these languages and getting comfortable with message passing programming paradigm. They will change how you approach and think about your application structure. Expect lots of Golang and some Erlang.

  7. Docker – Who is not into docker these days? If that’s not the case with you, leave this post right now and head over to docker user guide. That’s how important I find this piece of beauty (technology). Expect a lot about docker in my future posts.

  8. Startups – A lot of startup fun has kept me busy since ’10, some experiences and learnings are worth sharing.

  9. Android – I have been working full-time with mobiles (both Android and iOS) since ’12. Not much of application making but a lot of hacking with Adb protocol and libimobiledevice.

  10. System designing – Luckily, I happened to experience a lot of end-to-end system and network designing. This domain is of great interest once you start to have fun with Racks, Subnet, Routes, Switches, Firewalls, DNS, Multi-cast and entire suite of technology under this umbrella.

Will end this post with some interesting images from the past.

Fsck'd iPhone screen
Swollen iPhone screen due to high device temperature
Rackframe
Setting up Racks

How to perform X-FACEBOOK-PLATFORM and Google Talk X-OAUTH2 XMPP authentication with PHP Jaxl library

Ever since Jaxl library first introduced support for X-FACEBOOK-PLATFORM XMPP authentication mechanism, it has changed significantly. Also, Google Talk now supports OAuth 2.0 Authorization, an XMPP extension to allow users to log in using OAuth 2.0 credentials.

Both these mechanisms are a big win for XMPP developers, since real-time conversation experience can now be provided to their application users without asking them for their passwords. In this blog post, I will demonstrate how to perform X-FACEBOOK-PLATFORM and X-OAUTH2 XMPP authentication mechanism using Jaxl v3.x PHP Library.

X-FACEBOOK-PLATFORM XMPP Authentication
Here is a quick guide on how to perform X-FACEBOOK-PLATFORM XMPP authentication using xfacebook_platform_client.php which comes bundled with Jaxl v3.x examples:

  • Visit Facebook Developer Apps page and register your application
  • Once registered, visit access token tool to get required parameters to perform X-FACEBOOK-PLATFORM authentication Facebook Access Token Tool
  • Click on the debug button next to User Token and make sure xmpp_login is one of the extended permissions (scope)
  • Enter downloaded Jaxl library folder and run from command line as follows:

    $ php examples/xfacebook_platform_client.php fb_user_id_or_username fb_app_key fb_access_token

You can now take the source code of xfacebook_platform_client.php and customize it for your application needs.

Google Talk X-OAUTH2 XMPP Authentication
Here is a quick guide on how to perform Google Talk X-OAUTH2 XMPP authentication using xoauth2_gtalk_client.php which comes bundled with Jaxl v3.x examples:

  • Visit Google OAuth Playground and input https://www.googleapis.com/auth/googletalk as the required scope. Press “Authorize API” and then “Allow Access” button on the redirected page
  • In step 2, simply press “Exchange authorize code for tokens” and copy the access token
  • Enter downloaded Jaxl library folder and run from command line as follows:

    $ php examples/xoauth2_gtalk_client.php username@gmail.com access_token

You can now take the source code of xoauth2_gtalk_client.php and customize it for your application needs.

Wasn’t that simple 🙂

JAXLXml – Strophe style XML Builder : Working with Jaxl – A Networking Library in PHP – Part 2

Prior to Jaxl v3.x, the most ugliest piece of code inside Jaxl library was handling of XML packets. If you are working with XMPP protocol which is all about sending and receiving XML packets, it can become a nightmare if you don’t have a proper XML manipulation library in your toolkit. For Jaxl v3.x, first thing I decided to write was JAXLXml class, which is a custom XML packet implementation with no external dependencies and is an extension over the ideas from Strophe.Builder class written by Jack Moffitt.

JAXLXml is generic enough to find a place inside any PHP application that requires easy and elegant XML packet creation. In this blog post, I will give an exhaustive overview of how to create XML packets using JAXLXml class.

JAXLXml Constructor
Depending upon the need, there are several different ways of initializing a JAXLXml object:

  • $xml_obj = new JAXLXml($name, $ns, $attrs, $text);
  • $xml_obj = new JAXLXml($name, $ns, $attrs);
  • $xml_obj = new JAXLXml($name, $ns, $text);
  • $xml_obj = new JAXLXml($name, $attrs, $text);
  • $xml_obj = new JAXLXml($name, $attrs);
  • $xml_obj = new JAXLXml($name, $ns);
  • $xml_obj = new JAXLXml($name);

where:

  • $name – the XML node name
  • $ns – the XML namespace
  • $attrs – Key-Value (KV) pair of XML attributes
  • $text – XML content

Here are a few examples for each constructor style shown above:

JAXLXml will sanitize attributes and text values as shown below:

Manipulating Attributes, Child Nodes and Content
Below is an exhaustive list of methods available over initialized JAXLXml object $xml_obj for manipulating attributes, child nodes and content:

  • c($name, $ns=null, $attrs=array(), $text=null) : Append a child node at current rover and update the rover to point at newly added child node. Rover is nothing but a pointer indicating the level in the XML tree where this and other methods will perform. When an JAXLXml instance is initialized, rover points to the top level node.
  • cnode($node) : Append a child node given by $node (a JAXLXml object) at current rover and update the rover to point at newly added child node.
  • t($text, $append=FALSE) : Update text of the node pointed by current rover
  • top() : Move rover back to the top in the XML tree
  • up() : Move rover one step up the XML tree
  • attrs($attrs) : Merge new attributes specified as KV pair $attrs with existing attributes at the current rover.
  • match_attrs($attrs) : Accepts a KV pair of attributes $attrs, return bool if all keys exist and have same value as specified in the passed KV pair.
  • exists($name, $ns=null, $attrs=array()) : Checks if a child with $name exist. If found, return matching child as JAXLXml object otherwise false. If multiple children exist with same name, this function will return on first matching child
  • update($name, $ns=null, $attrs=array(), $text=null) : Update $ns, $attrs and $text (all at once) of an existing child node $name
  • to_string($parent_ns=null) : Return string representation of JAXLXml object

Method Chaining
The best thing one will find while working with JAXLXml class is that all the above methods are chain-able i.e. Any complex XML structure can be built with a single line of code.

Here is an example building a fairly nested XML structure in a single line of code:

Working with Jaxl – A Networking Library in PHP – Part 1 – An Introduction, Philosophy and History

Development of Jaxl library started way back in December’07 while I was working on a self-initiated project called Gtalkbots. The project is now dead, if you are interested in knowing more about it go through Gtalkbots BlogSpot. Jaxl v1.x was first released in Jan’09 and about a year later in Aug’10 Jaxl v2.x was released. First two versions were released as JAbber XMPP Library for writing clients and external server components.

While working on my startup Jaxl – A Platform As A Service (PAAS) for developing real-time applications, I started experiencing v2.x limitations when my external server side components were unable to process XMPP packets at the speed they were sent by ejabberd server. I started restructuring and refactoring the library which gave birth to Jaxl v3.x. Since v3.x was initially being used for developing the entire infrastructure, it shaped up as a networking library in PHP with stable support for XMPP protocol. However, later I had to rewrite several infrastructure components in Erlang Programming Language due to several issues that PHP as a language couldn’t solve (after all PHP wasn’t made for such tasks). Finally in April’12, Jaxl v3.x was open sourced.

Jaxl v3.x is an asynchronous, non-blocking, event based networking library in PHP for writing custom TCP/IP client and server implementations. From previous versions, Jaxl library inherits a full blown stable support for XMPP protocol stack. In v3.0, support for HTTP protocol stack was also introduced. At the heart of every protocol stack sits a Core stack. It contains all the building blocks for everything that we aim to do with Jaxl library. Both XMPP and HTTP protocol stacks are written on top of the Core stack. Infact the source code of these protocol implementations knows nothing about the standard (inbuilt) PHP socket and stream methods.

Philosophy
Jaxl is designed to work asynchronously in a non-blocking fashion and provides an event based callback API. Now what does all that mean?

By non-blocking and asynchronous it means, when a library function like:
$jaxl->send($stanza); is called, it will return immediately i.e. this function call will NOT block any further execution of your application script until $stanza has actually been sent over the connected TCP socket. Infact, when this function is called, passed $stanza object is put into an output buffer queue, which will be flushed as and when underlying TCP socket is available for writes. Similarly, most of the available methods (wherever required and possible) inside Jaxl library are non-blocking and asynchronous in nature.

By event based callback API it means, application code will need to register/add callbacks over necessary events as they occur inside Jaxl instance lifecycle. A list of available event callbacks with some explanation can be found here. For example, most of the XMPP applications will usually register a callback over on_auth_success event. As and when this event occurs inside Jaxl instance lifecycle, registered function will be callback’d with necessary parameters (if any).

Related Links

  • Read library documentation
  • Download the latest and greatest source from GitHub.
  • Have any Question? Want to discuss? Need Help? Use Google Group/Forum.
  • Found something missing or a bug in the source code? Kindly report an issue.
  • Fixed a bug? Want to submit a patch? Want to improve documentation? Checkout source code and contribute to the library

XMPP Application Examples

HTTP Application Examples

Stay Tuned
In coming weeks, under this series of blog posts titled “Working with Jaxl – A Networking Library in PHP”, I will cover following major topics with sample code:

  • Explanation of each Core stack class and how to use them
  • Design of each XMPP and HTTP stack class
  • XMPP over HTTP
  • XMPP File Transfer and Multimedia Sessions
  • Understanding and Using External Jabber Components
  • Asynchronous Job/Task Queues
  • Developing Concurrent and Parallel Systems

If you have any specific topic that you would like me to be cover, kindly let me know via your comments here.

Announcing Jaxl v3.x – asynchronous, non-blocking I/O, event based PHP client/server library

Jaxl v3.x is a successor of v2.x (and is NOT backward compatible), carrying a lot of code from v2.x while throwing away the ugly parts. A lot of components have been re-written keeping in mind the feedback from the developer community over the last 4 years. Also Jaxl shares a few philosophies from my experience with erlang and python languages.

Jaxl is an asynchronous, non-blocking I/O, event based PHP library for writing custom TCP/IP client and server implementations. From it’s previous versions, library inherits a full blown stable support for XMPP protocol stack. In v3.0, support for HTTP protocol stack was also added.

At the heart of every protocol stack sits a Core stack. It contains all the building blocks for everything that we aim to do with Jaxl library. Both XMPP and HTTP protocol stacks are written on top of the Core stack. Infact the source code of protocol implementations knows nothing about the standard (inbuilt) PHP socket and stream methods.

Source code on GitHub

Examples

Documentation

Group and Mailing List

Create a bug/issue

Read why v3.x was written and what traffic it has served in the past.

Releasing Jaxl 2.0 – Object oriented XMPP framework in PHP

After months of restructuring the Jaxl library, I am pleased to announce Jaxl 2.0, an object oriented XMPP framework in PHP for developing real time applications for browsers, desktops and hand held devices.

What’s new in Jaxl 2.0?

  • A lot of structural changes has been done from the previous version to make it more scalable, robust, flexible and easy to use
  • Library now provides an event mechanism, allowing developers to register callbacks for various xmpp events in their application code
  • Use integrated BOSH support to write real time web applications in minutes
  • More than 10 new implemented XMPP extensions (XEP’s) added
  • Development hosting moves to github, stable releases available at google code

Documentation for Jaxl users
Below is a list of getting started documentation for XMPP app developers:

Implemented XEP’s
A lot of new XEP’s has been implemented and packaged with Jaxl 2.0. Developers can use Jaxl event mechanism to implement new XEP’s without knowing the working of other core parts of the library.

Below is a list of released implemented XEP with Jaxl 2.0:

Documentation for project contributors
For developers interested in contributing to the Jaxl project, here is a list of insight documentation to get you started:

  • Jaxl core workflow and architecture (coming soon)
  • How to implement new XMPP extensions using Jaxl (coming soon)

Useful Links
For live help and discussion join jaxl@conference.psi-im.org chat room

WordPress Toolbar v 2.2 : Custom toolbar url, Support for WPMU and bug fixes

download-wordpress-plugin

WordPress toolbar plugin provide a facebook, digg style toolbar for all outgoing links from your blog posts. The toolbar url defaults to http://yourblog/wp-content/plugins/wordpress-toolbar/toolbar.php. However with version 2.2, blog admin can customize toolbar url to http://yourblog/wordpress-toolbar/ through the admin panel. A lot of other enhancements have been added like cross-plugin compatibility and support for WPMU hosted blogs. Check full feature list below.

What’s New?
Listed below is list of new features and bug fixes released with v 2.2:

  1. Support for customizing toolbar url through admin panel
  2. Support for WPMU hosted blogs
  3. Support for removing “Get this Plugin” widget from the toolbar through admin panel
  4. Security fix for possible XSS attack. Fix done by passing encoded hash string instead of plain text parameters. Also added various security checks on toolbar page to avoid possible XSS attacks.
  5. Bug fix where plugin didn’t work as expected because of cross plugin compatibility issues. Fix done by replacing server side toolbar logic with client side (using jquery) logic.
  6. Bug fix to show sociable share icons and tinyurl share link only for single posts and pages
  7. Bug fix for unrecognizable code in the toolbar when the encoding of hosted blog is different from utf-8. Fix done by using hosted blog settings instead of hardcoded utf-8.

Also core plugin code has been restructured (OOPS oriented now) so that maintainability and support becomes easier and quicker.

Steps to customize the default toolbar URL
Enable WordPress Toolbar v 2.2 plugin. Assuming you want to change default toolbar url from /wp-content/plugins/wordpress-toolbar/toolbar.php to /wordpress-toolbar, follow these steps:

  1. Enable apache mod_rewrite
  2. Add AllowOverride All in your blog virtual host config file and restart apache
  3. Add following apache rewrite rule by editing your blog .htaccess file
    RewriteRule ^wordpress-toolbar$ wp-content/plugins/wordpress-toolbar/toolbar.php
    RewriteRule ^wordpress-toolbar/$ wp-content/plugins/wordpress-toolbar/toolbar.php
  4. If you have blogs hosted using WPMU, add following apache rewrite rules in .htaccess file
    RewriteRule ^wordpress-toolbar$ wp-content/plugins/wordpress-toolbar/toolbar.php
    RewriteRule ^wordpress-toolbar/$ wp-content/plugins/wordpress-toolbar/toolbar.php
    RewriteRule ^([0-9a-zA-Z-]+)/wordpress-toolbar$ $1/wp-content/plugins/wordpress-toolbar/toolbar.php
    RewriteRule ^([0-9a-zA-Z-]+)/wordpress-toolbar/$ $1/wp-content/plugins/wordpress-toolbar/toolbar.php
  5. Manually check if rewrite rules are working. Open your custom toolbar url and you should see a result similar to this http://abhinavsingh.com/blog/wordpress-toolbar
  6. If for some reasons you DO NOT see “Working! Though required parameters are missing.” on toolbar page, it means rewrite rules didn’t worked as expected. Before you proceed with the setup, you SHOULD fix rewrite rules
  7. Go to wordpress admin and click "Wordpress Toolbar" under Settings tab
  8. Update your new custom toolbar url as shown: wordpress-toolbar-v-2.2-custom-toolbar-url-demo
  9. Clear cache and verify your toolbar

Enjoy and kindly let me know if you have issues installing plugin on your host.

Introducing MemChat: Open source group chat framework in PHP supporting Memcached, APC, SQLite, Flat Files and MySQL

MemChat is an open source group chat framework for personal and enterprise level websites. Written in PHP, MemChat can be configured to run with Memcached, APC, SQLite, Flat files and MySQL as it’s storage engine. With memcached, APC and Flat files serving as temporary storages and MySQL, SQLites being permanent storage engines.

MemChat uses MemBurger API for storing all the incoming messages in the storage engine. MemBurger is an open source PHP wrapper for all the storage engines mentioned above providing Collapsed Forwarding and Stale-While-Revalidate functionality.

MemChat can also be configured to notify the site owners at various event handlers provided. e.g. In case of a wordpress blog, site owner might want to get notified when someone post a new message on one of the blog post. Infact developers can write plugins for custom event handlers. MemChat provides two kind of notification methods. By default, MemChat uses XMPP protocol to notify blog owners. For these kind of notifications, JAXL an open source Jabber XMPP Client Library is used to send instant messages as notifications. MemChat can also be configured to send notifications using SMTP protocol a.k.a EMail.

MemChat also comes with a profanity word filter using PHProfane library i.e. one can configure MemChat to block all spam messages posted in the chat rooms. Developers can write a plugin to add custom spam words in the profanity filter.

MemChat Flow Diagram:
Below is a flow diagram showing how all the above components are clubbed in as MemChat:
MemChat Workflow Diagram

MemChat use cases:
MemChat framework requires a unique alphanumeric id for each group you want to create. For example, WordPress blog makes a good environment to setup MemChat since every blog post have a unique id. Hence to setup MemChat on a wordpress blog all we need to do is, call

$memchat_ui_html = memchat_init($memchat_group_id);

method, where $memchat_group_id = Blog post id.

Similarly MemChat can be setup on a number of places as listed below:

  1. Forums: Since each forum have a unique id per discussion thread, MemChat suits well here. Forum owners can setup MemChat for allowing current viewing users of the thread to chat in real time. If using temporary storage engines (since they scale up well), forum owners can easily write event handlers to save chat messages asynchronously so that interesting discussions can be made a part of actual discussion thread.
  2. Social Networking Websites: A social network will generally have group pages, fan pages, event pages and application pages. Since all pages will have a unique id associated with them, MemChat is a good fit here. Simply pass $blog_id = “group-1234”, for setting up a chat room on group page having id 1234. Similarly for fan, event and application pages.
  3. Blogs: A blog as discussed above is a perfect place for setting up MemChat. WordPress blog owners will be able to setup MemChat on their blog using WP-Chat plugin.

Setting up MemChat in 3 steps:
MemChat can be setup in 3 simple steps:

  1. Download: MemChat is hosted on Google Code.
  2. Update Configuration: Update MemChat config file to setup various default behavior and actions.
  3. Initialize: Include MemChat class file in your PHP template.
    include_once("/path/to/memchat.class.php")

    Next simply initialize MemChat

    $memchat_ui_html = memchat_init($memchat_group_id);

    which will return the user interface HTML code. Finally append it to your template and MemChat group is ready to serve.

The ease of setup can be imagine from the fact that, WP-Chat wordpress plugin development required only the above 3 steps. I was able to develop the plugin within 5 minutes.

MemChat Performance Benchmark
Initial benchmarking results show, On a single apache and single storage instance MemChat is capable of handling over 250 (MySQL) incoming chat messages per second. Capacity reaches in thousands when configured with Memcached or APC cache. Again Memcached configuration and APC no stat feature makes a difference in performance results.

Why still in alpha?
Yes, MemChat is still in alpha release. I want to test all MemChat features on a live traffic and hence I chose my blog for its testing. I have installed MemChat on my blog which is currently using APC cache as storage medium. Over two weeks or so I will configure MemChat with various other storage engines to have a final round of test. Also the event handler and PHProfane plugin feature is still under development. MemBurger API also requires a more modular design pattern so that extending it for any other storage types become trivial.

Try out MemChat on my blog, can be seen as a facebook type chat bar below. This will be soon available as WP-Chat plugin for wordpress blogs. WP-Chat user interface is customizable just like wordpress themes.

Let me know your views and suggestions for improvement.