Forum

Posted by
Oliver Smith  -  July 2010
I've recently found a portable, scalable, incredibly-high-performance, message-passing library called "ZeroMQ" http://www.zeromq.org/.

There are a slew of language wrappers for it (C, C++, Java, Perl, Python, Ruby, Lua, etc).

It presents a socket like API which means if you go look at it right now, you'll probably just think "meh, sockets?". But actually a whole lot more in a very tiny package.

For instance, it handles packetization of messages, ensuring that a recipient only receives complete packets.

More importantly, it handles multiplexing/load balancing of connections. One socket can represent many peers.

The beauty of ZeroMQ:
- Tiny, tiny footprint,
- Efficiency: it implements specific socket patterns rather than using homogenous socket types,
- Throughput: I'm blown away by some of the throughput achievable with ZeroMQ,
- One API fits all

For instance, you can replace the usual nightmarish headache of mutexes, condition variables and signals for inter-thread communications by using the Erlang-style approach of sending messages between threads.

Because this requires encapsulation, it produces excellent parallelism. The overhead gets amortized and - because it is lock free - reduced. Because you are using what operating systems recognize as IO, a thread waiting for data and using recv() blocks far more elegantly than a thread waiting on a mutex/futex.

Having a Roxen-created ZeroMQ Pike wrapper might help bring fresh attention to Roxen, since it would provide a very elegant way to build new Roxen modules which control remote apps using ZeroMQ.

--- zeromq multi-cast server, in C++ ---
zmq::context_t ctx(1) ; // ZeroMQ instance with 1 io thread.
zmq::socket_t  socket(ctx, ZMQ_PUB) ; // Published thread.
socket.bind("tcp://192.168.0.80:12345") ; // multicast over tcp
socket.bind("pgm://244.0.0.80:12345") ; // multicast via pgm library
// note: I bound one zmq socket to two destinations :)

char data[64] ;
do
{
  // Send a random number most updates
  if ( rand() % 10 > 0 )
    sprintf(data, "rand %u", rand());
  else
    sprintf(data, "time %u", time(NULL));

  zmq::message_t msg(data, strlen(data)+1, NULL) ;
  socket.send(msg) ;

  // Random delay.
  usleep(100 + rand() % 500) ;
}
--- end server ---

--- client in C++ that connects via PraGmatic Multicast library ---
zmq::context_t ctx(1) ;
zmq::socket_t socket(ctx, ZMQ_SUB) ; // Subscriber
socket.setsockopt(ZMQ_SUBSCRIBE, "", 0) ; // listen to all messages
socket.connect("pgm://244.0.0.80:12345") ;

zmq::message_t msg ;
while ( socket.recv(&msg) )
{
  printf("recvd: %s\n", (char*)msg.data()) ;
}
--- end C++ client ---

--- client in lua using multicast over tcp ---
require 'zmq'
local ctx = zmq.context(1)
local s = zmq.socket(ctx, ZMQ_SUB)
s.setsockopt(ZMQ_SUBSCRIBE, "time") -- Only listen to messages flagged as 'time'
s.connect("tcp://192.168.0.80:12345") -- multicast over tcp

local m = zmq.msg()
while zmq.recv(msg) do
  print("got update: " .. msg.data())
end
--- end lua client ---

Just to demonstrate the flexibility of the API, here is how you can trivially set up in-process threading with an API that could be converted to scalable simply by changing the socket name.

--- threaded app in C++ ---
#include <stdio.h>
#include <stdlib.h>
#include <zmq.hpp>
#include <pthread.h>

const char* const socketName = "inproc://work" ;

static void processWork(zmq::message_t& work)
{
  static int counter = 0 ;
  if ( (++counter % 1000) == 0 )
    printf("Received work %u\n", counter) ;
}

static void* threadFn(void* vptr)
{
  zmq::context_t& ctx = *(zmq::context_t*)vptr ;
  zmq::socket_t insock(ctx, ZMQ_UPSTREAM) ;
  insock.connect(socketName) ;
 
  zmq::message_t work ;
  while ( insock.recv(&work) )
  {
    _process(work) ;
  }
}

int main(int argc, const char* const argv[])
{
  zmq::context_ctx(0) ; // in-proc doesn't need an IO thread.
  zmq::socket_t outsock(ctx, ZMQ_DOWNSTREAM) ;
  outsock.bind(socketName) ;

  pthread_t thread ;
  pthread_create(&thread, NULL, threadFn, &ctx) ;

  for ( int i = 0 ; i < 1000000 ; ++i )
  {
    zmq::message_t msg(sizeof(int)) ;
    *(int*)msg.data() = (int)rand() ;
    outsock.send(msg) ;
    usleep(100 + rand() % 500) ;
  }

 return 0 ;
}
--- end ----

If you changed the socket name to "ipc://something" then, without any further work, you would have inter-process scalability.

Change it to "tcp://ip.add.re.ss:port" and you have a cross-machine scalable application.
 
Posted by
Marcus Agehall  -  July 2010
Seems like a neat project. While it sounds good in theory, it may not be all that good for Pike, and especially not for IPC.

Passing messages between threads is trivial in Pike, using Thread.Queue for example. If we were to introduce a library like this, we would need to encode all data that is to be passed as a message and then decode it, which would probably be much more expensive than simply handing the data over via a Thread.Queue object.

Even for non-IPC communication, it's not as usefull as one might think. Both the sender and the receiver must be able to understand the structures sent and that is going to be hard to accomplish from a pike level. Ofc, you could build an ADT.Struct object and send that, but then you could just as well do that using a Stdio.File object, i.e. there isn't much point from a Pike perspective.

These sorts of libraries are really useful on a C/C++ level, but they won't add much more than what Stdio.File and encode_value() already does.
 
Posted by
Oliver Smith  -  July 2010
The main reason for implementing a Pike wrapper is interoperability, specifically enabling Roxen modules that can interop with other ZeroMQ threads/processes/machines. That is, Pike as a ZeroMQ client.

ZeroMQ doesn't impose any definition on the message payload. It's just a message-passing utility.

The guys who developed ZeroMQ wanted the message-passing efficiency and portability that Erlang offers without ... Erlang. And it's at least 10-100x faster than native Erlang message passing (one of the first wrappers developed for it was for Erlang, hehe :)

It's lock free, and for IPC zero-copy too. And because it's IO based, a worker thread waiting for work to do doesn't have to goof around with mutexes etc. Instead, when it is ready, it sits blocking on an IO call (recv) which gets it best-of-bread thread scheduling by the OS.

Stdio.File doesn't provide the rich capabilities or anything like the performance that ZeroMQ offers. For instance, you're going to have a hard time opening a Stdio.File that connects to 8-9 different endpoints...

You'd have to check out the various ZeroMQ blogs to see some of the throughput benchmarks. I was pretty blown away.

It already has wrappers for a whole bunch of languages: Ada  Basic  C  C++  Common Lisp  Erlang  Go  Haskell  Java  Lua  .NET  Objective-C  ooc  Perl  PHP  Python  Ruby  Ruby (FFI)

Example:

---- python example ---
import zmq

# 1 IO thread
ctx = zmq.Context(1)

# "REP" sockets receive a request and send a reply.
sock = ctx.socket(zmq.REP)

# No need to explain what this does...
sock.bind("tcp://192.168.0.80:8080")

# But then you can also do this...
sock.bind("tcp://192.168.0.81:8080")

# "sock" now talks to two different endpoints.

# And maybe we want to send the same messages to
# other processes running on the same machine?

sock.bind("ipc:///tmp/my-zeromq-ipc-messages")

# So 'sock' is now talking to multiple endpoints.
# And each of those could, conceivably, actually
# have multiple servers listening on them.

sock.send("Hello")
--- end ---

FYI: The "Mongrel2" web server is written using ZeroMQ ( http://mongrel2.org/home ).

 
Posted by
Marcus Agehall  -  July 2010
Well, writing a simple wrapper that can only pass strings is a no-brainer, but not that useful as all the message processing would have to take place in Pike.

Allowing Pike to serve as a client might be useful for some things, but then you wouldn't have the speed you are talking about since you would have to parse all the messages you receive to understand the data within them.

A message bus can be really fast when it passes data to some other system where the data can simply be slotted into a struct and used, but that isn't the case in Pike.

For single process IPC, Pike already has lock-free mechanisms to pass any data structure between threads. I don't know where you get the idea that it's a complex or slow thing.

For multiprocess IPC, you would have to use encode/decode_value(), but then you can use Stdio.File just as well as this library. You can do multicast using Pike, so honestly, there isn't much difference.

That all said, I don't see Roxen writing a 0MQ wrapper anytime soon. I don't even think it fits in the standard Pike distribution. However, a wrapper module could very well be written and placed on got-pike.org. I might even be inclined to write it myself if I find the time...
 
Posted by
Oliver Smith  -  July 2010
Yeah, I'm not so much thinking of this for general purpose Pike usage. I'm thinking of Roxen-as-Management-server.

This just seemed the correct forum to suggest it.I figured I'd ask here incase any existing Roxen users were already thinking about doing it before I try and dig-up my old TeamSpeak2 PMOD and try to build one myself :)
 
1
Search this thread: