[Commotion-dev] Critical services

[Peter Folk]

I think that this type of project would have the most effect if it
included special-case optimizations for four critical services, in order
of importance:

   1. One-to-many updates (RSS including twitter and facebook; inbound)
   2. One-to-many updates (updating twitter, facebook, etc; outbound)
   3. Short one-to-one messages (IM, SMS)
   4. Medium-length one-to-one messages (email)

The first item could be implemented as follows:

*Caching Client-Server RSS Proxy* (CSSRSSP)
In net-accessible areas, servers run a daemon that accepts incoming SSH
connections (for compression as much as security), over which the
following protocol is run:

    * client sends monitor<space>N<cr><lf> (for int N) to indicate that
      it is monitoring N feeds.  N should be 10 digits, 0-padded on the
      left.
    * client sends N monitor notifications, consisting of
      A<space>B<space>C<space>D<cr><lf>, where A is the number of users
      monitoring the feed, B is the 64-bit UTC timestamp of the last
      update received for this feed, and C is the number of characters
      in the feed's URL, and D is that URL.  Numbers are 10, 20, and 5
      digits, respectively, 0-padded on the left, to make read() as easy
      as possible
    * server begins to monitor the feed urls, polling each feed (with
      higher frequency, down to the feeds minimum update time) in a
      round-robin fashion weighted by # of monitoring users
    * When a feed is updated, server sends A<space>B<space>C<cr><lf>,
      where A is the timestamp of the update (20 digits), B is 0 if this
      is the full feed, the length of the update (in bytes; 10 digits)
      otherwise, and C is the diff of the RSS feed from the timestamp
      last reported to be cached by this client
    * Client periodically updates server with new "monitor N" info.  A
      user count of 0 removes the feed from the client's list of
      monitored feeds.  Otherwise, an entry with an already-seen URL
      updates the timestamp and user count on record for that client
    * Server caches the most recent list of monitored feeds and metadata
      in a file, with name == MD5 checksum of the file
    * Client can resume a cached state (eg, after disconnection) by
      sending resume<space>A<cr><lf> with the A == the md5sum of a
      previous state, to which the server should reply either ok<cr><lf>
      if it has such a state or no<cr><lf> if it does not

On the client side, this general daemon can be used to implement a
caching twitter, facebook, and general RSS-feed proxy (each one would
require some custom code).


The second item could be implemented with custom code per updateable
service in the client-side interface to the proxy.  That might be
preferable to using standard HTTP interface because those short updates
could be prioritized above general HTTP traffic.


The third item is only hard if you actually need to proxy the sessions
to disallow or reprioritize file transfers (which should go via the same
priority as HTTP, for example).  That amount of prioritization might
come for free with classification by port with per-stream bandwidth
throttling.


The 4th item is basically a normal SMTP-hijacking mail server that is
configured to be pretty much an open relay, and deal well with being
disconnected (maybe have some batch-transfer mechanism like UUCP). 
Extra points for cacheing the login credentials and original target of
an SMTP request, so you can forward a message to its intended
destination and avoid relaying issues...

Some thoughts,
Peter


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chambana.net/pipermail/commotion-dev/attachments/20110201/a0f8e68b/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 552 bytes
Desc: OpenPGP digital signature
URL: <http://lists.chambana.net/pipermail/commotion-dev/attachments/20110201/a0f8e68b/attachment.sig>