[Commotion-dev] Critical services

[Tim Yardley]

Inline.

On Tue, Feb 1, 2011 at 8:48 AM, Peter Folk <peter at volo.net> wrote:
> I think that this type of project would have the most effect if it included
> special-case optimizations for four critical services, in order of
> importance:
>
> One-to-many updates (RSS including twitter and facebook; inbound)
> One-to-many updates (updating twitter, facebook, etc; outbound)
> Short one-to-one messages (IM, SMS)
> Medium-length one-to-one messages (email)
>

All of these sound like a good start.  The key is that localized
"caching" is limited by the hardware that we have (storage, etc).
With a flaky uplink, you want to be able to take an update and get it
where it needs to go whenever you have connectivity to do so.

> The first item could be implemented as follows:
>
> Caching Client-Server RSS Proxy (CSSRSSP)
> In net-accessible areas, servers run a daemon that accepts incoming SSH
> connections (for compression as much as security), over which the following
> protocol is run:
>
> client sends monitor<space>N<cr><lf> (for int N) to indicate that it is
> monitoring N feeds.  N should be 10 digits, 0-padded on the left.
> client sends N monitor notifications, consisting of
> A<space>B<space>C<space>D<cr><lf>, where A is the number of users monitoring
> the feed, B is the 64-bit UTC timestamp of the last update received for this
> feed, and C is the number of characters in the feed's URL, and D is that
> URL.  Numbers are 10, 20, and 5 digits, respectively, 0-padded on the left,
> to make read() as easy as possible
> server begins to monitor the feed urls, polling each feed (with higher
> frequency, down to the feeds minimum update time) in a round-robin fashion
> weighted by # of monitoring users
> When a feed is updated, server sends A<space>B<space>C<cr><lf>, where A is
> the timestamp of the update (20 digits), B is 0 if this is the full feed,
> the length of the update (in bytes; 10 digits) otherwise, and C is the diff
> of the RSS feed from the timestamp last reported to be cached by this client
> Client periodically updates server with new "monitor N" info.  A user count
> of 0 removes the feed from the client's list of monitored feeds.  Otherwise,
> an entry with an already-seen URL updates the timestamp and user count on
> record for that client
> Server caches the most recent list of monitored feeds and metadata in a
> file, with name == MD5 checksum of the file
> Client can resume a cached state (eg, after disconnection) by sending
> resume<space>A<cr><lf> with the A == the md5sum of a previous state, to
> which the server should reply either ok<cr><lf> if it has such a state or
> no<cr><lf> if it does not
>
> On the client side, this general daemon can be used to implement a caching
> twitter, facebook, and general RSS-feed proxy (each one would require some
> custom code).
>

I'm not familiar with this one, but it sounds useful.

>
> The second item could be implemented with custom code per updateable service
> in the client-side interface to the proxy.  That might be preferable to
> using standard HTTP interface because those short updates could be
> prioritized above general HTTP traffic.

I'd suggest just handling it on the server side if possible, as people
may or may not be able to customize apps on their resources.

>
>
> The third item is only hard if you actually need to proxy the sessions to
> disallow or reprioritize file transfers (which should go via the same
> priority as HTTP, for example).  That amount of prioritization might come
> for free with classification by port with per-stream bandwidth throttling.

Caching user credentials is more of the touchy situation.  You can
easily hold a desired HTTP post event...

>
>
> The 4th item is basically a normal SMTP-hijacking mail server that is
> configured to be pretty much an open relay, and deal well with being
> disconnected (maybe have some batch-transfer mechanism like UUCP).  Extra
> points for cacheing the login credentials and original target of an SMTP
> request, so you can forward a message to its intended destination and avoid
> relaying issues...

Agreed.

/tmy