[Commotion-dev] Python-based testing framework across Ubuntu/Debian and Android (and maybe OSX)?

Tue Apr 2 13:58:04 UTC 2013

Hey,

On 04/02/2013 04:20 AM, Ben West wrote:
> Hi All,
>
> I've created a repo of a few basic tests implemented as unittests in 
> Python 2.6+.  This repo contains just the Python scripts themselves, 
> so that it may be included later as a submodule into other repos 
> containing platform-specific packaging stuff.
> https://github.com/westbywest/commotion-tests-core
>
> Due please note I expect this repo to change, and even its location to 
> maybe switch to OTI's github account at some point soon.
>
> To serve as hopefully demonstrative examples of the test coverage 
> possible, the scripts I just checked in test for the following:
>
>   * ping localhost
>   * ping gateway IP (either default gateway or the one assigned by
>     OLSRd smartgateway)
>   * ping Google DNS (aka do we have Internet?)
>   * presence of an active olsrd process
>   * olsrd responds to jsoninfo requests
>   * link quality of the next hop is above a specified threshold
>
> These scripts should run under the following Python versions / platforms:
>
>   * Python v2.7+ under Debian/Ubuntu
>   * Python for Android (Py4A) app r5+ / Scripting Layer for Android
>     (SL4A) app r6+
>   * Python-mini v2.7+ under OpenWRT 12.09+
>
> ... to test the following Commotion implementations, respectively 
> (where each is enabled by the user, not by the Python script):
>
>   * commotion-mesh-applet
>   * Mesh Tether app
>   * Commotion-OpenWRT DR1, with *python*, *python-json*, and
>     *olsrd-mod-jsoninfo* modules installed
>
> Questions about how to proceed:
>
>   * Whether to proceed with Python-based testing framework? I tried to
>     heavily leverage the cross-platform compatibility, but is it
>     worthwhile?
>
>   * The Python for Android implementation chosen, Py4A, is only Python
>     v2.6.  Is having Python v2.7 on Android worth possibly compiling
>     it into a custom APK?
>   * Many of the desired test vectors, e.g. throughput testing, require
>     the test run simultaneously on at least 2 nodes.  It's pretty easy
>     to write a crude server in Python to function as one half of a
>     throughput test, but does the complexity of running different
>     Python scripts on different nodes simultaneously become unreasonable
>
Not at all, I think that it is necessary to have different nodes running 
tests simultaneously. In most of the openWRT testing situations we run 
currently, we have client devices running simultaneously to elucidate 
information about the mesh nodes.
>
>   * The python-mini module for OpenWRT chews up 1.5Mbytes of flash,
>     and it doesn't include unittest by default.  This unfortunately
>     appears to be /too much already/; it wouldn't fit on a Nanostation
>     flash w/o removing lots of stuff.  Maybe this size could be
>     trimmed down by modifying the python-mini package's Makefile, or
>     by distributing test scripts in bytecode form to OpenWRT nodes.
>
I think that loading python on to openWRT nodes is impractical. Having 
client devices running tests over the mesh with possible on-node 
analytics running for later collection will give us a good idea of QOS 
without having to load down the OpenWRT nodes, and as such, change their 
performance. We will not get the same level of detail as the android or 
desktop nodes, but it is really the best we can do while maintaining an 
accurate image of the nodes performance.

>  *
>
>
>
> On Fri, Mar 15, 2013 at 10:14 PM, Andrew Reynolds 
> <andrew at opentechinstitute.org <mailto:andrew at opentechinstitute.org>> 
> wrote:
>
>     It sounds reasonable. Could you sketch out what you have in mind? How
>     much of the network setup were you thinking of building into the test
>     framework vs. simply triggering and testing?
>
>     -andrew
>
>     On 03/15/2013 02:48 PM, Ben West wrote:
>     > Hi All,
>     >
>     > In lieu of recent progress towards getting Commotion to a
>     working state on
>     > Android, Ubuntu/Debian. and now possibly OSX, what thoughts
>     about building
>     > a simple and (to whatever degree feasible) cross-platform
>     testing framework
>     > in Python?
>     >
>     > The general idea is that python scripts could be used to start
>     hitting the
>     > test vectors listed here (note the server appears to be really
>     slow):
>     >
>     >
>     https://code.commotionwireless.net/projects/commotion/wiki/Testing#Mesh-Routing-Tech-Evaluations
>     >
>     https://code.commotionwireless.net/projects/commotion/wiki/Testing#Testbed-Requirements-based-on-test-suite-defined-above
>     >
>     https://code.commotionwireless.net/projects/commotion/wiki/Testing#Release-Candidate-Test-Regimen
>     >
>     > That is, assuming Ubuntu/Debian/OSX's python support as a
>     starting point,
>     > could these lighweight python implementations allow for some
>     unified test
>     > scripts across platforms?
>     >
>     > http://qpython.com/ (for Android)
>     > https://dev.openwrt.org/browser/packages/lang/python/Makefile
>     (for OpenWRT,
>     > to be compiled as module)
>     >
>     > Has anyone on the list had good experience with these Python
>     > implementations?
>     >
>     > My original thought for such testing scripts was to do them in shell
>     > scripting, but I'm guessing Python would be easier and more
>     powerful.
>     >
>     >
>     >
>     > _______________________________________________
>     > Commotion-dev mailing list
>     > Commotion-dev at lists.chambana.net
>     <mailto:Commotion-dev at lists.chambana.net>
>     > https://lists.chambana.net/mailman/listinfo/commotion-dev
>     >
>
>
>
>     _______________________________________________
>     Commotion-dev mailing list
>     Commotion-dev at lists.chambana.net
>     <mailto:Commotion-dev at lists.chambana.net>
>     https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
>
>
> -- 
> Ben West
> http://gowasabi.net
> ben at gowasabi.net <mailto:ben at gowasabi.net>
> 314-246-9434
>
>
> _______________________________________________
> Commotion-dev mailing list
> Commotion-dev at lists.chambana.net
> https://lists.chambana.net/mailman/listinfo/commotion-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chambana.net/pipermail/commotion-dev/attachments/20130402/b2fc4f21/attachment.html>