[Commotion-dev] Python-based testing framework across Ubuntu/Debian and Android (and maybe OSX)?
seamus tuohy
s2e at opentechinstitute.org
Tue Apr 2 14:44:59 UTC 2013
On 04/02/2013 10:31 AM, Ben West wrote:
> Hi Seamus and Dan,
>
> Replying about having Python on (and off) OpenWRT:
>
> The cross-platform nature seems like it would be very helpful to
> maintain a single collection of tests vectors to apply as Commotion
> finds its way onto more platforms (Windows, OSX). But, the obvious
> tradeoff here is the size of the Python interpreter.
Agreed!
>
> Also, the test scripts that query a node's OLSR instance via jsoninfo
> could also run /on the client/, rather than directly on the node. The
> node would just need to open up the jsoninfo port 9090 on its firewall
> behind the public AP interface (or have the client use its private
> AP). I can tweak the Python scripts (tho very little needed) to
> accommodate this option.
I think that is a great idea for the openWRT instances. Though, as we
get the other platforms up and running it will be less and less needed,
since we will be running more powerful devices as nodes.
>
> Besides that, I think I may be able to tweak the Python Makefile
> provided with OpenWRT, so that the compiled package is much smaller
> than its current 1.5MByte footprint. The Python v2.6 implementation
> bundled into Py4A (which runs all these tests fine) is /only 150kB/,
> by comparison. Still, admitted that installing another language on
> what is already disk- and RAM-bound devices is less than ideal.
I don't know what the consensus is, but as much as I would love to start
to play with python on the OpenWRT nodes, I think we will be able to use
the existing testing code you built, along with some client based tweaks
to accurately test a OpenWRT based mesh.
>
> Finally, sorry for using the term 'bytecode.' That's not really what
> we need; compiled Python bytecode is neither smaller than source nor
> stand-alone executable. What I meant is maybe compile the Python
> testsuite into a single standalone executable using a tool like these:
> http://cx-freeze.sourceforge.net/
> http://www.pyinstaller.org/
>
> On Tue, Apr 2, 2013 at 8:58 AM, seamus tuohy
> <s2e at opentechinstitute.org <mailto:s2e at opentechinstitute.org>> wrote:
>
> Hey,
>
>
> On 04/02/2013 04:20 AM, Ben West wrote:
>> Hi All,
>>
>> I've created a repo of a few basic tests implemented as unittests
>> in Python 2.6+. This repo contains just the Python scripts
>> themselves, so that it may be included later as a submodule into
>> other repos containing platform-specific packaging stuff.
>> https://github.com/westbywest/commotion-tests-core
>>
>> Due please note I expect this repo to change, and even its
>> location to maybe switch to OTI's github account at some point soon.
>>
>> To serve as hopefully demonstrative examples of the test coverage
>> possible, the scripts I just checked in test for the following:
>>
>> * ping localhost
>> * ping gateway IP (either default gateway or the one assigned
>> by OLSRd smartgateway)
>> * ping Google DNS (aka do we have Internet?)
>> * presence of an active olsrd process
>> * olsrd responds to jsoninfo requests
>> * link quality of the next hop is above a specified threshold
>>
>> These scripts should run under the following Python versions /
>> platforms:
>>
>> * Python v2.7+ under Debian/Ubuntu
>> * Python for Android (Py4A) app r5+ / Scripting Layer for
>> Android (SL4A) app r6+
>> * Python-mini v2.7+ under OpenWRT 12.09+
>>
>> ... to test the following Commotion implementations, respectively
>> (where each is enabled by the user, not by the Python script):
>>
>> * commotion-mesh-applet
>> * Mesh Tether app
>> * Commotion-OpenWRT DR1, with *python*, *python-json*, and
>> *olsrd-mod-jsoninfo* modules installed
>>
>> Questions about how to proceed:
>>
>> * Whether to proceed with Python-based testing framework? I
>> tried to heavily leverage the cross-platform compatibility,
>> but is it worthwhile?
>>
>> * The Python for Android implementation chosen, Py4A, is only
>> Python v2.6. Is having Python v2.7 on Android worth possibly
>> compiling it into a custom APK?
>> * Many of the desired test vectors, e.g. throughput testing,
>> require the test run simultaneously on at least 2 nodes.
>> It's pretty easy to write a crude server in Python to
>> function as one half of a throughput test, but does the
>> complexity of running different Python scripts on different
>> nodes simultaneously become unreasonable
>>
> Not at all, I think that it is necessary to have different nodes
> running tests simultaneously. In most of the openWRT testing
> situations we run currently, we have client devices running
> simultaneously to elucidate information about the mesh nodes.
>>
>> * The python-mini module for OpenWRT chews up 1.5Mbytes of
>> flash, and it doesn't include unittest by default. This
>> unfortunately appears to be /too much already/; it wouldn't
>> fit on a Nanostation flash w/o removing lots of stuff. Maybe
>> this size could be trimmed down by modifying the python-mini
>> package's Makefile, or by distributing test scripts in
>> bytecode form to OpenWRT nodes.
>>
> I think that loading python on to openWRT nodes is impractical.
> Having client devices running tests over the mesh with possible
> on-node analytics running for later collection will give us a good
> idea of QOS without having to load down the OpenWRT nodes, and as
> such, change their performance. We will not get the same level of
> detail as the android or desktop nodes, but it is really the best
> we can do while maintaining an accurate image of the nodes
> performance.
>
>
>> *
>>
>>
>>
>>
>>
>> On Fri, Mar 15, 2013 at 10:14 PM, Andrew Reynolds
>> <andrew at opentechinstitute.org
>> <mailto:andrew at opentechinstitute.org>> wrote:
>>
>> It sounds reasonable. Could you sketch out what you have in
>> mind? How
>> much of the network setup were you thinking of building into
>> the test
>> framework vs. simply triggering and testing?
>>
>> -andrew
>>
>> On 03/15/2013 02:48 PM, Ben West wrote:
>> > Hi All,
>> >
>> > In lieu of recent progress towards getting Commotion to a
>> working state on
>> > Android, Ubuntu/Debian. and now possibly OSX, what thoughts
>> about building
>> > a simple and (to whatever degree feasible) cross-platform
>> testing framework
>> > in Python?
>> >
>> > The general idea is that python scripts could be used to
>> start hitting the
>> > test vectors listed here (note the server appears to be
>> really slow):
>> >
>> >
>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Mesh-Routing-Tech-Evaluations
>> >
>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Testbed-Requirements-based-on-test-suite-defined-above
>> >
>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Release-Candidate-Test-Regimen
>> >
>> > That is, assuming Ubuntu/Debian/OSX's python support as a
>> starting point,
>> > could these lighweight python implementations allow for
>> some unified test
>> > scripts across platforms?
>> >
>> > http://qpython.com/ (for Android)
>> >
>> https://dev.openwrt.org/browser/packages/lang/python/Makefile
>> (for OpenWRT,
>> > to be compiled as module)
>> >
>> > Has anyone on the list had good experience with these Python
>> > implementations?
>> >
>> > My original thought for such testing scripts was to do them
>> in shell
>> > scripting, but I'm guessing Python would be easier and more
>> powerful.
>> >
>> >
>> >
>> > _______________________________________________
>> > Commotion-dev mailing list
>> > Commotion-dev at lists.chambana.net
>> <mailto:Commotion-dev at lists.chambana.net>
>> > https://lists.chambana.net/mailman/listinfo/commotion-dev
>> >
>>
>>
>>
>> _______________________________________________
>> Commotion-dev mailing list
>> Commotion-dev at lists.chambana.net
>> <mailto:Commotion-dev at lists.chambana.net>
>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>>
>>
>>
>>
>> --
>> Ben West
>> http://gowasabi.net
>> ben at gowasabi.net <mailto:ben at gowasabi.net>
>> 314-246-9434 <tel:314-246-9434>
>>
>>
>> _______________________________________________
>> Commotion-dev mailing list
>> Commotion-dev at lists.chambana.net <mailto:Commotion-dev at lists.chambana.net>
>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
> _______________________________________________
> Commotion-dev mailing list
> Commotion-dev at lists.chambana.net
> <mailto:Commotion-dev at lists.chambana.net>
> https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
>
>
> --
> Ben West
> http://gowasabi.net
> ben at gowasabi.net <mailto:ben at gowasabi.net>
> 314-246-9434
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chambana.net/pipermail/commotion-dev/attachments/20130402/945d8426/attachment-0001.html>
More information about the Commotion-dev
mailing list