[Commotion-dev] Python-based testing framework across Ubuntu/Debian and Android (and maybe OSX)?
Dan Staples
danstaples at opentechinstitute.org
Wed Apr 24 21:48:19 UTC 2013
What about porting some of this to Lua, since OpenWRT nodes already have
that interpreter? Would a lot be lost in the translation?
Also, we can look at the tests run at the latest BattleMesh last week. I
believe Seamus knows the repo where those are...I can't find them at the
moment.
On 04/24/2013 05:17 PM, Hans of Guardian wrote:
>
> Sorry to be late in weighing in this thread. I'm on the train to DC
> and diving back into a Commotion state of mind.
>
> I like having python test framework a lot, even if it doesn't run on
> all OpenWRT devices. To cover OpenWRT, you could set up OpenWRT on
> any device that has a lot of disk space, then run all the python stuff
> there. That would then give you a place to run the full test suite on
> OpenWRT. Then on the Ubuquiti hardware could be handled via some of
> the other ideas here, like shell scripts, or remote scraping via JSON,
> etc.
>
> Using python is just so vastly more productive that this approach will
> definitely be less work even though there would have to be some
> special cases for certain OpenWRT devices
>
> This reminds me, would this also then run in the VMs that you have
> been setting up? I think having some random box somewhere running
> like 20 or more various VMs with olsrd on them could be very nice as
> an automated testbed. This would not test the real world wireless
> conditions, but it would provide a way to run automated tests on olsrd
> meshes in a way that could be plugged into Jenkins builds. This kind
> of testbed would have caught the commit that broke meshing on ARM and
> amd64 that showed up in the more aggressive optimization of
> Ubuntu/quantal's compiler.
>
> .hc
>
> On Apr 2, 2013, at 10:31 AM, Ben West wrote:
>
>> Hi Seamus and Dan,
>>
>> Replying about having Python on (and off) OpenWRT:
>>
>> The cross-platform nature seems like it would be very helpful to
>> maintain a single collection of tests vectors to apply as Commotion
>> finds its way onto more platforms (Windows, OSX). But, the obvious
>> tradeoff here is the size of the Python interpreter.
>>
>> Also, the test scripts that query a node's OLSR instance via jsoninfo
>> could also run /on the client/, rather than directly on the node.
>> The node would just need to open up the jsoninfo port 9090 on its
>> firewall behind the public AP interface (or have the client use its
>> private AP). I can tweak the Python scripts (tho very little needed)
>> to accommodate this option.
>>
>> Besides that, I think I may be able to tweak the Python Makefile
>> provided with OpenWRT, so that the compiled package is much smaller
>> than its current 1.5MByte footprint. The Python v2.6 implementation
>> bundled into Py4A (which runs all these tests fine) is /only 150kB/,
>> by comparison. Still, admitted that installing another language on
>> what is already disk- and RAM-bound devices is less than ideal.
>>
>> Finally, sorry for using the term 'bytecode.' That's not really what
>> we need; compiled Python bytecode is neither smaller than source nor
>> stand-alone executable. What I meant is maybe compile the Python
>> testsuite into a single standalone executable using a tool like these:
>> http://cx-freeze.sourceforge.net/
>> http://www.pyinstaller.org/
>>
>> On Tue, Apr 2, 2013 at 8:58 AM, seamus tuohy
>> <s2e at opentechinstitute.org <mailto:s2e at opentechinstitute.org>> wrote:
>>
>> Hey,
>>
>>
>> On 04/02/2013 04:20 AM, Ben West wrote:
>>> Hi All,
>>>
>>> I've created a repo of a few basic tests implemented as
>>> unittests in Python 2.6+. This repo contains just the Python
>>> scripts themselves, so that it may be included later as a
>>> submodule into other repos containing platform-specific
>>> packaging stuff.
>>> https://github.com/westbywest/commotion-tests-core
>>>
>>> Due please note I expect this repo to change, and even its
>>> location to maybe switch to OTI's github account at some point soon.
>>>
>>> To serve as hopefully demonstrative examples of the test
>>> coverage possible, the scripts I just checked in test for the
>>> following:
>>>
>>> * ping localhost
>>> * ping gateway IP (either default gateway or the one assigned
>>> by OLSRd smartgateway)
>>> * ping Google DNS (aka do we have Internet?)
>>> * presence of an active olsrd process
>>> * olsrd responds to jsoninfo requests
>>> * link quality of the next hop is above a specified threshold
>>>
>>> These scripts should run under the following Python versions /
>>> platforms:
>>>
>>> * Python v2.7+ under Debian/Ubuntu
>>> * Python for Android (Py4A) app r5+ / Scripting Layer for
>>> Android (SL4A) app r6+
>>> * Python-mini v2.7+ under OpenWRT 12.09+
>>>
>>> ... to test the following Commotion implementations,
>>> respectively (where each is enabled by the user, not by the
>>> Python script):
>>>
>>> * commotion-mesh-applet
>>> * Mesh Tether app
>>> * Commotion-OpenWRT DR1, with *python*, *python-json*, and
>>> *olsrd-mod-jsoninfo* modules installed
>>>
>>> Questions about how to proceed:
>>>
>>> * Whether to proceed with Python-based testing framework? I
>>> tried to heavily leverage the cross-platform compatibility,
>>> but is it worthwhile?
>>>
>>> * The Python for Android implementation chosen, Py4A, is only
>>> Python v2.6. Is having Python v2.7 on Android worth
>>> possibly compiling it into a custom APK?
>>> * Many of the desired test vectors, e.g. throughput testing,
>>> require the test run simultaneously on at least 2 nodes.
>>> It's pretty easy to write a crude server in Python to
>>> function as one half of a throughput test, but does the
>>> complexity of running different Python scripts on different
>>> nodes simultaneously become unreasonable
>>>
>> Not at all, I think that it is necessary to have different nodes
>> running tests simultaneously. In most of the openWRT testing
>> situations we run currently, we have client devices running
>> simultaneously to elucidate information about the mesh nodes.
>>>
>>> * The python-mini module for OpenWRT chews up 1.5Mbytes of
>>> flash, and it doesn't include unittest by default. This
>>> unfortunately appears to be /too much already/; it wouldn't
>>> fit on a Nanostation flash w/o removing lots of stuff.
>>> Maybe this size could be trimmed down by modifying the
>>> python-mini package's Makefile, or by distributing test
>>> scripts in bytecode form to OpenWRT nodes.
>>>
>> I think that loading python on to openWRT nodes is impractical.
>> Having client devices running tests over the mesh with possible
>> on-node analytics running for later collection will give us a
>> good idea of QOS without having to load down the OpenWRT nodes,
>> and as such, change their performance. We will not get the same
>> level of detail as the android or desktop nodes, but it is really
>> the best we can do while maintaining an accurate image of the
>> nodes performance.
>>
>>
>>> *
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 15, 2013 at 10:14 PM, Andrew Reynolds
>>> <andrew at opentechinstitute.org
>>> <mailto:andrew at opentechinstitute.org>> wrote:
>>>
>>> It sounds reasonable. Could you sketch out what you have in
>>> mind? How
>>> much of the network setup were you thinking of building into
>>> the test
>>> framework vs. simply triggering and testing?
>>>
>>> -andrew
>>>
>>> On 03/15/2013 02:48 PM, Ben West wrote:
>>> > Hi All,
>>> >
>>> > In lieu of recent progress towards getting Commotion to a
>>> working state on
>>> > Android, Ubuntu/Debian. and now possibly OSX, what
>>> thoughts about building
>>> > a simple and (to whatever degree feasible) cross-platform
>>> testing framework
>>> > in Python?
>>> >
>>> > The general idea is that python scripts could be used to
>>> start hitting the
>>> > test vectors listed here (note the server appears to be
>>> really slow):
>>> >
>>> >
>>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Mesh-Routing-Tech-Evaluations
>>> >
>>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Testbed-Requirements-based-on-test-suite-defined-above
>>> >
>>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Release-Candidate-Test-Regimen
>>> >
>>> > That is, assuming Ubuntu/Debian/OSX's python support as a
>>> starting point,
>>> > could these lighweight python implementations allow for
>>> some unified test
>>> > scripts across platforms?
>>> >
>>> > http://qpython.com/ (for Android)
>>> >
>>> https://dev.openwrt.org/browser/packages/lang/python/Makefile (for
>>> OpenWRT,
>>> > to be compiled as module)
>>> >
>>> > Has anyone on the list had good experience with these Python
>>> > implementations?
>>> >
>>> > My original thought for such testing scripts was to do
>>> them in shell
>>> > scripting, but I'm guessing Python would be easier and
>>> more powerful.
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Commotion-dev mailing list
>>> > Commotion-dev at lists.chambana.net
>>> <mailto:Commotion-dev at lists.chambana.net>
>>> > https://lists.chambana.net/mailman/listinfo/commotion-dev
>>> >
>>>
>>>
>>>
>>> _______________________________________________
>>> Commotion-dev mailing list
>>> Commotion-dev at lists.chambana.net
>>> <mailto:Commotion-dev at lists.chambana.net>
>>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>>>
>>>
>>>
>>>
>>> --
>>> Ben West
>>> http://gowasabi.net <http://gowasabi.net/>
>>> ben at gowasabi.net <mailto:ben at gowasabi.net>
>>> 314-246-9434 <tel:314-246-9434>
>>>
>>>
>>> _______________________________________________
>>> Commotion-dev mailing list
>>> Commotion-dev at lists.chambana.net <mailto:Commotion-dev at lists.chambana.net>
>>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>>
>>
>> _______________________________________________
>> Commotion-dev mailing list
>> Commotion-dev at lists.chambana.net
>> <mailto:Commotion-dev at lists.chambana.net>
>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>>
>>
>>
>>
>> --
>> Ben West
>> http://gowasabi.net <http://gowasabi.net/>
>> ben at gowasabi.net <mailto:ben at gowasabi.net>
>> 314-246-9434
>> _______________________________________________
>> Commotion-dev mailing list
>> Commotion-dev at lists.chambana.net
>> <mailto:Commotion-dev at lists.chambana.net>
>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
>
> _______________________________________________
> Commotion-dev mailing list
> Commotion-dev at lists.chambana.net
> https://lists.chambana.net/mailman/listinfo/commotion-dev
--
Dan Staples
Open Technology Institute
https://commotionwireless.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chambana.net/pipermail/commotion-dev/attachments/20130424/54d8a5e3/attachment-0001.html>
More information about the Commotion-dev
mailing list