[Commotion-dev] Python-based testing framework across Ubuntu/Debian and Android (and maybe OSX)?
Josh Steiner
josh at vitriolix.com
Wed Apr 24 23:02:04 UTC 2013
I'll also chime in late (just landed in dc). My plan so far is to develop
the Windows client in python sharing as much code as possible with the
linux client. I'll also throw my backing behind python testing as I'll be
doing a certain amount of that as I work on the client, trying to be as TDD
as possible.
I also have no real kua knowhow other than haing read over some tutorials.
Josh
On Apr 24, 2013 5:58 PM, "Hans of Guardian" <hans at guardianproject.info>
wrote:
>
> Part of what makes python so productive is the vast amount of libraries
> available. If you can find Lua libs for everything, then that would be a
> starting place. From my personal perspective, I've never done any Lua, so
> I'd be unlikely to contribute much to Lua-based frameworks.
>
> .hc
>
> On Apr 24, 2013, at 5:48 PM, Dan Staples wrote:
>
> What about porting some of this to Lua, since OpenWRT nodes already have
> that interpreter? Would a lot be lost in the translation?
>
> Also, we can look at the tests run at the latest BattleMesh last week. I
> believe Seamus knows the repo where those are...I can't find them at the
> moment.
>
> On 04/24/2013 05:17 PM, Hans of Guardian wrote:
>
>
> Sorry to be late in weighing in this thread. I'm on the train to DC and
> diving back into a Commotion state of mind.
>
> I like having python test framework a lot, even if it doesn't run on all
> OpenWRT devices. To cover OpenWRT, you could set up OpenWRT on any device
> that has a lot of disk space, then run all the python stuff there. That
> would then give you a place to run the full test suite on OpenWRT. Then on
> the Ubuquiti hardware could be handled via some of the other ideas here,
> like shell scripts, or remote scraping via JSON, etc.
>
> Using python is just so vastly more productive that this approach will
> definitely be less work even though there would have to be some special
> cases for certain OpenWRT devices
>
> This reminds me, would this also then run in the VMs that you have been
> setting up? I think having some random box somewhere running like 20 or
> more various VMs with olsrd on them could be very nice as an automated
> testbed. This would not test the real world wireless conditions, but it
> would provide a way to run automated tests on olsrd meshes in a way that
> could be plugged into Jenkins builds. This kind of testbed would have
> caught the commit that broke meshing on ARM and amd64 that showed up in the
> more aggressive optimization of Ubuntu/quantal's compiler.
>
> .hc
>
> On Apr 2, 2013, at 10:31 AM, Ben West wrote:
>
> Hi Seamus and Dan,
>
> Replying about having Python on (and off) OpenWRT:
>
> The cross-platform nature seems like it would be very helpful to maintain
> a single collection of tests vectors to apply as Commotion finds its way
> onto more platforms (Windows, OSX). But, the obvious tradeoff here is the
> size of the Python interpreter.
>
> Also, the test scripts that query a node's OLSR instance via jsoninfo
> could also run *on the client*, rather than directly on the node. The
> node would just need to open up the jsoninfo port 9090 on its firewall
> behind the public AP interface (or have the client use its private AP). I
> can tweak the Python scripts (tho very little needed) to accommodate this
> option.
>
> Besides that, I think I may be able to tweak the Python Makefile
> provided with OpenWRT, so that the compiled package is much smaller than
> its current 1.5MByte footprint. The Python v2.6 implementation bundled
> into Py4A (which runs all these tests fine) is *only 150kB*, by
> comparison. Still, admitted that installing another language on what is
> already disk- and RAM-bound devices is less than ideal.
>
> Finally, sorry for using the term 'bytecode.' That's not really what we
> need; compiled Python bytecode is neither smaller than source nor
> stand-alone executable. What I meant is maybe compile the Python testsuite
> into a single standalone executable using a tool like these:
> http://cx-freeze.sourceforge.net/
> http://www.pyinstaller.org/
>
> On Tue, Apr 2, 2013 at 8:58 AM, seamus tuohy <s2e at opentechinstitute.org>wrote:
>
>> Hey,
>>
>>
>> On 04/02/2013 04:20 AM, Ben West wrote:
>>
>> Hi All,
>>
>> I've created a repo of a few basic tests implemented as unittests in
>> Python 2.6+. This repo contains just the Python scripts themselves, so
>> that it may be included later as a submodule into other repos containing
>> platform-specific packaging stuff.
>> https://github.com/westbywest/commotion-tests-core
>>
>> Due please note I expect this repo to change, and even its location to
>> maybe switch to OTI's github account at some point soon.
>>
>> To serve as hopefully demonstrative examples of the test coverage
>> possible, the scripts I just checked in test for the following:
>>
>> - ping localhost
>> - ping gateway IP (either default gateway or the one assigned by
>> OLSRd smartgateway)
>> - ping Google DNS (aka do we have Internet?)
>> - presence of an active olsrd process
>> - olsrd responds to jsoninfo requests
>> - link quality of the next hop is above a specified threshold
>>
>> These scripts should run under the following Python versions /
>> platforms:
>>
>> - Python v2.7+ under Debian/Ubuntu
>> - Python for Android (Py4A) app r5+ / Scripting Layer for Android
>> (SL4A) app r6+
>> - Python-mini v2.7+ under OpenWRT 12.09+
>>
>> ... to test the following Commotion implementations, respectively
>> (where each is enabled by the user, not by the Python script):
>>
>> - commotion-mesh-applet
>> - Mesh Tether app
>> - Commotion-OpenWRT DR1, with *python*, *python-json*, and *
>> olsrd-mod-jsoninfo* modules installed
>>
>> Questions about how to proceed:
>>
>> - Whether to proceed with Python-based testing framework? I tried to
>> heavily leverage the cross-platform compatibility, but is it worthwhile?
>>
>>
>> - The Python for Android implementation chosen, Py4A, is only Python
>> v2.6. Is having Python v2.7 on Android worth possibly compiling it into a
>> custom APK?
>> - Many of the desired test vectors, e.g. throughput testing, require
>> the test run simultaneously on at least 2 nodes. It's pretty easy to write
>> a crude server in Python to function as one half of a throughput test, but
>> does the complexity of running different Python scripts on different nodes
>> simultaneously become unreasonable
>>
>> Not at all, I think that it is necessary to have different nodes
>> running tests simultaneously. In most of the openWRT testing situations we
>> run currently, we have client devices running simultaneously to elucidate
>> information about the mesh nodes.
>>
>>
>> - The python-mini module for OpenWRT chews up 1.5Mbytes of flash, and
>> it doesn't include unittest by default. This unfortunately appears to be
>> *too much already*; it wouldn't fit on a Nanostation flash w/o
>> removing lots of stuff. Maybe this size could be trimmed down by modifying
>> the python-mini package's Makefile, or by distributing test scripts in
>> bytecode form to OpenWRT nodes.
>>
>> I think that loading python on to openWRT nodes is impractical. Having
>> client devices running tests over the mesh with possible on-node analytics
>> running for later collection will give us a good idea of QOS without having
>> to load down the OpenWRT nodes, and as such, change their performance. We
>> will not get the same level of detail as the android or desktop nodes, but
>> it is really the best we can do while maintaining an accurate image of the
>> nodes performance.
>>
>>
>>
>> -
>>
>>
>>
>> On Fri, Mar 15, 2013 at 10:14 PM, Andrew Reynolds <
>> andrew at opentechinstitute.org> wrote:
>>
>>> It sounds reasonable. Could you sketch out what you have in mind? How
>>> much of the network setup were you thinking of building into the test
>>> framework vs. simply triggering and testing?
>>>
>>> -andrew
>>>
>>> On 03/15/2013 02:48 PM, Ben West wrote:
>>> > Hi All,
>>> >
>>> > In lieu of recent progress towards getting Commotion to a working
>>> state on
>>> > Android, Ubuntu/Debian. and now possibly OSX, what thoughts about
>>> building
>>> > a simple and (to whatever degree feasible) cross-platform testing
>>> framework
>>> > in Python?
>>> >
>>> > The general idea is that python scripts could be used to start hitting
>>> the
>>> > test vectors listed here (note the server appears to be really slow):
>>> >
>>> >
>>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Mesh-Routing-Tech-Evaluations
>>> >
>>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Testbed-Requirements-based-on-test-suite-defined-above
>>> >
>>> https://code.commotionwireless.net/projects/commotion/wiki/Testing#Release-Candidate-Test-Regimen
>>> >
>>> > That is, assuming Ubuntu/Debian/OSX's python support as a starting
>>> point,
>>> > could these lighweight python implementations allow for some unified
>>> test
>>> > scripts across platforms?
>>> >
>>> > http://qpython.com/ (for Android)
>>> > https://dev.openwrt.org/browser/packages/lang/python/Makefile (for
>>> OpenWRT,
>>> > to be compiled as module)
>>> >
>>> > Has anyone on the list had good experience with these Python
>>> > implementations?
>>> >
>>> > My original thought for such testing scripts was to do them in shell
>>> > scripting, but I'm guessing Python would be easier and more powerful.
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Commotion-dev mailing list
>>> > Commotion-dev at lists.chambana.net
>>> > https://lists.chambana.net/mailman/listinfo/commotion-dev
>>> >
>>>
>>>
>>>
>>> _______________________________________________
>>> Commotion-dev mailing list
>>> Commotion-dev at lists.chambana.net
>>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>>>
>>>
>>
>>
>> --
>> Ben West
>> http://gowasabi.net
>> ben at gowasabi.net
>> 314-246-9434
>>
>>
>> _______________________________________________
>> Commotion-dev mailing listCommotion-dev at lists.chambana.nethttps://lists.chambana.net/mailman/listinfo/commotion-dev
>>
>>
>>
>> _______________________________________________
>> Commotion-dev mailing list
>> Commotion-dev at lists.chambana.net
>> https://lists.chambana.net/mailman/listinfo/commotion-dev
>>
>>
>
>
> --
> Ben West
> http://gowasabi.net
> ben at gowasabi.net
> 314-246-9434
> _______________________________________________
> Commotion-dev mailing list
> Commotion-dev at lists.chambana.net
> https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
>
>
> _______________________________________________
> Commotion-dev mailing listCommotion-dev at lists.chambana.nethttps://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
> --
> Dan Staples
>
> Open Technology Institutehttps://commotionwireless.net
>
> _______________________________________________
> Commotion-dev mailing list
> Commotion-dev at lists.chambana.net
> https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
>
> _______________________________________________
> Commotion-dev mailing list
> Commotion-dev at lists.chambana.net
> https://lists.chambana.net/mailman/listinfo/commotion-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.chambana.net/pipermail/commotion-dev/attachments/20130424/9d5c9717/attachment-0001.html>
More information about the Commotion-dev
mailing list