[Commotion-dev] [OTI-Tech] LTS Testing Update

Wed Apr 24 13:07:19 UTC 2013

Moving this discussion to commotion-dev...

When I was previously setting the wireless interfaces to use channel 9
instead of channel 5, the freifunk watchdog would routinely bring down
the wireless interfaces. And I have no idea why. The only way I got it
to work was uninstalling ff-watchdog. So see if that may be a reason why
wireless interfaces are unavailable...there should be a note about it in
logread.

I've also noticed that something is killing olsrd on DR1 nodes, without
any clue in the log. The routing table will still have stale routes in
it, indicating that olsrd isn't exiting cleanly. I wonder if it's being
killed by the out-of-memory watchdog. When I was troubleshooting this
before, I wrote a quick script that ran as a cronjob every minute, and
it would pgrep olsrd. If olsrd was running, it would redirect the output
of top into ~/top.out. If olsrd wasn't running, it would move the last
~/top.out as well as logread into a separate directory. That way,
whenever olsrd was killed, there would be a record of top the minute
before it crashed, as well as the log. Would this be useful for
troubleshooting the LTS nodes?

On Tue 23 Apr 2013 08:41:37 PM EDT, Andrew Reynolds wrote:
>
> You've probably already looked at this, but are there any cron jobs or
> watchdog scripts buried in openwrt that might be restarting networking
> without using commotiond, skipping the commotion profiles in the process?
>
> Also, is there any reason this troubleshooting discussion shouldn't be
> moved to commotion-dev?
>
> -andrew
>
> On 04/23/2013 07:48 PM, Will Hawkins wrote:
>>
>> Some firewall updates:
>>
>> It looks like adding a forwarding from wan to wan (for when the plug
>> gets an IP address from the dhcp server) seems to make a big difference.
>>
>> Some wireless updates:
>>
>> One of the node's wireless completely freaked out and lost the ability
>> to communicate securely with the other nodes. Station dump output showed
>> many stations but none of them were authorized. It took a reboot to get
>> it back into a good state (/etc/init.d/network restart) did not work.
>>
>> Now that I have some of the firewall issues straightened out and some of
>> the wireless straightened out, nodes are meshing.
>>
>> Will
>>
>> On 04/23/2013 07:31 PM, Will Hawkins wrote:
>>>
>>> Things are going from bad to worse to just plain crazy.
>>>
>>> Dan and I struggled around with the firewall and other settings today.
>>> We made some progress but, just as quickly, I fell backward into another
>>> mess.
>>>
>>> Here are a few thoughts:
>>>
>>> - It seems that consistently the wireless interface(s) are not around.
>>> This is consistent with what Preston was saying about his deployment in
>>> Detroit this afternoon. This happens after network restarts especially.
>>>
>>> - The IP addressing bug is still around.
>>>
>>> - The firewall configuration does not seem consistent. I find this
>>> especially curious and cannot seem to figure out what is going on! Dan
>>> thinks it is all in my head.
>>>
>>> - After running the QS, olsrd really wants to mesh over the plug
>>> interface. This is not an unreasonable thing to do, but it does not
>>> really follow semantically from what we are doing in the QS path.
>>>
>>> I will send more as I have it. It seems almost necessary to take a trip
>>> to LTS to actually work through some of these issues.
>>>
>>> Will
>>>
>>> On 04/22/2013 12:38 PM, Josh King wrote:
>>>>
>>>> Those are the old, ar231x nanos, not the newer ar71xx M-series versions
>>>> we typically use. The M-series ones have 8MB of flash, though you're
>>>> right that the older ones seem to have less.
>>>>
>>>>
>>>>
>>>> Dan Staples <danstaples at opentechinstitute.org> wrote:
>>>>
>>>> The Ubiquiti site lists them as having 4MB flash & 16MB SDRAM, which I
>>>> was surprised to see. http://www.ubnt.com/nanostation
>>>>
>>>> On Mon 22 Apr 2013 12:33:51 PM EDT, Josh King wrote:
>>>>
>>>> Actually, I believe the nanostation has 8MB as well. I've flashed
>>>> picos with nano images before, when we got the very first ones,
>>>> and I
>>>> was able to boot them but the network did not function correctly
>>>> as I
>>>> recall.
>>>>
>>>>
>>>>
>>>> Dan Staples <danstaples at opentechinstitute.org> wrote:
>>>>
>>>> I would suspect it didn't work becase nanostations have half the
>>>> flash storage (4 MB) of the picostation. Speaking of which, we
>>>> should figure out if our nanostation images for DR1 work with
>>>> nanostations, since the images are about 6 MB...
>>>>
>>>> On 04/22/2013 11:58 AM, Preston Rhea wrote:
>>>>
>>>> Would be interesting to know just what the problems were with
>>>> flashing a nano image to a bullet-compatible device. Also
>>>> did you
>>>> use the web interface flashing method or the old-school
>>>> paperclip
>>>> method?
>>>>
>>>>
>>>> On Fri, Apr 19, 2013 at 4:54 PM, Will Hawkins
>>>> <hawkinsw at opentechinstitute.org
>>>> <mailto:hawkinsw at opentechinstitute.org>> wrote:
>>>>
>>>> Some success: It helps when you use the correct image. Bullet
>>>> != Nano.
>>>> Things are going *better* now. I will keep you posted as I
>>>> progress.
>>>>
>>>> Will
>>>>
>>>> On 04/19/2013 04:05 PM, Josh King wrote:
>>>>
>>>> Could it be a timeout issue? What if we extend the timeout
>>>>
>>>> on the LTS nodes?
>>>>
>>>>
>>>>
>>>> Will Hawkins <hawkinsw at opentechinstitute.org
>>>> <
>>>>
>>>> mailto:hawkinsw at opentechinstitute.org>> wrote:
>>>>
>>>> Good news and bad news.
>>>>
>>>> The good news is that I am able to get the nodes
>>>>
>>>> meshing at LTS. The bad
>>>>
>>>> news is that it's inconsistent. I am having particular
>>>>
>>>> trouble with the
>>>>
>>>> plug interfaces.
>>>>
>>>> At boot time, the nodes do NOT get ip addresses from
>>>>
>>>> the dhcp server. I
>>>>
>>>> can watch the DHCP server logs and see that the node
>>>>
>>>> requests an IP
>>>>
>>>> address and that the server responds with a lease
>>>>
>>>> offer. However, the
>>>>
>>>> node does not think that it gets a response. It
>>>>
>>>> proceeds to set its own
>>>>
>>>> IP address for the plug interface and goes forward by
>>>>
>>>> starting up its
>>>>
>>>> own DHCP server.
>>>>
>>>> This is inconsistent with the be havior that I am seeing
>>>>
>>>> from the test
>>>>
>>>> nodes in the office. When they are connected to the NAF
>>>>
>>>> network they
>>>>
>>>> receive their an IP address offer and work correctly.
>>>>
>>>> Does this possibly have to do with the way that I am
>>>>
>>>> running through QS?
>>>>
>>>> Are there a particular set of "clicks" that I am doing
>>>> that
>>>> cause
>>>> behavior that seems consistent with this?
>>>>
>>>> I am continuing to investigate.
>>>>
>>>> Will
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> OTI-Tech mailing list
>>>> OTI-Tech at lists.opentechinstitute.org
>>>> <
>>>>
>>>> mailto:OTI-Tech at lists.opentechinstitute.org>
>>>>
>>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>>
>>>>
>>>> --
>>>> Josh King
>>>> Lead Technologist
>>>> The Open Technology Institute
>>>> New America Foundation
>>>>
>>>> *Sent from a mobile device; please excuse brevity, typos.*
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> OTI-Tech mailing list
>>>> OTI-Tech at lists.opentechinstitute.org
>>>> <
>>>>
>>>> mailto:OTI-Tech at lists.opentechinstitute.org>
>>>>
>>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> OTI-Tech mailing list
>>>> OTI-Tech at lists.opentechinstitute.org
>>>> <mailto:OTI-Tech at lists.opentechinstitute.org>
>>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> OTI-Tech mailing list
>>>> OTI-Tech at lists.opentechinstitute.org
>>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>>
>>>>
>>>> --
>>>> Dan Staples
>>>>
>>>> Open Technology Institute
>>>> https://commotionwireless.net
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>> OTI-Tech mailing list
>>>> OTI-Tech at lists.opentechinstitute.org
>>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>>
>>>>
>>>> --
>>>> Josh King
>>>> Lead Technologist
>>>> The Open Technology Institute
>>>> New America Foundation
>>>>
>>>> *Sent from a mobile device; please excuse brevity, typos.*
>>>>
>>>>
>>>> --
>>>> Dan Staples
>>>>
>>>> Open Technology Institute
>>>> https://commotionwireless.net
>>>>
>>>>
>>>> --
>>>> Josh King
>>>> Lead Technologist
>>>> The Open Technology Institute
>>>> New America Foundation
>>>>
>>>> *Sent from a mobile device; please excuse brevity, typos.*
>>>>
>>>>
>>>> _______________________________________________
>>>> OTI-Tech mailing list
>>>> OTI-Tech at lists.opentechinstitute.org
>>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>>
>>>
>>> _______________________________________________
>>> OTI-Tech mailing list
>>> OTI-Tech at lists.opentechinstitute.org
>>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>>
>>
>> _______________________________________________
>> OTI-Tech mailing list
>> OTI-Tech at lists.opentechinstitute.org
>> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
>>
>
>
>
>
>
> _______________________________________________
> OTI-Tech mailing list
> OTI-Tech at lists.opentechinstitute.org
> https://lists.opentechinstitute.org/mailman/listinfo/oti-tech
> -- 
> Dan Staples
>
> Open Technology Institute
> https://commotionwireless.net