Hi Will,<br><br>Glad that an apparent bug with dhcp was caught.<br><br>Please see my response about IBSS-RSN issues below <span style="color:rgb(0,102,0)">in green</span>.<br><br><div class="gmail_quote">On Wed, Apr 24, 2013 at 5:49 PM, Will Hawkins <span dir="ltr"><<a href="mailto:hawkinsw@opentechinstitute.org" target="_blank">hawkinsw@opentechinstitute.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">A few notes to consider after some testing today with LTS and at the office:<br>
<br>
- Collectd "stalled" one of the nodes at LTS (where collectd is still<br>
enabled)<br>
<br>
- Stations seem to lose their mind w.r.t IBSS RSN and authorization. I'm<br>
watching debugging output from wpa_supplicant on one of the nodes in the<br>
office to see if I can determine the problem. Of course, it's working<br>
great now :-)<br>
<br></blockquote><div><span style="color:rgb(0,102,0)"><br>I've observed a definite issue with nodes not reliably 'authorizing' themselves when joining an IBSS-RSN adhoc network, at least since r33202 or so. No idea on a cause, besides IBSS-RSN simply being buggy.<br>
<br>Besides doing something crude like putting 'sleep 30 ; wifi restart' in the file /etc/rc.local, I have also written a slightly less crude hotplug.d script that attempts to restart the wifi interface, if the node finds itself 'authorized' but not 'authenticated' on the adhoc network. I've attached that script to this email, and </span><span style="color:rgb(0,102,0)"><span style="color:rgb(0,102,0)">you can save it on a node as</span><span style="color:rgb(0,102,0)"><span style="color:rgb(0,102,0)"> <b>/etc/hotplug.d/firewall/20_mesh_auth_check</b>.<br>
</span></span> <br>Please note this is not a 100% effective solution, as the problem is not just with a newly powered on node not consistently authorizing itself, but <i>also</i> with some of the existing nodes in the adhoc network not consistently authorizing the new node on their end too. That is, I've seen instances where node A lists node B as both 'authenticated' and 'authorized', but where node B lists node A as 'authenticated' and <i>not</i> 'authorized.' Oy.<br>
<br>So, when a new node joins an RSN-encrypted adhoc network, it looks like the following steps must happen to ensure all nodes are authorized:<br><br>A. Newly powered-up node inspects output of 'iw wlan0 station dump' looking for entries where a remote node is 'authenticated' but <i>not</i> 'authorized,' and if so, restart the wifi and retry test. Ideally, the node would repeat this process X times until giving up. The script I'm attaching tries to do this, albeit without the option to give up after X times.</span><span style="color:rgb(0,102,0)"><span style="color:rgb(0,102,0)"><br>
</span> <br>B. All existing nodes periodically via cronjob check their own local output of 'iw wlan0 station dump', looking for new entries where a new remote node is 'authenticated' but <i>not</i> 'authorized,' If such an entry is found, restart the <i>local</i> node's wifi and retry test.<br>
<br>Having both of these steps occur simultaneously on all nodes clearly could lead to lots of ugly race conditions, so it's not ideal. Likewise, its even less ideal for a particular node with active clients to restart its wifi just because a new node powered on, but didn't get successfully authorized. Maybe an alternate way to achieve step B is just to have the newly powered-up node repeatedly restart its wifi <i>until</i> it can successfully ping all other nodes that appear on the adhoc network, tho would be tedious and make startup very slow.</span><br>
<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- luci_splash got itself into a nice "wedge" on one of the LTS nodes. I<br>
am going to do my best to get it unstuck. If we continue to see the<br>
problem, we'll have to take a hard look at pushing the splash rewrite to<br>
a higher priority.<br>
<br>
Will<br>
<div><br>
On 04/24/2013 04:26 PM, Ben West wrote:<br>
> Hi Will,<br>
><br>
> I just checked config on a node again (running Attitude Adjustment circa<br>
> r35xxx), and I found these lines in /etc/crontabs/root which had been<br>
> commented out:<br>
><br>
> #* * * * * /usr/sbin/ff_olsr_test_gw.sh<br>
> #*/5 * * * * /usr/sbin/ff_olsr_watchdog<br>
><br>
> So, be on the lookout for ff packages that deploy these scripts,<br>
</div>> although unfortunately it's not clear /which/ package includes these<br>
<div>> particular files. Maybe freifunk-common?<br>
><br>
> On Wed, Apr 24, 2013 at 2:28 PM, Will Hawkins<br>
</div>> <<a href="mailto:hawkinsw@opentechinstitute.org" target="_blank">hawkinsw@opentechinstitute.org</a> <mailto:<a href="mailto:hawkinsw@opentechinstitute.org" target="_blank">hawkinsw@opentechinstitute.org</a>>><br>
<div><div>> wrote:<br>
><br>
> Thanks for your response, Ben. We just recompiled an image w/o most of<br>
> the ff software. We are now testing that image to see if things are any<br>
> better. We will definitely note how ff-watchdog may be useful and how<br>
> ff-gw-check is the likely culprit ;-)<br>
><br>
> Will<br>
><br>
> On 04/24/2013 02:17 PM, Ben West wrote:<br>
> > The Freifunk watchdog package is actually a rather handy package,<br>
> since<br>
> > it will monitor any process you want (via periodic cronjob) and<br>
> restart<br>
> > that service if the active process disappears (aka crashes). To my<br>
> > knowledge, it doesn't directly start/stop any network interfaces.<br>
> But,<br>
> > ff-watchdog does need to be configured to monitor the processes<br>
> you care<br>
> > about, and to not conflict with any other watchdog-style task. That<br>
> > conflict may be indirectly causing interfaces to go down or even olsrd<br>
> > to stop in absence of a needed interface.<br>
> ><br>
> > Its config file is /etc/config/freifunk-watchfog, and here is an<br>
> example<br>
> > config I've used (for node using coovachilli):<br>
> ><br>
> > config process<br>
> > option process 'dropbear'<br>
> > option initscript '/etc/init.d/dropbear'<br>
> ><br>
> > config process<br>
> > option process 'crond'<br>
> > option initscript '/etc/init.d/cron'<br>
> ><br>
> > config process<br>
> > option process 'olsrd'<br>
> > option initscript '/etc/init.d/olsrd'<br>
> ><br>
> > config process<br>
> > option process 'chilli'<br>
> > option initscript '/etc/init.d/coovachilli'<br>
> ><br>
> > Are you sure you weren't having problems with the ff-gw-check package<br>
> > instead? I.e. un-installed that package at the same time as<br>
> > un-stinalling ff-watchdog? I think the gw-check package /will muck/<br>
> > with default routes and possibly also restart active network<br>
> interfaces<br>
> > if it can't get a successful ping to <a href="http://freifunk.net" target="_blank">freifunk.net</a><br>
</div></div>> <<a href="http://freifunk.net" target="_blank">http://freifunk.net</a>> <<a href="http://freifunk.net" target="_blank">http://freifunk.net</a>><br>
<div>> > or something.<br>
> ><br>
> > On Wed, Apr 24, 2013 at 8:07 AM, Dan Staples<br>
> > <<a href="mailto:danstaples@opentechinstitute.org" target="_blank">danstaples@opentechinstitute.org</a><br>
> <mailto:<a href="mailto:danstaples@opentechinstitute.org" target="_blank">danstaples@opentechinstitute.org</a>><br>
</div>> > <mailto:<a href="mailto:danstaples@opentechinstitute.org" target="_blank">danstaples@opentechinstitute.org</a><br>
<div><div>> <mailto:<a href="mailto:danstaples@opentechinstitute.org" target="_blank">danstaples@opentechinstitute.org</a>>>> wrote:<br>
> ><br>
> > Moving this discussion to commotion-dev...<br>
> ><br>
> > When I was previously setting the wireless interfaces to use<br>
> channel 9<br>
> > instead of channel 5, the freifunk watchdog would routinely<br>
> bring down<br>
> > the wireless interfaces. And I have no idea why. The only way<br>
> I got it<br>
> > to work was uninstalling ff-watchdog. So see if that may be a<br>
> reason why<br>
> > wireless interfaces are unavailable...there should be a note<br>
> about it in<br>
> > logread.<br>
> ><br>
> > I've also noticed that something is killing olsrd on DR1<br>
> nodes, without<br>
> > any clue in the log. The routing table will still have stale<br>
> routes in<br>
> > it, indicating that olsrd isn't exiting cleanly. I wonder if<br>
> it's being<br>
> > killed by the out-of-memory watchdog. When I was<br>
> troubleshooting this<br>
> > before, I wrote a quick script that ran as a cronjob every<br>
> minute, and<br>
> > it would pgrep olsrd. If olsrd was running, it would redirect<br>
> the output<br>
> > of top into ~/top.out. If olsrd wasn't running, it would move<br>
> the last<br>
> > ~/top.out as well as logread into a separate directory. That way,<br>
> > whenever olsrd was killed, there would be a record of top the<br>
> minute<br>
> > before it crashed, as well as the log. Would this be useful for<br>
> > troubleshooting the LTS nodes?<br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Ben West<br>
> > <a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
> > <a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a>><br>
</div></div>> <mailto:<a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a>>><br>
> > <a href="tel:314-246-9434" value="+13142469434" target="_blank">314-246-9434</a> <tel:<a href="tel:314-246-9434" value="+13142469434" target="_blank">314-246-9434</a>> <tel:<a href="tel:314-246-9434" value="+13142469434" target="_blank">314-246-9434</a> <tel:<a href="tel:314-246-9434" value="+13142469434" target="_blank">314-246-9434</a>>><br>
<div>> ><br>
> ><br>
> > _______________________________________________<br>
> > Commotion-dev mailing list<br>
> > <a href="mailto:Commotion-dev@lists.chambana.net" target="_blank">Commotion-dev@lists.chambana.net</a><br>
</div>> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net" target="_blank">Commotion-dev@lists.chambana.net</a>><br>
<div>> > <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
> ><br>
> _______________________________________________<br>
> Commotion-dev mailing list<br>
> <a href="mailto:Commotion-dev@lists.chambana.net" target="_blank">Commotion-dev@lists.chambana.net</a><br>
</div>> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net" target="_blank">Commotion-dev@lists.chambana.net</a>><br>
<div>> <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Ben West<br>
> <a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
</div>> <a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a>><br>
> <a href="tel:314-246-9434" value="+13142469434" target="_blank">314-246-9434</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Ben West<div><a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br><a href="mailto:ben@gowasabi.net" target="_blank">ben@gowasabi.net</a><br><a href="tel:314-246-9434" value="+13142469434" target="_blank">314-246-9434</a><br>
</div>