<div dir="ltr"><div><div>Hi Dan,<br><br></div>Thanks for offering more detail, especially that you see the nodes spontaneously reboot rather than simple have services crash.<br><br></div>I would again point out that the Picostations will have a finite limit for simultaneous clients. 15 to 20 clients is quite a few, each client requiring a portion of available of RAM. It may be a single Picostation is not going to be able to sustain all of them.<br>
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 6, 2013 at 10:58 AM, Dan Staples <span dir="ltr"><<a href="mailto:danstaples@opentechinstitute.org" target="_blank">danstaples@opentechinstitute.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Regarding logging, I'm not sure that will work well since the nodes are<br>
spontaneously rebooting themselves (due to OOM conditions), not the user<br>
rebooting them. What we're going to try to do is attach a serial console<br>
(thanks Will!) and try to slam the router with simultaneous users and<br>
traffic.<br>
<br>
Also, I don't think Dan is hosting local apps on the router itself<br>
(correct me if I'm wrong), but just advertising them using the Commotion<br>
apps portal. And that's just takes a little space for the Avahi service<br>
file...so hopefully that's not a problem.<br>
<br>
We'll certainly report what we find with our stress testing.<br>
<br>
Dan<br>
<div class="im"><br>
On 11/06/2013 10:37 AM, Ben West wrote:<br>
> I am also seeing sporadic memory consumption issues operating mesh nodes<br>
> running AA r38347 in WasabiNet on Nanostation Loco M2.<br>
><br>
> That is, using the same ath9k wifi driver and same underlying OS, but<br>
> without the Commotion-specific tools like commotiond and servald. I<br>
> will see nodes boot up with ~26Mbytes memory usage and then gradually<br>
> increase over the next few days until sporadic nodes start crashing with<br>
> page allocation failures (aka memory exhausted). This all is happening<br>
> despite having 3Mbytes of compressed swap space allocated. When I am<br>
> able to log into crashed nodes to inspect, I will occasionally find the<br>
</div>> current memory usage to be /less/ than the average observed on bootup,<br>
<div class="im">> along with ~500Kbytes sitting in swap.<br>
><br>
> This seems to suggest something is very sporadically allocating itself a<br>
> large chunk (multiple MBytes), but not residing in memory as such, and<br>
> causing other processes to crash in consequence. I do use the<br>
> coovachilli captive portal in WasabiNet, which could be a culprit and<br>
> thus unrelated to Commotion, but there could also be an underlying<br>
> memory leak in the kernel or wifi driver.<br>
><br>
> What are thoughts for having crashed nodes try to collect a debug report<br>
> about themselves when a crash condition is detected (e.g. no Internet<br>
> access, "page allocation failure" detected in syslog), and then write<br>
> that report to flash somewhere before the node get rebooted by its<br>
> frustrated user?<br>
><br>
> Besides that, do note that nodes with only 32MBytes of RAM, like UBNT<br>
> Picostations, are going to have difficulties hosting local apps for many<br>
> users. If Dan Hasting would be able to use an alternate device with<br>
> 64Mbytes+ RAM, e.g. a UBNT Rocket, Unifi, or even an indoor TP-Link<br>
> router (all of which should be able to run Commotion-OpenWRT), that may<br>
> be a viable workaround in cause chasing down memory leaks becomes too<br>
> ornery.<br>
><br>
><br>
><br>
> On Wed, Nov 6, 2013 at 8:54 AM, Dan Staples<br>
> <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>> wrote:<br>
><br>
> +commotion-dev<br>
><br>
> If your nodes are crashing w/ 15-20 clients, while both serval and<br>
> commotion-splash are disabled, that is very worrisome!<br>
><br>
> I propose to the Commotion dev team that we urgently need to come up<br>
> with a way to simulate network load, so we can identify and fix the<br>
> causes of these types of crashes. Does anyone have ideas or experiences<br>
> with this? Perhaps we can take the technical discussion over to the<br>
> commotion-dev list only.<br>
><br>
> And just an update for you Dan, earlier this week I found and fixed a<br>
> significant memory leak in Serval...not sure how much that will affect<br>
> the instability we've seen, but we'll soon know with some testing. The<br>
> fix will make its way into the nightly builds probably by the end of the<br>
> week.<br>
><br>
> As long as the rest of your network is DR1 or newer, the nightly builds<br>
> should be compatible.<br>
><br>
> Dan<br>
><br>
> On 11/06/2013 04:07 AM, Dan Hastings wrote:<br>
> > I was just checking to see if their had been any progress made on the<br>
> > nightly builds with fixing the memory overload causing the nodes to<br>
> > crash. To try and prevent my node from crashing I disabled serval and<br>
> > the splash page. However, whenever I have 15 to 20 students login to a<br>
> > local app at the start of class my node crashes instantly. I'm<br>
> wondering<br>
> > if upgrading to the latest nightly build might fix this issue. Lastly,<br>
> > if I upgrade to the latest nightly build will it still work with the<br>
> > other nodes that do not have the latest build or do I have to or is it<br>
> > recommend that I upgrade all of the other nodes to latest build as<br>
> > well? Thanks for all the hard work. Commotion is otherwise working<br>
> > wonders over here in the horn.<br>
> ><br>
> > Dan<br>
> ><br>
> > _______________________________________________<br>
> > Commotion-discuss mailing list<br>
> > <a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
</div></div>> <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>><br>
<div class="im">> > <a href="https://lists.chambana.net/mailman/listinfo/commotion-discuss" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-discuss</a><br>
> ><br>
><br>
> --<br>
> Dan Staples<br>
><br>
> Open Technology Institute<br>
> <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
> OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
> Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
> _______________________________________________<br>
> Commotion-dev mailing list<br>
> <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
</div>> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
<div class="im">> <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Ben West<br>
> <a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
</div>> <a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>><br>
> <a href="tel:314-246-9434" value="+13142469434">314-246-9434</a><br>
<div class="HOEnZb"><div class="h5"><br>
--<br>
Dan Staples<br>
<br>
Open Technology Institute<br>
<a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
_______________________________________________<br>
Commotion-dev mailing list<br>
<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
<a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div>Ben West</div><div><a href="mailto:me@benwest.name" target="_blank">me@benwest.name</a></div>
</div>