<div dir="ltr">I had exactly 18 students all simultaneously trying to connect to OwnCloud a local application hosted on our server not the routers at one time. I've had this happen a few random times in the past week or so but only once throughout the day almost always with the first class I have in the morning. The second class to follow had more students and we began the class repeating the same behavior of connecting to OwnCloud but I did not have any issues. <div>
<br></div><div>I'll ssh into the node tomorrow morning during my first class and see what commands are running if the node happens to crash. Also, I know I disabled the commotion-splash by checking "immediately authenticate" under Captive Portal but is there another way in command line to completely kill that process from running all the time? I just ran top on one of my nodes and I still see that no-dog splash is running. </div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 6, 2013 at 9:16 PM, <span dir="ltr"><<a href="mailto:commotion-discuss-request@lists.chambana.net" target="_blank">commotion-discuss-request@lists.chambana.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Commotion-discuss mailing list submissions to<br>
<a href="mailto:commotion-discuss@lists.chambana.net">commotion-discuss@lists.chambana.net</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.chambana.net/mailman/listinfo/commotion-discuss" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:commotion-discuss-request@lists.chambana.net">commotion-discuss-request@lists.chambana.net</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:commotion-discuss-owner@lists.chambana.net">commotion-discuss-owner@lists.chambana.net</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Commotion-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: [Commotion-dev] Memory Issues and Nightly Builds (Dan Staples)<br>
2. Re: [Commotion-dev] Memory Issues and Nightly Builds (Ryan Gerety)<br>
3. Re: [Commotion-dev] Memory Issues and Nightly Builds (Ben West)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 06 Nov 2013 12:54:10 -0500<br>
From: Dan Staples <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>><br>
To: <a href="mailto:me@benwest.name">me@benwest.name</a>, commotion-discuss<br>
<<a href="mailto:commotion-discuss@lists.chambana.net">commotion-discuss@lists.chambana.net</a>>, Commotion Development List<br>
<<a href="mailto:commotion-dev@lists.chambana.net">commotion-dev@lists.chambana.net</a>><br>
Subject: Re: [Commotion-discuss] [Commotion-dev] Memory Issues and<br>
Nightly Builds<br>
Message-ID: <<a href="mailto:527A8242.4040107@opentechinstitute.org">527A8242.4040107@opentechinstitute.org</a>><br>
Content-Type: text/plain; charset=UTF-8<br>
<br>
Since Dan Hastings has seen this happen with a lot of simultaneous<br>
clients and with high-memory components disabled, it sounds like that is<br>
likely the cause. Do you know exactly where that RAM is used for each<br>
connecting client?<br>
<br>
Dan, can you provide any more detailed info on exactly what was<br>
happening when you see the node crashing? How many simultaneous users,<br>
and what were they doing (viewing a webpage on the internet, or viewing<br>
the node's administrative web interface, etc)?<br>
<br>
On Wed 06 Nov 2013 12:40:06 PM EST, Ben West wrote:<br>
> Hi Dan,<br>
><br>
> Thanks for offering more detail, especially that you see the nodes<br>
> spontaneously reboot rather than simple have services crash.<br>
><br>
> I would again point out that the Picostations will have a finite limit<br>
> for simultaneous clients. 15 to 20 clients is quite a few, each<br>
> client requiring a portion of available of RAM. It may be a single<br>
> Picostation is not going to be able to sustain all of them.<br>
><br>
><br>
><br>
> On Wed, Nov 6, 2013 at 10:58 AM, Dan Staples<br>
> <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>> wrote:<br>
><br>
> Regarding logging, I'm not sure that will work well since the<br>
> nodes are<br>
> spontaneously rebooting themselves (due to OOM conditions), not<br>
> the user<br>
> rebooting them. What we're going to try to do is attach a serial<br>
> console<br>
> (thanks Will!) and try to slam the router with simultaneous users and<br>
> traffic.<br>
><br>
> Also, I don't think Dan is hosting local apps on the router itself<br>
> (correct me if I'm wrong), but just advertising them using the<br>
> Commotion<br>
> apps portal. And that's just takes a little space for the Avahi<br>
> service<br>
> file...so hopefully that's not a problem.<br>
><br>
> We'll certainly report what we find with our stress testing.<br>
><br>
> Dan<br>
><br>
> On 11/06/2013 10:37 AM, Ben West wrote:<br>
> > I am also seeing sporadic memory consumption issues operating<br>
> mesh nodes<br>
> > running AA r38347 in WasabiNet on Nanostation Loco M2.<br>
> ><br>
> > That is, using the same ath9k wifi driver and same underlying<br>
> OS, but<br>
> > without the Commotion-specific tools like commotiond and servald. I<br>
> > will see nodes boot up with ~26Mbytes memory usage and then<br>
> gradually<br>
> > increase over the next few days until sporadic nodes start<br>
> crashing with<br>
> > page allocation failures (aka memory exhausted). This all is<br>
> happening<br>
> > despite having 3Mbytes of compressed swap space allocated.<br>
> When I am<br>
> > able to log into crashed nodes to inspect, I will occasionally<br>
> find the<br>
> > current memory usage to be /less/ than the average observed on<br>
> bootup,<br>
> > along with ~500Kbytes sitting in swap.<br>
> ><br>
> > This seems to suggest something is very sporadically allocating<br>
> itself a<br>
> > large chunk (multiple MBytes), but not residing in memory as<br>
> such, and<br>
> > causing other processes to crash in consequence. I do use the<br>
> > coovachilli captive portal in WasabiNet, which could be a<br>
> culprit and<br>
> > thus unrelated to Commotion, but there could also be an underlying<br>
> > memory leak in the kernel or wifi driver.<br>
> ><br>
> > What are thoughts for having crashed nodes try to collect a<br>
> debug report<br>
> > about themselves when a crash condition is detected (e.g. no<br>
> Internet<br>
> > access, "page allocation failure" detected in syslog), and then<br>
> write<br>
> > that report to flash somewhere before the node get rebooted by its<br>
> > frustrated user?<br>
> ><br>
> > Besides that, do note that nodes with only 32MBytes of RAM, like<br>
> UBNT<br>
> > Picostations, are going to have difficulties hosting local apps<br>
> for many<br>
> > users. If Dan Hasting would be able to use an alternate device with<br>
> > 64Mbytes+ RAM, e.g. a UBNT Rocket, Unifi, or even an indoor TP-Link<br>
> > router (all of which should be able to run Commotion-OpenWRT),<br>
> that may<br>
> > be a viable workaround in cause chasing down memory leaks<br>
> becomes too<br>
> > ornery.<br>
> ><br>
> ><br>
> ><br>
> > On Wed, Nov 6, 2013 at 8:54 AM, Dan Staples<br>
> > <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>><br>
> > <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>>> wrote:<br>
> ><br>
> > +commotion-dev<br>
> ><br>
> > If your nodes are crashing w/ 15-20 clients, while both<br>
> serval and<br>
> > commotion-splash are disabled, that is very worrisome!<br>
> ><br>
> > I propose to the Commotion dev team that we urgently need to<br>
> come up<br>
> > with a way to simulate network load, so we can identify and<br>
> fix the<br>
> > causes of these types of crashes. Does anyone have ideas or<br>
> experiences<br>
> > with this? Perhaps we can take the technical discussion over<br>
> to the<br>
> > commotion-dev list only.<br>
> ><br>
> > And just an update for you Dan, earlier this week I found<br>
> and fixed a<br>
> > significant memory leak in Serval...not sure how much that<br>
> will affect<br>
> > the instability we've seen, but we'll soon know with some<br>
> testing. The<br>
> > fix will make its way into the nightly builds probably by<br>
> the end of the<br>
> > week.<br>
> ><br>
> > As long as the rest of your network is DR1 or newer, the<br>
> nightly builds<br>
> > should be compatible.<br>
> ><br>
> > Dan<br>
> ><br>
> > On 11/06/2013 04:07 AM, Dan Hastings wrote:<br>
> > > I was just checking to see if their had been any progress<br>
> made on the<br>
> > > nightly builds with fixing the memory overload causing the<br>
> nodes to<br>
> > > crash. To try and prevent my node from crashing I disabled<br>
> serval and<br>
> > > the splash page. However, whenever I have 15 to 20<br>
> students login to a<br>
> > > local app at the start of class my node crashes instantly. I'm<br>
> > wondering<br>
> > > if upgrading to the latest nightly build might fix this<br>
> issue. Lastly,<br>
> > > if I upgrade to the latest nightly build will it still<br>
> work with the<br>
> > > other nodes that do not have the latest build or do I have<br>
> to or is it<br>
> > > recommend that I upgrade all of the other nodes to latest<br>
> build as<br>
> > > well? Thanks for all the hard work. Commotion is<br>
> otherwise working<br>
> > > wonders over here in the horn.<br>
> > ><br>
> > > Dan<br>
> > ><br>
> > > _______________________________________________<br>
> > > Commotion-discuss mailing list<br>
> > > <a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
> <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>><br>
> > <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
> <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>>><br>
> > > <a href="https://lists.chambana.net/mailman/listinfo/commotion-discuss" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-discuss</a><br>
> > ><br>
> ><br>
> > --<br>
> > Dan Staples<br>
> ><br>
> > Open Technology Institute<br>
> > <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
> > OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
> > Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
> > _______________________________________________<br>
> > Commotion-dev mailing list<br>
> > <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
> > <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>>><br>
> > <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Ben West<br>
> > <a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
> > <a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>><br>
> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>>><br>
> > <a href="tel:314-246-9434" value="+13142469434">314-246-9434</a> <tel:<a href="tel:314-246-9434" value="+13142469434">314-246-9434</a>><br>
><br>
> --<br>
> Dan Staples<br>
><br>
> Open Technology Institute<br>
> <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
> OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
> Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
> _______________________________________________<br>
> Commotion-dev mailing list<br>
> <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
> <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Ben West<br>
> <a href="mailto:me@benwest.name">me@benwest.name</a> <mailto:<a href="mailto:me@benwest.name">me@benwest.name</a>><br>
--<br>
Dan Staples<br>
<br>
Open Technology Institute<br>
<a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 06 Nov 2013 13:00:58 -0500<br>
From: Ryan Gerety <<a href="mailto:gerety@opentechinstitute.org">gerety@opentechinstitute.org</a>><br>
To: <a href="mailto:commotion-discuss@lists.chambana.net">commotion-discuss@lists.chambana.net</a><br>
Subject: Re: [Commotion-discuss] [Commotion-dev] Memory Issues and<br>
Nightly Builds<br>
Message-ID: <<a href="mailto:527A83DA.6050507@opentechinstitute.org">527A83DA.6050507@opentechinstitute.org</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
I was under the impression that PicoStations could support roughly 30 users?<br>
<br>
<br>
On 11/6/2013 12:54 PM, Dan Staples wrote:<br>
> Since Dan Hastings has seen this happen with a lot of simultaneous<br>
> clients and with high-memory components disabled, it sounds like that is<br>
> likely the cause. Do you know exactly where that RAM is used for each<br>
> connecting client?<br>
><br>
> Dan, can you provide any more detailed info on exactly what was<br>
> happening when you see the node crashing? How many simultaneous users,<br>
> and what were they doing (viewing a webpage on the internet, or viewing<br>
> the node's administrative web interface, etc)?<br>
><br>
> On Wed 06 Nov 2013 12:40:06 PM EST, Ben West wrote:<br>
>> Hi Dan,<br>
>><br>
>> Thanks for offering more detail, especially that you see the nodes<br>
>> spontaneously reboot rather than simple have services crash.<br>
>><br>
>> I would again point out that the Picostations will have a finite limit<br>
>> for simultaneous clients. 15 to 20 clients is quite a few, each<br>
>> client requiring a portion of available of RAM. It may be a single<br>
>> Picostation is not going to be able to sustain all of them.<br>
>><br>
>><br>
>><br>
>> On Wed, Nov 6, 2013 at 10:58 AM, Dan Staples<br>
>> <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
>> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>> wrote:<br>
>><br>
>> Regarding logging, I'm not sure that will work well since the<br>
>> nodes are<br>
>> spontaneously rebooting themselves (due to OOM conditions), not<br>
>> the user<br>
>> rebooting them. What we're going to try to do is attach a serial<br>
>> console<br>
>> (thanks Will!) and try to slam the router with simultaneous users and<br>
>> traffic.<br>
>><br>
>> Also, I don't think Dan is hosting local apps on the router itself<br>
>> (correct me if I'm wrong), but just advertising them using the<br>
>> Commotion<br>
>> apps portal. And that's just takes a little space for the Avahi<br>
>> service<br>
>> file...so hopefully that's not a problem.<br>
>><br>
>> We'll certainly report what we find with our stress testing.<br>
>><br>
>> Dan<br>
>><br>
>> On 11/06/2013 10:37 AM, Ben West wrote:<br>
>> > I am also seeing sporadic memory consumption issues operating<br>
>> mesh nodes<br>
>> > running AA r38347 in WasabiNet on Nanostation Loco M2.<br>
>> ><br>
>> > That is, using the same ath9k wifi driver and same underlying<br>
>> OS, but<br>
>> > without the Commotion-specific tools like commotiond and servald. I<br>
>> > will see nodes boot up with ~26Mbytes memory usage and then<br>
>> gradually<br>
>> > increase over the next few days until sporadic nodes start<br>
>> crashing with<br>
>> > page allocation failures (aka memory exhausted). This all is<br>
>> happening<br>
>> > despite having 3Mbytes of compressed swap space allocated.<br>
>> When I am<br>
>> > able to log into crashed nodes to inspect, I will occasionally<br>
>> find the<br>
>> > current memory usage to be /less/ than the average observed on<br>
>> bootup,<br>
>> > along with ~500Kbytes sitting in swap.<br>
>> ><br>
>> > This seems to suggest something is very sporadically allocating<br>
>> itself a<br>
>> > large chunk (multiple MBytes), but not residing in memory as<br>
>> such, and<br>
>> > causing other processes to crash in consequence. I do use the<br>
>> > coovachilli captive portal in WasabiNet, which could be a<br>
>> culprit and<br>
>> > thus unrelated to Commotion, but there could also be an underlying<br>
>> > memory leak in the kernel or wifi driver.<br>
>> ><br>
>> > What are thoughts for having crashed nodes try to collect a<br>
>> debug report<br>
>> > about themselves when a crash condition is detected (e.g. no<br>
>> Internet<br>
>> > access, "page allocation failure" detected in syslog), and then<br>
>> write<br>
>> > that report to flash somewhere before the node get rebooted by its<br>
>> > frustrated user?<br>
>> ><br>
>> > Besides that, do note that nodes with only 32MBytes of RAM, like<br>
>> UBNT<br>
>> > Picostations, are going to have difficulties hosting local apps<br>
>> for many<br>
>> > users. If Dan Hasting would be able to use an alternate device with<br>
>> > 64Mbytes+ RAM, e.g. a UBNT Rocket, Unifi, or even an indoor TP-Link<br>
>> > router (all of which should be able to run Commotion-OpenWRT),<br>
>> that may<br>
>> > be a viable workaround in cause chasing down memory leaks<br>
>> becomes too<br>
>> > ornery.<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Wed, Nov 6, 2013 at 8:54 AM, Dan Staples<br>
>> > <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
>> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>><br>
>> > <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
>> <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>>> wrote:<br>
>> ><br>
>> > +commotion-dev<br>
>> ><br>
>> > If your nodes are crashing w/ 15-20 clients, while both<br>
>> serval and<br>
>> > commotion-splash are disabled, that is very worrisome!<br>
>> ><br>
>> > I propose to the Commotion dev team that we urgently need to<br>
>> come up<br>
>> > with a way to simulate network load, so we can identify and<br>
>> fix the<br>
>> > causes of these types of crashes. Does anyone have ideas or<br>
>> experiences<br>
>> > with this? Perhaps we can take the technical discussion over<br>
>> to the<br>
>> > commotion-dev list only.<br>
>> ><br>
>> > And just an update for you Dan, earlier this week I found<br>
>> and fixed a<br>
>> > significant memory leak in Serval...not sure how much that<br>
>> will affect<br>
>> > the instability we've seen, but we'll soon know with some<br>
>> testing. The<br>
>> > fix will make its way into the nightly builds probably by<br>
>> the end of the<br>
>> > week.<br>
>> ><br>
>> > As long as the rest of your network is DR1 or newer, the<br>
>> nightly builds<br>
>> > should be compatible.<br>
>> ><br>
>> > Dan<br>
>> ><br>
>> > On 11/06/2013 04:07 AM, Dan Hastings wrote:<br>
>> > > I was just checking to see if their had been any progress<br>
>> made on the<br>
>> > > nightly builds with fixing the memory overload causing the<br>
>> nodes to<br>
>> > > crash. To try and prevent my node from crashing I disabled<br>
>> serval and<br>
>> > > the splash page. However, whenever I have 15 to 20<br>
>> students login to a<br>
>> > > local app at the start of class my node crashes instantly. I'm<br>
>> > wondering<br>
>> > > if upgrading to the latest nightly build might fix this<br>
>> issue. Lastly,<br>
>> > > if I upgrade to the latest nightly build will it still<br>
>> work with the<br>
>> > > other nodes that do not have the latest build or do I have<br>
>> to or is it<br>
>> > > recommend that I upgrade all of the other nodes to latest<br>
>> build as<br>
>> > > well? Thanks for all the hard work. Commotion is<br>
>> otherwise working<br>
>> > > wonders over here in the horn.<br>
>> > ><br>
>> > > Dan<br>
>> > ><br>
>> > > _______________________________________________<br>
>> > > Commotion-discuss mailing list<br>
>> > > <a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
>> <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>><br>
>> > <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
>> <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>>><br>
>> > > <a href="https://lists.chambana.net/mailman/listinfo/commotion-discuss" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-discuss</a><br>
>> > ><br>
>> ><br>
>> > --<br>
>> > Dan Staples<br>
>> ><br>
>> > Open Technology Institute<br>
>> > <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
>> > OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
>> > Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
>> > _______________________________________________<br>
>> > Commotion-dev mailing list<br>
>> > <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
>> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
>> > <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
>> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>>><br>
>> > <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Ben West<br>
>> > <a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
>> > <a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>><br>
>> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>>><br>
>> > <a href="tel:314-246-9434" value="+13142469434">314-246-9434</a> <tel:<a href="tel:314-246-9434" value="+13142469434">314-246-9434</a>><br>
>><br>
>> --<br>
>> Dan Staples<br>
>><br>
>> Open Technology Institute<br>
>> <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
>> OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
>> Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
>> _______________________________________________<br>
>> Commotion-dev mailing list<br>
>> <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
>> <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
>> <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Ben West<br>
>> <a href="mailto:me@benwest.name">me@benwest.name</a> <mailto:<a href="mailto:me@benwest.name">me@benwest.name</a>><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 6 Nov 2013 12:09:48 -0600<br>
From: Ben West <<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>><br>
To: Dan Staples <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>><br>
Cc: commotion-discuss <<a href="mailto:commotion-discuss@lists.chambana.net">commotion-discuss@lists.chambana.net</a>>,<br>
Commotion Development List <<a href="mailto:commotion-dev@lists.chambana.net">commotion-dev@lists.chambana.net</a>><br>
Subject: Re: [Commotion-discuss] [Commotion-dev] Memory Issues and<br>
Nightly Builds<br>
Message-ID:<br>
<<a href="mailto:CADSh-SPWYwsXsXHkX-AiB4Mt%2B3beB8Ly62vm7EscfQZZb89THg@mail.gmail.com">CADSh-SPWYwsXsXHkX-AiB4Mt+3beB8Ly62vm7EscfQZZb89THg@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
It's relevant to point out that devices like Picostation and Nanostation<br>
are normally intended for use as thin hotspots in high-usage environments,<br>
i.e. DHCP and NAT routing not done on the device itself. So,<br>
Commotion-OpenWRT issuing DHCP leases and performing NAT one or 2 local<br>
LANs onboard does consume memory that otherwise would go to serving 802.11n<br>
clients. This is an inherent limitation of the chosen architecture.<br>
<br>
Besides that, I would assume at least these processes need to devote a<br>
portion of available RAM to each client on the public AP in<br>
Commotion-OpenWRT:<br>
<br>
- /proc/net/nf_conntrack entries<br>
- nodogsplash (although possibly only on initial portal page viewing)<br>
- uhttpd (again, only on portal page viewing)<br>
- the ath9k driver itself<br>
<br>
<br>
<br>
<br>
On Wed, Nov 6, 2013 at 11:54 AM, Dan Staples <<br>
<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>> wrote:<br>
<br>
> Since Dan Hastings has seen this happen with a lot of simultaneous<br>
> clients and with high-memory components disabled, it sounds like that is<br>
> likely the cause. Do you know exactly where that RAM is used for each<br>
> connecting client?<br>
><br>
> Dan, can you provide any more detailed info on exactly what was<br>
> happening when you see the node crashing? How many simultaneous users,<br>
> and what were they doing (viewing a webpage on the internet, or viewing<br>
> the node's administrative web interface, etc)?<br>
><br>
> On Wed 06 Nov 2013 12:40:06 PM EST, Ben West wrote:<br>
> > Hi Dan,<br>
> ><br>
> > Thanks for offering more detail, especially that you see the nodes<br>
> > spontaneously reboot rather than simple have services crash.<br>
> ><br>
> > I would again point out that the Picostations will have a finite limit<br>
> > for simultaneous clients. 15 to 20 clients is quite a few, each<br>
> > client requiring a portion of available of RAM. It may be a single<br>
> > Picostation is not going to be able to sustain all of them.<br>
> ><br>
> ><br>
> ><br>
> > On Wed, Nov 6, 2013 at 10:58 AM, Dan Staples<br>
> > <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
> > <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>> wrote:<br>
> ><br>
> > Regarding logging, I'm not sure that will work well since the<br>
> > nodes are<br>
> > spontaneously rebooting themselves (due to OOM conditions), not<br>
> > the user<br>
> > rebooting them. What we're going to try to do is attach a serial<br>
> > console<br>
> > (thanks Will!) and try to slam the router with simultaneous users and<br>
> > traffic.<br>
> ><br>
> > Also, I don't think Dan is hosting local apps on the router itself<br>
> > (correct me if I'm wrong), but just advertising them using the<br>
> > Commotion<br>
> > apps portal. And that's just takes a little space for the Avahi<br>
> > service<br>
> > file...so hopefully that's not a problem.<br>
> ><br>
> > We'll certainly report what we find with our stress testing.<br>
> ><br>
> > Dan<br>
> ><br>
> > On 11/06/2013 10:37 AM, Ben West wrote:<br>
> > > I am also seeing sporadic memory consumption issues operating<br>
> > mesh nodes<br>
> > > running AA r38347 in WasabiNet on Nanostation Loco M2.<br>
> > ><br>
> > > That is, using the same ath9k wifi driver and same underlying<br>
> > OS, but<br>
> > > without the Commotion-specific tools like commotiond and servald.<br>
> I<br>
> > > will see nodes boot up with ~26Mbytes memory usage and then<br>
> > gradually<br>
> > > increase over the next few days until sporadic nodes start<br>
> > crashing with<br>
> > > page allocation failures (aka memory exhausted). This all is<br>
> > happening<br>
> > > despite having 3Mbytes of compressed swap space allocated.<br>
> > When I am<br>
> > > able to log into crashed nodes to inspect, I will occasionally<br>
> > find the<br>
> > > current memory usage to be /less/ than the average observed on<br>
> > bootup,<br>
> > > along with ~500Kbytes sitting in swap.<br>
> > ><br>
> > > This seems to suggest something is very sporadically allocating<br>
> > itself a<br>
> > > large chunk (multiple MBytes), but not residing in memory as<br>
> > such, and<br>
> > > causing other processes to crash in consequence. I do use the<br>
> > > coovachilli captive portal in WasabiNet, which could be a<br>
> > culprit and<br>
> > > thus unrelated to Commotion, but there could also be an underlying<br>
> > > memory leak in the kernel or wifi driver.<br>
> > ><br>
> > > What are thoughts for having crashed nodes try to collect a<br>
> > debug report<br>
> > > about themselves when a crash condition is detected (e.g. no<br>
> > Internet<br>
> > > access, "page allocation failure" detected in syslog), and then<br>
> > write<br>
> > > that report to flash somewhere before the node get rebooted by its<br>
> > > frustrated user?<br>
> > ><br>
> > > Besides that, do note that nodes with only 32MBytes of RAM, like<br>
> > UBNT<br>
> > > Picostations, are going to have difficulties hosting local apps<br>
> > for many<br>
> > > users. If Dan Hasting would be able to use an alternate device<br>
> with<br>
> > > 64Mbytes+ RAM, e.g. a UBNT Rocket, Unifi, or even an indoor TP-Link<br>
> > > router (all of which should be able to run Commotion-OpenWRT),<br>
> > that may<br>
> > > be a viable workaround in cause chasing down memory leaks<br>
> > becomes too<br>
> > > ornery.<br>
> > ><br>
> > ><br>
> > ><br>
> > > On Wed, Nov 6, 2013 at 8:54 AM, Dan Staples<br>
> > > <<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
> > <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>><br>
> > > <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a><br>
> > <mailto:<a href="mailto:danstaples@opentechinstitute.org">danstaples@opentechinstitute.org</a>>>> wrote:<br>
> > ><br>
> > > +commotion-dev<br>
> > ><br>
> > > If your nodes are crashing w/ 15-20 clients, while both<br>
> > serval and<br>
> > > commotion-splash are disabled, that is very worrisome!<br>
> > ><br>
> > > I propose to the Commotion dev team that we urgently need to<br>
> > come up<br>
> > > with a way to simulate network load, so we can identify and<br>
> > fix the<br>
> > > causes of these types of crashes. Does anyone have ideas or<br>
> > experiences<br>
> > > with this? Perhaps we can take the technical discussion over<br>
> > to the<br>
> > > commotion-dev list only.<br>
> > ><br>
> > > And just an update for you Dan, earlier this week I found<br>
> > and fixed a<br>
> > > significant memory leak in Serval...not sure how much that<br>
> > will affect<br>
> > > the instability we've seen, but we'll soon know with some<br>
> > testing. The<br>
> > > fix will make its way into the nightly builds probably by<br>
> > the end of the<br>
> > > week.<br>
> > ><br>
> > > As long as the rest of your network is DR1 or newer, the<br>
> > nightly builds<br>
> > > should be compatible.<br>
> > ><br>
> > > Dan<br>
> > ><br>
> > > On 11/06/2013 04:07 AM, Dan Hastings wrote:<br>
> > > > I was just checking to see if their had been any progress<br>
> > made on the<br>
> > > > nightly builds with fixing the memory overload causing the<br>
> > nodes to<br>
> > > > crash. To try and prevent my node from crashing I disabled<br>
> > serval and<br>
> > > > the splash page. However, whenever I have 15 to 20<br>
> > students login to a<br>
> > > > local app at the start of class my node crashes instantly.<br>
> I'm<br>
> > > wondering<br>
> > > > if upgrading to the latest nightly build might fix this<br>
> > issue. Lastly,<br>
> > > > if I upgrade to the latest nightly build will it still<br>
> > work with the<br>
> > > > other nodes that do not have the latest build or do I have<br>
> > to or is it<br>
> > > > recommend that I upgrade all of the other nodes to latest<br>
> > build as<br>
> > > > well? Thanks for all the hard work. Commotion is<br>
> > otherwise working<br>
> > > > wonders over here in the horn.<br>
> > > ><br>
> > > > Dan<br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > Commotion-discuss mailing list<br>
> > > > <a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
> > <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>><br>
> > > <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
> > <mailto:<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a>>><br>
> > > ><br>
> <a href="https://lists.chambana.net/mailman/listinfo/commotion-discuss" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-discuss</a><br>
> > > ><br>
> > ><br>
> > > --<br>
> > > Dan Staples<br>
> > ><br>
> > > Open Technology Institute<br>
> > > <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
> > > OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
> > > Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
> > > _______________________________________________<br>
> > > Commotion-dev mailing list<br>
> > > <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
> > <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
> > > <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
> > <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>>><br>
> > > <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Ben West<br>
> > > <a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
> > > <a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>><br>
> > <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a> <mailto:<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a>>><br>
> > > <a href="tel:314-246-9434" value="+13142469434">314-246-9434</a> <tel:<a href="tel:314-246-9434" value="+13142469434">314-246-9434</a>><br>
> ><br>
> > --<br>
> > Dan Staples<br>
> ><br>
> > Open Technology Institute<br>
> > <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
> > OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
> > Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
> > _______________________________________________<br>
> > Commotion-dev mailing list<br>
> > <a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a><br>
> > <mailto:<a href="mailto:Commotion-dev@lists.chambana.net">Commotion-dev@lists.chambana.net</a>><br>
> > <a href="https://lists.chambana.net/mailman/listinfo/commotion-dev" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-dev</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Ben West<br>
> > <a href="mailto:me@benwest.name">me@benwest.name</a> <mailto:<a href="mailto:me@benwest.name">me@benwest.name</a>><br>
> --<br>
> Dan Staples<br>
><br>
> Open Technology Institute<br>
> <a href="https://commotionwireless.net" target="_blank">https://commotionwireless.net</a><br>
> OpenPGP key: <a href="http://disman.tl/pgp.asc" target="_blank">http://disman.tl/pgp.asc</a><br>
> Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9<br>
><br>
<br>
<br>
<br>
--<br>
Ben West<br>
<a href="http://gowasabi.net" target="_blank">http://gowasabi.net</a><br>
<a href="mailto:ben@gowasabi.net">ben@gowasabi.net</a><br>
<a href="tel:314-246-9434" value="+13142469434">314-246-9434</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.chambana.net/pipermail/commotion-discuss/attachments/20131106/f18dd5a7/attachment.html" target="_blank">http://lists.chambana.net/pipermail/commotion-discuss/attachments/20131106/f18dd5a7/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
Commotion-discuss mailing list<br>
<a href="mailto:Commotion-discuss@lists.chambana.net">Commotion-discuss@lists.chambana.net</a><br>
<a href="https://lists.chambana.net/mailman/listinfo/commotion-discuss" target="_blank">https://lists.chambana.net/mailman/listinfo/commotion-discuss</a><br>
<br>
<br>
------------------------------<br>
<br>
End of Commotion-discuss Digest, Vol 14, Issue 5<br>
************************************************<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><b>Dan Hastings</b><br><i>Abaarso School Computer Science Department</i><br><a href="mailto:dhastings@abaarsotech.org" target="_blank">dhastings@abaarsotech.org</a><br>
</div>
</div>