[Commotion-dev] Zram-swap and other memory strategies

[Dan Staples]

Ben, I wanted to dredge up this old post to see if you've had success with the zram-swap package on OpenWRT? We've been seeing frequent OOM conditions on the new Barrier Breaker images for Commotion, and I'm looking into solutions. This specifically happens on Picostations, but since they only have 8MB flash, I'm concerned adding even a compressed swap would be a bad idea.

I'm also looking at tuning overcommit and OOM kernel settings. One thing I'm trying now is setting /proc/sys/vm/overcommit_memory to 2 and adjusting /proc/sys/vm/overcommit_ratio. (see https://www.kernel.org/doc/Documentation/sysctl/vm.txt)

If that doesn't help, I'll try turning on /proc/sys/vm/oom_kill_allocating_task.

The strange thing is that memory usage, when the OOM killer engages and provides an info dump, looks almost identical to normal memory usage, so it's hard to figure out what is triggering the low memory conditions.

Dan

On 10/21/2013 12:43 PM, Ben West wrote:
> Here is a comment I added to a recent github issue re: node memory consumption and serval, suggesting enabling compressed swap as a psuedo-workaround.
> 
> https://github.com/opentechinstitute/commotion-openwrt/issues/40#issuecomment-25683085
> 
> ... I suggested adding the "zram-swap" package presently in OpenWRT trunk to the commotion packages feed, and then enabling swap in kernelconfig. This would let you enable compressed swap memory on nodes, and ideally make the memory limit somewhat softer (i.e. help nodes avoid OOM errors and processes crashing).
> 
> So, specifically, do make kernel_menuconfig and make these selections:
> General Setup -> Support for paging of anonymous memory (swap) *
> Device Drivers -> Staging drivers -> Compressed RAM block device support *
> 
> This kernel config change can also be done via a patch, and such a patch is buried somewhere in openwrt-devel listserv archives (i.e. when the zram package was originally announced).
> 
> Then copy the zram_swap package from trunk into commotionfeed and enable it.
> 
> For my nodes with 32MB of RAM, I specify 6MB swap in /etc/config/system:
> 
> |config system
>     ...
>     option 'zram_size_mb' '6'
> |
> 
> You can periodically check swap usage to ensure nothing is using excessive RAM:
> 
> |root at nsm5-b:~# free
>              total         used         free       shared      buffers
> Mem:         29184        24100         5084            0         2752
> -/+ buffers:              21348         7836
> Swap:         6140            0         6140
> 
> |
> 
> I should note this is only a /pseudo-workaround/, as it does not address the underlying issue of excessive RAM consumption.  At best, zrap-swap would allow the node to avoid OOM errors, and furthermore detect it is in a low-memory state and actively do something about it (e.g. reboot).
> 
> I am running zram-swap as described above on WasabiNet nodes, both 5.8GHz mesh backhaul and 2.4GHz mesh APs, and I can confirm that certain processes /do not like to be swapped out/, freezing or behaving erratically as a result.  hostapd, wpa_supplicant, olsrd, crond/busybox, and whatever your captive portal agent is (nodogsplash or coovachilli), all certainly shouldn't be swapped out.  Possibly commotiond too, although I've not had opportunity to test that.
> 
> So, the zram_swap method described above lacks a bit in robustness.  I think the *mlock* command can be used to prevent specific processes from being swapped out, although I'm uncertain of whether OpenWRT has this tool integrated.
> 
> 
> 
> On Sat, Oct 19, 2013 at 8:47 AM, Dan Staples <danstaples at opentechinstitute.org <mailto:danstaples at opentechinstitute.org>> wrote:
> 
>     Hey Dan,
> 
>     Disabling Serval won't affect the splash screen, but it will prevent
>     local app announcements from automatically propagating. Here's how to
>     turn it off/disable it:
> 
>     /etc/init.d/serval-dna disable && /etc/init.d/serval-dna stop
> 
>     Let me know if that doesn't solve the problem!
> 
>     best,
>     Dan
> 
>     On 10/19/2013 04:56 AM, Daniel Hastings wrote:
>     > I was wondering if there was a quick fix to the memory issue that causes
>     > the nodes to crash in DR2.  I know that Dan had mentioned that you could
>     > disable serval even though it disabled the splash screen.  Even that is
>     > the case I'm ok with it as long as if fixes the memory issue for the
>     > time being. Any quick simple explanation would be extremely helpful.  We
>     > are having some serious issues with nodes crashing repeatedly on end
>     > over here.
>     >
>     > Thanks again!
>     >
>     > Dan
>     >
>     > --
>     > *Dan Hastings*
>     > /Abaarso School Computer Science Department/
>     > dhastings at abaarsotech.org <mailto:dhastings at abaarsotech.org> <mailto:dhastings at abaarsotech.org <mailto:dhastings at abaarsotech.org>>
>     >
>     >
>     > _______________________________________________
>     > Commotion-discuss mailing list
>     > Commotion-discuss at lists.chambana.net <mailto:Commotion-discuss at lists.chambana.net>
>     > https://lists.chambana.net/mailman/listinfo/commotion-discuss
>     >
> 
>     --
>     Dan Staples
> 
>     Open Technology Institute
>     https://commotionwireless.net
>     OpenPGP key: http://disman.tl/pgp.asc
>     Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9
>     _______________________________________________
>     Commotion-discuss mailing list
>     Commotion-discuss at lists.chambana.net <mailto:Commotion-discuss at lists.chambana.net>
>     https://lists.chambana.net/mailman/listinfo/commotion-discuss
> 
> 
> 
> 
> -- 
> Ben West
> http://gowasabi.net
> ben at gowasabi.net <mailto:ben at gowasabi.net>
> 314-246-9434

-- 
Dan Staples

Open Technology Institute
https://commotionwireless.net
OpenPGP key: http://disman.tl/pgp.asc
Fingerprint: 2480 095D 4B16 436F 35AB 7305 F670 74ED BD86 43A9