Tracking down memory leaks



  • Are there any ways and or tools to help track down memory leaks? I cannot use the PRO version with the built in IDE as it doesn't have enough free memory for the program I am running. The app runs for about 7 minutes before throwing Memory full, aborting.Fatal error, restarting program...

    Any pointers would be appreciated.



  • So I have been doing some digging and I don't think this is a memory leak but rather a bunch of tasks happening at a point in time where the garbage collector has not run for a bit and then those tasks are too big to fit in memory. I have staggered the same tasks out over a loner period allowing the GC time to do it's thing and all is good. The problem with that is sometimes I need a few things to happen at the 'same' time but without the GC running, there is not enough memory. ('same' obviously sequentially but about the same time.) Is there any way to manually call the CG or at least influence when it runs?



  • You should be able to call require('lowsys').gc()

    However, the GC should be called automatically just in time. So, most probably there are some dangeling references. This can happen very easily in JavaScript, especially by using closures. One event handler defined in a closure keeps all variables of the function and the parent function and the parent function of memory, and then there are variables in these functions which also stay in memory, and they also have event handlers and so on.

    I remember a server side project I wrote which required gigabytes of memory. After realizing the problem memory usage went down 99%.

    And with 1 MB of RAM on the device, every such thing is a problem.

    Thomas



  • So calling gc seems to have resolved the issue. Basically I had 6 scheduled tasks that run at different intervals and if they run all at the same time (which happens statistically at 7 minutes based on how I had them set previously) the module runs out of memory if that also coincides with the gc not having run for some time since the memory was not low enough to trigger it. ie once it got down to like 200k, it was not enough for all 6 to run without gc. For testing purposes I call the gc every time I flash the running led (once every 2 seconds) and now there are no more reboots due to insufficient memory as when all 6 run, there is plenty. This is an extreme case as I would never have 6 of these tasks scheduled and they would never be run so frequently but I always test for extremes as it is the best way to catch things like this. I will play around and come up with a less aggressive gc collection but glad it is resolved. Thanks for the feedback.



  • I spoke too soon. I put the scheduled tasks back at a fairly frequent pace and again the memory runs out and the module reboots. Now after about 2.5 hours so better than before but still not right. Now however, the module got to a point of rebooting without even finishing the startup routine several times and is now dead. It cannot startup at all. Removed power and it doesn't even connect to wifi. What can I do from USB to try and find the state of the module and why it cannot start? Below is the last few rounds of attempted restarts until it gave up totally.

    a1ae9844-7e7c-45a1-a578-a8a3bd56836e-image.png



  • Is there anything I can do with this module to find out the state before wiping it and starting fresh?



  • I thought about this one for a while.

    On the neonious one you can skip auto start of the program via pressing the button.

    It seems like you have access to your device for a short while. You could disable auto start with lowsync settings set in that moment. https://neonious.com/lowjs/documentation/lowjs-settings.html

    Seems like memory full handling is not optimal yet. I will look into that besides other work.

    Thomas



  • Been working on the HW side of this project so just now getting back to this.

    Eventually I don't have access to the device. It is totally unresponsive and not connected to the WiFi network. The only thing I can do to recover it is a power cycle. The 'restarting program...' event doesn't seem to end up with all memory being reclaimed which would lead me to think something under the program itself is leaking memory.



  • Have you tried disabling auto-start as I wrote. That should allow you to get back in.

    Thomas



  • I am not sure this would help since it the first time it restarts it continues to work without an issue. It restarts many times before it finally fails to restart at all. That is why I tried to create the append file at boot with freemem but ran into the append file not working. This would give some indication of how many times it restarts and what the memory looks like after each restart.



  • So you do have access. OK, when you wrote "Is there anything I can do with this module to find out the state before wiping it and starting fresh?" it did not seem so.

    I can help when you pinpoint the problem more.



  • Well eventually it reboots to the point of breaking where there is no WiFi connection and I can do nothing but power cycle it. The screenshot above is when it gets to this point. It has restarted many times from an initial power cycle where the program runs fine for many restarts but eventually gets to a restart where there is no memory left to even restart the program and the module seems to hang. Hopefully being able to append to the log file rather than overwrite it will give more insights.


Log in to reply