Sounds like one or more serious bugs in your code. The OOM kernel module probably isn't going to help you here.
An infinite loop can be enough to cause CPU usage to go through the roof. Especially if you're using recursion in your program (where a function calls itself recursively).
But if your memory is rapidly decreasing - that also indicates that you almost certainly have one or more memory leaks in your code.
Fix the issues in your code and the problems you're having will go away.
What programming language are you using?
From the description of the problems, I'm assuming that either C or C++?
In which case, if you're dynamically allocating memory in a loop and never freeing/deleting that memory, you're creating memory leaks. You need to free/delete ALL dynamically allocated memory/objects as soon as they are no longer required.
If you're using C++ and your compiler supports the C++11 standard (or greater), then assigning new/dynamically created objects to smart pointers like std::unique_ptr can help to automate memory management and prevent memory leaks.
std::unique_ptr's will delete objects they point to when/if the smart pointer drops out of scope.
But if you're using C, you need to do ALL memory management manually in your code. There is no other way around it. Any dynamically allocated memory MUST ALWAYS be free'd by you as soon as it is no longer required.
And if you're using recursion, you should always put in some code to limit the amount of recursion. Because infinite recursion is NOT usually a good idea.
C and C++ are both extremely powerful programming languages. They
will do whatever you tell them to, even if what you're telling them to do is extremely stupid and/or ill advised.
They can also do things that many other programming languages either cannot do, or simply WILL not do because it's not necessarily safe to do. And because of this - it's very easy to get things wrong.
If you're not careful, you can cause all kinds of serious issues. Your application crashing is the least of your worries. If you do things REALLY badly - you can crash your entire system, which it looks like you may have just discovered. And it's possible to do even worse than that - at least you didn't accidentally overwrite your PC's boot sector, like I did once! Ha ha. Fortunately it wasn't one of my PC's. It was at college, many years ago. And they had backups of everything! Ha ha!
If you're having problems with your code, feel free to post any problematic bits in a reply here, along with some details about what your program's supposed to be doing. And myself, or one of the other programmers here will take a look at it and will try to help you to fix it.
Please note - If you do post code here - make sure you enclose the code in [code][/code] tags.
If you include the language in the opening tag - e.g. [code=c] for C, or [code=cpp] for C++, the code-block in your will have better syntax highlighting and the indentation/formatting of the code will be retained.
The best bet is to add the opening and closing tags to your post and then copy/paste your code inside it.
If you post code as plain text, without using code tags, the indentation/formatting will be lost and there will be no syntax highlighting. So please use code tags if you do decide to post your code.
e.g.
You would write a C++ snippet like this:
[code=cpp]
include <iostream>
int main()
{
std::cout << "Hello world\n";
return 0;
}
[/code]
Then in your post it will appear like this:
C++:
include <iostream>
int main()
{
std::cout << "Hello world\n";
return 0;
}
NOTE: Before anybody asks how I got the first code block to render as plain text - I used plain tags around the entire code block, to show how you'd include code whilst composing a post. But as mentioned above, the original indentation/formatting of the code in the first code block was lost because that block was rendered on the page as plain text, not code! The plain text rendering routines for posts here strip out leading whitespace. So if posting code here - code tags are essential!
