Low-level Node.js debugging on production
Sometimes when you want to run your Node.js application on production environment, you will receive:
Yeah… and now what?
Welcome to the dungeon, DevOps marine .
First of all, if you deployed an application on an OS different than SmartOS, you lost the ability to use many impressive tools which could definitely help you in that case (the most impressive ones are
dtrace, you can find details in the references section of this post).
So there is no
dtrace, no SmartOS. You can’t also use your favorite debugger like trace.gl, node-monkey or node-inspector. You have got only
gdb and your brain filled with low-level computer knowledge .
Before you dive deeper into the internals of your application, let’s begin the voodoo that is the process of setting the proper options in your OS. First you will remove the limit related with the core dump size:
Next, you set different path for these files (remember to put them on a partition with a huge amount of the free disk space):
Then, if you are using native Node.js extensions you have to recompile them with the debugging flag
-g (configuration file taken from the
Then, look at the dependencies (in our case
libxml2) and reinstall them in the development version (sometimes marked with a
After modifications and installing dependencies in the debugging mode, you have to run installation process inside the directory of the modified module (but only there, not on the upper level, because it will remove our changes in the modified module):
Somewhere inside the long output you will find information about flags used in compilation process (and you should see a
-g flag, responsible for attaching debugging informations).
Next, you have to simulate again situation with
Segmentation fault behavior. When application receive unwanted signal, it will dump file with the post-mortem internal structure, ready for using inside
And now you are inside the belly of the monster, so you can move around:
If you want to attach to the running process, you have to invoke the command specified below:
However, if your application is running in a cluster mode (or you are running more than one Node.js application on your machine), this will fail and instead of
pidof you have to pass a single PID value (from one application, master or one of the slaves, depends on what is interesting for you).
You can also run
strace in order to determine which system calls does your program invoke before death:
After gathering a certain amount of knowledge, armed with the stack traces, system call invocations and the memory footprint you have to dig deeper into the actual application code and maybe try to reproduce that behavior in a more debuggable environment .