how to collapse a layer by javascript?

As @Charu Rajput said, collapsing layers is almost certainly not possible.  I was however able to expand all layers with items in them.  As an action, record the command "Locate object" from the layers panel menu.  In the script, iterate the top level layers and call the action each time. 

var layers = app.activeDocument.layers;
for (var i = 0; i < layers.length; i++) {
    app.selection = null;
    layers[i].hasSelectedArtwork = true;
    app.doScript("Action 1", "Set 1");
}