Why I can not use all the 100% potential of CPU?

Question

I am using my computer clusters to run a MD C program but I can not use full potential of those clusters.

But the this node have 16 CPUs and I also only give 15 jobs for those CPUs. But I can not fully use those potentials.

below is the result of ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  23636  1624 ?        Ss   Jun15   0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/0]
root         6  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/0]
root         7  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/1]
root         8  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/1]
root         9  0.0  0.0      0     0 ?        S    Jun15   0:05 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/1]
root        11  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/2]
root        12  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/2]
root        13  0.0  0.0      0     0 ?        S    Jun15   0:01 [ksoftirqd/2]
root        14  0.0  0.0      0     0 ?        S    Jun15   0:02 [watchdog/2]
root        15  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/3]
root        16  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/3]
root        17  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/3]
root        18  0.0  0.0      0     0 ?        S    Jun15   0:17 [watchdog/3]
root        19  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/4]
root        20  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/4]
root        21  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/4]
root        22  0.0  0.0      0     0 ?        S    Jun15   0:02 [watchdog/4]
root        23  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/5]
root        24  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/5]
root        25  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/5]
root        26  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/5]
root        27  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/6]
root        28  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/6]
root        29  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/6]
root        30  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/6]
root        31  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/7]
root        32  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/7]
root        33  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/7]
root        34  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/7]
root        35  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/8]
root        36  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/8]
root        37  0.0  0.0      0     0 ?        S    Jun15   0:01 [ksoftirqd/8]
root        38  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/8]
root        39  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/9]
root        40  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/9]
root        41  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/9]
root        42  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/9]
root        43  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/10]
root        44  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/10]
root        45  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/10]
root        46  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/10]
root        47  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/11]
root        48  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/11]
root        49  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/11]
root        50  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/11]
root        51  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/12]
root        52  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/12]
root        53  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/12]
root        54  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/12]
root        55  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/13]
root        56  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/13]
root        57  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/13]
root        58  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/13]
root        59  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/14]
root        60  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/14]
root        61  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/14]
root        62  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/14]
root        63  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/15]
root        64  0.0  0.0      0     0 ?        S    Jun15   0:00 [migration/15]
root        65  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksoftirqd/15]
root        66  0.0  0.0      0     0 ?        S    Jun15   0:00 [watchdog/15]
root        67  0.0  0.0      0     0 ?        S    Jun15   0:00 [events/0]
root        68  0.0  0.0      0     0 ?        S    Jun15   0:16 [events/1]
root        69  0.0  0.0      0     0 ?        S    Jun15   0:26 [events/2]
root        70  0.0  0.0      0     0 ?        S    Jun15   1:24 [events/3]
root        71  0.0  0.0      0     0 ?        S    Jun15   4:17 [events/4]
root        72  0.0  0.0      0     0 ?        S    Jun15   4:11 [events/5]
root        73  0.0  0.0      0     0 ?        S    Jun15   0:31 [events/6]
root        74  0.0  0.0      0     0 ?        S    Jun15   2:34 [events/7]
root        75  0.0  0.0      0     0 ?        S    Jun15   1:11 [events/8]
root        76  0.0  0.0      0     0 ?        S    Jun15  11:39 [events/9]
root        77  0.0  0.0      0     0 ?        S    Jun15   1:15 [events/10]
root        78  0.0  0.0      0     0 ?        S    Jun15   0:01 [events/11]
root        79  0.0  0.0      0     0 ?        S    Jun15   0:00 [events/12]
root        80  0.0  0.0      0     0 ?        S    Jun15   0:01 [events/13]
root        81  0.0  0.0      0     0 ?        S    Jun15   0:00 [events/14]
root        82  0.0  0.0      0     0 ?        S    Jun15   0:00 [events/15]
root        83  0.0  0.0      0     0 ?        S    Jun15   0:00 [cpuset]
root        84  0.0  0.0      0     0 ?        S    Jun15   0:00 [khelper]
root        85  0.0  0.0      0     0 ?        S    Jun15   0:00 [netns]
root        86  0.0  0.0      0     0 ?        S    Jun15   0:00 [async/mgr]
root        87  0.0  0.0      0     0 ?        S    Jun15   0:00 [pm]
root        88  0.0  0.0      0     0 ?        S    Jun15   0:09 [sync_supers]
root        89  0.0  0.0      0     0 ?        S    Jun15   0:07 [bdi-default]
root        90  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/0]
root        91  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/1]
root        92  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/2]
root        93  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/3]
root        94  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/4]
root        95  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/5]
root        96  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/6]
root        97  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/7]
root        98  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/8]
root        99  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/9]
root       100  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/10]
root       101  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/11]
root       102  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/12]
root       103  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/13]
root       104  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/14]
root       105  0.0  0.0      0     0 ?        S    Jun15   0:00 [kintegrityd/15]
root       106  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/0]
root       107  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/1]
root       108  0.0  0.0      0     0 ?        S    Jun15   0:01 [kblockd/2]
root       109  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/3]
root       110  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/4]
root       111  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/5]
root       112  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/6]
root       113  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/7]
root       114  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/8]
root       115  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/9]
root       116  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/10]
root       117  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/11]
root       118  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/12]
root       119  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/13]
root       120  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/14]
root       121  0.0  0.0      0     0 ?        S    Jun15   0:00 [kblockd/15]
root       122  0.0  0.0      0     0 ?        S    Jun15   0:00 [kacpid]
root       123  0.0  0.0      0     0 ?        S    Jun15   0:00 [kacpi_notify]
root       124  0.0  0.0      0     0 ?        S    Jun15   0:00 [kacpi_hotplug]
root       125  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/0]
root       126  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/1]
root       127  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/2]
root       128  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/3]
root       129  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/4]
root       130  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/5]
root       131  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/6]
root       132  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/7]
root       133  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/8]
root       134  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/9]
root       135  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/10]
root       136  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/11]
root       137  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/12]
root       138  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/13]
root       139  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/14]
root       140  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata/15]
root       141  0.0  0.0      0     0 ?        S    Jun15   0:00 [ata_aux]
root       142  0.0  0.0      0     0 ?        S    Jun15   0:00 [ksuspend_usbd]
root       143  0.0  0.0      0     0 ?        S    Jun15   0:00 [khubd]
root       144  0.0  0.0      0     0 ?        S    Jun15   0:00 [kseriod]
root       145  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/0]
root       146  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/1]
root       147  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/2]
root       148  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/3]
root       149  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/4]
root       150  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/5]
root       151  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/6]
root       152  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/7]
root       153  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/8]
root       154  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/9]
root       155  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/10]
root       156  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/11]
root       157  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/12]
root       158  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/13]
root       159  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/14]
root       160  0.0  0.0      0     0 ?        S    Jun15   0:00 [md/15]
root       161  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/0]
root       162  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/1]
root       163  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/2]
root       164  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/3]
root       165  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/4]
root       166  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/5]
root       167  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/6]
root       168  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/7]
root       169  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/8]
root       170  0.0  0.0      0     0 ?        S    Jun15   0:00 [md_misc/9]
root       178  0.0  0.0      0     0 ?        S    Jun15   0:00 [kswapd0]
root       179  0.0  0.0      0     0 ?        S    Jun15   0:00 [kswapd1]
root       180  0.0  0.0      0     0 ?        SN   Jun15   0:00 [ksmd]
root       181  0.0  0.0      0     0 ?        SN   Jun15   0:10 [khugepaged]
root       182  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/0]
root       183  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/1]
root       184  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/2]
root       185  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/3]
root       186  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/4]
root       187  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/5]
root       188  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/6]
root       189  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/7]
root       190  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/8]
root       191  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/9]
root       192  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/10]
root       193  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/11]
root       194  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/12]
root       195  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/13]
root       196  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/14]
root       197  0.0  0.0      0     0 ?        S    Jun15   0:00 [aio/15]
root       198  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/0]
root       199  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/1]
root       200  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/2]
root       201  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/3]
root       202  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/4]
root       203  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/5]
root       204  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/6]
root       205  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/7]
root       206  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/8]
root       207  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/9]
root       208  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/10]
root       209  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/11]
root       210  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/12]
root       211  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/13]
root       212  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/14]
root       213  0.0  0.0      0     0 ?        S    Jun15   0:00 [crypto/15]
root       218  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/0]
root       219  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/1]
root       220  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/2]
root       221  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/3]
root       222  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/4]
root       223  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/5]
root       224  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/6]
root       225  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/7]
root       226  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/8]
root       227  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/9]
root       228  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/10]
root       229  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/11]
root       230  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/12]
root       231  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/13]
root       232  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/14]
root       233  0.0  0.0      0     0 ?        S    Jun15   0:00 [kthrotld/15]
root       246  0.0  0.0      0     0 ?        S    Jun15   0:00 [kpsmoused]
root       247  0.0  0.0      0     0 ?        S    Jun15   0:00 [usbhid_resumer]
root       277  0.0  0.0      0     0 ?        S    Jun15   0:00 [kstriped]
root       611  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_0]
root       612  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_1]
root       613  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_2]
root       614  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_3]
root       615  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_4]
root       616  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_5]
root       763  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_6]
root       764  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_wq_6]
root       769  0.0  0.0      0     0 ?        S    Jun15   0:00 [scsi_eh_7]
root       770  0.0  0.0      0     0 ?        S    Jun15   0:00 [fw_event0]
root       773  0.0  0.0      0     0 ?        S    Jun15   0:54 [poll_0_status]
root       818  0.0  0.0      0     0 ?        S    Jun15   0:12 [jbd2/sda3-8]
root       819  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       820  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       821  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       822  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       823  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       824  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       825  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       826  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       827  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       828  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       829  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       830  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       831  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       832  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       833  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       834  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root       921  0.0  0.0  11672  1652 ?        S<s  Jun15   0:00 /sbin/udevd -d
root      1292  0.0  0.0      0     0 ?        S    Jun15   1:03 [edac-poller]
root      1892  0.0  0.0      0     0 ?        S    Jun15   0:00 [mlx4]
root      1894  0.0  0.0      0     0 ?        S    Jun15   0:00 [mlx4_opreq]
root      1895  0.0  0.0      0     0 ?        S    Jun15   0:12 [flush-8:0]
root      1896  0.0  0.0      0     0 ?        S    Jun15   0:12 [mlx4_sense]
root      1905  0.0  0.0      0     0 ?        S    Jun15   0:00 [mlx4_en]
root      2077  0.0  0.0  11684  1664 ?        S<   Jun15   0:00 /sbin/udevd -d
root      2125  0.0  0.0      0     0 ?        S    Jun15   0:00 [jbd2/sda1-8]
root      2126  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2127  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2128  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2129  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2130  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2131  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2132  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2133  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2134  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2135  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2136  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2137  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2138  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2139  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2140  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2141  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2142  0.0  0.0      0     0 ?        S    Jun15   0:00 [jbd2/sda5-8]
root      2143  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2144  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2145  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2146  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2147  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2148  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2149  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2150  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2151  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2152  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2153  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2154  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2155  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2156  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2157  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2158  0.0  0.0      0     0 ?        S    Jun15   0:00 [ext4-dio-unwrit]
root      2202  0.0  0.0      0     0 ?        S    Jun15   0:00 [kauditd]
root      2250  0.0  0.0      0     0 ?        S    Jun15   0:00 [mthcacatas]
root      2253  0.0  0.0      0     0 ?        S    Jun15   0:00 [mlx4_ib]
root      2254  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_mad1]
root      2261  0.0  0.0      0     0 ?        S    Jun15   0:00 [iw_cxgb3]
root      2265  0.0  0.0      0     0 ?        S    Jun15   0:00 [nesewq]
root      2266  0.0  0.0      0     0 ?        S    Jun15   0:00 [nesdwq]
root      2270  0.0  0.0  11668  1660 ?        S<   Jun15   0:00 /sbin/udevd -d
root      2273  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_mcast]
root      2274  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_inform]
root      2275  0.0  0.0      0     0 ?        S    Jun15   0:00 [local_sa]
root      2276  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/0]
root      2277  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/1]
root      2278  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/2]
root      2279  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/3]
root      2280  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/4]
root      2281  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/5]
root      2282  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/6]
root      2283  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/7]
root      2284  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/8]
root      2285  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/9]
root      2286  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/10]
root      2287  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/11]
root      2288  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/12]
root      2289  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/13]
root      2290  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/14]
root      2291  0.0  0.0      0     0 ?        S    Jun15   0:00 [ib_cm/15]
root      2292  0.0  0.0      0     0 ?        S    Jun15   0:51 [ipoib]
root      2293  0.0  0.0      0     0 ?        S    Jun15   3:17 [ipoib_auto_mode]
root      2358  0.0  0.0      0     0 ?        S    Jun15   0:01 [ib_addr]
root      2359  0.0  0.0      0     0 ?        S    Jun15   0:00 [iw_cm_wq]
root      2360  0.0  0.0      0     0 ?        S    Jun15   0:00 [rdma_cm]
root      2594  0.0  0.0  93224   896 ?        S<sl Jun15   0:04 auditd
155       2664  0.0  0.0  60788  9360 ?        S    Jun15   0:11 /usr/libexec/systemtap/stap-serverd -r 2.6.32-220.el6.x86_64 -a x86_64 --log=/var/log/stap-server/log
root      2691  0.0  0.0 250856  1560 ?        Sl   Jun15   0:01 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4
root      2714  0.0  0.0      0     0 ?        S    Jun15   0:00 [kondemand/0]
root      2715  0.0  0.0      0     0 ?        S    Jun15   0:24 [kondemand/1]
root      2716  0.0  0.0      0     0 ?        S    Jun15   0:48 [kondemand/2]
root      2717  0.0  0.0      0     0 ?        S    Jun15   2:03 [kondemand/3]
root      2718  0.0  0.0      0     0 ?        S    Jun15   5:41 [kondemand/4]
root      2719  0.0  0.0      0     0 ?        S    Jun15   1:25 [kondemand/5]
root      2720  0.0  0.0      0     0 ?        S    Jun15   1:26 [kondemand/6]
root      2721  0.0  0.0      0     0 ?        S    Jun15   2:09 [kondemand/7]
root      2722  0.0  0.0      0     0 ?        S    Jun15   0:40 [kondemand/8]
root      2723  0.0  0.0      0     0 ?        S    Jun15   1:28 [kondemand/9]
root      2724  0.0  0.0      0     0 ?        S    Jun15   0:09 [kondemand/10]
root      2725  0.0  0.0      0     0 ?        S    Jun15   0:00 [kondemand/11]
root      2726  0.0  0.0      0     0 ?        S    Jun15   0:00 [kondemand/12]
root      2727  0.0  0.0      0     0 ?        S    Jun15   0:00 [kondemand/13]
root      2728  0.0  0.0      0     0 ?        S    Jun15   0:00 [kondemand/14]
root      2729  0.0  0.0      0     0 ?        S    Jun15   0:00 [kondemand/15]
root      2740  0.0  0.0   9204   644 ?        Ss   Jun15  17:34 irqbalance
rpc       2754  0.0  0.0  19024   984 ?        Ss   Jun15   0:02 rpcbind
rpcuser   2772  0.0  0.0  23200  1204 ?        Ss   Jun15   0:00 rpc.statd
root      2800  0.0  0.0      0     0 ?        S    Jun15   3:22 [rpciod/0]
root      2801  0.0  0.0      0     0 ?        S    Jun15   3:22 [rpciod/1]
root      2802  0.0  0.0      0     0 ?        S    Jun15   3:21 [rpciod/2]
root      2803  0.0  0.0      0     0 ?        S    Jun15   3:18 [rpciod/3]
root      2804  0.0  0.0      0     0 ?        S    Jun15   3:16 [rpciod/4]
root      2805  0.0  0.0      0     0 ?        S    Jun15   3:13 [rpciod/5]
root      2806  0.0  0.0      0     0 ?        S    Jun15   3:10 [rpciod/6]
root      2807  0.0  0.0      0     0 ?        S    Jun15   3:08 [rpciod/7]
root      2808  0.0  0.0      0     0 ?        S    Jun15  26:27 [rpciod/8]
root      2809  0.0  0.0      0     0 ?        S    Jun15   3:58 [rpciod/9]
root      2810  0.0  0.0      0     0 ?        S    Jun15   3:34 [rpciod/10]
root      2811  0.0  0.0      0     0 ?        S    Jun15   3:17 [rpciod/11]
root      2812  0.0  0.0      0     0 ?        S    Jun15   3:11 [rpciod/12]
root      2813  0.0  0.0      0     0 ?        S    Jun15   3:18 [rpciod/13]
root      2814  0.0  0.0      0     0 ?        S    Jun15   3:07 [rpciod/14]

Because the output excess the website limit I store the exact file on GoogleDrive if this may be useful.

output of ps

So are there some methods can solve this problem so I can fully use all those potential of CPUs?

How exactly do you figure out that "not all the CPU's potential is being used"? I see the system is pretty much maxed out. — vonbrand, Aug 7, 2015 at 12:27
@vonbrand Because when I type top I can see the CPU usage of the programe I want to use but the result is not very good, it shows that those programe can only use 50% CPU but before I can use 100% CPU — sikisis, Aug 7, 2015 at 14:10
@vonbrand I updated mu top screen capture to further show the result. obviously the usage is only 50% CPU but just before this time I can get 100%. — sikisis, Aug 7, 2015 at 14:17

casey · Accepted Answer · 2015-08-07 14:37:55Z

8

As mentioned in a comment and without seeing any of your code or other information (which would not be on-topic here anyway) all I can say is your program appears to be IO bound.

The means while your calculations could use more of your CPU, they are having to wait on data and spending many cycles waiting rather than calculating. This can be due to the way you write code (loop optimization, vectorization, etc). A common problem is accessing your data in a way that causes lots of cache misses. Your multiple cores may also share a L3 cache and if they are all working on different data you are probably running into a lot of misses there. Fetching memory from main DRAM is orders of magnitude slower than the on-die cache memory. If your data comes from the disks then you also have to deal with disk read latency in addition to DRAM latency. If the data is across ethernet or some other interconnect you also have to consider the latency in those reads. You can also incur lots of waiting on disk writes, particularly if you are writing often in small chunks rather than buffering for large writes.

In short, there are lot of IO performance considerations that are limiting your ability to maintain 100% CPU usage on your cores. My recommendation is to profile your code, figure out your IO limitations and make sure your code is efficient and go forward from there.

answered Aug 7, 2015 at 14:37

casey

14.5k5 gold badges45 silver badges62 bronze badges

Yeah,Thank you I will have a try. But if it is not in cluster I can fully use CPU in my own 4 CPU laptop,Why?
– sikisis
Aug 7, 2015 at 14:43
You laptop is probably slow enough (CPU capability) that the problem is CPU bound on your laptop. The cluster probably has much more powerful CPUs and your problem becomes IO bound. I'd be surprised if 100% on your 4 core laptop was driving more FLOPS than the cluster CPUs at 50%, but without code it is all speculation.
– casey
Aug 7, 2015 at 14:45
Both of You Two are right, I reduce the IO Frequency and That's really helpful!Now I can fully utilize the power of my CPU,thank you!
– sikisis
Aug 7, 2015 at 15:39
1

"A common problem is accessing your data in a way that causes lots of cache misses. Your multiple cores may also share a L3 cache and if they are all working on different data you are probably running into a lot of misses there." - as $ misses are transparent to processor I'd imagine they would be counted as CPU utilization in top as they are transparent to application/OS (setting aside profilers such as cachegrind) - even though it is I/O from CPU POV.
– Maciej Piechotka
Aug 7, 2015 at 15:40
Note that the I/O might be due to paging, or even due to memory <--> CPU traffic...
– vonbrand
Aug 7, 2015 at 17:38

| Show 1 more comment

PSkocik · Accepted Answer · 2015-08-07 14:33:08Z

2

That pretty much means your apps are IO bound--your hardrive/network, etc. can't keep up with your processor and consequently, the processor spends a lot of time waiting on IO data, not using its full potential.

If the IO your apps depend on is the network and you aren't downloading at your full bandwith, you might get an efficiency increase by adding more processes/threads, assuming that each process/thread adds a download channel that will increase your bandwidth usage towards your maximum.

answered Aug 7, 2015 at 14:33

PSkocik

27.5k13 gold badges81 silver badges139 bronze badges

But in fact I did not need to download any data and little IO happens in the program.It a normal C code to do a lot of calculation but little IO or network. AND if it is not in cluster I can fully use CPU in my own 4 CPU laptop.
– sikisis
Aug 7, 2015 at 14:39
All I can say is the little IO that does happen appears to be too slow for your cores to be maxed out.
– PSkocik
Aug 7, 2015 at 14:42
2

16 cores will be cumulatively much faster, but their IO channels (RAM and caches fall within that) probably won't be.
– PSkocik
Aug 7, 2015 at 14:45
SO if I close or reduce the frequency then it may help fully use the CPU capability?
– sikisis
Aug 7, 2015 at 14:50
Both of You Two are right, I reduce the IO Frequency and That's really helpful!Now I can fully utilize the power of my CPU,thank you!
– sikisis
Aug 7, 2015 at 15:38

Add a comment |

Stack Exchange Network

Why I can not use all the 100% potential of CPU?

2 Answers 2

Your Answer

Not the answer you're looking for? Browse other questions tagged
process
performance
cpu
ps
top
or ask your own question.

Hot Network Questions

Why I can not use all the 100% potential of CPU?

2 Answers 2

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged processperformancecpupstop or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
process
performance
cpu
ps
top
or ask your own question.