< Linux > A simple way to run program with mult-thread

Applicable conditions

If you want to deal with thousands of cases in a same way, you can use this method to accelerate the processing speed.
For example, when you want to load millions of images and process them independently, you can apply this method which may accelerate your processing speed hugely,

Firstly, just write a bash file

start_num=0
thread_num=8  #Number of threads you want apply
all_num=100000  #Numbers of samples you want to deal with
step_num=$[all_num/thread_num]
end_num=$step_num
all_num=$[all_num+step_num]
gpu_num=0
current_thread=0
echo $all_num

cd ${your_directory};  # Locate your code
while (($current_thread < $thread_num)) 
do 
python demo.py  $start_num $end_num  &  #Your interface with parameters
echo $start_num $end_num $gpu_num
start_num=$[start_num+step_num] 
end_num=$[start_num+step_num]
current_thread=$[current_thread + 1]
done

In this code snippet above, I take running python script as an example. You can modify the ‘all_num’ parameter according to the number of samples you want to deal with.

One important thing to mention is that ‘&’ could make your program running parallely which is the most important keyword in this code snippet.

Modify your code

You should modify your code in order to fit the bash code above. One thing to note is that your main function should take at least two input variables. The below code snippet is a python version.:

def main():
    if len(sys.argv) == 3:
        begin_num = int(sys.argv[1])
        end_num = int(sys.argv[2])
    else:
        begin_num = 0
        end_num = float("inf")

    for i, item in enumerate(all_items):
        if i < begin_num:
            continue
        if i >= end_num:
            break
	your_process_fun()....

The below code snippet is a C/C++ version:

int main(int argc, char** argv) {
    int begin_num = 0, end_num = INT_MAX;
    if (argc == 3) {
        begin_num = atoi(argv[1]);
        end_num = atoi(argv[2]);
    }
    int count_num = 0;
    while (Condition) {
        count_num++;
        if (count_num < begin_num) continue;
        if (count_num > end_num) break;
        your_process_fun()....
        }
	}