Subprocesses and multithreading
​

Concurrency is when a computer does many different things seemingly at the same time. For example, on a computer with one CPU core, the operating system will rapidly change which program is running on the single processor. This interleaves execution of the programs, providing the illusion that the programs are running simultaneously. 

Parallelism is actually doing many different things at the same time. Computers with multiple CPU cores can execute multiple programs simultaneously. 

Files used in the examples below:

1.wav
File Size:	981 kb
File Type:	wav

Download File

---

3.wav
File Size:	478 kb
File Type:	wav

Download File

---

Use subprocess to Manage Child Processes 
Child processes started by Python are able to run in parallel, enabling you to use Python to consume all of the CPU cores of your machine and maximize the throughput of your programs. 

you can use the run interface

>>> import subprocess
>>> subprocess.run(["pwd"], capture_output=True)
CompletedProcess(args=['pwd'], returncode=0,
 stdout=b'/Users/gulanurmatova/Desktop/python/lecture13\n', stderr=b'')

or more advanced popen interface:

import subprocess
proc = subprocess.Popen(
    ['echo', 'Hello from the child!'],
    stdout=subprocess.PIPE)
out, err = proc.communicate()
print(out.decode('utf-8'))

Why are we doing this?

let's look at a small program that plays sounds:

import os
os.system("afplay 1.wav")
os.system("afplay 3.wav")

the sounds play sequentially, but what if we wanted to play them at the same time?
How would you modify the first program to get the sounds play in parallel?

import subprocess
proc1 = subprocess.Popen(
    ['afplay', '1.wav'],
    stdout=subprocess.PIPE)
proc2 = subprocess.Popen(
    ['afplay', '3.wav'],
    stdout=subprocess.PIPE)
out1, err1 = proc1.communicate()
out2, err2 = proc2.communicate()
print(out1.decode('utf-8'))
if(err1 is not None):
	print(err1.decode('utf-8'))
print(out2.decode('utf-8'))
if(err2 is not None):
	print(err2.decode('utf-8'))

Child processes will run independently from their parent process, the Python interpreter. Their status can be polled periodically while Python does other work. 

import subprocess
import time

proc1 = subprocess.Popen(
    ['afplay', '1.wav'],
    stdout=subprocess.PIPE)
proc2 = subprocess.Popen(
    ['afplay', '3.wav'],
    stdout=subprocess.PIPE)

while proc1.poll() is None:
    print('Playing...')
    time.sleep(1)

out1, err1 = proc1.communicate()
out2, err2 = proc2.communicate()
print(out1.decode('utf-8'))
if(err1 is not None):
	print(err1.decode('utf-8'))
print(out2.decode('utf-8'))
if(err2 is not None):
	print(err2.decode('utf-8'))

If you’re worried about the child processes never finishing or somehow blocking on input or output pipes, then be sure to pass the timeout parameter to the communicate method. This will cause an exception to be raised if the child process hasn’t responded within a time period, giving you a chance to terminate the misbehaving child. 

import subprocess
import time

proc1 = subprocess.Popen(
    ['afplay', '1.wav'],
    stdout=subprocess.PIPE)
proc2 = subprocess.Popen(
    ['afplay', '3.wav'],
    stdout=subprocess.PIPE)

try:
    out1, err1 = proc1.communicate(timeout=2)
    out2, err2 = proc2.communicate(timeout=2)
except subprocess.TimeoutExpired:
    proc1.terminate()
    proc1.wait()
    proc2.terminate()
    proc2.wait()

​Multithreading

a simple example to start 3 threads

import threading
import time

def my_job():
	print(threading.current_thread().name)


t=threading.Thread(target=my_job)
t.start()

t1=threading.Thread(target=my_job)
t1.start()

t2=threading.Thread(target=my_job)
t2.start()

output:

Thread-1
Thread-2
Thread-3

You cannot distribute the threading module over multiple CPU's explicitly, let's modify our song player to use the threading module:

import threading
import os

def my_job(song):

	os.system("afplay "+song)

t1=threading.Thread(target=my_job, args=["1.wav"])
t1.start()
t2=threading.Thread(target=my_job, args=["3.wav"])
t2.start()

If you listen to the output, the songs will likely play sequentially. This is because both threads are tied to the same CPU. Then how would it be considered a parallel processing you may ask - the threads would be processed in parallel if one of them goes to sleep. That may not be the case with music, but very frequently, you'd be putting your threads to sleep. Let's look at few such examples and then we will move to the multiprocessing module of Python.

Following program is the medicine reminder, this does pass the processing to the other threads, because we put the threads to sleep 

import threading
import os
import time

def my_job(pill_name,time_to_wait):
	while(True):
		time.sleep(time_to_wait)
		print("It's time to take your pill " + pill_name)

t1=threading.Thread(target=my_job, args=["advil",0.03])
t1.start()
t2=threading.Thread(target=my_job, args=["tylenol",0.05])
t2.start()
t3=threading.Thread(target=my_job, args=["motrin",0.02])
t3.start()

Output:

It's time to take your pill motrin
It's time to take your pill advil
It's time to take your pill motrin
It's time to take your pill tylenol
It's time to take your pill advil
It's time to take your pill motrin
It's time to take your pill motrin
It's time to take your pill advil
It's time to take your pill tylenol
It's time to take your pill motrin
It's time to take your pill advil
It's time to take your pill motrin

Same program using multiprocessing:

import multiprocessing
import os
import time

def my_job(pill_name,time_to_wait):
	while(True):
		time.sleep(time_to_wait)
		print("It's time to take your pill " + pill_name)

t1=multiprocessing.Process(target=my_job, args=["advil",0.03])
t1.start()
t2=multiprocessing.Process(target=my_job, args=["tylenol",0.05])
t2.start()
t3=multiprocessing.Process(target=my_job, args=["motrin",0.02])
t3.start()

Now, let's see if we can get the music program to work in parallel:

import multiprocessing
import os
import time

def my_job(song):
	os.system("afplay "+song)

t1=multiprocessing.Process(target=my_job, args=["1.wav"])
t1.start()
t2=multiprocessing.Process(target=my_job, args=["3.wav"])
t2.start()