Scribd
Upload
95 views10 pages

PARCS-Python: Distributed Computing Setup

This document describes how to set up a distributed computing platform using PARCS-python. It discusses node types including a master node that dispatches work and worker nodes that perform work. It provides instructions for SDK setup in Google Cloud including creating a project, enabling APIs, and configuring zones and regions. It also provides instructions for creating firewall rules and instances including creating a master instance and multiple worker instances. Finally, it provides an example problem that maps work across workers and reduces the results.

Uploaded by

Max The human
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
95 views10 pages

PARCS-Python: Distributed Computing Setup

This document describes how to set up a distributed computing platform using PARCS-python. It discusses node types including a master node that dispatches work and worker nodes that perform work. It provides instructions for SDK setup in Google Cloud including creating a project, enabling APIs, and configuring zones and regions. It also provides instructions for creating firewall rules and instances including creating a master instance and multiple worker instances. Finally, it provides an example problem that maps work across workers and reduces the results.

Uploaded by

Max The human
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

PARCS-python

Distributed computations platform


Node types

• Master - dispatches work

• Worker - performs work


SDK setup

• Install gcloud sdk, login with gcloud auth login

• gcloud projects create parcs-python


• gcloud config set project parcs-python
• # enable billing and compute APIs for project
• gcloud config set compute/zone europe-north1-a
• gcloud config set compute/region europe-north1
Firewall setup

• gcloud compute

firewall-rules create allow-all

--direction=INGRESS --priority=1000

--network=default --action=ALLOW

--rules=all --source-ranges=[Link]/0
• 🎉🎉🎉
Instances

• gcloud compute instances



create-with-container master

--container-image=[Link]/
hummer12007/parcs-node

--container-env PARCS_ARGS="master"
• Record master’s IP, e.g. [Link]
• gcloud compute instances

create-with-container worker1 worker2 worker3…

--container-image=[Link]/
hummer12007/parcs-node

--container-env PARCS_ARGS="worker [Link]"
🎉🎉🎉🎉🎉🎉🎉🎉🎉
• Control panel accessible at [Link]
Example problem
[Link]

from Pyro4 import expose


@staticmethod
class Solver:
@expose
def __init__(self, workers=None, input_file_name=None, output_file_name=None):
def mymap(a, b):
self.input_file_name = input_file_name
print (a, b)
self.output_file_name = output_file_name
res = 0
[Link] = workers
for i in xrange(a, b):
print("Inited")
res += i
return res
def solve(self):
print("Job Started")
@staticmethod
print("Workers %d" % len([Link]))
@expose
n = self.read_input()
def myreduce(mapped):
step = n / len([Link])
output = 0
for x in mapped:
# map
output += [Link]
mapped = []
return output
for i in xrange(0, len([Link])):
[Link]([Link][i].mymap(i * step, i * step + step))
def read_input(self):
f = open(self.input_file_name, 'r')
print('Map finished: ', mapped)
line = [Link]()
[Link]()
# reduce
return int(line)
reduced = [Link](mapped)
print("Reduce finished: " + str(reduced))
def write_output(self, output):
f = open(self.output_file_name, 'w')
# output
[Link](str(output))
self.write_output(reduced)
[Link]('\n')
[Link]()
print("Job Finished")
Example problem
[Link]
mapped = []
for i in xrange(0, len([Link])):
[Link](

[Link][i].mymap(i * step, i * step + step))
print('Map finished: ', mapped)
# reduce
reduced = [Link](mapped)
print("Reduce finished: " + str(reduced))

@staticmethod @staticmethod
@expose @expose
def mymap(a, b): def myreduce(mapped):
print (a, b) output = 0
res = 0 for x in mapped:
for i in xrange(a, b): output += [Link]
res += i return output
return res
Done!

🎉
[Link]

You might also like