Projects > Peak-Two Cloud (P2C) >
Peak-Two Cloud Resources
Benchmarking
Report of performance of P2C for HPC using NAS Parallel Benchmarks (NPB) is now available here. |
Tutorial: Using Slurm to run MPI jobs
Quick start: First, learn to use screen.
The script above describes a job to compile and execute hello.c, an MPI program. The important parameters to change are --job-name, --output, and EXEC. The values of these parameters should describe your program. View node information
Submit a job
Check status of a job
Check job details
View result
|
Tutorial: Sharing a vcluster-deployed MPI cluster through Slurm
The main objective of vcluster is to provide on-demand provisioning of MPI clusters. However, sharing a cluster may be more efficient in some use cases, such as for training or teaching. This tutorial describes how to do this by using Slurm.
|
Tutorial: How to prevent freezing of SSH connection to your instance
First, to escape from a frozen ssh session, press and release (excluding the brackets) [ENTER], [~], [.] in succession. On the client machine and on the instance, set the mtu of the interface to 1454 bytes. $sudo ifconfig eth0 mtu 1454 If the above steps did not work, try the following. On your instance edit /etc/ssh/sshd_config then add the following lines at the end. Restart sshd then exit the session. $sudo service ssh restart $exit
Reconnect again. Lastly, if you are using gnome-terminal, set the custom command in Edit->Profile Preferences to the following. env TERM=xterm /bin/bash It is highly recommended to use screen after login. |
Tutorial: Running a Python Spark application in a vhadoop deployed hadoop cluster
"Apache Spark is a fast and general processing engine compatible with Hadoop
data. It can run in Hadoop clusters through YARN or Spark's standalone
mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any
Hadoop InputFormat. It is designed to perform both batch processing
(similar to MapReduce) and new workloads like streaming, interactive
queries, and machine learning."
"""CountUP.py""" from pyspark import SparkConf, SparkContext wiki = "/wc-in/tagalog.txt" conf = (SparkConf() .setMaster("yarn-client") .setAppName("CountUP") .set("spark.executor.memory", "128m")) sc = SparkContext(conf = conf) data = sc.textFile(wiki).cache() us = data.filter(lambda s: 'Unibersidad' in s).count() ps = data.filter(lambda s: 'Pilipinas' in s).count() print "Lines with Unibersidad: %i, lines with Pilipinas: %i" % (us, ps) $spark-submit CountUP.py |
Tutorial: How to upload files to your instance
If you need to upload files to your instance from the command line, the syntax is shown below. $scp -oUserKnownHostsFile=/dev/null -i <Key File>
For example: $scp -oUserKnownHostsFile=/dev/null -i user.pem
If you enabled password authentication in SSH and created a user. $scp
|
Tutorial: Enable SSH access to your instance using a username and password
As described in the User Guide, you can connect to your instance using a key pair as shown below. $ssh -oUserKnownHostsFile =/dev/null -i jachermocilla -guest- p2c.pem ubuntu@10.0.3.247 However, using a username and password to login maybe more convenient, especially if you are sharing access to others. This tutorial describes how to configure SSH for this on an Ubuntu instance. First login using the key pair as described above then become root. $sudo -s Second edit /etc/ssh/sshd_config. Set PasswordAuthentication to YES. #nano /etc/ssh/sshd_config Third, restart the SSH server. #service ssh restart Fourth, create a new user. You need to supply the password and other information for the user. #adduser newusername Fifth, add newusername ALL=(ALL:ALL) ALL to /etc/sudoers. This will give the new user sudo access. #nano /etc/sudoers Sixth, logout of the instance. #exit $exit That's it! You can now connect using username and password. $ssh -oUserKnownHostsFile =/dev/null newuser @10.0.3.247 |
Supported Tools
The table below shows the supported tools. They support the Message Passing Interface (MPI) for distributed processing of large data sets and computation-intensive operations. After creating a cluster in P2C using vcluster, the p2ctools utility can be used to activate any of these tools.
|
Tutorial: Running unattended jobs on P2C clusters using Screen
Some jobs you run on P2C will need a long time to finish(or your ssh session freezes due to network instability). When you connect to a cluster remotely (using SSH for example) and start your job, you will have to run the task in the background (adding & at the end of the command line) in order to logout of the SSH session without terminating your job early. An alternative way is to use Screen. (HINT: To escape from a frozen ssh session press in sequence: [ENTER] [~] [.] ) 1. Connect to the cluster using SSH. 2. Once you are logged in, create a new screen session using the command below. The -S option specifies the name of he screen session, in the example it is session-01 $screen -S session-01 3. Run a command continuously. Simple check the memory usage. $watch free 4. Detach from the session by pressing Ctrl-a, d (Press 'Ctrl' and 'a' at the same time, release, then press 'd'). You will be returned to your SSH login shell. Alternatively you can close the terminal where you ssh'd then go directly to step 7. 5. View the list of available screens. You can create additional screens using step 2. $screen -list 6. Exit from SSH. $exit 7. Connect using SSH again. Once logged in, view the available screens. $screen -list 8. Re attach to the session-01 screen. $screen -r session-01 9. You will observe that the watch command is still running even after you logged out of SSH. For more information read the man page of the Screen command. You can exit the screen session using exit. |
1-10 of 21