Introduction to CPU-intensive Cloud Parallelism

This tutorial is from the MBrace Starter Kit.

In this tutorial you learn how to perform CPU-intensive cloud-parallel workload on your MBrace cluster. The work will be computing prime numbers, though you can easily replace this with any code of your own.

First, you run some work on your local machine (single-threaded). Performance will depend on the spec of your machine.

Note that it is possible that your client scripting machine is more efficient than each individual machine in the cluster.

1: 
2: 
3: 
4:

let locallyComputedPrimes =
    [| for i in 1 .. 30 do
         let primes = Sieve.getPrimes 100000000
         yield sprintf "calculated %d primes: %A" primes.Length primes  |]

Next, you run the work on the cluster, on multiple workers. This exploits the the multiple machines (workers) in the cluster.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14:

let clusterPrimesTask =
    [| for i in 1 .. 30 -> 
         local { 
            let primes = Sieve.getPrimes 100000000
            return sprintf "calculated %d primes %A on machine '%s'" primes.Length primes Environment.MachineName 
         }
    |]
    |> Cloud.ParallelBalanced
    |> cluster.CreateProcess


clusterPrimesTask.ShowInfo()

let clusterPrimes = clusterPrimesTask.Result

Starting multiple jobs

The previous example used Cloud.Parallel to compose jobs into one combined job.

Alternatively, you could have started 30 independent jobs. This can be handy if you want to track each one independently:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10:

let jobs =  
    [ for i in 1 .. 10 -> 
         cloud { 
            let primes = Sieve.getPrimes 100000000
            return sprintf "calculated %d primes %A on machine '%s'" primes.Length primes Environment.MachineName 
         }
        |> cluster.CreateProcess ]

let jobResults = 
    [ for job in jobs -> job.Result ]

For example, you might want to track the exectuion time of each job:

1: 
2:

let jobTimes = 
    [ for job in jobs -> job.ExecutionTime ]

Tracking and controlling job failures

Jobs can fail for all sorts of reasons, normally by raising exceptions. For example, let's simulate this by injecting failures into our jobs:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10:

let makeJob i = 
    cloud { 
        if i % 8 = 0 then failwith "fail"
        let primes = Sieve.getPrimes 1000000
        return sprintf "calculated %d primes %A on machine '%s'" primes.Length primes Environment.MachineName 
    }

let jobs2 =  
    [ for i in 1 .. 10 -> 
         makeJob i |> cluster.CreateProcess ]

If you now attempt to access the results of the job, you will get the exception propagated back to your client scripting session:

1: 
2: 
3:

// raises an exception since some of the jobs failed
let jobResults2 = 
    [ for job in jobs2 -> job.Result ]

When this happens, it is useful to protect your jobs by a wrapper that captures information about the failing work:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21:

type Result<'T, 'Info> =
   | Success of 'T
   | Failure of 'Info * exn

let protectJob info (work: Cloud<_>) = 
    cloud { 
       try
           let! result = work 
           return Success result
       with exn -> 
           return Failure (info, exn)
    }

let jobs3 =  
    [ for i in 1 .. 10 -> 
         makeJob i
         |> protectJob ("job " + string i) 
         |> cluster.CreateProcess ]

let jobResults3 = 
    [ for job in jobs3 -> job.Result ]

The results will now show the success and failure of each job:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11:

val jobResults3 : Result<string,string> list =
  [Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Failure ("job 8", System.Exception: fail...)
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars];
   Success "calculated 78498 primes [|2; 3; 5; 7; 11; 13; 17; 19; 23; 29;"+[477 chars]]

You can then re-create adjusted versions of failing jobs and re-run them:

1:	`makeJob 8 \|> cluster.Run`

Summary

In this tutorial, you've learned how to do simple CPU-intensive work using MBrace.Azure. Continue with further samples to learn more about the MBrace programming model.

Note, you can use the above techniques from both scripts and compiled projects. To see the components referenced by this script, see ThespianCluster.fsx or AzureCluster.fsx.

namespace System

namespace System.IO

namespace MBrace

namespace MBrace.Core

namespace MBrace.Flow

val cluster : MBrace.Thespian.ThespianCluster

Full name: 3-cloud-parallel-cpu-intensive.cluster

module Config

val GetCluster : unit -> MBrace.Thespian.ThespianCluster

Full name: Config.GetCluster

Gets or creates a new Thespian cluster session.

val locallyComputedPrimes : string []

Full name: 3-cloud-parallel-cpu-intensive.locallyComputedPrimes

val i : int

val primes : int []

module Sieve

val getPrimes : nmax:int -> int []

Full name: Sieve.getPrimes

val sprintf : format:Printf.StringFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.sprintf

property Array.Length: int

val clusterPrimesTask : MBrace.Runtime.CloudProcess<obj>

Full name: 3-cloud-parallel-cpu-intensive.clusterPrimesTask

type Environment =
  static member CommandLine : string
  static member CurrentDirectory : string with get, set
  static member Exit : exitCode:int -> unit
  static member ExitCode : int with get, set
  static member ExpandEnvironmentVariables : name:string -> string
  static member FailFast : message:string -> unit + 1 overload
  static member GetCommandLineArgs : unit -> string[]
  static member GetEnvironmentVariable : variable:string -> string + 1 overload
  static member GetEnvironmentVariables : unit -> IDictionary + 1 overload
  static member GetFolderPath : folder:SpecialFolder -> string + 1 overload
  ...
  nested type SpecialFolder
  nested type SpecialFolderOption

Full name: System.Environment

property Environment.MachineName: string

member MBrace.Runtime.MBraceClient.CreateProcess : workflow:MBrace.Core.Cloud<'T> * ?cancellationToken:MBrace.Core.ICloudCancellationToken * ?faultPolicy:MBrace.Core.FaultPolicy * ?target:MBrace.Core.IWorkerRef * ?additionalResources:MBrace.Core.Internals.ResourceRegistry * ?taskName:string -> MBrace.Runtime.CloudProcess<'T>

member MBrace.Runtime.CloudProcess.ShowInfo : unit -> unit

val clusterPrimes : obj

Full name: 3-cloud-parallel-cpu-intensive.clusterPrimes

property MBrace.Runtime.CloudProcess.Result: obj

val jobs : MBrace.Runtime.CloudProcess<obj> list

Full name: 3-cloud-parallel-cpu-intensive.jobs

val jobResults : obj list

Full name: 3-cloud-parallel-cpu-intensive.jobResults

val job : MBrace.Runtime.CloudProcess<obj>

val jobTimes : TimeSpan option list

Full name: 3-cloud-parallel-cpu-intensive.jobTimes

property MBrace.Runtime.CloudProcess.ExecutionTime: TimeSpan option

val makeJob : i:'a -> 'b

Full name: 3-cloud-parallel-cpu-intensive.makeJob

val i : 'a

val failwith : message:string -> 'T

Full name: Microsoft.FSharp.Core.Operators.failwith

val jobs2 : MBrace.Runtime.CloudProcess<obj> list

Full name: 3-cloud-parallel-cpu-intensive.jobs2

val jobResults2 : obj list

Full name: cloudparallelcpuintensive.jobResults2

val job : obj

type Result<'T,'Info> =
| Success of 'T
| Failure of 'Info * exn

Full name: 3-cloud-parallel-cpu-intensive.Result<_,_>

union case Result.Success: 'T -> Result<'T,'Info>

Multiple items
union case Result.Failure: 'Info * exn -> Result<'T,'Info>

--------------------
active recognizer Failure: exn -> string option

Full name: Microsoft.FSharp.Core.Operators.( |Failure|_| )

type exn = Exception

Full name: Microsoft.FSharp.Core.exn

val protectJob : info:'a -> work:'b -> 'c

Full name: 3-cloud-parallel-cpu-intensive.protectJob

val info : 'a

val work : 'b

union case Result.Failure: 'Info * exn -> Result<'T,'Info>

val jobs3 : MBrace.Runtime.CloudProcess<obj> list

Full name: 3-cloud-parallel-cpu-intensive.jobs3

Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = String

Full name: Microsoft.FSharp.Core.string

val jobResults3 : obj list

Full name: 3-cloud-parallel-cpu-intensive.jobResults3

Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = System.String

Full name: Microsoft.FSharp.Core.string

type 'T list = List<'T>

Full name: Microsoft.FSharp.Collections.list<_>

Multiple items
val Failure : message:string -> exn

Full name: Microsoft.FSharp.Core.Operators.Failure

--------------------
active recognizer Failure: exn -> string option

Full name: Microsoft.FSharp.Core.Operators.( |Failure|_| )

Multiple items
type Exception =
  new : unit -> Exception + 2 overloads
  member Data : IDictionary
  member GetBaseException : unit -> Exception
  member GetObjectData : info:SerializationInfo * context:StreamingContext -> unit
  member GetType : unit -> Type
  member HelpLink : string with get, set
  member InnerException : Exception
  member Message : string
  member Source : string with get, set
  member StackTrace : string
  ...

Full name: System.Exception

--------------------
System.Exception() : unit
System.Exception(message: string) : unit
System.Exception(message: string, innerException: exn) : unit

member MBrace.Runtime.MBraceClient.Run : workflow:MBrace.Core.Cloud<'T> * ?cancellationToken:MBrace.Core.ICloudCancellationToken * ?faultPolicy:MBrace.Core.FaultPolicy * ?target:MBrace.Core.IWorkerRef * ?additionalResources:MBrace.Core.Internals.ResourceRegistry * ?taskName:string -> 'T

MBrace.Core and MBrace.Azure

Introduction to CPU-intensive Cloud Parallelism

Starting multiple jobs

Tracking and controlling job failures

Summary