GENIE:GENIEToolboxFunctions

Andrew Price, University of Southampton, ([mailto:a.r.price:soton.ac.uk a.r.price@soton.ac.uk])

gc_jobsubmit
Submit a compute job to progress the model state SYNTAX
 * [handle, retrieve] = gc_jobsubmit(metadata, runtime, resource, [restart], [prepare])

DESCRIPTION
 * Wrapper function for the submission of a GENIE simulation. The function returns a handle for the job and a retrieval data structure.


 * metadata - Passed straight through to write_configuration_files.m This structure contains the information to seed the model


 * runtime - Provides details of the local runtime environment and the archive containing the model binary and static input data


 * resource - Structure describing the remote resource to execute the model on. Run gc_createResource to generate an appropriate structure


 * restart - [Optional] Structure specifying details of the restart files to initialise the run


 * prepare - [Optional] If this string is set to 'prepare' the function performs all actions except the actual job submission. This allows multiple jobs to be set up so that submissions can be made to multi-processor or multi-core nodes.

See also gc_jobstatus, gc_jobretrieve, createResource, createRuntime, write_execute_script, write_configuration_files, write_condor_submission, write_multiplejobs_script

gc_jobstatus
Returns the status of a job SYNTAX
 * status = gc_jobstatus(handle, testFile)

DESCRIPTION
 * Processes the job handle and returns the current status of the job if possible. The status code meanings are taken from the Globus Toolkit even though we may be processing a local or Condor job. The return codes have the following meanings:

-1 is UNKNOWN 1 is PENDING 2 is ACTIVE 3 is DONE 4 is FAILED 5 is SUSPENDED 6 is UNSUBMITTED

See also gc_jobsubmit, gc_jobretrieve

gc_jobretrieve
Retrieve model output data SYNTAX
 * [success, resultsFiles] = gc_jobretrieve(retrieve, [subset])

DESCRIPTION
 * Processes a complete job by retrieving data files from the remote resource. This function interrogates the unique runtime directory and brings back any files that are not part of the runtime structure.

NOTES
 * Globus: The function will attempt to archive the results files using the Java Archive tool jar. If the resource description in the retrieve structure does not have a 'jarutil' field then the function will look for jar in a couple of possible locations. If the remote Globus resource does not have the jar utility installed at a known location then the function will resort to retrieving the results files individually.

See also gc_jobsubmit, gc_jobstatus

write_condor_submission
Writes the Condor submission file for a model run

SYNTAX
 * condor_submission_file = write_condor_submission(ExecuteScript, LocalRunDirUniq, targetOS, transferFiles, restartFiles)

DESCRIPTION
 * Constructs a Condor submission file in the given LocalRunDirUniq targetted for the given targetOS. The 'executable' field is set to the ExecuteScript and the list of files to be transferred is constructed from the transferFiles and restartFiles (optional) cell arrays.

See also gc_jobsubmit

write_configuration_files
Writes the configuration files using the metadata in the input structure

SYNTAX
 * configuration = write_configuration_files(metadata, LocalRunDirUniq)

DESCRIPTION
 * Parses the input metadata data structure and writes configuration files to the LocalRunDirUniq directory based on the information content. A cell array of configuration file names is returned relating to the files that have been created in the specified directory.

See also gc_jobsubmit, write_genie_configfile, write_genie_goinfile, write_genie_glimmerfile

write_execute_script
Writes the execution script that manages the GENIE binary and its output

SYNTAX
 * execFile = write_execute_script(configuration, runtime, resource, configFiles, restartFiles)

DESCRIPTION
 * Writes a script that will be invoked by the system in order to perform the GENIE simulation. The output script is responsible for the following tasks:


 * Extracting the contents of the archive containing the binary and input data files
 * If necessary, moving the configuration files and restart files to the appropriate locations relative to the binary
 * Invoking the model binary
 * Moving model output back to the root directory
 * Tidying up at the end of the run


 * The default version of this script unpacks the archive name defined in runtime.RuntimeArchive using 'unzip.exe' under Windows and 'tar zxvf' under Linux. The files described in configFiles and restartFiles are moved to the subdirectory 'genie_output/EXPID' (EXPID is a field of runtime) where the genie.exe binary is assumed to reside. The script will then change directory to this subdirectory and invoke genie.exe. Upon completion the output files are moved from all of the GENIE module output directories up to the root directory. (We conform to a Condor model of execution and do not retrieve a full directory hierarchy back to the submission machine but just the new files in the root directory of the remote execution node). Finally the script removes all sub-directories of the root as these are no longer needed.

NOTES
 * For most routine invocations of GENIE using the GENIE Toolbox this function should not need to be modified. If newer versions of GENIE (newer than rel-2-1-0) are being used, or if additional in-situ post-processing is required, then users may well need to augment this script to meet their needs.

See also gc_jobsubmit, write_multiplejobs_script, write_configuration_files, write_condor_submission

write_multiplejobs_script
Writes a master script to submit multiple jobs on a single node

SYNTAX
 * execFile = write_multiplejobs_script(LocalRunDirUniq, RemoteRunDirUniqs, RunScripts)

DESCRIPTION
 * Writes a master script that is executed on a remote host to invoke multiple jobs in the background. The jobs are assumed to have been prepared already by the gc_jobprepare function which should have created the unique runtime directory and GridFTP'd all files to the remote host.

See also gc_preparedJobSubmit

createResource
Utility script to specify a compute resource

SYNTAX
 * resource = createResource
 * resource = createResource('label')</tt>

DESCRIPTION
 * This interactive script will prompt the user for details of the compute resource they wish to define. Upon completion the function returns a data structure describing the resource in a format that the GENIE Toolbox can process. A number of pre-defined resource descriptions are available for common computer systems available to members of the project. A list of valid descriptors will be returned if the label is not recognised.

EXAMPLE
 * >> resource = createResource('NGSOxford')</tt>

type: 'globus' name: 'NGS Oxford' MaxJobs: 16 broker: 'PBS' RemoteTargetOS: 'linux' RemoteHost: 'grid-compute.oesc.ox.ac.uk' RemoteRunDir: '/home/ngs0165/' RemoteFileSep: '/' RemoteJobManager: 'jobmanager-pbs' RemoteMaxWallTime: 2880 jarutil: '/usr/bin/jar' JobsPerNode: 1

NOTES
 * Requires a valid proxy certificate in order to automatically probe for the user's home directory on a Globus resource.

See also createRuntime, gc_jobsubmit, gc_jobstatus, gc_jobretrieve

createRuntime
Utility script to specify the local runtime environment

SYNTAX
 * resource = createRuntime</tt>

DESCRIPTION
 * This interactive script will prompt the user for details of the local runtime environment. Upon completion the function returns a data structure describing the runtime in a format that the GENIE Toolbox can process.

See also createResource, gc_jobsubmit, gc_jobstatus, gc_jobretrieve

gc_findMyDBID
Returns the database ID associated with the current proxy certificate

SYNTAX
 * myDBID = gc_findMyDBID</tt>

DESCRIPTION
 * Returns the database user name associated with the user's X.509 certificate registered in the GENIE database. A valid proxy certificate must be instantiated for this function to work.

See also gd_createproxy, gd_query

gc_findMyHomeDir
Returns the home directory of current proxy on the specified Globus host

SYNTAX
 * fullpath = gc_findMyHomeDir(hoststr)</tt>

DESCRIPTION
 * Returns the home directory associated with the user's X.509 certificate on the Globus resource specified by hoststr. A valid proxy certificate must be instantiated for this function to work.

EXAMPLE
 * fullpath = gc_findMyHomeDir('grid-compute.oesc.ox.ac.uk')</tt>
 * /home/ngs0000

See also gd_createproxy

gc_version
Returns the current version of the GENIE Toolbox

SYNTAX
 * version = gc_version</tt>

DESCRIPTION
 * Returns the version of the current GENIE Toolbox release as a string of the form MAJOR.MINOR.POINT.

op_createParameterMapping
Specifies a mapping between the optimiser and problem domains

SYNTAX
 * map = op_createParameterMapping</tt>

DESCRIPTION
 * Returns a data structure describing the variable ranges in the optimiser domain and the problem domain. The mapping between the two ranges is specified for each variable by the cell array scaling. The available scalings are specified in the op_ParameterMapping function.

EXAMPLE
 * map = op_createParameterMapping</tt>

<ol> options_LVARS: [-1 -1] options_UVARS: [1 1] problem_LVARS: [0 100] problem_UVARS: [1 10000] scaling: {'linear' 'log10'} </ol>


 * The example mapping demonstrates the definition of two variables with lower and upper bounds of -1 and 1 respectively in the optimser domain. The two variables have lower and upper bounds of [0 100] and [1 1e4] respectively in the problem domain. The map.scaling field specifies a linear and log10 mapping between the two domains for the respective variables. The op_ParameterMapping function can be used to map a VARS vector between the two domains. The op_GENIE2OPTIONS and op_OPTIONS2GENIE utility scripts will load the mapping specified in this file and convert a VARS vector or matrix from one domain to the other.

See also op_ParameterMapping, op_GENIE2OPTIONS, op_OPTIONS2GENIE

op_GENIE2OPTIONS
Maps the variable values from the GENIE domain to the optimiser domain

SYNTAX
 * OPTIONSVARS = op_GENIE2OPTIONS(GENIEVARS, map)</tt>

DESCRIPTION
 * Wrapping of the op_ParameterMapping function to convert a GENIEVARS vector or matrix of variables in the problem (GENIE) domain to the optimiser domain using the mapping specified in the op_createParameterMapping script.

EXAMPLE
 * map = op_createParameterMapping</tt>

<ol> options_LVARS: [-1 -1] options_UVARS: [1 1] problem_LVARS: [0 100] problem_UVARS: [1 10000] scaling: {'linear' 'log10'} </ol>
 * OPTIONSVARS = op_GENIE2OPTIONS([0.5 1000], map)</tt>

<ol> 0    0 </ol>

See also op_OPTIONS2GENIE, op_createParameterMapping, op_ParameterMapping

op_genieparallel2
GENIE model wrapping for invocation using optjobparallel2

SYNTAX
 * retrieve = op_genieparallel2(VARS)</tt>

DESCRIPTION
 * This function wraps the submission of a specific GENIE simulation using the input VARS vector to configure the instance of the model. The function returns a data structure that can be used by the OptionsMatlab optjobparallel2 job management function to monitor each invocation and retrieve the results when complete.

NOTES
 * A mapping may need to be performed between the variables in the Options (optimiser) domain to the variables in the GENIE (problem) domain. If this is the case then this script should invoke the op_OPTIONS2GENIE on the input VARS vector using an appropriate map data structure before allocating values to the model configuration.

See also op_genieparallel2_parse2, op_GENIE2OPTIONS, OptionsMatlab, gc_jobsubmit

op_genieparallel2_parse2
Parses the results of a model invocation submitted by optjobparallel2

SYNTAX
 * EVAL = op_genieparallel2_parse2(retrieve)</tt>

DESCRIPTION
 * This function polls the job handle contained in the retrieve data structure and processes the data if the job has finished. An empty EVAL result is returned if the job is still running so that the optimiser will return to this job later. Once the job is detected as complete or failed this function retrieves the model output results and returns the objective function value(s) in EVAL.

See also op_genieparallel2, OptionsMatlab, gc_jobstatus, gc_jobretrieve

op_OPTIONS2GENIE
Maps the variable values from the optimiser domain to the GENIE domain

SYNTAX
 * GENIEVARS = op_OPTIONS2GENIE(OPTIONSVARS, map)</tt>

DESCRIPTION
 * Wrapping of the op_ParameterMapping function to convert a OPTIONSVARS vector or matrix of variables in the optimiser domain to the problem (GENIE) domain using the mapping specified in the op_createParameterMapping function.

EXAMPLE
 * map = op_createParameterMapping</tt>

<ol> options_LVARS: [-1 -1] options_UVARS: [1 1] problem_LVARS: [0 100] problem_UVARS: [1 10000] scaling: {'linear' 'log10'} </ol>


 * <tt>GENIEVARS = op_OPTIONS2GENIE([0 0], map)</tt>

<ol> 1.0e+003 *

0.0005   1.0000 </ol>

See also op_GENIE2OPTIONS, op_createParameterMapping, op_ParameterMapping

op_ParameterMapping
Maps variables to/from the problem domain from/to the optimiser domain

SYNTAX
 * <tt>outVARS = op_ParameterMapping(inVARS, direction, options_LVARS, options_UVARS, SCL, problem_LVARS, problem_UVARS)</tt>

DESCRIPTION
 * Utility function to map the variables used by the optimiser into the user-defined problem domain and back again. The function maps the vector or matrix of variables inVARS in the specified direction. The optimiser domain bounds and problem domain bounds are specified in the vectors options_LVARS, options_UVARS and problem_LVARS, problem_UVARS repectively. The scaling between the domains for each variables is specified in the SCL cell array.


 * Valid values for the direction argument are:
 * 'toDomain'   - maps inVARS to the problem domain
 * 'toOptimiser' - maps inVARS to the optimiser domain


 * Valid values for the SCL cell array are:
 * 'linear'     - performs a linear mapping between the domains
 * 'log10'      - performs a log10 mapping between the domains

EXAMPLE
 * <tt>outVARS=op_ParameterMapping(0,'toDomain',-1,1,{'linear'},0,2)</tt>

<ol> 1 </ol>


 * <tt>outVARS=op_ParameterMapping(1,'toOptimiser',-1,1,{'linear'},0,2)</tt>

<ol> 0 </ol>

See also op_createParameterMapping, op_GENIE2OPTIONS, op_OPTIONS2GENIE