1 / 7

main sphinx design concepts

The Database Warehouse is the information depot for all of Sphinx ... There is no direct communication between different internal Sphinx modules ...

erika
Download Presentation

main sphinx design concepts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. Main Sphinx Design Concepts

    There are two primary design components which comprise Sphinx The Database Warehouse The Control Process The Database Warehouse is the information depot for all of Sphinx All information is stored/retrieved via the database (or possibly multiple databases) There is no direct communication between different internal Sphinx modules The API for all internal modules is SQL along with the database schema The Control Process is the main point of activity It is a loop which calls different Sphinx modules All modules in the control process operate on stateful entities in the database, changing the state of an entity from one to another state Sphinx operates as a Finite State Machine! Uses data mining principles in the design: The API for all modules is SQL along the database schema Modules mine the database for information and then take some action.

    2. More Details

    The components are currently split (packaged) in the following way: Sphinx Client Clarens Web Service (XML-RPC) + VDT-Client Sends a Chimera DAX to Sphinx Server Receives individual “concrete Jobs” from Sphinx Server Submits “Job” (= “micro DAG”) to Condor-G/DAGMan “micro DAG” (for now…could use Grid-shell or equivalent in the future) Job to create environment Job to move input data locally (if required) Job to run application Job to move output data (if required) Job to register data in RLS Job to clean-up environment Sphinx Server Clarens Web Service (XML-RPC) Receives a Chimera DAX Returns individual concrete “Jobs” (=“micro DAG”) Provides heuristic algorithms and Satisfies constraints for: policies, input data existence, required resources grid weather

    3. Version 1 (current configuration): Sphinx Server with Sphinx Submission Client

    Sphinx Server VDT Client VDT Server Site Monitoring Service Resource Repl. Loc. Serv. Condor-G/DAGMan Multi-threaded Control Process Job Execution Planner DAG Reducer DAG Tracker Job Predictor DAG Data Repl. Planner Job Admission Control Message Interface DAG Predictor DAG Admission Control Policies Accounting Resource Prop. Resource Hier. Resource Dist. Distance Entity-Name Entity Hier. Job-Entity Prop. Job DAG Rel. Data Rel. Appl. Job Tracking Grid Weather Data Warehouse Schedule Prediction Data Management Information Gatherer Sphinx Client Message Interface Job Submitter Job Tracker Message indicates current place-holder implementation

    4. Version 1: Sphinx Control Process – Simple Finite State Machine

    DAG: unreduced JOB: unpredicted DAG: unpredicted JOB: unpredicted DAG: unaccepted JOB: unaccepted DAG: unpredicted JOB: unaccepted DAG: unaccepted JOB: unplanned DAG: unplanned JOB: unplanned yes no Await Mesg. Reduce DAG Predict Job no no yes yes yes no Predict DAG Admit Job Admit DAG no no yes yes DAG: unfinished JOB: unplanned DAG: unfinished JOB: unsent DAG: remove JOB: remove DAG: unfinished JOB: remove yes no Plan Dag Plan Job Submit Job no no yes yes yes no Remove Job Remove DAG

    Sphinx Server VDT Client VDT Server Site Monitoring Service Resource Repl. Loc. Serv. Condor-G/DAGMan Multi-threaded Control Process Job Execution Planner DAG Reducer DAG Tracker Job Predictor DAG Data Repl. Planner Job Admission Control Message Interface DAG Predictor DAG Admission Control Policies Accounting Resource Prop. Resource Hier. Resource Dist. Distance Entity-Name Entity Hier. Job-Entity Prop. Job DAG Rel. Data Rel. Appl. Job Tracking Grid Weather Data Warehouse Schedule Prediction Data Management Information Gatherer Sphinx Client Message Interface Job Submitter Job Tracker Message

    5. Examples of how planning modules can be moved inside/outside the Sphinx Server/Client

    Sphinx Server/VDT Client UI Machine VDT Server Site Monitoring Service Resource Multi-threaded Control Process Job Execution Planner DAG Reducer DAG Tracker Job Predictor DAG Data Repl. Planner Job Admission Control Job Submitter Message Interface DAG Predictor DAG Admission Control Job Tracker Data Management Information Gatherer Repl. Loc. Serv. Condor-G/DAGMan Policies Accounting Resource Prop. Resource Hier. Resource Dist. Distance Entity-Name Entity Hier. Job-Entity Prop. Job DAG Rel. Data Rel. Appl. Job Tracking Grid Weather Data Warehouse Schedule Prediction Sphinx Client Message Interface Message

    6. Examples of how planning modules can be moved inside/outside the Sphinx Server/Client

    Sphinx Server/VDT Client VO Specific External Planning Service VDT Server Site Monitoring Service Resource Multi-threaded Control Process DAG Reducer DAG Tracker Job Predictor DAG Data Repl. Planner Job Admission Control Job Submitter Message Interface DAG Predictor DAG Admission Control Job Tracker Data Management Information Gatherer Repl. Loc. Serv. Condor-G/DAGMan Policies Accounting Resource Prop. Resource Hier. Resource Dist. Distance Entity-Name Entity Hier. Job-Entity Prop. Job DAG Rel. Data Rel. Appl. Job Tracking Grid Weather Data Warehouse Schedule Prediction Sphinx Client Message Interface Message Job Execution Planner

    7. Examples of how planning modules can be moved inside/outside the Sphinx Server/Client

More Related