IPSL - PRODIGUER. Messaging Platform Design

Transcription

1 IPSL - PRODIGUER Messaging Platform Design

2 I - Platform Overview

3 Aujhourd hui

4 TGCC IDRIS CINES SSH IPSL Command Line

5 Demain ( = Aujhourd hui + Messaging Platform)

6 IPSL IPSL TGCC IDRIS CINES CNRM XXX MQ Relay MQ Relay MQ Relay MQ Relay MQ Relay msg msg msg msg msg MQ Cluster MQ Apps DB s API json IPSL Browser Command Line Desktop

7 Solution = Durable Message Queues AMQP : Advanced Message Queue Protocol RabbitMQ : AMQP Implementation 1. Erlang i.e. parallel & scalable; 2. Persistence i.e. durable; 3. Multi-OS i.e. ubiquitous; 4. Python py-amqlib / txampq

8

9 Performances

10 II - Message Flow

11 Message Flow 1: CC.NODE ---> CC.MQ Compute Computing Centre simul (fortran) args libligcm (ksh) args MQ client (c) tcp/ip (port = 5762) msg MQ Computing Centre (amqp + rabbit-mq + shovel)

12 Message Flow 2: CC.MQ ---> IPSL.MQ MQ Computing Centre (amqp + rabbit-mq + shovel) shovel (ini) failover??? msg ssl msg shovel (ini) MQ IPSL (amqps + rabbit-mq + shovel)

13 Message Flow 3: IPSL.MQ <---> IPSL.MQ-APP MQ IPSL (amqps + rabbit-mq + shovel) msg tcp/ip (port=5672) msg sim-mon metrics esg-f es-doc api smtp img primary secondary MQ IPSL

14 Message Flow 4: IPSL.MQ-APP ---> Other MQ IPSL primary secondary sim-mon metrics esg-f es-doc api smtp img tcp/ip https wss smtp??? IPSL ESG-F ES-DOC brows er IPSL File IPSL

15 Message Flow 4: IPSL.API ---> IPSL.FE IPSL (tornado http & ws server) sim-mon metrics xxx https/wss json json json sim-mon (javascript) metrics (javascript) xxx (javascript) browser (jquery, backbone, underscore, bootstrap)

16 III - Message Content

17 Message Content Header = AMPQ Basic Properties

18 Message Content Body = text blob

19 IV - Security

20 Security Transport Level = SSL

21 Security : Transport Level AMQPS = encrypted advanced messaging protocol Communication channel is encrypted via SSL SSL Certificate Authority = CNRS Example RabbitMQ server configuration: [ {rabbit, [ {ssl_listeners, [5671]}, # port number {ssl_options, [ {cacertfile,"/path/to/ca/cacert.pem"}, # certifcate authority.pem file {certfile,"/path/to/server/cert.pem"}, # server certificaye.pem file {keyfile,"/path/to/server/key.pem"}, # server certificate key file {verify,verify_peer}, # requires client to present a certificate {fail_if_no_peer_cert,false} # validates presented client certificate ]} ]} ].

22 Security : Transport Level Clients pass SSL credentials via connection params: ssl = flag indicating whether to enable SSL handshake ssl_options = dictionary of SSL options cacertfile : "/path/to/ca/cacert.pem" certfile : "/path/to/client/cert.pem" keyfile : "/path/to/client/key.pem" verify : verify_peer fail_if_no_peer_cert : true

23 Security : Transport Level SSL TODO Obtain CNRS root authority certificate Create signed server certificates Deploy server certificates Update Rabbit MQ server config file & restart Create signed client certificates (one per client?) Deploy client certificates alongside client binaries Update clients to update AMQP connection strings ISSUE certificate expiration management?

24 Security Authentication = SASL

25 Security : Authentication Authentication = SASL PLAIN SASL = Simple Authentication and Security Layer PLAIN = clear-text username & password SASL PLAIN = default for RabbitMQ servers and clients SASL PLAIN requires SSL to be secure Servers are setup with user accounts Server authentication is at the MQ exchange level Clients pass in credentials via amqps connection string Clients should pass base64 encoded SHA-256 encrypted credentials User must be created in target RabbitMQ server

26 Security Authorization = Permissions

27 Security : Authorization User names map to server permissions defined against: MQ Virtual hosts MQ Exchanges MQ Queues Virtual Host = prodiguer Exchanges = in, internal, out Queues = logger, main, smon etc Each user account (publisher / consumer) will have to assigned relevant permissions Permissions assigned using either RabbitMQ management plugin or.json config file

28 V - Message Durability

29 VI - Message Routing

30 VII RabbitMQ

31 RabbitMQ - Configuration Environment variables rabbitmq-env.config RabbitMQ initialises several environment variables Unlikely they need to be changed Server Configuration file rabbitmq.config No need to change general options Management options include a json file for initialising exchanges, queues, virtual hosts, users etc.

32 RabbitMQ - Operations RabbitMQ management plugin rabbitmq-plugins enable rabbitmq_management front-end API Can view connections, channels, exchanges, queues Can view, create, update, delete virtual hosts Can view, create, update, delete users Can enable / disable statistics HTTP REST service Can perform all tasks supported by front-end e.g. create a virtual host curl -i -u guest:guest -H "content-type:application/json" \ -XPUT Command Line Interface Wraps the API e.g. rabbitmqadmin V prodiguer list exchanges

33 VIII Further Info

34 Further Info AMPQ RabbitMQ Security - SSL ssl.html ssl.html Security SASL Simple_Authentication_and_Security_Layer Security Access Control control.html Configuration