08 – RabbitMQ

●
[Link]
part1-rabbitmq-for-beginners-what-is-
[Link]
– Also parts 2, 3 and 4
●
[Link]
●
[Link]
 Communication guarantees
●
Reliable ●
How well does UDP fare w/ the list on the
left? TCP?
●
Secure
●
Common needs led enterprise developers
●
Durable to adopt Message Oriented Middleware
●
Asynchronous delivery (MOM)
●
MOM needs to be everywhere to be
●
Scalable
useful
●
Available – dominant solutions are proprietary
●
Routable (Multicast etc) – too expensive for everyday use (Cloud-scale)
– they don’t interoperate
●
Addressable (naming)
– has resulted in lots of ad-hoc home-brew
●
Semantic guarantees ●
Middleware Hell
●
Flexible – 100’s of applications
●
Manageable – 10,000’s of links
– every connection different
●
Filterable
– massive waste of effort
●
Transformable
 AMQP
Message Oriented Middleware
●
Set up by JPMorgan in 2006
– Goal to make Message Oriented Middleware
pervasive
– Make it practical, useful, interoperable
– Bring together users and vendors to solve the problem
●
AMQP is an “Internet Protocol for Business
Messaging”
– so end users feel a connection to the technology.
●
AMQP aspires to define MOM
AMQP Working Group
Strong Governance
Protocol Products
Credit-Suisse, JPMorgan,
Deutsche Borse Systems, Goldman
iMatix Red Hat Apache Rabbit Cisco
Sachs, TWIST, 29West, Envoy,
Novell, Tervela, WSO2,..

Community
Feedback

iMatix Red Hat MRG Apache Rabbit MQ Cisco AON

End Users OpenAMQ Qpid
 AMQP Working Group – Strong Governance
Protocol Products
Credit-Suisse, JPMorgan,
Deutsche Borse Systems, Goldman
iMatix Red Hat Apache Rabbit Cisco
Sachs, TWIST, 29West, Envoy,
Novell, Tervela, WSO2,..

Community
Feedback

iMatix Red Hat MRG Apache Rabbit MQ Cisco AON

End Users OpenAMQ Qpid

AMQP Working Group

controls the standard Diverse products implement the standard

Page 5 [Link]
 RabbitMQ
●
Message-queueing software
– message broker or queue
manager.
●
queues can be defined
●
applications may connect to the
queue and transfer a message
onto it.
 AMQP – Requirements
UBIQUITOUS AND PERVASIVE
●
Open internet protocol standard
●
Binary WIRE protocol so that it can be ubiquitous, fast,
embedded
●
Unambiguous core functionality for business message
routing and delivery within Internet infrastructure
●
Scalable, so that it can be a basis for high performance
fault-tolerant lossless messaging infrastructure, i.e
without requiring other messaging technology
●
Fits into existing enterprise messaging applications
environments in a practical way
 AMQP – Requirements
SAFETY
●
Infrastructure for a secure and trusted global
transaction network
– Consisting of business messages that are tamper proof
– Supporting message durability independent of receivers
being connected
●
Transport business transactions of any financial
value
●
Sender and Receiver are mutually agreed upon
counter parties
– No possibility for injection of Spam
 AMQP – Requirements
FIDELITY
●
Well-stated message queuing and delivery semantics
covering
– at-most-once
– at-least-once
– and once-and-only-once (e.g. 'reliable’, ‘assured’, ‘guaranteed’)
●
Well-stated message ordering semantics describing what
a sender can expect
– a receiver to observe
– a queue manager to observe
●
Well-stated reliable failure semantics
– so exceptions can be managed
 AMQP – Requirements
UNIFIED
●
AMQP aspires to be the sole business messaging tool for organizations
●
Global addressing standardizing end-to-end delivery across any network
scope
●
Any AMQP client can initiate communication with, and then communicate
with, any AMQP broker over TCP/IP
●
Optionally, extendable to alternate transports via negotiation
●
Provide a core set of messaging patterns via a single manageable protocol:
– asynchronous directed messaging
– request/reply, publish/subscribe
– store-and-forward
●
Provide for Hub-and-Spoke messaging topology within and across business
boundaries
●
Provide for Hub-to-Hub message relay across business boundaries through
enactment of explicit agreements between broker authorities
 AMQP – Requirements
INTEROPERABILITY
●
Multiple stable and interoperating broker implementations
– Each with a completely independent provenance (min. 2 to move to
Final)
– Each broker implementation is conformant with the specification, for all
mandatory functionality, including fidelity semantics
●
Stable core (client-broker) wire protocol so that brokers do not
require upgrade during 1.x feature evolution: Any 1.x client will
work with any 1.y broker if y >= x
●
Stable extended (broker-broker) wire protocol so that brokers do
not require upgrade during 1.x feature evolution: Any two brokers
versions 1.x, 1.y can communicate using protocol 1.x if x<y
●
Layered architecture, so features & network transports can be
independently extended by separated communities of use
 AMQP – Requirements
MANAGEABLE
●
Decentralized deployment with independent local
governance
●
Intermediated:
– supports routing and relay management,
– traffic flow management
– quality of service management
●
Interaction with the message delivery system is
possible, sufficient to integrate with prevailing
business operations that administer messaging
systems using management standards.
 AMQP – Requirements
Banking Security Requirements
●
SSL support
●
Service Context (incl. Security Context):
– A standard Message property for for propagation of Security Tokens
●
Support for carrying Security Tokens:
– Principal-ID, SAML, Kerberos ticket, etc.
– Carried within the Service Context in the Message
●
Unique Security Token per Message:
– Enables multiplexing of different Security Contexts on a given messaging session (e.g. for
proxying)
●
Hash and sign of Message (including Security Context)
– Assure authenticity of the contents in addition to encryption (content verified by final-
destination).
– Full-path privacy for business transactions that might pass through a number of hubs enroute
to the final destination, where you would not want to have the exposed content of the
message sitting in some queue and disk along the way.
●
Chains of trust within trust realms - optional
 AMQP Functionality
●
AMQP is Message Oriented Middleware (MOM)
●
Transfers application data units from senders to receivers –
layer 7
●
Expectations
– message transfer is via trusted intermediaries
– messages will be delivered unchanged
– security
– Applications can be separated by (large amounts) of space and time
●
Abstract from the underlying technology
– Physical network limits should be hidden (message size, node
location)
– Technology concerns should be hidden (platform, language, OS
 AMQP Functionality
●
AMQP is Message Oriented Middleware (MOM)
●
The intermediaries offer various delivery options,
– as defined by either the sender or the receiver (s)
●
The intermediaries provided various defined
qualities of service for the sender and the
receiver (s)
●
Provide stability and backwards compatibility
(10yrs+)
 AMQP Architecture
●
Broker
– Applications Connect to a Broker to participate in the AMQP network
– Receive messages from publishers
●
applications that publish them, also known as producers
– Routes messages to consumers
●
applications that process them).
– The Connection is used to establish a Session
●
Sessions provide state between Connections, establish identity, ease failover
– Connections are further subdivide into Channels
●
Multiple threads of control within an Application can share one Connection
●
Queues
– Applications logic interacts ONLY with Queues
– Queues have well known Names == Addressable
– Applications do not need to know how messages get in/out of Queues
– Applications will assign implied semantics to Queues (e.g. “StockOrderQueue”)
 AMQP Architecture
●
Links
– Links move Messages between Queues and/or Applications
– Contain Routing and Predicate Evaluation Logic – similar to
Complex Event Processing
●
Exchange provided the core routing
– Exchange == Input Queue -> Links -> Output Queues
●
Input Queue provide an abstract Address
●
Links contain a Function to evaluate Messages
●
Function parameterised by the Link predicates
●
Output Queue = Link( message, predicates)
– Moves complexity from Clients to Brokers
●
AMQP Architecture
AMQP Architecture

Client Client

Client Broker Internet Broker Client

Client Client
Firewalls

Broker
Client

Client

Client
 AMQP Exchanges
●
Exchanges are AMQP entities where messages are sent.
●
Exchanges take a message and route it into zero or more
queues.
●
The routing algorithm used depends on the exchange
type and rules called bindings.
●
AMQP 0-9-1 provide four ●
exchanges are declared with a number of
attributes
exchange types: – Name
– Direct exchange – Durability (exchanges survive broker restart)
– Fanout exchange – Auto-delete (exchange is deleted when last
queue is unbound from it)
– Topic exchange – Arguments (optional, used by plugins and
broker-specific features)
– Headers exchange
 AMQP - Programmable Protocol
●
AMQP is a Programmable Protocol
●
AMQP entities and routing schemes are primarily defined by
applications themselves, not a broker administrator.
●
Provision is made for protocol operations that declare queues and
exchanges, define bindings between them, subscribe to queues and so
on.
– Application dependent
– Application defined
●
This gives application developers a lot of freedom but also requires
them to be aware of potential definition conflicts.
●
Applications
– declare the AMQP 0-9-1 entities that they need
– define necessary routing schemes
– May delete AMQP 0-9-1 entities when they are no longer used.
 AMQP Queue Delivery
●
Point-to-point AMQP
AMQPBroker
Broker
Client
Client Link Tail Entry 3
●
Producer
Producer Session Entry 2
Entry 1 Head Link Client
Client
Queue (source) Session Consumer
Consumer
Consummer and producer agree on queue -Persistent

Only “Source” queue is required and can be

read directly by consumer over Link.

●
Abstracted Point-to-point Queue
AMQP
AMQPBroker
Broker

Client
Client Link Tail Entry 3 Tail

Producer Entry 2 Link Entry 2

Producer Session
Entry 1 Head Entry 1 Head

All messages are automatically forward on to the Queue (Source)

-Persistent
Queue (worker)
-Persistent
real worker queue.
Allows internal topology to change without the
outside world seeing
 AMQP Queue Delivery
●
Load-Balanced Point-to-point
AMQP
AMQPBroker
Broker
Client Link Entry 3 Link Client
Client
Client Tail
Consumer
Producer
Producer Session Entry 2
1 Head
Session Consumer
Entry 1 or 2 ?
Queue (source)
-Persistent Link Client
Client
Session Consumer
Consumer
 AMQP Queue Delivery
●
Dynamic (non-persistent) Pub/Sub
●

Messages are “garbage collected” in an

implementation specific manner after delivery.
AMQP makes some guarantees about how
long messages are valid for.

●
Durable (persistent) Pub/Sub
 How to use AMQP
●
Install Server(s)
– For examploe rabitmq
●
Develop consummers/producer
●
Configure
– server queues (in/out)
– exchanges
 RabbitMQ examples
[Link]

●
Point to point ●
Routing

●
Topics
●
Pub/Sub

●
Work Queues ●
RPC
 When to use
●
Allows fast respose instead of being forced to
perform resource-heavy procedures
●
Is also good when you want to distribute a
message to multiple recipients for consumption
●
Is also good when you want to balance loads
between workers.
●
Is good to add low coupling between the sender
and the receiver.
 When to use

●
The consumer can take a message of the queue and start the
processing of the PDF
– at the same time as the producer is queueing up new messages on the queue.
●
The consumer can be on a totally different server than the publisher,
– or they can be located on the same server.
●
The request can be created in one programming language and
handled in another programming language
 Concepts
●
Producer: Application that sends the messages.
●
Consumer: Application that receives the messages.
●
Queue: Buffer that stores messages.
●
Message: Information that is sent from the producer to a consumer through
RabbitMQ.
●
Connection: A connection is a TCP connection between your application and the
RabbitMQ broker.
●
Channel: A channel is a virtual connection inside a connection. When you are
publishing or consuming messages from a queue - it's all done over a channel.
●
Exchange: Receives messages from producers and pushes them to queues
depending on rules defined by the exchange type. To receive messages, a queue
needs to be bound to at least one exchange.
●
Binding: A binding is a link between a queue and an exchange.
●
Routing key: The routing key is a key that the exchange looks at to decide how to
route the message to queues. The routing key is like an address for the message.
 Setup
●
Install RabbitMQ
– zypper install rabbitmq-server
– zypper install rabbitmq-server-plugins
●
Enable plugins
– rabbitmq-plugins enable rabbitmq_management
●
Run rabbitmq
– Rabbitmq-server
●
Access console
– [Link] (guest / guest)
●
Install client
– pip3 install pika