1005 cern-active mq-v2

1005 cern-active mq-v2

• 1.
  James CaseyCERN ITDepartmentGrid Technologies GroupFUSE Community Day, London, 2010Using ActiveMQ at CERNfor the Large Hadron Collider
• 2.
  OverviewWhat we doat CERNCurrent ActiveMQ UsageMonitoring a distributed infrastructureLessons LearnedFuture ActiveMQ UsageBuilding a generic messaging service
• 3.
  LHC is avery large scientific instrument…CMSLHCbALICEATLASLarge Hadron Collider27 km circumferenceLake Geneva
• 4.
  … based onadvanced technology27 km of superconducting magnetscooled in superfluid helium at 1.9 K
• 5.
  What are welooking for?To answer fundamental questions about the construction of the universeWhy have we got mass ? (Higgs Boson)Search for a Grand Unified TheorySupersymmetryDark Matter, Dark EnergyAntimatter/matter asymmetry
• 6.
  This Requires…….1. Accelerators: powerful machines that accelerate particles to extremely high energies and then bring them into collision with other particles2. Detectors : gigantic instruments that record the resulting particles as they “stream” out from the point of collision.4. People : Only a worldwide collaboration of thousands of scientists, engineers, technicians and support staff can design, build and operate the complex “machines”3. Computers :to collect, store, distribute and analyse the vast amount of data produced by the detectors
• 7.
  View of theATLAS detector during constructionLength : ~ 46 mRadius : ~ 12 mWeight : ~ 7000 tons~108 electronic channels
• 8.
  A collisionat LHCBunches, each containing 100 billion protons, cross 40 million times a second in the centre of each experiment1 billion proton-proton interactions per second in ATLAS & CMS !Large Numbers of collisions per event ~ 1000 tracks stream into the detector every 25 ns a large number of channels (~ 100 M ch)  ~ 1 MB/25ns i.e. 40 TB/s !8
• 9.
  The Data AcquisitionCannotpossibly extract and record 40 TB/s. Essentially 2 stages of selection - dedicated custom designed hardware processors  40 MHz  100 kHz- then each ‘event’ sent to a free core in a farm of ~ 30k CPU-cores 100 kHz  few 100 Hz9
• 10.
  Tier 0 atCERN: Acquisition, First pass processing, Storage & DistributionIan.Bird@cern.ch10
• 11.
  First Beam day– 10 Sep. 2008
• 13.
  The LHC ComputingChallengeExperiments will produce about 15 Million Gigabytes (15 PB) of data each year (about 20 million CDs!)LHC data analysis requires a computing power equivalent to ~100,000 of today's fastest PC processors (140MSi2K)Analysis carried out at more than 140 computing centres 12 large centres for primary data management: CERN (Tier-0) and eleven Tier-1s38 federations of smaller Tier-2 centres
• 14.
  Solution: the GridUsethe Grid to unite computing resources of particle physics institutions around the worldThe World Wide Web provides seamless access to information that is stored in many millions of different geographical locationsThe Grid is an infrastructure that provides seamless access to computing power and data storage capacity distributed over the globe.It makes multiple computer centres look like a single system to the end-user.
• 15.
  LHC Computing Gridproject (WLCG)The grid is complexHighly distributedNo central controlLots of software in many languagesGrid middleware SLOC – 1.7M TotalC++ 850K, C 550K, SH 160K, Java 115K, Python 50K, Perl 35KExperiment code e.g. ATLAS – C++ 7M SLOCComplex services dependencies
• 16.
  My Problem -Monitoring the operational grid infrastructureTools for Operations and MonitoringBuild and run monitoring infrastructure for WLCGOperational tools for management of grid infrastructuresExamples:Configuration databaseHelpdesk/ ticketingMonitoringAvailability reportingEarly design decision: Use messaging as an integration framework
• 17.
  Open Source tothe coreDesign and develop services for Open Science based on:Open source softwareOpen protocolsFunded by a series of EU ProjectsEDG, EGEE, EGI.eu, EMIBacked by industry supportAll our code is open source and freely availableResults published in Open Access journals
• 18.
  Use Case– AvailabilityMonitoring and ReportingMonitoring of reliability and availability of European distributed computing infrastructure Data must be reliableDefinitive source of availability and accounting reportsDistributed operations modelGrid implies ‘cross-administrative domain’No root login !Global ticketingDistributed operations dashboards
• 19.
  SolutionDistributed monitoring basedon NagiosTied together with ActiveMQNetwork of 4 brokers in 3 countriesLinked to ticketing and alarm systemsMessage level signing + encryption for verification of identityUses STOMP for all communicationCode in Perl & pythonTopics with Virtual Consumers All persistent messagesTopic naming used for filtering and selection
• 20.
  Architecture
• 21.
  Component drilldown
• 22.
  Current Status16 nationallevel Nagios serversWill grow to ~40 in next 3 monthsClients distributed across 40 countries315 sites5K services500,000 test results/day3 consumers of full data stream to database for analysis and post processing40 distributed alarm dashboards with filtered feeds
• 23.
  Lessons (1)Just usingSTOMP is sub-optimalPros:Very simpleGood for lightweight clients in many languagesConsHard to write reliable long-lived clientsNo NACK, No heartbeatAmbiguities in the specificationContent-length and TextMessageContent-encodingNot really broker independent in practiceInterested in contributing to STOMP 1.1/2.0
• 24.
  Lessons (2) JMS Durableconsumer suckFragile in Network of BrokersMany problems fixed now by FUSEVirtual Topics solve the problemPros:Just like a queueCan monitor queue length, purge ConsIssues with selectorsStartup race conditions (solvable via config)
• 25.
  Lessons (3)Network ofbrokers seem attractivePros:It’s all a cloudClients connect anywhere and it “just works”Cons:It’s a very complicated area of codeOften you need to “ask the computer”Or a core ActiveMQ developerTrade off between resilience/scaling and complexity
• 26.
  Lessons (4)Know thecodeMost of it is very simpleEven for non-java developersIf you keep away from “java-ish” stuffJTA, XA, SpringPlugin architecture is very easy to work withMost things can be implemented by a pluginE.g. Monitoring, logging, restricting features, AuthN/AuthZDocs currently don’t explain everythingEspecially the interactions between plugins/features
• 27.
  Lessons (5)Stay inthe ballparkIf it’s not in tests:Think twice about using the feature in that way…Write a test for it !ExamplesSSL and network connectorsNetwork of Brokers with odd topologiesSTOMP/Openwiredifferences in feature support
• 28.
  Nagios for ActiveMQWeuse Nagios to monitor BrokersProducer/consumersUses jmx4perl to reduce JVM load on Nagios machineExposes JMX information as JSONSimple perl interface to write clientsGeneric nagios checksLooking how to make more available for the community
• 29.
  Broker MonitoringStandard OSinformationFilesystem full, processes running, socket counts, open file countsJMX for broker statisticsStore usage, JVM stats, inactive durable subs, queues with pending messagesJMX based scripts to manage brokersRemove unwanted advisoriesPurge queues with no consumers
• 30.
  Virtual Topic monitoringFulltesting of consumers from producers on all brokers in Network of BrokersConsumers instrumented to reply to test messagesAddressed to a single client-id on a topicSend message to topic in Reply-ToNagios sends messages to all brokers for a topicChecks they all come backUseful to check that all brokers in network are forwarding correctly
• 31.
  Nagios broker statuscheck
• 32.
  To the future– a generic messaging serviceMany concurrent applications …… in many languages …… over the WAN …… with little control over the usersNot a typical messaging problem ?
• 33.
  Isolate clients frommessaging via filesystemParticularly in the WANAlways assume messaging could be uncontactableKeeps “core” broker network smallAnd keeps complexity isolatedAllows all clients to use best language/protocol to talk to messaging Design thoughts – File Based Queue
• 34.
  Design Thoughts– AMQPstyle remote messagingQueues bound to broker nodes IP-like routing sends messages to destinationsClients connect to specific instancesBetter determinacy in networkEasier to manage explicit connections between brokers
• 35.
  SummaryActiveMQ is akey technology choice for operating and monitoring the WLCG grid infrastructureIt provides a scalable and adaptable platform for building a wide range of messaging based applicationsFUSE fits our model of open source software with industrial support
• 36.
  Thank you foryour attentionQuestions?