CHEP An update on the scalability limits of the Condor batch system. by Igor Sfiligoi for the UWisc and Red Hat Condor teams doing the real work

Transcription

1 CHEP 2010 An update on the scalability limits of the Condor batch system by Igor Sfiligoi for the UWisc and Red Hat Condor teams doing the real work CHEP 2010 Update on Condor scalability 1

2 What is Condor 1 Workload management system (a.k.a. Batch system) Used both for Dedicated compute farms Resource scavenging e.g. glideins Highly decentralized architecture Submit node Schedd Negotiator Collector Submit node Schedd Worker node Startd User job Worker node Startd User job CHEP 2010 Update on Condor scalability 2

3 What is Condor 2 Can also bridge WMS technologies Condor-G De-facto standard client in Open Science Grid GRAM Submit node CREAM Schedd EC2 CHEP 2010 Update on Condor scalability 3 LSF Condor

4 Limits reported at last CHEP Schedd Max 23k running jobs Hard limit due to excessive port use 100k jobs per day Schedd would not go faster than that Collector Max 25k glideins Soft limit... did not really push over that Condor-G GT2 speeds (to be finished) CHEP 2010 Update on Condor scalability 4

5 Pushing the schedd limit 1 Major limit was port usage Condor adds shared_port daemon A single listening port needed Paired with CCB (reversed connections) No ephemeral (i.e. client) ports used either Managed to reach 49k running jobs 49k Now firewall friendly Reached but not stable CHEP 2010 Update on Condor scalability 5

6 Pushing the schedd limit 2 The highest stable level reached is 41k 41k Required a node with 64GB of memory (thanks CMS T1@FNAL) Testing with 8h jobs (on glideins) Continuous submissions since dagman was used 1*min condor_q and condor_status Ran into scalability limits for my test tools! CHEP 2010 Update on Condor scalability 6

7 Pushing the schedd limit 3 Submit node still memory hungry About 1M per running job (for shadows) Exact amount depends if 32bit or 64bit binaries Testing on 32GB memory node 64bit binaries Max stable = 27k 32bit binaries Max stable = 37k 27k 37k CHEP 2010 Update on Condor scalability 7

8 Schedd transaction rate 1 Improvements that helped raise the transaction rate Reduced fsync frequency (was a disk killer!) Daemoncore optimizations Reduced protocol overhead between shadows and the schedd Shadow recycling Asynchronous matchmaking (schedd would be unresponsive for O(10s)) CHEP 2010 Update on Condor scalability 8

9 Schedd transaction rate 2 Now much easier to keep the resources busy with short jobs Condor 7.4.X Managed to keep 2.5k slots busy with 15s jobs 2.5k 2.5k Condor 7.5.X CHEP 2010 Update on Condor scalability 9

10 Schedd transaction rate 2 Using more HEP like lengths (1.5h average) Managed to run XX jobs in YY time Equivalent to ZZ job/day Still running the test CHEP 2010 Update on Condor scalability 10

11 Collector scalability 1 Improvements in this area Moving to new classads Richer semantics and about 20% faster Optimized classad insertion and removal Can reduce collector load a lot when using Condor's slot partitioning hard to quantify in typical HEP installation, since collector already uses very little CPU CHEP 2010 Update on Condor scalability 11

12 Collector scalability 2 Reached and maintained 50k without problems Using 2 schedds Using 10G of memory for collector(s)+negotiator (using collector tree with GSI authentication) 50k CHEP 2010 Update on Condor scalability 12

13 Negotiator improvements Indirect benefits from Schedd and Collector code changes Negotiation cycles now faster slots: 7056 jobs per auto cluster: 1 jobs matched: negotiator time: 664s negotiator time: 527s No need to limit the negotiation cycle either (required before in order not to kill the schedd) CHEP 2010 Update on Condor scalability 13

14 Something about bulk advertise To be written CHEP 2010 Update on Condor scalability 14

15 Rate (Hz) Condor-G improvements 1 Double queue One for submission, one for termination Eliminates long startup wait times after rapid Grid job termination Peak rates: Start: 3.5Hz End: 2.5Hz Jobs Running CHEP 2010 Update on Condor scalability 15

16 Condor-G improvements 2 GRAM5 support 1 Faster startup rates and termination rates 500 GRAM2 1k GRAM5 CHEP 2010 Update on Condor scalability 16

17 Condor-G improvements 3 GRAM5 support 2 Allows for both better usage of resources and better monitoring GRAM2 GRAM5 Finished after ~9h 30 minute jobs 3k slots total Finished after 6h 20mins CHEP 2010 Update on Condor scalability 17

18 Condor-G improvements 4 Many CREAM improvements in the last year Better speed Client initialed file transfer Better reliability CREAM finally usable from Condor-G Seems much faster than GRAM2/5 Old test... only partially significant due to high error rate But the best we have right now CHEP 2010 Update on Condor scalability 18

19 Conclusions To be written CHEP 2010 Update on Condor scalability 19

20 Acknowledgements Many thanks to the FNAL CMS Tier1 team for providing the test hardware This work has been partially funded by the US Department of Energy under Grant No. DE- FC02-06ER41436 subcontract No. 647F290 (OSG), and the US National Science Foundation under Grants No. PHY (DISUN), PHY (CMS Maintenance & Operations), and OCI (STCI). CHEP 2010 Update on Condor scalability 20