CHEP An update on the scalability limits of the Condor batch system

Transcription

1 CHEP 2010 An update on the scalability limits of the Condor batch system by Igor Sfiligoi for the UWisc Condor and Red Hat MRG Grid teams doing the real work CHEP 2010 Update on Condor scalability 1

2 What is Condor 1 Workload management system (a.k.a. Batch system) Used both for Dedicated compute farms Resource scavenging e.g. glideins Highly decentralized architecture Submit node Schedd Negotiator Collector Submit node Schedd Worker node Startd User job Worker node Startd User job CHEP 2010 Update on Condor scalability 2

3 What is Condor 2 Can also bridge WMS technologies Condor-G De-facto standard client in Open Science Grid GRAM Submit node CREAM Schedd EC2 CHEP 2010 Update on Condor scalability 3 LSF Condor

4 Limits reported at last CHEP Schedd Max 23k running jobs Hard limit due to excessive port use 100k jobs per day Schedd would not go faster than that Collector Max 25k glideins Soft limit... did not really push over that Condor-G GT2: FIFO job handling CHEP 2010 Update on Condor scalability 4

5 Pushing the schedd limit 1 Major limit was port usage Condor adds shared_port daemon A single listening port needed Paired with CCB (reversed connections) No ephemeral (i.e. client) ports used either Managed to reach 49k running jobs 49k Now firewall friendly Reached but not stable CHEP 2010 Update on Condor scalability 5

6 Pushing the schedd limit 2 The highest stable level reached is 41k 41k Required a node with 64GB of memory (thanks CMS T1@FNAL) Testing with 8h jobs (on glideins) Continuous submissions since dagman was used 1*min condor_q and condor_status Ran into scalability limits for my test tools! CHEP 2010 Update on Condor scalability 6

7 Pushing the schedd limit 3 Submit node still memory hungry About 1M per running job (for shadows) Exact amount depends if 32bit or 64bit binaries Testing on 32GB memory node 64bit binaries Max stable = 27k 32bit binaries Max stable = 37k 27k 37k CHEP 2010 Update on Condor scalability 7

8 Schedd transaction rate 1 Improvements that helped raise the transaction rate Reduced fsync frequency (was a disk killer!) Daemoncore optimizations Reduced protocol overhead between shadows and the schedd Shadow recycling Asynchronous matchmaking (schedd would be unresponsive for O(10s)) CHEP 2010 Update on Condor scalability 8

9 Schedd transaction rate 2 Now much easier to keep the resources busy with short jobs Condor 7.4.X Managed to keep 2.5k slots busy with 15s jobs Soft limit 2.5k 2.5k Condor 7.5.X CHEP 2010 Update on Condor scalability 9

10 Schedd transaction rate 2 Using more HEP like lengths (1.5h average) Kept 25k slots busy no problems 25k Soft limit 275 jobs/minute Equivalent to 400k jobs/day Jobs/minute :30 00:00 00:30 01:00 01:30 02:00 02:30 Start rate End rate CHEP 2010 Update on Condor scalability 10

11 Schedd transaction rate 3 Pushing the limits: 30 min jobs Soft limit Still able to keep 25k slots busy 25k If the disk can keep up This test on ramdisk 870 jobs/minute Equivalent to 1.2M jobs/day Jobs/minute :30 23:00 23:30 00:00 00:30 Start rate End rate CHEP 2010 Update on Condor scalability 11

12 Collector scalability 1 Improvements in this area Moving to new classads Richer semantics and about 20% faster Optimized classad insertion and removal Can reduce collector load a lot when using Condor's slot partitioning Hard to quantify in typical HEP installation, since collector already uses very little CPU But HTPC may need this CHEP 2010 Update on Condor scalability 12

13 Collector scalability 2 Reached and maintained 90k without problems Using 5 schedds Soft limit Using 16G of memory for collector(s)+negotiator (using collector tree with GSI authentication) 70k 90k 70k limit with CCBs (fd limited) CHEP 2010 Update on Condor scalability 13

14 Negotiator improvements Indirect benefits from Schedd and Collector code changes Negotiation cycles now faster Test setup 1 Slots: 7056 Jobs per auto cluster: 1 Jobs matched: neg. time: 664s neg. time: 527s Test setup 2 Slots: 50k Unclaimed: 1k Preemption disabled Condor schedds Match rate: 50 jobs/s Test setup , single schedd 1.6k slots in 150s No need to limit the negotiation cycle either (required before in order not to kill the schedd) 15k CHEP 2010 Update on Condor scalability 14

15 Client improvements condor_advertise Added bulk ClassAd uploading Reduced CMS glideinwms frontend cycle from ~2 minutes to ~5 seconds TCP updates Adds reliability Especially useful in bulk mode condor_q/condor_status Reduced network traffic Faster response times CHEP 2010 Update on Condor scalability 15

16 Rate (Hz) Condor-G improvements 1 Double queue One for submission, one for termination Eliminates long startup wait times after rapid Grid job termination Peak rates about the same: Start: 0.5Hz End: 0.3Hz Jobs Running CHEP 2010 Update on Condor scalability 16

17 Condor-G improvements 2 GRAM5 support 1 Faster startup rates and termination rates 500 GRAM2 1k GRAM5 CHEP 2010 Update on Condor scalability 17

18 Condor-G improvements 3 GRAM5 support 2 Allows for both better usage of resources and better monitoring GRAM2 GRAM5 Finished after ~9h 30 minute jobs 3k slots total Finished after 6h 20mins CHEP 2010 Update on Condor scalability 18

19 Condor-G improvements 4 Many CREAM improvements in the last year Better speed Client initiated file transfer Better reliability CREAM finally usable from Condor-G Seems much faster than GRAM2/5 Old test... only partially significant due to high error rate But the best we have right now CHEP 2010 Update on Condor scalability 19

20 Conclusions Condor has undergone many small incremental scalability related improvements Together they make a big difference All CMS requirements met Condor now also much more Firewall friendly Condor-G added support for several new Grid types CREAM was especially long desired CHEP 2010 Update on Condor scalability 20

21 Acknowledgements Many thanks to the FNAL CMS Tier1 team for providing the test hardware This work has been partially funded by the US National Science Foundation under Grants No (Condor), PHY (DISUN), PHY (CMS Maintenance & Operations), and OCI (STCI), and the US Department of Energy under Grant No. DE- FC02-06ER41436 subcontract No. 647F290 (OSG) CHEP 2010 Update on Condor scalability 21