3ńZc@sddlZddlZdefdYZdefdYZdefdYZdefd YZd efd YZd efd YZ defdYZ defdYZ dS(iNtRequestExceededExceptioncBseZdZRS(cCs<||_||_d||f}tt|j|dS(sError when requested amount exceeds what is allowed The request that raised this error should be retried after waiting the time specified by ``retry_time``. :type requested_amt: int :param requested_amt: The originally requested byte amount :type retry_time: float :param retry_time: The length in time to wait to retry for the requested amount s<Request amount %s exceeded the amount available. Retry in %sN(t requested_amtt retry_timetsuperRt__init__(tselfRRtmsg((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRs   (t__name__t __module__R(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRst RequestTokencBseZdZRS(sDA token to pass as an identifier when consuming from the LeakyBucket(RRt__doc__(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR (st TimeUtilscBseZdZdZRS(cCs tjS(sgGet the current time back :rtype: float :returns: The current time in seconds (ttime(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR .scCs tj|S(swSleep for a designated time :type value: float :param value: The time to sleep for in seconds (R tsleep(Rtvalue((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR 6s(RRR R (((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR -s tBandwidthLimitercBs eZddZedZRS(cCs1||_||_|dkr-t|_ndS(sLimits bandwidth for shared S3 transfers :type leaky_bucket: LeakyBucket :param leaky_bucket: The leaky bucket to use limit bandwidth :type time_utils: TimeUtils :param time_utils: Time utility to use for interacting with time. N(t _leaky_buckett _time_utilstNoneR (Rt leaky_buckett time_utils((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR@s   cCs2t||j||j}|s.|jn|S(sWraps a fileobj in a bandwidth limited stream wrapper :type fileobj: file-like obj :param fileobj: The file-like obj to wrap :type transfer_coordinator: s3transfer.futures.TransferCoordinator param transfer_coordinator: The coordinator for the general transfer that the wrapped stream is a part of :type enabled: boolean :param enabled: Whether bandwidth limiting should be enabled to start (tBandwidthLimitedStreamRRtdisable_bandwidth_limiting(Rtfileobjttransfer_coordinatortenabledtstream((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pytget_bandwith_limited_streamNs    N(RRRRtTrueR(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR?s RcBszeZdddZdZdZdZdZdZdZ d Z d Z d Z d Z d ZRS(iicCsj||_||_||_||_|dkr?t|_nt|_t|_ d|_ ||_ dS(s[Limits bandwidth for reads on a wrapped stream :type fileobj: file-like object :param fileobj: The file like object to wrap :type leaky_bucket: LeakyBucket :param leaky_bucket: The leaky bucket to use to throttle reads on the stream :type transfer_coordinator: s3transfer.futures.TransferCoordinator param transfer_coordinator: The coordinator for the general transfer that the wrapped stream is a part of :type time_utils: TimeUtils :param time_utils: The time utility to use for interacting with time iN( t_fileobjRt_transfer_coordinatorRRR Rt_bandwidth_limiting_enabledR t_request_tokent _bytes_seent_bytes_threshold(RRRRRtbytes_threshold((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRes        cCs t|_dS(s0Enable bandwidth limiting on reads to the streamN(RR(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pytenable_bandwidth_limitingscCs t|_dS(s1Disable bandwidth limiting on reads to the streamN(tFalseR(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRscCsd|js|jj|S|j|7_|j|jkrJ|jj|S|j|jj|S(shRead a specified amount Reads will only be throttled if bandwidth limiting is enabled. (RRtreadR!R"t_consume_through_leaky_bucket(Rtamount((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR&s  cCsvxo|jjsey*|jj|j|jd|_dSWqtk ra}|jj|j qXqW|jjdS(Ni( Rt exceptionRtconsumeR!R RRR R(Rte((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR's  cCs|jdS(s6Signal that data being read is being transferred to S3N(R$(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pytsignal_transferringscCs|jdS(s:Signal that data being read is not being transferred to S3N(R(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pytsignal_not_transferringscCs|jj|dS(N(Rtseek(Rtwhere((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR.scCs |jjS(N(Rttell(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR0scCs0|jr|jr|jn|jjdS(N(RR!R'Rtclose(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR1s cCs|S(N((R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyt __enter__scOs|jdS(N(R1(Rtargstkwargs((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyt__exit__sNi(RRRRR$RR&R'R,R-R.R0R1R2R5(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRds         t LeakyBucketcBsGeZddddZdZdZdZdZdZRS(cCst||_||_|dkr3t|_ntj|_||_|dkrft |_n||_ |dkrt |_ ndS(s9A leaky bucket abstraction to limit bandwidth consumption :type rate: int :type rate: The maximum rate to allow. This rate is in terms of bytes per second. :type time_utils: TimeUtils :param time_utils: The time utility to use for interacting with time :type rate_tracker: BandwidthRateTracker :param rate_tracker: Tracks bandwidth consumption :type consumption_scheduler: ConsumptionScheduler :param consumption_scheduler: Schedules consumption retries when necessary N( tfloatt _max_rateRRR t threadingtLockt_lockt _rate_trackertBandwidthRateTrackert_consumption_schedulertConsumptionScheduler(Rtmax_rateRt rate_trackertconsumption_scheduler((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRs      cCs|jq|jj}|jj|r>|j|||S|j||rf|j|||n|j||SWdQXdS(scConsume an a requested amount :type amt: int :param amt: The amount of bytes to request to consume :type request_token: RequestToken :param request_token: The token associated to the consumption request that is used to identify the request. So if a RequestExceededException is raised the token should be used in subsequent retry consume() request. :raises RequestExceededException: If the consumption amount would exceed the maximum allocated bandwidth :rtype: int :returns: The amount consumed N( R;RR R>t is_scheduledt,_release_requested_amt_for_scheduled_requestt_projected_to_exceed_max_ratet!_raise_request_exceeded_exceptiont_release_requested_amt(Rtamtt request_tokenttime_now((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR*s  cCs"|jj||}||jkS(N(R<tget_projected_rateR8(RRHRJtprojected_rate((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRE scCs |jj||j||S(N(R>tprocess_scheduled_consumptionRG(RRHRIRJ((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRDs cCsD|t|j}|jj|||}td|d|dS(NRR(R7R8R>tschedule_consumptionR(RRHRIRJtallocated_timeR((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRFs  cCs|jj|||S(N(R<trecord_consumption_rate(RRHRJ((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRGsN( RRRRR*RERDRFRG(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR6s     R?cBs,eZdZdZdZdZRS(cCsi|_d|_dS(s*Schedules when to consume a desired amountiN(t _tokens_to_scheduled_consumptiont _total_wait(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR!s cCs ||jkS(sIndicates if a consumption request has been scheduled :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. (RQ(Rttoken((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRC&scCs4|j|7_i|jd6|d6|j|<|jS(sSchedules a wait time to be able to consume an amount :type amt: int :param amt: The amount of bytes scheduled to be consumed :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. :type time_to_consume: float :param time_to_consume: The desired time it should take for that specific request amount to be consumed in regardless of previously scheduled consumption requests :rtype: float :returns: The amount of time to wait for the specific request before actually consuming the specified amount. t wait_durationttime_to_consume(RRRQ(RRHRSRU((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRN/s  cCs3|jj|}t|j|dd|_dS(sProcesses a scheduled consumption request that has completed :type token: RequestToken :param token: The token associated to the consumption request that is used to identify the request. RUiN(RQtpoptmaxRR(RRStscheduled_retry((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRMIs(RRRRCRNRM(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR? s  R=cBsGeZddZedZdZdZdZdZRS(g?cCs||_d|_d|_dS(sTracks the rate of bandwidth consumption :type a: float :param a: The constant to use in calculating the exponentional moving average of the bandwidth rate. Specifically it is used in the following calculation: current_rate = alpha * new_rate + (1 - alpha) * current_rate This value of this constant should be between 0 and 1. N(t_alphaRt _last_timet _current_rate(Rtalpha((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRVs  cCs|jdkrdS|jS(smThe current transfer rate :rtype: float :returns: The current tracked transfer rate gN(RZRR[(R((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyt current_ratefscCs#|jdkrdS|j||S(sZGet the projected rate using a provided amount and time :type amt: int :param amt: The proposed amount to consume :type time_at_consumption: float :param time_at_consumption: The proposed time to consume at :rtype: float :returns: The consumption rate if that amt and time were consumed gN(RZRt*_calculate_exponential_moving_average_rate(RRHttime_at_consumption((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRKqs cCsG|jdkr%||_d|_dS|j|||_||_dS(sRecord the consumption rate based off amount and time point :type amt: int :param amt: The amount that got consumed :type time_at_consumption: float :param time_at_consumption: The time at which the amount was consumed gN(RZRR[R^(RRHR_((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyRPs   cCs+||j}|dkr#tdS||S(Nitinf(RZR7(RRHR_t time_delta((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyt_calculate_rates   cCs/|j||}|j|d|j|jS(Ni(RbRYR[(RRHR_tnew_rate((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR^s( RRRtpropertyR]RKRPRbR^(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyR=Us     ( R R9t ExceptionRtobjectR R RRR6R?R=(((s8/usr/lib/python2.7/site-packages/s3transfer/bandwidth.pyt s  %jR5