3ńZc@s ddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddl mZdZdd(Zdd)ZejeZd d Zd Zd ZddZdZdZdZdefdYZdefdYZdefdYZ defdYZ!defdYZ"defdYZ#defdYZ$d e%fd!YZ&d"efd#YZ'd$e'fd%YZ(d&efd'YZ)dS(*iN(t defaultdict(t rename_file(tseekablei'iiiiicCsdjdt|DS(Ntcss!|]}tjtjVqdS(N(trandomtchoicetstringt hexdigits(t.0t_((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pys 's(tjointrange(t num_digits((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytrandom_file_extension&scKs2|dkr.t|jdr.|jjndS(Nt PutObjectt UploadParttsignal_not_transferring(RR(thasattrtbodyR(trequesttoperation_nametkwargs((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR*s cKs2|dkr.t|jdr.|jjndS(NRRtsignal_transferring(s PutObjects UploadPart(RRR(RRR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR0s cCsd||}||dkrBd}|dk rPt|d}qPn||d}d||f}|S(sCalculate the range parameter for multipart downloads/copies :type part_size: int :param part_size: The size of the part :type part_index: int :param part_index: The index for which this parts starts. This index starts at zero :type num_parts: int :param num_parts: The total number of parts in the transfer :returns: The value to use for Range parameter on downloads or the CopySourceRange parameter for copies iRs bytes=%s-%sN(tNonetstr(t part_sizet part_indext num_partst total_sizet start_ranget end_ranget range_param((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytcalculate_range_parameter6s  cCseg}xX|jjjD]G}d|}t||r|jtjt||d|qqW|S(s Retrieves callbacks from a subscriber :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future the subscriber is associated to. :type callback_type: str :param callback_type: The type of callback to retrieve from the subscriber. Valid types include: * 'queued' * 'progress' * 'done' :returns: A list of callbacks for the type specified. All callbacks are preinjected with the transfer future. ton_tfuture(tmetat call_argst subscribersRtappendt functoolstpartialtgetattr(ttransfer_futuret callback_typet callbackst subscribert callback_name((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt get_callbacksSs cCs+|r'x|D]}|d|q WndS(sCalls all progress callbacks :param callbacks: A list of progress callbacks to invoke :param bytes_transferred: The number of bytes transferred. This is passed to the callbacks. If no bytes were transferred the callbacks will not be invoked because no progress was achieved. It is also possible to receive a negative amount which comes from retrying a transfer request. tbytes_transferredN((R,R0tcallback((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytinvoke_progress_callbacksqs  cCs@i}x3|jD]%\}}||kr|||(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR:stFunctionContainercBs)eZdZdZdZdZRS(sAn object that contains a function and any args or kwargs to call it When called the provided function will be called with provided args and kwargs. cOs||_||_||_dS(N(t_funct_argst_kwargs(R<tfunctargsR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>s  cCsd|j|j|jfS(Ns'Function: %s with args %s and kwargs %s(RBRCRD(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt__repr__scCs|j|j|jS(N(RBRCRD(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt__call__s(R?R@t__doc__R>RGRH(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRAs  tCountCallbackInvokercBsAeZdZdZedZdZdZdZRS(sAn abstraction to invoke a callback when a shared count reaches zero :param callback: Callback invoke when finalized count reaches zero cCs.tj|_||_d|_t|_dS(Ni(t threadingtLockt_lockt _callbackt_counttFalset _is_finalized(R<R1((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>s  cCs|j |jSWdQXdS(N(RMRO(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt current_counts cCs;|j,|jr"tdn|jd7_WdQXdS(sIncrement the count by ones;Counter has been finalized it can no longer be incremented.iN(RMRQt RuntimeErrorRO(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt increments    cCsf|jW|jdkr(tdn|jd8_|jr\|jdkr\|jnWdQXdS(sDecrement the count by oneis,Counter is at zero. It cannot dip below zeroiN(RMRORSRQRN(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt decrements  cCs9|j*t|_|jdkr/|jnWdQXdS(sFinalize the counter Once finalized, the counter never be incremented and the callback can be invoked once the count reaches zero iN(RMtTrueRQRORN(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytfinalizes  ( R?R@RIR>tpropertyRRRTRURW(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRJs   tOSUtilscBsJeZdZdZddZdZdZdZdZ RS(cCstjj|S(N(tostpathtgetsize(R<tfilename((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt get_file_sizescCstj||||dtS(Ntenable_callbacks(t ReadFileChunkt from_filenameRP(R<R]t start_bytetsizeR,((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytopen_file_chunk_readers  c Cs"t|||d|dtd|S(NR,R_tclose_callbacks(R`RP(R<tfileobjt chunk_sizetfull_file_sizeR,Re((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt#open_file_chunk_reader_from_fileobjs  cCs t||S(N(topen(R<R]tmode((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRjscCs)ytj|Wntk r$nXdS(s+Remove a file, noop if file does not exist.N(RZtremovetOSError(R<R]((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt remove_files cCst||dS(N(R(R<tcurrent_filenamet new_filename((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCsxtjj|stStj|j}tj|r;tStj|rNtStj |ratStj |rttStS(sChecks to see if a file is a special UNIX file. It checks if the file is a character special device, block special device, FIFO, or socket. :param filename: Name of the file :returns: True if the file is a special file. False, if is not. ( RZR[texistsRPtstattst_modetS_ISCHRRVtS_ISBLKtS_ISFIFOtS_ISSOCK(tclsR]Rk((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytis_special_files N( R?R@R^RdRRiRjRnRRy(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRYs     tDeferredOpenFilecBsteZddedZdZedZd dZdZ dZ dZ d Z d Z d ZRS( itrbcCs1||_d|_||_||_||_dS(sA class that defers the opening of a file till needed This is useful for deferring opening of a file till it is needed in a separate thread, as there is a limit of how many open files there can be in a single thread for most operating systems. The file gets opened in the following methods: ``read()``, ``seek()``, and ``__enter__()`` :type filename: str :param filename: The name of the file to open :type start_byte: int :param start_byte: The byte to seek to when the file is opened. :type mode: str :param mode: The mode to use to open the file :type open_function: function :param open_function: The function to use to open the file N(t _filenameRt_fileobjt _start_bytet_modet_open_function(R<R]RbRkt open_function((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>!s     cCsV|jdkrR|j|j|j|_|jdkrR|jj|jqRndS(Ni(R}RRR|RR~tseek(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt_open_if_needed<scCs|jS(N(R|(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytnameBscCs|j|jj|S(N(RR}tread(R<tamount((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRFs cCs|j|jj|dS(N(RR}twrite(R<tdata((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRJs cCs|j|jj|dS(N(RR}R(R<twhere((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRNs cCs#|jdkr|jS|jjS(N(R}RR~ttell(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRRscCs|jr|jjndS(N(R}tclose(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRWs cCs|j|S(N(R(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt __enter__[s cOs|jdS(N(R(R<RFR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt__exit___sN(R?R@RjR>RRXRRRRRRRRR(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRz s       R`cBseZdeddZededZdZddZdZ dZ dZ dZ dZ d Zd Zd Zd Zd ZdZRS(cCs||_|jj|_|j|jd|d|jd||_d|_||_|dkrog|_n||_||_ |dkr||_ ndS(s Given a file object shown below:: |___________________________________________________| 0 | | full_file_size |----chunk_size---| f.tell() :type fileobj: file :param fileobj: File like object :type chunk_size: int :param chunk_size: The max chunk size to read. Trying to read pass the end of the chunk size will behave like you've reached the end of the file. :type full_file_size: int :param full_file_size: The entire content length associated with ``fileobj``. :type callbacks: A list of function(amount_read) :param callbacks: Called whenever data is read from this object in the order provided. :type enable_callbacks: boolean :param enable_callbacks: True if to run callbacks. Otherwise, do not run callbacks :type close_callbacks: A list of function() :param close_callbacks: Called when close is called. The function should take no arguments. trequested_sizeRbtactual_file_sizeiN( R}RR~t_calculate_file_sizet_sizet _amount_readt _callbacksRt_callbacks_enabledt_close_callbacks(R<RfRgRhR,R_Re((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>ds#        cCsJt|d}|j|tj|jj}||||||S(s[Convenience factory function to create from a filename. :type start_byte: int :param start_byte: The first byte from which to start reading. :type chunk_size: int :param chunk_size: The max chunk size to read. Trying to read pass the end of the chunk size will behave like you've reached the end of the file. :type full_file_size: int :param full_file_size: The entire content length associated with ``fileobj``. :type callbacks: function(amount_read) :param callbacks: Called whenever data is read from this object. :type enable_callbacks: bool :param enable_callbacks: Indicate whether to invoke callback during read() calls. :rtype: ``ReadFileChunk`` :return: A new instance of ``ReadFileChunk`` R{(RjRRZtfstattfilenotst_size(RxR]RbRgR,R_tft file_size((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRas cCs||}t||S(N(tmin(R<RfRRbRtmax_chunk_size((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs cCs|dkr|j|j}nt|j|j|}|jj|}|jt|7_|jdk r|jrt |jt|n|S(N( RRRRR}RtlenRRR2(R<Rtamount_to_readR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs cCs0|jt|jdr,|jjndS(NR(tenable_callbackRR}R(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs cCs0|jt|jdr,|jjndS(NR(tdisable_callbackRR}R(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs cCs t|_dS(N(RVR(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCs t|_dS(N(RPR(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCsY|jj|j||jdk rL|jrLt|jd||jn||_dS(NR0(R}RR~RRRR2R(R<R((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs cCsG|jdk r6|jr6x|jD] }|q"Wn|jjdS(N(RRRR}R(R<R1((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCs|jS(N(R(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCs|jS(N(R(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt__len__scCs|S(N((R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscOs|jdS(N(R(R<RFR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCs tgS(N(titer(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt__iter__sN(R?R@RRVR>t classmethodRaRRRRRRRRRRRRR(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR`cs"0           tStreamReaderProgresscBs#eZdZddZdZRS(s<Wrapper for a read only stream that adds progress callbacks.cCs.||_||_|dkr*g|_ndS(N(t_streamRR(R<tstreamR,((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>s   cOs/|jj||}t|jt||S(N(RRR2RR(R<RFRR8((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRsN(R?R@RIRR>R(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs tNoResourcesAvailablecBseZRS((R?R@(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR st TaskSemaphorecBs&eZdZedZdZRS(cCstj||_dS(sqA semaphore for the purpose of limiting the number of tasks :param count: The size of semaphore N(RKt Semaphoret _semaphore(R<tcount((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>scCs9tjd||jj|s5td|ndS(sAcquire the semaphore :param tag: A tag identifying what is acquiring the semaphore. Note that this is not really needed to directly use this class but is needed for API compatibility with the SlidingWindowSemaphore implementation. :param block: If True, block until it can be acquired. If False, do not block and raise an exception if cannot be aquired. :returns: A token (can be None) to use when releasing the semaphore s Acquiring %ssCannot acquire tag '%s'N(tloggertdebugRtacquireR(R<ttagtblocking((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs cCs(tjd||f|jjdS(stRelease the semaphore :param tag: A tag identifying what is releasing the semaphore :param acquire_token: The token returned from when the semaphore was acquired. Note that this is not really needed to directly use this class but is needed for API compatibility with the SlidingWindowSemaphore implementation. sReleasing acquire %s/%sN(RRRtrelease(R<Rt acquire_token((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR(s (R?R@R>RVRR(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs  tSlidingWindowSemaphorecBs5eZdZdZdZedZdZRS(sA semaphore used to coordinate sequential resource access. This class is similar to the stdlib BoundedSemaphore: * It's initialized with a count. * Each call to ``acquire()`` decrements the counter. * If the count is at zero, then ``acquire()`` will either block until the count increases, or if ``blocking=False``, then it will raise a NoResourcesAvailable exception indicating that it failed to acquire the semaphore. The main difference is that this semaphore is used to limit access to a resource that requires sequential access. For example, if I want to access resource R that has 20 subresources R_0 - R_19, this semaphore can also enforce that you only have a max range of 10 at any given point in time. You must also specify a tag name when you acquire the semaphore. The sliding window semantics apply on a per tag basis. The internal count will only be incremented when the minimum sequence number for a tag is released. cCsR||_tt|_i|_tj|_tj|j|_ i|_ dS(N( RORtintt_tag_sequencest_lowest_sequenceRKRLRMt Conditiont _conditiont_pending_release(R<R((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>Ks   cCs|j |jSWdQXdS(N(RMRO(R<((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRRUs cCstjd||jjz|jdkrn|sHtd|qnx#|jdkrj|jjqKWn|j|}|dkr||j|RRRVRR(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR5s   tChunksizeAdjustercBs8eZeeedZddZdZdZ RS(cCs||_||_||_dS(N(tmax_sizetmin_sizet max_parts(R<RRR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyR>s  cCs4|}|dk r'|j||}n|j|S(sGet a chunksize close to current that fits within all S3 limits. :type current_chunksize: int :param current_chunksize: The currently configured chunksize. :type file_size: int or None :param file_size: The size of the file to upload. This might be None if the object being transferred has an unknown size. :returns: A valid chunksize that fits within configured limits. N(Rt_adjust_for_max_partst_adjust_for_chunksize_limits(R<tcurrent_chunksizeRt chunksize((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pytadjust_chunksizes  cCsh||jkr0tjd|j|f|jS||jkr`tjd|j|f|jS|SdS(Ns@Chunksize greater than maximum chunksize. Setting to %s from %s.s=Chunksize less than minimum chunksize. Setting to %s from %s.(RRRR(R<R((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRscCs|}ttj|t|}x<||jkrc|d9}ttj|t|}q(W||krtjd||fn|S(Nis[Chunksize would result in the number of parts exceeding the maximum. Setting to %s from %s.(RtmathtceiltfloatRRR(R<RRRR((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs # N( R?R@tMAX_SINGLE_UPLOAD_SIZEtMIN_UPLOAD_CHUNKSIZEt MAX_PARTSR>RRRR(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyRs   i@i(*RttimeR'RRZRrRtloggingRKtiot collectionsRts3transfer.compatRRRRRt getLoggerR?RR RRRR R/R2R9tobjectR:RARJRYRzR`Rt ExceptionRRRR(((s4/usr/lib/python2.7/site-packages/s3transfer/utils.pyt sD                    0>C%_