rmLbc@sddlZddlZejjddddlmZddlmZmZddl m Z ddl m Z ddl m Z dd l mZdd l mZdd lmZdd lmZdd lmZmZdefdYZdefdYZdefdYZdefdYZdefdYZdefdYZde fdYZde fdYZde fdYZdS( iNis/usr/lib/fence-agents/bundled(tsix(tseekabletreadable(tIN_MEMORY_UPLOAD_TAG(tTask(tSubmissionTask(tCreateMultipartUploadTask(tCompleteMultipartUploadTask(t get_callbacks(tget_filtered_dict(tDeferredOpenFiletChunksizeAdjustertAggregatedProgressCallbackcBs/eZddZdZdZdZRS(iicCs||_||_d|_dS(sAggregates progress updates for every provided progress callback :type callbacks: A list of functions that accepts bytes_transferred as a single argument :param callbacks: The callbacks to invoke when threshold is reached :type threshold: int :param threshold: The progress threshold in which to take the aggregated progress and invoke the progress callback with that aggregated progress total iN(t _callbackst _thresholdt _bytes_seen(tselft callbackst threshold((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt__init__s  cCs2|j|7_|j|jkr.|jndS(N(RRt_trigger_callbacks(Rtbytes_transferred((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt__call__/scCs |jdkr|jndS(s@Flushes out any progress that has not been sent to its callbacksiN(RR(R((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pytflush4scCs1x!|jD]}|d|jq Wd|_dS(NRi(R R(Rtcallback((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR9si(t__name__t __module__RRRR(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR s   tInterruptReadercBsPeZdZdZddZdZdZdZdZ dZ RS( sWrapper that can interrupt reading using an error It uses a transfer coordinator to propogate an error if it notices that a read is being made while the file is being read from. :type fileobj: file-like obj :param fileobj: The file-like object to read from :type transfer_coordinator: s3transfer.futures.TransferCoordinator :param transfer_coordinator: The transfer coordinator to use if the reader needs to be interrupted. cCs||_||_dS(N(t_fileobjt_transfer_coordinator(Rtfileobjttransfer_coordinator((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRLs cCs+|jjr|jjn|jj|S(N(Rt exceptionRtread(Rtamount((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR!Ps cCs|jj|dS(N(Rtseek(Rtwhere((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR#ZscCs |jjS(N(Rttell(R((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR%]scCs|jjdS(N(Rtclose(R((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR&`scCs|S(N((R((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt __enter__cscOs|jdS(N(R&(Rtargstkwargs((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt__exit__fsN( RRt__doc__RtNoneR!R#R%R&R'R*(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR?s      tUploadInputManagercBsqeZdZd dZedZdZdZdZ dZ dZ dZ d Z d ZRS( sJBase manager class for handling various types of files for uploads This class is typically used for the UploadSubmissionTask class to help determine the following: * How to determine the size of the file * How to determine if a multipart upload is required * How to retrieve the body for a PutObject * How to retrieve the bodies for a set of UploadParts The answers/implementations differ for the various types of file inputs that may be accepted. All implementations must subclass and override public methods from this class. cCs||_||_||_dS(N(t_osutilRt_bandwidth_limiter(RtosutilRtbandwidth_limiter((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRys  cCstddS(sDetermines if the source for the upload is compatible with manager :param upload_source: The source for which the upload will pull data from. :returns: True if the manager can handle the type of source specified otherwise returns False. smust implement _is_compatible()N(tNotImplementedError(tclst upload_source((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt is_compatible~s cCstddS(sWhether the body it provides are stored in-memory :type operation_name: str :param operation_name: The name of the client operation that the body is being used for. Valid operation_names are ``put_object`` and ``upload_part``. :rtype: boolean :returns: True if the body returned by the manager will be stored in memory. False if the manager will not directly store the body in memory. s%must implement store_body_in_memory()N(tNotImplemented(Rtoperation_name((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pytstores_body_in_memorys cCstddS(sProvides the transfer size of an upload :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request s&must implement provide_transfer_size()N(R2(Rttransfer_future((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pytprovide_transfer_sizescCstddS(sDetermines where a multipart upload is required :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: boolean :returns: True, if the upload should be multipart based on configuartion and size. False, otherwise. s*must implement requires_multipart_upload()N(R2(RR9tconfig((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pytrequires_multipart_uploads cCstddS(sReturns the body to use for PutObject :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type config: s3transfer.manager.TransferConfig :param config: The config associated to the transfer manager :rtype: s3transfer.utils.ReadFileChunk :returns: A ReadFileChunk including all progress callbacks associated with the transfer future. s$must implement get_put_object_body()N(R2(RR9((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pytget_put_object_bodys cCstddS(sYields the part number and body to use for each UploadPart :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The future associated with upload request :type chunksize: int :param chunksize: The chunksize to use for this upload. :rtype: int, s3transfer.utils.ReadFileChunk :returns: Yields the part number and the ReadFileChunk including all progress callbacks associated with the transfer future for that specific yielded part. s)must implement yield_upload_part_bodies()N(R2(RR9t chunksize((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pytyield_upload_part_bodiesscCs@t||j}|jr<|jj||jdt}n|S(Ntenabled(RRR/tget_bandwith_limited_streamtFalse(RR((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt _wrap_fileobjs   cCs&t|d}|r"t|gSgS(Ntprogress(RR (RR9R((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt_get_progress_callbackss cCsg|D]}|j^qS(N(R(Rtaggregated_progress_callbacksR((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt_get_close_callbackssN(RRR+R,Rt classmethodR5R8R:R<R=R?RCRERG(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR-js        tUploadFilenameInputManagercBsneZdZedZdZdZdZdZdZ dZ dZ d Z d Z RS( sUpload utility for filenamescCst|tjS(N(t isinstanceRt string_types(R3R4((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR5scCstS(N(RB(RR7((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR8scCs)|jj|jj|jjjdS(N(tmetaR:R.t get_file_sizet call_argsR(RR9((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR:s  cCs|jj|jkS(N(RLtsizetmultipart_threshold(RR9R;((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR<sc Csy|j|\}}|j|}|j|}|j|}|jj}|jjd|d|d|d|d|S(NRt chunk_sizetfull_file_sizeRtclose_callbacks(t&_get_put_object_fileobj_with_full_sizeRCRERGRLROR.t#open_file_chunk_reader_from_fileobj(RR9Rt full_sizeRRSRO((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR=s  c cs|jj}|j||}xtd|dD]}|j|}|j|}||d}|j|jjjd|d|d|\} } |j | } |j j d| d|d| d|d|} || fVq2WdS( Nit start_bytet part_sizeRRRRQRRS( RLROt_get_num_partstrangeRERGt'_get_upload_part_fileobj_with_full_sizeRNRRCR.RU( RR9R>RRt num_partst part_numberRRSRWRRVtread_file_chunk((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR?s     cCst||d|jj}|S(Nt open_function(R R.topen(RRRW((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt_get_deferred_open_file scCs1|jjj}|jj}|j|d|fS(Ni(RLRNRRORa(RR9RRO((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRT%s cKs*|d}|d}|j|||fS(NRWRR(Ra(RRR)RWRV((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR[*s  cCs#ttj|jjt|S(N(tinttmathtceilRLROtfloat(RR9RX((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRY/s(RRR+RHR5R8R:R<R=R?RaRTR[RY(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRIs        tUploadSeekableInputManagercBsAeZdZedZdZdZdZdZRS(s&Upload utility for an open file objectcCst|ot|S(N(RR(R3R4((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR56scCs|dkrtStSdS(Nt put_object(RBtTrue(RR7((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR8:s cCs\|jjj}|j}|jdd|j}|j||jj||dS(Nii(RLRNRR%R#R:(RR9Rtstart_positiont end_position((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR:@s    cKs,|j|d}tj|t|fS(NRX(R!RtBytesIOtlen(RRR)tdata((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR[LscCs/|jjj}|j|jj}||fS(N(RLRNRR%RO(RR9RRO((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRT[s( RRR+RHR5R8R:R[RT(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRf4s   tUploadNonSeekableInputManagercBskeZdZd dZedZdZdZdZ dZ dZ e dZ d ZRS( s7Upload utility for a file-like object that cannot seek.cCs)tt|j|||d|_dS(Nt(tsuperRnRt _initial_data(RR0RR1((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRes cCs t|S(N(R(R3R4((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR5jscCstS(N(Rh(RR7((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR8nscCsdS(N((RR9((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR:qscCsv|jjdk r%|jj|jkS|jjj}|j}|j||t|_t |j|krntSt SdS(N( RLROR,RPRNRt_readRBRqRlRh(RR9R;RR((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR<vs cCs\|j|}|j|}|jjj}|j|j|j||}d|_|S(N( RERGRLRNRt _wrap_dataRqR!R,(RR9RRSRtbody((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR=s c cs|jjj}d}xttr|j|}|j|}|d7}|j||}|sbPn|j|||}d}||fVqWdS(Nii( RLRNRRhRERGRrRsR,( RR9R>t file_objectR]RRSt part_contentt part_object((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR?s  cCst|jdkr"|j|S|t|jkra|j| }|r]|j||_n|S|t|j}|j|j|}|rd|_n|S(s= Reads a specific amount of data from a stream and returns it. If there is any data in initial_data, that will be popped out first. :type fileobj: A file-like object that implements read :param fileobj: The stream to read from. :type amount: int :param amount: The number of bytes to read from the stream. :type truncate: bool :param truncate: Whether or not to truncate initial_data after reading from it. :return: Generator which generates part bodies from the initial data. iRo(RlRqR!(RRR"ttruncateRmtamount_to_read((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRrs   c CsO|jtj|}|jjd|dt|dt|d|d|S(s Wraps data with the interrupt reader and the file chunk reader. :type data: bytes :param data: The data to wrap. :type callbacks: list :param callbacks: The callbacks associated with the transfer future. :type close_callbacks: list :param close_callbacks: The callbacks to be called when closing the wrapper for the data. :return: Fully wrapped data. RRQRRRRS(RCRRkR.RURl(RRmRRSR((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRss N(RRR+R,RRHR5R8R:R<R=R?RhRrRs(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRncs      *tUploadSubmissionTaskcBskeZdZddddgZdgZdZd dZdZdZ d Z d Z d Z RS( s.Task for submitting tasks to execute an uploadtSSECustomerKeytSSECustomerAlgorithmtSSECustomerKeyMD5t RequestPayercCsbtttg}|jjj}x!|D]}|j|r%|Sq%Wtd|t|fdS(snRetieves a class for managing input for an upload based on file type :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future for the request :rtype: class of UploadInputManager :returns: The appropriate class to use for managing a specific type of input for uploads. s&Input %s of type: %s is not supported.N( RIRfRnRLRNRR5t RuntimeErrorttype(RR9tupload_manager_resolver_chainRtupload_manager_cls((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt_get_upload_input_manager_clss   cCs|j|||j|}|jjdkr@|j|n|j||sq|j||||||n|j||||||dS(s :param client: The client associated with the transfer manager :type config: s3transfer.manager.TransferConfig :param config: The transfer config associated with the transfer manager :type osutil: s3transfer.utils.OSUtil :param osutil: The os utility associated to the transfer manager :type request_executor: s3transfer.futures.BoundedExecutor :param request_executor: The request executor associated with the transfer manager :type transfer_future: s3transfer.futures.TransferFuture :param transfer_future: The transfer future associated with the transfer request that tasks are being submitted for N( RRRLROR,R:R<t_submit_upload_requestt_submit_multipart_request(RtclientR;R0trequest_executorR9R1tupload_input_manager((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt_submit s  c Cs|jj}|j|d}|jj|td|jdi|d6|j|d6|jd6|jd6|j d6d t d |dS( NRgRt main_kwargsRRtbuckettkeyt extra_argstis_finalttag( RLRNt_get_upload_task_tagRtsubmitt PutObjectTaskR=RRRRh( RRR;R0RR9RRNtput_object_tag((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyR4s          c Cs|jj}|jj|td|jdi|d6|jd6|jd6|jd6}g} |j|j} |j |d} |jj } t } | j |j | }|j||}x|D]}\}}| j|jj|td|jdi|d6|d6|jd6|jd6|d 6| d6d i|d 6d | qW|j|j}|jj|td|jdi|d6|jd6|jd6|d6d i|d 6| d 6dtdS(NRRRRRRt upload_partRR]tpending_main_kwargst upload_idRtpartsR(RLRNRRRRRRt_extra_upload_part_argsRROR tadjust_chunksizetmultipart_chunksizeR?tappendtUploadPartTaskt_extra_complete_multipart_argsRRh(RRR;R0RR9RRNtcreate_multipart_futuret part_futurestextra_part_argstupload_part_tagROtadjusterR>t part_iteratorR]Rtcomplete_multipart_extra_args((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRNsd                      cCst||jS(N(R tUPLOAD_PART_ARGS(RR((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRscCst||jS(N(R tCOMPLETE_MULTIPART_ARGS(RR((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRscCs"d}|j|rt}n|S(N(R,R8R(RRR7R((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRs N( RRR+RRRR,RRRRRR(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRzs    &  M  RcBseZdZdZRS(s Task to do a nonmultipart uploadc Cs2|&}|jd|d|d||WdQXdS(sP :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param extra_args: A dictionary of any extra arguments that may be used in the upload. tBuckettKeytBodyN(Rg(RRRRRRRt((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt_mains (RRR+R(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRsRcBseZdZdZRS(s+Task to upload a part in a multipart uploadc CsX|4}|jd|d|d|d|d||} WdQX| d} i| d6|d6S(s :param client: The client to use when calling PutObject :param fileobj: The file to upload. :param bucket: The name of the bucket to upload to :param key: The name of the key to upload to :param upload_id: The id of the upload :param part_number: The number representing the part of the multipart upload :param extra_args: A dictionary of any extra arguments that may be used in the upload. :rtype: dict :returns: A dictionary representing a part:: {'Etag': etag_value, 'PartNumber': part_number} This value can be appended to a list to be used to complete the multipart upload. RRtUploadIdt PartNumberRNtETag(R( RRRRRRR]RRttresponsetetag((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRs     (RRR+R(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyRs(Rctsystpathtinserttbotocore.compatRts3transfer.compatRRts3transfer.futuresRts3transfer.tasksRRRRts3transfer.utilsRR R R tobjectR RR-RIRfRnRzRR(((s5/usr/lib/python2.7/site-packages/s3transfer/upload.pyt s*  !+yQ/