Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org LearningPython WOW! eBook www.wowebook.org TableofContents LearningPython Credits AbouttheAuthor Acknowledgements AbouttheReviewers www.PacktPub.com Supportfiles,eBooks,discountoffers,andmore Whysubscribe? FreeaccessforPacktaccountholders Preface Whatthisbookcovers Whatyouneedforthisbook Whothisbookisfor Conventions Readerfeedback Customersupport Downloadingtheexamplecode Errata Piracy Questions 1.IntroductionandFirstSteps–TakeaDeepBreath Aproperintroduction EnterthePython AboutPython Portability Coherence Developerproductivity Anextensivelibrary Softwarequality WOW! eBook www.wowebook.org Softwareintegration Satisfactionandenjoyment Whatarethedrawbacks? WhoisusingPythontoday? Settinguptheenvironment Python2versusPython3–thegreatdebate InstallingPython SettingupthePythoninterpreter Aboutvirtualenv Yourfirstvirtualenvironment Yourfriend,theconsole HowyoucanrunaPythonprogram RunningPythonscripts RunningthePythoninteractiveshell RunningPythonasaservice RunningPythonasaGUIapplication HowisPythoncodeorganized Howdoweusemodulesandpackages Python’sexecutionmodel Namesandnamespaces Scopes Objectandclasses Guidelinesonhowtowritegoodcode ThePythonculture AnoteontheIDEs Summary 2.Built-inDataTypes Everythingisanobject Mutableorimmutable?Thatisthequestion Numbers Integers WOW! eBook www.wowebook.org Booleans Reals Complexnumbers Fractionsanddecimals Immutablesequences Stringsandbytes Encodinganddecodingstrings Indexingandslicingstrings Tuples Mutablesequences Lists Bytearrays Settypes Mappingtypes–dictionaries Thecollectionsmodule Namedtuples Defaultdict ChainMap Finalconsiderations Smallvaluescaching Howtochoosedatastructures Aboutindexingandslicing Aboutthenames Summary 3.IteratingandMakingDecisions Conditionalprogramming Aspecializedelse:elif Theternaryoperator Looping Theforloop Iteratingoverarange WOW! eBook www.wowebook.org Iteratingoverasequence Iteratorsanditerables Iteratingovermultiplesequences Thewhileloop Thebreakandcontinuestatements Aspecialelseclause Puttingthisalltogether Example1–aprimegenerator Example2–applyingdiscounts Aquickpeekattheitertoolsmodule Infiniteiterators Iteratorsterminatingontheshortestinputsequence Combinatoricgenerators Summary 4.Functions,theBuildingBlocksofCode Whyusefunctions? Reducecodeduplication Splittingacomplextask Hideimplementationdetails Improvereadability Improvetraceability Scopesandnameresolution Theglobalandnonlocalstatements Inputparameters Argumentpassing Assignmenttoargumentnamesdon’taffectthecaller Changingamutableaffectsthecaller Howtospecifyinputparameters Positionalarguments Keywordargumentsanddefaultvalues Variablepositionalarguments WOW! eBook www.wowebook.org Variablekeywordarguments Keyword-onlyarguments Combininginputparameters Avoidthetrap!Mutabledefaults Returnvalues Returningmultiplevalues Afewusefultips Recursivefunctions Anonymousfunctions Functionattributes Built-infunctions Onefinalexample Documentingyourcode Importingobjects Relativeimports Summary 5.SavingTimeandMemory map,zip,andfilter map zip filter Comprehensions Nestedcomprehensions Filteringacomprehension dictcomprehensions setcomprehensions Generators Generatorfunctions Goingbeyondnext Theyieldfromexpression Generatorexpressions WOW! eBook www.wowebook.org Someperformanceconsiderations Don’toverdocomprehensionsandgenerators Namelocalization Generationbehaviorinbuilt-ins Onelastexample Summary 6.AdvancedConcepts–OOP,Decorators,andIterators Decorators Adecoratorfactory Object-orientedprogramming ThesimplestPythonclass Classandobjectnamespaces Attributeshadowing I,me,andmyself–usingtheselfvariable Initializinganinstance OOPisaboutcodereuse Inheritanceandcomposition Accessingabaseclass Multipleinheritance Methodresolutionorder Staticandclassmethods Staticmethods Classmethods Privatemethodsandnamemangling Thepropertydecorator Operatoroverloading Polymorphism–abriefoverview Writingacustomiterator Summary 7.Testing,Profiling,andDealingwithExceptions Testingyourapplication WOW! eBook www.wowebook.org Theanatomyofatest Testingguidelines Unittesting Writingaunittest Mockobjectsandpatching Assertions Aclassicunittestexample Makingatestfail Interfacetesting Comparingtestswithandwithoutmocks Boundariesandgranularity Amoreinterestingexample Test-drivendevelopment Exceptions ProfilingPython Whentoprofile? Summary 8.TheEdges–GUIsandScripts Firstapproach–scripting Theimports Parsingarguments Thebusinesslogic Secondapproach–aGUIapplication Theimports Thelayoutlogic Thebusinesslogic Fetchingthewebpage Savingtheimages Alertingtheuser Howtoimprovetheapplication? Wheredowegofromhere? WOW! eBook www.wowebook.org Thetkinter.tixmodule Theturtlemodule wxPython,PyQt,andPyGTK Theprincipleofleastastonishment Threadingconsiderations Summary 9.DataScience IPythonandJupyternotebook Dealingwithdata Settingupthenotebook Preparingthedata Cleaningthedata CreatingtheDataFrame Unpackingthecampaignname Unpackingtheuserdata Cleaningeverythingup SavingtheDataFrametoafile Visualizingtheresults Wheredowegofromhere? Summary 10.WebDevelopmentDoneRight WhatistheWeb? HowdoestheWebwork? TheDjangowebframework Djangodesignphilosophy Themodellayer Theviewlayer Thetemplatelayer TheDjangoURLdispatcher Regularexpressions Aregexwebsite WOW! eBook www.wowebook.org SettingupDjango Startingtheproject Creatingusers AddingtheEntrymodel Customizingtheadminpanel Creatingtheform Writingtheviews Thehomeview Theentrylistview Theformview TyingupURLsandviews Writingthetemplates Thefutureofwebdevelopment WritingaFlaskview BuildingaJSONquoteserverinFalcon Summary 11.DebuggingandTroubleshooting Debuggingtechniques Debuggingwithprint Debuggingwithacustomfunction Inspectingthetraceback UsingthePythondebugger Inspectinglogfiles Othertechniques Profiling Assertions Wheretofindinformation Troubleshootingguidelines Usingconsoleeditors Wheretoinspect Usingteststodebug WOW! eBook www.wowebook.org Monitoring Summary 12.SummingUp–ACompleteExample Thechallenge Ourimplementation ImplementingtheDjangointerface Thesetup Themodellayer Asimpleform Theviewlayer Importsandhomeview Listingallrecords Creatingrecords Updatingrecords Deletingrecords SettinguptheURLs Thetemplatelayer Homeandfootertemplates Listingallrecords Creatingandeditingrecords TalkingtotheAPI Deletingrecords ImplementingtheFalconAPI Themainapplication Writingthehelpers Codingthepasswordvalidator Codingthepasswordgenerator Writingthehandlers Codingthepasswordvalidatorhandler Codingthepasswordgeneratorhandler RunningtheAPI WOW! eBook www.wowebook.org TestingtheAPI Testingthehelpers Testingthehandlers Wheredoyougofromhere? Summary Awordoffarewell Index WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org LearningPython WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org LearningPython Copyright©2015PacktPublishing Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem, ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthe publisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews. Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyofthe informationpresented.However,theinformationcontainedinthisbookissoldwithout warranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,andits dealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecaused directlyorindirectlybythisbook. PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthe companiesandproductsmentionedinthisbookbytheappropriateuseofcapitals. However,PacktPublishingcannotguaranteetheaccuracyofthisinformation. Firstpublished:December2015 Productionreference:1171215 PublishedbyPacktPublishingLtd. LiveryPlace 35LiveryStreet BirminghamB32PB,UK. ISBN978-1-78355-171-2 www.packtpub.com WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Credits Author FabrizioRomano Reviewers SimoneBurol JulioVicenteTrigoGuijarro VeitHeller CommissioningEditor AkramHussain AcquisitionEditor IndrajitDas ContentDevelopmentEditors SamanthaGonsalves AdrianRaposo TechnicalEditor SiddhiRane CopyEditors JanbalDharmaraj KevinMcGowan ProjectCoordinator KinjalBari Proofreader SafisEditing Indexer PriyaSane Graphics KirkD’Penha AbhinashSahu ProductionCoordinator MelwynD’sa CoverWork WOW! eBook www.wowebook.org MelwynD’sa WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org AbouttheAuthor FabrizioRomanowasborninItalyin1975.Heholdsamaster’sdegreeincomputer scienceengineeringfromtheUniversityofPadova.HeisalsoacertifiedScrummaster. BeforePython,hehasworkedwithseveralotherlanguages,suchasC/C++,Java,PHP, andC#. In2011,hemovedtoLondonandstartedworkingasaPythondeveloperforGlasses Direct,oneofEurope’sleadingonlineprescriptionglassesretailers. HethenworkedasaseniorPythondeveloperforTBG(nowSprinklr),oneoftheworld’s leadingcompaniesinsocialmediaadvertising.AtTBG,heandhisteamcollaboratedwith FacebookandTwitter.TheywerethefirstintheworldtogetaccesstotheTwitter advertisingAPI.Hewrotethecodethatpublishedthefirstgeo-narrowcastedpromoted tweetintheworldusingtheAPI. HecurrentlyworksasaseniorplatformdeveloperatStudent.com,acompanythatis revolutionizingthewayinternationalstudentsfindtheirperfecthomeallaroundtheworld HehasdeliveredtalksonTeachingPythonandTDDwithPythonatthelasttwoeditions ofEuroPythonandatSkillsmatterinLondon. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Acknowledgements IwouldliketothankAdrianRaposoandIndrajitDasfromPacktPublishingfortheirhelp andsupportandgivingmetheopportunitytolivethisadventure.Iwouldalsoliketo thankeveryoneatPacktPublishingwhohavecontributedtotherealizationofthisbook. SpecialthanksgotoSiddhiRane,mytechnicaleditor.Thankyouforyourkindness,for workingveryhard,andforgoingtheextramilejusttomakemehappy. IwouldliketoexpressmydeepestgratitudetoSimoneBurolandJulioTrigo,whohave giftedmewithsomeoftheirpreciousfreetime.Theyhavereviewedthebookand providedmewithinvaluablefeedback. Abigthankyoutomyteammates,MattBennettandJakubKubaBorys,fortheirinterest inthisbookandfortheirsupportandfeedbackthatmakesmeabettercodereveryday. AheartfeltthankyoutoMarco“Tex”Beri,whointroducedmetoPythonwithan enthusiasmsecondtonone. AspecialthankstoDr.NaomiCeder,fromwhomIlearnedsomuchoverthelastyear.She hasgivenmeprecioussuggestionsandhasencouragedmetoembracethisopportunity. Finally,Iwouldliketothankallmyfriendswhohavesupportedmeinanyway. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org AbouttheReviewers SimoneBurolisanItaliansoftwaredeveloperwhowasborninTreviso(Italy)in1978. Heobtainedamaster’sdegreeincomputerscienceengineeringfromtheUniversityof Padua(Italy),andsincethenworkedinbankingfor5yearsinVenice(Italy).In2010,he movedtoLondon(UnitedKingdom),whereheworkedinwarehouseautomationfor OcadoTechnologyandtheninbankingforAlgomi. JulioVicenteTrigoGuijarroisacomputerscientistandsoftwareengineerwithalmosta decadeofexperienceinsoftwaredevelopment.HeisalsoacertifiedScrummaster,who enjoysthebenefitsofusingagilesoftwaredevelopment(ScrumandXP). Hecompletedhisstudiesincomputerscienceandsoftwareengineeringfromthe UniversityofAlicante,Spain,in2007.Sincethen,hehasworkedwithseveral technologiesandlanguages,includingMicrosoftDynamicsNAV,Java,JavaScript,and Python. SomeoftheapplicationscoveredbyJulioduringhiscareerincludeRESTfulAPIs,ERPs, billingplatforms,paymentgateways,ande-commercewebsites. HehasbeenusingPythononbothpersonalandprofessionalprojectssince2012,andheis passionateaboutsoftwaredesign,softwarequality,andcodingstandards. Iwouldliketothankmyparentsfortheirlove,goodadvice,andcontinuoussupport. IwouldalsoliketothankallmyfriendsthatImetalongtheway,whoenrichedmylife, formotivatingmeandhelpingmeprogress. VeitHellerisafullstackdeveloper,mostlyworkingonthebackendsideofwebprojects. HecurrentlyresidesinBerlinandworksforaprototypicalPythonistacompanynamed Bright.Inhisfreetime,hewritesinterpretersforvariousprogramminglanguages. IwouldliketothankthepeopleatBrightforbeingawelcomingcompanythatsupports meinallmyendeavors,myfriendsandmyfamilyforcopingwithmystrangeness,and manufacturersofcaffeinateddrinksworldwide. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org www.PacktPub.com WOW! eBook www.wowebook.org Supportfiles,eBooks,discountoffers,and more Forsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com. DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFand ePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandas aprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwith usat<[email protected]>formoredetails. Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signup forarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooks andeBooks. https://www2.packtpub.com/books/subscription/packtlib DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigital booklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks. WOW! eBook www.wowebook.org Whysubscribe? FullysearchableacrosseverybookpublishedbyPackt Copyandpaste,print,andbookmarkcontent Ondemandandaccessibleviaawebbrowser WOW! eBook www.wowebook.org FreeaccessforPacktaccountholders IfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccess PacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsfor immediateaccess. ToAlanTuring,thefatherofComputerScience. ToGuidoVanRossum,thefatherofPython. ToAdrianoRomano,myfather,mybiggestfan. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Preface ShortlyafterIstartedwriting,afriendaskedmeiftherereallywasaneedofanother LearningPythonbook. Anexcellentquestionthatwecouldalsoexpressinanotherform:Whathasthisbookto offer?WhatmakesthisbookdifferentfromtheaverageintroductorybookonPython? Ithinktherearetwomaindifferencesandmanygoodreasonswhyyouwouldwanttoread it. Firstly,westartwithintroducingsomeimportantprogrammingconcepts.Webuildasolid foundationbycoveringthecriticalaspectsofthiswonderfullanguage. Thepacegraduallyincreases,alongwiththedifficultyofthesubjectspresented.Bythe endofChapter7,Testing,Profiling,andDealingwithExceptions,wewillcoverallthe fundamentals. FromChapter8,TheEdges–GUIsandScripts,onward,thebooktakesasteepturn, whichbringsustodifferencenumbertwo. Toconsolidatetheknowledgeacquired,thereisnothinglikeworkingonasmallproject. So,inthesecondpartofthebook,eachchapterdeliversaprojectonadifferentsubject. Weexplorescripting,graphicalinterfaces,datascience,andwebprogramming. Eachprojectissmallenoughtofitwithinachapterandyetbigenoughtoberelevant. Eachchapterisinteresting,conveysamessage,andteachessomethingvaluable. Afterashortsectionondebugging,thebookendswithacompleteexamplethatwraps thingsup.Itriedtocraftitsothatyouwillbeabletoexpanditinseveralways. So,thisisdefinitelynottheusualLearningPythonbook.Itsapproachismuchmore “hands-on”andpractical. IwantedtoempoweryoutohelpyoubecomeatruePythonninja.ButIalsodidmybest toentertainyouandfosteryourlogicalthinkingandcreativityalongtheway. Now,haveIansweredthequestion? WOW! eBook www.wowebook.org Whatthisbookcovers Chapter1,IntroductionandFirstSteps–TakeaDeepBreath,introducesyouto fundamentalprogrammingconcepts.ItguidesyoutogettingPythonupandrunningon yourcomputerandintroducesyoutosomeofitsconstructs. Chapter2,Built-inDataTypes,introducesyoutoPythonbuilt-indatatypes.Pythonhasa veryrichsetofnativedatatypesandthischapterwillgiveyouadescriptionandashort exampleforeachofthem. Chapter3,IteratingandMakingDecisions,teachesyouhowtocontroltheflowofyour codebyinspectingconditions,applyinglogic,andperformingloops. Chapter4,Functions,theBuildingBlocksofCode,teachesyouhowtowritefunctions. Functionsarethekeystoreusingcode,toreducingdebuggingtime,andingeneral,to writingbettercode. Chapter5,SavingTimeandMemory,introducesyoutothefunctionalaspectsofPython programming.Thischapterteachesyouhowtowritecomprehensionsandgenerators, whicharepowerfultoolsthatyoucanusetospeedupyourcodeandsavememory. Chapter6,AdvancedConcepts–OOP,Decorators,andIterators,teachesyouthebasics ofobject-orientedprogrammingwithPython.Itshowsyouthekeyconceptsandallthe potentialsofthisparadigm.Italsoshowsyouoneofthemostbelovedcharacteristicsof Python:decorators.Finally,italsocoverstheconceptofiterators. Chapter7,Testing,Profiling,andDealingwithExceptions,teachesyouhowtomakeyour codemorerobust,fast,andstableusingtechniquessuchastestingandprofiling.Italso formallydefinestheconceptofexceptions. Chapter8,TheEdges–GUIsandScripts,guidesyouthroughanexamplefromtwo differentpointsofview.Theyareattheextremitiesofaspectrum:oneimplementationisa scriptandtheotheroneapropergraphicaluserinterfaceapplication. Chapter9,DataScience,introducesafewkeyconceptsandaveryspecialtool,the JupyterNotebook. Chapter10,WebDevelopmentDoneRight,introducesthefundamentalsofweb developmentanddeliversaprojectusingtheDjangowebframework.Theexamplewillbe basedonregularexpressions. Chapter11,DebuggingandTroubleshooting,showsyouthemainmethodstodebugyour codeandsomeexamplesonhowtoapplythem. Chapter12,SummingUp–ACompleteExample,presentsaDjangowebsitethatactsasan interfacetoanunderlyingslimAPIwrittenwiththeFalconwebframework.Thischapter takesalltheconceptscoveredinthebooktothenextlevelandsuggestswheretogotodig deeperandtakethenextsteps. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Whatyouneedforthisbook Youareencouragedtofollowtheexamplesinthisbook.Inordertodoso,youwillneeda computer,anInternetconnection,andabrowser.ThebookiswritteninPython3.4,butit shouldalsoworkwithanyPython3.*version.Ihavewritteninstructionsonhowtoinstall Pythononthethreemainoperatingsystemsusedtoday:Windows,Mac,andLinux.Ihave alsoexplainedhowtoinstallalltheextralibrariesusedinthevariousexamplesand providedsuggestionsifthereaderfindsanyissuesduringtheinstallationofanyofthem. Noparticulareditorisrequiredtotypethecode;however,Isuggestthatthosewhoare interestedinfollowingtheexamplesshouldconsideradoptingapropercoding environment.Ihavegivensuggestionsonthismatterinthefirstchapter. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Whothisbookisfor Pythonisthemostpopularintroductoryteachinglanguageinthetopcomputerscience universitiesintheUS,soifyouarenewtosoftwaredevelopmentorifyouhavelittle experienceandwouldliketostartoffontherightfoot,thenthislanguageandthisbook arewhatyouneed.Itsamazingdesignandportabilitywillhelpyoubecomeproductive regardlessoftheenvironmentyouchoosetoworkwith. IfyouhavealreadyworkedwithPythonoranyotherlanguage,thisbookcanstillbe usefultoyoubothasareferencetoPython’sfundamentalsandtoprovideawiderangeof considerationsandsuggestionscollectedovertwodecadesofexperience. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Conventions Inthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkinds ofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheir meaning. Codewordsintext,databasetablenames,foldernames,filenames,fileextensions, pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Open upaPythonconsole,andtypeimportthis.” Ablockofcodeissetasfollows: #wedefineafunction,calledlocal deflocal(): m=7 print(m) m=5 print(m) Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevant linesoritemsaresetinbold: #wedefineafunction,calledlocal deflocal(): m=7 print(m) m=5 print(m) Anycommand-lineinputoroutputiswrittenasfollows: >>>frommathimportfactorial >>>factorial(5) 120 Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen, forexample,inmenusordialogboxes,appearinthetextlikethis:“Toopentheconsole onWindows,gototheStartmenu,chooseRun,andtypecmd.” Note Warningsorimportantnotesappearinaboxlikethis. Tip Tipsandtricksappearlikethis. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Readerfeedback Feedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthis book—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsus developtitlesthatyouwillreallygetthemostoutof. Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthe book’stitleinthesubjectofyourmessage. Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingor contributingtoabook,seeourauthorguideatwww.packtpub.com/authors. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Customersupport NowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelp youtogetthemostfromyourpurchase. WOW! eBook www.wowebook.org Downloadingtheexamplecode Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.com forallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbook elsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou. WOW! eBook www.wowebook.org Errata Althoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdo happen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthe code—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveother readersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufind anyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata, selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthe detailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedand theerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataunderthe Erratasectionofthattitle. Toviewthepreviouslysubmittederrata,goto https://www.packtpub.com/books/content/supportandenterthenameofthebookinthe searchfield.TherequiredinformationwillappearundertheErratasection. WOW! eBook www.wowebook.org Piracy PiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.At Packt,wetaketheprotectionofourcopyrightandlicensesveryseriously.Ifyoucome acrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswith thelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy. Pleasecontactusat<[email protected]>withalinktothesuspectedpirated material. Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluable content. WOW! eBook www.wowebook.org Questions Ifyouhaveaproblemwithanyaspectofthisbook,youcancontactusat <[email protected]>,andwewilldoourbesttoaddresstheproblem. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter1.IntroductionandFirstSteps– TakeaDeepBreath “Giveamanafishandyoufeedhimforaday.Teachamantofishandyoufeedhimforalifetime.” —Chineseproverb AccordingtoWikipedia,computerprogrammingis: “…aprocessthatleadsfromanoriginalformulationofacomputingproblemto executablecomputerprograms.Programminginvolvesactivitiessuchasanalysis, developingunderstanding,generatingalgorithms,verificationofrequirementsof algorithmsincludingtheircorrectnessandresourcesconsumption,and implementation(commonlyreferredtoascoding)ofalgorithmsinatarget programminglanguage”. Inanutshell,codingistellingacomputertodosomethingusingalanguageitunderstands. Computersareverypowerfultools,butunfortunately,theycan’tthinkforthemselves.So theyneedtobetoldeverything.Theyneedtobetoldhowtoperformatask,howto evaluateaconditiontodecidewhichpathtofollow,howtohandledatathatcomesfroma devicesuchasthenetworkoradisk,andhowtoreactwhensomethingunforeseen happens,say,somethingisbrokenormissing. Youcancodeinmanydifferentstylesandlanguages.Isithard?Iwouldsay“yes”and “no”.It’sabitlikewriting.Everybodycanlearnhowtowrite,andyoucantoo.Butwhat ifyouwantedtobecomeapoet?Thenwritingaloneisnotenough.Youhavetoacquirea wholeothersetofskillsandthiswilltakealongerandgreatereffort. Intheend,itallcomesdowntohowfaryouwanttogodowntheroad.Codingisnotjust puttingtogethersomeinstructionsthatwork.Itissomuchmore! Goodcodeisshort,fast,elegant,easytoreadandunderstand,simple,easytomodifyand extend,easytoscaleandrefactor,andeasytotest.Ittakestimetobeabletowritecode thathasallthesequalitiesatthesametime,butthegoodnewsisthatyou’retakingthe firststeptowardsitatthisverymomentbyreadingthisbook.AndIhavenodoubtyou candoit.Anyonecan,infact,weallprogramallthetime,onlywearen’tawareofit. Wouldyoulikeanexample? Sayyouwanttomakeinstantcoffee.Youhavetogetamug,theinstantcoffeejar,a teaspoon,water,andthekettle.Evenifyou’renotawareofit,you’reevaluatingalotof data.You’remakingsurethatthereiswaterinthekettleaswellasthekettleisplugged-in, thatthemugisclean,andthatthereisenoughcoffeeinthejar.Then,youboilthewater andmaybeinthemeantimeyouputsomecoffeeinthemug.Whenthewaterisready,you pouritintothecup,andstir. So,howisthisprogramming? WOW! eBook www.wowebook.org Well,wegatheredresources(thekettle,coffee,water,teaspoon,andmug)andweverified someconditionsonthem(kettleisplugged-in,mugisclean,thereisenoughcoffee).Then westartedtwoactions(boilingthewaterandputtingcoffeeinthemug),andwhenbothof themwerecompleted,wefinallyendedtheprocedurebypouringwaterinthemugand stirring. Canyouseeit?Ihavejustdescribedthehigh-levelfunctionalityofacoffeeprogram.It wasn’tthathardbecausethisiswhatthebraindoesalldaylong:evaluateconditions, decidetotakeactions,carryouttasks,repeatsomeofthem,andstopatsomepoint.Clean objects,putthemback,andsoon. Allyouneednowistolearnhowtodeconstructallthoseactionsyoudoautomaticallyin reallifesothatacomputercanactuallymakesomesenseofthem.Andyouneedtolearna languageaswell,toinstructit. Sothisiswhatthisbookisfor.I’lltellyouhowtodoitandI’lltrytodothatbymeansof manysimplebutfocusedexamples(myfavoritekind). WOW! eBook www.wowebook.org Aproperintroduction IlovetomakereferencestotherealworldwhenIteachcoding;Ibelievetheyhelppeople retaintheconceptsbetter.However,nowisthetimetobeabitmorerigorousandseewhat codingisfromamoretechnicalperspective. Whenwewritecode,we’reinstructingacomputeronwhatarethethingsithastodo. Wheredoestheactionhappen?Inmanyplaces:thecomputermemory,harddrives, networkcables,CPU,andsoon.It’sawhole“world”,whichmostofthetimeisthe representationofasubsetoftherealworld. Ifyouwriteapieceofsoftwarethatallowspeopletobuyclothesonline,youwillhaveto representrealpeople,realclothes,realbrands,sizes,andsoonandsoforth,withinthe boundariesofaprogram. Inordertodoso,youwillneedtocreateandhandleobjectsintheprogramyou’rewriting. Apersoncanbeanobject.Acarisanobject.Apairofsocksisanobject.Luckily,Python understandsobjectsverywell. Thetwomainfeaturesanyobjecthasarepropertiesandmethods.Let’stakeaperson objectasanexample.Typicallyinacomputerprogram,you’llrepresentpeopleas customersoremployees.Thepropertiesthatyoustoreagainstthemarethingslikethe name,theSSN,theage,iftheyhaveadrivinglicense,theire-mail,gender,andsoon.Ina computerprogram,youstoreallthedatayouneedinordertouseanobjectforthepurpose you’reserving.Ifyouarecodingawebsitetosellclothes,youprobablywanttostorethe heightandweightaswellasothermeasuresofyourcustomerssothatyoucansuggestthe appropriateclothesforthem.So,propertiesarecharacteristicsofanobject.Weusethem allthetime:“Couldyoupassmethatpen?”–“Whichone?”–“Theblackone.”Here,we usedthe“black”propertyofapentoidentifyit(mostlikelyamongstablueandared one). Methodsarethingsthatanobjectcando.Asaperson,Ihavemethodssuchasspeak,walk, sleep,wake-up,eat,dream,write,read,andsoon.AllthethingsthatIcandocouldbe seenasmethodsoftheobjectsthatrepresentsme. So,nowthatyouknowwhatobjectsareandthattheyexposemethodsthatyoucanrun andpropertiesthatyoucaninspect,you’rereadytostartcoding.Codinginfactissimply aboutmanagingthoseobjectsthatliveinthesubsetoftheworldthatwe’rereproducingin oursoftware.Youcancreate,use,reuse,anddeleteobjectsasyouplease. AccordingtotheDataModelchapterontheofficialPythondocumentation: “ObjectsarePython’sabstractionfordata.AlldatainaPythonprogramis representedbyobjectsorbyrelationsbetweenobjects.” We’lltakeacloserlookatPythonobjectsinChapter6,AdvancedConcepts–OOP, Decorators,andIterators.Fornow,allweneedtoknowisthateveryobjectinPythonhas anID(oridentity),atype,andavalue. WOW! eBook www.wowebook.org Oncecreated,theidentityofanobjectisneverchanged.It’sauniqueidentifierforit,and it’susedbehindthescenesbyPythontoretrievetheobjectwhenwewanttouseit. Thetypeaswell,neverchanges.Thetypetellswhatoperationsaresupportedbythe objectandthepossiblevaluesthatcanbeassignedtoit. We’llseePython’smostimportantdatatypesinChapter2,Built-inDataTypes. Thevaluecaneitherchangeornot.Ifitcan,theobjectissaidtobemutable,whilewhen itcannot,theobjectissaidtobeimmutable. Howdoweuseanobject?Wegiveitanameofcourse!Whenyougiveanobjectaname, thenyoucanusethenametoretrievetheobjectanduseit. Inamoregenericsense,objectssuchasnumbers,strings(text),collections,andsoonare associatedwithaname.Usually,wesaythatthisnameisthenameofavariable.Youcan seethevariableasbeinglikeabox,whichyoucanusetoholddata. So,youhavealltheobjectsyouneed:whatnow?Well,weneedtousethem,right?We maywanttosendthemoveranetworkconnectionorstoretheminadatabase.Maybe displaythemonawebpageorwritethemintoafile.Inordertodoso,weneedtoreactto auserfillinginaform,orpressingabutton,oropeningawebpageandperforminga search.Wereactbyrunningourcode,evaluatingconditionstochoosewhichpartsto execute,howmanytimes,andunderwhichcircumstances. Andtodoallthis,basicallyweneedalanguage.That’swhatPythonisfor.Pythonisthe languagewe’llusetogetherthroughoutthisbooktoinstructthecomputertodosomething forus. Now,enoughofthistheoreticalstuff,let’sgetstarted. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org EnterthePython PythonisthemarvelouscreatureofGuidoVanRossum,aDutchcomputerscientistand mathematicianwhodecidedtogifttheworldwithaprojecthewasplayingaroundwith overChristmas1989.Thelanguageappearedtothepublicsomewherearound1991,and sincethenhasevolvedtobeoneoftheleadingprogramminglanguagesusedworldwide today. IstartedprogrammingwhenIwas7yearsold,onaCommodoreVIC20,whichwaslater replacedbyitsbiggerbrother,theCommodore64.ThelanguagewasBASIC.Lateron,I landedonPascal,Assembly,C,C++,Java,JavaScript,VisualBasic,PHP,ASP,ASP .NET,C#,andotherminorlanguagesIcannotevenremember,butonlywhenIlandedon Python,Ifinallyhadthatfeelingthatyouhavewhenyoufindtherightcouchintheshop. Whenallofyourbodypartsareyelling,“Buythisone!Thisoneisperfectforus!” Ittookmeaboutadaytogetusedtoit.ItssyntaxisabitdifferentfromwhatIwasused to,andingeneral,Iveryrarelyworkedwithalanguagethatdefinesscopingwith indentation.Butaftergettingpastthatinitialfeelingofdiscomfort(likehavingnew shoes),Ijustfellinlovewithit.Deeply.Let’sseewhy. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org AboutPython Beforewegetintothegorydetails,let’sgetasenseofwhysomeonewouldwanttouse Python(IwouldrecommendyoutoreadthePythonpageonWikipediatogetamore detailedintroduction). Tomymind,Pythonexposesthefollowingqualities. WOW! eBook www.wowebook.org Portability Pythonrunseverywhere,andportingaprogramfromLinuxtoWindowsorMacisusually justamatteroffixingpathsandsettings.Pythonisdesignedforportabilityandittakes careofoperatingsystem(OS)specificquirksbehindinterfacesthatshieldyoufromthe painofhavingtowritecodetailoredtoaspecificplatform. WOW! eBook www.wowebook.org Coherence Pythonisextremelylogicalandcoherent.Youcanseeitwasdesignedbyabrilliant computerscientist.Mostofthetimeyoucanjustguesshowamethodiscalled,ifyou don’tknowit. Youmaynotrealizehowimportantthisisrightnow,especiallyifyouareatthebeginning, butthisisamajorfeature.Itmeanslessclutteringinyourhead,lessskimmingthroughthe documentation,andlessneedformappinginyourbrainwhenyoucode. WOW! eBook www.wowebook.org Developerproductivity AccordingtoMarkLutz(LearningPython,5thEdition,O’ReillyMedia),aPython programistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++code.This meansthejobgetsdonefaster.Andfasterisgood.Fastermeansafasterresponseonthe market.Lesscodenotonlymeanslesscodetowrite,butalsolesscodetoread(and professionalcodersreadmuchmorethantheywrite),lesscodetomaintain,todebug,and torefactor. AnotherimportantaspectisthatPythonrunswithouttheneedoflengthyandtime consumingcompilationandlinkagesteps,soyoudon’thavetowaittoseetheresultsof yourwork. WOW! eBook www.wowebook.org Anextensivelibrary Pythonhasanincrediblywidestandardlibrary(it’ssaidtocomewith“batteries included”).Ifthatwasn’tenough,thePythoncommunityallovertheworldmaintainsa bodyofthirdpartylibraries,tailoredtospecificneeds,whichyoucanaccessfreelyatthe PythonPackageIndex(PyPI).WhenyoucodePythonandyourealizethatyouneeda certainfeature,inmostcases,thereisatleastonelibrarywherethatfeaturehasalready beenimplementedforyou. WOW! eBook www.wowebook.org Softwarequality Pythonisheavilyfocusedonreadability,coherence,andquality.Thelanguageuniformity allowsforhighreadabilityandthisiscrucialnowadayswherecodeismoreofacollective effortthanasoloexperience.AnotherimportantaspectofPythonisitsintrinsicmultiparadigmnature.Youcanuseitasscriptinglanguage,butyoualsocanexploitobjectoriented,imperative,andfunctionalprogrammingstyles.Itisversatile. WOW! eBook www.wowebook.org Softwareintegration AnotherimportantaspectisthatPythoncanbeextendedandintegratedwithmanyother languages,whichmeansthatevenwhenacompanyisusingadifferentlanguageastheir mainstreamtool,Pythoncancomeinandactasaglueagentbetweencomplex applicationsthatneedtotalktoeachotherinsomeway.Thisiskindofanadvancedtopic, butintherealworld,thisfeatureisveryimportant. WOW! eBook www.wowebook.org Satisfactionandenjoyment Lastbutnotleast,thefunofit!WorkingwithPythonisfun.Icancodefor8hoursand leavetheofficehappyandsatisfied,alientothestruggleothercodershavetoendure becausetheyuselanguagesthatdon’tprovidethemwiththesameamountofwelldesigneddatastructuresandconstructs.Pythonmakescodingfun,nodoubtaboutit.And funpromotesmotivationandproductivity. ThesearethemajoraspectswhyIwouldrecommendPythontoeveryonefor.Ofcourse, therearemanyothertechnicalandadvancedfeaturesthatIcouldhavetalkedabout,but theydon’treallypertaintoanintroductorysectionlikethisone.Theywillcomeup naturally,chapterafterchapter,inthisbook. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Whatarethedrawbacks? Probably,theonlydrawbackthatonecouldfindinPython,whichisnotduetopersonal preferences,istheexecutionspeed.Typically,Pythonisslowerthanitscompiledbrothers. ThestandardimplementationofPythonproduces,whenyourunanapplication,a compiledversionofthesourcecodecalledbytecode(withtheextension.pyc),whichis thenrunbythePythoninterpreter.Theadvantageofthisapproachisportability,whichwe payforwithaslowdownduetothefactthatPythonisnotcompileddowntomachine levelasareotherlanguages. However,Pythonspeedisrarelyaproblemtoday,henceitswideuseregardlessofthis suboptimalfeature.Whathappensisthatinreallife,hardwarecostisnolongeraproblem, andusuallyit’seasyenoughtogainspeedbyparallelizingtasks.Whenitcomesto numbercrunchingthough,onecanswitchtofasterPythonimplementations,suchasPyPy, whichprovidesanaverage7-foldspeedupbyimplementingadvancedcompilation techniques(checkhttp://pypy.org/forreference). Whendoingdatascience,you’llmostlikelyfindthatthelibrariesthatyouusewith Python,suchasPandasandNumpy,achievenativespeedduetothewaytheyare implemented. Ifthatwasn’tagoodenoughargument,youcanalwaysconsiderthatPythonisdrivingthe backendofservicessuchasSpotifyandInstagram,whereperformanceisaconcern. Nonetheless,Pythondoesitsjobperfectlyadequately. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org WhoisusingPythontoday? Notyetconvinced?Let’stakeaverybrieflookatthecompaniesthatareusingPython today:Google,YouTube,Dropbox,Yahoo,ZopeCorporation,IndustrialLight&Magic, WaltDisneyFeatureAnimation,Pixar,NASA,NSA,RedHat,Nokia,IBM,Netflix,Yelp, Intel,Cisco,HP,Qualcomm,andJPMorganChase,justtonameafew. EvengamessuchasBattlefield2,Civilization4,andQuArKareimplementedusing Python. Pythonisusedinmanydifferentcontexts,suchassystemprogramming,web programming,GUIapplications,gamingandrobotics,rapidprototyping,system integration,datascience,databaseapplications,andmuchmore. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Settinguptheenvironment BeforewetalkaboutinstallingPythononyoursystem,letmetellyouaboutwhichPython versionI’llbeusinginthisbook. WOW! eBook www.wowebook.org Python2versusPython3–thegreatdebate Pythoncomesintwomainversions—Python2,whichisthepast—andPython3,whichis thepresent.Thetwoversions,thoughverysimilar,areincompatibleonsomeaspects. Intherealworld,Python2isactuallyquitefarfrombeingthepast.Inshort,eventhough Python3hasbeenoutsince2008,thetransitionphaseisstillfarfrombeingover.Thisis mostlyduetothefactthatPython2iswidelyusedintheindustry,andofcourse, companiesaren’tsokeenonupdatingtheirsystemsjustforthesakeofupdating, followingtheifitain’tbroke,don’tfixitphilosophy.Youcanreadallaboutthetransition betweenthetwoversionsontheWeb. Anotherissuethatwashinderingthetransitionistheavailabilityofthird-partylibraries. Usually,aPythonprojectreliesontensofexternallibraries,andofcourse,whenyoustart anewproject,youneedtobesurethatthereisalreadyaversion3compatiblelibraryfor anybusinessrequirementthatmaycomeup.Ifthat’snotthecase,startingabrandnew projectinPython3meansintroducingapotentialrisk,whichmanycompaniesarenot happytotake. Atthetimeofwriting,themajorityofthemostwidelyusedlibrarieshavebeenportedto Python3,andit’squitesafetostartaprojectinPython3formostcases.Manyofthe librarieshavebeenrewrittensothattheyarecompatiblewithbothversions,mostly harnessingthepowerofthesix(2x3)library,whichhelpsintrospectingandadaptingthe behavioraccordingtotheversionused. OnmyLinuxbox(Ubuntu14.04),IhavethefollowingPythonversion: >>>importsys >>>print(sys.version) 3.4.0(default,Apr112014,13:05:11) [GCC4.8.2] SoyoucanseethatmyPythonversionis3.4.0.TheprecedingtextisalittlebitofPython codethatItypedintomyconsole.We’lltalkaboutitinamoment. AlltheexamplesinthisbookwillberunusingthisPythonversion.Mostofthemwillrun alsoinPython2(Ihaveversion2.7.6installedaswell),andthosethatwon’twilljust requiresomeminoradjustmentstocaterforthesmallincompatibilitiesbetweenthetwo versions.AnotherreasonbehindthischoiceisthatIthinkit’sbettertolearnPython3,and then,ifyouneedto,learnthedifferencesithaswithPython2,ratherthangoingtheother wayaround. Don’tworryaboutthisversionthingthough:it’snotthatbiganissueinpractice. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org InstallingPython Ineverreallygotthepointofhavingasetupsectioninabook,regardlessofwhatitisthat youhavetosetup.Mostofthetime,betweenthetimetheauthorwritestheinstruction andthetimeyouactuallytrythemout,monthshavepassed.Thatis,ifyou’relucky.One versionchangeandthingsmaynotworkthewayitisdescribedinthebook.Luckily,we havetheWebnow,soinordertohelpyougetupandrunning,I’lljustgiveyoupointers andobjectives. Tip IfanyoftheURLsorresourcesI’llpointyoutoarenolongertherebythetimeyouread thisbook,justremember:Googleisyourfriend. WOW! eBook www.wowebook.org SettingupthePythoninterpreter Firstofall,let’stalkaboutyourOS.Pythonisfullyintegratedandmostlikelyalready installedinbasicallyalmosteveryLinuxdistribution.IfyouhaveaMac,it’slikelythat Pythonisalreadythereaswell(however,possiblyonlyPython2.7),whereasifyou’re usingWindows,youprobablyneedtoinstallit. GettingPythonandthelibrariesyouneedupandrunningrequiresabitofhandiwork. LinuxhappenstobethemostuserfriendlyOSforPythonprogrammers,Windowsonthe otherhandistheonethatrequiresthebiggesteffort,Macbeingsomewhereinbetween. Forthisreason,ifyoucanchoose,IsuggestyoutouseLinux.Ifyoucan’t,andyouhavea Mac,thengoforitanyway.IfyouuseWindows,you’llbefinefortheexamplesinthis book,butingeneralworkingwithPythonwillrequireyouabitmoretweaking. MyOSisUbuntu14.04,andthisiswhatIwillusethroughoutthebook,alongwith Python3.4.0. TheplaceyouwanttostartistheofficialPythonwebsite:https://www.python.org.This websitehoststheofficialPythondocumentationandmanyotherresourcesthatyouwill findveryuseful.Takethetimetoexploreit. Tip Anotherexcellent,resourcefulwebsiteonPythonanditsecosystemishttp://docs.pythonguide.org. FindthedownloadsectionandchoosetheinstallerforyourOS.IfyouareonWindows, makesurethatwhenyouruntheinstaller,youchecktheoptioninstallpip(actually,I wouldsuggesttomakeacompleteinstallation,justtobesafe,ofallthecomponentsthe installerholds).We’lltalkaboutpiplater. NowthatPythonisinstalledinyoursystem,theobjectiveistobeabletoopenaconsole andrunthePythoninteractiveshellbytypingpython. Note PleasenotethatIusuallyrefertothePythoninteractiveshellsimplyasPythonconsole. ToopentheconsoleinWindows,gototheStartmenu,chooseRun,andtypecmd.Ifyou encounteranythingthatlookslikeapermissionproblemwhileworkingontheexamples ofthisbook,pleasemakesureyouarerunningtheconsolewithadministratorrights. OntheMacOSX,youcanstartaterminalbygoingtoApplications|Utilities| Terminal. IfyouareonLinux,youknowallthatthereistoknowabouttheconsole. Note IwillusethetermconsoleinterchangeablytoindicatetheLinuxconsole,theWindows commandprompt,andtheMacterminal.Iwillalsoindicatethecommand-lineprompt withtheLinuxdefaultformat,likethis: WOW! eBook www.wowebook.org $sudoapt-getupdate Whateverconsoleyouopen,typepythonattheprompt,andmakesurethePython interactiveshellshowsup.Typeexit()toquit.Keepinmindthatyoumayhaveto specifypython3ifyourOScomeswithPython2.*preinstalled. ThisishowitshouldlookonWindows7: AndthisishowitshouldlookonLinux: NowthatPythonissetupandyoucanrunit,it’stimetomakesureyouhavetheothertool thatwillbeindispensabletofollowtheexamplesinthebook:virtualenv. WOW! eBook www.wowebook.org Aboutvirtualenv Asyouprobablyhaveguessedbyitsname,virtualenvisallaboutvirtualenvironments. Letmeexplainwhattheyareandwhyweneedthemandletmedoitbymeansofasimple example. YouinstallPythononyoursystemandyoustartworkingonawebsiteforclientX.You createaprojectfolderandstartcoding.Alongthewayyoualsoinstallsomelibraries,for exampletheDjangoframework,whichwe’llseeindepthinChapter10,Web DevelopmentDoneRight.Let’ssaytheDjangoversionyouinstallforprojectXis1.7.1. Now,yourwebsiteissogoodthatyougetanotherclient,Y.Hewantsyoutobuildanother website,soyoustartprojectYand,alongtheway,youneedtoinstallDjangoagain.The onlyissueisthatnowtheDjangoversionis1.8andyoucannotinstallitonyoursystem becausethiswouldreplacetheversionyouinstalledforprojectX.Youdon’twanttorisk introducingincompatibilityissues,soyouhavetwochoices:eitheryoustickwiththe versionyouhavecurrentlyonyourmachine,oryouupgradeitandmakesurethefirst projectisstillfullyworkingcorrectlywiththenewversion. Let’sbehonest,neitheroftheseoptionsisveryappealing,right?Definitelynot.So,here’s thesolution:virtualenv! virtualenvisatoolthatallowsyoutocreateavirtualenvironment.Inotherwords,itisa tooltocreateisolatedPythonenvironments,eachofwhichisafolderthatcontainsallthe necessaryexecutablestousethepackagesthataPythonprojectwouldneed(thinkof packagesaslibrariesforthetimebeing). SoyoucreateavirtualenvironmentforprojectX,installallthedependencies,andthen youcreateavirtualenvironmentforprojectY,installingallitsdependencieswithoutthe slightestworrybecauseeverylibraryyouinstallendsupwithintheboundariesofthe appropriatevirtualenvironment.Inourexample,projectXwillholdDjango1.7.1,while projectYwillholdDjango1.8. Note Itisofvitalimportancethatyouneverinstalllibrariesdirectlyatthesystemlevel.Linux forexamplereliesonPythonformanydifferenttasksandoperations,andifyoufiddle withthesysteminstallationofPython,youriskcompromisingtheintegrityofthewhole system(guesstowhomthishappened…).Sotakethisasarule,suchasbrushingyour teethbeforegoingtobed:always,alwayscreateavirtualenvironmentwhenyoustarta newproject. Toinstallvirtualenvonyoursystem,thereareafewdifferentways.OnaDebian-based distributionofLinuxforexample,youcaninstallitwiththefollowingcommand: $sudoapt-getinstallpython-virtualenv Probably,theeasiestwayistousepipthough,withthefollowingcommand: $sudopipinstallvirtualenv#sudomaybyoptional WOW! eBook www.wowebook.org pipisapackagemanagementsystemusedtoinstallandmanagesoftwarepackages writteninPython. Python3hasbuilt-insupportforvirtualenvironments,butinpractice,theexternal librariesarestillthedefaultonproductionsystems.Ifyouhavetroublegettingvirtualenv upandrunning,pleaserefertothevirtualenvofficialwebsite:https://virtualenv.pypa.io. WOW! eBook www.wowebook.org Yourfirstvirtualenvironment Itisveryeasytocreateavirtualenvironment,butaccordingtohowyoursystemis configuredandwhichPythonversionyouwantthevirtualenvironmenttorun,youneedto runthecommandproperly.Anotherthingyouwillneedtodowithavirtualenv,whenyou wanttoworkwithit,istoactivateit.Activatingavirtualenvbasicallyproducessomepath jugglingbehindthescenessothatwhenyoucallthePythoninterpreter,you’reactually callingtheactivevirtualenvironmentone,insteadofthemeresystemone. I’llshowyouafullexampleonbothLinuxandWindows.Wewill: 1. Createafoldernamedlearning.pythonunderyourprojectroot(whichinmycaseis afoldercalledsrv,inmyhomefolder).Pleaseadaptthepathsaccordingtothesetup youfancyonyourbox. 2. Withinthelearning.pythonfolder,wewillcreateavirtualenvironmentcalled .lpvenv. Note Somedevelopersprefertocallallvirtualenvironmentsusingthesamename(for example,.venv).Thiswaytheycanrunscriptsagainstanyvirtualenvbyjust knowingthenameoftheprojecttheydwellin.Thisisaverycommontechniquethat Iuseaswell.Thedotin.venvisbecauseinLinux/Macprependinganamewitha dotmakesthatfileorfolderinvisible. 3. Aftercreatingthevirtualenvironment,wewillactivateit(thisisslightlydifferent betweenLinux,Mac,andWindows). 4. Then,we’llmakesurethatwearerunningthedesiredPythonversion(3.4.*)by runningthePythoninteractiveshell. 5. Finally,wewilldeactivatethevirtualenvironmentusingthedeactivatecommand. Thesefivesimplestepswillshowyouallyouhavetodotostartanduseaproject. Here’sanexampleofhowthosestepsmightlooklikeonLinux(commandsthatstartwith a#arecomments): WOW! eBook www.wowebook.org NoticethatIhadtoexplicitlytellvirtualenvtousethePython3.4interpreterbecauseon myboxPython2.7isthedefaultone.HadInotdonethat,Iwouldhavehadavirtual environmentwithPython2.7insteadofPython3.4. Youcancombinethetwoinstructionsforstep2inonesinglecommandlikethis: $virtualenv-p$(whichpython3.4).lpvenv Ipreferredtobeexplicitlyverboseinthisinstance,tohelpyouunderstandeachbitofthe procedure. Anotherthingtonoticeisthatinordertoactivateavirtualenvironment,weneedtorun the/bin/activatescript,whichneedstobesourced(whenascriptis“sourced”,itmeans thatitseffectsstickaroundwhenit’sdonerunning).Thisisveryimportant.Alsonotice howthepromptchangesafterweactivatethevirtualenvironment,showingitsnameon theleft(andhowitdisappearswhenwedeactivate).InMacOS,thestepsarethesameso Iwon’trepeatthemhere. Nowlet’shavealookathowwecanachievethesameresultinWindows.Youwill probablyhavetoplayaroundabit,especiallyifyouhaveadifferentWindowsorPython versionthanI’musinghere.Thisisallgoodexperiencethough,sotryandthinkpositively attheinitialstrugglethateverycoderhastogothroughinordertogetthingsgoing. WOW! eBook www.wowebook.org Here’showitshouldlookonWindows(commandsthatstartwith::arecomments): NoticethereareafewsmalldifferencesfromtheLinuxversion.Apartfromthe commandstocreateandnavigatethefolders,oneimportantdifferenceishowyouactivate yourvirtualenv.Also,inWindowsthereisnowhichcommand,soweusedthewhere command. Atthispoint,youshouldbeabletocreateandactivateavirtualenvironment.Pleasetry andcreateanotheronewithoutmeguidingyou,getacquaintedtothisprocedurebecause it’ssomethingthatyouwillalwaysbedoing:weneverworksystem-widewithPython, remember?It’sextremelyimportant. So,withthescaffoldingoutoftheway,we’rereadytotalkabitmoreaboutPythonand howyoucanuseit.Beforewedoitthough,allowmetospendafewwordsaboutthe console. WOW! eBook www.wowebook.org Yourfriend,theconsole InthiseraofGUIsandtouchscreendevices,itseemsalittleridiculoustohavetoresortto atoolsuchastheconsole,wheneverythingisjustaboutoneclickaway. Butthetruthiseverytimeyouremoveyourrighthandfromthekeyboard(ortheleftone, ifyou’realefty)tograbyourmouseandmovethecursorovertothespotyouwantto click,you’relosingtime.Gettingthingsdonewiththeconsole,counter-intuitivelyasit maybe,resultsinhigherproductivityandspeed.Iknow,youhavetotrustmeonthis. Speedandproductivityareimportantandpersonally,Ihavenothingagainstthemouse, butthereisanotherverygoodreasonforwhichyoumaywanttogetwellacquaintedwith theconsole:whenyoudevelopcodethatendsuponsomeserver,theconsolemightbethe onlyavailabletool.Ifyoumakefriendswithit,Ipromiseyou,youwillnevergetlost whenit’sofutmostimportancethatyoudon’t(typically,whenthewebsiteisdownand youhavetoinvestigateveryquicklywhat’sgoingon). Soit’sreallyuptoyou.Ifyou’reindoubt,pleasegrantmethebenefitofthedoubtand giveitatry.It’seasierthanyouthink,andyou’llneverregretit.Thereisnothingmore pitifulthanagooddeveloperwhogetslostwithinanSSHconnectiontoaserverbecause theyareusedtotheirowncustomsetoftools,andonlytothat. Now,let’sgetbacktoPython. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org HowyoucanrunaPythonprogram ThereareafewdifferentwaysinwhichyoucanrunaPythonprogram. WOW! eBook www.wowebook.org RunningPythonscripts Pythoncanbeusedasascriptinglanguage.Infact,italwaysprovesitselfveryuseful. Scriptsarefiles(usuallyofsmalldimensions)thatyounormallyexecutetodosomething likeatask.Manydevelopersenduphavingtheirownarsenaloftoolsthattheyfirewhen theyneedtoperformatask.Forexample,youcanhavescriptstoparsedatainaformat andrenderitintoanotherdifferentformat.Oryoucanuseascripttoworkwithfilesand folders.Youcancreateormodifyconfigurationfiles,andmuchmore.Technically,thereis notmuchthatcannotbedoneinascript. It’squitecommontohavescriptsrunningataprecisetimeonaserver.Forexample,if yourwebsitedatabaseneedscleaningevery24hours(forexample,thetablethatstoresthe usersessions,whichexpireprettyquicklybutaren’tcleanedautomatically),youcouldset upacronjobthatfiresyourscriptat3:00A.M.everyday. Note AccordingtoWikipedia,thesoftwareutilityCronisatime-basedjobschedulerinUnixlikecomputeroperatingsystems.Peoplewhosetupandmaintainsoftwareenvironments usecrontoschedulejobs(commandsorshellscripts)torunperiodicallyatfixedtimes, dates,orintervals. IhavePythonscriptstodoallthemenialtasksthatwouldtakememinutesormoretodo manually,andatsomepoint,Idecidedtoautomate.Forexample,Ihavealaptopthat doesn’thaveaFnkeytotogglethetouchpadonandoff.Ifindthisveryannoying,andI don’twanttogoclickingaboutthroughseveralmenuswhenIneedtodoit,soIwrotea smallscriptthatissmartenoughtotellmysystemtotogglethetouchpadactivestate,and nowIcandoitwithonesimpleclickfrommylauncher.Priceless. We’lldevotehalfofChapter8,TheEdges–GUIsandScriptsonscriptingwithPython. WOW! eBook www.wowebook.org RunningthePythoninteractiveshell AnotherwayofrunningPythonisbycallingtheinteractiveshell.Thisissomethingwe alreadysawwhenwetypedpythononthecommandlineofourconsole. Soopenaconsole,activateyourvirtualenvironment(whichbynowshouldbesecond naturetoyou,right?),andtypepython.Youwillbepresentedwithacoupleoflinesthat shouldlooklikethis(ifyouareonLinux): Python3.4.0(default,Apr112014,13:05:11) [GCC4.8.2]onlinux Type"help","copyright","credits"or"license"formoreinformation. Those>>>arethepromptoftheshell.TheytellyouthatPythoniswaitingforyoutotype something.Ifyoutypeasimpleinstruction,somethingthatfitsinoneline,that’sallyou’ll see.However,ifyoutypesomethingthatrequiresmorethanonelineofcode,theshell willchangethepromptto...,givingyouavisualcluethatyou’retypingamultiline statement(oranythingthatwouldrequiremorethanonelineofcode). Goon,tryitout,let’sdosomebasicmaths: >>>2+4 6 >>>10/4 2.5 >>>2**1024 179769313486231590772930519078902473361797697894230657273430081157732675805 500963132708477322407536021120113879871393357658789768814416622492847430639 474124377767893424865485276302219601246094119453082952085005768838150682342 462881473913110540827237163350510684586298239947245938479716304835356329624 224137216 Thelastoperationisshowingyousomethingincredible.Weraise2tothepowerof1024, andPythonishandlingthistaskwithnotroubleatall.TrytodoitinJava,C++,orC#.It won’twork,unlessyouusespeciallibrariestohandlesuchbignumbers. Iusetheinteractiveshelleveryday.It’sextremelyusefultodebugveryquickly,for example,tocheckifadatastructuresupportsanoperation.Ormaybetoinspectorruna pieceofcode. WhenyouuseDjango(awebframework),theinteractiveshelliscoupledwithitand allowsyoutoworkyourwaythroughtheframeworktools,toinspectthedatainthe database,andmanymorethings.Youwillfindthattheinteractiveshellwillsoonbecome oneofyourdearestfriendsonthejourneyyouareembarkingon. Anothersolution,whichcomesinamuchnicergraphiclayout,istouseIDLE (IntegratedDeveLopmentEnvironment).It’squiteasimpleIDE,whichisintended mostlyforbeginners.Ithasaslightlylargersetofcapabilitiesthanthenakedinteractive shellyougetintheconsole,soyoumaywanttoexploreit.Itcomesforfreeinthe WindowsPythoninstallerandyoucaneasilyinstallitinanyothersystem.Youcanfind informationaboutitonthePythonwebsite. WOW! eBook www.wowebook.org GuidoVanRossumnamedPythonaftertheBritishcomedygroupMontyPython,soit’s rumoredthatthenameIDLEhasbeenchoseninhonorofErikIdle,oneofMonty Python’sfoundingmembers. WOW! eBook www.wowebook.org RunningPythonasaservice Apartfrombeingrunasascript,andwithintheboundariesofashell,Pythoncanbecoded andrunaspropersoftware.We’llseemanyexamplesthroughoutthebookaboutthis mode.Andwe’llunderstandmoreaboutitinamoment,whenwe’lltalkabouthow Pythoncodeisorganizedandrun. WOW! eBook www.wowebook.org RunningPythonasaGUIapplication PythoncanalsoberunasaGUI(GraphicalUserInterface).Thereareseveral frameworksavailable,someofwhicharecross-platformandsomeothersareplatformspecific.InChapter8,TheEdges–GUIsandScripts,we’llseeanexampleofaGUI applicationcreatedusingTkinter,whichisanobject-orientedlayerthatlivesontopofTk (TkintermeansTkInterface). Note Tkisagraphicaluserinterfacetoolkitthattakesdesktopapplicationdevelopmenttoa higherlevelthantheconventionalapproach.ItisthestandardGUIforTcl(Tool CommandLanguage),butalsoformanyotherdynamiclanguagesandcanproducerich nativeapplicationsthatrunseamlesslyunderWindows,Linux,MacOSX,andmore. TkintercomesbundledwithPython,thereforeitgivestheprogrammereasyaccesstothe GUIworld,andforthesereasons,IhavechosenittobetheframeworkfortheGUI examplesthatI’llpresentinthisbook. AmongtheotherGUIframeworks,wefindthatthefollowingarethemostwidelyused: PyQt wxPython PyGtk Describingthemindetailisoutsidethescopeofthisbook,butyoucanfindallthe informationyouneedonthePythonwebsiteintheGUIProgrammingsection.IfGUIsare whatyou’relookingfor,remembertochoosetheoneyouwantaccordingtosome principles.Makesurethey: Offerallthefeaturesyoumayneedtodevelopyourproject Runonalltheplatformsyoumayneedtosupport Relyonacommunitythatisaswideandactiveaspossible Wrapgraphicdrivers/toolsthatyoucaneasilyinstall/access WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org HowisPythoncodeorganized Let’stalkalittlebitabouthowPythoncodeisorganized.Inthisparagraph,we’llstart goingdowntherabbitholealittlebitmoreandintroduceabitmoretechnicalnamesand concepts. Startingwiththebasics,howisPythoncodeorganized?Ofcourse,youwriteyourcode intofiles.Whenyousaveafilewiththeextension.py,thatfileissaidtobeaPython module. Tip Ifyou’reonWindowsorMac,whichtypicallyhidefileextensionstotheuser,please makesureyouchangetheconfigurationsothatyoucanseethecompletenameofthe files.Thisisnotstrictlyarequirement,butaheartysuggestion. Itwouldbeimpracticaltosaveallthecodethatitisrequiredforsoftwaretoworkwithin onesinglefile.Thatsolutionworksforscripts,whichareusuallynotlongerthanafew hundredlines(andoftentheyarequiteshorterthanthat). AcompletePythonapplicationcanbemadeofhundredsofthousandsoflinesofcode,so youwillhavetoscatteritthroughdifferentmodules.Better,butnotnearlygoodenough.It turnsoutthatevenlikethisitwouldstillbeimpracticaltoworkwiththecode.SoPython givesyouanotherstructure,calledpackage,whichallowsyoutogroupmodulestogether. Apackageisnothingmorethanafolder,whichmustcontainaspecialfile,__init__.py thatdoesn’tneedtoholdanycodebutwhosepresenceisrequiredtotellPythonthatthe folderisnotjustsomefolder,butit’sactuallyapackage(notethatasofPython3.3 __init__.pyisnotstrictlyrequiredanymore). Asalways,anexamplewillmakeallofthismuchclearer.Ihavecreatedanexample structureinmybookproject,andwhenItypeinmyLinuxconsole: $tree-vexample Igetatreerepresentationofthecontentsofthech1/examplefolder,whichholdsthecode fortheexamplesofthischapter.Here’showastructureofarealsimpleapplicationcould looklike: example/ ├──core.py ├──run.py └──util ├──__init__.py ├──db.py ├──math.py └──network.py Youcanseethatwithintherootofthisexample,wehavetwomodules,core.pyand run.py,andonepackage:util.Withincore.py,theremaybethecorelogicofour application.Ontheotherhand,withintherun.pymodule,wecanprobablyfindthelogic tostarttheapplication.Withintheutilpackage,Iexpecttofindvariousutilitytools,and WOW! eBook www.wowebook.org infact,wecanguessthatthemodulestherearecalledbythetypeoftoolstheyhold: db.pywouldholdtoolstoworkwithdatabases,math.pywouldofcoursehold mathematicaltools(maybeourapplicationdealswithfinancialdata),andnetwork.py wouldprobablyholdtoolstosend/receivedataonnetworks. Asexplainedbefore,the__init__.pyfileistherejusttotellPythonthatutilisa packageandnotjustamerefolder. Hadthissoftwarebeenorganizedwithinmodulesonly,itwouldhavebeenmuchharderto inferitsstructure.Iputamoduleonlyexampleunderthech1/files_onlyfolder,seeitfor yourself: $tree-vfiles_only Thisshowsusacompletelydifferentpicture: files_only/ ├──core.py ├──db.py ├──math.py ├──network.py └──run.py Itisalittlehardertoguesswhateachmoduledoes,right?Now,considerthatthisisjusta simpleexample,soyoucanguesshowmuchharderitwouldbetounderstandareal applicationifwecouldn’torganizethecodeinpackagesandmodules. WOW! eBook www.wowebook.org Howdoweusemodulesandpackages Whenadeveloperiswritinganapplication,itisverylikelythattheywillneedtoapply thesamepieceoflogicindifferentpartsofit.Forexample,whenwritingaparserforthe datathatcomesfromaformthatausercanfillinawebpage,theapplicationwillhaveto validatewhetheracertainfieldisholdinganumberornot.Regardlessofhowthelogicfor thiskindofvalidationiswritten,it’sverylikelythatitwillbeneededinmorethanone place.Forexampleinapollapplication,wheretheuserisaskedmanyquestion,it’slikely thatseveralofthemwillrequireanumericanswer.Forexample: Whatisyourage Howmanypetsdoyouown Howmanychildrendoyouhave Howmanytimeshaveyoubeenmarried Itwouldbeverybadpracticetocopypaste(or,moreproperlysaid:duplicate)the validationlogicineveryplacewhereweexpectanumericanswer.Thiswouldviolatethe DRY(Don’tRepeatYourself)principle,whichstatesthatyoushouldneverrepeatthe samepieceofcodemorethanonceinyourapplication.Ifeeltheneedtostressthe importanceofthisprinciple:youshouldneverrepeatthesamepieceofcodemorethan onceinyourapplication(gottheirony?). Thereareseveralreasonswhyrepeatingthesamepieceoflogiccanbeverybad,themost importantonesbeing: Therecouldbeabuginthelogic,andtherefore,youwouldhavetocorrectitinevery placethatlogicisapplied. Youmaywanttoamendthewayyoucarryoutthevalidation,andagainyouwould havetochangeitineveryplaceitisapplied. Youmayforgettofix/amendapieceoflogicbecauseyoumisseditwhensearching forallitsoccurrences.Thiswouldleavewrong/inconsistentbehaviorinyour application. Yourcodewouldbelongerthanneeded,fornogoodreason. Pythonisawonderfullanguageandprovidesyouwithallthetoolsyouneedtoapplyall thecodingbestpractices.Forthisparticularexample,weneedtobeabletoreuseapiece ofcode.Tobeabletoreuseapieceofcode,weneedtohaveaconstructthatwillholdthe codeforussothatwecancallthatconstructeverytimeweneedtorepeatthelogicinside it.Thatconstructexists,andit’scalledfunction. I’mnotgoingtoodeepintothespecificshere,sopleasejustrememberthatafunctionisa blockoforganized,reusablecodewhichisusedtoperformatask.Functionscanassume manyformsandnames,accordingtowhatkindofenvironmenttheybelongto,butfor nowthisisnotimportant.We’llseethedetailswhenweareabletoappreciatethem,later on,inthebook.Functionsarethebuildingblocksofmodularityinyourapplication,and theyarealmostindispensable(unlessyou’rewritingasupersimplescript,you’lluse functionsallthetime).We’llexplorefunctionsinChapter4,Functions,theBuilding BlocksofCode. WOW! eBook www.wowebook.org Pythoncomeswithaveryextensivelibrary,asIalreadysaidafewpagesago.Now, maybeit’sagoodtimetodefinewhatalibraryis:alibraryisacollectionoffunctions andobjectsthatprovidefunctionalitiesthatenrichtheabilitiesofalanguage. Forexample,withinPython’smathlibrarywecanfindaplethoraoffunctions,oneof whichisthefactorialfunction,whichofcoursecalculatesthefactorialofanumber. Note Inmathematics,thefactorialofanon-negativeintegernumberN,denotedasN!,is definedastheproductofallpositiveintegerslessthanorequaltoN.Forexample,the factorialof5iscalculatedas: 5!=5*4*3*2*1=120 Thefactorialof0is0!=1,torespecttheconventionforanemptyproduct. So,ifyouwantedtousethisfunctioninyourcode,allyouwouldhavetodoistoimportit andcallitwiththerightinputvalues.Don’tworrytoomuchifinputvaluesandthe conceptofcallingisnotveryclearfornow,pleasejustconcentrateontheimportpart. Note Weusealibrarybyimportingwhatweneedfromit,andthenweuseit. InPython,tocalculatethefactorialofnumber5,wejustneedthefollowingcode: >>>frommathimportfactorial >>>factorial(5) 120 Note Whateverwetypeintheshell,ifithasaprintablerepresentation,willbeprintedonthe consoleforus(inthiscase,theresultofthefunctioncall:120). So,let’sgobacktoourexample,theonewithcore.py,run.py,util,andsoon. Inourexample,thepackageutilisourutilitylibrary.Ourcustomutilitybeltthatholds allthosereusabletools(thatis,functions),whichweneedinourapplication.Someof themwilldealwithdatabases(db.py),somewiththenetwork(network.py),andsome willperformmathematicalcalculations(math.py)thatareoutsidethescopeofPython’s standardmathlibraryandtherefore,wehadtocodethemforourselves. Wewillseeindetailhowtoimportfunctionsandusethemintheirdedicatedchapter. Let’snowtalkaboutanotherveryimportantconcept:Python’sexecutionmodel. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Python’sexecutionmodel Inthisparagraph,Iwouldliketointroduceyoutoafewveryimportantconcepts,suchas scope,names,andnamespaces.YoucanreadallaboutPython’sexecutionmodelinthe officialLanguagereference,ofcourse,butIwouldarguethatitisquitetechnicaland abstract,soletmegiveyoualessformalexplanationfirst. WOW! eBook www.wowebook.org Namesandnamespaces Sayyouarelookingforabook,soyougotothelibraryandasksomeoneforthebookyou wanttofetch.Theytellyousomethinglike“secondfloor,sectionX,rowthree”.Soyou goupthestairs,lookforsectionX,andsoon. Itwouldbeverydifferenttoenteralibrarywhereallthebooksarepiledtogetherin randomorderinonebigroom.Nofloors,nosections,norows,noorder.Fetchingabook wouldbeextremelyhard. Whenwewritecodewehavethesameissue:wehavetotryandorganizeitsothatitwill beeasyforsomeonewhohasnopriorknowledgeaboutittofindwhatthey’relookingfor. Whensoftwareisstructuredcorrectly,italsopromotescodereuse.Ontheotherhand, disorganizedsoftwareismorelikelytoexposescatteredpiecesofduplicatedlogic. Firstofall,let’sstartwiththebook.WerefertoabookbyitstitleandinPythonlingo,that wouldbeaname.Pythonnamesaretheclosestabstractiontowhatotherlanguagescall variables.Namesbasicallyrefertoobjectsandareintroducedbynamebindingoperations. Let’smakeaquickexample(noticethatanythingthatfollowsa#isacomment): >>>n=3#integernumber >>>address="221bBakerStreet,NW16XE,London"#S.Holmes >>>employee={ ...'age':45, ...'role':'CTO', ...'SSN':'AB1234567', ...} >>>#let'sprintthem >>>n 3 >>>address '221bBakerStreet,NW16XE,London' >>>employee {'role':'CTO','SSN':'AB1234567','age':45} >>>#whatifItrytoprintanameIdidn'tdefine? >>>other_name Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> NameError:name'other_name'isnotdefined Wedefinedthreeobjectsintheprecedingcode(doyourememberwhatarethethree featureseveryPythonobjecthas?): Anintegernumbern(type:int,value:3) Astringaddress(type:str,value:SherlockHolmes’address) Adictionaryemployee(type:dict,value:adictionarywhichholdsthreekey/value pairs) Don’tworry,Iknowyou’renotsupposedtoknowwhatadictionaryis.We’llseeinthe nextchapterthatit’sthekingofPythondatastructures. Tip WOW! eBook www.wowebook.org Haveyounoticedthatthepromptchangedfrom>>>to...whenItypedinthedefinition ofemployee?That’sbecausethedefinitionspansovermultiplelines. So,whataren,addressandemployee?Theyarenames.Namesthatwecanuseto retrievedatawithinourcode.Theyneedtobekeptsomewheresothatwheneverweneed toretrievethoseobjects,wecanusetheirnamestofetchthem.Weneedsomespaceto holdthem,hence:namespaces! Anamespaceisthereforeamappingfromnamestoobjects.Examplesarethesetofbuiltinnames(containingfunctionsthatarealwaysaccessibleforfreeinanyPythonprogram), theglobalnamesinamodule,andthelocalnamesinafunction.Eventhesetofattributes ofanobjectcanbeconsideredanamespace. Thebeautyofnamespacesisthattheyallowyoutodefineandorganizeyournameswith clarity,withoutoverlappingorinterference.Forexample,thenamespaceassociatedwith thatbookwewerelookingforinthelibrarycanbeusedtoimportthebookitself,likethis: fromlibrary.second_floor.section_x.row_threeimportbook Westartfromthelibrarynamespace,andbymeansofthedot(.)operator,wewalkinto thatnamespace.Withinthisnamespace,welookforsecond_floor,andagainwewalk intoitwiththe.operator.Wethenwalkintosection_x,andfinallywithinthelast namespace,row_tree,wefindthenamewewerelookingfor:book. Walkingthroughanamespacewillbeclearerwhenwe’llbedealingwithrealcode examples.Fornow,justkeepinmindthatnamespacesareplaceswherenamesare associatedtoobjects. Thereisanotherconcept,whichiscloselyrelatedtothatofanamespace,whichI’dliketo brieflytalkabout:thescope. WOW! eBook www.wowebook.org Scopes AccordingtoPython’sdocumentation,ascopeisatextualregionofaPythonprogram, whereanamespaceisdirectlyaccessible.Directlyaccessiblemeansthatwhenyou’re lookingforanunqualifiedreferencetoaname,Pythontriestofinditinthenamespace. Scopesaredeterminedstatically,butactuallyduringruntimetheyareuseddynamically. Thismeansthatbyinspectingthesourcecodeyoucantellwhatthescopeofanobjectis, butthisdoesn’tpreventthesoftwaretoalterthatduringruntime.Therearefourdifferent scopesthatPythonmakesaccessible(notnecessarilyallofthempresentatthesametime, ofcourse): Thelocalscope,whichistheinnermostoneandcontainsthelocalnames. Theenclosingscope,thatis,thescopeofanyenclosingfunction.Itcontainsnonlocalnamesandalsonon-globalnames. Theglobalscopecontainstheglobalnames. Thebuilt-inscopecontainsthebuilt-innames.Pythoncomeswithasetoffunctions thatyoucanuseinaoff-the-shelffashion,suchasprint,all,abs,andsoon.They liveinthebuilt-inscope. Theruleisthefollowing:whenwerefertoaname,Pythonstartslookingforitinthe currentnamespace.Ifthenameisnotfound,Pythoncontinuesthesearchtotheenclosing scopeandthiscontinueuntilthebuilt-inscopeissearched.Ifanamehasn’tbeenfound aftersearchingthebuilt-inscope,thenPythonraisesaNameErrorexception,which basicallymeansthatthenamehasn’tbeendefined(yousawthisinthepreceding example). Theorderinwhichthenamespacesarescannedwhenlookingforanameistherefore: local,enclosing,global,built-in(LEGB). Thisisallverytheoretical,solet’sseeanexample.InordertoshowyouLocaland Enclosingnamespaces,Iwillhavetodefineafewfunctions.Don’tworryifyouarenot familiarwiththeirsyntaxforthemoment,we’llstudyfunctionsinChapter4,Functions, theBuildingBlocksofCode.Justrememberthatinthefollowingcode,whenyouseedef, itmeansI’mdefiningafunction. scopes1.py #LocalversusGlobal #wedefineafunction,calledlocal deflocal(): m=7 print(m) m=5 print(m) #wecall,or`execute`thefunctionlocal local() WOW! eBook www.wowebook.org Intheprecedingexample,wedefinethesamenamem,bothintheglobalscopeandinthe localone(theonedefinedbythefunctionlocal).Whenweexecutethisprogramwiththe followingcommand(haveyouactivatedyourvirtualenv?): $pythonscopes1.py Weseetwonumbersprintedontheconsole:5and7. WhathappensisthatthePythoninterpreterparsesthefile,toptobottom.First,itfindsa coupleofcommentlines,whichareskipped,thenitparsesthedefinitionofthefunction local.Whencalled,thisfunctiondoestwothings:itsetsupanametoanobject representingnumber7andprintsit.ThePythoninterpreterkeepsgoinganditfinds anothernamebinding.Thistimethebindinghappensintheglobalscopeandthevalueis 5.Thenextlineisacalltotheprintfunction,whichisexecuted(andsowegetthefirst valueprintedontheconsole:5). Afterthis,thereisacalltothefunctionlocal.Atthispoint,Pythonexecutesthefunction, soatthistime,thebindingm=7happensandit’sprinted. Oneveryimportantthingtonoticeisthatthepartofthecodethatbelongstothedefinition ofthefunctionlocalisindentedbyfourspacesontheright.Pythoninfactdefinesscopes byindentingthecode.Youwalkintoascopebyindentingandwalkoutofitby unindenting.Somecodersusetwospaces,othersthree,butthesuggestednumberof spacestouseisfour.It’sagoodmeasuretomaximizereadability.We’lltalkmoreabout alltheconventionsyoushouldembracewhenwritingPythoncodelater. Whatwouldhappenifweremovedthatm=7line?RemembertheLEGBrule.Python wouldstartlookingforminthelocalscope(functionlocal),and,notfindingit,itwould gotothenextenclosingscope.Thenextoneinthiscaseistheglobalonebecausethereis noenclosingfunctionwrappedaroundlocal.Therefore,wewouldseetwonumber5 printedontheconsole.Let’sactuallyseehowthecodewouldlooklike: scopes2.py #LocalversusGlobal deflocal(): #mdoesn'tbelongtothescopedefinedbythelocalfunction #soPythonwillkeeplookingintothenextenclosingscope. #misfinallyfoundintheglobalscope print(m,'printingfromthelocalscope') m=5 print(m,'printingfromtheglobalscope') local() Runningscopes2.pywillprintthis: (.lpvenv)fab@xps:ch1$pythonscopes2.py 5printingfromtheglobalscope 5printingfromthelocalscope WOW! eBook www.wowebook.org Asexpected,Pythonprintsmthefirsttime,thenwhenthefunctionlocaliscalled,misn’t foundinitsscope,soPythonlooksforitfollowingtheLEGBchainuntilmisfoundinthe globalscope. Let’sseeanexamplewithanextralayer,theenclosingscope: scopes3.py #Local,EnclosingandGlobal defenclosing_func(): m=13 deflocal(): #mdoesn'tbelongtothescopedefinedbythelocal #functionsoPythonwillkeeplookingintothenext #enclosingscope.Thistimemisfoundintheenclosing #scope print(m,'printingfromthelocalscope') #callingthefunctionlocal local() m=5 print(m,'printingfromtheglobalscope') enclosing_func() Runningscopes3.pywillprintontheconsole: (.lpvenv)fab@xps:ch1$pythonscopes3.py 5printingfromtheglobalscope 13printingfromthelocalscope Asyoucansee,theprintinstructionfromthefunctionlocalisreferringtomasbefore.m isstillnotdefinedwithinthefunctionitself,soPythonstartswalkingscopesfollowingthe LEGBorder.Thistimemisfoundintheenclosingscope. Don’tworryifthisisstillnotperfectlyclearfornow.Itwillcometoyouaswegothrough theexamplesinthebook.TheClassessectionofthePythontutorial(official documentation)hasaninterestingparagraphaboutscopesandnamespaces.Makesureyou readitatsomepointifyouwishforadeeperunderstandingofthesubject. Beforewefinishoffthischapter,Iwouldliketotalkabitmoreaboutobjects.Afterall, basicallyeverythinginPythonisanobject,soIthinktheydeserveabitmoreattention. Tip Downloadingtheexamplecode Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.com forallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbook elsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou. WOW! eBook www.wowebook.org Objectandclasses WhenIintroducedobjectsintheAproperintroductionsection,Isaidthatweusethemto representreal-lifeobjects.Forexample,wesellgoodsofanykindontheWebnowadays andweneedtobeabletohandle,store,andrepresentthemproperly.Butobjectsare actuallysomuchmorethanthat.Mostofwhatyouwilleverdo,inPython,hastodowith manipulatingobjects. So,withoutgoingtoomuchintodetail(we’lldothatinChapter6,AdvancedConcepts– OOP,Decorators,andIterators),Iwanttogiveyoutheinanutshellkindofexplanation aboutclassesandobjects. We’vealreadyseenthatobjectsarePython’sabstractionfordata.Infact,everythingin Pythonisanobject.Numbers,strings(datastructuresthatholdtext),containers, collections,evenfunctions.Youcanthinkofthemasiftheywereboxeswithatleastthree features:anID(unique),atype,andavalue. Buthowdotheycometolife?Howdowecreatethem?Howtowewriteourowncustom objects?Theanswerliesinonesimpleword:classes. Objectsare,infact,instancesofclasses.ThebeautyofPythonisthatclassesareobjects themselves,butlet’snotgodownthisroad.Itleadstooneofthemostadvancedconcepts ofthislanguage:metaclasses.We’lltalkverybrieflyabouttheminChapter6,Advanced Concepts–OOP,Decorators,andIterators.Fornow,thebestwayforyoutogetthe differencebetweenclassesandobjects,isbymeansofanexample. Sayafriendtellsyou“Iboughtanewbike!”Youimmediatelyunderstandwhatshe’s talkingabout.Haveyouseenthebike?No.Doyouknowwhatcoloritis?Nope.The brand?Nope.Doyouknowanythingaboutit?Nope.Butatthesametime,youknow everythingyouneedinordertounderstandwhatyourfriendmeantwhenshetoldyoushe boughtanewbike.Youknowthatabikehastwowheelsattachedtoaframe,asaddle, pedals,handlebars,brakes,andsoon.Inotherwords,evenifyouhaven’tseenthebike itself,youknowtheconceptofbike.Anabstractsetoffeaturesandcharacteristicsthat togetherformsomethingcalledbike. Incomputerprogramming,thatiscalledaclass.It’sthatsimple.Classesareusedtocreate objects.Infact,objectsaresaidtobeinstancesofclasses. Inotherwords,weallknowwhatabikeis,weknowtheclass.ButthenIhavemyown bike,whichisaninstanceoftheclassbike.Andmybikeisanobjectwithitsown characteristicsandmethods.Youhaveyourownbike.Sameclass,butdifferentinstance. Everybikeevercreatedintheworldisaninstanceofthebikeclass. Let’sseeanexample.Wewillwriteaclassthatdefinesabikeandthenwe’llcreatetwo bikes,oneredandoneblue.I’llkeepthecodeverysimple,butdon’tfretifyoudon’t understandeverythingaboutit;allyouneedtocareaboutatthismomentistounderstand thedifferencebetweenclassandobject(orinstanceofaclass): bike.py WOW! eBook www.wowebook.org #let'sdefinetheclassBike classBike: def__init__(self,colour,frame_material): self.colour=colour self.frame_material=frame_material defbrake(self): print("Braking!") #let'screateacoupleofinstances red_bike=Bike('Red','Carbonfiber') blue_bike=Bike('Blue','Steel') #let'sinspecttheobjectswehave,instancesoftheBikeclass. print(red_bike.colour)#prints:Red print(red_bike.frame_material)#prints:Carbonfiber print(blue_bike.colour)#prints:Blue print(blue_bike.frame_material)#prints:Steel #let'sbrake! red_bike.brake()#prints:Braking! Tip IhopebynowIdon’tneedtotellyoutorunthefileeverytime,right?Thefilenameis indicatedinthefirstlineofthecodeblock.Justrun$pythonfilename,andyou’llbe fine. Somanyinterestingthingstonoticehere.Firstthingsfirst;thedefinitionofaclass happenswiththeclassstatement(highlightedinthecode).Whatevercodecomesafter theclassstatement,andisindented,iscalledthebodyoftheclass.Inourcase,thelast linethatbelongstotheclassdefinitionistheprint("Braking!")one. Afterhavingdefinedtheclasswe’rereadytocreateinstances.Youcanseethattheclass bodyhoststhedefinitionoftwomethods.Amethodisbasically(andsimplistically)a functionthatbelongstoaclass. Thefirstmethod,__init__isaninitializer.ItusessomePythonmagictosetupthe objectswiththevalueswepasswhenwecreateit. Note Everymethodthathasleadingandtrailingdoubleunderscore,inPython,iscalledmagic method.MagicmethodsareusedbyPythonforamultitudeofdifferentpurposes,hence it’sneveragoodideatonameacustommethodusingtwoleadingandtrailing underscores.ThisnamingconventionisbestlefttoPython. Theothermethodwedefined,brake,isjustanexampleofanadditionalmethodthatwe couldcallifwewantedtobrakethebike.Itcontainsjustaprintstatement,ofcourse,it’s anexample. Wecreatedtwobikesthen.Onehasredcolorandacarbonfiberframe,andtheotherone hasbluecolorandsteelframe.Wepassthosevaluesuponcreation.Aftercreation,we printoutthecolorpropertyandframetypeoftheredbike,andtheframetypeoftheblue WOW! eBook www.wowebook.org onejustasanexample.Wealsocallthebrakemethodofthered_bike. Onelastthingtonotice.YourememberItoldyouthatthesetofattributesofanobjectis consideredtobeanamespace?Ihopeit’sclearernow,whatImeant.Youseethatby gettingtotheframe_typepropertythroughdifferentnamespaces(red_bike,blue_bike) weobtaindifferentvalues.Nooverlapping,noconfusion. Thedot(.)operatorisofcoursethemeansweusetowalkintoanamespace,inthecaseof objectsaswell. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Guidelinesonhowtowritegoodcode Writinggoodcodeisnotaseasyasitseems.AsIalreadysaidbefore,goodcodeexposes alonglistofqualitiesthatisquitehardtoputtogether.Writinggoodcodeis,tosome extent,anart.Regardlessofwhereonthepathyouwillbehappytosettle,thereis somethingthatyoucanembracewhichwillmakeyourcodeinstantlybetter:PEP8. AccordingtoWikipedia: “Python’sdevelopmentisconductedlargelythroughthePythonEnhancement Proposal(PEP)process.ThePEPprocessistheprimarymechanismforproposing majornewfeatures,forcollectingcommunityinputonanissue,andfordocumenting thedesigndecisionsthathavegoneintoPython.” AmongallthePEPs,probablythemostfamousoneisPEP8.Itlaysoutasimplebut effectivesetofguidelinestodefinePythonaestheticsothatwewritebeautifulPython code.Ifyoutakeonesuggestionoutofthischapter,pleaseletitbethis:useit.Embraceit. Youwillthankmelater. Codingtodayisnolongeracheck-in/check-outbusiness.Rather,it’smoreofasocial effort.Severaldeveloperscollaboratetoapieceofcodethroughtoolslikegitand mercurial,andtheresultiscodethatisfatheredbymanydifferenthands. Note GitandMercurialareprobablythemostuseddistributedrevisioncontrolsystemstoday. Theyareessentialtoolsdesignedtohelpteamsofdeveloperscollaborateonthesame software. Thesedays,morethanever,weneedtohaveaconsistentwayofwritingcode,sothat readabilityismaximized.WhenalldevelopersofacompanyabidewithPEP8,it’snot uncommonforanyofthemlandingonapieceofcodetothinktheywroteitthemselves.It actuallyhappenstomeallthetime(IalwaysforgetthecodeIwrite). Thishasatremendousadvantage:whenyoureadcodethatyoucouldhavewritten yourself,youreaditeasily.Withoutaconvention,everycoderwouldstructurethecode thewaytheylikemost,orsimplythewaytheyweretaughtorareusedto,andthiswould meanhavingtointerpreteverylineaccordingtosomeoneelse’sstyle.Itwouldmean havingtolosemuchmoretimejusttryingtounderstandit.ThankstoPEP8,wecanavoid this.I’msuchafanofitthatIwon’tsignoffacodereviewifthecodedoesn’trespectit. Sopleasetakethetimetostudyit,it’sveryimportant. Intheexamplesofthisbook,IwilltrytorespectitasmuchasIcan.Unfortunately,Idon’t havetheluxuryof79characters(whichisthemaximumlinelengthsuggestedbyPEP*), andIwillhavetocutdownonblanklinesandotherthings,butIpromiseyouI’lltryto layoutmycodesothatit’sasreadableaspossible. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org ThePythonculture Pythonhasbeenadoptedwidelyinallcodingindustries.It’susedbymanydifferent companiesformanydifferentpurposes,andit’salsousedineducation(it’sanexcellent languageforthatpurpose,becauseofitsmanyqualitiesandthefactthatit’seasytolearn). OneofthereasonsPythonissopopulartodayisthatthecommunityarounditisvast, vibrant,andfullofbrilliantpeople.Manyeventsareorganizedallovertheworld,mostly eitheraroundPythonoritsmainwebframework,Django. Pythonisopen,andveryoftensoarethemindsofthosewhoembraceit.Checkoutthe communitypageonthePythonwebsiteformoreinformationandgetinvolved! ThereisanotheraspecttoPythonwhichrevolvesaroundthenotionofbeingPythonic.It hastodowiththefactthatPythonallowsyoutousesomeidiomsthataren’tfound elsewhere,atleastnotinthesameformoreasinessofuse(Ifeelquiteclaustrophobic whenIhavetocodeinalanguagewhichisnotPythonnow). Anyway,overtheyears,thisconceptofbeingPythonichasemergedand,thewayI understandit,issomethingalongthelinesofdoingthingsthewaytheyaresupposedtobe doneinPython. TohelpyouunderstandalittlebitmoreaboutPython’scultureandaboutbeingPythonic,I willshowyoutheZenofPython.AlovelyEastereggthatisverypopular.Openupa Pythonconsoleandtypeimportthis.Whatfollowsistheresultofthisline: >>>importthis TheZenofPython,byTimPeters Beautifulisbetterthanugly. Explicitisbetterthanimplicit. Simpleisbetterthancomplex. Complexisbetterthancomplicated. Flatisbetterthannested. Sparseisbetterthandense. Readabilitycounts. Specialcasesaren'tspecialenoughtobreaktherules. Althoughpracticalitybeatspurity. Errorsshouldneverpasssilently. Unlessexplicitlysilenced. Inthefaceofambiguity,refusethetemptationtoguess. Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit. Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch. Nowisbetterthannever. Althoughneverisoftenbetterthan*right*now. Iftheimplementationishardtoexplain,it'sabadidea. Iftheimplementationiseasytoexplain,itmaybeagoodidea. Namespacesareonehonkinggreatidea—let'sdomoreofthose! Therearetwolevelsofreadinghere.Oneistoconsideritasasetofguidelinesthathave beenputdowninafunway.Theotheroneistokeepitinmind,andmaybereaditoncein awhile,tryingtounderstandhowitreferstosomethingdeeper.SomePython WOW! eBook www.wowebook.org characteristicsthatyouwillhavetounderstanddeeplyinordertowritePythontheway it’ssupposedtobewritten.Startwiththefunlevel,andthendigdeeper.Alwaysdig deeper. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org AnoteontheIDEs JustafewwordsaboutIntegratedDevelopmentEnvironments(IDEs).Tofollowthe examplesinthisbookyoudon’tneedone,anytexteditorwilldofine.Ifyouwanttohave moreadvancedfeaturessuchassyntaxcoloringandautocompletion,youwillhaveto fetchyourselfanIDE.YoucanfindacomprehensivelistofopensourceIDEs(just Google“pythonides”)onthePythonwebsite.IpersonallyuseSublimeTexteditor.It’s freetotryoutanditcostsjustafewdollars.IhavetriedmanyIDEsinmylife,butthisis theonethatmakesmemostproductive. Twoextremelyimportantpiecesofadvice: WhateverIDEyouwillchosetouse,trytolearnitwellsothatyoucanexploitits strengths,butdon’tdependonit.ExerciseyourselftoworkwithVIM(oranyother texteditor)onceinawhile,learntobeabletodosomeworkonanyplatform,with anysetoftools. Whatevertexteditor/IDEyouwilluse,whenitcomestowritingPython,indentation isfourspaces.Don’tusetabs,don’tmixthemwithspaces.Usefourspaces,nottwo, notthree,notfive.Justusefour.Thewholeworldworkslikethat,andyoudon’t wanttobecomeanoutcastbecauseyouwerefondofthethree-spacelayout. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,westartedtoexploretheworldofprogrammingandthatofPython.We’ve barelyscratchedthesurface,justalittle,touchingconceptsthatwillbediscussedlateron inthebookingreaterdetail. WetalkedaboutPython’smainfeatures,whoisusingitandforwhat,andwhatarethe differentwaysinwhichwecanwriteaPythonprogram. Inthelastpartofthechapter,weflewoverthefundamentalnotionsofnamespace,scope, class,andobject.WealsosawhowPythoncodecanbeorganizedusingmodulesand packages. Onapracticallevel,welearnedhowtoinstallPythononoursystem,howtomakesurewe havethetoolsweneed,pipandvirtualenv,andwealsocreatedandactivatedourfirst virtualenvironment.Thiswillallowustoworkinaself-containedenvironmentwithout theriskofcompromisingthePythonsysteminstallation. Nowyou’rereadytostartthisjourneywithme.Allyouneedisenthusiasm,anactivated virtualenvironment,thisbook,yourfingers,andsomecoffee. Trytofollowtheexamples,I’llkeepthemsimpleandshort.Ifyouputthemunderyour fingertips,youwillretainthemmuchbetterthanifyoujustreadthem. Inthenextchapter,wewillexplorePython’srichsetofbuilt-indatatypes.There’smuch tocoverandmuchtolearn! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter2.Built-inDataTypes “Data!Data!Data!”hecriedimpatiently.“Ican’tmakebrickswithoutclay.” —SherlockHolmes-TheAdventureoftheCopperBeeches Everythingyoudowithacomputerismanagingdata.Datacomesinmanydifferent shapesandflavors.It’sthemusicyoulisten,themovieyoustream,thePDFsyouopen. Eventhechapteryou’rereadingatthisverymomentisjustafile,whichisdata. Datacanbesimple,anintegernumbertorepresentanage,orcomplex,likeanorder placedonawebsite.Itcanbeaboutasingleobjectoraboutacollectionofthem. Datacanevenbeaboutdata,thatis,metadata.Datathatdescribesthedesignofotherdata structuresordatathatdescribesapplicationdataoritscontext. InPython,objectsareabstractionfordata,andPythonhasanamazingvarietyofdata structuresthatyoucanusetorepresentdata,orcombinethemtocreateyourowncustom data.Beforewedelveintothespecifics,Iwantyoutobeveryclearaboutobjectsin Python,solet’stalkalittlebitmoreaboutthem. WOW! eBook www.wowebook.org Everythingisanobject Aswealreadysaid,everythinginPythonisanobject.Butwhatreallyhappenswhenyou typeaninstructionlikeage=42inaPythonmodule? Tip Ifyougotohttp://pythontutor.com/,youcantypethatinstructionintoatextboxandget itsvisualrepresentation.Keepthiswebsiteinmind,it’sveryusefultoconsolidateyour understandingofwhatgoesonbehindthescenes. So,whathappensisthatanobjectiscreated.Itgetsanid,thetypeissettoint(integer number),andthevalueto42.Anameageisplacedintheglobalnamespace,pointingto thatobject.Therefore,wheneverweareintheglobalnamespace,aftertheexecutionof thatline,wecanretrievethatobjectbysimplyaccessingitthroughitsname:age. Ifyouweretomovehouse,youwouldputalltheknives,forks,andspoonsinaboxand labelitcutlery.Canyouseeit’sexactlythesameconcept?Here’sascreenshotofhowit maylooklike(youmayhavetotweakthesettingstogettothesameview): So,fortherestofthischapter,wheneveryoureadsomethingsuchasname=some_value, thinkofanameplacedinthenamespacethatistiedtothescopeinwhichtheinstruction waswritten,withanicearrowpointingtoanobjectthathasanid,atype,andavalue. Thereisalittlebitmoretosayaboutthismechanism,butit’smucheasiertotalkaboutit overanexample,sowe’llgetbacktothislater. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Mutableorimmutable?Thatisthe question AfirstfundamentaldistinctionthatPythonmakesondataisaboutwhetherornotthe valueofanobjectchanges.Ifthevaluecanchange,theobjectiscalledmutable,whileif thevaluecannotchange,theobjectiscalledimmutable. Itisveryimportantthatyouunderstandthedistinctionbetweenmutableandimmutable becauseitaffectsthecodeyouwrite,sohere’saquestion: >>>age=42 >>>age 42 >>>age=43#A >>>age 43 Intheprecedingcode,ontheline#A,haveIchangedthevalueofage?Well,no.Butnow it’s43(Ihearyousay…).Yes,it’s43,but42wasanintegernumber,ofthetypeint, whichisimmutable.So,whathappenedisreallythatonthefirstline,ageisanamethatis settopointtoanintobject,whosevalueis42.Whenwetypeage=43,whathappensis thatanotherobjectiscreated,ofthetypeintandvalue43(also,theidwillbedifferent), andthenameageissettopointtoit.So,wedidn’tchangethat42to43.Weactuallyjust pointedagetoadifferentlocation:thenewintobjectwhosevalueis43.Let’sseethe samecodealsoprintingtheIDs: >>>age=42 >>>id(age) 10456352 >>>age=43 >>>id(age) 10456384 NoticethatweprinttheIDsbycallingthebuilt-inidfunction.Asyoucansee,theyare different,asexpected.Bearinmindthatagepointstooneobjectatatime:42first,then 43.Nevertogether. Now,let’sseethesameexampleusingamutableobject.Forthisexample,let’sjustusea Personobject,thathasapropertyage: >>>fab=Person(age=39) >>>fab.age 39 >>>id(fab) 139632387887456 >>>fab.age=29#Iwish! >>>id(fab) 139632387887456#stillthesameid Inthiscase,IsetupanobjectfabwhosetypeisPerson(acustomclass).Oncreation,the objectisgiventheageof39.I’mprintingit,alongwiththeobjectid,rightafterwards. WOW! eBook www.wowebook.org Noticethat,evenafterIchangeagetobe29,theIDoffabstaysthesame(whiletheIDof agehaschanged,ofcourse).CustomobjectsinPythonaremutable(unlessyoucodethem nottobe).Keepthisconceptinmind,it’sveryimportant.I’llremindyouaboutitthrough therestofthechapter. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Numbers Let’sstartbyexploringPython’sbuilt-indatatypesfornumbers.Pythonwasdesignedby amanwithamaster’sdegreeinmathematicsandcomputerscience,soit’sonlylogical thatithasamazingsupportfornumbers. Numbersareimmutableobjects. WOW! eBook www.wowebook.org Integers Pythonintegershaveunlimitedrange,subjectonlytotheavailablevirtualmemory.This meansthatitdoesn’treallymatterhowbiganumberyouwanttostore:aslongasitcanfit inyourcomputer’smemory,Pythonwilltakecareofit.Integernumberscanbepositive, negative,and0(zero).Theysupportallthebasicmathematicaloperations,asshowninthe followingexample: >>>a=12 >>>b=3 >>>a+b#addition 15 >>>b-a#subtraction -9 >>>a//b#integerdivision 4 >>>a/b#truedivision 4.0 >>>a*b#multiplication 36 >>>b**a#poweroperator 531441 >>>2**1024#averybignumber,Pythonhandlesitgracefully 17976931348623159077293051907890247336179769789423065727343008115 77326758055009631327084773224075360211201138798713933576587897688 14416622492847430639474124377767893424865485276302219601246094119 45308295208500576883815068234246288147391311054082723716335051068 4586298239947245938479716304835356329624224137216 Theprecedingcodeshouldbeeasytounderstand.Justnoticeoneimportantthing:Python hastwodivisionoperators,oneperformstheso-calledtruedivision(/),whichreturnsthe quotientoftheoperands,andtheotherone,theso-calledintegerdivision(//),which returnstheflooredquotientoftheoperands.Seehowthatisdifferentforpositiveand negativenumbers: >>>7/4#truedivision 1.75 >>>7//4#integerdivision,flooringreturns1 1 >>>-7/4#truedivisionagain,resultisoppositeofprevious -1.75 >>>-7//4#integerdiv.,resultnottheoppositeofprevious -2 Thisisaninterestingexample.Ifyouwereexpectinga-1onthelastline,don’tfeelbad, it’sjustthewayPythonworks.TheresultofanintegerdivisioninPythonisalways roundedtowardsminusinfinity.Ifinsteadofflooringyouwanttotruncateanumbertoan integer,youcanusethebuilt-inintfunction,likeshowninthefollowingexample: >>>int(1.75) 1 >>>int(-1.75) -1 WOW! eBook www.wowebook.org Noticethattruncationisdonetowards0. Thereisalsoanoperatortocalculatetheremainderofadivision.It’scalledmodulo operator,andit’srepresentedbyapercent(%): >>>10%3#remainderofthedivision10//3 1 >>>10%4#remainderofthedivision10//4 2 WOW! eBook www.wowebook.org Booleans Booleanalgebraisthatsubsetofalgebrainwhichthevaluesofthevariablesarethetruth values:trueandfalse.InPython,TrueandFalsearetwokeywordsthatareusedto representtruthvalues.Booleansareasubclassofintegers,andbehaverespectivelylike1 and0.TheequivalentoftheintclassforBooleansistheboolclass,whichreturnseither TrueorFalse.Everybuilt-inPythonobjecthasavalueintheBooleancontext,which meanstheybasicallyevaluatetoeitherTrueorFalsewhenfedtotheboolfunction.We’ll seeallaboutthisinChapter3,IteratingandMakingDecisions. BooleanvaluescanbecombinedinBooleanexpressionsusingthelogicaloperatorsand, or,andnot.Again,we’llseetheminfullinthenextchapter,sofornowlet’sjustseea simpleexample: >>>int(True)#Truebehaveslike1 1 >>>int(False)#Falsebehaveslike0 0 >>>bool(1)#1evaluatestoTrueinabooleancontext True >>>bool(-42)#andsodoeseverynon-zeronumber True >>>bool(0)#0evaluatestoFalse False >>>#quickpeakattheoperators(and,or,not) >>>notTrue False >>>notFalse True >>>TrueandTrue True >>>FalseorTrue True YoucanseethatTrueandFalsearesubclassesofintegerswhenyoutrytoaddthem. Pythonupcaststhemtointegersandperformsaddition: >>>1+True 2 >>>False+42 42 >>>7-True 6 Note Upcastingisatypeconversionoperationthatgoesfromasubclasstoitsparent.Inthe examplepresentedhere,TrueandFalse,whichbelongtoaclassderivedfromtheinteger class,areconvertedbacktointegerswhenneeded.Thistopicisaboutinheritanceandwill beexplainedindetailinChapter6,AdvancedConcepts–OOP,Decorators,andIterators. WOW! eBook www.wowebook.org Reals Realnumbers,orfloatingpointnumbers,arerepresentedinPythonaccordingtotheIEEE 754double-precisionbinaryfloating-pointformat,whichisstoredin64bitsof informationdividedintothreesections:sign,exponent,andmantissa. Note QuenchyourthirstforknowledgeaboutthisformatonWikipedia: http://en.wikipedia.org/wiki/Double-precision_floating-point_format Usuallyprogramminglanguagesgivecoderstwodifferentformats:singleanddouble precision.Theformertakingup32bitsofmemory,andthelatter64.Pythonsupportsonly thedoubleformat.Let’sseeasimpleexample: >>>pi=3.1415926536#howmanydigitsofPIcanyouremember? >>>radius=4.5 >>>area=pi*(radius**2) >>>area 63.61725123519331 Note Inthecalculationofthearea,Iwrappedtheradius**2withinbraces.Eventhoughthat wasn’tnecessarybecausethepoweroperatorhashigherprecedencethanthe multiplicationone,Ithinktheformulareadsmoreeasilylikethat. Thesys.float_infostructsequenceholdsinformationabouthowfloatingpointnumbers willbehaveonyoursystem.ThisiswhatIseeonmybox: >>>importsys >>>sys.float_info sys.float_info(max=1.7976931348623157e+308,max_exp=1024,max_10_exp=308, min=2.2250738585072014e-308,min_exp=-1021,min_10_exp=-307,dig=15, mant_dig=53,epsilon=2.220446049250313e-16,radix=2,rounds=1) Let’smakeafewconsiderationshere:wehave64bitstorepresentfloatnumbers.This meanswecanrepresentatmost2**64==18,446,744,073,709,551,616numbers withthatamountofbits.Takealookatthemaxandepsilonvalueforthefloatnumbers, andyou’llrealizeit’simpossibletorepresentthemall.Thereisjustnotenoughspaceso theyareapproximatedtotheclosestrepresentablenumber.Youprobablythinkthatonly extremelybigorextremelysmallnumberssufferfromthisissue.Well,thinkagain: >>>3*0.1–0.3#thisshouldbe0!!! 5.551115123125783e-17 Whatdoesthistellyou?Ittellsyouthatdoubleprecisionnumberssufferfrom approximationissuesevenwhenitcomestosimplenumberslike0.1or0.3.Whyisthis important?Itcanbeabigproblemifyou’rehandlingprices,orfinancialcalculations,or anykindofdatathatneedsnottobeapproximated.Don’tworry,Pythongivesyouthe Decimaltype,whichdoesn’tsufferfromtheseissues,we’llseetheminabit. WOW! eBook www.wowebook.org Complexnumbers Pythongivesyoucomplexnumberssupportoutofthebox.Ifyoudon’tknowwhat complexnumbersare,youcanlookthemupontheWeb.Theyarenumbersthatcanbe expressedintheforma+ibwhereaandbarerealnumbers,andi(orjifyou’rean engineer)istheimaginaryunit,thatis,thesquarerootof-1.aandbarecalled respectivelytherealandimaginarypartofthenumber. It’sactuallyunlikelyyou’llbeusingthem,unlessyou’recodingsomethingscientific. Let’sseeasmallexample: >>>c=3.14+2.73j >>>c.real#realpart 3.14 >>>c.imag#imaginarypart 2.73 >>>c.conjugate()#conjugateofA+BjisA-Bj (3.14-2.73j) >>>c*2#multiplicationisallowed (6.28+5.46j) >>>c**2#poweroperationaswell (2.4067000000000007+17.1444j) >>>d=1+1j#additionandsubtractionaswell >>>c-d (2.14+1.73j) WOW! eBook www.wowebook.org Fractionsanddecimals Let’sfinishthetourofthenumberdepartmentwithalookatfractionsanddecimals. Fractionsholdarationalnumeratoranddenominatorintheirlowestforms.Let’sseea quickexample: >>>fromfractionsimportFraction >>>Fraction(10,6)#madhatter? Fraction(5,3)#noticeit'sbeenreducedtolowestterms >>>Fraction(1,3)+Fraction(2,3)#1/3+2/3=3/3=1/1 Fraction(1,1) >>>f=Fraction(10,6) >>>f.numerator 5 >>>f.denominator 3 Althoughtheycanbeveryusefulattimes,it’snotthatcommontospotthemin commercialsoftware.Mucheasierinstead,istoseedecimalnumbersbeingusedinall thosecontextswhereprecisioniseverything,forexample,scientificandfinancial calculations. Note It’simportanttorememberthatarbitraryprecisiondecimalnumberscomeatapricein performance,ofcourse.Theamountofdatatobestoredforeachnumberisfargreater thanitisforfractionsorfloatsaswellasthewaytheyarehandled,whichrequiresthe Pythoninterpretermuchmoreworkbehindthescenes.Anotherinterestingthingtoknow isthatyoucangetandsettheprecisionbyaccessingdecimal.getcontext().prec. Let’sseeaquickexamplewithDecimalnumbers: >>>fromdecimalimportDecimalasD#renameforbrevity >>>D(3.14)#pi,fromfloat,soapproximationissues Decimal('3.140000000000000124344978758017532527446746826171875') >>>D('3.14')#pi,fromastring,sonoapproximationissues Decimal('3.14') >>>D(0.1)*D(3)-D(0.3)#fromfloat,westillhavetheissue Decimal('2.775557561565156540423631668E-17') >>>D('0.1')*D(3)-D('0.3')#fromstring,allperfect Decimal('0.0') NoticethatwhenweconstructaDecimalnumberfromafloat,ittakesonallthe approximationissuesthefloatmaycomefrom.Ontheotherhand,whentheDecimalhas noapproximationissues,forexample,whenwefeedanintorastringrepresentationto theconstructor,thenthecalculationhasnoquirkybehavior.Whenitcomestomoney,use decimals. Thisconcludesourintroductiontobuilt-innumerictypes,let’snowseesequences. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Immutablesequences Let’sstartwithimmutablesequences:strings,tuples,andbytes. WOW! eBook www.wowebook.org Stringsandbytes TextualdatainPythonishandledwithstrobjects,morecommonlyknownasstrings. Theyareimmutablesequencesofunicodecodepoints.Unicodecodepointscanrepresent acharacter,butcanalsohaveothermeanings,suchasformattingdataforexample. Python,unlikeotherlanguages,doesn’thaveachartype,soasinglecharacterisrendered simplybyastringoflength1.Unicodeisanexcellentwaytohandledata,andshouldbe usedfortheinternalsofanyapplication.Whenitcomestostoretextualdatathough,or senditonthenetwork,youmaywanttoencodeit,usinganappropriateencodingforthe mediumyou’reusing.StringliteralsarewritteninPythonusingsingle,doubleortriple quotes(bothsingleordouble).Ifbuiltwithtriplequotes,astringcanspanonmultiple lines.Anexamplewillclarifythepicture: >>>#4waystomakeastring >>>str1='Thisisastring.Webuiltitwithsinglequotes.' >>>str2="Thisisalsoastring,butbuiltwithdoublequotes." >>>str3='''Thisisbuiltusingtriplequotes, ...soitcanspanmultiplelines.''' >>>str4="""Thistoo ...isamultilineone ...builtwithtripledouble-quotes.""" >>>str4#A 'Thistoo\nisamultilineone\nbuiltwithtripledouble-quotes.' >>>print(str4)#B Thistoo isamultilineone builtwithtripledouble-quotes. In#Aand#B,weprintstr4,firstimplicitly,thenexplicitlyusingtheprintfunction.A niceexercisewouldbetofindoutwhytheyaredifferent.Areyouuptothechallenge? (hint,lookupthestrfunction) Strings,likeanysequence,havealength.Youcangetthisbycallingthelenfunction: >>>len(str1) 49 Encodinganddecodingstrings Usingtheencode/decodemethods,wecanencodeunicodestringsanddecodebytes objects.Utf-8isavariablelengthcharacterencoding,capableofencodingallpossible unicodecodepoints.ItisthedominantencodingfortheWeb(andnotonly).Noticealso thatbyaddingaliteralbinfrontofastringdeclaration,we’recreatingabytesobject. >>>s="Thisisüŋíc0de"#unicodestring:codepoints >>>type(s) <class'str'> >>>encoded_s=s.encode('utf-8')#utf-8encodedversionofs >>>encoded_s b'Thisis\xc3\xbc\xc5\x8b\xc3\xadc0de'#result:bytesobject >>>type(encoded_s)#anotherwaytoverifyit <class'bytes'> >>>encoded_s.decode('utf-8')#let'sreverttotheoriginal WOW! eBook www.wowebook.org 'Thisisüŋíc0de' >>>bytes_obj=b"Abytesobject"#abytesobject >>>type(bytes_obj) <class'bytes'> Indexingandslicingstrings Whenmanipulatingsequences,it’sverycommontohavetoaccessthematoneprecise position(indexing),ortogetasubsequenceoutofthem(slicing).Whendealingwith immutablesequences,bothoperationsareread-only. Whileindexingcomesinoneform,azero-basedaccesstoanypositionwithinthe sequence,slicingcomesindifferentforms.Whenyougetasliceofasequence,youcan specifythestartandstoppositions,andthestep.Theyareseparatedwithacolon(:) likethis:my_sequence[start:stop:step].Alltheargumentsareoptional,startis inclusive,stopisexclusive.It’smucheasiertoshowanexample,ratherthanexplainthem furtherinwords: >>>s="Thetroubleisyouthinkyouhavetime." >>>s[0]#indexingatposition0,whichisthefirstchar 'T' >>>s[5]#indexingatposition5,whichisthesixthchar 'r' >>>s[:4]#slicing,wespecifyonlythestopposition 'The' >>>s[4:]#slicing,wespecifyonlythestartposition 'troubleisyouthinkyouhavetime.' >>>s[2:14]#slicing,bothstartandstoppositions 'etroubleis' >>>s[2:14:3]#slicing,start,stopandstep(every3chars) 'erb' >>>s[:]#quickwayofmakingacopy 'Thetroubleisyouthinkyouhavetime.' Ofallthelines,thelastoneisprobablythemostinteresting.Ifyoudon’tspecifya parameter,Pythonwillfillinthedefaultforyou.Inthiscase,startwillbethestartofthe string,stopwillbetheendofthesting,andstepwillbethedefault1.Thisisaneasyand quickwayofobtainingacopyofthestrings(samevalue,butdifferentobject).Canyou findawaytogetthereversedcopyofastringusingslicing?(don’tlookitup,finditfor yourself) WOW! eBook www.wowebook.org Tuples Thelastimmutablesequencetypewe’regoingtoseeisthetuple.Atupleisasequenceof arbitraryPythonobjects.Inatuple,itemsareseparatedbycommas.Theyareused everywhereinPython,becausetheyallowforpatternsthatarehardtoreproduceinother languages.Sometimestuplesareusedimplicitly,forexampletosetupmultiplevariables ononeline,ortoallowafunctiontoreturnmultipledifferentobjects(usuallyafunction returnsoneobjectonly,inmanyotherlanguages),andeveninthePythonconsole,you canusetuplesimplicitlytoprintmultipleelementswithonesingleinstruction.We’llsee examplesforallthesecases: >>>t=()#emptytuple >>>type(t) <class'tuple'> >>>one_element_tuple=(42,)#youneedthecomma! >>>three_elements_tuple=(1,3,5) >>>a,b,c=1,2,3#tupleformultipleassignment >>>a,b,c#implicittupletoprintwithoneinstruction (1,2,3) >>>3inthree_elements_tuple#membershiptest True Noticethatthemembershipoperatorincanalsobeusedwithlists,strings,dictionaries, andingeneralwithcollectionandsequenceobjects. Note Noticethattocreateatuplewithoneitem,weneedtoputthatcommaaftertheitem.The reasonisthatwithoutthecommathatitemisjustitselfwrappedinbraces,kindofina redundantmathematicalexpression.Noticealsothatonassignment,bracesareoptionalso my_tuple=1,2,3isthesameasmy_tuple=(1,2,3). Onethingthattupleassignmentallowsustodo,isone-lineswaps,withnoneedforathird temporaryvariable.Let’sseefirstamoretraditionalwayofdoingit: >>>a,b=1,2 >>>c=a#weneedthreelinesandatemporaryvarc >>>a=b >>>b=c >>>a,b#aandbhavebeenswapped (2,1) Andnowlet’sseehowwewoulddoitinPython: >>>a,b=b,a#thisisthePythonicwaytodoit >>>a,b (1,2) TakealookatthelinethatshowsyouthePythonicwayofswappingtwovalues:doyou rememberwhatIwroteinChapter1,IntroductionandFirstSteps–TakeaDeepBreath. APythonprogramistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++ code,andfeatureslikeone-lineswapscontributetothis.Pythoniselegant,whereelegance WOW! eBook www.wowebook.org inthiscontextmeansalsoeconomy. Becausetheyareimmutable,tuplescanbeusedaskeysfordictionaries(we’llseethis shortly).Thedictobjectsneedkeystobeimmutablebecauseiftheycouldchange,then thevaluetheyreferencewouldn’tbefoundanymore(becausethepathtoitdependson thekey).Ifyouareintodatastructures,youknowhowniceafeaturethisoneistohave. Tome,tuplesarePython’sbuilt-indatathatmostcloselyrepresentamathematicalvector. Thisdoesn’tmeanthatthiswasthereasonforwhichtheywerecreatedthough.Tuples usuallycontainanheterogeneoussequenceofelements,whileontheotherhandlistsare mostofthetimeshomogeneous.Moreover,tuplesarenormallyaccessedviaunpackingor indexing,whilelistsareusuallyiteratedover. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Mutablesequences Mutablesequencesdifferfromtheirimmutablesistersinthattheycanbechangedafter creation.TherearetwomutablesequencetypesinPython:listsandbytearrays.Isaid beforethatthedictionaryisthekingofdatastructuresinPython.Iguessthismakesthe listitsrightfulqueen. WOW! eBook www.wowebook.org Lists Pythonlistsaremutablesequences.Theyareverysimilartotuples,buttheydon’thave therestrictionsduetoimmutability.Listsarecommonlyusedtostorecollectionsof homogeneousobjects,butthereisnothingpreventingyoutostoreheterogeneous collectionsaswell.Listscanbecreatedinmanydifferentways,let’sseeanexample: >>>[]#emptylist [] >>>list()#sameas[] [] >>>[1,2,3]#aswithtuples,itemsarecommaseparated [1,2,3] >>>[x+5forxin[2,3,4]]#Pythonismagic [7,8,9] >>>list((1,3,5,7,9))#listfromatuple [1,3,5,7,9] >>>list('hello')#listfromastring ['h','e','l','l','o'] Inthepreviousexample,Ishowedyouhowtocreatealistusingdifferenttechniques.I wouldlikeyoutotakeagoodlookatthelinethatsaysPythonismagic,whichIamnot expectingyoutofullyunderstandatthispoint(unlessyoucheatedandyou’renota novice!).Thatiscalledalistcomprehension,averypowerfulfunctionalfeatureof Python,whichwe’llseeindetailinChapter5,SavingTimeandMemory.Ijustwantedto makeyourmouthwateratthispoint. Creatinglistsisgood,buttherealfuncomeswhenweusethem,solet’sseethemain methodstheygiftuswith: >>>a=[1,2,1,3] >>>a.append(13)#wecanappendanythingattheend >>>a [1,2,1,3,13] >>>a.count(1)#howmany`1`arethereinthelist? 2 >>>a.extend([5,7])#extendthelistbyanother(orsequence) >>>a [1,2,1,3,13,5,7] >>>a.index(13)#positionof`13`inthelist(0-basedindexing) 4 >>>a.insert(0,17)#insert`17`atposition0 >>>a [17,1,2,1,3,13,5,7] >>>a.pop()#pop(removeandreturn)lastelement 7 >>>a.pop(3)#popelementatposition3 1 >>>a [17,1,2,3,13,5] >>>a.remove(17)#remove`17`fromthelist >>>a [1,2,3,13,5] WOW! eBook www.wowebook.org >>>a.reverse()#reversetheorderoftheelementsinthelist >>>a [5,13,3,2,1] >>>a.sort()#sortthelist >>>a [1,2,3,5,13] >>>a.clear()#removeallelementsfromthelist >>>a [] Theprecedingcodegivesyouaroundupoflist’smainmethods.Iwanttoshowyouhow powerfultheyare,usingextendasanexample.Youcanextendlistsusinganysequence type: >>>a=list('hello')#makesalistfromastring >>>a ['h','e','l','l','o'] >>>a.append(100)#append100,heterogeneoustype >>>a ['h','e','l','l','o',100] >>>a.extend((1,2,3))#extendusingtuple >>>a ['h','e','l','l','o',100,1,2,3] >>>a.extend('...')#extendusingstring >>>a ['h','e','l','l','o',100,1,2,3,'.','.','.'] Now,let’sseewhatarethemostcommonoperationsyoucandowithlists: >>>a=[1,3,5,7] >>>min(a)#minimumvalueinthelist 1 >>>max(a)#maximumvalueinthelist 7 >>>sum(a)#sumofallvaluesinthelist 16 >>>len(a)#numberofelementsinthelist 4 >>>b=[6,7,8] >>>a+b#`+`withlistmeansconcatenation [1,3,5,7,6,7,8] >>>a*2#`*`hasalsoaspecialmeaning [1,3,5,7,1,3,5,7] Thelasttwolinesintheprecedingcodearequiteinterestingbecausetheyintroduceusto aconceptcalledoperatoroverloading.Inshort,itmeansthatoperatorssuchas+,-.*,%, andsoon,mayrepresentdifferentoperationsaccordingtothecontexttheyareusedin.It doesn’tmakeanysensetosumtwolists,right?Therefore,the+signisusedto concatenatethem.Hence,the*signisusedtoconcatenatethelisttoitselfaccordingtothe rightoperand.Now,let’stakeastepfurtherdowntherabbitholeandseesomethingalittle moreinteresting.Iwanttoshowyouhowpowerfulthesortmethodcanbeandhoweasyit isinPythontoachieveresultsthatrequireagreatdealofeffortinotherlanguages: >>>fromoperatorimportitemgetter >>>a=[(5,3),(1,3),(1,2),(2,-1),(4,9)] WOW! eBook www.wowebook.org >>>sorted(a) [(1,2),(1,3),(2,-1),(4,9),(5,3)] >>>sorted(a,key=itemgetter(0)) [(1,3),(1,2),(2,-1),(4,9),(5,3)] >>>sorted(a,key=itemgetter(0,1)) [(1,2),(1,3),(2,-1),(4,9),(5,3)] >>>sorted(a,key=itemgetter(1)) [(2,-1),(1,2),(5,3),(1,3),(4,9)] >>>sorted(a,key=itemgetter(1),reverse=True) [(4,9),(5,3),(1,3),(1,2),(2,-1)] Theprecedingcodedeservesalittleexplanation.Firstofall,aisalistoftuples.This meanseachelementinaisatuple(a2-tuple,tobepicky).Whenwecall sorted(some_list),wegetasortedversionofsome_list.Inthiscase,thesortingona2tupleworksbysortingthemonthefirstiteminthetuple,andonthesecondwhenthefirst oneisthesame.Youcanseethisbehaviorintheresultofsorted(a),whichyields[(1, 2),(1,3),...].Pythonalsogivesustheabilitytocontrolonwhichelement(s)ofthe tuplethesortingmustberunagainst.Noticethatwhenweinstructthesortedfunctionto workonthefirstelementofeachtuple(bykey=itemgetter(0)),theresultisdifferent: [(1,3),(1,2),...].Thesortingisdoneonlyonthefirstelementofeachtuple (whichistheoneatposition0).Ifwewanttoreplicatethedefaultbehaviorofasimple sorted(a)call,weneedtousekey=itemgetter(0,1),whichtellsPythontosortfirston theelementsatposition0withinthetuples,andthenonthoseatposition1.Comparethe resultsandyou’llseetheymatch. Forcompleteness,Iincludedanexampleofsortingonlyontheelementsatposition1,and thesamebutinreverseorder.IfyouhaveeverseensortinginJava,Iexpectyoutobeon yourkneescryingwithjoyatthisverymoment. ThePythonsortingalgorithmisverypowerful,anditwaswrittenbyTimPeters(we’ve alreadyseenthisname,canyourecallwhen?).ItisaptlynamedTimsort,anditisablend betweenmergeandinsertionsortandhasbettertimeperformancesthanmostother algorithmsusedformainstreamprogramminglanguages.Timsortisastablesorting algorithm,whichmeansthatwhenmultiplerecordshavethesamekey,theiroriginalorder ispreserved.We’veseenthisintheresultofsorted(a,key=itemgetter(0))whichhas yielded[(1,3),(1,2),...]inwhichtheorderofthosetwotupleshasbeen preservedbecausetheyhavethesamevalueatposition0. WOW! eBook www.wowebook.org Bytearrays Toconcludeouroverviewofmutablesequencetypes,let’sspendacoupleofminuteson thebytearraytype.Basically,theyrepresentthemutableversionofbytesobjects.They exposemostoftheusualmethodsofmutablesequencesaswellasmostofthemethodsof thebytestype.Itemsareintegersintherange[0,256). Note Whenitcomestointervals,I’mgoingtousethestandardnotationforopen/closedranges. Asquarebracketononeendmeansthatthevalueisincluded,whilearoundbracemeans it’sexcluded.Thegranularityisusuallyinferredbythetypeoftheedgeelementsso,for example,theinterval[3,7]meansallintegersbetween3and7,inclusive.Ontheother hand,(3,7)meansallintegersbetween3and7exclusive(hence4,5,and6).Itemsina bytearraytypeareintegersbetween0and256,0isincluded,256isnot.Onereason intervalsareoftenexpressedlikethisistoeasecoding.Ifwebreakarange[a,b)intoN consecutiveranges,wecaneasilyrepresenttheoriginaloneasaconcatenationlikethis: Themiddlepoints(ki)beingexcludedononeend,andincludedontheotherend,allow foreasyconcatenationandsplittingwhenintervalsarehandledinthecode. Let’sseeaquickexamplewiththetypebytearray: >>>bytearray()#emptybytearrayobject bytearray(b'') >>>bytearray(10)#zero-filledinstancewithgivenlength bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00') >>>bytearray(range(5))#bytearrayfromiterableofintegers bytearray(b'\x00\x01\x02\x03\x04') >>>name=bytearray(b'Lina')#A-bytearrayfrombytes >>>name.replace(b'L',b'l') bytearray(b'lina') >>>name.endswith(b'na') True >>>name.upper() bytearray(b'LINA') >>>name.count(b'L') 1 Asyoucanseeintheprecedingcode,thereareafewwaystocreateabytearrayobject. Theycanbeusefulinmanysituations,forexample,whenreceivingdatathroughasocket, theyeliminatetheneedtoconcatenatedatawhilepolling,hencetheyproveveryhandy. Ontheline#A,Icreatedthenamebytearrayfromthestringb'Lina'toshowyouhowthe bytearrayobjectexposesmethodsfrombothsequencesandstrings,whichisextremely handy.Ifyouthinkaboutit,theycanbeconsideredasmutablestrings. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Settypes Pythonalsoprovidestwosettypes,setandfrozenset.Thesettypeismutable,while frozensetisimmutable.Theyareunorderedcollectionsofimmutableobjects. Hashabilityisacharacteristicthatallowsanobjecttobeusedasasetmemberaswellas akeyforadictionary,aswe’llseeverysoon. Note Anobjectishashableifithasahashvaluewhichneverchangesduringitslifetime. Objectsthatcompareequallymusthavethesamehashvalue.Setsareverycommonly usedtotestformembership,solet’sintroducetheinoperatorinthefollowingexample: >>>small_primes=set()#emptyset >>>small_primes.add(2)#addingoneelementatatime >>>small_primes.add(3) >>>small_primes.add(5) >>>small_primes {2,3,5} >>>small_primes.add(1)#LookwhatI'vedone,1isnotaprime! >>>small_primes {1,2,3,5} >>>small_primes.remove(1)#solet'sremoveit >>>3insmall_primes#membershiptest True >>>4insmall_primes False >>>4notinsmall_primes#negatedmembershiptest True >>>small_primes.add(3)#tryingtoadd3again >>>small_primes {2,3,5}#nochange,duplicationisnotallowed >>>bigger_primes=set([5,7,11,13])#fastercreation >>>small_primes|bigger_primes#unionoperator`|` {2,3,5,7,11,13} >>>small_primes&bigger_primes#intersectionoperator`&` {5} >>>small_primes-bigger_primes#differenceoperator`-` {2,3} Intheprecedingcode,youcanseetwodifferentwaystocreateaset.Onecreatesan emptysetandthenaddselementsoneatatime.Theothercreatesthesetusingalistof numbersasargumenttotheconstructor,whichdoesalltheworkforus.Ofcourse,you cancreateasetfromalistortuple(oranyiterable)andthenyoucanaddandremove membersfromthesetasyouplease. Anotherwayofcreatingasetisbysimplyusingthecurlybracesnotation,likethis: >>>small_primes={2,3,5,5,3} >>>small_primes {2,3,5} WOW! eBook www.wowebook.org NoticeIaddedsomeduplicationtoemphasizethattheresultsetwon’thaveany. Note We’llseeiterableobjectsanditerationinthenextchapter.Fornow,justknowthatiterable objectsareobjectsyoucaniterateoninadirection. Let’sseeanexampleabouttheimmutablecounterpartofthesettype:frozenset. >>>small_primes=frozenset([2,3,5,7]) >>>bigger_primes=frozenset([5,7,11]) >>>small_primes.add(11)#wecannotaddtoafrozenset Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> AttributeError:'frozenset'objecthasnoattribute'add' >>>small_primes.remove(2)#neitherwecanremove Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> AttributeError:'frozenset'objecthasnoattribute'remove' >>>small_primes&bigger_primes#intersect,union,etc.allowed frozenset({5,7}) Asyoucansee,frozensetobjectsarequitelimitedinrespectoftheirmutable counterpart.Theystillproveveryeffectiveformembershiptest,union,intersectionand differenceoperations,andforperformancereasons. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Mappingtypes–dictionaries Ofallthebuilt-inPythondatatypes,thedictionaryisprobablythemostinterestingone. It’stheonlystandardmappingtype,anditisthebackboneofeveryPythonobject. Adictionarymapskeystovalues.Keysneedtobehashableobjects,whilevaluescanbe ofanyarbitrarytype.Dictionariesaremutableobjects. Therearequiteafewdifferentwaystocreateadictionary,soletmegiveyouasimple exampleofhowtocreateadictionaryequalto{'A':1,'Z':-1}infivedifferentways: >>>a=dict(A=1,Z=-1) >>>b={'A':1,'Z':-1} >>>c=dict(zip(['A','Z'],[1,-1])) >>>d=dict([('A',1),('Z',-1)]) >>>e=dict({'Z':-1,'A':1}) >>>a==b==c==d==e#aretheyallthesame? True#indeed! Haveyounoticedthosedoubleequals?Assignmentisdonewithoneequal,whiletocheck whetheranobjectisthesameasanotherone(or5inonego,inthiscase),weusedouble equals.Thereisalsoanotherwaytocompareobjects,whichinvolvestheisoperator,and checkswhetherthetwoobjectsarethesame(iftheyhavethesameID,notjustthevalue), butunlessyouhaveagoodreasontouseit,youshouldusethedoubleequalinstead.In theprecedingcode,Ialsousedonenicefunction:zip.Itisnamedafterthereal-lifezip, whichgluestogethertwothingstakingoneelementfromeachatatime.Letmeshowyou anexample: >>>list(zip(['h','e','l','l','o'],[1,2,3,4,5])) [('h',1),('e',2),('l',3),('l',4),('o',5)] >>>list(zip('hello',range(1,6)))#equivalent,morePythonic [('h',1),('e',2),('l',3),('l',4),('o',5)] Intheprecedingexample,Ihavecreatedthesamelistintwodifferentways,onemore explicit,andtheotheralittlebitmorePythonic.ForgetforamomentthatIhadtowrap thelistconstructoraroundthezipcall(thereasonisbecausezipreturnsaniterator,not alist),andconcentrateontheresult.Seehowziphascoupledthefirstelementsofits twoargumentstogether,thenthesecondones,thenthethirdones,andsoonandsoforth? Takealookatyourpants(oratyourpurseifyou’realady)andyou’llseethesame behaviorinyouractualzip.Butlet’sgobacktodictionariesandseehowmanywonderful methodstheyexposeforallowingustomanipulatethemaswewant.Let’sstartwiththe basicoperations: >>>d={} >>>d['a']=1#let'ssetacoupleof(key,value)pairs >>>d['b']=2 >>>len(d)#howmanypairs? 2 >>>d['a']#whatisthevalueof'a'? 1 >>>d#howdoes`d`looknow? WOW! eBook www.wowebook.org {'a':1,'b':2} >>>deld['a']#let'sremove`a` >>>d {'b':2} >>>d['c']=3#let'sadd'c':3 >>>'c'ind#membershipischeckedagainstthekeys True >>>3ind#notthevalues False >>>'e'ind False >>>d.clear()#let'scleaneverythingfromthisdictionary >>>d {} Noticehowaccessingkeysofadictionary,regardlessofthetypeofoperationwe’re performing,isdonethroughsquarebrackets.Doyourememberstrings,list,andtuples? Wewereaccessingelementsatsomepositionthroughsquarebracketsaswell.Yetanother exampleofPython’sconsistency. Let’sseenowthreespecialobjectscalleddictionaryviews:keys,values,anditems. Theseobjectsprovideadynamicviewofthedictionaryentriesandtheychangewhenthe dictionarychanges.keys()returnsallthekeysinthedictionary,values()returnsallthe valuesinthedictionary,anditems()returnsallthe(key,value)pairsinthedictionary. Note It’sveryimportanttoknowthat,evenifadictionaryisnotintrinsicallyordered,according tothePythondocumentation:“Keysandvaluesareiteratedoverinanarbitraryorder whichisnon-random,variesacrossPythonimplementations,anddependsonthe dictionary’shistoryofinsertionsanddeletions.Ifkeys,valuesanditemsviewsareiterated overwithnointerveningmodificationstothedictionary,theorderofitemswilldirectly correspond.” Enoughwiththischatter,let’sputallthisdownintocode: >>>d=dict(zip('hello',range(5))) >>>d {'e':1,'h':0,'o':4,'l':3} >>>d.keys() dict_keys(['e','h','o','l']) >>>d.values() dict_values([1,0,4,3]) >>>d.items() dict_items([('e',1),('h',0),('o',4),('l',3)]) >>>3ind.values() True >>>('o',4)ind.items() True Afewthingstonoticeintheprecedingcode.First,noticehowwe’recreatingadictionary byiteratingoverthezippedversionofthestring'hello'andthelist[0,1,2,3,4]. Thestring'hello'hastwo'l'charactersinside,andtheyarepairedupwiththevalues2 WOW! eBook www.wowebook.org and3bythezipfunction.Noticehowinthedictionary,thesecondoccurrenceofthe'l' key(theonewithvalue3),overwritesthefirstone(theonewithvalue2).Anotherthingto noticeisthatwhenaskingforanyview,theoriginalorderislost,butisconsistentwithin theviews,asexpected.Noticealsothatyoumayhavedifferentresultswhenyoutrythis codeonyourmachine.Pythondoesn’tguaranteethat,itonlyguaranteestheconsistencyof theorderinwhichtheviewsarepresented. We’llseehowtheseviewsarefundamentaltoolswhenwetalkaboutiteratingover collections.Let’stakealooknowatsomeothermethodsexposedbyPython’s dictionaries,there’splentyofthemandtheyareveryuseful: >>>d {'e':1,'h':0,'o':4,'l':3} >>>d.popitem()#removesarandomitem ('e',1) >>>d {'h':0,'o':4,'l':3} >>>d.pop('l')#removeitemwithkey`l` 3 >>>d.pop('not-a-key')#removeakeynotindictionary:KeyError Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> KeyError:'not-a-key' >>>d.pop('not-a-key','default-value')#withadefaultvalue? 'default-value'#wegetthedefaultvalue >>>d.update({'another':'value'})#wecanupdatedictthisway >>>d.update(a=13)#orthisway(likeafunctioncall) >>>d {'a':13,'another':'value','h':0,'o':4} >>>d.get('a')#sameasd['a']butifkeyismissingnoKeyError 13 >>>d.get('a',177)#defaultvalueusedifkeyismissing 13 >>>d.get('b',177)#likeinthiscase 177 >>>d.get('b')#keyisnotthere,soNoneisreturned Allthesemethodsarequitesimpletounderstand,butit’sworthtalkingaboutthatNone, foramoment.EveryfunctioninPythonreturnsNone,unlessthereturnstatementis explicitlyused,butwe’llseethiswhenweexplorefunctions.Noneisfrequentlyusedto representtheabsenceofavalue,aswhendefaultargumentsarenotpassedtoafunction. SomeinexperiencedcoderssometimeswritecodethatreturnseitherFalseorNone.Both FalseandNoneevaluatetoFalsesoitmayseemthereisnotmuchdifferencebetween them.Butactually,Iwouldarguethereisquiteanimportantdifference:Falsemeansthat wehaveinformation,andtheinformationwehaveisFalse.Nonemeansnoinformation. Andnoinformationisverydifferentfromaninformation,whichisFalse.Inlayman’s terms,ifyouaskyourmechanic“ismycarready?”thereisabigdifferencebetweenthe answer“No,it’snot”(False)and“Ihavenoidea”(None). OnelastmethodIreallylikeofdictionariesissetdefault.Itbehaveslikeget,butalso setsthekeywiththegivenvalueifitisnotthere.Let’sseeandexample: WOW! eBook www.wowebook.org >>>d={} >>>d.setdefault('a',1)#'a'ismissing,wegetdefaultvalue 1 >>>d {'a':1}#also,thekey/valuepair('a',1)hasnowbeenadded >>>d.setdefault('a',5)#let'strytooverridethevalue 1 >>>d {'a':1}#didn'twork,asexpected So,we’renowattheendofthistour.Testyourknowledgeaboutdictionariestryingto foreseehowdlookslikeafterthisline. >>>d={} >>>d.setdefault('a',{}).setdefault('b',[]).append(1) It’snotthatcomplicated,butdon’tworryifyoudon’tgetitimmediately.Ijustwantedto spuryoutoexperimentwithdictionaries. Thisconcludesourtourofbuilt-indatatypes.BeforeImakesomeconsiderationsabout whatwe’veseeninthischapter,Iwanttobrieflytakeapeekatthecollectionsmodule. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Thecollectionsmodule WhenPythongeneralpurposebuilt-incontainers(tuple,list,set,anddict)aren’t enough,wecanfindspecializedcontainerdatatypesinthecollectionsmodule.They are: Datatype Description namedtuple() Afactoryfunctionforcreatingtuplesubclasseswithnamedfields deque Alist-likecontainerwithfastappendsandpopsoneitherend ChainMap Adict-likeclassforcreatingasingleviewofmultiplemappings Counter Adictsubclassforcountinghashableobjects OrderedDict Adictsubclassthatrememberstheorderentrieswereadded defaultdict Adictsubclassthatcallsafactoryfunctiontosupplymissingvalues UserDict Awrapperarounddictionaryobjectsforeasierdictsubclassing UserList Awrapperaroundlistobjectsforeasierlistsubclassing UserString Awrapperaroundstringobjectsforeasierstringsubclassing Wedon’thavetheroomtocoverallofthem,butyoucanfindplentyofexamplesinthe officialdocumentation,sohereI’lljustgiveasmallexampletoshowyounamedtuple, defaultdict,andChainMap. WOW! eBook www.wowebook.org Namedtuples Anamedtupleisatuple-likeobjectthathasfieldsaccessiblebyattributelookupaswell asbeingindexableanditerable(it’sactuallyasubclassoftuple).Thisissortofa compromisebetweenafull-fledgedobjectandatuple,anditcanbeusefulinthosecases whereyoudon’tneedthefullpowerofacustomobject,butyouwantyourcodetobe morereadablebyavoidingweirdindexing.Anotherusecaseiswhenthereisachancethat itemsinthetupleneedtochangetheirpositionafterrefactoring,forcingthecoderto refactoralsoallthelogicinvolved,whichcanbeverytricky.Asusual,anexampleis betterthanathousandwords(orwasitapicture?).Saywearehandlingdataabouttheleft andrighteyeofapatient.Wesaveonevalueforthelefteye(position0)andoneforthe righteye(position1)inaregulartuple.Here’showthatmightbe: >>>vision=(9.5,8.8) >>>vision (9.5,8.8) >>>vision[0]#lefteye(implicitpositionalreference) 9.5 >>>vision[1]#righteye(implicitpositionalreference) 8.8 Nowlet’spretendwehandlevisionobjectallthetime,andatsomepointthedesigner decidestoenhancethembyaddinginformationforthecombinedvision,sothatavision objectstoresdatainthisformat:(lefteye,combined,righteye). Doyouseethetroublewe’reinnow?Wemayhavealotofcodethatdependson vision[0]beingthelefteyeinformation(whichstillis)andvision[1]beingtheright eyeinformation(whichisnolongerthecase).Wehavetorefactorourcodewhereverwe handletheseobjects,changingvision[1]tovision[2],anditcanbepainful.Wecould haveprobablyapproachedthisabitbetterfromthebeginning,byusinganamedtuple.Let meshowyouwhatImean: >>>fromcollectionsimportnamedtuple >>>Vision=namedtuple('Vision',['left','right']) >>>vision=Vision(9.5,8.8) >>>vision[0] 9.5 >>>vision.left#sameasvision[0],butexplicit 9.5 >>>vision.right#sameasvision[1],butexplicit 8.8 Ifwithinourcodewerefertoleftandrighteyeusingvision.leftandvision.right,all weneedtodotofixthenewdesignissueistochangeourfactoryandthewaywecreate instances.Therestofthecodewon’tneedtochange. >>>Vision=namedtuple('Vision',['left','combined','right']) >>>vision=Vision(9.5,9.2,8.8) >>>vision.left#stillperfect 9.5 >>>vision.right#stillperfect(thoughnowisvision[2]) WOW! eBook www.wowebook.org 8.8 >>>vision.combined#thenewvision[1] 9.2 Youcanseehowconvenientitistorefertothosevaluesbynameratherthanbyposition. Afterall,awisemanoncewrote“Explicitisbetterthanimplicit”(canyourecallwhere? Thinkzenifyoudon’t…).Thisexamplemaybealittleextreme,ofcourseit’snotlikely thatourcodedesignerwillgoforachangelikethis,butyou’dbeamazedtoseehow frequentlyissuessimilartothisonehappeninaprofessionalenvironment,andhow painfulitistorefactorthem. WOW! eBook www.wowebook.org Defaultdict Thedefaultdictdatatypeisoneofmyfavorites.Itallowsyoutoavoidcheckingifakey isinadictionarybysimplyinsertingitforyouonyourfirstaccessattempt,withadefault valuewhosetypeyoupassoncreation.Insomecases,thistoolcanbeveryhandyand shortenyourcodealittle.Let’sseeaquickexample:sayweareupdatingthevalueofage, byaddingoneyear.Ifageisnotthere,weassumeitwas0andweupdateitto1. >>>d={} >>>d['age']=d.get('age',0)+1#agenotthere,weget0+1 >>>d {'age':1} >>>d={'age':39} >>>d['age']=d.get('age',0)+1#disthere,weget40 >>>d {'age':40} Nowlet’sseehowitwouldworkwithadefaultdictdatatype.Thesecondlineis actuallytheshortversionofa4-lineslongifclausethatwewouldhavetowriteif dictionariesdidn’thavethegetmethod.We’llseeallaboutifclausesinChapter3, IteratingandMakingDecisions. >>>fromcollectionsimportdefaultdict >>>dd=defaultdict(int)#intisthedefaulttype(0thevalue) >>>dd['age']+=1#shortfordd['age']=dd['age']+1 >>>dd defaultdict(<class'int'>,{'age':1})#1,asexpected >>>dd['age']=39 >>>dd['age']+=1 >>>dd defaultdict(<class'int'>,{'age':40})#40,asexpected Noticehowwejustneedtoinstructthedefaultdictfactorythatwewantanintnumber tobeusedincasethekeyismissing(we’llget0,whichisthedefaultfortheinttype). Also,noticethateventhoughinthisexamplethereisnogainonthenumberoflines,there isdefinitelyagaininreadability,whichisveryimportant.Youcanalsouseadifferent techniquetoinstantiateadefaultdictdatatype,whichinvolvescreatingafactoryobject. Fordiggingdeeper,pleaserefertotheofficialdocumentation. WOW! eBook www.wowebook.org ChainMap TheChainMapisanextremelynicedatatypewhichwasintroducedinPython3.3.It behaveslikeanormaldictionarybutaccordingtothePythondocumentation:isprovided forquicklylinkinganumberofmappingssotheycanbetreatedasasingleunit.Thisis usuallymuchfasterthancreatingonedictionaryandrunningmultipleupdatecallsonit. ChainMapcanbeusedtosimulatenestedscopesandisusefulintemplating.The underlyingmappingsarestoredinalist.Thatlistispublicandcanbeaccessedorupdated usingthemapsattribute.Lookupssearchtheunderlyingmappingssuccessivelyuntilakey isfound.Incontrast,writes,updates,anddeletionsonlyoperateonthefirstmapping. Averycommonusecaseisprovidingdefaults,solet’sseeanexample: >>>fromcollectionsimportChainMap >>>default_connection={'host':'localhost','port':4567} >>>connection={'port':5678} >>>conn=ChainMap(connection,default_connection)#mapcreation >>>conn['port']#portisfoundinthefirstdictionary 5678 >>>conn['host']#hostisfetchedfromtheseconddictionary 'localhost' >>>conn.maps#wecanseethemappingobjects [{'port':5678},{'host':'localhost','port':4567}] >>>conn['host']='packtpub.com'#let'saddhost >>>conn.maps [{'host':'packtpub.com','port':5678}, {'host':'localhost','port':4567}] >>>delconn['port']#let'sremovetheportinformation >>>conn.maps [{'host':'packtpub.com'}, {'host':'localhost','port':4567}] >>>conn['port']#nowportisfetchedfromtheseconddictionary 4567 >>>dict(conn)#easytomergeandconverttoregulardictionary {'host':'packtpub.com','port':4567} IjustlovehowPythonmakesyourlifeeasy.YouworkonaChainMapobject,configure thefirstmappingasyouwant,andwhenyouneedacompletedictionarywithallthe defaultsaswellasthecustomizeditems,youjustfeedtheChainMapobjecttoadict constructor.Ifyouhavenevercodedinotherlanguages,suchasJavaorC++,you probablywon’tbeabletofullyappreciatehowpreciousthisis,howPythonmakesyour lifesomucheasier.Ido,IfeelclaustrophobiceverytimeIhavetocodeinsomeother language. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Finalconsiderations That’sit.Nowyouhaveseenaverygoodportionofthedatastructuresthatyouwillusein Python.IencourageyoutotakeadiveintothePythondocumentationandexperiment furtherwitheachandeverydatatypewe’veseeninthischapter.It’sworthit,believeme. Everythingyou’llwritewillbeabouthandlingdata,somakesureyourknowledgeaboutit isrocksolid. Beforeweleapintothenextchapter,I’dliketomakesomefinalconsiderationsabout differentaspectsthattomymindareimportantandnottobeneglected. WOW! eBook www.wowebook.org Smallvaluescaching Whenwediscussedobjectsatthebeginningofthischapter,wesawthatwhenweassigned anametoanobject,Pythoncreatestheobject,setsitsvalue,andthenpointsthenameto it.Wecanassigndifferentnamestothesamevalueandweexpectdifferentobjectstobe created,likethis: >>>a=1000000 >>>b=1000000 >>>id(a)==id(b) False Intheprecedingexample,aandbareassignedtotwointobjects,whichhavethesame valuebuttheyarenotthesameobject,asyoucansee,theiridisnotthesame.Solet’sdo itagain: >>>a=5 >>>b=5 >>>id(a)==id(b) True Ohoh!IsPythonbroken?Whyarethetwoobjectsthesamenow?Wedidn’tdoa=b= 5,wesetthemupseparately.Well,theanswerisperformances.Pythoncachesshort stringsandsmallnumbers,toavoidhavingmanycopiesofthemcloggingupthesystem memory.Everythingishandledproperlyunderthehoodsoyoudon’tneedtoworryabit, butmakesurethatyourememberthisbehaviorshouldyourcodeeverneedtofiddlewith IDs. WOW! eBook www.wowebook.org Howtochoosedatastructures Aswe’veseen,Pythonprovidesyouwithseveralbuilt-indatatypesandsometimes,if you’renotthatexperienced,choosingtheonethatservesyoubestcanbetricky,especially whenitcomestocollections.Forexample,sayyouhavemanydictionariestostore,each ofwhichrepresentsacustomer.Withineachcustomerdictionarythere’san'id':'code' uniqueidentificationcode.Inwhatkindofcollectionwouldyouplacethem?Well,unless Iknowmoreaboutthesecustomers,it’sveryhardtoanswer.WhatkindofaccesswillI need?WhatsortofoperationswillIhavetoperformoneachofthem,andhowmany times?Willthecollectionchangeovertime?WillIneedtomodifythecustomer dictionariesinanyway?WhatisgoingtobethemostfrequentoperationIwillhaveto performonthecollection? Ifyoucananswertheprecedingquestions,thenyouwillknowwhattochoose.Ifthe collectionnevershrinksorgrows(inotherwords,itwon’tneedtoadd/deleteany customerobjectaftercreation)orshuffles,thentuplesareapossiblechoice.Otherwise listsareagoodcandidate.Everycustomerdictionaryhasauniqueidentifierthough,so evenadictionarycouldwork.Letmedrafttheseoptionsforyou: #examplecustomerobjects customer1={'id':'abc123','full_name':'MasterYoda'} customer2={'id':'def456','full_name':'Obi-WanKenobi'} customer3={'id':'ghi789','full_name':'AnakinSkywalker'} #collecttheminatuple customers=(customer1,customer2,customer3) #orcollecttheminalist customers=[customer1,customer2,customer3] #ormaybewithinadictionary,theyhaveauniqueidafterall customers={ 'abc123':customer1, 'def456':customer2, 'ghi789':customer3, } Somecustomerswehavethere,right?Iprobablywouldn’tgowiththetupleoption,unless Iwantedtohighlightthatthecollectionisnotgoingtochange.I’dsayusuallyalistis better,itallowsformoreflexibility. Anotherfactortokeepinmindisthattuplesandlistsareorderedcollections,whileifyou useadictionaryorasetyoulosetheordering,soyouneedtoknowiforderingis importantinyourapplication. Whataboutperformances?Forexampleinalist,operationssuchasinsertionand membershipcantakeO(n),whiletheyareO(1)foradictionary.It’snotalwayspossibleto usedictionariesthough,ifwedon’thavetheguaranteethatwecanuniquelyidentifyeach itemofthecollectionbymeansofoneofitsproperties,andthatthepropertyinquestionis hashable(soitcanbeakeyindict). Note Ifyou’rewonderingwhatO(n)andO(1)mean,pleaseGoogle“bigOnotation”andgeta WOW! eBook www.wowebook.org gistofitfromanywhere.Inthiscontext,let’sjustsaythatifperforminganoperationOp to onadatastructuretakesO(f(n)),itwouldmeanthatOptakesatmostatime complete,wherecissomepositiveconstant,nisthesizeoftheinput,andfissome function.So,thinkofO(…)asanupperboundfortherunningtimeofanoperation(itcan beusedalsotosizeothermeasurablequantities,ofcourse). Anotherwayofunderstandingifyouhavechosentherightdatastructureisbylookingat thecodeyouhavetowriteinordertomanipulateit.Ifeverythingcomeseasilyandflows naturally,thenyouprobablyhavechosencorrectly,butifyoufindyourselfthinkingyour codeisgettingunnecessarilycomplicated,thenyouprobablyshouldtryanddecide whetheryouneedtoreconsideryourchoices.It’squitehardtogiveadvicewithouta practicalcasethough,sowhenyouchooseadatastructureforyourdata,trytokeepease ofuseandperformanceinmindandgiveprecedencetowhatmattersmostinthecontext youare. WOW! eBook www.wowebook.org Aboutindexingandslicing Atthebeginningofthischapter,wesawslicingappliedonstrings.Slicingingeneral appliestoasequence,sotuples,lists,strings,etc.Withlists,slicingcanalsobeusedfor assignment.I’vealmostneverseenthisusedinprofessionalcode,butstill,youknowyou can.Couldyouslicedictionariesorsets?Ihearyouscream“Ofcoursenot!Theyarenot ordered!“.Excellent,Iseewe’reonthesamepagehere,solet’stalkaboutindexing. ThereisonecharacteristicaboutPythonindexingIhaven’tmentionedbefore.I’llshow youbyexample.Howdoyouaddressthelastelementofacollection?Let’ssee: >>>a=list(range(10))#`a`has10elements.Lastoneis9. >>>a [0,1,2,3,4,5,6,7,8,9] >>>len(a)#itslengthis10elements 10 >>>a[len(a)-1]#positionoflastoneislen(a)-1 9 >>>a[-1]#butwedon'tneedlen(a)!Pythonrocks! 9 >>>a[-2]#equivalenttolen(a)-2 8 >>>a[-3]#equivalenttolen(a)-3 7 Ifthelistahas10elements,becauseofthe0-indexpositioningsystemofPython,thefirst oneisatposition0andthelastoneisatposition9.Intheprecedingexample,theelements areconvenientlyplacedinapositionequaltotheirvalue:0isatposition0,1atposition1, andsoon. So,inordertofetchthelastelement,weneedtoknowthelengthofthewholelist(or tuple,orstring,andsoon)andthensubtract1.Hence:len(a)–1.Thisissocommonan operationthatPythonprovidesyouwithawaytoretrieveelementsusingnegative indexing.Thisprovesveryusefulwhenyoudosomeseriousdatamanipulation.Here’sa nicediagramabouthowindexingworksonthestring"HelloThere": Tryingtoaddressindexesgreaterthan9orsmallerthan-10willraiseanIndexError,as expected. WOW! eBook www.wowebook.org Aboutthenames Youmayhavenoticedthat,inordertokeeptheexampleasshortaspossible,Ihavecalled manyobjectsusingsimpleletters,likea,b,c,d,andsoon.Thisisperfectlyokwhenyou debugontheconsoleorwhenyoushowthata+b==7,butit’sbadpracticewhenit comestoprofessionalcoding(oranytypeofcoding,forallthatmatter).Ihopeyouwill indulgemeifIsometimesdoit,thereasonistopresentthecodeinamorecompactway. Inarealenvironmentthough,whenyouchoosenamesforyourdata,youshouldchoose themcarefullyandtheyshouldreflectwhatthedataisabout.So,ifyouhaveacollection ofCustomerobjects,customersisaperfectlygoodnameforit.Wouldcustomers_list, customers_tuple,orcustomers_collectionworkaswell?Thinkaboutitforasecond. Isitgoodtotiethenameofthecollectiontothedatatype?Idon’tthinkso,atleastin mostcases.SoI’dsayifyouhaveanexcellentreasontodosogoahead,otherwisedon’t. Thereasonis,oncethatcustomers_tuplestartsbeingusedindifferentplacesofyour code,andyourealizeyouactuallywanttousealistinsteadofatuple,you’reupforsome funrefactoring(alsoknownaswastedtime).Namesfordatashouldbenouns,andnames forfunctionsshouldbeverbs.Namesshouldbeasexpressiveaspossible.Pythonis actuallyaverygoodexamplewhenitcomestonames.Mostofthetimeyoucanjustguess whatafunctioniscalledifyouknowwhatitdoes.Crazy,huh? Chapter2,MeaningfulNamesofCleanCode,RobertC.Martin,PrenticeHallisentirely dedicatedtonames.It’sanamazingbookthathelpedmeimprovemycodingstylein manydifferentways,amustreadifyouwanttotakeyourcodingtothenextlevel. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,we’veexploredthebuilt-indatatypesofPython.We’veseenhowmany theyareandhowmuchcanbeachievedbyjustusingthemindifferentcombinations. We’veseennumbertypes,sequences,sets,mappings,collections,we’veseenthat everythingisanobject,we’velearnedthedifferencebetweenmutableandimmutable,and we’vealsolearnedaboutslicingandindexing(and,proudly,negativeindexingaswell). We’vepresentedsimpleexamples,butthere’smuchmorethatyoucanlearnaboutthis subject,sostickyournoseintotheofficialdocumentationandexplore. Mostofall,Iencourageyoutotryoutalltheexercisesbyyourself,getyourfingersusing thatcode,buildsomemusclememory,andexperiment,experiment,experiment.Learn whathappenswhenyoudividebyzero,whenyoucombinedifferentnumbertypesintoa singleexpression,whenyoumanagestrings.Playwithalldatatypes.Exercisethem,break them,discoveralltheirmethods,enjoythemandlearnthemwell,damnwell. Ifyourfoundationisnotrocksolid,howgoodcanyourcodebe?Anddataisthe foundationforeverything.Datashapeswhatdancesaroundit. Themoreyouprogresswiththebook,themoreit’slikelythatyouwillfindsome discrepanciesormaybeasmalltypohereandthereinmycode(oryours).Youwillgetan errormessage,somethingwillbreak.That’swonderful!Whenyoucode,thingsbreakall thetime,youdebugandfixallthetime,soconsidererrorsasusefulexercisestolearn somethingnewaboutthelanguageyou’reusing,andnotasfailuresorproblems.Errors willkeepcomingupuntilyourverylastlineofcode,that’sforsure,soyoumayaswell startmakingyourpeacewiththemnow. Thenextchapterisaboutiteratingandmakingdecisions.We’llseehowtoactuallyput thosecollectionsinuse,andtakedecisionsbasedonthedatawe’representedwith.We’ll starttogoalittlefasternowthatyourknowledgeisbuildingup,somakesureyou’re comfortablewiththecontentsofthischapterbeforeyoumovetothenextone.Oncemore, havefun,explore,breakthings.It’saverygoodwaytolearn. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter3.IteratingandMaking Decisions “Insanity:doingthesamethingoverandoveragainandexpectingdifferentresults.” —AlbertEinstein Inthepreviouschapter,we’veseenPythonbuilt-indatatypes.Nowthatyou’refamiliar withdatainitsmanyformsandshapes,it’stimetostartlookingathowaprogramcanuse it. AccordingtoWikipedia: Incomputerscience,controlflow(oralternatively,flowofcontrol)referstothe specificationoftheorderinwhichtheindividualstatements,instructionsorfunction callsofanimperativeprogramareexecutedorevaluated. Inordertocontroltheflowofaprogram,wehavetwomainweapons:conditional programming(alsoknownasbranching)andlooping.Wecanusetheminmany differentcombinationsandvariations,butinthischapter,insteadofgoingthroughall possiblevariousformsofthosetwoconstructsina“documentation”fashion,I’drather giveyouthebasicsandthenI’llwriteacoupleofsmallscriptswithyou.Inthefirstone, we’llseehowtocreatearudimentaryprimenumbergenerator,whileinthesecondone, we’llseehowtoapplydiscountstocustomersbasedoncoupons.Thiswayyoushouldget abetterfeelingabouthowconditionalprogrammingandloopingcanbeused. WOW! eBook www.wowebook.org Conditionalprogramming Conditionalprogramming,orbranching,issomethingyoudoeveryday,everymoment. It’saboutevaluatingconditions:ifthelightisgreen,thenIcancross,ifit’sraining,then I’mtakingtheumbrella,andifI’mlateforwork,thenI’llcallmymanager. Themaintoolistheifstatement,whichcomesindifferentformsandcolors,butbasically whatitdoesisevaluateanexpressionand,basedontheresult,choosewhichpartofthe codetoexecute.Asusual,let’sseeanexample: conditional.1.py late=True iflate: print('Ineedtocallmymanager!') Thisispossiblythesimplestexample:whenfedtotheifstatement,lateactsasa conditionalexpression,whichisevaluatedinaBooleancontext(exactlylikeifwewere callingbool(late)).IftheresultoftheevaluationisTrue,thenweenterthebodyofcode immediatelyaftertheifstatement.Noticethattheprintinstructionisindented:this meansitbelongstoascopedefinedbytheifclause.Executionofthiscodeyields: $pythonconditional.1.py Ineedtocallmymanager! SincelateisTrue,theprintstatementwasexecuted.Let’sexpandonthisexample: conditional.2.py late=False iflate: print('Ineedtocallmymanager!')#1 else: print('noneedtocallmymanager…')#2 ThistimeIsetlate=False,sowhenIexecutethecode,theresultisdifferent: $pythonconditional.2.py noneedtocallmymanager… Dependingontheresultofevaluatingthelateexpression,wecaneitherenterblock#1or block#2,butnotboth.Block#1isexecutedwhenlateevaluatestoTrue,whileblock#2 isexecutedwhenlateevaluatestoFalse.TryassigningFalse/Truevaluestothelate name,andseehowtheoutputforthiscodechangesaccordingly. Theprecedingexamplealsointroducestheelseclause,whichbecomesveryhandywhen wewanttoprovideanalternativesetofinstructionstobeexecutedwhenanexpression evaluatestoFalsewithinanifclause.Theelseclauseisoptional,asit’sevidentby comparingtheprecedingtwoexamples. WOW! eBook www.wowebook.org Aspecializedelse:elif Sometimesallyouneedistodosomethingifaconditionismet(simpleifclause).Other timesyouneedtoprovideanalternative,incasetheconditionisFalse(if/elseclause), buttherearesituationswhereyoumayhavemorethantwopathstochoosefrom,so,since callingthemanager(ornotcallingthem)iskindofabinarytypeofexample(eitheryou calloryoudon’t),let’schangethetypeofexampleandkeepexpanding.Thistimewe decidetaxpercentages.Ifmyincomeislessthen10k,Iwon’tpayanytaxes.Ifitis between10kand30k,I’llpay20%taxes.Ifitisbetween30kand100k,I’llpay35% taxes,andover100k,I’ll(gladly)pay45%taxes.Let’sputthisalldownintobeautiful Pythoncode: taxes.py income=15000 ifincome<10000: tax_coefficient=0.0#1 elifincome<30000: tax_coefficient=0.2#2 elifincome<100000: tax_coefficient=0.35#3 else: tax_coefficient=0.45#4 print('Iwillpay:',income*tax_coefficient,'intaxes') Executingtheprecedingcodeyields: $pythontaxes.py Iwillpay:3000.0intaxes Let’sgothroughtheexamplelinebyline:westartbysettinguptheincomevalue.Inthe example,myincomeis15k.Weentertheifclause.Noticethatthistimewealso introducedtheelifclause,whichisacontractionforelse-if,andit’sdifferentfroma bareelseclauseinthatitalsohasitsowncondition.So,theifexpressionincome< 10000,evaluatestoFalse,thereforeblock#1isnotexecuted.Thecontrolpassestothe nextconditionevaluator:elifincome<30000.ThisoneevaluatestoTrue,therefore block#2isexecuted,andbecauseofthis,Pythonthenresumesexecutionafterthewhole if/elif/elif/elseclause(whichwecanjustcallifclausefromnowon).Thereisonly oneinstructionaftertheifclause,theprintcall,whichtellsusIwillpay3kintaxesthis year(15k*20%).Noticethattheorderismandatory:ifcomesfirst,then(optionally)as manyelifasyouneed,andthen(optionally)anelseclause. Interesting,right?Nomatterhowmanylinesofcodeyoumayhavewithineachblock, whenoneoftheconditionsevaluatestoTrue,theassociatedblockisexecutedandthen executionresumesafterthewholeclause.IfnoneoftheconditionsevaluatestoTrue(for example,income=200000),thenthebodyoftheelseclausewouldbeexecuted(block #4).Thisexampleexpandsourunderstandingofthebehavioroftheelseclause.Itsblock ofcodeisexecutedwhennoneoftheprecedingif/elif/…/elifexpressionshas evaluatedtoTrue. WOW! eBook www.wowebook.org Trytomodifythevalueofincomeuntilyoucancomfortablyexecuteallblocksatyour will(oneperexecution,ofcourse).Andthentrytheboundaries.Thisiscrucial, wheneveryouhaveconditionsexpressedasequalitiesorinequalities(==,!=,<,>,<=, >=),thosenumbersrepresentboundaries.Itisessentialtotestboundariesthoroughly. ShouldIallowyoutodriveat18or17?AmIcheckingyouragewithage<18,orage <=18?Youcan’timaginehowmanytimesIhadtofixsubtlebugsthatstemmedfrom usingthewrongoperator,sogoaheadandexperimentwiththeprecedingcode.Change some<to<=andsetincometobeoneoftheboundaryvalues(10k,30k,100k)aswellas anyvalueinbetween.Seehowtheresultchanges,getagoodunderstandingofitbefore proceeding. Beforewemovetothenexttopic,let’sseeanotherexamplethatshowsushowtonestif clauses.Sayyourprogramencountersanerror.Ifthealertsystemistheconsole,weprint theerror.Ifthealertsystemisane-mail,wesenditaccordingtotheseverityoftheerror. Ifthealertsystemisanythingotherthanconsoleore-mail,wedon’tknowwhattodo, thereforewedonothing.Let’sputthisintocode: errorsalert.py alert_system='console'#othervaluecanbe'email' error_severity='critical'#othervalues:'medium'or'low' error_message='OMG!Somethingterriblehappened!' ifalert_system=='console': print(error_message)#1 elifalert_system=='email': iferror_severity=='critical': send_email('[email protected]',error_message)#2 eliferror_severity=='medium': send_email('[email protected]',error_message)#3 else: send_email('[email protected]',error_message)#4 Theprecedingexampleisquiteinteresting,initssilliness.Itshowsustwonestedif clauses(outerandinner).Italsoshowsustheouterifclausedoesn’thaveanyelse, whiletheinneronedoes.Noticehowindentationiswhatallowsustonestoneclause withinanotherone. Ifalert_system=='console',body#1isexecuted,andnothingelsehappens.Onthe otherhand,ifalert_system=='email',thenweenterintoanotherifclause,whichwe calledinner.Intheinnerifclause,accordingtoerror_severity,wesendane-mailto eitheranadmin,first-levelsupport,orsecond-levelsupport(blocks#2,#3,and#4).The send_emailfunctionisnotdefinedinthisexample,thereforetryingtorunitwouldgive youanerror.Inthesourcecodeofthebook,whichyoucandownloadfromthewebsite,I includedatricktoredirectthatcalltoaregularprintfunction,justsoyoucanexperiment ontheconsolewithoutactuallysendingane-mail.Trychangingthevaluesandseehowit allworks. WOW! eBook www.wowebook.org Theternaryoperator OnelastthingIwouldliketoshowyoubeforemovingontothenextsubject,isthe ternaryoperatoror,inlayman’sterms,theshortversionofanif/elseclause.Whenthe valueofanameistobeassignedaccordingtosomecondition,sometimesit’seasierand morereadabletousetheternaryoperatorinsteadofaproperifclause.Inthefollowing example,thetwocodeblocksdoexactlythesamething: ternary.py order_total=247#GBP #classicif/elseform iforder_total>100: discount=25#GBP else: discount=0#GBP print(order_total,discount) #ternaryoperator discount=25iforder_total>100else0 print(order_total,discount) Forsimplecaseslikethis,Ifinditverynicetobeabletoexpressthatlogicinoneline insteadoffour.Remember,asacoder,youspendmuchmoretimereadingcodethen writingit,soPythonconcisenessisinvaluable. Areyouclearonhowtheternaryoperatorworks?Basicallyisname=somethingif conditionelsesomething-else.Sonameisassignedsomethingifconditionevaluates toTrue,andsomething-elseifconditionevaluatestoFalse. Nowthatyouknoweverythingaboutcontrollingthepathofthecode,let’smoveontothe nextsubject:looping. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Looping Ifyouhaveanyexperiencewithloopinginotherprogramminglanguages,youwillfind Python’swayofloopingabitdifferent.Firstofall,whatislooping?Loopingmeansbeing abletorepeattheexecutionofacodeblockmorethanonce,accordingtotheloop parameterswe’regiven.Therearedifferentloopingconstructs,whichservedifferent purposes,andPythonhasdistilledallofthemdowntojusttwo,whichyoucanuseto achieveeverythingyouneed.Thesearetheforandwhilestatements. Whileit’sdefinitelypossibletodoeverythingyouneedusingeitherofthem,theyserve differentpurposesandthereforethey’reusuallyusedindifferentcontexts.We’llexplore thisdifferencethoroughlythroughthischapter. WOW! eBook www.wowebook.org Theforloop Theforloopisusedwhenloopingoverasequence,likealist,tuple,oracollectionof objects.Let’sstartwithasimpleexamplethatismorelikeC++style,andthenlet’s graduallyseehowtoachievethesameresultsinPython(you’lllovePython’ssyntax). simple.for.py fornumberin[0,1,2,3,4]: print(number) Thissimplesnippetofcode,whenexecuted,printsallnumbersfrom0to4.Theforloop isfedthelist[0,1,2,3,4]andateachiteration,numberisgivenavaluefromthe sequence(whichisiteratedsequentially,inorder),thenthebodyoftheloopisexecuted (theprintline).numberchangesateveryiteration,accordingtowhichvalueiscoming nextfromthesequence.Whenthesequenceisexhausted,theforloopterminates,andthe executionofthecoderesumesnormallywiththecodeaftertheloop. Iteratingoverarange Sometimesweneedtoiterateoverarangeofnumbers,anditwouldbequiteunpleasantto havetodosobyhardcodingthelistsomewhere.Insuchcases,therangefunctioncomes totherescue.Let’sseetheequivalentoftheprevioussnippetofcode: simple.for.py fornumberinrange(5): print(number) TherangefunctionisusedextensivelyinPythonprogramswhenitcomestocreating sequences:youcancallitbypassingonevalue,whichactsasstop(countingfrom0),or youcanpasstwovalues(startandstop),oreventhree(start,stop,andstep).Check outthefollowingexample: >>>list(range(10))#onevalue:from0tovalue(excluded) [0,1,2,3,4,5,6,7,8,9] >>>list(range(3,8))#twovalues:fromstarttostop(excluded) [3,4,5,6,7] >>>list(range(-10,10,4))#threevalues:stepisadded [-10,-6,-2,2,6] Forthemoment,ignorethatweneedtowraprange(...)withinalist.Therangeobject isalittlebitspecial,butinthiscasewe’rejustinterestedinunderstandingwhatarethe valuesitwillreturntous.Youseethatthedealisthesamewithslicing:startisincluded, stopexcluded,andoptionallyyoucanaddastepparameter,whichbydefaultis1. Trymodifyingtheparametersoftherange()callinoursimple.for.pycodeandsee whatitprints,getcomfortablewithit. Iteratingoverasequence Nowwehaveallthetoolstoiterateoverasequence,solet’sbuildonthatexample: simple.for.2.py WOW! eBook www.wowebook.org surnames=['Rivest','Shamir','Adleman'] forpositioninrange(len(surnames)): print(position,surnames[position]) Theprecedingcodeaddsalittlebitofcomplexitytothegame.Executionwillshowthis result: $pythonsimple.for.2.py 0Rivest 1Shamir 2Adleman Let’susetheinside-outtechniquetobreakitdown,ok?Westartfromtheinnermostpart ofwhatwe’retryingtounderstand,andweexpandoutwards.So,len(surnames)isthe lengthofthesurnameslist:3.Therefore,range(len(surnames))isactuallytransformed intorange(3).Thisgivesustherange[0,3),whichisbasicallyasequence(0,1,2). Thismeansthattheforloopwillrunthreeiterations.Inthefirstone,positionwilltake value0,whileinthesecondone,itwilltakevalue1,andfinallyvalue2inthethirdand lastiteration.Whatis(0,1,2),ifnotthepossibleindexingpositionsforthesurnames list?Atposition0wefind'Rivest',atposition1,'Shamir',andatposition2, 'Adleman'.Ifyouarecuriousaboutwhatthesethreemencreatedtogether,change print(position,surnames[position])toprint(surnames[position][0],end='') addafinalprint()outsideoftheloop,andrunthecodeagain. Now,thisstyleofloopingisactuallymuchclosertolanguageslikeJavaorC++.In Pythonit’squiteraretoseecodelikethis.Youcanjustiterateoveranysequenceor collection,sothereisnoneedtogetthelistofpositionsandretrieveelementsoutofa sequenceateachiteration.It’sexpensive,needlesslyexpensive.Let’schangetheexample intoamorePythonicform: simple.for.3.py surnames=['Rivest','Shamir','Adleman'] forsurnameinsurnames: print(surname) Nowthat’ssomething!It’spracticallyEnglish.Theforloopcaniterateoverthesurnames list,anditgivesbackeachelementinorderateachinteraction.Runningthiscodewill printthethreesurnames,oneatatime.It’smucheasiertoread,right? Whatifyouwantedtoprintthepositionaswellthough?Orwhatifyouactuallyneededit foranyreason?Shouldyougobacktotherange(len(...))form?No.Youcanusethe enumeratebuilt-infunction,likethis: simple.for.4.py surnames=['Rivest','Shamir','Adleman'] forposition,surnameinenumerate(surnames): print(position,surname) Thiscodeisveryinterestingaswell.Noticethatenumerategivesbacka2-tuple (position,surname)ateachiteration,butstill,it’smuchmorereadable(andmore efficient)thantherange(len(...))example.Youcancallenumeratewithastart WOW! eBook www.wowebook.org parameter,likeenumerate(iterable,start),anditwillstartfromstart,ratherthan0. Justanotherlittlethingthatshowsyouhowmuchthoughthasbeengivenindesigning Pythonsothatitmakesyourlifeeasy. Usingaforloopitispossibletoiterateoverlists,tuples,andingeneralanythingthatin Pythoniscallediterable.Thisisaveryimportantconcept,solet’stalkaboutitabitmore. WOW! eBook www.wowebook.org Iteratorsanditerables AccordingtothePythondocumentation,aniterableis: “Anobjectcapableofreturningitsmembersoneatatime.Examplesofiterables includeallsequencetypes(suchaslist,str,andtuple)andsomenon-sequence typeslikedict,fileobjects,andobjectsofanyclassesyoudefinewithan __iter__()or__getitem__()method.Iterablescanbeusedinaforloopandin manyotherplaceswhereasequenceisneeded(zip(),map(),…).Whenaniterable objectispassedasanargumenttothebuilt-infunctioniter(),itreturnsaniterator fortheobject.Thisiteratorisgoodforonepassoverthesetofvalues.Whenusing iterables,itisusuallynotnecessarytocalliter()ordealwithiteratorobjects yourself.Theforstatementdoesthatautomaticallyforyou,creatingatemporary unnamedvariabletoholdtheiteratorforthedurationoftheloop.” Simplyput,whathappenswhenyouwriteforkinsequence:...body…,isthatthe forloopaskssequenceforthenextelement,itgetssomethingback,itcallsthat somethingk,andthenexecutesitsbody.Then,onceagain,theforloopaskssequence againforthenextelement,itcallsitkagain,andexecutesthebodyagain,andsoonand soforth,untilthesequenceisexhausted.Emptysequenceswillresultinzeroexecutionsof thebody. Somedatastructures,wheniteratedover,producetheirelementsinorder,likelists,tuples, andstrings,whilesomeothersdon’t,likesetsanddictionaries. Pythongivesustheabilitytoiterateoveriterables,usingatypeofobjectcallediterator. Accordingtotheofficialdocumentation,aniteratoris: “Anobjectrepresentingastreamofdata.Repeatedcallstotheiterator’s__next__() method(orpassingittothebuilt-infunctionnext())returnsuccessiveitemsinthe stream.WhennomoredataareavailableaStopIterationexceptionisraised instead.Atthispoint,theiteratorobjectisexhaustedandanyfurthercallstoits __next__()methodjustraiseStopIterationagain.Iteratorsarerequiredtohave an__iter__()methodthatreturnstheiteratorobjectitselfsoeveryiteratorisalso iterableandmaybeusedinmostplaceswhereotheriterablesareaccepted.One notableexceptioniscodewhichattemptsmultipleiterationpasses.Acontainerobject (suchasalist)producesafreshnewiteratoreachtimeyoupassittotheiter() functionoruseitinaforloop.Attemptingthiswithaniteratorwilljustreturnthe sameexhaustediteratorobjectusedinthepreviousiterationpass,makingitappear likeanemptycontainer.” Don’tworryifyoudon’tfullyunderstandalltheprecedinglegalese,youwillinduetime. Iputithereasahandyreferenceforthefuture. Inpractice,thewholeiterable/iteratormechanismissomewhathiddenbehindthecode. Unlessyouneedtocodeyourowniterableoriteratorforsomereason,youwon’thaveto worryaboutthistoomuch.Butit’sveryimportanttounderstandhowPythonhandlesthis WOW! eBook www.wowebook.org keyaspectofcontrolflowbecauseitwillshapethewayyouwillwriteyourcode. WOW! eBook www.wowebook.org Iteratingovermultiplesequences Let’sseeanotherexampleofhowtoiterateovertwosequencesofthesamelength,in ordertoworkontheirrespectiveelementsinpairs.Saywehavealistofpeopleandalist ofnumbersrepresentingtheageofthepeopleinthefirstlist.Wewanttoprintapair person/ageononelineforallofthem.Let’sstartwithanexampleandlet’srefineit gradually. multiple.sequences.py people=['Jonas','Julio','Mike','Mez'] ages=[25,30,31,39] forpositioninrange(len(people)): person=people[position] age=ages[position] print(person,age) Bynow,thiscodeshouldbeprettystraightforwardforyoutounderstand.Weneedto iterateoverthelistofpositions(0,1,2,3)becausewewanttoretrieveelementsfromtwo differentlists.Executingitwegetthefollowing: $pythonmultiple.sequences.py Jonas25 Julio30 Mike31 Mez39 ThiscodeisbothinefficientandnotPythonic.Inefficientbecauseretrievinganelement giventhepositioncanbeanexpensiveoperation,andwe’redoingitfromscratchateach iteration.Themailmandoesn’tgobacktothebeginningoftheroadeachtimehedelivers aletter,right?Hemovesfromhousetohouse.Fromonetothenextone.Let’strytomake itbetterusingenumerate: multiple.sequences.enumerate.py people=['Jonas','Julio','Mike','Mez'] ages=[25,30,31,39] forposition,personinenumerate(people): age=ages[position] print(person,age) Better,butstillnotperfect.Andstillabitugly.We’reiteratingproperlyonpeople,but we’restillfetchingageusingpositionalindexing,whichwewanttoloseaswell.Well,no worries,Pythongivesyouthezipfunction,remember?Let’suseit! multiple.sequences.zip.py people=['Jonas','Julio','Mike','Mez'] ages=[25,30,31,39] forperson,ageinzip(people,ages): print(person,age) Ah!Somuchbetter!Onceagain,comparetheprecedingcodewiththefirstexampleand admirePython’selegance.ThereasonIwantedtoshowthisexampleistwofold.Onthe WOW! eBook www.wowebook.org onehand,IwantedtogiveyouanideaofhowshorterthecodeinPythoncanbecompared tootherlanguageswherethesyntaxdoesn’tallowyoutoiterateoversequencesor collectionsaseasily.Andontheotherhand,andmuchmoreimportantly,noticethatwhen theforloopaskszip(sequenceA,sequenceB)forthenextelement,itgetsbackatuple, notjustasingleobject.Itgetsbackatuplewithasmanyelementsasthenumberof sequenceswefeedtothezipfunction.Let’sexpandalittleonthepreviousexamplein twoways:usingexplicitandimplicitassignment: multiple.sequences.explicit.py people=['Jonas','Julio','Mike','Mez'] ages=[25,30,31,39] nationalities=['Belgium','Spain','England','Bangladesh'] forperson,age,nationalityinzip(people,ages,nationalities): print(person,age,nationality) Intheprecedingcode,weaddedthenationalitieslist.Nowthatwefeedthreesequencesto thezipfunction,theforloopgetsbacka3-tupleateachiteration.Noticethattheposition oftheelementsinthetuplerespectsthepositionofthesequencesinthezipcall. Executingthecodewillyieldthefollowingresult: $pythonmultiple.sequences.explicit.py Jonas25Belgium Julio30Spain Mike31England Mez39Bangladesh Sometimes,forreasonsthatmaynotbeclearinasimpleexampleliketheprecedingone, youmaywanttoexplodethetuplewithinthebodyoftheforloop.Ifthatisyourdesire, it’sperfectlypossibletodoso. multiple.sequences.implicit.py people=['Jonas','Julio','Mike','Mez'] ages=[25,30,31,39] nationalities=['Belgium','Spain','England','Bangladesh'] fordatainzip(people,ages,nationalities): person,age,nationality=data print(person,age,nationality) It’sbasicallydoingwhattheforloopdoesautomaticallyforyou,butinsomecasesyou maywanttodoityourself.Here,the3-tupledatathatcomesfromzip(...),isexploded withinthebodyoftheforloopintothreevariables:person,age,andnationality. WOW! eBook www.wowebook.org Thewhileloop Intheprecedingpages,wesawtheforloopinaction.It’sincrediblyusefulwhenyou needtoloopoverasequenceoracollection.Thekeypointtokeepinmind,whenyou needtobeabletodiscriminatewhichloopingconstructtouse,isthattheforlooprocks whenyouhavetoiterateoverafiniteamountofelements.Itcanbeahugeamount,but still,somethingthatatsomepointends. Thereareothercasesthough,whenyoujustneedtoloopuntilsomeconditionissatisfied, orevenloopindefinitelyuntiltheapplicationisstopped.Caseswherewedon’treallyhave somethingtoiterateon,andthereforetheforloopwouldbeapoorchoice.Butfearnot, forthesecasesPythonprovidesuswiththewhileloop. Thewhileloopissimilartotheforloop,inthattheybothloopandateachiterationthey executeabodyofinstructions.Whatisdifferentbetweenthemisthatthewhileloop doesn’tloopoverasequence(itcan,butyouhavetomanuallywritethelogicandit wouldn’tmakeanysense,youwouldjustwanttouseaforloop),rather,itloopsaslong asacertainconditionissatisfied.Whentheconditionisnolongersatisfied,theloopends. Asusual,let’sseeanexamplewhichwillclarifyeverythingforus.Wewanttoprintthe binaryrepresentationofapositivenumber.Inordertodoso,werepeatedlydividethe numberbytwo,collectingtheremainder,andthenproducetheinverseofthelistof remainders.Letmegiveyouasmallexampleusingnumber6,whichis110inbinary. 6/2=3(remainder:0) 3/2=1(remainder:1) 1/2=0(remainder:1) Listofremainders:0,1,1. Inverseis1,1,0,whichisalsothebinaryrepresentationof6:110 Let’swritesomecodetocalculatethebinaryrepresentationfornumber39:1001112. binary.py n=39 remainders=[] whilen>0: remainder=n%2#remainderofdivisionby2 remainders.append(remainder)#wekeeptrackofremainders n//=2#wedividenby2 #reassignthelisttoitsreversedcopyandprintit remainders=remainders[::-1] print(remainders) Intheprecedingcode,Ihighlightedtwothings:n>0,whichistheconditiontokeep looping,andremainders[::-1]whichisaniceandeasywaytogetthereversedversion ofalist(missingstartandendparameters,step=-1,producesthesamelist,fromend tostart,inreverseorder).Wecanmakethecodealittleshorter(andmorePythonic),by usingthedivmodfunction,whichiscalledwithanumberandadivisor,andreturnsatuple withtheresultoftheintegerdivisionanditsremainder.Forexample,divmod(13,5) WOW! eBook www.wowebook.org wouldreturn(2,3),andindeed5*2+3=13. binary.2.py n=39 remainders=[] whilen>0: n,remainder=divmod(n,2) remainders.append(remainder) #reassignthelisttoitsreversedcopyandprintit remainders=remainders[::-1] print(remainders) Intheprecedingcode,wehavereassignedntotheresultofthedivisionby2,andthe remainder,inonesingleline. Noticethattheconditioninawhileloopisaconditiontocontinuelooping.Ifitevaluates toTrue,thenthebodyisexecutedandthenanotherevaluationfollows,andsoon,untilthe conditionevaluatestoFalse.Whenthathappens,theloopisexitedimmediatelywithout executingitsbody. Note IftheconditionneverevaluatestoFalse,theloopbecomesasocalledinfiniteloop. Infiniteloopsareusedforexamplewhenpollingfromnetworkdevices:youaskthesocket ifthereisanydata,youdosomethingwithitifthereisany,thenyousleepforasmall amountoftime,andthenyouaskthesocketagain,overandoveragain,withoutever stopping. Havingtheabilitytoloopoveracondition,ortoloopindefinitely,isthereasonwhythe forloopaloneisnotenough,andthereforePythonprovidesthewhileloop. Tip Bytheway,ifyouneedthebinaryrepresentationofanumber,checkoutthebinfunction. Justforfun,let’sadaptoneoftheexamples(multiple.sequences.py)usingthewhile logic. multiple.sequences.while.py people=['Jonas','Julio','Mike','Mez'] ages=[25,30,31,39] position=0 whileposition<len(people): person=people[position] age=ages[position] print(person,age) position+=1 Intheprecedingcode,Ihavehighlightedtheinitialization,condition,andupdateofthe variableposition,whichmakesitpossibletosimulatetheequivalentforloopcodeby handlingtheiterationvariablemanually.Everythingthatcanbedonewithaforloopcan alsobedonewithawhileloop,eventhoughyoucanseethere’sabitofboilerplateyou WOW! eBook www.wowebook.org havetogothroughinordertoachievethesameresult.Theoppositeisalsotrue,but simulatinganeverendingwhileloopusingaforlooprequiressomerealtrickery,sowhy wouldyoudothat?Usetherighttoolforthejob,and99.9%ofthetimesyou’llbefine. So,torecap,useaforloopwhenyouneedtoiterateoverone(oracombinationof) iterable,andawhileloopwhenyouneedtoloopaccordingtoaconditionbeingsatisfied ornot.Ifyoukeepinmindthedifferencebetweenthetwopurposes,youwillnever choosethewrongloopingconstruct. Let’snowseehowtoalterthenormalflowofaloop. WOW! eBook www.wowebook.org Thebreakandcontinuestatements Accordingtothetaskathand,sometimesyouwillneedtoaltertheregularflowofaloop. Youcaneitherskipasingleiteration(asmanytimesyouwant),oryoucanbreakoutof theloopentirely.Acommonusecaseforskippingiterationsisforexamplewhenyou’re iteratingoveralistofitemsandyouneedtoworkoneachofthemonlyifsomecondition isverified.Ontheotherhand,ifyou’reiteratingoveracollectionofitems,andyouhave foundoneofthemthatsatisfiessomeneedyouhave,youmaydecidenottocontinuethe loopentirelyandthereforebreakoutofit.Therearecountlesspossiblescenarios,soit’s bettertoseeacoupleofexamples. Let’ssayyouwanttoapplya20%discounttoallproductsinabasketlistforthosewhich haveanexpirationdateoftoday.Thewayyouachievethisistousethecontinue statement,whichtellstheloopingconstruct(fororwhile)toimmediatelystopexecution ofthebodyandgotothenextiteration,ifany.Thisexamplewilltakeusalittledeeper downtherabbitwhole,sobereadytojump. discount.py fromdatetimeimportdate,timedelta today=date.today() tomorrow=today+timedelta(days=1)#today+1dayistomorrow products=[ {'sku':'1','expiration_date':today,'price':100.0}, {'sku':'2','expiration_date':tomorrow,'price':50}, {'sku':'3','expiration_date':today,'price':20}, ] forproductinproducts: ifproduct['expiration_date']!=today: continue product['price']*=0.8#equivalenttoapplying20%discount print( 'Priceforsku',product['sku'], 'isnow',product['price']) Youseewestartbyimportingthedateandtimedeltaobjects,thenwesetupour products.Thosewithsku1and3haveanexpirationdateoftoday,whichmeanswewant toapply20%discountonthem.Weloopovereachproductandweinspecttheexpiration date.Ifitisnot(inequalityoperator,!=)today,wedon’twanttoexecutetherestofthe bodysuite,sowecontinue. Noticethatisnotimportantwhereinthebodysuiteyouplacethecontinuestatement (youcanevenuseitmorethanonce).Whenyoureachit,executionstopsandgoesbackto thenextiteration.Ifwerunthediscount.pymodule,thisistheoutput: $pythondiscount.py Priceforsku1isnow80.0 Priceforsku3isnow16.0 Whichshowsyouthatthelasttwolinesofthebodyhaven’tbeenexecutedforskunumber 2. WOW! eBook www.wowebook.org Let’snowseeanexampleofbreakingoutofaloop.Saywewanttotellifatleastanyof theelementsinalistevaluatestoTruewhenfedtotheboolfunction.Giventhatweneed toknowifthereisatleastone,whenwefinditwedon’tneedtokeepscanningthelistany further.InPythoncode,thistranslatestousingthebreakstatement.Let’swritethisdown intocode: any.py items=[0,None,0.0,True,0,7]#Trueand7evaluatetoTrue found=False#thisiscalled"flag" foriteminitems: print('scanningitem',item) ifitem: found=True#weupdatetheflag break iffound:#weinspecttheflag print('AtleastoneitemevaluatestoTrue') else: print('AllitemsevaluatetoFalse') Theprecedingcodeissuchacommonpatterninprogramming,youwillseeitalot.When youinspectitemsthisway,basicallywhatyoudoistosetupaflagvariable,thenstartthe inspection.Ifyoufindoneelementthatmatchesyourcriteria(inthisexample,that evaluatestoTrue),thenyouupdatetheflagandstopiterating.Afteriteration,youinspect theflagandtakeactionaccordingly.Executionyields: $pythonany.py scanningitem0 scanningitemNone scanningitem0.0 scanningitemTrue AtleastoneitemevaluatestoTrue SeehowexecutionstoppedafterTruewasfound? Thebreakstatementactsexactlylikethecontinueone,inthatitstopsexecutingthebody oftheloopimmediately,butalso,preventsanyotheriterationtorun,effectivelybreaking outoftheloop. Thecontinueandbreakstatementscanbeusedtogetherwithnolimitationintheir number,bothintheforandwhileloopingconstructs. Tip Bytheway,thereisnoneedtowritecodetodetectifthereisatleastoneelementina sequencethatevaluatestoTrue.Justcheckouttheanybuilt-infunction. WOW! eBook www.wowebook.org Aspecialelseclause OneofthefeaturesI’veseenonlyinthePythonlanguageistheabilitytohaveelse clausesafterwhileandforloops.It’sveryrarelyused,butit’sdefinitelynicetohave.In short,youcanhaveanelsesuiteafterafororwhileloop.Iftheloopendsnormally, becauseofexhaustionoftheiterator(forloop)orbecausetheconditionisfinallynotmet (whileloop),thentheelsesuite(ifpresent)isexecuted.Incaseexecutionisinterrupted byabreakstatement,theelseclauseisnotexecuted.Let’stakeanexampleofaforloop thatiteratesoveragroupofitems,lookingforonethatwouldmatchsomecondition.In casewedon’tfindatleastonethatsatisfiesthecondition,wewanttoraiseanexception. Thismeanswewanttoarresttheregularexecutionoftheprogramandsignalthatthere wasanerror,orexception,thatwecannotdealwith.Exceptionswillbethesubjectof Chapter7,Testing,Profiling,andDealingwithExceptions,sodon’tworryifyoudon’t fullyunderstandthemnow.Justbearinmindthattheywillaltertheregularflowofthe code.Letmenowshowyoutwoexamplesthatdoexactlythesamething,butoneofthem isusingthespecialfor…elsesyntax.Saythatwewanttofindamongacollectionof peopleonethatcoulddriveacar. for.no.else.py classDriverException(Exception): pass people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)] driver=None forperson,ageinpeople: ifage>=18: driver=(person,age) break ifdriverisNone: raiseDriverException('Drivernotfound.') Noticetheflagpatternagain.WesetdrivertobeNone,thenifwefindoneweupdatethe driverflag,andthen,attheendoftheloop,weinspectittoseeifonewasfound.Ikind ofhavethefeelingthatthosekidswoulddriveaverymetalliccar,butanyway,noticethat ifadriverisnotfound,aDriverExceptionisraised,signalingtheprogramthatexecution cannotcontinue(we’relackingthedriver). Thesamefunctionalitycanberewrittenabitmoreelegantlyusingthefollowingcode: for.else.py classDriverException(Exception): pass people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)] forperson,ageinpeople: ifage>=18: driver=(person,age) break else: WOW! eBook www.wowebook.org raiseDriverException('Drivernotfound.') Noticethatwearen’tforcedtousetheflagpatternanymore.Theexceptionisraisedas partoftheforlooplogic,whichmakesgoodsensebecausetheforloopischeckingon somecondition.Allweneedistosetupadriverobjectincasewefindone,becausethe restofthecodeisgoingtousethatinformationsomewhere.Noticethecodeisshorterand moreelegant,becausethelogicisnowcorrectlygroupedtogetherwhereitbelongs. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Puttingthisalltogether Nowthatyouhaveseenallthereistoseeaboutconditionalsandloops,it’stimetospice thingsupalittle,andseethosetwoexamplesIanticipatedatthebeginningofthischapter. We’llmixandmatchhere,soyoucanseehowonecanusealltheseconceptstogether. Let’sstartbywritingsomecodetogeneratealistofprimenumbersuptosomelimit. PleasebearinmindthatI’mgoingtowriteaveryinefficientandrudimentaryalgorithmto detectprimes.Theimportantthingforyouistoconcentrateonthosebitsinthecodethat belongtothischapter’ssubject. WOW! eBook www.wowebook.org Example1–aprimegenerator AccordingtoWikipedia: “Aprimenumber(oraprime)isanaturalnumbergreaterthan1thathasnopositive divisorsotherthan1anditself.Anaturalnumbergreaterthan1thatisnotaprime numberiscalledacompositenumber.” Basedonthisdefinition,ifweconsiderthefirst10naturalnumbers,wecanseethat2,3, 5,and7areprimes,while1,4,6,8,9,10arenot.Inordertohaveacomputertellyouifa numberNisprime,youcandividethatnumberbyallnaturalnumbersintherange[2,N). Ifanyofthosedivisionsyieldszeroasaremainder,thenthenumberisnotaprime. Enoughchatter,let’sgetdowntobusiness.I’llwritetwoversionsofthis,thesecondof whichwillexploitthefor…elsesyntax. primes.py primes=[]#thiswillcontaintheprimesintheend upto=100#thelimit,inclusive forninrange(2,upto+1): is_prime=True#flag,newateachiterationofouterfor fordivisorinrange(2,n): ifn%divisor==0: is_prime=False break ifis_prime:#checkonflag primes.append(n) print(primes) Lotsofthingstonoticeintheprecedingcode.Firstofallwesetupanemptylistprimes, whichwillcontaintheprimesattheend.Thelimitis100,andyoucanseeit’sinclusivein thewaywecallrange()intheouterloop.Ifwewroterange(2,upto)thatwouldbe[2, upto),right?Thereforerange(2,upto+1)givesus[2,upto+1)==[2,upto]. So,twoforloops.Intheouteroneweloopoverthecandidateprimes,thatis,allnatural numbersfrom2toupto.Insideeachiterationofthisouterloopwesetupaflag(whichis settoTrueateachiteration),andthenstartdividingthecurrentnbyallnumbersfrom2to n–1.Ifwefindaproperdivisorforn,itmeansniscomposite,andthereforewesetthe flagtoFalseandbreaktheloop.Noticethatwhenwebreaktheinnerone,theouterone keepsongoingnormally.Thereasonwhywebreakafterhavingfoundaproperdivisorfor nisthatwedon’tneedanyfurtherinformationtobeabletotellthatnisnotaprime. Whenwecheckontheis_primeflag,ifitisstillTrue,itmeanswecouldn’tfindany numberin[2,n)thatisaproperdivisorforn,thereforenisaprime.Weappendntothe primeslist,andhop!Anotheriteration,untilnequals100. Runningthiscodeyields: $pythonprimes.py [2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67, 71,73,79,83,89,97] WOW! eBook www.wowebook.org Beforeweproceed,onequestion:ofalliterationsoftheouterloop,oneofthemis differentthanalltheothers.Couldyoutellwhichone,andwhy?Thinkaboutitfora second,gobacktothecodeandtrytofigureitoutforyourself,andthenkeepreadingon. Didyoufigureitout?Ifnot,don’tfeelbad,it’sperfectlynormal.Iaskedyoutodoitasa smallexercisebecauseit’swhatcodersdoallthetime.Theskilltounderstandwhatthe codedoesbysimplylookingatitissomethingyoubuildovertime.It’sveryimportant,so trytoexerciseitwheneveryoucan.I’lltellyoutheanswernow:theiterationthatbehaves differentlyfromallothersisthefirstone.Thereasonisbecauseinthefirstiteration,nis 2.Thereforetheinnermostforloopwon’tevenrun,becauseit’saforloopwhichiterates overrange(2,2),andwhatisthatifnot[2,2)?Tryitoutforyourself,writeasimplefor loopwiththatiterable,putaprintinthebodysuite,andseeifanythinghappens(it won’t…). Now,fromanalgorithmicpointofviewthiscodeisinefficientsolet’satleastmakeit morebeautiful: primes.else.py primes=[] upto=100 forninrange(2,upto+1): fordivisorinrange(2,n): ifn%divisor==0: break else: primes.append(n) print(primes) Muchnicer,right?Theis_primeflagiscompletelygone,andweappendntotheprimes listwhenweknowtheinnerforloophasn’tencounteredanybreakstatements.Seehow thecodelookscleanerandreadsbetter? WOW! eBook www.wowebook.org Example2–applyingdiscounts Inthisexample,IwanttoshowyouatechniqueIlikealot.Inmanyprogramming languages,otherthantheif/elif/elseconstructs,inwhateverformorsyntaxtheymay come,youcanfindanotherstatement,usuallycalledswitch/case,thatinPythonis missing.Itistheequivalentofacascadeofif/elif/…/elif/elseclauses,withasyntax similartothis(warning!JavaScriptcode!): switch.js switch(day_number){ case1: case2: case3: case4: case5: day="Weekday"; break; case6: day="Saturday"; break; case0: day="Sunday"; break; default: day=""; alert(day_number+'isnotavaliddaynumber.') } Intheprecedingcode,weswitchonavariablecalledday_number.Thismeanswegetits valueandthenwedecidewhatcaseitfitsin(ifany).From1to5thereisacascade,which meansnomatterthenumber,[1,5]allgodowntothebitoflogicthatsetsdayas "Weekday".Thenwehavesinglecasesfor0and6andadefaultcasetopreventerrors, whichalertsthesystemthatday_numberisnotavaliddaynumber,thatis,notin[0,6]. Pythonisperfectlycapableofrealizingsuchlogicusingif/elif/elsestatements: switch.py if1<=day_number<=5: day='Weekday' elifday_number==6: day='Saturday' elifday_number==0: day='Sunday' else: day='' raiseValueError( str(day_number)+'isnotavaliddaynumber.') Intheprecedingcode,wereproducethesamelogicoftheJavaScriptsnippet,inPython, usingif/elif/elsestatements.IraisedValueErrorexceptionjustasanexampleatthe end,ifday_numberisnotin[0,6].Thisisonepossiblewayoftranslatingtheswitch/case logic,butthereisalsoanotherone,sometimescalleddispatching,whichIwillshowyou WOW! eBook www.wowebook.org inthelastversionofthenextexample. Tip Bytheway,didyounoticethefirstlineoftheprevioussnippet?Haveyounoticedthat Pythoncanmakedouble(actually,evenmultiple)comparisons?It’sjustwonderful! Let’sstartthenewexamplebysimplywritingsomecodethatassignsadiscountto customersbasedontheircouponvalue.I’llkeepthelogicdowntoaminimumhere, rememberthatallwereallycareaboutisconditionalsandloops. coupons.py customers=[ dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20 dict(id=2,total=150,coupon_code='P30'),#P30:percent,30% dict(id=3,total=100,coupon_code='P50'),#P50:percent,50% dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15 ] forcustomerincustomers: code=customer['coupon_code'] ifcode=='F20': customer['discount']=20.0 elifcode=='F15': customer['discount']=15.0 elifcode=='P30': customer['discount']=customer['total']*0.3 elifcode=='P50': customer['discount']=customer['total']*0.5 else: customer['discount']=0.0 forcustomerincustomers: print(customer['id'],customer['total'],customer['discount']) Westartbysettingupsomecustomers.Theyhaveanordertotal,acouponcode,andanid. Imadeupfourdifferenttypesofcoupon,twoarefixedandtwoarepercentagebased.You canseethatintheif/elif/elsecascadeIapplythediscountaccordingly,andIsetitasa 'discount'keyinthecustomerdict. AttheendIjustprintoutpartofthedatatoseeifmycodeisworkingproperly. $pythoncoupons.py 120020.0 215045.0 310050.0 411015.0 Thiscodeissimpletounderstand,butallthoseclausesarekindofclutteringthelogic.It’s noteasytoseewhat’sgoingonatafirstglance,andIdon’tlikeit.Incaseslikethis,you canexploitadictionarytoyouradvantage,likethis: coupons.dict.py customers=[ dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20 WOW! eBook www.wowebook.org dict(id=2,total=150,coupon_code='P30'),#P30:percent,30% dict(id=3,total=100,coupon_code='P50'),#P50:percent,50% dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15 ] discounts={ 'F20':(0.0,20.0),#eachvalueis(percent,fixed) 'P30':(0.3,0.0), 'P50':(0.5,0.0), 'F15':(0.0,15.0), } forcustomerincustomers: code=customer['coupon_code'] percent,fixed=discounts.get(code,(0.0,0.0)) customer['discount']=percent*customer['total']+fixed forcustomerincustomers: print(customer['id'],customer['total'],customer['discount']) Runningtheprecedingcodeyieldsexactlythesameresultwehadfromthesnippetbefore it.Wesparedtwolines,butmoreimportantly,wegainedalotinreadability,asthebodyof theforloopnowisjustthreelineslong,andveryeasytounderstand.Theconcepthereis touseadictionaryasdispatcher.Inotherwords,wetrytofetchsomethingfromthe dictionarybasedonacode(ourcoupon_code),andbyusingdict.get(key,default), wemakesurewealsocaterforwhenthecodeisnotinthedictionaryandweneeda defaultvalue. NoticethatIhadtoapplysomeverysimplelinearalgebrainordertocalculatethe discountproperly.Eachdiscounthasapercentageandfixedpartinthedictionary, representedbya2-tuple.Byapplyingpercent*total+fixed,wegetthecorrect discount.Whenpercentis0,theformulajustgivesthefixedamount,anditgives percent*totalwhenfixedis0.Simplebuteffective. Thistechniqueisimportantbecauseitisalsousedinothercontexts,withfunctions,where itactuallybecomesmuchmorepowerfulthanwhatwe’veseenintheprecedingsnippet.If it’snotcompletelycleartoyouhowitworks,Isuggestyoutotakeyourtimeand experimentwithit.Changevaluesandaddprintstatementstoseewhat’sgoingonwhile theprogramisrunning. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Aquickpeekattheitertoolsmodule Achapteraboutiterables,iterators,conditionallogic,andloopingwouldn’tbecomplete withoutspendingafewwordsabouttheitertoolsmodule.Ifyouareintoiterating,this isakindofheaven. AccordingtothePythonofficialdocumentation,theitertoolsmoduleis: “Amodulewhichimplementsanumberofiteratorbuildingblocksinspiredby constructsfromAPL,Haskell,andSML.Eachhasbeenrecastinaformsuitablefor Python.Themodulestandardizesacoresetoffast,memoryefficienttoolsthatare usefulbythemselvesorincombination.Together,theyforman“iteratoralgebra” makingitpossibletoconstructspecializedtoolssuccinctlyandefficientlyinpure Python.” BynomeansdoIhavetheroomheretoshowyouallthegoodiesyoucanfindinthis module,soIencourageyoutogoandcheckitoutforyourself,Ipromiseyou’llenjoyit. Inanutshell,itprovidesyouwiththreebroadcategoriesofiterators.Iwillgiveyouavery smallexampleofoneiteratortakenfromeachoneofthem,justtomakeyourmouthwater alittle. WOW! eBook www.wowebook.org Infiniteiterators Infiniteiteratorsallowyoutoworkwithaforloopinadifferentfashion,likeifitwasa whileloop. infinite.py fromitertoolsimportcount fornincount(5,3): ifn>20: break print(n,end=',')#insteadofnewline,commaandspace Runningthecodegivesthis: $pythoninfinite.py 5,8,11,14,17,20, Thecountfactoryclassmakesaniteratorthatjustgoesonandoncounting.Itstartsfrom 5andkeepsadding3toit.Weneedtomanuallybreakitifwedon’twanttogetstuckin aninfiniteloop. WOW! eBook www.wowebook.org Iteratorsterminatingontheshortestinput sequence Thiscategoryisveryinteresting.Itallowsyoutocreateaniteratorbasedonmultiple iterators,combiningtheirvaluesaccordingtosomelogic.Thekeypointhereisthat amongthoseiterators,incaseanyofthemareshorterthantherest,theresultingiterator won’tbreak,itwillsimplystopassoonastheshortestiteratorisexhausted.Thisisvery theoretical,Iknow,soletmegiveyouanexampleusingcompress.Thisiteratorgivesyou backthedataaccordingtoacorrespondingiteminaselectorbeingTrueorFalse: compress('ABC',(1,0,1))wouldgiveback'A'and'C',becausetheycorrespondto the1's.Let’sseeasimpleexample: compress.py fromitertoolsimportcompress data=range(10) even_selector=[1,0]*10 odd_selector=[0,1]*10 even_numbers=list(compress(data,even_selector)) odd_numbers=list(compress(data,odd_selector)) print(odd_selector) print(list(data)) print(even_numbers) print(odd_numbers) Noticethatodd_selectorandeven_selectorare20elementslong,whiledataisjust10 elementslong.compresswillstopassoonasdatahasyieldeditslastelement.Running thiscodeproducesthefollowing: $pythoncompress.py [0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1] [0,1,2,3,4,5,6,7,8,9] [0,2,4,6,8] [1,3,5,7,9] It’saveryfastandnicewayofselectingelementsoutofaniterable.Thecodeisvery simple,justnoticethatinsteadofusingaforlooptoiterateovereachvaluethatisgiven backbythecompresscalls,weusedlist(),whichdoesthesame,butinsteadof executingabodyofinstructions,putsallthevaluesintoalistandreturnsit. WOW! eBook www.wowebook.org Combinatoricgenerators Lastbutnotleast,combinatoricgenerators.Thesearereallyfun,ifyouareintothiskind ofthing.Let’sjustseeasimpleexampleonpermutations. AccordingtoWolframMathworld: “Apermutation,alsocalledan“arrangementnumber”or“order”,isa rearrangementoftheelementsofanorderedlistSintoaone-to-onecorrespondence withSitself.” Forexample,thepermutationsofABCare6:ABC,ACB,BAC,BCA,CAB,andCBA. IfasethasNelements,thenthenumberofpermutationsofthemisN!(Nfactorial).For thestringABCthepermutationsare3!=3*2*1=6.Let’sdoitinPython: permutations.py fromitertoolsimportpermutations print(list(permutations('ABC'))) Thisveryshortsnippetofcodeproducesthefollowingresult: $pythonpermutations.py [('A','B','C'),('A','C','B'),('B','A','C'),('B','C','A'),('C', 'A','B'),('C','B','A')] Beverycarefulwhenyouplaywithpermutation.Theirnumbergrowsataratethatis proportionaltothefactorialofthenumberoftheelementsyou’repermuting,andthat numbercangetreallybig,reallyfast. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,we’vetakenanotherstepforwardtoexpandourcodingvocabulary.We’ve seenhowtodrivetheexecutionofthecodebyevaluatingconditions,andwe’veseenhow toloopanditerateoversequencesandcollectionsofobjects.Thisgivesusthepowerto controlwhathappenswhenourcodeisrun,whichmeanswearegettinganideaonhowto shapeitsothatitdoeswhatwewantanditreactstodatathatchangesdynamically. We’vealsoseenhowtocombineeverythingtogetherinacoupleofsimpleexamples,and intheendwehavetakenabrieflookattheitertoolsmodule,whichisfullofinteresting iteratorswhichcanenrichourabilitieswithPythonevenmore. Nowit’stimetoswitchgears,totakeanotherstepforwardandtalkaboutfunctions.The nextchapterisallaboutthembecausetheyareextremelyimportant.Makesureyou’re comfortablewithwhathasbeendoneuptonow:Iwanttoprovideyouwithinteresting examples,soI’llhavetogoalittlefaster.Ready?Turnthepage. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter4.Functions,theBuildingBlocks ofCode “Tocreatearchitectureistoputinorder.Putwhatinorder?Functionandobjects.” —LeCorbusier Inthischapter,we’regoingtoexplorefunctions.Wealreadysaidthateverythingisan objectinPython,andfunctionsarenoexceptiontothis.But,whatexactlyisafunction?A functionisasequenceofinstructionsthatperformatask,bundledasaunit.Thisunitcan thenbeimportedandusedwhereverit’sneeded.Therearemanyadvantagestousing functionsinyourcode,aswe’llseeshortly. Ibelievethesaying,apictureisworthonethousandwords,isparticularlytruewhen explainingfunctionstosomeonewhoisnewtothisconcept,sopleasetakealookatthe followingimage: Asyoucansee,afunctionisablockofinstructions,packagedasawhole,likeabox. Functionscanacceptinputargumentsandproduceoutputvalues.Bothoftheseare optional,aswe’llseeintheexamplesinthischapter. AfunctioninPythonisdefinedbyusingthedefkeyword,afterwhichthenameofthe functionfollows,terminatedbyapairofbraces(whichmayormaynotcontaininput parameters)and,finally,acolon(:)signalstheendofthefunctiondefinitionline. Immediatelyafterwards,indentedbyfourspaces,wefindthebodyofthefunction,which isthesetofinstructionsthatthefunctionwillexecutewhencalled. Note Notethattheindentationbyfourspacesisnotmandatory,butitistheamountofspaces suggestedbyPEP8,and,inpractice,itisthemostwidelyusedspacingmeasure. Afunctionmayormaynotreturnoutput.Ifafunctionwantstoreturnoutput,itdoessoby usingthereturnkeyword,followedbythedesiredoutput.Ifyouhaveaneagleeye,you mayhavenoticedthelittle*afterOptionalintheoutputsectionoftheprecedingpicture. ThisisbecauseafunctionalwaysreturnssomethinginPython,evenifyoudon’texplicitly usethereturnclause.Ifthefunctionhasnoreturnstatementinitsbody,it’sreturnvalue WOW! eBook www.wowebook.org isNone.Thereasonsbehindthisdesignchoiceareoutofthescopeofanintroductory chapter,soallyouneedtoknowisthatthisbehaviorwillmakeyourlifeeasier,asalways, thankyouPython. WOW! eBook www.wowebook.org Whyusefunctions? Functionsareamongthemostimportantconceptsandconstructsofanylanguage,solet megiveyouafewreasonswhyweneedthem: Theyreducecodeduplicationinaprogram.Byhavingaspecifictasktakencareof byaniceblockofpackagedcodethatwecanimportandcallwheneverwewant,we don’tneedtoduplicateitsimplementation. Theyhelpinsplittingacomplextaskorprocedureintosmallerblocks,eachofwhich becomesafunction. Theyhidetheimplementationdetailsfromtheirusers. Theyimprovetraceability. Theyimprovereadability. Let’slookatafewexamplestogetabetterunderstandingofeachpoint. WOW! eBook www.wowebook.org Reducecodeduplication Imaginethatyouarewritingapieceofscientificsoftware,andyouneedtocalculate primesuptoalimit,aswedidinthepreviouschapter.Youwriteseveralalgorithmsand primenumbers,beingthebasisofmanydifferenttypesofcalculations,keepcreepinginto yourcode.Well,youhaveanicealgorithmtocalculatethem,soyoucopyandpasteitto whereveryouneed.Oneday,though,yourfriendMisterSmartygivesyouabetter algorithmtocalculateprimenumbers,andthiswillsaveyoualotoftime.Atthispoint, youneedtogooveryourwholecodebaseandreplacetheoldcodewiththenewcode. Thisisactuallyaverybadwaytogoaboutit.It’serror-prone,youneverknowwhatlines youarechoppingoutorleavingtherebymistakewhenyoucutandpastecodeinother code,andyoumayalsoriskmissingoneoftheplaceswhereprimecalculationwasdone, leavingyoursoftwarewithdifferentversions.Canyouimagineifyoudiscoveredthatthe oldwaywasbuggy?Youwouldhaveanundetectedbuginyourcode,andbugslikethis arequitehardtospot,especiallyinbigcodebases. So,whatshouldyoudo?Simple!Youwriteafunction,get_prime_numbers(upto),and useitanywhereyouneedalistofprimes.WhenMisterSmartycomestoyouandgives youthenewcode,allyouhavetodoisreplacethebodyofthatfunctionwiththenew implementation,andyou’redone!Therestofthesoftwarewillautomaticallyadapt,since it’sjustcallingthefunction. Yourcodewillbeshorter,itwillnotsufferfrominconsistenciesbetweenoldandnew waysofperformingatask,orundetectedbugsduetocopyandpastefailuresoroversights. Usefunctions,andyou’llonlygainfromit,Ipromise. WOW! eBook www.wowebook.org Splittingacomplextask Functionsareveryusefulalsotosplitalongorcomplextaskintosmallerpieces.Theend resultisthatthecodebenefitsfromitinseveralways,forexample,readability,testability, andreuse.Togiveyouasimpleexample,imaginethatyou’repreparingareport.Your codeneedstofetchdatafromadatasource,parseit,filterit,polishit,andthenawhole seriesofalgorithmsneedstoberunagainstit,inordertoproducetheresultswhichwill feedtheReportclass.It’snotuncommontoreadprocedureslikethisthatarejustonebig functiondo_report(data_source).Therearetensorhundredsoflinesofcodewhichend withreturnreport. Situationslikethisarecommonincodeproducedbyscientists.Theyhavebrilliantminds andtheycareaboutthecorrectnessoftheendresultbut,unfortunately,sometimesthey havenotraininginprogrammingtheory.Itisnottheirfault,onecannotknoweverything. Now,pictureinyourheadsomethinglikeafewhundredlinesofcode.It’sveryhardto followthrough,tofindtheplaceswherethingsarechangingcontext(likefinishingone taskandstartingthenextone).Doyouhavethepictureinyourmind?Good.Don’tdoit! Instead,lookatthiscode: data.science.example.py defdo_report(data_source): #fetchandpreparedata data=fetch_data(data_source) parsed_data=parse_data(data) filtered_data=filter_data(parsed_data) polished_data=polish_data(filtered_data) #runalgorithmsondata final_data=analyse(polished_data) #createandreturnreport report=Report(final_data) returnreport Thepreviousexampleisfictitious,ofcourse,butcanyouseehoweasyitwouldbetogo throughthecode?Iftheendresultlookswrong,itwouldbeveryeasytodebugeachofthe singledataoutputsinthedo_reportfunction.Moreover,it’seveneasiertoexcludepartof theprocesstemporarilyfromthewholeprocedure(youjustneedtocommentouttheparts youneedtosuspend).Codelikethisiseasiertodealwith. WOW! eBook www.wowebook.org Hideimplementationdetails Let’sstaywiththeprecedingexampletotalkaboutthispointaswell.Youcanseethat,by goingthroughthecodeofthedo_reportfunction,youcangetaprettygood understandingwithoutreadingonesinglelineofimplementation.Thisisbecause functionshidetheimplementationdetails.Thisfeaturemeansthat,ifyoudon’tneedto delveintodetails,youarenotforcedto,inthewayyouwouldifdo_reportwasjustone bigfatfunction.Inordertounderstandwhatwasgoingon,youwouldhavetoreadthe implementationdetails.Youdon’tneedtowithfunctions.Thisreducesthetimeyouspend readingthecodeandsince,inaprofessionalenvironment,readingcodetakesmuchmore timethanactuallywritingit,it’sveryimportanttoreduceitasmuchaswecan. WOW! eBook www.wowebook.org Improvereadability Coderssometimesdon’tseethepointinwritingafunctionwithabodyofoneortwolines ofcode,solet’slookatanexamplethatshowsyouwhyyoushoulddoit. Imaginethatyouneedtomultiplytwomatrices: Wouldyouprefertohavetoreadthiscode: matrix.multiplication.nofunc.py a=[[1,2],[3,4]] b=[[5,1],[2,1]] c=[[sum(i*jfori,jinzip(r,c))forcinzip(*b)] forrina] Orwouldyoupreferthisone: matrix.multiplication.func.py #thisfunctioncouldalsobedefinedinanothermodule defmatrix_mul(a,b): return[[sum(i*jfori,jinzip(r,c))forcinzip(*b)] forrina] a=[[1,2],[3,4]] b=[[5,1],[2,1]] c=matrix_mul(a,b) It’smucheasiertounderstandthatcistheresultofthemultiplicationbetweenaandbin thesecondexample.It’smucheasiertoreadthroughthecodeand,ifyoudon’tneedto modifythatpart,youdon’tevenneedtogointotheimplementationdetails. Therefore,readabilityisimprovedherewhile,inthefirstsnippet,youwouldhaveto spendtimetryingtounderstandwhatthatcomplicatedlistcomprehensionwasdoing. Note Don’tworryifyoudon’tunderstandlistcomprehensions,we’llstudytheminthenext chapter. WOW! eBook www.wowebook.org Improvetraceability Imaginethatyouhavewrittenane-commercewebsite.Youhavedisplayedtheproduct pricesalloverthepages.ImaginethatthepricesinyourdatabasearestoredwithnoVAT, butyouwanttodisplaythemonthewebsitewithVATat20%.Here’safewwaysof calculatingtheVAT-inclusivepricefromtheVAT-exclusiveprice. vat.py price=100#GBP,noVAT final_price1=price*1.2 final_price2=price+price/5.0 final_price3=price*(100+20)/100.0 final_price4=price+price*0.2 AllthesefourdifferentwaysofcalculatingaVAT-inclusivepriceareperfectlyacceptable, andIpromiseyouIhavefoundthemallinmycolleagues’code,overtheyears.Now, imaginethatyouhavestartedsellingyourproductsindifferentcountriesandsomeof themhavedifferentVATratessoyouneedtorefactoryourcode(throughoutthewebsite) inordertomakethatVATcalculationdynamic. HowdoyoutracealltheplacesinwhichyouareperformingaVATcalculation?Coding todayisacollaborativetaskandyoucannotbesuretheVAThasbeencalculatedusing onlyoneofthoseforms.It’sgoingtobehell,believeme. So,let’swriteafunctionthattakestheinputvalues,vatandprice(VAT-exclusive),and returnsaVAT-inclusiveprice. vat.function.py defcalculate_price_with_vat(price,vat): returnprice*(100+vat)/100 Nowyoucanimportthatfunctionandapplyitinanyplaceofyourwebsitewhereyou needtocalculateaVAT-inclusivepriceandwhenyouneedtotracethosecalls,youcan searchforcalculate_price_with_vat. Note Notethat,intheprecedingexample,priceisassumedtobeVAT-exclusive,andvathasa percentagevalue(forexample,19,20,23,andsoon). WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Scopesandnameresolution Doyourememberwhenwetalkedaboutscopesandnamespacesinthefirstchapter? We’regoingtoexpandonthatconceptnow.Finally,wecantalkaboutfunctionsandthis willmakeeverythingeasiertounderstand.Let’sstartwithaverysimpleexample. scoping.level.1.py defmy_function(): test=1#thisisdefinedinthelocalscopeofthefunction print('my_function:',test) test=0#thisisdefinedintheglobalscope my_function() print('global:',test) Ihavedefinedthenametestintwodifferentplacesinthepreviousexample.Itisactually intwodifferentscopes.Oneistheglobalscope(test=0),andtheotheristhelocal scopeofthefunctionmy_function(test=1).Ifyouexecutethecode,you’llseethis: $pythonscoping.level.1.py my_function:1 global:0 It’sclearthattest=1shadowstheassignmenttest=0inmy_function.Intheglobal context,testisstill0,asyoucanseefromtheoutputoftheprogrambutwedefinethe nametestagaininthefunctionbody,andwesetittopointtoanintegerofvalue1.Both thetwotestnamesthereforeexist,oneintheglobalscope,pointingtoanintobjectwith value0,theotherinthemy_functionscope,pointingtoanintobjectwithvalue1.Let’s commentoutthelinewithtest=1.Pythongoesandsearchesforthenametestinthe nextenclosingnamespace(recalltheLEGBrule:Local,Enclosing,Global,Built-in describedinChapter1,IntroductionandFirstSteps–TakeaDeepBreath)and,inthis case,wewillseethevalue0printedtwice.Tryitinyourcode. Now,let’sraisethestakeshereandlevelup: scoping.level.2.py defouter(): test=1#outerscope definner(): test=2#innerscope print('inner:',test) inner() print('outer:',test) test=0#globalscope outer() print('global:',test) Intheprecedingcode,wehavetwolevelsofshadowing.Onelevelisinthefunction outer,andtheotheroneisinthefunctioninner.Itisfarfromrocketscience,butitcan betricky.Ifwerunthecode,weget: WOW! eBook www.wowebook.org $pythonscoping.level.2.py inner:2 outer:1 global:0 Trycommentingoutthelinetest=1.Whatdoyouthinktheresultwillbe?Well,when reachingthelineprint('outer:',test),Pythonwillhavetolookfortestinthenext enclosingscope,thereforeitwillfindandprint0,insteadof1.Makesureyoucomment outtest=2aswell,toseeifyouunderstandwhathappens,andiftheLEGBruleis clear,beforeproceeding. AnotherthingtonoteisthatPythongivesyoutheabilitytodefineafunctioninanother function.Theinnerfunction’snameisdefinedwithinthenamespaceoftheouterfunction, exactlyaswouldhappenwithanyothername. WOW! eBook www.wowebook.org Theglobalandnonlocalstatements Goingbacktotheprecedingexample,wecanalterwhathappenstotheshadowingofthe testnamebyusingoneofthesetwospecialstatements:globalandnonlocal.Asyoucan seefromthepreviousexample,whenwedefinetest=2inthefunctioninner,we overwritetestneitherinthefunctionouter,norintheglobalscope.Wecangetread accesstothosenamesifweusetheminanestedscopethatdoesn’tdefinethem,butwe cannotmodifythembecause,whenwewriteanassignmentinstruction,we’reactually defininganewnameinthecurrentscope. Howdowechangethisbehavior?Well,wecanusethenonlocalstatement.Accordingto theofficialdocumentation: “Thenonlocalstatementcausesthelistedidentifierstorefertopreviouslybound variablesinthenearestenclosingscopeexcludingglobals.” Let’sintroduceitinthefunctioninner,andseewhathappens: scoping.level.2.nonlocal.py defouter(): test=1#outerscope definner(): nonlocaltest test=2#nearestenclosingscope print('inner:',test) inner() print('outer:',test) test=0#globalscope outer() print('global:',test) NoticehowinthebodyofthefunctioninnerIhavedeclaredthetestnametobe nonlocal.Runningthiscodeproducesthefollowingresult: $pythonscoping.level.2.nonlocal.py inner:2 outer:2 global:0 Wow,lookatthatresult!Itmeansthat,bydeclaringtesttobenonlocalinthefunction inner,weactuallygettobindthenametesttothatdeclaredinthefunctionouter.Ifwe removedthenonlocaltestlinefromthefunctioninnerandtriedthesametrickinthe functionouter,wewouldgetaSyntaxError,becausethenonlocalstatementworkson enclosingscopesexcludingtheglobalone. Isthereawaytogettothattest=0intheglobalnamespacethen?Ofcourse,wejust needtousetheglobalstatement.Let’stryit. scoping.level.2.global.py WOW! eBook www.wowebook.org defouter(): test=1#outerscope definner(): globaltest test=2#globalscope print('inner:',test) inner() print('outer:',test) test=0#globalscope outer() print('global:',test) Notethatwehavenowdeclaredthenametesttobeglobal,whichwillbasicallybindit totheonewedefinedintheglobalnamespace(test=0).Runthecodeandyoushould getthefollowing: $pythonscoping.level.2.global.py inner:2 outer:1 global:2 Thisshowsthatthenameaffectedbytheassignmenttest=2isnowtheglobalone. Thistrickwouldalsoworkintheouterfunctionbecause,inthiscase,we’rereferringto theglobalscope.Tryitforyourselfandseewhatchanges,getcomfortablewithscopes andnameresolution,it’sveryimportant. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Inputparameters Atthebeginningofthischapter,wesawthatafunctioncantakeinputparameters.Before wedelveintoallpossibletypeofparameters,let’smakesureyouhaveaclear understandingofwhatpassingaparametertoafunctionmeans.Therearethreekeypoints tokeepinmind: Argumentpassingisnothingmorethanassigninganobjecttoalocalvariablename Assigninganobjecttoanargumentnameinsideafunctiondoesn’taffectthecaller Changingamutableobjectargumentinafunctionaffectsthecaller Let’slookatanexampleforeachofthesepoints. WOW! eBook www.wowebook.org Argumentpassing Takealookatthefollowingcode.Wedeclareanamexintheglobalscope,thenwe declareafunctionfunc(y)andwecallit,passingx.Ihighlightedthecallinthecode. key.points.argument.passing.py x=3 deffunc(y): print(y) func(x)#prints:3 Whenfunciscalledwithx,whathappensisthatwithinitslocalscope,anameyis created,andit’spointedtothesameobjectxispointingto.Thisisbetterclarifiedbythe followingpicture: Therightpartoftheprecedingpicturedepictsthestateoftheprogramwhenexecutionhas reachedtheend,afterfunchasreturned(None).TakealookattheFramescolumn,and notethatwehavetwonames,xandfunc,intheglobalnamespace(Globalframe), pointingtoanint(withavalueofthree)andtoafunctionobject,respectively.Right belowit,intherectangletitledfunc,wecanseethefunction’slocalnamespace,inwhich onlyonenamehasbeendefined:y.Becausewehavecalledfuncwithx(line5intheleft partofthepicture),yispointingtothesameobjectthatxispointingto.Thisiswhat happensunderthehoodwhenanargumentispassedtoafunction.Ifwehadusedthe namexinsteadofyinthefunctiondefinition,thingswouldhavebeenexactlythesame (onlymaybeabitconfusingatfirst),therewouldbealocalxinthefunction,andaglobal xoutside,aswesawintheScopesandnameresolutionsection. So,inanutshell,whatreallyhappensisthatthefunctioncreatesinitslocalscopethe namesdefinedasargumentsand,whenwecallit,webasicallytellPythonwhichobjects thosenamesmustbepointedtowards. WOW! eBook www.wowebook.org Assignmenttoargumentnamesdon’taffectthe caller Thisissomethingthatcanbetrickytounderstandatfirst,solet’slookatanexample. key.points.assignment.py x=3 deffunc(x): x=7#definingalocalx,notchangingtheglobalone func(x) print(x)#prints:3 Intheprecedingcode,whenthelinex=7isexecuted,whathappensisthatwithinthe localscopeofthefunctionfunc,thenamexispointedtoanintegerwithvalue7,leaving theglobalxunaltered. WOW! eBook www.wowebook.org Changingamutableaffectsthecaller Thisisthefinalpoint,andit’sveryimportantbecausePythonapparentlybehaves differentlywithmutables(justapparentlythough).Let’slookatanexample: key.points.mutable.py x=[1,2,3] deffunc(x): x[1]=42#thisaffectsthecaller! func(x) print(x)#prints:[1,42,3] Wow,weactuallychangedtheoriginalobject!Ifyouthinkaboutit,thereisnothingweird inthisbehavior.Thenamexinthefunctionissettopointtothecallerobjectbythe functioncallandwithinthebodyofthefunction,we’renotchangingx,inthatwe’renot changingitsreference,or,inotherwords,wearenotchangingtheobjectxispointingto. Whatwe’redoingisaccessingthatobject’selementatposition1,andchangingitsvalue. Rememberpoint#2:“Assigninganobjecttoanargumentnamewithinafunctiondoesn’t affectthecaller“.Ifthatiscleartoyou,thefollowingcodeshouldnotbesurprising. key.points.mutable.assignment.py x=[1,2,3] deffunc(x): x[1]=42#thischangesthecaller! x='somethingelse'#thispointsxtoanewstringobject func(x) print(x)#stillprints:[1,42,3] TakealookatthetwolinesIhavehighlighted.Atfirst,wejustaccessthecallerobject again,atposition1,andchangeitsvaluetonumber42.Then,wereassignxtopointtothe string'somethingelse'.Thisleavesthecallerunaltered,accordingtopoint#2,and,in fact,theoutputisthesameasthatoftheprevioussnippet. Takeyourtimetoplayaroundwiththisconceptandexperimentwithprintsandcallsto theidfunctionuntileverythingisclearinyourmind.Thisisoneofthekeyaspectsof Pythonanditmustbeveryclear,otherwiseyouriskintroducingsubtlebugsintoyour code. Nowthatwehaveagoodunderstandingofinputparametersandhowtheybehave,let’s seehowwecanspecifythem. WOW! eBook www.wowebook.org Howtospecifyinputparameters Therearefivedifferentwaysofspecifyinginputparameters.Let’slookatthemoneby one. Positionalarguments Positionalargumentsarereadfromlefttorightandtheyarethemostcommontypeof arguments. arguments.positional.py deffunc(a,b,c): print(a,b,c) func(1,2,3)#prints:123 Thereisnotmuchelsetosay.Theycanbeasnumerousasyouwantandtheyareassigned byposition.Inthefunctioncall,1comesfirst,2comessecondand3comesthird, thereforetheyareassignedtoa,bandcrespectively. Keywordargumentsanddefaultvalues Keywordargumentsareassignedbykeywordusingthename=valuesyntax. arguments.keyword.py deffunc(a,b,c): print(a,b,c) func(a=1,c=2,b=3)#prints:132 Keywordargumentsactwhencallingthefunctioninsteadofrespectingtheleft-to-right positionalassignment,k.Keywordargumentsarematchedbyname,evenwhentheydon’t respectthedefinition’soriginalposition(we’llseethatthereisalimitationtothisbehavior later,whenwemixandmatchdifferenttypesofarguments). Thecounterpartofkeywordarguments,onthedefinitionside,isdefaultvalues.The syntaxisthesame,name=value,andallowsustonothavetoprovideanargumentifwe arehappywiththegivendefault. arguments.default.py deffunc(a,b=4,c=88): print(a,b,c) func(1)#prints:1488 func(b=5,a=7,c=9)#prints:759 func(42,c=9)#prints:4249 Thearetwothingstonotice,whichareveryimportant.Firstofall,youcannotspecifya defaultargumentontheleftofapositionalone.Second,notehowintheexamples,when anargumentispassedwithoutusingtheargument_name=valuesyntax,itmustbethefirst oneinthelist,,anditisalwaysassignedtoa.Tryandscramblethoseargumentsandsee whathappens.Pythonerrormessagesareverygoodattellingyouwhat’swrong.So,for example,ifyoutriedsomethinglikethis: WOW! eBook www.wowebook.org func(b=1,c=2,42)#positionalargumentafterkeywordone Youwouldgetthefollowingerror: SyntaxError:non-keywordargafterkeywordarg Thisinformsyouthatyou’vecalledthefunctionincorrectly. Variablepositionalarguments Sometimesyoumaywanttopassavariablenumberofpositionalargumentstoafunction andPythonprovidesyouwiththeabilitytodoit.Let’slookataverycommonusecase, theminimumfunction.Thisisafunctionthatcalculatestheminimumofitsinputvalues. arguments.variable.positional.py defminimum(*n): #print(n)#nisatuple ifn:#explainedafterthecode mn=n[0] forvalueinn[1:]: ifvalue<mn: mn=value print(mn) minimum(1,3,-7,9)#n=(1,3,-7,9)-prints:-7 minimum()#n=()-prints:nothing Asyoucansee,whenwespecifyaparameterprependinga*toitsname,wearetelling Pythonthatthatparameterwillbecollectingavariablenumberofpositionalarguments, accordingtohowthefunctioniscalled.Withinthefunction,nisatuple.Uncommentthe print(n)toseeforyourselfandplayaroundwithitforabit. Note Haveyounoticedhowwecheckedifnwasn’temptywithasimpleifn:?Thisisdueto thefactthatcollectionobjectsevaluatetoTruewhennon-empty,andotherwiseFalsein Python.Thisistruefortuples,sets,lists,dictionaries,andsoon. Oneotherthingtonoteisthatwemaywanttothrowanerrorwhenwecallthefunction withnoarguments,insteadofsilentlydoingnothing.Inthiscontext,we’renotconcerned aboutmakingthisfunctionrobust,butinunderstandingvariablepositionalarguments. Let’smakeanotherexampletoshowyoutwothingsthat,inmyexperience,areconfusing tothosewhoarenewtothis. arguments.variable.positional.unpacking.py deffunc(*args): print(args) values=(1,3,-7,9) func(values)#equivalentto:func((1,3,-7,9)) func(*values)#equivalentto:func(1,3,-7,9) Takeagoodlookatthelasttwolinesoftheprecedingexample.Inthefirstone,wecall WOW! eBook www.wowebook.org funcwithoneargument,afourelementstuple.Inthesecondexample,byusingthe* syntax,we’redoingsomethingcalledunpacking,whichmeansthatthefourelements tupleisunpacked,andthefunctioniscalledwithfourarguments:1,3,-7,9. ThisbehaviorispartofthemagicPythondoestoallowyoutodoamazingthingswhen callingfunctionsdynamically. Variablekeywordarguments Variablekeywordargumentsareverysimilartovariablepositionalarguments.Theonly differenceisthesyntax(**insteadof*)andthattheyarecollectedinadictionary. Collectionandunpackingworkinthesameway,solet’slookatanexample: arguments.variable.keyword.py deffunc(**kwargs): print(kwargs) #Allcallsequivalent.Theyprint:{'a':1,'b':42} func(a=1,b=42) func(**{'a':1,'b':42}) func(**dict(a=1,b=42)) Allthecallsareequivalentintheprecedingexample.Youcanseethataddinga**infront oftheparameternameinthefunctiondefinitiontellsPythontousethatnametocollecta variablenumberofkeywordparameters.Ontheotherhand,whenwecallthefunction,we caneitherpassname=valueargumentsexplicitly,orunpackadictionaryusingthesame** syntax. Thereasonwhybeingabletopassavariablenumberofkeywordparametersisso importantmaynotbeevidentatthemoment,so,howaboutamorerealisticexample? Let’sdefineafunctionthatconnectstoadatabase.Wewanttoconnecttoadefault databasebysimplycallingthisfunctionwithnoparameters.Wealsowanttoconnectto anyotherdatabasebypassingthefunctiontheappropriatearguments.Beforeyoureadon, spendacoupleofminutesfiguringoutasolutionbyyourself. arguments.variable.db.py defconnect(**options): conn_params={ 'host':options.get('host','127.0.0.1'), 'port':options.get('port',5432), 'user':options.get('user',''), 'pwd':options.get('pwd',''), } print(conn_params) #wethenconnecttothedb(commentedout) #db.connect(**conn_params) connect() connect(host='127.0.0.42',port=5433) connect(port=5431,user='fab',pwd='gandalf') Noteinthefunctionwecanprepareadictionaryofconnectionparameters(conn_params) inthefunctionusingdefaultvaluesasfallback,allowingthemtobeoverwritteniftheyare WOW! eBook www.wowebook.org providedinthefunctioncall.Therearebetterwaystodothiswithfewerlinesofcodebut we’renotconcernedwiththatnow.Runningtheprecedingcodeyieldsthefollowing result: $pythonarguments.variable.db.py {'host':'127.0.0.1','pwd':'','user':'','port':5432} {'host':'127.0.0.42','pwd':'','user':'','port':5433} {'host':'127.0.0.1','pwd':'gandalf','user':'fab','port':5431} Notethecorrespondencebetweenthefunctioncallsandtheoutput.Notehowdefault valuesareeitherthereoroverridden,accordingtowhatwaspassedtothefunction. Keyword-onlyarguments Python3allowsforanewtypeofparameter:thekeyword-onlyparameter.Wearegoing tostudythemonlybrieflyastheirusecasesarenotthatfrequent.Therearetwowaysof specifyingthem,eitherafterthevariablepositionalarguments,orafterabare*.Let’ssee anexampleofboth. arguments.keyword.only.py defkwo(*a,c): print(a,c) kwo(1,2,3,c=7)#prints:(1,2,3)7 kwo(c=4)#prints:()4 #kwo(1,2)#breaks,invalidsyntax,withthefollowingerror #TypeError:kwo()missing1requiredkeyword-onlyargument:'c' defkwo2(a,b=42,*,c): print(a,b,c) kwo2(3,b=7,c=99)#prints:3799 kwo2(3,c=13)#prints:34213 #kwo2(3,23)#breaks,invalidsyntax,withthefollowingerror #TypeError:kwo2()missing1requiredkeyword-onlyargument:'c' Asanticipated,thefunction,kwo,takesavariablenumberofpositionalarguments(a)and akeyword-onlyfunction,c.Theresultsofthecallsarestraightforwardandyoucan uncommentthethirdcalltoseewhaterrorPythonreturns. Thesameappliestothefunction,kwo2,whichdiffersfromkwointhatittakesapositional argumenta,akeywordargumentb,andthenakeyword-onlyargument,c.Youcan uncommentthethirdcalltoseetheerror. Nowthatyouknowhowtospecifydifferenttypesofinputparameters,let’sseehowyou cancombinetheminfunctiondefinitions. Combininginputparameters Youcancombineinputparameters,aslongasyoufollowtheseorderingrules: Whendefiningafunction,normalpositionalargumentscomefirst(name),thenany defaultarguments(name=value),thenthevariablepositionalarguments(*name,or simply*),thenanykeyword-onlyarguments(eithernameorname=valueformis WOW! eBook www.wowebook.org good),thenanyvariablekeywordarguments(**name). Ontheotherhand,whencallingafunction,argumentsmustbegiveninthefollowing order:positionalargumentsfirst(value),thenanycombinationofkeyword arguments(name=value),variablepositionalarguments(*name),thenvariable keywordarguments(**name). Sincethiscanbeabittrickywhenlefthanginginthetheoreticalworld,let’slookata coupleofquickexamples. arguments.all.py deffunc(a,b,c=7,*args,**kwargs): print('a,b,c:',a,b,c) print('args:',args) print('kwargs:',kwargs) func(1,2,3,*(5,7,9),**{'A':'a','B':'b'}) func(1,2,3,5,7,9,A='a',B='b')#sameaspreviousone Notetheorderoftheparametersinthefunctiondefinition,andthatthetwocallsare equivalent.Inthefirstone,we’reusingtheunpackingoperatorsforiterablesand dictionaries,whileinthesecondonewe’reusingamoreexplicitsyntax.Theexecutionof thisyields(Iprintedonlytheresultofonecall): $pythonarguments.all.py a,b,c:123 args:(5,7,9) kwargs:{'A':'a','B':'b'} Let’snowlookatanexamplewithkeyword-onlyarguments. arguments.all.kwonly.py deffunc_with_kwonly(a,b=42,*args,c,d=256,**kwargs): print('a,b:',a,b) print('c,d:',c,d) print('args:',args) print('kwargs:',kwargs) #bothcallsequivalent func_with_kwonly(3,42,c=0,d=1,*(7,9,11),e='E',f='F') func_with_kwonly(3,42,*(7,9,11),c=0,d=1,e='E',f='F') NotethatIhavehighlightedthekeyword-onlyargumentsinthefunctiondeclaration.They comeafterthevariablepositionalargument*args,anditwouldbethesameiftheycame rightafterasingle*(inwhichcasetherewouldn’tbeavariablepositionalargument).The executionofthisyields(Iprintedonlytheresultofonecall): $pythonarguments.all.kwonly.py a,b:342 c,d:01 args:(7,9,11) kwargs:{'f':'F','e':'E'} OneotherthingtonotearethenamesIgavetothevariablepositionalandkeyword WOW! eBook www.wowebook.org arguments.You’refreetochoosedifferently,butbeawarethatargsandkwargsarethe conventionalnamesgiventotheseparameters,atleastgenerically.Nowthatyouknow howtodefineafunctioninallpossibleflavors,letmeshowyousomethingtricky: mutabledefaults. Avoidthetrap!Mutabledefaults OnethingtobeveryawareofwithPythonisthatdefaultvaluesarecreatedatdeftime, therefore,subsequentcallstothesamefunctionwillpossiblybehavedifferentlyaccording tothemutabilityoftheirdefaultvalues.Let’slookatanexample: arguments.defaults.mutable.py deffunc(a=[],b={}): print(a) print(b) print('#'*12) a.append(len(a))#thiswillaffecta'sdefaultvalue b[len(a)]=len(a)#andthiswillaffectb'sone func() func() func() Theparametersbothhavemutabledefaultvalues.Thismeansthat,ifyouaffectthose objects,anymodificationwillstickaroundinsubsequentfunctioncalls.Seeifyoucan understandtheoutputofthosecalls: $pythonarguments.defaults.mutable.py [] {} ############ [0] {1:1} ############ [0,1] {1:1,2:2} ############ It’sinteresting,isn’tit?Whilethisbehaviormayseemveryweirdatfirst,itactuallymakes sense,andit’sveryhandy,forexample,whenusingmemoizationtechniques(Googlean exampleofthat,ifyou’reinterested). Evenmoreinterestingiswhathappenswhen,betweenthecalls,weintroduceonethat doesn’tusedefaults,likethis: arguments.defaults.mutable.intermediate.call.py func() func(a=[1,2,3],b={'B':1}) func() Whenwerunthiscode,thisistheoutput: $pythonarguments.defaults.mutable.intermediate.call.py [] WOW! eBook www.wowebook.org {} ############ [1,2,3] {'B':1} ############ [0] {1:1} ############ Thisoutputshowsusthatthedefaultsareretainedevenifwecallthefunctionwithother values.Onequestionthatcomestomindis,howdoIgetafreshemptyvalueeverytime? Well,theconventionisthefollowing: arguments.defaults.mutable.no.trap.py deffunc(a=None): ifaisNone: a=[] #dowhateveryouwantwith`a`... Notethat,byusingtheprecedingtechnique,ifaisn’tpassedwhencallingthefunction, youalwaysgetabrandnewemptylist. Okay,enoughwiththeinput,let’slookattheothersideofthecoin,theoutput. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Returnvalues ReturnvaluesoffunctionsareoneofthosethingswherePythonislightyearsaheadof mostotherlanguages.Functionsareusuallyallowedtoreturnoneobject(onevalue)but, inPython,youcanreturnatuple,andthisimpliesthatyoucanreturnwhateveryouwant. Thisfeatureallowsacodertowritesoftwarethatwouldbemuchhardertowriteinany otherlanguage,orcertainlymoretedious.We’vealreadysaidthattoreturnsomething fromafunctionweneedtousethereturnstatement,followedbywhatwewanttoreturn. Therecanbeasmanyreturnstatementsasneededinthebodyofafunction. Ontheotherhand,ifwithinthebodyofafunctionwedon’treturnanything,thefunction willreturnNone.Thisbehaviorisharmlessand,eventhoughIdon’thavetheroomhereto gointodetailexplainingwhyPythonwasdesignedlikethis,letmejusttellyouthatthis featureallowsforseveralinterestingpatterns,andconfirmsPythonasaveryconsistent language. Isayit’sharmlessbecauseyouareneverforcedtocollecttheresultofafunctioncall.I’ll showyouwhatImeanwithanexample: return.none.py deffunc(): pass func()#thereturnofthiscallwon'tbecollected.It'slost. a=func()#thereturnofthisoneinsteadiscollectedinto`a` print(a)#prints:None Notethatthewholebodyofthefunctioniscomprisedonlyofthepassstatement.Asthe officialdocumentationtellsus,passisanulloperation.Whenitisexecuted,nothing happens.Itisusefulasaplaceholderwhenastatementisrequiredsyntactically,butno codeneedstobeexecuted.Inotherlanguages,wewouldprobablyjustindicatethatwitha pairofcurlybraces({}),whichdefineanemptyscopebutinPythonascopeisdefinedby indentingcode,thereforeastatementsuchaspassisnecessary. Noticealsothatthefirstcallofthefunctionfuncreturnsavalue(None)whichwedon’t collect.AsIsaidbefore,collectingthereturnvalueofafunctioncallisnotmandatory. Now,that’sgoodbutnotveryinterestingso,howaboutwewriteaninterestingfunction? RememberthatinChapter1,IntroductionandFirstSteps–TakeaDeepBreath,wetalked aboutthefactorialofafunction.Let’swriteourownhere(forsimplicity,Iwillassumethe functionisalwayscalledcorrectlywithappropriatevaluessoIwon’tsanity-checkonthe inputargument): return.single.value.py deffactorial(n): ifnin(0,1): return1 result=n forkinrange(2,n): result*=k WOW! eBook www.wowebook.org returnresult f5=factorial(5)#f5=120 Notethatwehavetwopointsofreturn.Ifniseither0or1(inPythonit’scommontouse theintypeofcheckasIdidinsteadofthemoreverboseifn==0orn==1:),we return1.Otherwise,weperformtherequiredcalculation,andwereturnresult.Canwe writethisfunctionalittlebitmorePythonically?Yes,butI’llletyoufigureoutthatfor yourself,asanexercise. return.single.value.2.py fromfunctoolsimportreduce fromoperatorimportmul deffactorial(n): returnreduce(mul,range(1,n+1),1) f5=factorial(5)#f5=120 Iknowwhatyou’rethinking,oneline?Pythoniselegant,andconcise!Ithinkthis functionisreadableevenifyouhaveneverseenreduceormul,butifyoucan’treaditor understandit,setasideafewminutesanddosomeresearchonthePythondocumentation untilitsbehavioriscleartoyou.Beingabletolookupfunctionsinthedocumentationand understandcodewrittenbysomeoneelseisataskeverydeveloperneedstobeableto perform,sothinkofthisasagoodexercise,andgoodluck! Tip Tothisend,makesureyoulookupthehelpfunction,whichcomesinveryhandy exploringwiththeconsole. WOW! eBook www.wowebook.org Returningmultiplevalues Unlikeinmostotherlanguages,inPythonit’sveryeasytoreturnmultipleobjectsfroma function.Thisfeatureopensupawholeworldofpossibilitiesandallowsyoutocodeina stylethatishardtoreproducewithotherlanguages.Ourthinkingislimitedbythetools weuse,thereforewhenPythongivesyoumorefreedomthanotherlanguages,itisactually boostingyourowncreativityaswell.Toreturnmultiplevaluesisveryeasy,youjustuse tuples(eitherexplicitlyorimplicitly).Let’slookatasimpleexamplethatmimicsthe divmodbuilt-infunction: return.multiple.py defmoddiv(a,b): returna//b,a%b print(moddiv(20,7))#prints(2,6) Icouldhavewrappedthehighlightedpartintheprecedingcodeinbraces,makingitan explicittuple,butthere’snoneedforthat.Theprecedingfunctionreturnsboththeresult andtheremainderofthedivision,atthesametime. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Afewusefultips Whenwritingfunctions,it’sveryusefultofollowguidelinessothatyouwritethemwell. I’llquicklypointsomeofthemouthere: Functionsshoulddoonething:Functionsthatdoonethingareeasytodescribein oneshortsentence.Functionswhichdomultiplethingscanbesplitintosmaller functionswhichdoonething.Thesesmallerfunctionsareusuallyeasiertoreadand understand.Rememberthedatascienceexamplewesawafewpagesago. Functionsshouldbesmall:Thesmallertheyare,theeasieritistotestthemandto writethemsothattheydoonething. Thefewerinputparameters,thebetter:Functionswhichtakealotofarguments quicklybecomehardertomanage(amongotherissues). Functionsshouldbeconsistentintheirreturnvalues:ReturningFalseorNoneis notthesamething,evenifwithinaBooleancontexttheybothevaluatetoFalse. Falsemeansthatwehaveinformation(False),whileNonemeansthatthereisno information.Trywritingfunctionswhichreturninaconsistentway,nomatterwhat happensintheirbody. Functionsshouldn’thavesideeffects:Inotherwords,functionsshouldnotaffect thevaluesyoucallthemwith.Thisisprobablythehardeststatementtounderstandat thispoint,soI’llgiveyouanexampleusinglists.Inthefollowingcode,notehow numbersisnotsortedbythesortedfunction,whichactuallyreturnsasortedcopyof numbers.Conversely,thelist.sort()methodisactingonthenumbersobjectitself, andthatisfinebecauseitisamethod(afunctionthatbelongstoanobjectand thereforehastherightstomodifyit): >>>numbers=[4,1,7,5] >>>sorted(numbers)#won'tsorttheoriginal`numbers`list [1,4,5,7] >>>numbers#let'sverify [4,1,7,5]#good,untouched >>>numbers.sort()#thiswillactonthelist >>>numbers [1,4,5,7] Followtheseguidelinesandyou’llwritebetterfunctions,whichwillserveyouwell. Note Chapter3,FunctionsinCleanCodebyRobertC.Martin,PrenticeHallisdedicatedto functionsandit’sprobablythebestsetofguidelinesI’veeverreadonthesubject. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Recursivefunctions Whenafunctioncallsitselftoproducearesult,itissaidtoberecursive.Sometimes recursivefunctionsareveryusefulinthattheymakeiteasiertowritecode.Some algorithmsareveryeasytowriteusingtherecursiveparadigm,whileothersarenot.There isnorecursivefunctionthatcannotberewritteninaniterativefashion,soit’susuallyupto theprogrammertochoosethebestapproachforthecaseathand. Arecursivefunctionusuallyhasasetofbasecasesforwhichthereturnvaluedoesn’t dependonasubsequentcalltothefunctionitselfandasetofrecursivecases,forwhich thereturnvalueiscalculatedwithoneormorecallstothefunctionitself. Asanexample,wecanconsiderthe(hopefullyfamiliarbynow)factorialfunctionN!. ThebasecaseiswhenNiseither0or1.Thefunctionreturns1withnoneedforfurther calculation.Ontheotherhand,inthegeneralcase,N!returnstheproduct1*2*…*(N1)*N.Ifyouthinkaboutit,N!canberewrittenlikethis:N!=(N-1)!*N.Asapractical example,consider5!=1*2*3*4*5=(1*2*3*4)*5=4!*5. Let’swritethisdownincode: recursive.factorial.py deffactorial(n): ifnin(0,1):#basecase return1 returnfactorial(n-1)*n#recursivecase Note Whenwritingrecursivefunctions,alwaysconsiderhowmanynestedcallsyoumake,there isalimit.Forfurtherinformationonthis,checkoutsys.getrecursionlimit()and sys.setrecursionlimit(). Recursivefunctionsareusedalotwhenwritingalgorithmsandtheycanbereallyfunto write.Asagoodexercise,trytosolveacoupleofsimpleproblemsusingbotharecursive andaniterativeapproach. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Anonymousfunctions OnelasttypeoffunctionsthatIwanttotalkaboutareanonymousfunctions.These functions,whicharecalledlambdasinPython,areusuallyusedwhenafully-fledged functionwithitsownnamewouldbeoverkill,andallwewantisaquick,simpleone-liner thatdoesthejob. ImaginethatyouwantalistofallthenumbersuptoNwhicharemultiplesoffive. Imaginethatyouwanttofilterthoseoutusingthefilterfunction,whichtakesafunction andaniterableandconstructsafilterobjectwhichyoucaniterateon,fromthoseelements ofiterableforwhichthefunctionreturnsTrue.Withoutusingananonymousfunction,you woulddosomethinglikethis: filter.regular.py defis_multiple_of_five(n): returnnotn%5 defget_multiples_of_five(n): returnlist(filter(is_multiple_of_five,range(n))) print(get_multiples_of_five(50)) Ihavehighlightedthemainlogicofget_multiples_of_five.Notehowthefilteruses is_multiple_of_fivetofilterthefirstnnaturalnumbers.Thisseemsabitexcessive,the taskissimpleandwedon’tneedtokeeptheis_multiple_of_fivefunctionaroundfor anythingelse.Let’srewriteitusingalambdafunction: filter.lambda.py defget_multiples_of_five(n): returnlist(filter(lambdak:notk%5,range(n))) print(get_multiples_of_five(50)) Thelogicisexactlythesamebutthefilteringfunctionisnowalambda.Defininga lambdaisveryeasyandfollowsthisform:func_name=lambda[parameter_list]: expression.Afunctionobjectisreturned,whichisequivalenttothis:def func_name([parameter_list]):returnexpression. Note Notethatoptionalparametersareindicatedfollowingthecommonsyntaxofwrapping theminsquarebrackets. Let’slookatanothercoupleofexamplesofequivalentfunctionsdefinedinthetwoforms: lambda.explained.py #example1:adder defadder(a,b): returna+b #isequivalentto: adder_lambda=lambdaa,b:a+b #example2:touppercase defto_upper(s): WOW! eBook www.wowebook.org returns.upper() #isequivalentto: to_upper_lambda=lambdas:s.upper() Theprecedingexamplesareverysimple.Thefirstoneaddstwonumbers,andthesecond oneproducestheuppercaseversionofastring.NotethatIassignedwhatisreturnedby thelambdaexpressionstoaname(adder_lambda,to_upper_lambda),butthereisnoneed forthatwhenyouuselambdasinthewaywedidinthefilterexamplebefore. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Functionattributes Everyfunctionisafully-fledgedobjectand,assuch,theyhavemanyattributes.Someof themarespecialandcanbeusedinanintrospectivewaytoinspectthefunctionobjectat runtime.Thefollowingscriptisanexamplethatshowsallofthemandhowtodisplay theirvalueforanexamplefunction: func.attributes.py defmultiplication(a,b=1): """Returnamultipliedbyb.""" returna*b special_attributes=[ "__doc__","__name__","__qualname__","__module__", "__defaults__","__code__","__globals__","__dict__", "__closure__","__annotations__","__kwdefaults__", ] forattributeinspecial_attributes: print(attribute,'->',getattr(multiplication,attribute)) Iusedthebuilt-ingetattrfunctiontogetthevalueofthoseattributes.getattr(obj, attribute)isequivalenttoobj.attributeandcomesinhandywhenweneedtogetan attributeatruntimeusingitsstringname.Runningthisscriptyields: $pythonfunc.attributes.py __doc__->Returnamultipliedbyb. __name__->multiplication __qualname__->multiplication __module__->__main__ __defaults__->(1,) __code__-><codeobjectmultiplicationat0x7ff529e79300,file "ch4/func.attributes.py",line1> __globals__->{...omitted…} __dict__->{} __closure__->None __annotations__->{} __kwdefaults__->None Ihaveomittedthevalueofthe__globals__attribute,itwastoobig.Anexplanationof themeaningofthisattributecanbefoundinthetypessectionofthePythonDataModel documentationpage. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Built-infunctions Pythoncomeswithalotofbuilt-infunctions.Theyareavailableanywhereandyoucan getalistofthembyinspectingthebuiltinmodulewithdir(__builtin__),orbygoing totheofficialPythondocumentation.Unfortunately,Idon’thavetheroomtogothrough allofthemhere.Someofthemwe’vealreadyseen,suchasany,bin,bool,divmod, filter,float,getattr,id,int,len,list,min,print,set,tuple,type,andzip,but therearemanymore,whichyoushouldreadatleastonce. Getfamiliarwiththem,experiment,writeasmallpieceofcodeforeachofthem,make sureyouhavethematthetipofyourfingerssothatyoucanusethemwhenyouneed them. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Onefinalexample Beforewefinishoffthischapter,howaboutafinalexample?Iwasthinkingwecould writeafunctiontogeneratealistofprimenumbersuptoalimit.We’vealreadyseenthe codeforthissolet’smakeitafunctionand,tokeepitinteresting,let’soptimizeitabit. Itturnsoutthatyoudon’tneedtodivideitbyallnumbersfrom2toN-1todecideifa numberNisprime.Youcanstopat .Moreover,youdon’tneedtotestthedivisionfor ,youcanjustusetheprimesinthatrange.I’llleaveittoyouto allnumbersfrom2to figureoutwhythisworks,ifyou’reinterested.Let’sseehowthecodechanges: primes.py frommathimportsqrt,ceil defget_primes(n): """Calculatealistofprimesupton(included).""" primelist=[] forcandidateinrange(2,n+1): is_prime=True root=int(ceil(sqrt(candidate)))#divisionlimit forprimeinprimelist:#wetryonlytheprimes ifprime>root:#noneedtocheckanyfurther break ifcandidate%prime==0: is_prime=False break ifis_prime: primelist.append(candidate) returnprimelist Thecodeisthesameasinthepreviouschapter.Wehavechangedthedivisionalgorithm sothatweonlytestdivisibilityusingthepreviouslycalculatedprimesandwestopped oncethetestingdivisorwasgreaterthantherootofthecandidate.Weusedtheresultlist primelisttogettheprimesforthedivision.Wecalculatedtherootvalueusingafancy formula,theintegervalueoftheceilingoftherootofthecandidate.Whileasimpleint(k **0.5)+1wouldhaveservedourpurposeaswell,theformulaIchoseiscleanerand requiresmetouseacoupleofimports,whichIwantedtoshowyou.Checkoutthe functionsinthemathmodule,theyareveryinteresting! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Documentingyourcode I’mabigfanofcodethatdoesn’tneeddocumentation.Whenyouprogramcorrectly, choosetherightnamesandtakecareofthedetails,yourcodeshouldcomeoutasselfexplanatoryanddocumentationshouldnotbeneeded.Sometimesacommentisvery usefulthough,andsoissomedocumentation.Youcanfindtheguidelinesfor documentingPythoninPEP257–Docstringconventions,butI’llshowyouthebasics here. Pythonisdocumentedwithstrings,whichareaptlycalleddocstrings.Anyobjectcanbe documented,andyoucanuseeitherone-lineormulti-linedocstrings.One-linersarevery simple.Theyshouldnotprovideanothersignatureforthefunction,butclearlystateits purpose. docstrings.py defsquare(n): """Returnthesquareofanumbern.""" returnn**2 defget_username(userid): """Returntheusernameofausergiventheirid.""" returndb.get(user_id=userid).username Usingtripledouble-quotedstringsallowsyoutoexpandeasilylateron.Usesentencesthat endinaperiod,anddon’tleaveblanklinesbeforeorafter. Multi-linecommentsarestructuredinasimilarway.Thereshouldbeaone-linerthat brieflygivesyouthegistofwhattheobjectisabout,andthenamoreverbosedescription. Asanexample,Ihavedocumentedafictitiousconnectfunction,usingtheSphinx notation,inthefollowingexample. Note SphinxisprobablythemostwidelyusedtoolforcreatingPythondocumentation.Infact, theofficialPythondocumentationwaswrittenwithit.It’sdefinitelyworthspendingsome timecheckingitout. docstrings.py defconnect(host,port,user,password): """Connecttoadatabase. ConnecttoaPostgreSQLdatabasedirectly,usingthegiven parameters. :paramhost:ThehostIP. :paramport:Thedesiredport. :paramuser:Theconnectionusername. :parampassword:Theconnectionpassword. :return:Theconnectionobject. """ #bodyofthefunctionhere… WOW! eBook www.wowebook.org returnconnection WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Importingobjects Nowthatyouknowalotaboutfunctions,let’sseehowtousethem.Thewholepointof writingfunctionsistobeabletolaterreusethem,andthisinPythontranslatesto importingthemintothenamespaceinwhichyouneedthem.Therearemanydifferent waystoimportobjectsintoanamespace,butthemostcommononesarejusttwo:import module_nameandfrommodule_nameimportfunction_name.Ofcourse,thesearequite simplisticexamples,butbearwithmeforthetimebeing. Theformimportmodule_namefindsthemodulemodule_nameanddefinesanameforitin thelocalnamespacewheretheimportstatementisexecuted. Theformfrommodule_nameimportidentifierisalittlebitmorecomplicatedthan that,butbasicallydoesthesamething.Itfindsmodule_nameandsearchesforanattribute (orasubmodule)andstoresareferencetoidentifierinthelocalnamespace. Bothformshavetheoptiontochangethenameoftheimportedobjectusingtheasclause, likethis: frommymoduleimportmyfuncasbetter_named_func Justtogiveyouaflavorofwhatimportinglookslike,here’sanexamplefromatest moduleofanumbertheorylibraryIwrotesomeyearsago(it’savailableonBitbucket): karma/test_nt.py importunittest#importstheunittestmodule frommathimportsqrt#importsonefunctionfrommath fromrandomimportrandint,sample#twoimportsatonce frommockimportpatch fromnose.toolsimport(#multilineimport assert_equal, assert_list_equal, assert_not_in, ) fromkarmaimportnt,utils IcommentedsomeofthemandIhopeit’seasytofollow.Whenyouhaveastructureof filesstartingintherootofyourproject,youcanusethedotnotationtogettotheobject youwanttoimportintoyourcurrentnamespace,beitapackage,amodule,aclass,a function,oranythingelse.Thefrommoduleimportsyntaxalsoallowsacatch-allclause frommoduleimport*,whichissometimesusedtogetallthenamesfromamoduleinto thecurrentnamespaceatonce,butit’sfrowneduponforseveralreasons:performances, theriskofsilentlyshadowingothernames,andsoon.Youcanreadallthatthereisto knowaboutimportsintheofficialPythondocumentationbut,beforeweleavethesubject, letmegiveyouabetterexample. Imaginethatyouhavedefinedacoupleoffunctions:square(n)andcube(n)inamodule, funcdef.py,whichisinthelibfolder.Youwanttousetheminacoupleofmodules WOW! eBook www.wowebook.org whichareatthesamelevelofthelibfolder,calledfunc_import.py,andfunc_from.py. Showingthetreestructureofthatprojectproducessomethinglikethis: ├──func_from.py ├──func_import.py ├──lib ├──funcdef.py └──__init__.py BeforeIshowyouthecodeofeachmodule,pleaserememberthatinordertotellPython thatitisactuallyapackage,weneedtoputa__init__.pymoduleinit. Note Therearetwothingstonoteaboutthe__init__.pyfile.Firstofall,itisafullyfledged Pythonmodulesoyoucanputcodeintoitasyouwouldwithanyothermodule.Second, asofPython3.3,itspresenceisnolongerrequiredtomakeafolderbeinterpretedasa Pythonpackage. Thecodeisasfollows: funcdef.py defsquare(n): returnn**2 defcube(n): returnn**3 func_import.py importlib.funcdef print(lib.funcdef.square(10)) print(lib.funcdef.cube(10)) func_from.py fromlib.funcdefimportsquare,cube print(square(10)) print(cube(10)) Boththesefiles,whenexecuted,print100and1000.Youcanseehowdifferentlywethen accessthesquareandcubefunctions,accordingtohowandwhatweimportedinthe currentscope. WOW! eBook www.wowebook.org Relativeimports Theimportswe’veseenuntilnowarecalledabsolute,thatistosaytheydefinethewhole pathofthemodulethatwewanttoimport,orfromwhichwewanttoimportanobject. ThereisanotherwayofimportingobjectsintoPython,whichiscalledrelativeimport.It’s helpfulinsituationsinwhichwewanttorearrangethestructureoflargepackageswithout havingtoeditsub-packages,orwhenwewanttomakeamoduleinsideapackageableto importitself.Relativeimportsaredonebyaddingasmanyleadingdotsinfrontofthe moduleasthenumberoffoldersweneedtobacktrack,inordertofindwhatwe’re searchingfor.Simplyput,itissomethinglikethis: from.mymoduleimportmyfunc Foracompleteexplanationofrelativeimports,refertoPEP328 (https://www.python.org/dev/peps/pep-0328). Inlaterchapters,we’llcreateprojectsusingdifferentlibrariesandwe’lluseseveral differenttypesofimports,includingrelativeones,somakesureyoutakeabitoftimeto readupaboutitintheofficialPythondocumentation. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,finallyweexploredtheworldoffunctions.Theyareextremelyimportant and,fromnowon,we’llusethembasicallyeverywhere.Wetalkedaboutthemainreasons forusingthem,themostimportantofwhicharecodereuseandimplementationhiding. Wesawthatafunctionobjectislikeaboxthattakesoptionalinputandproducesoutput. Wecanfeedinputvaluestoafunctioninmanydifferentways,usingpositionaland keywordarguments,andusingvariablesyntaxforbothtypes. Nowyoushouldknowhowtowriteafunction,howtodocumentit,importitintoyour code,andcallit. ThenextchapterwillforcemetopushmyfootdownonthethrottleevenmoresoI suggestyoutakeanyopportunityyougettoconsolidateandenrichtheknowledgeyou’ve gathereduntilnowbyputtingyournoseintothePythonofficialdocumentation. Readyforthecoolstuff?Let’sgo! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter5.SavingTimeandMemory “It’snotthedailyincreasebutdailydecrease.Hackawayattheunessential.” —BruceLee IlovethisquotefromBruceLee,hewassuchawiseman!Especially,thesecondpart, hackawayattheunessential,istomewhatmakesacomputerprogramelegant.Afterall, ifthereisabetterwayofdoingthingssothatwedon’twastetimeormemory,whynot? Sometimes,therearevalidreasonsfornotpushingourcodeuptothemaximumlimit:for example,sometimestoachieveanegligibleimprovement,wehavetosacrificeon readabilityormaintainability.Doesitmakeanysensetohaveawebpageservedin1 secondwithunreadable,complicatedcode,whenwecanserveitin1.05secondswith readable,cleancode?No,itmakesnosense. Ontheotherhand,sometimesit’sperfectlylicittotryandshaveoffamillisecondfroma function,especiallywhenthefunctionismeanttobecalledthousandsoftimes.Every millisecondyousavetheremeansonesecondsavedperthousandofcalls,andthiscould bemeaningfulforyourapplication. Inlightoftheseconsiderations,thefocusofthischapterwillnotbetogiveyouthetools topushyourcodetotheabsolutelimitsofperformanceandoptimization“nomatter what”,butrather,togiveyouthetoolstowriteefficient,elegantcodethatreadswell,runs fast,anddoesn’twasteresourcesinanobviousway. Inthischapter,Iwillperformseveralmeasurementsandcomparisons,andcautiously drawsomeconclusions.Pleasedokeepinmindthatonadifferentboxwithadifferent setuporadifferentoperatingsystem,resultsmayvary.Takealookatthiscode: squares.py defsquare1(n): returnn**2#squaringthroughthepoweroperator defsquare2(n): returnn*n#squaringthroughmultiplication Bothfunctionsreturnthesquareofn,butwhichisfaster?FromasimplebenchmarkIran onthem,itlookslikethesecondisslightlyfaster.Ifyouthinkaboutit,itmakessense: calculatingthepowerofanumberinvolvesmultiplicationandtherefore,whatever algorithmyoumayusetoperformthepoweroperation,it’snotlikelytobeatasimple multiplicationliketheoneinsquare2. Dowecareaboutthisresult?Inmostcasesno.Ifyou’recodingane-commercewebsite, chancesareyouwon’teverevenneedtoraiseanumbertothesecondpower,andifyou do,youprobablywillhavetodoitafewtimesperpage.Youdon’tneedtoconcern yourselfonsavingafewmicrosecondsonafunctionyoucallafewtimes. So,whendoesoptimizationbecomeimportant?Oneverycommoncaseiswhenyouhave todealwithhugecollectionsofdata.Ifyou’reapplyingthesamefunctiononamillion WOW! eBook www.wowebook.org customerobjects,thenyouwantyourfunctiontobetuneduptoitsbest.Gaining1/10ofa secondonafunctioncalledonemilliontimessavesyou100,000seconds,whichareabout 27.7hours.That’snotthesame,right?So,let’sfocusoncollections,andlet’sseewhich toolsPythongivesyoutohandlethemwithefficiencyandgrace. Note Manyoftheconceptswewillseeinthischapterarebasedonthoseofiteratorand iterable.Simplyput,theabilityforanobjecttoreturnitsnextelementwhenasked,andto raiseaStopIterationexceptionwhenexhausted.We’llseehowtocodeacustomiterator anditerableobjectsinthenextchapter. WOW! eBook www.wowebook.org map,zip,andfilter We’llstartbyreviewingmap,filter,andzip,whicharethemainbuilt-infunctionsone canemploywhenhandlingcollections,andthenwe’lllearnhowtoachievethesame resultsusingtwoveryimportantconstructs:comprehensionsandgenerators.Fastenyour seatbelt! WOW! eBook www.wowebook.org map AccordingtotheofficialPythondocumentation: map(function,iterable,...)returnsaniteratorthatappliesfunctiontoevery itemofiterable,yieldingtheresults.Ifadditionaliterableargumentsarepassed, functionmusttakethatmanyargumentsandisappliedtotheitemsfromalliterables inparallel.Withmultipleiterables,theiteratorstopswhentheshortestiterableis exhausted. Wewillexplaintheconceptofyieldinglateroninthechapter.Fornow,let’stranslatethis intocode:we’llusealambdafunctionthattakesavariablenumberofpositional arguments,andjustreturnsthemasatuple.Also,asmapreturnsaniterator,we’llneedto wrapeachcalltoitwithinalistconstructorsothatweexhausttheiterablebyputtingall ofitselementsintoalist(you’llseeanexampleofthisinthecode): map.example.py >>>map(lambda*a:a,range(3))#withoutwrappinginlist… <mapobjectat0x7f563513b518>#wegettheiteratorobject >>>list(map(lambda*a:a,range(3)))#wrappinginlist… [(0,),(1,),(2,)]#wegetalistwithitselements >>>list(map(lambda*a:a,range(3),'abc'))#2iterables [(0,'a'),(1,'b'),(2,'c')] >>>list(map(lambda*a:a,range(3),'abc',range(4,7)))#3 [(0,'a',4),(1,'b',5),(2,'c',6)] >>>#mapstopsattheshortestiterator >>>list(map(lambda*a:a,(),'abc'))#emptytupleisshortest [] >>>list(map(lambda*a:a,(1,2),'abc'))#(1,2)shortest [(1,'a'),(2,'b')] >>>list(map(lambda*a:a,(1,2,3,4),'abc'))#'abc'shortest [(1,'a'),(2,'b'),(3,'c')] Intheprecedingcodeyoucanseewhy,inordertopresentyouwiththeresults,Ihaveto wrapthecallstomapwithinalistconstructor,otherwiseIgetthestringrepresentationof amapobject,whichisnotreallyusefulinthiscontext,isit? Youcanalsonoticehowtheelementsofeachiterableareappliedtothefunction:atfirst, thefirstelementofeachiterable,thenthesecondoneofeachiterable,andsoon.Notice alsothatmapstopswhentheshortestoftheiterableswecalleditwithisexhausted.Thisis actuallyaverynicebehavior:itdoesn’tforceustoleveloffalltheiterablestoacommon length,anditdoesn’tbreakiftheyaren’tallthesamelength. mapisveryusefulwhenyouhavetoapplythesamefunctiontooneormorecollectionsof objects.Asamoreinterestingexample,let’sseethedecorate-sort-undecorateidiom (alsoknownasSchwartziantransform).It’satechniquethatwasextremelypopular whenPythonsortingwasn’tprovidingkey-functions,andthereforetodayislessused,but it’sacooltrickthatstillcomesathandonceinawhile. Let’sseeavariationofitinthenextexample:wewanttosortindescendingorderbythe WOW! eBook www.wowebook.org sumofcreditsaccumulatedbystudents,sotohavethebeststudentatposition0.Wewrite afunctiontoproduceadecoratedobject,wesort,andthenweundecorate.Eachstudent hascreditsinthree(possiblydifferent)subjects.Todecorateanobjectmeanstotransform it,eitheraddingextradatatoit,orputtingitintoanotherobject,inawaythatallowsusto beabletosorttheoriginalobjectsthewaywewant.Afterthesorting,werevertthe decoratedobjectstogettheoriginalonesfromthem.Thisiscalledtoundecorate. decorate.sort.undecorate.py students=[ dict(id=0,credits=dict(math=9,physics=6,history=7)), dict(id=1,credits=dict(math=6,physics=7,latin=10)), dict(id=2,credits=dict(history=8,physics=9,chemistry=10)), dict(id=3,credits=dict(math=5,physics=5,geography=7)), ] defdecorate(student): #createa2-tuple(sumofcredits,student)fromstudentdict return(sum(student['credits'].values()),student) defundecorate(decorated_student): #discardsumofcredits,returnoriginalstudentdict returndecorated_student[1] students=sorted(map(decorate,students),reverse=True) students=list(map(undecorate,students)) Intheprecedingcode,Ihighlightedthetrickyandimportantparts.Let’sstartby understandingwhateachstudentobjectis.Infact,let’sprintthefirstone: {'credits':{'history':7,'math':9,'physics':6},'id':0} Youcanseethatit’sadictionarywithtwokeys:idandcredit.Thevalueofcreditis alsoadictionaryinwhichtherearethreesubject/gradekey/valuepairs.AsI’msureyou recallfromourvisitinthedatastructuresworld,callingdict.values()returnsanobject similartoaniterable,withonlythevalues.Therefore, sum(student['credits'].values()),forthefirststudentisequivalenttosum(9,6,7) (oranypermutationofthosenumbersbecausedictionariesdon’tretainorder,butluckily forus,additioniscommutative). Withthatoutoftheway,it’seasytoseewhatistheresultofcallingdecoratewithanyof thestudents.Let’sprinttheresultofdecorate(students[0]): (22,{'credits':{'history':7,'math':9,'physics':6},'id':0}) That’snice!Ifwedecorateallthestudentslikethis,wecansortthemontheirtotalamount ofcreditsbutjustsortingthelistoftuples.Inordertoapplythedecorationtoeachitemin students,wecallmap(decorate,students).Thenwesorttheresult,andthenwe undecorateinasimilarfashion.Ifyouhavegonethroughthepreviouschapterscorrectly, understandingthiscodeshouldn’tbetoohard. Printingstudentsafterrunningthewholecodeyields: $pythondecorate.sort.undecorate.py WOW! eBook www.wowebook.org [{'credits':{'chemistry':10,'history':8,'physics':9},'id':2}, {'credits':{'latin':10,'math':6,'physics':7},'id':1}, {'credits':{'history':7,'math':9,'physics':6},'id':0}, {'credits':{'geography':7,'math':5,'physics':5},'id':3}] Andyoucansee,bytheorderofthestudentobjects,thattheyhaveindeedbeensortedby thesumoftheircredits. Note Formoreonthedecorate-sort-undecorateidiom,there’saveryniceintroductioninthe sortinghow-tosectionoftheofficialPythondocumentation (https://docs.python.org/3.4/howto/sorting.html#the-old-way-using-decorate-sortundecorate). Onethingtonoticeaboutthesortingpart:whatiftwoormorestudentssharethesame totalsum?Thesortingalgorithmwouldthenproceedsortingthetuplesbycomparingthe studentobjectswitheachother.Thisdoesn’tmakeanysense,andinmorecomplexcases couldleadtounpredictableresults,orevenerrors.Ifyouwanttobesuretoavoidthis issue,onesimplesolutionistocreatea3-tupleinsteadofa2-tuple,havingthesumof creditsinthefirstposition,thepositionofthestudentobjectinthestudentslistinthe secondone,andthestudentobjectitselfinthethirdone.Thisway,ifthesumofcreditsis thesame,thetupleswillbesortedagainsttheposition,whichwillalwaysbedifferentand thereforeenoughtoresolvethesortingbetweenanypairoftuples.Formore considerationsonthistopic,pleasecheckoutthesortinghow-tosectionontheofficial Pythondocumentation. WOW! eBook www.wowebook.org zip We’vealreadycoveredzipinthepreviouschapters,solet’sjustdefineitproperlyand thenIwanttoshowyouhowyoucouldcombineitwithmap. AccordingtothePythondocumentation: zip(*iterables)returnsaniteratoroftuples,wherethei-thtuplecontainsthei-th elementfromeachoftheargumentsequencesoriterables.Theiteratorstopswhen theshortestinputiterableisexhausted.Withasingleiterableargument,itreturnsan iteratorof1-tuples.Withnoarguments,itreturnsanemptyiterator. Let’sseeanexample: zip.grades.py >>>grades=[18,23,30,27,15,9,22] >>>avgs=[22,21,29,24,18,18,24] >>>list(zip(avgs,grades)) [(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)] >>>list(map(lambda*a:a,avgs,grades))#equivalenttozip [(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)] Intheprecedingcode,we’rezippingtogethertheaverageandthegradeforthelastexam, pereachstudent.Noticehowthecodeinsidethetwolistcallsproducesexactlythesame result,showinghoweasyitistoreproducezipusingmap.Noticealsothat,aswedofor map,wehavetofeedtheresultofthezipcalltoalistconstructor. Asimpleexampleonthecombineduseofmapandzipcouldbeawayofcalculatingthe element-wisemaximumamongstsequences,thatis,themaximumofthefirstelementof eachsequence,thenthemaximumofthesecondone,andsoon: maxims.py >>>a=[5,9,2,4,7] >>>b=[3,7,1,9,2] >>>c=[6,8,0,5,3] >>>maxs=map(lambdan:max(*n),zip(a,b,c)) >>>list(maxs) [6,9,2,9,7] Noticehoweasyitistocalculatethemaxvaluesofthreesequences.zipisnotstrictly neededofcourse,wecouldjustusemap,butthiswouldrequireustowriteamuchmore complicatedfunctiontofeedmapwith.Sometimeswemaybeinasituationwhere changingthefunctionwefeedtomapisnotevenpossible.Incaseslikethese,beingableto massagethedata(likewe’redoinginthisexamplewithzip)isveryhelpful. WOW! eBook www.wowebook.org filter AccordingtothePythondocumentation: filter(function,iterable)constructaniteratorfromthoseelementsofiterable forwhichfunctionreturnsTrue.iterablemaybeeitherasequence,acontainerwhich supportsiteration,oraniterator.IffunctionisNone,theidentityfunctionisassumed, thatis,allelementsofiterablethatarefalseareremoved. Let’sseeaveryquickexample: filter.py >>>test=[2,5,8,0,0,1,0] >>>list(filter(None,test)) [2,5,8,1] >>>list(filter(lambdax:x,test))#equivalenttopreviousone [2,5,8,1] >>>list(filter(lambdax:x>4,test))#keeponlyitems>4 [5,8] Intheprecedingcode,noticehowthesecondcalltofilterisequivalenttothefirstone.If wepassafunctionthattakesoneargumentandreturnstheargumentitself,onlythose argumentsthatareTruewillmakethefunctionreturnTrue,thereforethisbehavioris exactlythesameaspassingNone.It’softenaverygoodexercisetomimicsomeofthe built-inPythonbehaviors.Whenyousucceedyoucansayyoufullyunderstandhow Pythonbehavesinaspecificsituation. Armedwithmap,zip,andfilter(andseveralotherfunctionsfromthePythonstandard library)wecanmassagesequencesveryeffectively.Butthosefunctionsarenottheonly waytodoit.Solet’sseeoneofthenicestfeaturesofPython:comprehensions. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Comprehensions Pythonoffersyoudifferenttypesofcomprehensions:list,dict,andset. We’llconcentrateonthefirstonefornow,andthenitwillbeeasytoexplaintheother two. Alistcomprehensionisaquickwayofmakingalist.Usuallythelististheresultof someoperationthatmayinvolveapplyingafunction,filtering,orbuildingadifferentdata structure. Let’sstartwithaverysimpleexampleIwanttocalculatealistwiththesquaresofthefirst 10naturalnumbers.Howwouldyoudoit?Thereareacoupleofequivalentways: squares.map.py #IfyoucodelikethisyouarenotaPythonguy!;) >>>squares=[] >>>forninrange(10): ...squares.append(n**2) ... >>>list(squares) [0,1,4,9,16,25,36,49,64,81] #Thisisbetter,oneline,niceandreadable >>>squares=map(lambdan:n**2,range(10)) >>>list(squares) [0,1,4,9,16,25,36,49,64,81] Theprecedingexampleshouldbenothingnewforyou.Let’sseehowtoachievethesame resultusingalistcomprehension: squares.comprehension.py >>>[n**2forninrange(10)] [0,1,4,9,16,25,36,49,64,81] Assimpleasthat.Isn’titelegant?Basicallywehaveputaforloopwithinsquare brackets.Let’snowfilterouttheoddsquares.I’llshowyouhowtodoitwithmapand filter,andthenusingalistcomprehensionagain. even.squares.py #usingmapandfilter sq1=list( filter(lambdan:notn%2,map(lambdan:n**2,range(10))) ) #equivalent,butusinglistcomprehensions sq2=[n**2forninrange(10)ifnotn%2] print(sq1,sq1==sq2)#prints:[0,4,16,36,64]True Ithinkthatnowthedifferenceinreadabilityisevident.Thelistcomprehensionreads muchbetter.It’salmostEnglish:givemeallsquares(n**2)fornbetween0and9ifnis even. WOW! eBook www.wowebook.org AccordingtothePythondocumentation: Alistcomprehensionconsistsofbracketscontaininganexpressionfollowedbyafor clause,thenzeroormorefororifclauses.Theresultwillbeanewlistresulting fromevaluatingtheexpressioninthecontextoftheforandifclauseswhichfollow it”. WOW! eBook www.wowebook.org Nestedcomprehensions Let’sseeanexampleofnestedloops.It’sverycommonwhendealingwithalgorithmsto havetoiterateonasequenceusingtwoplaceholders.Thefirstonerunsthroughthewhole sequence,lefttoright.Thesecondoneaswell,butitstartsfromthefirstone,insteadof0. Theconceptisthatoftestingallpairswithoutduplication.Let’sseetheclassicalforloop equivalent. pairs.for.loop.py items='ABCDE' pairs=[] forainrange(len(items)): forbinrange(a,len(items)): pairs.append((items[a],items[b])) Ifyouprintpairsattheend,youget: [('A','A'),('A','B'),('A','C'),('A','D'),('A','E'),('B','B'), ('B','C'),('B','D'),('B','E'),('C','C'),('C','D'),('C','E'), ('D','D'),('D','E'),('E','E')] Allthetupleswiththesameletterarethoseforwhichbisatthesamepositionasa.Now, let’sseehowwecantranslatethisinalistcomprehension: pairs.list.comprehension.py items='ABCDE' pairs=[(items[a],items[b]) forainrange(len(items))forbinrange(a,len(items))] Thisversionisjusttwolineslongandachievesthesameresult.Noticethatinthis particularcase,becausetheforloopoverbhasadependencyona,itmustfollowthefor loopoverainthecomprehension.Ifyouswapthemaround,you’llgetanameerror. WOW! eBook www.wowebook.org Filteringacomprehension Wecanapplyfilteringtoacomprehension.Let’sfirstdoitwithfilter.Let’sfindall Pythagoreantripleswhoseshortsidesarenumberssmallerthan10.Weobviouslydon’t wanttotestacombinationtwice,andthereforewe’lluseatrickliketheonewesawinthe previousexample. Note APythagoreantripleisatriple(a,b,c)ofintegernumberssatisfyingtheequation . pythagorean.triple.py frommathimportsqrt #thiswillgenerateallpossiblepairs mx=10 legs=[(a,b,sqrt(a**2+b**2)) forainrange(1,mx)forbinrange(a,mx)] #thiswillfilteroutallnonpythagoreantriples legs=list( filter(lambdatriple:triple[2].is_integer(),legs)) print(legs)#prints:[(3,4,5.0),(6,8,10.0)] Intheprecedingcode,wegeneratedalistof3-tuples,legs.Eachtuplecontainstwo integernumbers(thelegs)andthehypotenuseofthePythagoreantrianglewhoselegsare thefirsttwonumbersinthetuple.Forexample,whena=3andb=4,thetuplewillbe(3, 4,5.0),andwhena=5andb=7,thetuplewillbe(5,7,8.602325267042627). Afterhavingallthetriplesdone,weneedtofilteroutallthosethatdon’thavea hypotenusethatisanintegernumber.Inordertodothis,wefilterbasedon float_number.is_integer()beingTrue.ThismeansthatofthetwoexampletuplesI showedyoubefore,theonewithhypotenuse5.0willberetained,whiletheonewith hypotenuse8.602325267042627willbediscarded. Thisisgood,butIdon’tlikethatthetriplehastwointegernumbersandafloat.Theyare supposedtobeallintegers,solet’susemaptofixthis: pythagorean.triple.int.py frommathimportsqrt mx=10 legs=[(a,b,sqrt(a**2+b**2)) forainrange(1,mx)forbinrange(a,mx)] legs=filter(lambdatriple:triple[2].is_integer(),legs) #thiswillmakethethirdnumberinthetuplesinteger legs=list( map(lambdatriple:triple[:2]+(int(triple[2]),),legs)) print(legs)#prints:[(3,4,5),(6,8,10)] Noticethestepweadded.Wetakeeachelementinlegsandwesliceit,takingonlythe firsttwoelementsinit.Then,weconcatenatetheslicewitha1-tuple,inwhichweputthe integerversionofthatfloatnumberthatwedidn’tlike. WOW! eBook www.wowebook.org Seemslikealotofwork,right?Indeeditis.Let’sseehowtodoallthiswithalist comprehension: pythagorean.triple.comprehension.py frommathimportsqrt #thisstepisthesameasbefore mx=10 legs=[(a,b,sqrt(a**2+b**2)) forainrange(1,mx)forbinrange(a,mx)] #herewecombinefilterandmapinoneCLEANlistcomprehension legs=[(a,b,int(c))fora,b,cinlegsifc.is_integer()] print(legs)#prints:[(3,4,5),(6,8,10)] Iknow.It’smuchbetter,isn’tit?It’sclean,readable,shorter.Inotherwords,elegant. Tip I’mgoingquitefasthere,asanticipatedinthesummaryofthelastchapter.Areyou playingwiththiscode?Ifnot,Isuggestyoudo.It’sveryimportantthatyouplayaround, breakthings,changethings,seewhathappens.Makesureyouhaveaclearunderstanding ofwhatisgoingon.Youwanttobecomeaninja,right? WOW! eBook www.wowebook.org dictcomprehensions Dictionaryandsetcomprehensionsworkexactlylikethelistones,onlythereisalittle differenceinthesyntax.Thefollowingexamplewillsufficetoexplaineverythingyou needtoknow: dictionary.comprehensions.py fromstringimportascii_lowercase lettermap=dict((c,k)fork,cinenumerate(ascii_lowercase,1)) Ifyouprintlettermap,youwillseethefollowing(Iomittedthemiddleresults,youget thegist): {'a':1, 'b':2, 'c':3, ...omittedresults… 'x':24, 'y':25, 'z':26} Whathappensintheprecedingcodeisthatwe’refeedingthedictconstructorwitha comprehension(technically,ageneratorexpression,we’llseeitinabit).Wetellthedict constructortomakekey/valuepairsfromeachtupleinthecomprehension.Weenumerate thesequenceofalllowercaseASCIIletters,startingfrom1,usingenumerate.Pieceof cake.Thereisalsoanotherwaytodothesamething,whichisclosertotheother dictionarysyntax: lettermap={c:kfork,cinenumerate(ascii_lowercase,1)} Itdoesexactlythesamething,withaslightlydifferentsyntaxthathighlightsabitmoreof thekey:valuepart. Dictionariesdonotallowduplicationinthekeys,asshowninthefollowingexample: dictionary.comprehensions.duplicates.py word='Hello' swaps={c:c.swapcase()forcinword} print(swaps)#prints:{'o':'O','l':'L','e':'E','H':'h'} Wecreateadictionarywithkeys,thelettersinthestring'Hello',andvaluesofthesame letters,butwiththecaseswapped.Noticethereisonlyone'l':'L'pair.Theconstructor doesn’tcomplain,simplyreassignsduplicatestothelatestvalue.Let’smakethisclearer withanotherexample;let’sassigntoeachkeyitspositioninthestring: dictionary.comprehensions.positions.py word='Hello' positions={c:kfork,cinenumerate(word)} print(positions)#prints:{'l':3,'o':4,'e':1,'H':0} Noticethevalueassociatedtotheletter'l':3.Thepair'l':2isn’tthere,ithasbeen overriddenby'l':3. WOW! eBook www.wowebook.org setcomprehensions Setcomprehensionsareverysimilartolistanddictionaryones.Pythonallowsboththe set()constructortobeused,ortheexplicit{}syntax.Let’sseeonequickexample: set.comprehensions.py word='Hello' letters1=set(cforcinword) letters2={cforcinword} print(letters1)#prints:{'l','o','H','e'} print(letters1==letters2)#prints:True Noticehowforsetcomprehensions,asfordictionaries,duplicationisnotallowedand thereforetheresultingsethasonlyfourletters.Also,noticethattheexpressionsassigned toletters1andletters2produceequivalentsets. Thesyntaxusedtocreateletters2isverysimilartotheonewecanusetocreatea dictionarycomprehension.Youcanspotthedifferenceonlybythefactthatdictionaries requirekeysandvalues,separatedbycolumns,whilesetsdon’t. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Generators GeneratorsareoneverypowerfultoolthatPythongiftsuswith.Theyarebasedonthe conceptsofiteration,aswesaidbefore,andtheyallowforcodingpatternsthatcombine elegancewithefficiency. Generatorsareoftwotypes: Generatorfunctions:Theseareverysimilartoregularfunctions,butinsteadof returningresultsthroughreturnstatements,theyuseyield,whichallowsthemto suspendandresumetheirstatebetweeneachcall Generatorexpressions:Theseareverysimilartothelistcomprehensionswe’ve seeninthischapter,butinsteadofreturningalisttheyreturnanobjectthatproduces resultsonebyone WOW! eBook www.wowebook.org Generatorfunctions Generatorfunctionscomeunderallaspectslikeregularfunctions,withonedifference: insteadofcollectingresultsandreturningthematonce,theycanstartthecomputation, yieldonevalue,suspendtheirstatesavingeverythingtheyneedtobeabletoresumeand, ifcalledagain,resumeandperformanotherstep.Generatorfunctionsareautomatically turnedintotheirowniteratorsbyPython,soyoucancallnextonthem. Thisisallverytheoreticalso,let’smakeitclearwhysuchamechanismissopowerful, andthenlet’sseeanexample. SayIaskedyoutocountoutloudfrom1toamillion.Youstart,andatsomepointIask youtostop.Aftersometime,Iaskyoutoresume.Atthispoint,whatistheminimum informationyouneedtobeabletoresumecorrectly?Well,youneedtorememberthelast numberyoucalled.IfIstoppedyouafter31415,youwilljustgoonwith31416,andso on. Thepointis,youdon’tneedtorememberallthenumbersyousaidbefore31415,nordo youneedthemtobewrittendownsomewhere.Well,youmaynotknowit,butyou’re behavinglikeageneratoralready! Takeagoodlookatthefollowingcode: first.n.squares.py defget_squares(n):#classicfunctionapproach return[x**2forxinrange(n)] print(get_squares(10)) defget_squares_gen(n):#generatorapproach forxinrange(n): yieldx**2#weyield,wedon'treturn print(list(get_squares_gen(10))) Theresultoftheprintswillbethesame:[0,1,4,9,16,25,36,49,64,81].But thereisahugedifferencebetweenthetwofunctions.get_squaresisaclassicfunction thatcollectsallthesquaresofnumbersin[0,n)inalist,andreturnsit.Ontheotherhand, get_squares_genisagenerator,andbehavesverydifferently.Eachtimetheinterpreter reachestheyieldline,itsexecutionissuspended.Theonlyreasonthoseprintsreturnthe sameresultisbecausewefedget_squares_gentothelistconstructor,whichwhen calledlikethatexhauststhegeneratorcompletelybyaskingthenextelementuntila StopIterationisraised.Let’sseethisindetail: first.n.squares.manual.py defget_squares_gen(n): forxinrange(n): yieldx**2 squares=get_squares_gen(4)#thiscreatesageneratorobject print(squares)#<generatorobjectget_squares_genat0x7f158…> print(next(squares))#prints:0 print(next(squares))#prints:1 WOW! eBook www.wowebook.org print(next(squares))#prints:4 print(next(squares))#prints:9 #thefollowingraisesStopIteration,thegeneratorisexhausted, #anyfurthercalltonextwillkeepraisingStopIteration print(next(squares)) Intheprecedingcode,eachtimewecallnextonthegeneratorobject,weeitherstartit (firstnext)ormakeitresumefromthelastsuspensionpoint(anyothernext). Thefirsttimewecallnextonit,weget0,whichisthesquareof0,then1,then4,then9 andsincetheforloopstopsafterthat(nis4),thenthegeneratornaturallyends.Aclassic functionwouldatthatpointjustreturnNone,butinordertocomplywiththeiteration protocol,ageneratorwillinsteadraiseaStopIterationexception. Thisexplainshowaforloopworksforexample.Whenyoucallforkinrange(n), whathappensunderthehoodisthattheforloopgetsaniteratoroutofrange(n)and startscallingnextonit,untilStopIterationisraised,whichtellstheforloopthatthe iterationhasreacheditsend. Havingthisbehaviorbuilt-inineveryiterationaspectofPythonmakesgeneratorseven morepowerfulbecauseoncewewritethem,we’llbeabletoplugtheminwhatever iterationmechanismwewant. Atthispoint,you’reprobablyaskingyourselfwhywouldyouwanttouseagenerator insteadofaregularfunction.Well,thetitleofthischaptershouldsuggesttheanswer.I’ll talkaboutperformanceslater,sofornowlet’sconcentrateonanotheraspect:sometimes generatorsallowyoutodosomethingthatwouldn’tbepossiblewithasimplelist.For example,sayyouwanttoanalyzeallpermutationsofasequence.Ifthesequencehas lengthN,thenthenumberofitspermutationsisN!.Thismeansthatifthesequenceis10 elementslong,thenumberofpermutationsis3628800.Butasequenceof20elements wouldhave2432902008176640000permutations.Theygrowfactorially. Nowimagineyouhaveaclassicfunctionthatisattemptingtocalculateallpermutations, puttheminalist,andreturnittoyou.With10elements,itwouldrequireprobablyafew tensofseconds,butfor20elementsthereissimplynowaythatitcanbedone. Ontheotherhand,ageneratorfunctionwillbeabletostartthecomputationandgiveyou backthefirstpermutation,thenthesecond,andsoon.Ofcourseyouwon’thavethetime toparsethemall,theyaretoomany,butatleastyou’llbeabletoworkwithsomeofthem. Rememberwhenweweretalkingaboutthebreakstatementinforloops?Whenwefound anumberdividingacandidateprimewewerebreakingtheloop,noneedtogoon. Sometimesit’sexactlythesame,onlytheamountofdatayouhavetoiterateoverisso hugethatyoucannotkeepitallinmemoryinalist.Inthiscase,generatorsareinvaluable: theymakepossiblewhatwouldn’tbepossibleotherwise. So,inordertosavememory(andtime),usegeneratorfunctionswheneverpossible. It’salsoworthnotingthatyoucanusethereturnstatementinageneratorfunction.Itwill produceaStopIterationexceptiontoberaised,effectivelyendingtheiteration.Thisis extremelyimportant.Ifareturnstatementwereactuallytomakethefunctionreturn WOW! eBook www.wowebook.org something,itwouldbreaktheiterationprotocol.Pythonconsistencypreventsthis,and allowsusgreateasewhencoding.Let’sseeaquickexample: gen.yield.return.py defgeometric_progression(a,q): k=0 whileTrue: result=a*q**k ifresult<=100000: yieldresult else: return k+=1 forningeometric_progression(2,5): print(n) , ,…. Theprecedingcodeyieldsalltermsofthegeometricprogressiona,aq, Whentheprogressionproducesatermthatisgreaterthan100,000,thegeneratorstops (withareturnstatement).Runningthecodeproducesthefollowingresult: $pythongen.yield.return.py 2 10 50 250 1250 6250 31250 Thenexttermwouldhavebeen156250,whichistoobig. WOW! eBook www.wowebook.org Goingbeyondnext Atthebeginningofthischapter,Itoldyouthatgeneratorobjectsarebasedontheiteration protocol.We’llseeinthenextchapteracompleteexampleofhowtowriteacustom iterator/iterableobject.Fornow,Ijustwantyoutounderstandhownext()works. Whathappenswhenyoucallnext(generator)isthatyou’recallingthe generator.__next__()method.Remember,amethodisjustafunctionthatbelongstoan object,andobjectsinPythoncanhavespecialmethods.Ourfriend__next__()isjustone oftheseanditspurposeistoreturnthenextelementoftheiteration,ortoraise StopIterationwhentheiterationisoverandtherearenomoreelementstoreturn. Note InPython,anobject’sspecialmethodsarealsocalledmagicmethods,ordunder(from “doubleunderscore”)methods. Whenwewriteageneratorfunction,Pythonautomaticallytransformsitintoanobjectthat isverysimilartoaniterator,andwhenwecallnext(generator),thatcallistransformed ingenerator.__next__().Let’srevisitthepreviousexampleaboutgeneratingsquares: first.n.squares.manual.method.py defget_squares_gen(n): forxinrange(n): yieldx**2 squares=get_squares_gen(3) print(squares.__next__())#prints:0 print(squares.__next__())#prints:1 print(squares.__next__())#prints:4 #thefollowingraisesStopIteration,thegeneratorisexhausted, #anyfurthercalltonextwillkeepraisingStopIteration print(squares.__next__()) Theresultisexactlyasthepreviousexample,onlythistimeinsteadofusingtheproxycall next(squares),we’redirectlycallingsquares.__next__(). Generatorobjectshavealsothreeothermethodsthatallowcontrollingtheirbehavior: send,throw,andclose.sendallowsustocommunicateavaluebacktothegenerator object,whilethrowandcloserespectivelyallowraisinganexceptionwithinthegenerator andclosingit.TheiruseisquiteadvancedandIwon’tbecoveringthemhereindetail,but Iwanttospendafewwordsatleastaboutsend,withasimpleexample. Takealookatthefollowingcode: gen.send.preparation.py defcounter(start=0): n=start whileTrue: yieldn n+=1 WOW! eBook www.wowebook.org c=counter() print(next(c))#prints:0 print(next(c))#prints:1 print(next(c))#prints:2 Theprecedingiteratorcreatesageneratorobjectthatwillrunforever.Youcankeep callingit,itwillneverstop.Alternatively,youcanputitinaforloop,forexample,forn incounter():...anditwillgoonforeveraswell. Now,whatifyouwantedtostopitatsomepoint?Onesolutionistouseavariableto controlthewhileloop.Somethinglikethis: gen.send.preparation.stop.py stop=False defcounter(start=0): n=start whilenotstop: yieldn n+=1 c=counter() print(next(c))#prints:0 print(next(c))#prints:1 stop=True print(next(c))#raisesStopIteration Thiswilldoit.Westartwithstop=False,anduntilwechangeittoTrue,thegenerator willjustkeepgoing,likebefore.ThemomentwechangestoptoTruethough,thewhile loopwillexit,andthenextcallwillraiseaStopIterationexception.Thistrickworks, butIdon’tlikeit.Wedependonanexternalvariable,andthiscanleadtoissues:whatif anotherfunctionchangesthatstop?Moreover,thecodeisscattered.Inanutshell,this isn’tgoodenough. Wecanmakeitbetterbyusinggenerator.send().Whenwecallgenerator.send(),the valuethatwefeedtosendwillbepassedintothegenerator,executionisresumed,andwe canfetchitviatheyieldexpression.Thisisallverycomplicatedwhenexplainedwith words,solet’sseeanexample: gen.send.py defcounter(start=0): n=start whileTrue: result=yieldn#A print(type(result),result)#B ifresult=='Q': break n+=1 c=counter() print(next(c))#C print(c.send('Wow!'))#D print(next(c))#E print(c.send('Q'))#F WOW! eBook www.wowebook.org Executionoftheprecedingcodeproducesthefollowing: $pythongen.send.py 0 <class'str'>Wow! 1 <class'NoneType'>None 2 <class'str'>Q Traceback(mostrecentcalllast): File"gen.send.py",line14,in<module> print(c.send('Q'))#F StopIteration Ithinkit’sworthgoingthroughthiscodelinebyline,likeifwewereexecutingit,andsee ifwecanunderstandwhat’sgoingon. Westartthegeneratorexecutionwithacalltonext(#C).Withinthegenerator,nissetto thesamevalueofstart.Thewhileloopisentered,executionstops(#A)andn(0)is yieldedbacktothecaller.0isprintedontheconsole. Wethencallsend(#D),executionresumesandresultissetto'Wow!'(still#A),thenits typeandvalueareprintedontheconsole(#B).resultisnot'Q',thereforenis incrementedby1andexecutiongoesbacktothewhilecondition,which,beingTrue, evaluatestoTrue(thatwasn’thardtoguess,right?).Anotherloopcyclebegins,execution stopsagain(#A),andn(1)isyieldedbacktothecaller.1isprintedontheconsole. Atthispoint,wecallnext(#E),executionisresumedagain(#A),andbecausewearenot sendinganythingtothegeneratorexplicitly,Pythonbehavesexactlylikefunctionsthatare notusingthereturnstatement:theyieldnexpression(#A)returnsNone.result thereforeissettoNone,anditstypeandvalueareyetagainprintedontheconsole(#B). Executioncontinues,resultisnot'Q'sonisincrementedby1,andwestartanotherloop again.Executionstopsagain(#A)andn(2)isyieldedbacktothecaller.2isprintedonthe console. Andnowforthegrandfinale:wecallsendagain(#F),butthistimewepassin'Q', thereforewhenexecutionisresumed,resultissetto'Q'(#A).Itstypeandvalueare printedontheconsole(#B),andthenfinallytheifclauseevaluatestoTrueandthewhile loopisstoppedbythebreakstatement.Thegeneratornaturallyterminatesandthismeans aStopIterationexceptionisraised.Youcanseetheprintofitstracebackonthelastfew linesprintedontheconsole. Thisisnotatallsimpletounderstandatfirst,soifit’snotcleartoyou,don’tbe discouraged.Youcankeepreadingonandthenyoucancomebacktothisexampleafter sometime. Usingsendallowsforinterestingpatterns,andit’sworthnotingthatsendcanonlybe usedtoresumetheexecution,nottostartit.Onlynextstartstheexecutionofagenerator. WOW! eBook www.wowebook.org Theyieldfromexpression Anotherinterestingconstructistheyieldfromexpression.Thisexpressionallowsyouto yieldvaluesfromasubiterator.Itsuseallowsforquiteadvancedpatterns,solet’sjustsee averyquickexampleofit: gen.yield.for.py defprint_squares(start,end): forninrange(start,end): yieldn**2 forninprint_squares(2,5): print(n) Thepreviouscodeprintsthenumbers4,9,16ontheconsole(onseparatelines).Bynow,I expectyoutobeabletounderstanditbyyourself,butlet’squicklyrecapwhathappens. Theforloopoutsidethefunctiongetsaniteratorfromprint_squares(2,5)andcalls nextonituntiliterationisover.Everytimethegeneratoriscalled,executionissuspended (andlaterresumed)onyieldn**2,whichreturnsthesquareofthecurrentn. Let’sseehowwecantransformthiscodebenefitingfromtheyieldfromexpression: gen.yield.from.py defprint_squares(start,end): yieldfrom(n**2forninrange(start,end)) forninprint_squares(2,5): print(n) Thiscodeproducesthesameresult,butasyoucanseetheyieldfromisactuallyrunning asubiterator(n**2…).Theyieldfromexpressionreturnstothecallereachvaluethe subiteratorisproducing.It’sshorteranditreadsbetter. WOW! eBook www.wowebook.org Generatorexpressions Let’snowtalkabouttheothertechniquestogeneratevaluesoneatatime. Thesyntaxisexactlythesameaslistcomprehensions,only,insteadofwrappingthe comprehensionwithsquarebrackets,youwrapitwithroundbraces.Thatiscalleda generatorexpression. Ingeneral,generatorexpressionsbehavelikeequivalentlistcomprehensions,butthereis oneveryimportantthingtoremember:generatorsallowforoneiterationonly,thenthey willbeexhausted.Let’sseeanexample: generator.expressions.py >>>cubes=[k**3forkinrange(10)]#regularlist >>>cubes [0,1,8,27,64,125,216,343,512,729] >>>type(cubes) <class'list'> >>>cubes_gen=(k**3forkinrange(10))#createasgenerator >>>cubes_gen <generatorobject<genexpr>at0x7ff26b5db990> >>>type(cubes_gen) <class'generator'> >>>list(cubes_gen)#thiswillexhaustthegenerator [0,1,8,27,64,125,216,343,512,729] >>>list(cubes_gen)#nothingmoretogive [] Lookatthelineinwhichthegeneratorexpressioniscreatedandassignedthename cubes_gen.Youcanseeit’sageneratorobject.Inordertoseeitselements,wecanusea forloop,amanualsetofcallstonext,orsimply,feedittoalistconstructor,whichis whatIdid. Noticehow,oncethegeneratorhasbeenexhausted,thereisnowaytorecoverthesame elementsfromitagain.Weneedtorecreateit,ifwewanttouseitfromscratchagain. Inthenextfewexamples,let’sseehowtoreproducemapandfilterusinggenerator expressions: gen.map.py defadder(*n): returnsum(n) s1=sum(map(lambdan:adder(*n),zip(range(100),range(1,101)))) s2=sum(adder(*n)forninzip(range(100),range(1,101))) Inthepreviousexample,s1ands2areexactlythesame:theyarethesumofadder(0, 1),adder(1,2),adder(2,3),andsoon,whichtranslatestosum(1,3,5,...).The syntaxisdifferentthough,Ifindthegeneratorexpressiontobemuchmorereadable: gen.filter.py cubes=[x**3forxinrange(10)] odd_cubes1=filter(lambdacube:cube%2,cubes) WOW! eBook www.wowebook.org odd_cubes2=(cubeforcubeincubesifcube%2) Inthepreviousexample,odd_cubes1andodd_cubes2arethesame:theygeneratea sequenceofoddcubes.Yetagain,Ipreferthegeneratorsyntax.Thisshouldbeevident whenthingsgetalittlemorecomplicated: gen.map.filter.py N=20 cubes1=map( lambdan:(n,n**3), filter(lambdan:n%3==0orn%5==0,range(N)) ) cubes2=( (n,n**3)forninrange(N)ifn%3==0orn%5==0) Theprecedingcodecreatestogeneratorscubes1andcubes2.Theyareexactlythesame, andreturn2-tuples(n, )whennisamultipleof3or5. Ifyouprintthelist(cubes1),youget:[(0,0),(3,27),(5,125),(6,216),(9, 729),(10,1000),(12,1728),(15,3375),(18,5832)]. Seehowmuchbetterthegeneratorexpressionreads?Itmaybedebatablewhenthingsare verysimple,butassoonasyoustartnestingfunctionsabit,likewedidinthisexample, thesuperiorityofthegeneratorsyntaxisevident.Shorter,simpler,moreelegant. Now,letmeaskyouaquestion:whatisthedifferencebetweenthefollowinglinesof code? sum.example.py s1=sum([n**2forninrange(10**6)]) s2=sum((n**2forninrange(10**6))) s3=sum(n**2forninrange(10**6)) Strictlyspeaking,theyallproducethesamesum.Theexpressionstogets2ands3are exactlythesamebecausethebracesins2areredundant.Theyarebothgenerator expressionsinsidethesumfunction.Theexpressiontogets1isdifferentthough.Inside sum,wefindalistcomprehension.Thismeansthatinordertocalculates1,thesum functionhastocallnextonalist,amilliontimes. Doyouseewherewe’reloosingtimeandmemory?Beforesumcanstartcallingnexton thatlist,thelistneedstohavebeencreated,whichisawasteoftimeandspace.It’smuch betterforsumtocallnextonasimplegeneratorexpression.Thereisnoneedtohaveall thenumbersfromrange(10**6)storedinalist. So,watchoutforextraparentheseswhenyouwriteyourexpressions:sometimesit’seasy toskiponthesedetails,whichmakesourcodemuchdifferent.Don’tbelieveme? sum.example.2.py s=sum([n**2forninrange(10**8)])#thisiskilled #s=sum(n**2forninrange(10**8))#thissucceeds print(s) WOW! eBook www.wowebook.org Tryrunningtheprecedingexample.IfIrunthefirstline,thisiswhatIget: $pythonsum.example.2.py Killed Ontheotherhand,ifIcommentoutthefirstline,anduncommentthesecondone,thisis theresult: $pythonsum.example.2.py 333333328333333350000000 Sweetgeneratorexpressions.Thedifferencebetweenthetwolinesisthatinthefirstone,a listwiththesquaresofthefirsthundredmillionnumbersmustbemadebeforebeingable tosumthemup.Thatlistishuge,andwerunoutofmemory(atleast,myboxdid,ifyours doesn’ttryabiggernumber),thereforePythonkillstheprocessforus.Sadface. Butwhenweremovethesquarebrackets,wedon’tmakealistanymore.Thesum functionreceives0,1,4,9,andsoonuntilthelastone,andsumsthemup.Noproblems, happyface. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Someperformanceconsiderations So,we’veseenthatwehavemanydifferentwaystoachievethesameresult.Wecanuse anycombinationofmap,zip,filter,orchoosetogowithacomprehension,ormaybe choosetouseagenerator,eitherfunctionorexpression.Wemayevendecidetogowith forloops:whenthelogictoapplytoeachrunningparameterisn’tsimple,theymaybethe bestoption. Otherthanreadabilityconcernsthough,let’stalkaboutperformances.Whenitcomesto performances,usuallytherearetwofactorswhichplayamajorrole:spaceandtime. Spacemeansthesizeofthememorythatadatastructureisgoingtotakeup.Thebestway tochooseistoaskyourselfifyoureallyneedalist(ortuple)orifasimplegenerator functionwouldworkaswell.Iftheanswerisyes,gowiththegenerator,it’llsavealotof space.Samegoeswithfunctions:ifyoudon’tactuallyneedthemtoreturnalistortuple, thenyoucantransformthemingeneratorfunctionsaswell. Sometimes,youwillhavetouselists(ortuples),forexampletherearealgorithmsthat scansequencesusingmultiplepointersormaybetheyrunoverthesequencemorethan once.Ageneratorfunction(orexpression)canbeiteratedoveronlyonceandthenit’s exhausted,sointhesesituations,itwouldn’tbetherightchoice. Timeisabitharderthanspacebecauseitdependsonmorevariablesandthereforeitisn’t possibletostatethatXisfasterthanYwithabsolutecertaintyforallcases.However, basedontestsrunonPythontoday,wecansaythatmapcallscanbetwiceasfastas equivalentforloops,andlistcomprehensionscanbe(alwaysgenerallyspeaking)even fasterthanequivalentmapcalls. Inordertofullyappreciatethereasonbehindthesestatements,weneedtounderstandhow Pythonworks,andthisisabitoutsidethescopeofthisbook,forit’stootechnicalin detail.Let’sjustsaythatmapandlistcomprehensionsrunatClanguagespeedwithinthe interpreter,whileaPythonforloopisrunasPythonbytecodewithinthePythonVirtual Machine,whichisoftenmuchslower. Note ThereareseveraldifferentimplementationsofPython.Theoriginalone,andstillthemost commonone,istheonewritteninC.Cisoneofthemostpowerfulandpopular programminglanguagesstillusedtoday. TheseclaimsImadecomefrombooksandarticlesthatyoucanfindontheWeb,buthow aboutwedoasmallexerciseandtrytofindoutforourselves?Iwillwriteasmallpieceof codethatcollectstheresultsofdivmod(a,b)foracertainsetofintegerpairs(a,b).I willusethetimefunctionfromthetimemoduletocalculatetheelapsedtimeofthe operationsthatIwillperform.Let’sgo! performances.py fromtimeimporttime mx=5500#thisisthemaxIcouldreachwithmycomputer… WOW! eBook www.wowebook.org t=time()#starttimefortheforloop dmloop=[] forainrange(1,mx): forbinrange(a,mx): dmloop.append(divmod(a,b)) print('forloop:{:.4f}s'.format(time()-t))#elapsedtime t=time()#starttimeforthelistcomprehension dmlist=[ divmod(a,b)forainrange(1,mx)forbinrange(a,mx)] print('listcomprehension:{:.4f}s'.format(time()-t)) t=time()#starttimeforthegeneratorexpression dmgen=list( divmod(a,b)forainrange(1,mx)forbinrange(a,mx)) print('generatorexpression:{:.4f}s'.format(time()-t)) #verifycorrectnessofresultsandnumberofitemsineachlist print(dmloop==dmlist==dmgen,len(dmloop)) Asyoucansee,we’recreatingthreelists:dmloop,dmlist,dmgen(divmod-forloop, divmod-listcomprehension,divmod-generatorexpression).Westartwiththeslowest option,theforloops.Thenwehavealistcomprehension,andfinallyagenerator expression.Let’sseetheoutput: $pythonperformances.py forloop:4.3433s listcomprehension:2.7238s generatorexpression:3.1380s True15122250 Thelistcomprehensionrunsin63%ofthetimetakenbytheforloop.That’s impressive.Thegeneratorexpressioncamequiteclosetothat,withagood72%.The reasonthegeneratorexpressionissloweristhatweneedtofeedittothelist() constructorandthishasalittlebitmoreoverheadcomparedtoasheerlistcomprehension. Iwouldnevergowithageneratorexpressioninasimilarcasethough,thereisnopointif attheendwewantalist.Iwouldjustusealistcomprehension,andtheresultofthe previousexampleprovesmeright.Ontheotherhand,ifIjusthadtodothosedivmod calculationswithoutretainingtheresults,thenageneratorexpressionwouldbethewayto gobecauseinsuchasituationalistcomprehensionwouldunnecessarilyconsumealotof space. So,torecap:generatorsareveryfastandallowyoutosaveonspace.Listcomprehensions areingeneralevenfaster,butdon’tsaveonspace.PurePythonforloopsaretheslowest option.Let’sseeasimilarexamplethatcomparesaforloopandamapcall: performances.map.py fromtimeimporttime mx=2*10**7 t=time() WOW! eBook www.wowebook.org absloop=[] forninrange(mx): absloop.append(abs(n)) print('forloop:{:.4f}s'.format(time()-t)) t=time() abslist=[abs(n)forninrange(mx)] print('listcomprehension:{:.4f}s'.format(time()-t)) t=time() absmap=list(map(abs,range(mx))) print('map:{:.4f}s'.format(time()-t)) print(absloop==abslist==absmap) Thiscodeisconceptuallyverysimilartothepreviousexample.Theonlythingthathas changedisthatwe’reapplyingtheabsfunctioninsteadofthedivmodone,andwehave onlyoneloopinsteadoftwonestedones.Executiongivesthefollowingresult: $pythonperformances.map.py forloop:3.1283s listcomprehension:1.3966s map:1.2319s True Andmapwinstherace!AsItoldyoubefore,givingastatementofwhatisfasterthanwhat isverytricky.Inthiscase,themapcallisfasterthanthelistcomprehension. Apartfromthecasebycaselittledifferencesthough,it’squiteclearthattheforloop optionistheslowestone,solet’sseewhatarethereasonswestillwanttouseit. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Don’toverdocomprehensionsand generators We’veseenhowpowerfullistcomprehensionsandgeneratorexpressionscanbe.Andthey are,don’tgetmewrong,butthefeelingthatIhavewhenIdealwiththemisthattheir complexitygrowsexponentially.Themoreyoutrytodowithinasinglecomprehensionor ageneratorexpression,theharderitbecomestoread,understand,andthereforeto maintainorchange. OpenaPythonconsoleandtypeinimportthis,let’sreadtheZenofPythonagain,in particular,thereareafewlinesthatIthinkareveryimportanttokeepinmind: >>>importthis TheZenofPython,byTimPeters Beautifulisbetterthanugly. Explicitisbetterthanimplicit.# Simpleisbetterthancomplex.# Complexisbetterthancomplicated. Flatisbetterthannested. Sparseisbetterthandense. Readabilitycounts.# Specialcasesaren'tspecialenoughtobreaktherules. Althoughpracticalitybeatspurity. Errorsshouldneverpasssilently. Unlessexplicitlysilenced. Inthefaceofambiguity,refusethetemptationtoguess. Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit. Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch. Nowisbetterthannever. Althoughneverisoftenbetterthan*right*now. Iftheimplementationishardtoexplain,it'sabadidea.# Iftheimplementationiseasytoexplain,itmaybeagoodidea. Namespacesareonehonkinggreatidea—let'sdomoreofthose! Ihaveputacommentsignontherightofthemainfocuspointshere.Comprehensionsand generatorexpressionsbecomehardtoread,moreimplicitthanexplicit,complex,andthey canbehardtoexplain.Sometimesyouhavetobreakthemapartusingtheinside-out technique,tounderstandwhytheyproducetheresulttheyproduce. Togiveyouanexample,let’stalkabitmoreaboutPythagoreantriples.Justtoremind you,aPythagoreantripleisatupleofpositiveintegers(a,b,c)suchthat . Wesawearlierinthischapterhowtocalculatethem,butwediditinaveryinefficient waybecausewewerescanningallpairsofnumbersbelowacertainthreshold,calculating thehypotenuse,andfilteringoutthosethatwerenotproducingatriple. AbetterwaytogetalistofPythagoreantriplesistodirectlygeneratethem.Thereare manydifferentformulastodothisandwe’lluseoneofthem:theEuclideanformula. WOW! eBook www.wowebook.org Thisformulasaysthatanytriple(a,b,c),where ,b=2mn, ,with mandnpositiveintegerssuchthatm>n,isaPythagoreantriple.Forexample,whenm= 2andn=1,wefindthesmallesttriple:(3,4,5). Thereisonecatchthough:considerthetriple(6,8,10),thatisjustlike(3,4,5)withall ,but thenumbersmultipliedby2.ThistripleisdefinitelyPythagorean,since wecanderiveitfrom(3,4,5)simplybymultiplyingeachofitselementsby2.Samegoes for(9,12,15),(12,16,20),andingeneralforallthetriplesthatwecanwriteas(3k,4k, 5k),withkbeingapositiveintegergreaterthan1. Atriplethatcannotbeobtainedbymultiplyingtheelementsofanotheronebysomefactor k,iscalledprimitive.Anotherwayofstatingthisis:ifthethreeelementsofatripleare coprime,thenthetripleisprimitive.Twonumbersarecoprimewhentheydon’tshareany primefactoramongsttheirdivisors,thatis,theirgreatestcommondivisor(GCD)is1. Forexample,3and5arecoprime,while3and6arenot,becausetheyarebothdivisible by3. So,theEuclideanformulatellsusthatifmandnarecoprime,andm–nisodd,thetriple theygenerateisprimitive.Inthefollowingexample,wewillwriteageneratorexpression tocalculatealltheprimitivePythagoreantripleswhosehypotenuse(c)islessthanorequal tosomeintegerN.Thismeanswewantalltriplesforwhich ,whichmeanswecanapproximatethecalculationwith formulalookslikethis: anupperboundof .Whennis1,the . So,torecap:mmustbegreaterthann,theymustalsobecoprime,andtheirdifferencemnmustbeodd.Moreover,inordertoavoiduselesscalculationswe’llputtheupperbound formatfloor(sqrt(N))+1. Note Thefunctionfloorforarealnumberxgivesthemaximumintegernsuchthatn<x,for example,floor(3.8)=3,floor(13.1)=13.Takingthefloor(sqrt(N))+1meanstakingthe integerpartofthesquarerootofNandaddingaminimalmarginjusttomakesurewe don’tmissoutanynumber. Let’sputallofthisintocode,stepbystep.Let’sstartbywritingasimplegcdfunctionthat usesEuclid’salgorithm: functions.py defgcd(a,b): """CalculatetheGreatestCommonDivisorof(a,b).""" whileb!=0: a,b=b,a%b returna TheexplanationofEuclid’salgorithmisavailableontheWeb,soIwon’tspendanytime heretalkingaboutit;weneedtoconcentrateonthegeneratorexpression.Thenextstepis tousetheknowledgewegatheredbeforetogeneratealistofprimitivePythagorean WOW! eBook www.wowebook.org triples: pythagorean.triple.generation.py fromfunctionsimportgcd N=50 triples=sorted(#1 ((a,b,c)fora,b,cin(#2 ((m**2-n**2),(2*m*n),(m**2+n**2))#3 forminrange(1,int(N**.5)+1)#4 forninrange(1,m)#5 if(m-n)%2andgcd(m,n)==1#6 )ifc<=N),key=lambda*triple:sum(*triple)#7 ) print(triples) Thereyougo.It’snoteasytoread,solet’sgothroughitlinebyline.At#3,westarta generatorexpressionthatiscreatingtriples.Youcanseefrom#4and#5thatwe’relooping onmin[1,M]withMbeingtheintegerpartofsqrt(N),plus1.Ontheotherhand,nloops within[1,m),torespectthem>nrule.WorthnotinghowIcalculatedsqrt(N),thatis, N**.5,whichisjustanotherwaytodoitthatIwantedtoshowyou. At#6,youcanseethefilteringconditionstomakethetriplesprimitive:(m-n)%2 evaluatestoTruewhen(m-n)isodd,andgcd(m,n)==1meansmandnarecoprime. Withtheseinplace,weknowthetripleswillbeprimitive.Thistakescareoftheinnermost generatorexpression.Theoutermostonestartsat#2,andfinishesat#7.Wetakethetriples (a,b,c)in(…innermostgenerator…)suchthatc<=N.Thisisnecessarybecause isthelowestupperboundthatwecanapply,butitdoesn’tguaranteethatcwill actuallybelessthanorequaltoN. Finally,at#1weapplysorting,topresentthelistinorder.At#7,aftertheoutermost generatorexpressionisclosed,youcanseethatwespecifythesortingkeytobethesuma +b+c.Thisisjustmypersonalpreference,thereisnomathematicalreasonbehindit. So,whatdoyouthink?Wasitstraightforwardtoread?Idon’tthinkso.Andbelieveme, thisisstillasimpleexample;Ihaveseenexpressionswaymorecomplicatedthanthisone. Unfortunatelysomeprogrammersthinkthatwritingcodelikethisiscool,thatit’ssome sortofdemonstrationoftheirsuperiorintellectualpowers,oftheirabilitytoquicklyread anddigestintricatecode. Withinaprofessionalenvironmentthough,Ifindmyselfhavingmuchmorerespectfor thosewhowriteefficient,cleancode,andmanagetokeepegooutthedoor.Conversely, thosewhodon’t,willproducelinesatwhichyouwillstareforalongtimewhileswearing inthreelanguages(atleastthisiswhatIdo). Now,let’sseeifwecanrewritethiscodeintosomethingeasiertoread: pythagorean.triple.generation.for.py fromfunctionsimportgcd WOW! eBook www.wowebook.org defgen_triples(N): forminrange(1,int(N**.5)+1):#1 forninrange(1,m):#2 if(m-n)%2andgcd(m,n)==1:#3 c=m**2+n**2#4 ifc<=N:#5 a=m**2-n**2#6 b=2*m*n#7 yield(a,b,c)#8 triples=sorted( gen_triples(50),key=lambda*triple:sum(*triple))#9 print(triples) Ifeelsomuchbetteralready.Let’sgothroughthiscodeaswell,linebyline.You’llsee howeasieritistounderstand. Westartloopingat#1and#2,inexactlythesamewaywewereloopingintheprevious example.Online#3,wehavethefilteringforprimitivetriples.Online#4,wedeviatea bitfromwhatweweredoingbefore:wecalculatec,andonline#5,wefilteroncbeing lessthanorequaltoN.Onlywhencsatisfiesthatcondition,wecalculateaandb,and yieldtheresultingtuple.It’salwaysgoodtodelayallcalculationsforasmuchaspossible sothatwedon’twastetime,incaseeventuallywehavetodiscardthoseresults. Onthelastline,beforeprintingtheresult,weapplysortingwiththesamekeywewere usinginthegeneratorexpressionexample. Ihopeyouagree,thisexampleiseasiertounderstand.AndIpromiseyou,ifyouhaveto modifythecodeoneday,you’llfindthatmodifyingthisoneiseasy,whiletomodifythe otherversionwilltakemuchlonger(anditwillbemoreerrorprone). Bothexamples,whenrun,printthefollowing: $pythonpythagorean.triple.generation.py [(3,4,5),(5,12,13),(15,8,17),(7,24,25),(21,20,29),(35,12, 37),(9,40,41)] Themoralofthestoryis,tryandusecomprehensionsandgeneratorexpressionsasmuch asyoucan,butifthecodestartstobecomplicatedtomodifyortoread,youmaywantto refactorintosomethingmorereadable.Thereisnothingwrongwiththis. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Namelocalization Nowthatwearefamiliarwithalltypesofcomprehensionsandgeneratorexpression,let’s talkaboutnamelocalizationwithinthem.Python3.*localizesloopvariablesinallfour formsofcomprehensions:list,dict,set,andgeneratorexpressions.Thisbehavioris thereforedifferentfromthatoftheforloop.Let’sseeasimpleexampletoshowallthe cases: scopes.py A=100 ex1=[AforAinrange(5)] print(A)#prints:100 ex2=list(AforAinrange(5)) print(A)#prints:100 ex3=dict((A,2*A)forAinrange(5)) print(A)#prints:100 ex4=set(AforAinrange(5)) print(A)#prints:100 s=0 forAinrange(5): s+=A print(A)#prints:4 Intheprecedingcode,wedeclareaglobalnameA=100,andthenweexercisethefour comprehensions:list,generatorexpression,dictionary,andset.Noneofthemalterthe globalnameA.Conversely,youcanseeattheendthattheforloopmodifiesit.Thelast printstatementprints4. Let’sseewhathappensifAwasn’tthere: scopes.noglobal.py ex1=[AforAinrange(5)] print(A)#breaks:NameError:name'A'isnotdefined Theprecedingcodewouldworkthesamewithanyofthefourtypesofcomprehensions. Afterwerunthefirstline,Aisnotdefinedintheglobalnamespace. Onceagain,theforloopbehavesdifferently: scopes.for.py s=0 forAinrange(5): s+=A print(A)#prints:4 print(globals()) Theprecedingcodeshowsthatafteraforloop,iftheloopvariablewasn’tdefinedbefore it,wecanfinditintheglobalframe.Tomakesureofit,let’stakeapeekatitbycalling WOW! eBook www.wowebook.org theglobals()built-infunction: $pythonscopes.for.py 4 {'__spec__':None,'__name__':'__main__','s':10,'A':4,'__doc__': None,'__cached__':None,'__package__':None,'__file__':'scopes.for.py', '__loader__':<_frozen_importlib.SourceFileLoaderobjectat 0x7f05a5a183c8>,'__builtins__':<module'builtins'(built-in)>} Togetherwithalotofotherboilerplatestuff,wecanspot'A':4. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Generationbehaviorinbuilt-ins Amongstthebuilt-intypes,thegenerationbehaviorisnowquitecommon.Thisisamajor differencebetweenPython2andPython3.Alotoffunctionssuchasmap,zip,and filterhavebeentransformedsothattheyreturnobjectsthatbehavelikeiterables.The ideabehindthischangeisthatifyouneedtomakealistofthoseresultsyoucanalways wrapthecallinalist()class,andyou’redone.Ontheotherhand,ifyoujustneedto iterateandwanttokeeptheimpactonmemoryaslightaspossible,youcanusethose functionssafely. Anothernotableexampleistherangefunction.InPython2itreturnsalist,andthereis anotherfunctioncalledxrangethatreturnsanobjectthatyoucaniterateon,which generatesthenumbersonthefly.InPython3thisfunctionhasgone,andrangenow behaveslikeit. Butthisconceptingeneralisnowquitewidespread.Youcanfinditintheopen() function,whichisusedtooperateonfileobjects(we’llseeitinoneofthenextchapters), butalsoinenumerate,inthedictionarykeys,values,anditemsmethods,andseveral otherplaces. Itallmakessense:Python’saimistotryandreducethememoryfootprintbyavoiding wastingspacewhereverispossible,especiallyinthosefunctionsandmethodsthatare usedextensivelyinmostsituations. Doyourememberatthebeginningofthischapter?Isaidthatitmakesmoresenseto optimizetheperformancesofcodethathastodealwithalotofobjects,ratherthan shavingoffafewmillisecondsfromafunctionthatwecalltwiceaday. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Onelastexample Beforewepartfromthischapter,I’llshowyouasimpleproblemthatIsubmittedto candidatesforaPythondeveloperroleinacompanyIusedtoworkfor. Theproblemisthefollowing:giventhesequence01123581321…writeafunction thatwouldreturnthetermsofthissequenceuptosomelimitN. Ifyouhaven’trecognizedit,thatistheFibonaccisequence,whichisdefinedasF(0)=0, F(1)=1and,foranyn>1,F(n)=F(n-1)+F(n-2).Thissequenceisexcellenttotest knowledgeaboutrecursion,memoizationtechniquesandothertechnicaldetails,butinthis caseitwasagoodopportunitytocheckwhetherthecandidateknewaboutgenerators(and toomanysocalledPythoncodersdidn’t,whenIwasinterviewingthem). Let’sstartfromarudimentaryversionofafunction,andthenimproveonit: fibonacci.first.py deffibonacci(N): """ReturnallfibonaccinumbersuptoN.""" result=[0] next_n=1 whilenext_n<=N: result.append(next_n) next_n=sum(result[-2:]) returnresult print(fibonacci(0))#[0] print(fibonacci(1))#[0,1,1] print(fibonacci(50))#[0,1,1,2,3,5,8,13,21,34] Fromthetop:wesetuptheresultlisttoastartingvalueof[0].Thenwestartthe iterationfromthenextelement(next_n),whichis1.Whilethenextelementisnotgreater thanN,wekeepappendingittothelistandcalculatingthenext.Wecalculatethenext elementbytakingasliceofthelasttwoelementsintheresultlistandpassingittothe sumfunction.Addsomeprintstatementshereandthereifthisisnotcleartoyou,butby nowIwouldexpectitnottobeanissue. WhentheconditionofthewhileloopevaluatestoFalse,weexittheloopandreturn result.Youcanseetheresultofthoseprintstatementsinthecommentsnexttoeachof them. Atthispoint,Iwouldaskthecandidatethefollowingquestion:“WhatifIjustwantedto iterateoverthosenumbers?”Agoodcandidatewouldthenchangethecodelikethenext listing(anexcellentcandidatewouldhavestartedwithit!): fibonacci.second.py deffibonacci(N): """ReturnallfibonaccinumbersuptoN.""" yield0 ifN==0: return WOW! eBook www.wowebook.org a=0 b=1 whileb<=N: yieldb a,b=b,a+b print(list(fibonacci(0)))#[0] print(list(fibonacci(1)))#[0,1,1] print(list(fibonacci(50)))#[0,1,1,2,3,5,8,13,21,34] ThisisactuallyoneofthesolutionsIwasgiven.Idon’tknowwhyIkeptit,butI’mgladI didsoIcanshowittoyou.Now,thefibonaccifunctionisageneratorfunction.Firstwe yield0,thenifNis0wereturn(thiswillcauseaStopIterationexceptiontoberaised).If that’snotthecase,westartiterating,yieldingbateveryloopcycle,andthenupdatinga andb.Allweneedinordertobeabletoproducethenextelementofthesequenceisthe pasttwo:aandb,respectively. Thiscodeismuchbetter,hasalightermemoryfootprintandallwehavetodotogetalist ofFibonaccinumbersistowrapthecallwithlist(),asusual. Butwhataboutelegance?Icannotleavethecodelikethat.Itwasdecentforaninterview, wherethefocusismoreonfunctionalitythanelegance,buthereI’dliketoshowyoua nicerversion: fibonacci.elegant.py deffibonacci(N): """ReturnallfibonaccinumbersuptoN.""" a,b=0,1 whilea<=N: yielda a,b=b,a+b Muchbetter.Thewholebodyofthefunctionisfourlines,fiveifyoucountthedocstring. Noticehowinthiscaseusingtupleassignment(a,b=0,1anda,b=b,a+b) helpsinmakingthecodeshorter,andmorereadable.It’soneofthefeaturesofPythonI likealot. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,weexploredtheconceptofiterationandgenerationabitmoredeeply.We sawthemap,zipandfilterfunctionsquiteindetail,andhowtousethemasan alternativetoaregularforloopapproach. Thenwesawtheconceptofcomprehensions,forlists,dictionaries,andsets.Wesawtheir syntaxandhowtousethemasanalternativetoboththeclassicforloopapproachand alsototheuseofmap,zip,andfilterfunctions. Finally,wetalkedabouttheconceptofgeneration,intwoforms:generatorfunctionsand expressions.Welearnedhowtosavetimeandspacebyusinggenerationtechniquesand sawhowtheycanmakepossiblewhatwouldn’tnormallybeifweusedaconventional approachbasedonlists. Wetalkedaboutperformances,andsawthatforloopsarelastintermsofspeed,butthey providethebestreadabilityandflexibilitytochange.Ontheotherhand,functionssuchas mapandfiltercanbemuchfaster,andcomprehensionsmaybeevenbetter. Thecomplexityofthecodewrittenusingthesetechniquesgrowsexponentiallyso,in ordertofavorreadabilityandeaseofmaintainability,westillneedtousetheclassicfor loopapproachattimes.Anotherdifferenceisinthenamelocalization,wheretheforloop behavesdifferentlyfromallothertypesofcomprehensions. Thenextchapterwillbeallaboutobjectsandclasses.Structurallysimilartothisone,in thatwewon’texploremanydifferentsubjects,rather,justafewofthem,butwe’lltryto divealittlebitmoredeeply. Makesureyouunderstandwelltheconceptsofthischapterbeforejumpingtothenext one.We’rebuildingawallbrickbybrick,andifthefoundationisnotsolid,wewon’tget veryfar. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter6.AdvancedConcepts–OOP, Decorators,andIterators “Laclassenonèacqua.(Classwillout)” —Italiansaying Icouldprobablywriteasmallbookaboutobject-orientedprogramming(referredtoas OOPhenceforth)andclasses.Inthischapter,I’mfacingthehardchallengeoffindingthe balancebetweenbreadthanddepth.Therearesimplytoomanythingstotell,andthere’s plentyofthemthatwouldtakemorethanthiswholechapterifIdescribedthemalonein depth.Therefore,IwilltrytogiveyouwhatIthinkisagoodpanoramicviewofthe fundamentals,plusafewthingsthatmaycomeinhandyinthenextchapters.Python’s officialdocumentationwillhelpinfillingthegaps. We’regoingtoexplorethreeimportantconceptsinthischapter:decorators,OOP,and iterators. WOW! eBook www.wowebook.org Decorators Inthepreviouschapter,Imeasuredtheexecutiontimeofvariousexpressions.Ifyou recall,Ihadtoinitializeavariabletothestarttime,andsubtractitfromthecurrenttime afterexecutioninordertocalculatetheelapsedtime.Ialsoprinteditontheconsoleafter eachmeasurement.Thatwasverytedious. Everytimeyoufindyourselfrepeatingthings,analarmbellshouldgooff.Canyouput thatcodeinafunctionandavoidrepetition?Theanswermostofthetimeisyes,solet’s lookatanexample. decorators/time.measure.start.py fromtimeimportsleep,time deff(): sleep(.3) defg(): sleep(.5) t=time() f() print('ftook:',time()-t)#ftook:0.3003859519958496 t=time() g() print('gtook:',time()-t)#gtook:0.5005719661712646 Intheprecedingcode,Idefinedtwofunctions,fandg,whichdonothingbutsleep(by0.3 and0.5secondsrespectively).Iusedthesleepfunctiontosuspendtheexecutionforthe desiredamountoftime.Ialsohighlightedhowwecalculatethetimeelapsedbysettingt tothecurrenttimeandthensubtractingitwhenthetaskisdone.Youcanseethatthe measureisprettyaccurate. Now,howdoweavoidrepeatingthatcodeandthosecalculations?Onefirstpotential approachcouldbethefollowing: decorators/time.measure.dry.py fromtimeimportsleep,time deff(): sleep(.3) defg(): sleep(.5) defmeasure(func): t=time() func() print(func.__name__,'took:',time()-t) measure(f)#ftook:0.30041074752807617 WOW! eBook www.wowebook.org measure(g)#gtook:0.5006198883056641 Ah,muchbetternow.Thewholetimingmechanismhasbeenencapsulatedintoafunction sowedon’trepeatcode.Weprintthefunctionnamedynamicallyandit’seasyenoughto code.Whatifweneedtopassargumentstothefunctionwemeasure?Thiscodewouldget justabitmorecomplicated,solet’sseeanexample. decorators/time.measure.arguments.py fromtimeimportsleep,time deff(sleep_time=0.1): sleep(sleep_time) defmeasure(func,*args,**kwargs): t=time() func(*args,**kwargs) print(func.__name__,'took:',time()-t) measure(f,sleep_time=0.3)#ftook:0.3004162311553955 measure(f,0.2)#ftook:0.20028162002563477 Now,fisexpectingtobefedsleep_time(withadefaultvalueof0.1).Ialsohadto changethemeasurefunctionsothatitisnowacceptingafunction,anyvariablepositional arguments,andanyvariablekeywordarguments.Inthisway,whateverwecallmeasure with,weredirectthoseargumentstothecalltofwedoinside. Thisisverygood,butwecanpushitalittlebitfurther.Let’ssaywewanttosomehow havethattimingbehaviorbuilt-inintheffunction,sothatwecouldjustcallitandhave thatmeasuretaken.Here’showwecoulddoit: decorators/time.measure.deco1.py fromtimeimportsleep,time deff(sleep_time=0.1): sleep(sleep_time) defmeasure(func): defwrapper(*args,**kwargs): t=time() func(*args,**kwargs) print(func.__name__,'took:',time()-t) returnwrapper f=measure(f)#decorationpoint f(0.2)#ftook:0.2002875804901123 f(sleep_time=0.3)#ftook:0.3003721237182617 print(f.__name__)#wrapper<-ouch! Theprecedingcodeisprobablynotsostraightforward.Iconfessthat,eventoday,it sometimesrequiresmesomeseriousconcentrationtounderstandsomedecorators,they canbeprettynasty.Let’sseewhathappenshere.Themagicisinthedecorationpoint. Webasicallyreassignfwithwhateverisreturnedbymeasurewhenwecallitwithfasan WOW! eBook www.wowebook.org argument.Withinmeasure,wedefineanotherfunction,wrapper,andthenwereturnit.So, theneteffectisthatafterthedecorationpoint,whenwecallf,we’reactuallycalling wrapper.Sincethewrapperinsideiscallingfunc,whichisf,weareactuallyclosingthe looplikethat.Ifyoudon’tbelieveme,takealookatthelastline. wrapperisactually…awrapper.Ittakesvariableandpositionalarguments,andcallsf withthem.Italsodoesthetimemeasurementtrickaroundthecall. Thistechniqueiscalleddecoration,andmeasureis,atalleffects,adecorator.This paradigmbecamesopopularandwidelyusedthatatsomepoint,Pythonaddedaspecial syntaxforit(checkPEP318).Let’sexplorethreecases:onedecorator,twodecorators, andonedecoratorthattakesarguments. decorators/syntax.py deffunc(arg1,arg2,...): pass func=decorator(func) #isequivalenttothefollowing: @decorator deffunc(arg1,arg2,...): pass Basically,insteadofmanuallyreassigningthefunctiontowhatwasreturnedbythe decorator,weprependthedefinitionofthefunctionwiththespecialsyntax @decorator_name. Wecanapplymultipledecoratorstothesamefunctioninthefollowingway: decorators/syntax.py deffunc(arg1,arg2,...): pass func=deco1(deco2(func)) #isequivalenttothefollowing: @deco1 @deco2 deffunc(arg1,arg2,...): pass Whenapplyingmultipledecorators,payattentiontotheorder,shoulditmatter.Inthe precedingexample,funcisdecoratedwithdeco2first,andtheresultisdecoratedwith deco1.Agoodruleofthumbis:thecloserthedecoratortothefunction,thesooneritis applied. Somedecoratorscantakearguments.Thistechniqueisgenerallyusedtoproduceother decorators.Let’slookatthesyntax,andthenwe’llseeanexampleofit. decorators/syntax.py deffunc(arg1,arg2,...): WOW! eBook www.wowebook.org pass func=decoarg(argA,argB)(func) #isequivalenttothefollowing: @decoarg(argA,argB) deffunc(arg1,arg2,...): pass Asyoucansee,thiscaseisabitdifferent.Firstdecoargiscalledwiththegiven arguments,andthenitsreturnvalue(theactualdecorator)iscalledwithfunc.BeforeI giveyouanotherexample,let’sfixonethingthatisbotheringme.Idon’twanttolosethe originalfunctionnameanddocstring(andtheotherattributesaswell,checkthe documentationforthedetails)whenIdecorateit.Butbecauseinsideourdecoratorwe returnwrapper,theoriginalattributesfromfuncarelostandfendsupbeingassignedthe attributesofwrapper.Thereisaneasyfixforthatfromfunctools,awonderfulmodule fromthePythonstandardlibrary.Iwillfixthelastexample,andIwillalsorewriteits syntaxtousethe@operator. decorators/time.measure.deco2.py fromtimeimportsleep,time fromfunctoolsimportwraps defmeasure(func): @wraps(func) defwrapper(*args,**kwargs): t=time() func(*args,**kwargs) print(func.__name__,'took:',time()-t) returnwrapper @measure deff(sleep_time=0.1): """I'macat.Ilovetosleep!""" sleep(sleep_time) f(sleep_time=0.3)#ftook:0.30039525032043457 print(f.__name__,':',f.__doc__) #f:I'macat.Ilovetosleep! Nowwe’retalking!Asyoucansee,allweneedtodoistotellPythonthatwrapper actuallywrapsfunc(bymeansofthewrapsfunction),andyoucanseethattheoriginal nameanddocstringarenowmaintained. Let’sseeanotherexample.Iwantadecoratorthatprintsanerrormessagewhentheresult ofafunctionisgreaterthanathreshold.Iwillalsotakethisopportunitytoshowyouhow toapplytwodecoratorsatonce. decorators/two.decorators.py fromtimeimportsleep,time fromfunctoolsimportwraps defmeasure(func): WOW! eBook www.wowebook.org @wraps(func) defwrapper(*args,**kwargs): t=time() result=func(*args,**kwargs) print(func.__name__,'took:',time()-t) returnresult returnwrapper defmax_result(func): @wraps(func) defwrapper(*args,**kwargs): result=func(*args,**kwargs) ifresult>100: print('Resultistoobig({0}).Maxallowedis100.' .format(result)) returnresult returnwrapper @measure @max_result defcube(n): returnn**3 print(cube(2)) print(cube(5)) Tip Takeyourtimeinstudyingtheprecedingexampleuntilyouaresureyouunderstandit well.Ifyoudo,Idon’tthinkthereisanydecoratoryouwon’tbeabletowriteafterwards. Ihadtoenhancethemeasuredecorator,sothatitswrappernowreturnstheresultofthe calltofunc.Themax_resultdecoratordoesthataswell,butbeforereturning,itchecks thatresultisnotgreaterthan100,whichisthemaximumallowed. Idecoratedcubewithbothofthem.First,max_resultisapplied,thenmeasure.Running thiscodeyieldsthisresult: $pythontwo.decorators.py cubetook:7.62939453125e-06# 8# Resultistoobig(125).Maxallowedis100. cubetook:1.1205673217773438e-05 125 Foryourconvenience,Iputa#totherightoftheresultsofthefirstcall:print(cube(2)). Theresultis8,andthereforeitpassesthethresholdchecksilently.Therunningtimeis measuredandprinted.Finally,weprinttheresult(8). Onthesecondcall,theresultis125,sotheerrormessageisprinted,theresultreturned, andthenit’stheturnofmeasure,whichprintstherunningtimeagain,andfinally,weprint theresult(125). HadIdecoratedthecubefunctionwiththesametwodecoratorsbutinadifferentorder, theerrormessagewouldfollowthelinethatprintstherunningtime,insteadofpreceding WOW! eBook www.wowebook.org it. WOW! eBook www.wowebook.org Adecoratorfactory Let’ssimplifythisexamplenow,goingbacktoasingledecorator:max_result.Iwantto makeitsothatIcandecoratedifferentfunctionswithdifferentthresholds,andIdon’t wanttowriteonedecoratorforeachthreshold.Let’samendmax_resultsothatitallows ustodecoratefunctionsspecifyingthethresholddynamically. decorators/decorators.factory.py fromfunctoolsimportwraps defmax_result(threshold): defdecorator(func): @wraps(func) defwrapper(*args,**kwargs): result=func(*args,**kwargs) ifresult>threshold: print( 'Resultistoobig({0}).Maxallowedis{1}.' .format(result,threshold)) returnresult returnwrapper returndecorator @max_result(75) defcube(n): returnn**3 print(cube(5)) Thisprecedingcodeshowsyouhowtowriteadecoratorfactory.Ifyourecall, decoratingafunctionwithadecoratorthattakesargumentsisthesameaswritingfunc= decorator(argA,argB)(func),sowhenwedecoratecubewithmax_result(75),we’re doingcube=max_result(75)(cube). Let’sgothroughwhathappens,stepbystep.Whenwecallmax_result(75),weenterits body.Adecoratorfunctionisdefinedinside,whichtakesafunctionasitsonlyargument. Insidethatfunction,theusualdecoratortrickisperformed.Wedefineawrapper,insideof whichwechecktheresultoftheoriginalfunction’scall.Thebeautyofthisapproachis thatfromtheinnermostlevel,wecanstillrefertobothfuncandthreshold,whichallows ustosetthethresholddynamically. wrapperreturnsresult,decoratorreturnswrapper,andmax_resultreturnsdecorator. Thismeansthatourcallcube=max_result(75)(cube),actuallybecomescube= decorator(cube).Notjustanydecoratorthough,butoneforwhichthresholdhasthe value75.Thisisachievedbyamechanismcalledclosure,whichisoutsideofthescopeof thischapterbutnonethelessveryinteresting,soImentioneditforyoutodosomeresearch onit. Runningthelastexampleproducesthefollowingresult: $pythondecorators.factory.py WOW! eBook www.wowebook.org Resultistoobig(125).Maxallowedis75. 125 Theprecedingcodeallowsmetousethemax_resultdecoratorwithdifferentthresholds atmyownwill,likethis: decorators/decorators.factory.py @max_result(75) defcube(n): returnn**3 @max_result(100) defsquare(n): returnn**2 @max_result(1000) defmultiply(a,b): returna*b Notethateverydecorationusesadifferentthresholdvalue. DecoratorsareverypopularinPython.Theyareusedquiteoftenandtheysimplify(and beautify,Idaresay)thecodealot. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Object-orientedprogramming It’sbeenquitealongandhopefullynicejourneyand,bynow,weshouldbereadyto exploreobject-orientedprogramming.I’llusethedefinitionfromKindler,E.;Krivy,I. (2011).Object-OrientedSimulationofsystemswithsophisticatedcontrol.International JournalofGeneralSystems,andadaptittoPython: Object-orientedprogramming(OOP)isaprogrammingparadigmbasedonthe conceptof“objects”,whicharedatastructuresthatcontaindata,intheformof attributes,andcode,intheformoffunctionsknownasmethods.Adistinguishing featureofobjectsisthatanobject’smethodcanaccessandoftenmodifythedata attributesoftheobjectwithwhichtheyareassociated(objectshaveanotionof “self”).InOOprogramming,computerprogramsaredesignedbymakingthemout ofobjectsthatinteractwithoneanother. Pythonhasfullsupportforthisparadigm.Actually,aswehavealreadysaid,everythingin Pythonisanobject,sothisshowsthatOOPisnotjustsupportedbyPython,butit’spartof itsverycore. ThetwomainplayersinOOPareobjectsandclasses.Classesareusedtocreateobjects (objectsareinstancesoftheclasseswithwhichtheywerecreated),sowecouldseethem asinstancefactories.Whenobjectsarecreatedbyaclass,theyinherittheclassattributes andmethods.Theyrepresentconcreteitemsintheprogram’sdomain. WOW! eBook www.wowebook.org ThesimplestPythonclass IwillstartwiththesimplestclassyoucouldeverwriteinPython. oop/simplest.class.py classSimplest():#whenempty,thebracesareoptional pass print(type(Simplest))#whattypeisthisobject? simp=Simplest()#wecreateaninstanceofSimplest:simp print(type(simp))#whattypeissimp? #issimpaninstanceofSimplest? print(type(simp)==Simplest)#There'sabetterwayforthis Let’sruntheprecedingcodeandexplainitlinebyline: $pythonoop/simplest.class.py <class'type'> <class'__main__.Simplest'> True TheSimplestclassIdefinedonlyhasthepassinstructionforitsbody,whichmeansit doesn’thaveanycustomattributesormethods.Iwillprintitstype(__main__isthename ofthescopeinwhichtop-levelcodeexecutes),andIamawarethat,inthecomment,I wroteobjectinsteadofclass.Itturnsoutthat,asyoucanseebytheresultofthatprint, classesareactuallyobjects.Tobeprecise,theyareinstancesoftype.Explainingthis conceptwouldleadtoatalkaboutmetaclassesandmetaprogramming,advanced conceptsthatrequireasolidgraspofthefundamentalstobeunderstoodandalasthisis beyondthescopeofthischapter.Asusual,Imentionedittoleaveapointerforyou,for whenyou’llbereadytodigdeeper. Let’sgobacktotheexample:IusedSimplesttocreateaninstance,simp.Youcansee thatthesyntaxtocreateaninstanceisthesameweusetocallafunction. Thenweprintwhattypesimpbelongstoandweverifythatsimpisinfactaninstanceof Simplest.I’llshowyouabetterwayofdoingthislateroninthechapter. Uptonow,it’sallverysimple.WhathappenswhenwewriteclassClassName():pass, though?Well,whatPythondoesiscreateaclassobjectandassignitaname.Thisisvery similartowhathappenswhenwedeclareafunctionusingdef. WOW! eBook www.wowebook.org Classandobjectnamespaces Aftertheclassobjecthasbeencreated(whichusuallyhappenswhenthemoduleisfirst imported),itbasicallyrepresentsanamespace.Wecancallthatclasstocreateits instances.Eachinstanceinheritstheclassattributesandmethodsandisgivenitsown namespace.Wealreadyknowthat,towalkanamespace,allweneedtodoistousethedot (.)operator. Let’slookatanotherexample: oop/class.namespaces.py classPerson(): species='Human' print(Person.species)#Human Person.alive=True#Addeddynamically! print(Person.alive)#True man=Person() print(man.species)#Human(inherited) print(man.alive)#True(inherited) Person.alive=False print(man.alive)#False(inherited) man.name='Darth' man.surname='Vader' print(man.name,man.surname)#DarthVader Intheprecedingexample,Ihavedefinedaclassattributecalledspecies.Anyvariable definedinthebodyofaclassisanattributethatbelongstothatclass.Inthecode,Ihave alsodefinedPerson.alive,whichisanotherclassattribute.Youcanseethatthereisno restrictiononaccessingthatattributefromtheclass.Youcanseethatman,whichisan instanceofPerson,inheritsbothofthem,andreflectstheminstantlywhentheychange. manhasalsotwoattributeswhichbelongtoitsownnamespaceandthereforearecalled instanceattributes:nameandsurname. Note Classattributesaresharedamongstallinstances,whileinstanceattributesarenot; therefore,youshoulduseclassattributestoprovidethestatesandbehaviorstobeshared byallinstances,anduseinstanceattributesfordatathatbelongsjusttoonespecificobject. WOW! eBook www.wowebook.org Attributeshadowing Whenyousearchforanattributeinanobject,ifitisnotfound,Pythonkeepssearchingin theclassthatwasusedtocreatethatobject(andkeepssearchinguntilit’seitherfoundor theendoftheinheritancechainisreached).Thisleadstoaninterestingshadowing behavior.Let’slookatanexample: oop/class.attribute.shadowing.py classPoint(): x=10 y=7 p=Point() print(p.x)#10(fromclassattribute) print(p.y)#7(fromclassattribute) p.x=12#pgetsitsown'x'attribute print(p.x)#12(nowfoundontheinstance) print(Point.x)#10(classattributestillthesame) delp.x#wedeleteinstanceattribute print(p.x)#10(nowsearchhastogoagaintofindclassattr) p.z=3#let'smakeita3Dpoint print(p.z)#3 print(Point.z) #AttributeError:typeobject'Point'hasnoattribute'z' Theprecedingcodeisveryinteresting.WehavedefinedaclasscalledPointwithtwo classattributes,xandy.Whenwecreateaninstance,p,youcanseethatwecanprintboth xandyfromp‘snamespace(p.xandp.y).WhathappenswhenwedothatisthatPython doesn’tfindanyxoryattributesontheinstance,andthereforesearchestheclass,and findsthemthere. Thenwegivepitsownxattributebyassigningp.x=12.Thisbehaviormayappearabit weirdatfirst,butifyouthinkaboutit,it’sexactlythesameaswhathappensinafunction thatdeclaresx=12whenthereisaglobalx=10outside.Weknowthatx=12won’t affecttheglobalone,andforclassesandinstances,itisexactlythesame. Afterassigningp.x=12,whenweprintit,thesearchdoesn’tneedtoreadtheclass attributes,becausexisfoundontheinstance,thereforeweget12printedout. WealsoprintPoint.xwhichreferstoxintheclassnamespace. Andthen,wedeletexfromthenamespaceofp,whichmeansthat,onthenextline,when weprintitagain,Pythonwillgoagainandsearchforitintheclass,becauseitwon’tbe foundintheinstanceanymore. Thelastthreelinesshowyouthatassigningattributestoaninstancedoesn’tmeanthat theywillbefoundintheclass.Instancesgetwhateverisintheclass,buttheoppositeis nottrue. WOW! eBook www.wowebook.org Whatdoyouthinkaboutputtingthexandycoordinatesasclassattributes?Doyouthink itwasagoodidea? WOW! eBook www.wowebook.org I,me,andmyself–usingtheselfvariable Fromwithinaclassmethodwecanrefertoaninstancebymeansofaspecialargument, calledselfbyconvention.selfisalwaysthefirstattributeofaninstancemethod.Let’s examinethisbehaviortogetherwithhowwecanshare,notjustattributes,butmethods withallinstances. oop/class.self.py classSquare(): side=8 defarea(self):#selfisareferencetoaninstance returnself.side**2 sq=Square() print(sq.area())#64(sideisfoundontheclass) print(Square.area(sq))#64(equivalenttosq.area()) sq.side=10 print(sq.area())#100(sideisfoundontheinstance) Notehowtheareamethodisusedbysq.Thetwocalls,Square.area(sq)and sq.area(),areequivalent,andteachushowthemechanismworks.Eitheryoupassthe instancetothemethodcall(Square.area(sq)),whichwithinthemethodwillbecalled self,oryoucanuseamorecomfortablesyntax:sq.area()andPythonwilltranslatethat foryoubehindthecurtains. Let’slookatabetterexample: oop/class.price.py classPrice(): deffinal_price(self,vat,discount=0): """Returnspriceafterapplyingvatandfixeddiscount.""" return(self.net_price*(100+vat)/100)-discount p1=Price() p1.net_price=100 print(Price.final_price(p1,20,10))#110(100*1.2-10) print(p1.final_price(20,10))#equivalent Theprecedingcodeshowsyouthatnothingpreventsusfromusingargumentswhen declaringmethods.Wecanusetheexactsamesyntaxasweusedwiththefunction,butwe needtorememberthatthefirstargumentwillalwaysbetheinstance. WOW! eBook www.wowebook.org Initializinganinstance Haveyounoticedhow,beforecallingp1.final_price(...),wehadtoassignnet_price top1?Thereisabetterwaytodoit.Inotherlanguages,thiswouldbecalleda constructor,butinPython,it’snot.Itisactuallyaninitializer,sinceitworksonan alreadycreatedinstance,andthereforeit’scalled__init__.It’samagicmethod,whichis runrightaftertheobjectiscreated.Pythonobjectsalsohavea__new__method,whichis theactualconstructor.Inpractice,it’snotsocommontohavetooverrideitthough,it’sa practicethatismostlyusedwhencodingmetaclasses,whichisafairlyadvancedtopicthat wewon’texploreinthebook. oop/class.init.py classRectangle(): def__init__(self,sideA,sideB): self.sideA=sideA self.sideB=sideB defarea(self): returnself.sideA*self.sideB r1=Rectangle(10,4) print(r1.sideA,r1.sideB)#104 print(r1.area())#40 r2=Rectangle(7,3) print(r2.area())#21 Thingsarefinallystartingtotakeshape.Whenanobjectiscreated,the__init__method isautomaticallyrunforus.Inthiscase,Icodeditsothatwhenwecreateanobject(by callingtheclassnamelikeafunction),wepassargumentstothecreationcall,likewe wouldonanyregularfunctioncall.Thewaywepassparametersfollowsthesignatureof the__init__method,andtherefore,inthetwocreationstatements,10and7willbesideA forr1andr2respectively,while4and3willbesideB.Youcanseethatthecalltoarea() fromr1andr2reflectsthattheyhavedifferentinstancearguments. Settingupobjectsinthiswayismuchnicerandconvenient. WOW! eBook www.wowebook.org OOPisaboutcodereuse Bynowitshouldbeprettyclear:OOPisallaboutcodereuse.Wedefineaclass,we createinstances,andthoseinstancesusemethodsthataredefinedonlyintheclass.They willbehavedifferentlyaccordingtohowtheinstanceshavebeensetupbytheinitializer. Inheritanceandcomposition Butthisisjusthalfofthestory,OOPismuchmorepowerful.Wehavetwomaindesign constructstoexploit:inheritanceandcomposition. InheritancemeansthattwoobjectsarerelatedbymeansofanIs-Atypeofrelationship. Ontheotherhand,compositionmeansthattwoobjectsarerelatedbymeansofaHas-A typeofrelationship.It’sallveryeasytoexplainwithanexample: oop/class.inheritance.py classEngine(): defstart(self): pass defstop(self): pass classElectricEngine(Engine):#Is-AEngine pass classV8Engine(Engine):#Is-AEngine pass classCar(): engine_cls=Engine def__init__(self): self.engine=self.engine_cls()#Has-AEngine defstart(self): print( 'Startingengine{0}forcar{1}...Wroom,wroom!' .format( self.engine.__class__.__name__, self.__class__.__name__) ) self.engine.start() defstop(self): self.engine.stop() classRaceCar(Car):#Is-ACar engine_cls=V8Engine classCityCar(Car):#Is-ACar engine_cls=ElectricEngine classF1Car(RaceCar):#Is-ARaceCarandalsoIs-ACar WOW! eBook www.wowebook.org engine_cls=V8Engine car=Car() racecar=RaceCar() citycar=CityCar() f1car=F1Car() cars=[car,racecar,citycar,f1car] forcarincars: car.start() """Prints: StartingengineEngineforcarCar...Wroom,wroom! StartingengineV8EngineforcarRaceCar...Wroom,wroom! StartingengineElectricEngineforcarCityCar...Wroom,wroom! StartingengineV8EngineforcarF1Car...Wroom,wroom! """ TheprecedingexampleshowsyouboththeIs-AandHas-Atypesofrelationshipsbetween objects.Firstofall,let’sconsiderEngine.It’sasimpleclassthathastwomethods,start andstop.WethendefineElectricEngineandV8Engine,whichbothinheritfromEngine. Youcanseethatbythefactthatwhenwedefinethem,weputEnginewithinthebraces aftertheclassname. ThismeansthatbothElectricEngineandV8Engineinheritattributesandmethodsfrom theEngineclass,whichissaidtobetheirbaseclass. Thesamehappenswithcars.CarisabaseclassforbothRaceCarandCityCar.RaceCaris alsothebaseclassforF1Car.AnotherwayofsayingthisisthatF1Carinheritsfrom RaceCar,whichinheritsfromCar.Therefore,F1CarIs-ARaceCarandRaceCarIs-ACar. Becauseofthetransitiveproperty,wecansaythatF1CarIs-ACaraswell.CityCartoo,IsACar. WhenwedefineclassA(B):pass,wesayAisthechildofB,andBistheparentofA. parentandbasearesynonyms,aswellaschildandderived.Also,wesaythataclass inheritsfromanotherclass,orthatitextendsit. Thisistheinheritancemechanism. Ontheotherhand,let’sgobacktothecode.Eachclasshasaclassattribute,engine_cls, whichisareferencetotheengineclasswewanttoassigntoeachtypeofcar.Carhasa genericEngine,whilethetworacecarshaveapowerfulV8engine,andthecitycarhasan electricone. Whenacariscreatedintheinitializermethod__init__,wecreateaninstanceof whateverengineclassisassignedtothecar,andsetitasengineinstanceattribute. Itmakessensetohaveengine_clssharedamongstallclassinstancesbecauseit’squite likelythatthesameinstancesofacarwillhavethesamekindofengine.Ontheother hand,itwouldn’tbegoodtohaveonesingleengine(aninstanceofanyEngineclass)asa classattribute,becausewewouldbesharingoneengineamongstallinstances,whichis incorrect. WOW! eBook www.wowebook.org ThetypeofrelationshipbetweenacaranditsengineisaHas-Atype.AcarHas-Aengine. Thisiscalledcomposition,andreflectsthefactthatobjectscanbemadeofmanyother objects.AcarHas-Aengine,gears,wheels,aframe,doors,seats,andsoon. WhendesigningOOPcode,itisofvitalimportancetodescribeobjectsinthiswaysothat wecanuseinheritanceandcompositioncorrectlytostructureourcodeinthebestway. Beforeweleavethisparagraph,let’scheckifItoldyouthetruthwithanotherexample: oop/class.issubclass.isinstance.py car=Car() racecar=RaceCar() f1car=F1Car() cars=[(car,'car'),(racecar,'racecar'),(f1car,'f1car')] car_classes=[Car,RaceCar,F1Car] forcar,car_nameincars: forclass_incar_classes: belongs=isinstance(car,class_) msg='isa'ifbelongselse'isnota' print(car_name,msg,class_.__name__) """Prints: carisaCar carisnotaRaceCar carisnotaF1Car racecarisaCar racecarisaRaceCar racecarisnotaF1Car f1carisaCar f1carisaRaceCar f1carisaF1Car """ Asyoucansee,carisjustaninstanceofCar,whileracecarisaninstanceofRaceCar (andofCarbyextension)andf1carisaninstanceofF1Car(andofbothRaceCarandCar, byextension).AbananaisaninstanceofBanana.But,also,itisaFruit.Also,itisFood, right?Thisisthesameconcept. Tocheckifanobjectisaninstanceofaclass,usetheisinstancemethod.Itis recommendedoversheertypecomparison(type(object)==Class). Let’salsocheckinheritance,samesetup,withdifferentforloops: oop/class.issubclass.isinstance.py forclass1incar_classes: forclass2incar_classes: is_subclass=issubclass(class1,class2) msg='{0}asubclassof'.format( 'is'ifis_subclasselse'isnot') print(class1.__name__,msg,class2.__name__) """Prints: CarisasubclassofCar WOW! eBook www.wowebook.org CarisnotasubclassofRaceCar CarisnotasubclassofF1Car RaceCarisasubclassofCar RaceCarisasubclassofRaceCar RaceCarisnotasubclassofF1Car F1CarisasubclassofCar F1CarisasubclassofRaceCar F1CarisasubclassofF1Car """ Interestingly,welearnthataclassisasubclassofitself.Checktheoutputofthepreceding exampletoseethatitmatchestheexplanationIprovided. Note Onethingtonoticeaboutconventionsisthatclassnamesarealwayswrittenusing CapWords,whichmeansThisWayIsCorrect,asopposedtofunctionsandmethods,which arewrittenthis_way_is_correct.Also,wheninthecodeyouwanttouseanamewhichisa Python-reservedkeywordorbuilt-infunctionorclass,theconventionistoaddatrailing underscoretothename.Inthefirstforloopexample,I’mloopingthroughtheclass namesusingforclass_in…,becauseclassisareservedword.Butyoualreadyknewall thisbecauseyouhavethoroughlystudiedPEP8,right? TohelpyoupicturethedifferencebetweenIs-AandHas-A,takealookatthefollowing diagram: WOW! eBook www.wowebook.org Accessingabaseclass We’vealreadyseenclassdeclarationslikeclassClassA:passandclass ClassB(BaseClassName):pass.Whenwedon’tspecifyabaseclassexplicitly,Python willsetthespecialobjectclassasthebaseclassfortheonewe’redefining.Ultimately,all classesderivefromobject.Notethat,ifyoudon’tspecifyabaseclass,bracesare optional. Therefore,writingclassA:passorclassA():passorclassA(object):passis exactlythesamething.objectisaspecialclassinthatithasthemethodsthatarecommon toallPythonclasses,anditdoesn’tallowyoutosetanyattributesonit. Let’sseehowwecanaccessabaseclassfromwithinaclass. oop/super.duplication.py classBook: def__init__(self,title,publisher,pages): self.title=title self.publisher=publisher self.pages=pages classEbook(Book): def__init__(self,title,publisher,pages,format_): self.title=title self.publisher=publisher self.pages=pages self.format_=format_ Takealookattheprecedingcode.IhighlightedthepartofEbookinitializationthatis duplicatedfromitsbaseclassBook.Thisisquitebadpracticebecausewenowhavetwo setsofinstructionsthataredoingthesamething.Moreover,anychangeinthesignatureof Book.__init__willnotreflectinEbook.WeknowthatEbookIs-ABook,andthereforewe wouldprobablywantchangestobereflectedinthechildrenclasses. Let’sseeonewaytofixthisissue: oop/super.explicit.py classBook: def__init__(self,title,publisher,pages): self.title=title self.publisher=publisher self.pages=pages classEbook(Book): def__init__(self,title,publisher,pages,format_): Book.__init__(self,title,publisher,pages) self.format_=format_ ebook=Ebook('LearningPython','PacktPublishing',360,'PDF') print(ebook.title)#LearningPython print(ebook.publisher)#PacktPublishing print(ebook.pages)#360 print(ebook.format_)#PDF WOW! eBook www.wowebook.org Now,that’sbetter.Wehaveremovedthatnastyduplication.Basically,wetellPythonto callthe__init__methodoftheBookclass,andwefeedselftothecall,makingsurethat webindthatcalltothepresentinstance. Ifwemodifythelogicwithinthe__init__methodofBook,wedon’tneedtotouchEbook, itwillautoadapttothechange. Thisapproachisgood,butwecanstilldoabitbetter.SaythatwechangeBook‘snameto Liber,becausewe’vefalleninlovewithLatin.Wehavetochangethe__init__method ofEbooktoreflectthechange.Thiscanbeavoidedbyusingsuper. oop/super.implicit.py classBook: def__init__(self,title,publisher,pages): self.title=title self.publisher=publisher self.pages=pages classEbook(Book): def__init__(self,title,publisher,pages,format_): super().__init__(title,publisher,pages) #Anotherwaytodothesamethingis: #super(Ebook,self).__init__(title,publisher,pages) self.format_=format_ ebook=Ebook('LearningPython','PacktPublishing',360,'PDF') print(ebook.title)#LearningPython print(ebook.publisher)#PacktPublishing print(ebook.pages)#360 print(ebook.format_)#PDF superisafunctionthatreturnsaproxyobjectthatdelegatesmethodcallstoaparentor siblingclass.Inthiscase,itwilldelegatethatcallto__init__totheBookclass,andthe beautyofthismethodisthatnowwe’reevenfreetochangeBooktoLiberwithouthaving totouchthelogicinthe__init__methodofEbook. Nowthatweknowhowtoaccessabaseclassfromachild,let’sexplorePython’smultiple inheritance. WOW! eBook www.wowebook.org Multipleinheritance Apartfromcomposingaclassusingmorethanonebaseclass,whatisofinteresthereis howanattributesearchisperformed.Takealookatthefollowingdiagram: WOW! eBook www.wowebook.org Asyoucansee,ShapeandPlotteractasbaseclassesforalltheothers.Polygoninherits directlyfromthem,RegularPolygoninheritsfromPolygon,andbothRegularHexagon andSquareinheritfromRegulaPolygon.NotealsothatShapeandPlotterimplicitly inheritfromobject,thereforewehavewhatiscalledadiamondor,insimplerterms, morethanonepathtoreachabaseclass.We’llseewhythismattersinafewmoments. Let’stranslateitintosomesimplecode: oop/multiple.inheritance.py classShape: geometric_type='GenericShape' defarea(self):#Thisactsasplaceholderfortheinterface raiseNotImplementedError defget_geometric_type(self): returnself.geometric_type classPlotter: defplot(self,ratio,topleft): #Imaginesomeniceplottinglogichere… print('Plottingat{},ratio{}.'.format( topleft,ratio)) classPolygon(Shape,Plotter):#baseclassforpolygons geometric_type='Polygon' classRegularPolygon(Polygon):#Is-APolygon geometric_type='RegularPolygon' def__init__(self,side): self.side=side classRegularHexagon(RegularPolygon):#Is-ARegularPolygon geometric_type='RegularHexagon' defarea(self): return1.5*(3**.5*self.side**2) classSquare(RegularPolygon):#Is-ARegularPolygon geometric_type='Square' defarea(self): returnself.side*self.side hexagon=RegularHexagon(10) print(hexagon.area())#259.8076211353316 print(hexagon.get_geometric_type())#RegularHexagon hexagon.plot(0.8,(75,77))#Plottingat(75,77),ratio0.8. square=Square(12) print(square.area())#144 print(square.get_geometric_type())#Square square.plot(0.93,(74,75))#Plottingat(74,75),ratio0.93. WOW! eBook www.wowebook.org Takealookattheprecedingcode:theclassShapehasoneattribute,geometric_type,and twomethods:areaandget_geometric_type.It’squitecommontousebaseclasses(like Shape,inourexample)todefineaninterface:methodsforwhichchildrenmustprovidean implementation.Therearedifferentandbetterwaystodothis,butIwanttokeepthis exampleassimpleaspossible. WealsohavethePlotterclass,whichaddstheplotmethod,therebyprovidingplotting capabilitiesforanyclassthatinheritsfromit.Ofcourse,theplotimplementationisjusta dummyprintinthisexample.ThefirstinterestingclassisPolygon,whichinheritsfrom bothShapeandPlotter. Therearemanytypesofpolygons,oneofwhichistheregularone,whichisboth equiangular(allanglesareequal)andequilateral(allsidesareequal),sowecreatethe RegularPolygonclassthatinheritsfromPolygon.Becauseforaregularpolygon,allsides areequal,wecanimplementasimple__init__methodonRegularPolygon,whichtakes thelengthoftheside.Finally,wecreatetheRegularHexagonandSquareclasses,which bothinheritfromRegularPolygon. Thisstructureisquitelong,buthopefullygivesyouanideaofhowtospecializethe classificationofyourobjectswhenyoudesignthecode. Now,pleasetakealookatthelasteightlines.NotethatwhenIcalltheareamethodon hexagonandsquare,Igetthecorrectareaforboth.Thisisbecausetheybothprovidethe correctimplementationlogicforit.Also,Icancallget_geometric_typeonbothofthem, eventhoughitisnotdefinedontheirclasses,andPythonhastogoallthewayuptoShape tofindanimplementationforit.Notethat,eventhoughtheimplementationisprovidedin theShapeclass,theself.geometric_typeusedforthereturnvalueiscorrectlytaken fromthecallerinstance. Theplotmethodcallsarealsointeresting,andshowyouhowyoucanenrichyourobjects withcapabilitiestheywouldn’totherwisehave.Thistechniqueisverypopularinweb frameworkssuchasDjango(whichwe’llexploreintwolaterchapters),whichprovides specialclassescalledmixins,whosecapabilitiesyoucanjustuseoutofthebox.Allyou havetodoistodefinethedesiredmixinasonethebaseclassesforyourown,andthat’sit. Multipleinheritanceispowerful,butcanalsogetreallymessy,soweneedtomakesure weunderstandwhathappenswhenweuseit. Methodresolutionorder Bynow,weknowthatwhenyouaskforsomeobject.attribute,andattributeisnot foundonthatobject,Pythonstartssearchingintheclasssomeobjectwascreatedfrom.If it’snotthereeither,Pythonsearchesuptheinheritancechainuntileitherattributeis foundortheobjectclassisreached.Thisisquitesimpletounderstandiftheinheritance chainisonlycomprisedofsingleinheritancesteps,whichmeansthatclasseshaveonly oneparent.However,whenmultipleinheritanceisinvolved,therearecaseswhenit’snot straightforwardtopredictwhatwillbethenextclassthatwillbesearchedforifan attributeisnotfound. WOW! eBook www.wowebook.org Pythonprovidesawaytoalwaysknowwhatistheorderinwhichclassesaresearchedon attributelookup:themethodresolutionorder. Note Themethodresolutionorder(MRO)istheorderinwhichbaseclassesaresearchedfor amemberduringlookup.Fromversion2.3PythonusesanalgorithmcalledC3,which guaranteesmonotonicity. InPython2.2,new-styleclasseswereintroduced.Thewayyouwriteanew-styleclassin Python2.*istodefineitwithanexplicitobjectbaseclass.Classicclasseswerenot explicitlyinheritingfromobjectandhavebeenremovedinPython3. Oneofthedifferencesbetweenclassicandnewstyle-classesinPython2.*isthatnewstyleclassesaresearchedwiththenewMRO. Withregardstothepreviousexample,let’sseewhatistheMROfortheSquareclass: oop/multiple.inheritance.py print(square.__class__.__mro__) #prints: #(<class'__main__.Square'>,<class'__main__.RegularPolygon'>, #<class'__main__.Polygon'>,<class'__main__.Shape'>, #<class'__main__.Plotter'>,<class'object'>) TogettotheMROofaclass,wecangofromtheinstancetoits__class__attributeand fromthattoits__mro__attribute.Alternatively,wecouldhavecalledSquare.__mro__,or Square.mro()directly,butifyouhavetodoitdynamically,it’smorelikelyyouwillhave anobjectinyourhandsratherthanaclass. NotethattheonlypointofdoubtisthebisectionafterPolygon,wheretheinheritance chainbreaksintotwoways,oneleadstoShapeandtheothertoPlotter.Weknowby scanningtheMROfortheSquareclassthatShapeissearchedbeforePlotter. Whyisthisimportant?Well,imaginethefollowingcode: oop/mro.simple.py classA: label='a' classB(A): label='b' classC(A): label='c' classD(B,C): pass d=D() print(d.label)#Hypotheticallythiscouldbeeither'b'or'c' BothBandCinheritfromA,andDinheritsfrombothBandC.Thismeansthatthelookup WOW! eBook www.wowebook.org forthelabelattributecanreachthetop(A)throughbothBorC.Accordingtowhichis reachedfirst,wegetadifferentresult. So,intheprecedingexampleweget'b',whichiswhatwewereexpecting,sinceBisthe leftmostoneamongstbaseclassesofD.ButwhathappensifIremovethelabelattribute fromB?Thiswouldbetheconfusingsituation:WillthealgorithmgoallthewayuptoAor willitgettoCfirst?Let’sfindout! oop/mro.py classA: label='a' classB(A): pass#was:label='b' classC(A): label='c' classD(B,C): pass d=D() print(d.label)#'c' print(d.__class__.mro())#noticeanotherwaytogettheMRO #prints: #[<class'__main__.D'>,<class'__main__.B'>, #<class'__main__.C'>,<class'__main__.A'>,<class'object'>] So,welearnthattheMROisD-B-C-A-(object),whichmeanswhenweaskford.label, weget'c',whichiscorrect. Indaytodayprogramming,itisnotquitecommontohavetodealwiththeMRO,butthe firsttimeyoufightagainstsomemixinfromaframework,Ipromiseyou’llbegladIspent aparagraphexplainingit. WOW! eBook www.wowebook.org Staticandclassmethods Untilnow,wehavecodedclasseswithattributesintheformofdataandinstancemethods, buttherearetwoothertypesofmethodsthatwecanplaceinsideaclass:staticmethods andclassmethods. Staticmethods Asyoumayrecall,whenyoucreateaclassobject,Pythonassignsanametoit.Thatname actsasanamespace,andsometimesitmakessensetogroupfunctionalitiesunderit.Static methodsareperfectforthisusecasesinceunlikeinstancemethods,theyarenotpassed anyspecialargument.Let’slookatanexampleofanimaginaryStringclass. oop/static.methods.py classString: @staticmethod defis_palindrome(s,case_insensitive=True): #weallowonlylettersandnumbers s=''.join(cforcinsifc.isalnum())#Studythis! #Forcaseinsensitivecomparison,welower-cases ifcase_insensitive: s=s.lower() forcinrange(len(s)//2): ifs[c]!=s[-c-1]: returnFalse returnTrue @staticmethod defget_unique_words(sentence): returnset(sentence.split()) print(String.is_palindrome( 'Radar',case_insensitive=False))#False:CaseSensitive print(String.is_palindrome('Anutforajaroftuna'))#True print(String.is_palindrome('NeverOdd,OrEven!'))#True print(String.is_palindrome( 'InGirumImusNocteEtConsumimurIgni')#Latin!Show-off! )#True print(String.get_unique_words( 'Ilovepalindromes.Ireallyreallylovethem!')) #{'them!','really','palindromes.','I','love'} Theprecedingcodeisquiteinteresting.Firstofall,welearnthatstaticmethodsare createdbysimplyapplyingthestaticmethoddecoratortothem.Youcanseethatthey aren’tpassedanyspecialargumentso,apartfromthedecoration,theyreallyjustlooklike functions. Wehaveaclass,String,whichactsasacontainerforfunctions.Anotherapproachwould betohaveaseparatemodulewithfunctionsinside.It’sreallyamatterofpreferencemost ofthetime. WOW! eBook www.wowebook.org Thelogicinsideis_palindromeshouldbestraightforwardforyoutounderstandbynow, but,justincase,let’sgothroughit.Firstweremoveallcharactersfromsthatarenot eitherlettersornumbers.Inordertodothis,weusethejoinmethodofastringobject(an emptystringobject,inthiscase).Bycallingjoinonanemptystring,theresultisthatall elementsintheiterableyoupasstojoinwillbeconcatenatedtogether.Wefeedjoina generatorexpressionthatsays,takeanycharacterfromsifthecharacteriseither alphanumericoranumber.Ihopeyouhavebeenabletofindthatoutbyyourself,maybe usingtheinside-outtechniqueIshowedyouinoneoftheprecedingchapters. Wethenlowercasesifcase_insensitiveisTrue,andthenweproceedtocheckifitisa palindrome.Inordertodothis,wecomparethefirstandlastcharacters,thenthesecond andthesecondtolast,andsoon.Ifatanypointwefindadifference,itmeansthestring isn’tapalindromeandthereforewecanreturnFalse.Ontheotherhand,ifweexitthefor loopnormally,itmeansnodifferenceswerefound,andwecanthereforesaythestringisa palindrome. Noticethatthiscodeworkscorrectlyregardlessofthelengthofthestring,thatis,ifthe lengthisoddoreven.len(s)//2reacheshalfofs,andifsisanoddamountof characterslong,themiddleonewon’tbechecked(likeinRaDaR,Disnotchecked),but wedon’tcare;itwouldbecomparedwithitselfsoit’salwayspassingthatcheck. get_unique_wordsismuchsimpler,itjustreturnsasettowhichwefeedalistwiththe wordsfromasentence.Thesetclassremovesanyduplicationforus,thereforewedon’t needtodoanythingelse. TheStringclassprovidesusanicecontainernamespaceformethodsthataremeantto workonstrings.IcouldhavecodedasimilarexamplewithaMathclass,andsomestatic methodstoworkonnumbers,butIwantedtoshowyousomethingdifferent. Classmethods Classmethodsareslightlydifferentfrominstancemethodsinthattheyalsotakeaspecial firstargument,butinthiscase,itistheclassobjectitself.Twoverycommonusecasesfor codingclassmethodsaretoprovidefactorycapabilitytoaclassandtoallowbreakingup staticmethods(whichyouhavetothencallusingtheclassname)withouthavingto hardcodetheclassnameinyourlogic.Let’slookatanexampleofbothofthem. oop/class.methods.factory.py classPoint: def__init__(self,x,y): self.x=x self.y=y @classmethod deffrom_tuple(cls,coords):#clsisPoint returncls(*coords) @classmethod deffrom_point(cls,point):#clsisPoint returncls(point.x,point.y) WOW! eBook www.wowebook.org p=Point.from_tuple((3,7)) print(p.x,p.y)#37 q=Point.from_point(p) print(q.x,q.y)#37 Intheprecedingcode,Ishowedyouhowtouseaclassmethodtocreateafactoryforthe class.Inthiscase,wewanttocreateaPointinstancebypassingbothcoordinates(regular creationp=Point(3,7)),butwealsowanttobeabletocreateaninstancebypassinga tuple(Point.from_tuple)oranotherinstance(Point.from_point). Withinthetwoclassmethods,theclsargumentreferstothePointclass.Aswithinstance method,whichtakeselfasthefirstargument,classmethodtakeaclsargument.Both selfandclsarenamedafteraconventionthatyouarenotforcedtofollowbutare stronglyencouragedtorespect.ThisissomethingthatnoPythoncoderwouldchange becauseitissostrongaconventionthatparsers,linters,andanytoolthatautomatically doessomethingwithyourcodewouldexpect,soit’smuchbettertosticktoit. Let’slookatanexampleoftheotherusecase:splittingastaticmethod. oop/class.methods.split.py classString: @classmethod defis_palindrome(cls,s,case_insensitive=True): s=cls._strip_string(s) #Forcaseinsensitivecomparison,welower-cases ifcase_insensitive: s=s.lower() returncls._is_palindrome(s) @staticmethod def_strip_string(s): return''.join(cforcinsifc.isalnum()) @staticmethod def_is_palindrome(s): forcinrange(len(s)//2): ifs[c]!=s[-c-1]: returnFalse returnTrue @staticmethod defget_unique_words(sentence): returnset(sentence.split()) print(String.is_palindrome('Anutforajaroftuna'))#True print(String.is_palindrome('Anutforajarofbeans'))#False Comparethiscodewiththepreviousversion.Firstofallnotethateventhough is_palindromeisnowaclassmethod,wecallitinthesamewaywewerecallingitwhen itwasastaticone.Thereasonwhywechangedittoaclassmethodisthatafterfactoring outacoupleofpiecesoflogic(_strip_stringand_is_palindrome),weneedtogeta WOW! eBook www.wowebook.org referencetothemandifwehavenoclsinourmethod,theonlyoptionwouldbetocall themlikethis:String._strip_string(...)andString._is_palindrome(...),whichis notgoodpractice,becausewewouldhardcodetheclassnameintheis_palindrome method,therebyputtingourselvesintheconditionofhavingtomodifyitwheneverwe wouldchangetheclassname.Usingclswillactastheclassname,whichmeansourcode won’tneedanyamendments. Notealsothat,bynamingthefactored-outmethodswithaleadingunderscore,Iam hintingthatthosemethodsarenotsupposedtobecalledfromoutsidetheclass,butthis willbethesubjectofthenextparagraph. WOW! eBook www.wowebook.org Privatemethodsandnamemangling IfyouhaveanybackgroundwithlanguageslikeJava,C#,C++,orsimilar,thenyouknow theyallowtheprogrammertoassignaprivacystatustoattributes(bothdataandmethods). Eachlanguagehasitsownslightlydifferentflavorforthis,butthegististhatpublic attributesareaccessiblefromanypointinthecode,whileprivateonesareaccessibleonly withinthescopetheyaredefinedin. InPython,thereisnosuchthing.Everythingispublic;therefore,werelyonconventions andonamechanismcallednamemangling. Theconventionisasfollows:ifanattribute’snamehasnoleadingunderscoresitis consideredpublic.Thismeansyoucanaccessitandmodifyitfreely.Whenthenamehas oneleadingunderscore,theattributeisconsideredprivate,whichmeansit’sprobably meanttobeusedinternallyandyoushouldnotuseitormodifyitfromtheoutside.Avery commonusecaseforprivateattributesarehelpermethodsthataresupposedtobeusedby publicones(possiblyincallchainsinconjunctionwithothermethods),andinternaldata, likescalingfactors,oranyotherdatathatideallywewouldputinaconstant(avariable thatcannotchange,but,surprise,surprise,Pythondoesn’thavethoseeither). Thischaracteristicusuallyscarespeoplefromotherbackgroundsoff;theyfeelthreatened bythelackofprivacy.Tobehonest,inmywholeprofessionalexperiencewithPython, I’veneverheardanyonescreamingOhmyGod,wehaveaterriblebugbecausePython lacksprivateattributes!Notonce,Iswear. Thatsaid,thecallforprivacyactuallymakessensebecausewithoutit,yourisk introducingbugsintoyourcodeforreal.Let’slookatasimpleexample: oop/private.attrs.py classA: def__init__(self,factor): self._factor=factor defop1(self): print('Op1withfactor{}...'.format(self._factor)) classB(A): defop2(self,factor): self._factor=factor print('Op2withfactor{}...'.format(self._factor)) obj=B(100) obj.op1()#Op1withfactor100… obj.op2(42)#Op2withfactor42… obj.op1()#Op1withfactor42…<-ThisisBAD Intheprecedingcode,wehaveanattributecalled_factor,andlet’spretendit’svery importantthatitisn’tmodifiedatruntimeaftertheinstanceiscreated,becauseop1 dependsonittofunctioncorrectly.We’venameditwithaleadingunderscore,butthe issuehereisthatwhenwecallobj.op2(42),wemodifyit,andthisreflectsinsubsequent WOW! eBook www.wowebook.org callstoop1. Let’sfixthisundesiredbehaviorbyaddinganotherleadingunderscore: oop/private.attrs.fixed.py classA: def__init__(self,factor): self.__factor=factor defop1(self): print('Op1withfactor{}...'.format(self.__factor)) classB(A): defop2(self,factor): self.__factor=factor print('Op2withfactor{}...'.format(self.__factor)) obj=B(100) obj.op1()#Op1withfactor100… obj.op2(42)#Op2withfactor42… obj.op1()#Op1withfactor100…<-Wohoo!Nowit'sGOOD! Wow,lookatthat!Nowit’sworkingasdesired.Pythoniskindofmagicandinthiscase, whatishappeningisthatthenamemanglingmechanismhaskickedin. Namemanglingmeansthatanyattributenamethathasatleasttwoleadingunderscores andatmostonetrailingunderscore,like__my_attr,isreplacedwithanamethatincludes anunderscoreandtheclassnamebeforetheactualname,like_ClassName__my_attr. Thismeansthatwhenyouinheritfromaclass,themanglingmechanismgivesyour privateattributetwodifferentnamesinthebaseandchildclassessothatnamecollisionis avoided.Everyclassandinstanceobjectstoresreferencestotheirattributesinaspecial attributecalled__dict__,solet’sinspectobj.__dict__toseenamemanglinginaction: oop/private.attrs.py print(obj.__dict__.keys()) #dict_keys(['_factor']) Thisisthe_factorattributethatwefindintheproblematicversionofthisexample.But lookattheonethatisusing__factor: oop/private.attrs.fixed.py print(obj.__dict__.keys()) #dict_keys(['_A__factor','_B__factor']) See?objhastwoattributesnow,_A__factor(mangledwithintheAclass),and _B__factor(mangledwithintheBclass).Thisisthemechanismthatmakespossiblethat whenyoudoobj.__factor=42,__factorinAisn’tchanged,becauseyou’reactually touching_B__factor,whichleaves_A__factorsafeandsound. Ifyou’redesigningalibrarywithclassesthataremeanttobeusedandextendedbyother developers,youwillneedtokeepthisinmindinordertoavoidunintentionaloverriding WOW! eBook www.wowebook.org ofyourattributes.Bugslikethesecanbeprettysubtleandhardtospot. WOW! eBook www.wowebook.org Thepropertydecorator Anotherthingthatwouldbeacrimenottomentionisthepropertydecorator.Imagine thatyouhaveanageattributeinaPersonclassandatsomepointyouwanttomakesure thatwhenyouchangeitsvalue,you’realsocheckingthatageiswithinaproperrange, like[18,99].Youcanwriteaccessormethods,likeget_age()andset_age()(alsocalled gettersandsetters)andputthelogicthere.get_age()willmostlikelyjustreturnage, whileset_age()willalsodotherangecheck.Theproblemisthatyoumayalreadyhavea lotofcodeaccessingtheageattributedirectly,whichmeansyou’renowuptosomegood (andpotentiallydangerousandtedious)refactoring.LanguageslikeJavaovercomethis problembyusingtheaccessorpatternbasicallybydefault.ManyJavaIntegrated DevelopmentEnvironments(IDEs)autocompleteanattributedeclarationbywriting getterandsetteraccessormethodsstubsforyouonthefly. Pythonissmarter,anddoesthiswiththepropertydecorator.Whenyoudecoratea methodwithproperty,youcanusethenameofthemethodasifitwasadataattribute. Becauseofthis,it’salwaysbesttorefrainfromputtinglogicthatwouldtakeawhileto completeinsuchmethodsbecause,byaccessingthemasattributes,wearenotexpecting towait. Let’slookatanexample: oop/property.py classPerson: def__init__(self,age): self.age=age#anyonecanmodifythisfreely classPersonWithAccessors: def__init__(self,age): self._age=age defget_age(self): returnself._age defset_age(self): if18<=age<=99: self._age=age else: raiseValueError('Agemustbewithin[18,99]') classPersonPythonic: def__init__(self,age): self._age=age @property defage(self): returnself._age @age.setter defage(self,age): if18<=age<=99: WOW! eBook www.wowebook.org self._age=age else: raiseValueError('Agemustbewithin[18,99]') person=PersonPythonic(39) print(person.age)#39-Noticeweaccessasdataattribute person.age=42#Noticeweaccessasdataattribute print(person.age)#42 person.age=100#ValueError:Agemustbewithin[18,99] ThePersonclassmaybethefirstversionwewrite.Thenwerealizeweneedtoputthe rangelogicinplaceso,withanotherlanguage,wewouldhavetorewritePersonasthe PersonWithAccessorsclass,andrefactorallthecodethatwasusingPerson.age.In Python,werewritePersonasPersonPythonic(younormallywouldn’tchangethename, ofcourse)sothattheageisstoredinaprivate_agevariable,andwedefineproperty gettersandsettersusingthatdecoration,whichallowustokeepusingtheperson instancesaswewerebefore.Agetterisamethodthatiscalledwhenweaccessan attributeforreading.Ontheotherhand,asetterisamethodthatiscalledwhenweaccess anattributetowriteit.Inotherlanguages,likeJavaforexample,it’scustomarytodefine themasget_age()andset_age(intvalue),butIfindthePythonsyntaxmuchneater.It allowsyoutostartwritingsimplecodeandrefactorlateron,onlywhenyouneedit,there isnoneedtopolluteyourcodewithaccessorsonlybecausetheymaybehelpfulinthe future. Thepropertydecoratoralsoallowsforread-onlydata(nosetter)andforspecialactions whentheattributeisdeleted.Pleaserefertotheofficialdocumentationtodigdeeper. WOW! eBook www.wowebook.org Operatoroverloading IfindPython’sapproachtooperatoroverloadingtobebrilliant.Tooverloadanoperator meanstogiveitameaningaccordingtothecontextinwhichitisused.Forexample,the+ operatormeansadditionwhenwedealwithnumbers,butconcatenationwhenwedeal withsequences. InPython,whenyouuseoperators,you’remostlikelycallingthespecialmethodsofsome objectsbehindthescenes.Forexample,thecalla[k]roughlytranslatesto type(a).__getitem__(a,k). Asanexample,let’screateaclassthatstoresastringandevaluatestoTrueif'42'ispart ofthatstring,andFalseotherwise.Also,let’sgivetheclassalengthpropertywhich correspondstothatofthestoredstring. oop/operator.overloading.py classWeird: def__init__(self,s): self._s=s def__len__(self): returnlen(self._s) def__bool__(self): return'42'inself._s weird=Weird('Hello!Iam9yearsold!') print(len(weird))#24 print(bool(weird))#False weird2=Weird('Hello!Iam42yearsold!') print(len(weird2))#25 print(bool(weird2))#True Thatwasfun,wasn’tit?Forthecompletelistofmagicmethodsthatyoucanoverridein ordertoprovideyourcustomimplementationofoperatorsforyourclasses,pleasereferto thePythondatamodelintheofficialdocumentation. WOW! eBook www.wowebook.org Polymorphism–abriefoverview ThewordpolymorphismcomesfromtheGreekpolys(many,much)andmorphē(form, shape),anditsmeaningistheprovisionofasingleinterfaceforentitiesofdifferenttypes. Inourcarexample,wecallengine.start(),regardlessofwhatkindofengineitis.As longasitexposesthestartmethod,wecancallit.That’spolymorphisminaction. Inotherlanguages,likeJava,inordertogiveafunctiontheabilitytoacceptdifferent typesandcallamethodonthem,thosetypesneedtobecodedinsuchawaythatthey shareaninterface.Inthisway,thecompilerknowsthatthemethodwillbeavailable regardlessofthetypeoftheobjectthefunctionisfed(aslongasitextendstheproper interface,ofcourse). InPython,thingsaredifferent.Polymorphismisimplicit,nothingpreventsyoutocalla methodonanobject,therefore,technically,thereisnoneedtoimplementinterfacesor otherpatterns. Thereisaspecialkindofpolymorphismcalledadhocpolymorphism,whichiswhatwe sawinthelastparagraph:operatoroverloading.Theabilityofanoperatortochange shape,accordingtothetypeofdataitisfed. Icannotspendtoomuchtimeonpolymorphism,butIencourageyoutocheckitoutby yourself,itwillexpandyourunderstandingofOOP.Goodluck! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Writingacustomiterator Nowwehaveallthetoolstoappreciatehowwecanwriteourowncustomiterator.Let’s firstdefinewhatisaniterableandaniterator: Iterable:Anobjectissaidtobeiterableifit’scapableofreturningitsmembersone atatime.Lists,tuples,strings,dicts,arealliterables.Customobjectsthatdefine eitherof__iter__or__getitem__methodsarealsoiterables. Iterator:Anobjectissaidtobeaniteratorifitrepresentsastreamofdata.Acustom iteratorisrequiredtoprovideanimplementationfor__iter__thatreturnstheobject itself,andanimplementationfor__next__,whichreturnsthenextitemofthedata streamuntilthestreamisexhausted,atwhichpointallsuccessivecallsto__next__ simplyraisetheStopIterationexception.Built-infunctionssuchasiterandnext aremappedtocall__iter__and__next__onanobject,behindthescenes. Let’swriteaniteratorthatreturnsalltheoddcharactersfromastringfirst,andthenthe evenones. iterators/iterator.py classOddEven: def__init__(self,data): self._data=data self.indexes=(list(range(0,len(data),2))+ list(range(1,len(data),2))) def__iter__(self): returnself def__next__(self): ifself.indexes: returnself._data[self.indexes.pop(0)] raiseStopIteration oddeven=OddEven('ThIsIsCoOl!') print(''.join(cforcinoddeven))#TIICO!hssol oddeven=OddEven('HoLa')#ormanually… it=iter(oddeven)#thiscallsoddeven.__iter__internally print(next(it))#H print(next(it))#L print(next(it))#o print(next(it))#a So,weneededtoprovideanimplementationfor__iter__whichreturnedtheobjectitself, andthenonefor__next__.Let’sgothroughit.Whatneedstohappenisthatwereturn _data[0],_data[2],_data[4],…,_data[1],_data[3],_data[5],…untilwehave returnedeveryiteminthedata.Inordertodothis,wepreparealist,indexes,like[0,2,4, 6,…,1,3,5,…],andwhilethereisatleastanelementinit,wepopthefirstoneand returntheelementfromthedatathatisatthatposition,therebyachievingourgoal.When indexesisempty,weraiseStopIteration,asrequiredbytheiteratorprotocol. WOW! eBook www.wowebook.org Thereareotherwaystoachievethesameresult,sogoaheadandtrytocodeadifferent oneyourself.Makesuretheendresultworksforalledgecases,emptysequences, sequencesoflength1,2,andsoon. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,wesawdecorators,discoveredthereasonsforhavingthem,andafew examplesusingoneormoreatthesametime.Wealsosawdecoratorsthattakearguments, whichareusuallyusedasdecoratorfactories. Wescratchedthesurfaceofobject-orientedprogramminginPython.Wecoveredallthe basicsinawaythatyoushouldnowbeabletounderstandfairlyeasilythecodethatwill comeinfuturechapters.Wetalkedaboutallkindsofmethodsandattributesthatonecan writeinaclass,weexploredinheritanceversuscomposition,methodoverriding, properties,operatoroverloading,andpolymorphism. Attheend,weverybrieflytouchedbaseoniterators,sonowyouhavealltheknowledge toalsounderstandgeneratorsmoredeeply. Inthenextchapter,wetakeasteepturn.Itwillstartthesecondhalfofthebook,whichis muchmoreproject-orientedso,fromnowon,itwillbelesstheoryandmorecode,Ihope youwillenjoyfollowingtheexamplesandgettingyourhandsdirty,verydirty. Theysaythatasmoothseanevermadeaskillfulsailor,sokeepexploring,breakthings, readtheerrormessagesaswellasthedocumentation,andlet’sseeifwecangettoseethat whiterabbit. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter7.Testing,Profiling,andDealing withExceptions “Codewithouttestsisbrokenbydesign.” —JacobKaplan-Moss JacobKaplan-MossisoneofthecoredevelopersoftheDjangowebframework.We’re goingtoexploreitinthenextchapters.Istronglyagreewiththisquoteofhis.Ibelieve codewithouttestsshouldn’tbedeployedtoproduction. Whyaretestssoimportant?Well,forone,theygiveyoupredictability.Or,atleast,they helpyouachievehighpredictability.Unfortunately,thereisalwayssomebugthatsneaks intoourcode.Butwedefinitelywantourcodetobeaspredictableaspossible.Whatwe don’twantistohaveasurprise,ourcodebehavinginanunpredictableway.Wouldyoube happytoknowthatthesoftwarethatchecksonthesensorsoftheplanethatistakingyou onholidayssometimesgoescrazy?No,probablynot. Thereforeweneedtotestourcode,weneedtocheckthatitsbehavioriscorrect,thatit worksasexpectedwhenitdealswithedgecases,thatitdoesn’thangwhenthe componentsit’stalkingtoaredown,thattheperformancesarewellwithintheacceptable range,andsoon. Thischapterisallaboutthistopic,makingsurethatyourcodeispreparedtofacethe scaryoutsideworld,thatisfastenoughandthatitcandealwithunexpectedorexceptional conditions. We’regoingtoexploretesting,includingabriefintroductiontotest-drivendevelopment (TDD),whichisoneofmyfavoriteworkingmethodologies.Then,we’regoingtoexplore theworldofexceptions,andfinallywe’regoingtotalkalittlebitaboutperformancesand profiling.Deepbreath,andherewego… WOW! eBook www.wowebook.org Testingyourapplication Therearemanydifferentkindsoftests,somanyinfactthatcompaniesoftenhavea dedicateddepartment,calledqualityassurance(QA),madeupofindividualsthatspend theirdaytestingthesoftwarethecompanydevelopersproduce. Tostartmakinganinitialclassification,wecandividetestsintotwobroadcategories: white-boxandblack-boxtests. White-boxtestsarethosewhichexercisetheinternalsofthecode,theyinspectitdownto averyfinelevelofgranularity.Ontheotherhand,black-boxtestsarethosewhich considerthesoftwareundertestingasifbeingwithinabox,theinternalsofwhichare ignored.Eventhetechnology,orthelanguageusedinsidetheboxisnotimportantfor black-boxtests.Whattheydoistopluginputtooneendoftheboxandverifytheoutput attheotherend,andthat’sit. Note Thereisalsoanin-betweencategory,calledgray-boxtesting,thatinvolvestestinga systeminthesamewaywedowiththeblack-boxapproach,buthavingsomeknowledge aboutthealgorithmsanddatastructuresusedtowritethesoftwareandonlypartialaccess toitssourcecode. Therearemanydifferentkindsoftestsinthesecategories,eachofwhichservesadifferent purpose.Justtogiveyouanidea,here’safew: Front-endtestsmakesurethattheclientsideofyourapplicationisexposingthe informationthatitshould,allthelinks,thebuttons,theadvertising,everythingthat needstobeshowntotheclient.Itmayalsoverifythatitispossibletowalkacertain paththroughtheuserinterface. Scenariotestsmakeuseofstories(orscenarios)thathelpthetesterworkthrougha complexproblemortestapartofthesystem. Integrationtestsverifythebehaviorofthevariouscomponentsofyourapplication whentheyareworkingtogethersendingmessagesthroughinterfaces. Smoketestsareparticularlyusefulwhenyoudeployanewupdateonyour application.Theycheckwhetherthemostessential,vitalpartsofyourapplicationare stillworkingastheyshouldandthattheyarenotonfire.Thistermcomesfromwhen engineerstestedcircuitsbymakingsurenothingwassmoking. Acceptancetests,oruseracceptancetesting(UAT)iswhatadeveloperdoeswitha productowner(forexample,inaSCRUMenvironment)todetermineiftheworkthat wascommissionedwascarriedoutcorrectly. Functionaltestsverifythefeaturesorfunctionalitiesofyoursoftware. Destructiveteststakedownpartsofyoursystem,simulatingafailure,inorderto establishhowwelltheremainingpartsofthesystemperform.Thesekindsoftestsare performedextensivelybycompaniesthatneedtoprovideanextremelyreliable service,suchasAmazon,forexample. Performancetestsaimtoverifyhowwellthesystemperformsunderaspecificload WOW! eBook www.wowebook.org ofdataortrafficsothat,forexample,engineerscangetabetterunderstandingof whicharethebottlenecksinthesystemthatcouldbringitdowntoitskneesina heavyloadsituation,orthosewhichpreventscalability. Usabilitytests,andthecloselyrelateduserexperience(UX)tests,aimtocheckif theuserinterfaceissimpleandeasytounderstandanduse.Theyaimtoprovideinput tothedesignerssothattheuserexperienceisimproved. Securityandpenetrationtestsaimtoverifyhowwellthesystemisprotected againstattacksandintrusions. Unittestshelpthedevelopertowritethecodeinarobustandconsistentway, providingthefirstlineoffeedbackanddefenseagainstcodingmistakes,refactoring mistakes,andsoon. Regressiontestsprovidethedeveloperwithusefulinformationaboutafeaturebeing compromisedinthesystemafteranupdate.Someofthecausesforasystembeing saidtohavearegressionareanoldbugcomingbacktolife,anexistingfeaturebeing compromised,oranewissuebeingintroduced. Manybooksandarticleshavebeenwrittenabouttesting,andIhavetopointyoutothose resourcesifyou’reinterestedinfindingoutmoreaboutallthedifferentkindsoftests.In thischapter,wewillconcentrateonunittests,sincetheyarethebackboneofsoftware craftingandformthevastmajorityofteststhatarewrittenbyadeveloper. Testingisanart,anartthatyoudon’tlearnfrombooks,I’mafraid.Youcanlearnallthe definitions(andyoushould),andtryandcollectasmuchknowledgeabouttestingasyou canbutIpromiseyou,youwillbeabletotestyoursoftwareproperlyonlywhenyouhave doneitforlongenoughinthefield. Whenyouarehavingtroublerefactoringabitofcode,becauseeverylittlethingyoutouch makesatestblowup,youlearnhowtowritelessrigidandlimitingtests,whichstillverify thecorrectnessofyourcodebut,atthesametime,allowyouthefreedomandjoytoplay withit,toshapeitasyouwant. Whenyouarebeingcalledtoooftentofixunexpectedbugsinyourcode,youlearnhow towritetestsmorethoroughly,howtocomeupwithamorecomprehensivelistofedge cases,andstrategiestocopewiththembeforetheyturnintobugs. Whenyouarespendingtoomuchtimereadingtestsandtryingtorefactortheminorderto changeasmallfeatureinthecode,youlearntowritesimpler,shorter,andbetterfocused tests. Icouldgoonwiththiswhenyou…youlearn…,butIguessyougetthepicture.Youneed togetyourhandsdirtyandbuildexperience.Mysuggestion?Studythetheoryasmuchas youcan,andthenexperimentusingdifferentapproaches.Also,trytolearnfrom experiencedcoders;it’sveryeffective. WOW! eBook www.wowebook.org Theanatomyofatest Beforeweconcentrateonunittests,let’sseewhatatestis,andwhatitspurposeis. Atestisapieceofcodewhosepurposeistoverifysomethinginoursystem.Itmaybe thatwe’recallingafunctionpassingtwointegers,thatanobjecthasapropertycalled donald_duck,orthatwhenyouplaceanorderonsomeAPI,afteraminuteyoucanseeit dissectedintoitsbasicelements,inthedatabase. Atestistypicallycomprisedofthreesections: Preparation:Thisiswhereyousetupthescene.Youprepareallthedata,the objects,theservicesyouneedintheplacesyouneedthemsothattheyarereadytobe used. Execution:Thisiswhereyouexecutethebitoflogicthatyou’recheckingagainst. Youperformanactionusingthedataandtheinterfacesyouhavesetupinthe preparationphase. Verification:Thisiswhereyouverifytheresultsandmakesuretheyareaccordingto yourexpectations.Youcheckthereturnedvalueofafunction,orthatsomedataisin thedatabase,someisnot,somehaschanged,arequesthasbeenmade,somethinghas happened,amethodhasbeencalled,andsoon. WOW! eBook www.wowebook.org Testingguidelines Likesoftware,testscanbegoodorbad,withthewholerangeofshadesinthemiddle.In ordertowritegoodtests,herearesomeguidelines: Keepthemassimpleaspossible:It’sokaytoviolatesomegoodcodingrules,such ashardcodingvaluesorduplicatingcode.Testsneedfirstandforemosttobeas readableaspossibleandeasytounderstand.Whentestsarehardtoreador understand,youcanneverbesureiftheyareactuallymakingsureyourcodeis performingcorrectly. Testsshouldverifyonethingandonethingonly:It’sveryimportantthatyoukeep themshortandcontained.It’sperfectlyfinetowritemultipleteststoexerciseasingle objectorfunction.Justmakesurethateachtesthasoneandonlyonepurpose. Testsshouldnotmakeanyunnecessaryassumptionwhenverifyingdata:Thisis trickytounderstandatfirst,butsayyouaretestingthereturnvalueofafunctionand itisanunorderedlistofnumbers(like[2,3,1]).Iftheorderinthatlistisrandom, inthetestyoumaybetemptedtosortitandcompareitwith[1,2,3].Ifyoudo, youwillintroduceanextraassumptionontheorderingoftheresultofyourfunction call,andthisisbadpractice.Youshouldalwaysfindawaytoverifythingswithout introducinganyassumptionsoranyfeaturethatdoesn’tbelongintheusecaseyou’re describingwithyourtest. Testsshouldexercisethewhat,ratherthanthehow:Testsshouldfocuson checkingwhatafunctionissupposedtodo,ratherthanhowitisdoingit.For example,focusonthefactthatit’scalculatingthesquarerootofanumber(thewhat), insteadofonthefactthatitiscallingmath.sqrttodoit(thehow).Unlessyou’re writingperformancetestsoryouhaveaparticularneedtoverifyhowacertainaction isperformed,trytoavoidthistypeoftestingandfocusonthewhat.Testingthehow leadstorestrictivetestsandmakesrefactoringhard.Moreover,thetypeoftestyou havetowritewhenyouconcentrateonthehowismorelikelytodegradethequality ofyourtestingcodebasewhenyouamendyoursoftwarefrequently(moreonthis later). Testsshouldassumetheleastpossibleinthepreparationphase:Sayyouhave10 teststhatarecheckinghowadatastructureismanipulatedbyafunction.Andlet’s saythisdatastructureisadictwithfivekey/valuepairs.Ifyouputthecompletedict ineachtest,themomentyouhavetochangesomethinginthatdict,youalsohaveto amendalltentests.Ontheotherhand,ifyoustripdownthetestdataasmuchasyou can,youwillfindthat,mostofthetime,it’spossibletohavethemajorityoftests checkingonlyapartialversionofthedata,andonlyafewrunningwithafullversion ofit.Thismeansthatwhenyouneedtochangeyourdata,youwillhavetoamend onlythoseteststhatareactuallyexercisingit. Testshouldrunasfastaspossible:Agoodtestcodebasecouldendupbeingmuch longerthanthecodebeingtesteditself.Itvariesaccordingtothesituationandthe developerbutwhateverthelength,you’llenduphavinghundreds,ifnotthousands, ofteststorun,whichmeansthefastertheyrun,thefasteryoucangetbacktowriting code.WhenusingTDD,forexample,youruntestsveryoften,sospeedisessential. WOW! eBook www.wowebook.org Testsshoulduseuptheleastpossibleamountofresources:Thereasonforthisis thateverydeveloperwhochecksoutyourcodeshouldbeabletorunyourtests,no matterhowpowerfultheirboxis.Itcouldbeaskinnyvirtualmachineoraneglected Jenkinsbox,yourtestsshouldrunwithoutchewinguptoomanyresources. Note AJenkinsboxisamachinethatrunsJenkins,softwarethatiscapableof,amongst manyotherthings,runningyourtestsautomatically.Jenkinsisfrequentlyusedin companieswheredevelopersusepracticeslikecontinuousintegration,extreme programming,andsoon. WOW! eBook www.wowebook.org Unittesting Nowthatyouhaveanideaaboutwhattestingisandwhyweneedit,let’sfinallyintroduce thedeveloper’sbestfriend:theunittest. Beforeweproceedwiththeexamples,allowmetospendsomewordsofcaution:I’lltryto giveyouthefundamentalsaboutunittesting,butIdon’tfollowanyparticularschoolof thoughtormethodologytotheletter.Overtheyears,Ihavetriedmanydifferenttesting approaches,eventuallycomingupwithmyownwayofdoingthings,whichisconstantly evolving.ToputitasBruceLeewouldhave: “Absorbwhatisuseful,discardwhatisuselessandaddwhatisspecificallyyour own”. Writingaunittest Inordertoexplainhowtowriteaunittest,let’shelpourselveswithasimplesnippet: data.py defget_clean_data(source): data=load_data(source) cleaned_data=clean_data(data) returncleaned_data Thefunctionget_clean_dataisresponsibleforgettingdatafromsource,cleaningit,and returningittothecaller.Howdowetestthisfunction? Onewayofdoingthisistocallitandthenmakesurethatload_datawascalledoncewith sourceasitsonlyargument.Thenwehavetoverifythatclean_datawascalledonce, withthereturnvalueofload_data.And,finally,wewouldneedtomakesurethatthe returnvalueofclean_dataiswhatisreturnedbytheget_clean_datafunctionaswell. Inordertodothis,weneedtosetupthesourceandrunthiscode,andthismaybea problem.Oneofthegoldenrulesofunittestingisthatanythingthatcrossesthe boundariesofyourapplicationneedstobesimulated.Wedon’twanttotalktoarealdata source,andwedon’twanttoactuallyrunrealfunctionsiftheyarecommunicatingwith anythingthatisnotcontainedinourapplication.Afewexampleswouldbeadatabase,a searchservice,anexternalAPI,afileinthefilesystem,andsoon. Weneedtheserestrictionstoactasashield,sothatwecanalwaysrunourtestssafely withoutthefearofdestroyingsomethinginarealdatasource. Anotherreasonisthatitmaybequitedifficultforasingledevelopertoreproducethe wholearchitectureontheirbox.Itmayrequirethesettingupofdatabases,APIs,services, filesandfolders,andsoonandsoforth,andthiscanbedifficult,timeconsuming,or sometimesnotevenpossible. Note Verysimplyput,anapplicationprogramminginterface(API)isasetoftoolsfor WOW! eBook www.wowebook.org buildingsoftwareapplications.AnAPIexpressesasoftwarecomponentintermsofits operations,inputsandoutputs,andunderlyingtypes.Forexample,ifyoucreateasoftware thatneedstointerfacewithadataproviderservice,it’sverylikelythatyouwillhavetogo throughtheirAPIinordertogainaccesstothedata. Therefore,inourunittests,weneedtosimulateallthosethingsinsomeway.Unittests needtoberunbyanydeveloperwithouttheneedforthewholesystemtobesetupon theirbox. Adifferentapproach,whichIalwaysfavorwhenit’spossibletodoso,istosimulate entitieswithoutusingfakeobjects,butusingspecialpurposetestobjectsinstead.For example,ifyourcodetalkstoadatabase,insteadoffakingallthefunctionsandmethods thattalktothedatabaseandprogrammingthefakeobjectssothattheyreturnwhatthereal oneswould,I’dmuchratherprefertospawnatestdatabase,setupthetablesanddataI need,andthenpatchtheconnectionsettingssothatmytestsarerunningrealcode,against thetestdatabase,therebydoingnoharmatall.In-memorydatabasesareexcellentoptions forthesecases. Note Oneoftheapplicationsthatallowyoutospawnadatabasefortesting,isDjango.Within thedjango.testpackageyoucanfindseveraltoolsthathelpyouwriteyourtestssothat youwon’thavetosimulatethedialogwithadatabase.Bywritingteststhisway,youwill alsobeabletocheckontransactions,encodings,andallotherdatabaserelatedaspectsof programming.Anotheradvantageofthisapproachconsistsintheabilityofchecking againstthingsthatcanchangefromonedatabasetoanother. Sometimes,though,it’sstillnotpossible,andweneedtousefakes,thereforelet’stalk aboutthem. Mockobjectsandpatching Firstofall,inPython,thesefakeobjectsarecalledmocks.Uptoversion3.3,themock librarywasathird-partylibrarythatbasicallyeveryprojectwouldinstallviapipbut,from version3.3,ithasbeenincludedinthestandardlibraryundertheunittestmodule,and rightfullyso,givenitsimportanceandhowwidespreaditis. Theactofreplacingarealobjectorfunction(oringeneral,anypieceofdatastructure) withamock,iscalledpatching.Themocklibraryprovidesthepatchtool,whichcanact asafunctionorclassdecorator,andevenasacontextmanager(moreonthisinChapter8, TheEdges–GUIsandScripts),thatyoucanusetomockthingsout.Onceyouhave replacedeverythingyouneednottorun,withsuitablemocks,youcanpasstothesecond phaseofthetestandrunthecodeyouareexercising.Aftertheexecution,youwillbeable tocheckthosemockstoverifythatyourcodehasworkedcorrectly. Assertions Theverificationphaseisdonethroughtheuseofassertions.Anassertionisafunction(or method)thatyoucanusetoverifyequalitybetweenobjects,aswellasotherconditions. Whenaconditionisnotmet,theassertionwillraiseanexceptionthatwillmakeyourtest WOW! eBook www.wowebook.org fail.Youcanfindalistofassertionsintheunittestmoduledocumentation,andtheir correspondingPythonicversioninthenosethird-partylibrary,whichprovidesafew advantagesoverthesheerunittestmodule,startingfromanimprovedtestdiscovery strategy(whichisthewayatestrunnerdetectsanddiscoversthetestsinyourapplication). Aclassicunittestexample Mocks,patches,andassertionsarethebasictoolswe’llbeusingtowritetests.So,finally, let’sseeanexample.I’mgoingtowriteafunctionthattakesalistofintegersandfilters outallthosewhicharen’tpositive. filter_funcs.py deffilter_ints(v): return[numfornuminvifis_positive(num)] defis_positive(n): returnn>0 Intheprecedingexample,wedefinethefilter_intsfunction,whichbasicallyusesalist comprehensiontoretainallthenumbersinvthatarepositive,discardingzerosand negativeones.Ihope,bynow,anyfurtherexplanationofthecodewouldbeinsulting. Whatisinteresting,though,istostartthinkingabouthowwecantestit.Well,howabout wecallfilter_intswithalistofnumbersandwemakesurethatis_positiveiscalled foreachofthem?Ofcourse,wewouldhavetotestis_positiveaswell,butIwillshow youlateronhowtodothat.Let’swriteasimpletestforfilter_intsnow. Note Justtobesurewe’reonthesamepage,Iamputtingthecodeforthischapterinafolder calledch7,whichlieswithintherootofourproject.Atthesamelevelofch7,Ihave createdafoldercalledtests,inwhichIhaveplacedafoldercalledtest_ch7.Inthis folderIhaveonetestfile,calledtest_filter_func.py. Basically,withinthetestsfolder,IwillrecreatethetreestructureofthecodeI’mtesting, prependingeverythingwithtest_.Thisway,findingtestsisreallyeasy,aswellasis keepingthemtidy. tests/test_ch7/test_filter_funcs.py fromunittestimportTestCase#1 fromunittest.mockimportpatch,call#2 fromnose.toolsimportassert_equal#3 fromch7.filter_funcsimportfilter_ints#4 classFilterIntsTestCase(TestCase):#5 @patch('ch7.filter_funcs.is_positive')#6 deftest_filter_ints(self,is_positive_mock):#7 #preparation v=[3,-4,0,5,8] #execution WOW! eBook www.wowebook.org filter_ints(v)#8 #verification assert_equal( [call(3),call(-4),call(0),call(5),call(8)], is_positive_mock.call_args_list )#9 My,ohmy,solittlecode,andyetsomuchtosay.Firstofall:#1.TheTestCaseclassisthe baseclassthatweusetohaveacontainedentityinwhichtorunourtests.It’snotjusta barecontainer;itprovidesyouwithmethodstowritetestsmoreeasily. On#2,weimportpatchandcallfromtheunittest.mockmodule.patchisresponsible forsubstitutinganobjectwithaMockinstance,therebygivingustheabilitytocheckonit aftertheexecutionphasehasbeencompleted.callprovidesuswithanicewayof expressinga(forexample,function)call. On#3,youcanseethatIprefertouseassertionsfromnose,ratherthantheonesthatcome withtheunittestmodule.Togiveyouanexample,assert_equal(...)wouldbecome self.assertEqual(...)ifIdidn’tusenose.Idon’tenjoytypingself.foranyassertion, ifthereisawaytoavoidit,andIdon’tparticularlyenjoycamelcase,thereforeIalways prefertousenosetomakemyassertions. assert_equalisafunctionthattakestwoparameters(andanoptionalthirdonethatacts asamessage)andverifiesthattheyarethesame.Iftheyareequal,nothinghappens,butif theydiffer,thenanAssertionErrorexceptionisraised,tellingussomethingiswrong. WhenIwritemytests,Ialwaysputtheexpectedvalueasthefirstargument,andthereal oneasthesecond.ThisconventionsavesmetimewhenI’mreadingtests. On#4,weimportthefunctionwewanttotest,andthen(#5)weproceedtocreatetheclass whereourtestswilllive.Eachmethodofthisclassstartingwithtest_,willbeinterpreted asatest.Asyoucansee,weneedtodecoratetest_filter_intswithpatch(#6). Understandingthispartiscrucial,weneedtopatchtheobjectwhereitisactuallyused.In thiscase,thepathisverysimple:ch7.filter_func.is_positive. Tip Patchingcanbeverytricky,soIurgeyoutoreadtheWheretopatchsectioninthemock documentation:https://docs.python.org/3/library/unittest.mock.html#where-to-patch. Whenwedecorateafunctionusingpatch,likeinourexample,wegetanextraargument inthetestsignature(#7),whichIliketocallasthepatchedfunctionname,plusa_mock suffix,justtomakeitclearthattheobjecthasbeenpatched(ormockedout).). Finally,wegettothebodyofthetest,andwehaveaverysimplepreparationphasein whichwesetupalistwithatleastonerepresentativeofalltheintegernumbercategories (negative,zero,andpositive). Then,in#8,weperformtheexecutionphase,whichrunsthefilter_intsfunction, withoutcollectingitsresults.Ifallhasgoneasexpected,thefakeis_positivefunction musthavebeencalledwitheachoftheintegersinv. WOW! eBook www.wowebook.org Wecanverifythisbycomparingalistofcallobjectstothecall_args_listattributeon themock(#9).Thisattributeisthelistofallthecallsperformedontheobjectsinceits creation. Let’srunthistest.Firstofall,makesurethatyouinstallnose($pipfreezewilltellyou ifyouhaveitalready): $pipinstallnose Then,changeintotherootoftheproject(mineiscalledlearning.python),andrunthe testslikethis: $noseteststests/test_ch7/ . -----------------------------------------------------------Ran1testin0.006s OK Theoutputshowsonedot(eachdotisatest),aseparationline,andthetimetakentorun thewholetestsuite.ItalsosaysOKattheend,whichmeansthatourtestswereall successful. Makingatestfail Good,sojustforfunlet’smakeonefail.Inthetestfile,changethelastcallfromcall(8) tocall(9),andrunthetestsagain: $noseteststests/test_ch7/ F ============================================================ FAIL:test_filter_ints(test_filter_funcs.FilterIntsTestCase) -----------------------------------------------------------Traceback(mostrecentcalllast): File"/usr/lib/python3.4/unittest/mock.py",line1125,inpatched returnfunc(*args,**keywargs) File"/home/fab/srv/learning.python/tests/test_ch7/test_filter_funcs.py", line21,intest_filter_ints is_positive_mock.call_args_list AssertionError:[call(3),call(-4),call(0),call(5),call(9)]!=[call(3), call(-4),call(0),call(5),call(8)] -----------------------------------------------------------Ran1testin0.008s FAILED(failures=1) Wow,wemadethebeastangry!Somuchwonderfulinformation,though.Thistellsyou thatthetesttest_filter_ints(withthepathtoit),wasrunandthatitfailed(thebigFat thetop,wherethedotwasbefore).ItgivesyouaTraceback,thattellsyouthatinthe test_filter_funcs.pymodule,atline21,whenassertingon is_positive_mock.call_args_list,wehaveadiscrepancy.Thetestexpectsthelistof callstoendwithacall(9)instance,butthereallistendswithacall(8).Thisisnothing lessthanwonderful. Ifyouhaveatestlikethis,canyouimaginewhatwouldhappenifyourefactoredand introducedabugintoyourfunctionbymistake?Well,yourtestswillbreak!Theywilltell WOW! eBook www.wowebook.org youthatyouhavescrewedsomethingup,andhere’sthedetails.So,yougoandcheckout whatyoubroke. Interfacetesting Let’saddanothertestthatchecksonthereturnedvalue.It’sanothermethodintheclass, soIwon’treproducethewholecodeagain: tests/test_ch7/test_filter_funcs.py deftest_filter_ints_return_value(self): v=[3,-4,0,-2,5,0,8,-1] result=filter_ints(v) assert_list_equal([3,5,8],result) Thistestisabitdifferentfromthepreviousone.Firstly,wecannotmocktheis_positive function,otherwisewewouldn’tbeabletocheckontheresult.Secondly,wedon’tcheck oncalls,butonlyontheresultofthefunctionwheninputisgiven. Ilikethistestmuchmorethanthepreviousone.Thistypeoftestiscalledaninterface testbecauseitchecksontheinterface(thesetofinputsandoutputs)ofthefunctionwe’re testing.Itdoesn’tuseanymocks,whichiswhyIusethistechniquemuchmorethanthe previousone.Let’srunthenewtestsuiteandthenlet’sseewhyIlikeinterfacetesting morethanthosewithmocks. $noseteststests/test_ch7/ .. -----------------------------------------------------------Ran2testsin0.006s OK Twotestsran,allgood(Ichangedthat9backtoan8inthefirsttest,ofcourse). Comparingtestswithandwithoutmocks Now,let’sseewhyIdon’treallylikemocksandusethemonlywhenIhavenochoice. Let’srefactorthecodeinthisway: filter_funcs_refactored.py deffilter_ints(v): v=[numfornuminvifnum!=0]#1 return[numfornuminvifis_positive(num)] Thecodeforis_positiveisthesameasbefore.Butthelogicinfilter_intshasnow changedinawaythatis_positivewillneverbecalledwitha0,sincetheyareallfiltered outin#1.Thisleadstoaninterestingresult,solet’srunthetestsagain: $noseteststests/test_ch7/test_filter_funcs_refactored.py F. ============================================================ FAIL:test_filter_ints(test_filter_funcs_refactored.FilterIntsTestCase) -----------------------------------------------------------...omit… WOW! eBook www.wowebook.org AssertionError:[call(3),call(-4),call(0),call(5),call(8)]!=[call(3), call(-4),call(5),call(8)] -----------------------------------------------------------Ran2testsin0.002s FAILED(failures=1) Onetestsucceededbuttheotherone,theonewiththemockedis_positivefunction, failed.TheAssertionErrormessageshowsusthatwenowneedtoamendthelistof expectedcalls,removingcall(0),becauseitisnolongerperformed. Thisisnotgood.Wehavechangedneithertheinterfaceofthefunctionnoritsbehavior. Thefunctionisstillkeepingtoitsoriginalcontract.Whatwe’vedonebytestingitwitha mockedobjectislimitourselves.Infact,wenowhavetoamendthetestinordertousethe newlogic. Thisisjustasimpleexamplebutitshowsoneimportantflawinthewholemock mechanism.Youmustkeepyourmocksup-to-dateandinsyncwiththecodetheyare replacing,otherwiseyouriskhavingissuesliketheprecedingone,orevenworse.Your testsmaynotfailbecausetheyareusingmockedobjectsthatperformfine,butbecausethe realones,nownotinsyncanymore,areactuallyfailing. Sousemocksonlywhennecessary,onlywhenthereisnootherwayoftestingyour functions.Whenyoucrosstheboundariesofyourapplicationinatest,trytousea replacement,likeatestdatabase,orafakeAPI,andonlywhenit’snotpossible,resortto mocks.Theyareverypowerful,butalsoverydangerouswhennothandledproperly. So,let’sremovethatfirsttestandkeeponlythesecondone,sothatIcanshowyou anotherissueyoucouldrunintowhenwritingtests.Thewholetestmodulenowlookslike this: tests/test_ch7/test_filter_funcs_final.py fromunittestimportTestCase fromnose.toolsimportassert_list_equal fromch7.filter_funcsimportfilter_ints classFilterIntsTestCase(TestCase): deftest_filter_ints_return_value(self): v=[3,-4,0,-2,5,0,8,-1] result=filter_ints(v) assert_list_equal([3,5,8],result) Ifwerunit,itwillpass. Abriefchatabouttriangulation.Nowletmeaskyou:whathappensifIchangemy filter_intsfunctiontothis? filter_funcs_triangulation.py deffilter_ints(v): return[3,5,8] Ifyourunthetestsuite,thetestwehavewillstillpass!YoumaythinkI’mcrazybutI’m showingyouthisbecauseIwanttotalkaboutaconceptcalledtriangulation,whichis WOW! eBook www.wowebook.org veryimportantwhendoinginterfacetestingwithTDD. Thewholeideaistoremovecheatingcode,orbadlyperformingcode,bypinpointingit fromdifferentangles(likegoingtoonevertexofatrianglefromtheothertwo)inaway thatmakesitimpossibleforourcodetocheat,andthebugisexposed.Wecansimply modifythetestlikethis: tests/test_ch7/test_filter_funcs_final_triangulation.py deftest_filter_ints_return_value(self): v1=[3,-4,0,-2,5,0,8,-1] v2=[7,-3,0,0,9,1] assert_list_equal([3,5,8],filter_ints(v1)) assert_list_equal([7,9,1],filter_ints(v2)) Ihavemovedtheexecutionsectionintheassertionsdirectly,andyoucanseethatwe’re nowpinpointingourfunctionfromtwodifferentangles,therebyrequiringthatthereal codebeinit.It’snolongerpossibleforourfunctiontocheat. Triangulationisaverypowerfultechniquethatteachesustoalwaystrytoexerciseour codefrommanydifferentangles,tocoverallpossibleedgecasestoexposeanyproblems. Boundariesandgranularity Let’snowaddatestfortheis_positivefunction.Iknowit’saone-liner,butitpresents uswithopportunitytodiscusstwoveryimportantconcepts:boundariesandgranularity. That0inthebodyofthefunctionisaboundary,the>intheinequalityishowwebehave withregardstothisboundary.Typically,whenyousetaboundary,youdividethespace intothreeareas:whatliesbeforetheboundary,aftertheboundary,andontheboundary itself.Intheexample,beforetheboundarywefindthenegativenumbers,theboundaryis theelement0and,aftertheboundary,wefindthepositivenumbers.Weneedtotesteach oftheseareastobesurewe’retestingthefunctioncorrectly.So,let’sseeonepossible solution(Iwilladdthetesttotheclass,butIwon’tshowtherepeatedcode): tests/test_ch7/test_filter_funcs_is_positive_loose.py deftest_is_positive(self): assert_equal(False,is_positive(-2))#beforeboundary assert_equal(False,is_positive(0))#ontheboundary assert_equal(True,is_positive(2))#aftertheboundary Youcanseethatweareexercisingonenumberforeachdifferentareaaroundthe boundary.Doyouthinkthistestisgood?Thinkaboutitforaminutebeforereadingon. Theanswerisno,thistestisnotgood.Notgoodenough,anyway.IfIchangethebodyof theis_positivefunctiontoreadreturnn>1,Iwouldexpectmytesttofail,butit won’t.-2isstillFalse,aswellas0,and2isstillTrue.Whydoesthathappen?Itis becausewehaven’ttakengranularityproperlyintoaccount.We’redealingwithintegers, sowhatistheminimumgranularitywhenwemovefromoneintegertothenextone?It’s 1.Therefore,whenwesurroundtheboundary,takingallthreeareasintoaccountisnot enough.Weneedtodoitwiththeminimumpossiblegranularity.Let’schangethetest: WOW! eBook www.wowebook.org tests/test_ch7/test_filter_funcs_is_positive_correct.py deftest_is_positive(self): assert_equal(False,is_positive(-1)) assert_equal(False,is_positive(0)) assert_equal(True,is_positive(1)) Ah,nowit’sbetter.Nowifwechangethebodyofis_positivetoreadreturnn>1,the thirdassertionwillfail,whichiswhatwewant.Canyouthinkofabettertest? tests/test_ch7/test_filter_funcs_is_positive_better.py deftest_is_positive(self): assert_equal(False,is_positive(0)) forninrange(1,10**4): assert_equal(False,is_positive(-n)) assert_equal(True,is_positive(n)) Thistestisevenbetter.Wetestthefirsttenthousandintegers(bothpositiveandnegative) and0.Itbasicallyprovidesuswithabettercoveragethanjusttheoneacrosstheboundary. So,keepthisinmind.Zoomcloselyaroundeachboundarywithminimalgranularity,but trytoexpandaswell,findingagoodcompromisebetweenoptimalcoverageand executionspeed.Wewouldlovetocheckthefirstbillionintegers,butwecan’twaitdays forourteststorun. Amoreinterestingexample Okay,thiswasasgentleanintroductionasIcouldgiveyou,solet’smoveontosomething moreinteresting.Let’swriteandtestafunctionthatflattensanesteddictionarystructure. Foracoupleofyears,IhaveworkedverycloselywithTwitterandFacebookAPIs. Handlingsuchhumongousdatastructuresisnoteasy,especiallysincethey’reoftendeeply nested.Itturnsoutthatit’smucheasiertoflattentheminawaythatyoucanworkon themwithoutlosingtheoriginalstructuralinformation,andthenrecreatethenested structurefromtheflatone.Togiveyouanexample,wewantsomethinglikethis: data_flatten.py nested={ 'fullname':'Alessandra', 'age':41, 'phone-numbers':['+447421234567','+447423456789'], 'residence':{ 'address':{ 'first-line':'AlexandraRd', 'second-line':'', }, 'zip':'N80PP', 'city':'London', 'country':'UK', }, } flat={ 'fullname':'Alessandra', 'age':41, WOW! eBook www.wowebook.org 'phone-numbers':['+447421234567','+447423456789'], 'residence.address.first-line':'AlexandraRd', 'residence.address.second-line':'', 'residence.zip':'N80PP', 'residence.city':'London', 'residence.country':'UK', } Astructurelikeflatismuchsimplertomanipulate.Beforewritingtheflattener,let’s makesomeassumptions:thekeysarestrings,weleaveeverydatastructureasitisunless it’sadictionary,inwhichcaseweflattenit,weusethedotasseparator,butwewanttobe abletopassadifferentonetoourfunction.Here’sthecode: data_flatten.py defflatten(data,prefix='',separator='.'): """Flattensanesteddictstructure.""" ifnotisinstance(data,dict): return{prefix:data}ifprefixelsedata result={} for(key,value)indata.items(): result.update( flatten( value, _get_new_prefix(prefix,key,separator), separator=separator)) returnresult def_get_new_prefix(prefix,key,separator): return(separator.join((prefix,str(key))) ifprefixelsestr(key)) Theprecedingexampleisnotdifficult,butalsonottrivialsolet’sgothroughit.Atfirst, wecheckifdataisadictionary.Ifit’snotadictionary,thenit’sdatathatdoesn’tneedto beflattened;therefore,wesimplyreturneitherdataor,ifprefixisnotanemptystring,a dictionarywithonekey/valuepair:prefix/data. Ifinsteaddataisadict,weprepareanemptyresultdicttoreturn,thenweparsethelist ofdata‘sitems,which,atI’msureyouwillremember,are2-tuples(key,value).Foreach (key,value)pair,werecursivelycallflattenonthem,andweupdatetheresultdictwith what’sreturnedbythatcall.Recursionisexcellentwhenrunningthroughnested structures. Ataglance,canyouunderstandwhatthe_get_new_prefixfunctiondoes?Let’susethe inside-outtechniqueonceagain.Iseeaternaryoperatorthatreturnsthestringifiedkey whenprefixisanemptystring.Ontheotherhand,whenprefixisanon-emptystring, weusetheseparatortojointheprefixwiththestringifiedversionofkey.Noticethat thebracesinsidethecalltojoinaren’tredundant,weneedthem.Canyoufigureoutwhy? Let’swriteacoupleoftestsforthisfunction: tests/test_ch7/test_data_flatten.py #...importsomitted… WOW! eBook www.wowebook.org classFlattenTestCase(TestCase): deftest_flatten(self): test_cases=[ ({'A':{'B':'C','D':[1,2,3],'E':{'F':'G'}}, 'H':3.14, 'J':['K','L'], 'M':'N'}, {'A.B':'C', 'A.D':[1,2,3], 'A.E.F':'G', 'H':3.14, 'J':['K','L'], 'M':'N'}), (0,0), ('Hello','Hello'), ({'A':None},{'A':None}), ] for(nested,flat)intest_cases: assert_equal(flat,flatten(nested)) deftest_flatten_custom_separator(self): nested={'A':{'B':{'C':'D'}}} assert_equal( {'A#B#C':'D'},flatten(nested,separator='#')) Let’sstartfromtest_flatten.Idefinedalistof2-tuples(nested,flat),eachofwhich representsatestcase(Ihighlightednestedtoeasereading).Ihaveonebigdictwiththree levelsofnesting,andthensomesmallerdatastructuresthatwon’tchangewhenpassedto theflattenfunction.Thesetestcasesareprobablynotenoughtocoveralledgecases,but theyshouldgiveyouagoodideaofhowyoucouldstructureatestlikethis.Withasimple forloop,Icyclethrougheachtestcaseandassertthattheresultofflatten(nested)is equaltoflat. Tip Onethingtosayaboutthisexampleisthat,whenyourunit,itwillshowyouthattwotests havebeenrun.Thisisactuallynotcorrectbecauseeveniftechnicallytherewereonlytwo testsrunning,inoneofthemwehavemultipletestcases.Itwouldbenicertohavethem runinawaythattheywererecognizedasseparate.Thisispossiblethroughtheuseof librariessuchasnose-parameterized,whichIencourageyoutocheckout.It’son https://pypi.python.org/pypi/nose-parameterized. Ialsoprovidedasecondtesttomakesurethecustomseparatorfeatureworked.Asyou cansee,Iusedonlyonedatastructure,whichismuchsmaller.Wedon’tneedtogobig again,nortotestotheredgecases.Remember,testsshouldmakesureofonethingandone thingonly,andtest_flatten_custom_separatorjusttakescareofverifyingwhetheror notwecanfeedtheflattenfunctionadifferentseparator. IcouldkeepblatheringonabouttestsforaboutanotherbookifonlyIhadthespace,but unfortunately,weneedtostophere.Ihaven’ttoldyouaboutdoctests(testswritteninthe documentationusingaPythoninteractiveshellstyle),andaboutanotherhalfamillion WOW! eBook www.wowebook.org thingsthatcouldbesaidaboutthissubject.You’llhavetodiscoverthatforyourself. Takealookatthedocumentationfortheunittestmodule,thenoseandnoseparameterizedlibraries,andpytest(http://pytest.org/),andyouwillbefine.Inmy experience,mockingandpatchingseemtobequitehardtogetagoodgraspoffor developerswhoarenewtothem,soallowyourselfalittletimetodigestthesetechniques. Tryandlearnthemgradually. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Test-drivendevelopment Let’stalkbrieflyabouttest-drivendevelopmentorTDD.Itisamethodologythatwas rediscoveredbyKentBeck,whowroteTestDrivenDevelopmentbyExample,Addison Wesley–2002,whichIencourageyoutocheckoutifyouwanttolearnaboutthe fundamentalsofthissubject,whichI’mquiteobsessedwith. TDDisasoftwaredevelopmentmethodologythatisbasedonthecontinuous repetitionofaveryshortdevelopmentcycle. Atfirst,thedeveloperwritesatest,andmakesitrun.Thetestissupposedtochecka featurethatisnotyetpartofthecode.Maybeisanewfeaturetobeadded,orsomething toberemovedoramended.Runningthetestwillmakeitfailand,becauseofthis,this phaseiscalledRed. Whenthetesthasfailed,thedeveloperwritestheminimalamountofcodetomakeitpass. Whenrunningthetestsucceeds,wehavetheso-calledGreenphase.Inthisphase,itis okaytowritecodethatcheats,justtomakethetestpass(that’swhyyouwouldthenuse triangulation).Thistechniqueiscalled,fakeit‘tilyoumakeit. Thelastpieceofthiscycleiswherethedevelopertakescareofboththecodeandthetests (inseparatetimes)andrefactorsthemuntiltheyareinthedesiredstate.Thislastphaseis calledRefactor. TheTDDmantrathereforerecites,Red-Green-Refactor. Atfirst,itfeelsreallyweirdtowritetestsbeforethecode,andImustconfessittookmea whiletogetusedtoit.Ifyousticktoit,though,andforceyourselftolearnthisslightly counter-intuitivewayofworking,atsomepointsomethingalmostmagicalhappens,and youwillseethequalityofyourcodeincreaseinawaythatwouldn’tbepossible otherwise. Whenyouwriteyourcodebeforethetests,youhavetotakecareofwhatthecodehasto doandhowithastodoit,bothatthesametime.Ontheotherhand,whenyouwritetests beforethecode,youcanconcentrateonthewhatpartalone,whileyouwritethem.When youwritethecodeafterwards,youwillmostlyhavetotakecareofhowthecodehasto implementwhatisrequiredbythetests.Thisshiftinfocusallowsyourmindto concentrateonthewhatandhowpartsinseparatemoments,yieldingabrainpowerboost thatwillsurpriseyou. Thereareseveralotherbenefitsthatcomefromtheadoptionofthistechnique: Youwillrefactorwithmuchmoreconfidence:Becausewhenyoutouchyourcode youknowthatifyouscrewthingsup,youwillbreakatleastonetest.Moreover,you willbeabletotakecareofthearchitecturaldesignintherefactorphase,where havingteststhatactasguardianswillallowyoutoenjoymassagingthecodeuntilit reachesastatethatsatisfiesyou. Thecodewillbemorereadable:Thisiscrucialinourtime,whencodingisasocial WOW! eBook www.wowebook.org activityandeveryprofessionaldeveloperspendsmuchmoretimereadingcodethan writingit. Thecodewillbemoreloose-coupledandeasiertotestandmaintain:Thisis simplybecausewritingthetestsfirstforcesyoutothinkmoredeeplyaboutits structure. Writingtestsfirstrequiresyoutohaveabetterunderstandingofthebusiness requirements:Thisisfundamentalindeliveringwhatwasactuallyaskedfor.Ifyour understandingoftherequirementsislackinginformation,you’llfindwritingatest extremelychallengingandthissituationactsasasentinelforyou. Havingeverythingunittestedmeansthecodewillbeeasiertodebug:Moreover, smalltestsareperfectforprovidingalternativedocumentation.Englishcanbe misleading,butfivelinesofPythoninasimpletestareveryhardtobe misunderstood. Higherspeed:It’sfastertowritetestsandcodethanitistowritethecodefirstand thenlosetimedebuggingit.Ifyoudon’twritetests,youwillprobablydeliverthe codesooner,butthenyouwillhavetotrackthebugsdownandsolvethem(and,rest assured,therewillbebugs).Thecombinedtimetakentowritethecodeandthen debugitisusuallylongerthanthetimetakentodevelopthecodewithTDD,where havingtestsrunningbeforethecodeiswritten,ensuringthattheamountofbugsinit willbemuchlowerthanintheothercase. Ontheotherhand,themainshortcomingsofthistechniqueare: Thewholecompanyneedstobelieveinit:Otherwiseyouwillhavetoconstantly arguewithyourboss,whowillnotunderstandwhyittakesyousolongtodeliver. Thetruthis,itmaytakeyouabitlongertodeliverintheshortterm,butinthelong termyougainalotwithTDD.However,itisquitehardtoseethelongtermbecause it’snotunderournosesliketheshorttermis.Ihavefoughtbattleswithstubborn bossesinmycareer,tobeabletocodeusingTDD.Sometimesithasbeenpainful, butalwayswellworthit,andIhaveneverregretteditbecause,intheend,thequality oftheresulthasalwaysbeenappreciated. Ifyoufailtounderstandthebusinessrequirements,thiswillreflectinthetests youwrite,andthereforeitwillreflectinthecodetoo:Thiskindofproblemis quitehardtospotuntilyoudoUAT,butonethingthatyoucandotoreducethe likelihoodofithappeningistopairwithanotherdeveloper.Pairingwillinevitably requirediscussionsaboutthebusinessrequirements,andthiswillhelphavingabetter ideaaboutthembeforethetestsarewritten. Badlywrittentestsarehardtomaintain:Thisisafact.Testswithtoomanymocks orwithextraassumptionsorbadlystructureddatawillsoonbecomeaburden.Don’t letthisdiscourageyou;justkeepexperimentingandchangethewayyouwritethem untilyoufindawaythatdoesn’trequireyouahugeamountofworkeverytimeyou touchyourcode. I’msopassionateaboutTDDthatwhenIinterviewforajob,Ialwaysaskifthecompany I’mabouttojoinadoptsit.Iftheanswerisno,it’skindofadeal-breakerforme.I encourageyoutocheckitoutanduseit.Useituntilyoufeelsomethingclickinginyour WOW! eBook www.wowebook.org mind.Youwon’tregretit,Ipromise. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Exceptions EventhoughIhaven’tformallyintroducedthemtoyou,bynowIexpectyoutoatleast haveavagueideaofwhatanexceptionis.Inthepreviouschapters,we’veseenthatwhen aniteratorisexhausted,callingnextonitraisesaStopIterationexception.We’vemet IndexErrorwhenwetriedaccessingalistatapositionthatwasoutsidethevalidrange. We’vealsometAttributeErrorwhenwetriedaccessinganattributeonanobjectthat didn’thaveit,andKeyErrorwhenwedidthesamewithakeyandadictionary.We’ve alsojustmetAssertionErrorwhenrunningtests. Now,thetimehascomeforustotalkaboutexceptions. Sometimes,eventhoughanoperationorapieceofcodeiscorrect,thereareconditionsin whichsomethingmaygowrong.Forexample,ifwe’reconvertinguserinputfromstring toint,theusercouldaccidentallytypealetterinplaceofadigit,makingitimpossiblefor ustoconvertthatvalueintoanumber.Whendividingnumbers,wemaynotknowin advanceifwe’reattemptingadivisionbyzero.Whenopeningafile,itcouldbemissing orcorrupted. Whenanerrorisdetectedduringexecution,itiscalledanexception.Exceptionsarenot necessarilylethal;infact,we’veseenthatStopIterationisdeeplyintegratedinPython generatoranditeratormechanisms.Normally,though,ifyoudon’ttakethenecessary precautions,anexceptionwillcauseyourapplicationtobreak.Sometimes,thisisthe desiredbehaviorbutinothercases,wewanttopreventandcontrolproblemssuchas these.Forexample,wemayalerttheuserthatthefilethey’retryingtoopeniscorrupted orthatitismissingsothattheycaneitherfixitorprovideanotherfile,withouttheneed fortheapplicationtodiebecauseofthisissue.Let’sseeanexampleofafewexceptions: exceptions/first.example.py >>>gen=(nforninrange(2)) >>>next(gen) 0 >>>next(gen) 1 >>>next(gen) Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> StopIteration >>>print(undefined_var) Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> NameError:name'undefined_var'isnotdefined >>>mylist=[1,2,3] >>>mylist[5] Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> IndexError:listindexoutofrange >>>mydict={'a':'A','b':'B'} >>>mydict['c'] Traceback(mostrecentcalllast): WOW! eBook www.wowebook.org File"<stdin>",line1,in<module> KeyError:'c' >>>1/0 Traceback(mostrecentcalllast): File"<stdin>",line1,in<module> ZeroDivisionError:divisionbyzero Asyoucansee,thePythonshellisquiteforgiving.WecanseetheTraceback,sothatwe haveinformationabouttheerror,buttheprogramdoesn’tdie.Thisisaspecialbehavior,a regularprogramorascriptwouldnormallydieifnothingweredonetohandleexceptions. Tohandleanexception,Pythongivesyouthetrystatement.Whathappenswhenyou enterthetryclauseisthatPythonwillwatchoutforoneormoredifferenttypesof exceptions(accordingtohowyouinstructit),andiftheyareraised,itwillallowyouto react.Thetrystatementiscomprisedofthetryclause,whichopensthestatement;oneor moreexceptclauses(alloptional)thatdefinewhattodowhenanexceptioniscaught;an elseclause(optional),whichisexecutedwhenthetryclauseisexitedwithoutany exceptionraised;andafinallyclause(optional),whosecodeisexecutedregardlessof whateverhappenedintheotherclauses.Thefinallyclauseistypicallyusedtocleanup resources.Mindtheorder,it’simportant.Also,trymustbefollowedbyatleastone exceptclauseorafinallyclause.Let’sseeanexample: exceptions/try.syntax.py deftry_syntax(numerator,denominator): try: print('Inthetryblock:{}/{}' .format(numerator,denominator)) result=numerator/denominator exceptZeroDivisionErroraszde: print(zde) else: print('Theresultis:',result) returnresult finally: print('Exiting') print(try_syntax(12,4)) print(try_syntax(11,0)) Theprecedingexampledefinesasimpletry_syntaxfunction.Weperformthedivisionof twonumbers.WearepreparedtocatchaZeroDivisionErrorexceptionifwecallthe functionwithdenominator=0.Initially,thecodeentersthetryblock.Ifdenominatoris not0,resultiscalculatedandtheexecution,afterleavingthetryblock,resumesinthe elseblock.Weprintresultandreturnit.Takealookattheoutputandyou’llnoticethat justbeforereturningresult,whichistheexitpointofthefunction,Pythonexecutesthe finallyclause. Whendenominatoris0,thingschange.Weentertheexceptblockandprintzde.The elseblockisn’texecutedbecauseanexceptionwasraisedinthetryblock.Before (implicitly)returningNone,westillexecutethefinallyblock.Takealookattheoutput andseeifitmakessensetoyou: WOW! eBook www.wowebook.org $pythonexceptions/try.syntax.py Inthetryblock:12/4 Theresultis:3.0 Exiting 3.0 Inthetryblock:11/0 divisionbyzero Exiting None Whenyouexecuteatryblock,youmaywanttocatchmorethanoneexception.For example,whentryingtodecodeaJSONobject,youmayincurintoValueErrorfor malformedJSON,orTypeErrorifthetypeofthedatayou’refeedingtojson.loads()is notastring.Inthiscase,youmaystructureyourcodelikethis: exceptions/json.example.py importjson json_data='{}' try: data=json.loads(json_data) except(ValueError,TypeError)ase: print(type(e),e) ThiscodewillcatchbothValueErrorandTypeError.Trychangingjson_data='{}'to json_data=2orjson_data='{{',andyou’llseethedifferentoutput. Note JSONstandsforJavaScriptObjectNotationandit’sanopenstandardformatthatuses human-readabletexttotransmitdataobjectsconsistingofkey/valuepairs.It’san exchangeformatwidelyusedwhenmovingdataacrossapplications,especiallywhendata needstobetreatedinalanguageorplatform-agnosticway. Ifyouwanttohandlemultipleexceptionsdifferently,youcanjustaddmoreexcept clauses,likethis: exceptions/multiple.except.py try: #somecode exceptException1: #reacttoException1 except(Exception2,Exception3): #reacttoException2andException3 exceptException3: #reacttoException3… Keepinmindthatanexceptionishandledinthefirstblockthatdefinesthatexception classoranyofitsbases.Therefore,whenyoustackmultipleexceptclauseslikewe’ve justdone,makesurethatyouputspecificexceptionsatthetopandgenericonesatthe bottom.InOOPterms,childrenontop,grandparentsatthebottom.Moreover,remember thatonlyoneexcepthandlerisexecutedwhenanexceptionisraised. Youcanalsowritecustomexceptions.Inordertodothat,youjusthavetoinheritfrom WOW! eBook www.wowebook.org anyotherexceptionclass.Pythonbuilt-inexceptionsaretoomanytobelistedhere,soI havetopointyoutowardstheofficialdocumentation.Oneimportantthingtoknowisthat everyPythonexceptionderivesfromBaseException,butyourcustomexceptionsshould neverinheritdirectlyfromthatone.Thereasonforitisthathandlingsuchanexception willtrapalsosystem-exitingexceptionssuchasSystemExitandKeyboardInterrupt, whichderivefromBaseException,andthiscouldleadtosevereissues.Incaseofdisaster, youwanttobeabletoCtrl+Cyourwayoutofanapplication. YoucaneasilysolvetheproblembyinheritingfromException,whichinheritsfrom BaseException,butdoesn’tincludeanysystem-exitingexceptioninitschildrenbecause theyaresiblingsinthebuilt-inexceptionshierarchy(see https://docs.python.org/3/library/exceptions.html#exception-hierarchy). Programmingwithexceptionscanbeverytricky.Youcouldinadvertentlysilenceout errors,ortrapexceptionsthataren’tmeanttobehandled.Playitsafebykeepinginminda fewguidelines:alwaysputinthetryclauseonlythecodethatmaycausetheexception(s) thatyouwanttohandle.Whenyouwriteexceptclauses,beasspecificasyoucan,don’t justresorttoexceptExceptionbecauseit’seasy.Useteststomakesureyourcode handlesedgecasesinawaythatrequirestheleastpossibleamountofexceptionhandling. Writinganexceptstatementwithoutspecifyinganyexceptionwouldcatchanyexception, thereforeexposingyourcodetothesamerisksyouincurwhenyouderiveyourcustom exceptionsfromBaseException. Youwillfindinformationaboutexceptionsalmosteverywhereontheweb.Somecoders usethemabundantly,otherssparingly(Ibelongtothelattercategory).Findyourownway ofdealingwiththembytakingexamplesfromotherpeople’ssourcecode.There’splenty ofinterestingprojectswhosesourcesareopen,andyoucanfindthemoneitherGitHub (https://github.com)orBitbucket(https://bitbucket.org/). Beforewetalkaboutprofiling,letmeshowyouanunconventionaluseofexceptions,just togiveyousomethingtohelpyouexpandyourviewsonthem.Theyarenotjustsimply errors. exceptions/for.loop.py n=100 found=False forainrange(n): iffound:break forbinrange(n): iffound:break forcinrange(n): if42*a+17*b+c==5096: found=True print(a,b,c)#799995 Theprecedingcodeisquiteacommonidiomifyoudealwithnumbers.Youhavetoiterate overafewnestedrangesandlookforaparticularcombinationofa,b,andcthatsatisfies acondition.Intheexample,conditionisatriviallinearequation,butimaginesomething muchcoolerthanthat.Whatbugsmeishavingtocheckifthesolutionhasbeenfoundat WOW! eBook www.wowebook.org thebeginningofeachloop,inordertobreakoutofthemasfastaswecanwhenitis.The breakoutlogicinterfereswiththerestofthecodeandIdon’tlikeit,soIcameupwitha differentsolutionforthis.Takealookatit,andseeifyoucanadaptittoothercasestoo. exceptions/for.loop.py classExitLoopException(Exception): pass try: n=100 forainrange(n): forbinrange(n): forcinrange(n): if42*a+17*b+c==5096: raiseExitLoopException(a,b,c) exceptExitLoopExceptionasele: print(ele)#(79,99,95) Canyouseehowmuchmoreelegantitis?Nowthebreakoutlogicisentirelyhandledwith asimpleexceptionwhosenameevenhintsatitspurpose.Assoonastheresultisfound, weraiseit,andimmediatelythecontrolisgiventotheexceptclausewhichhandlesit. Thisisfoodforthought.Thisexampleindirectlyshowsyouhowtoraiseyourown exceptions.Readupontheofficialdocumentationtodiveintothebeautifuldetailsofthis subject. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org ProfilingPython ThereareafewdifferentwaystoprofileaPythonapplication.Profilingmeanshavingthe applicationrunwhilekeepingtrackofseveraldifferentparameters,likethenumberof timesafunctioniscalled,theamountoftimespentinsideit,andsoon.Profilingcanhelp usfindthebottlenecksinourapplication,sothatwecanimproveonlywhatisreally slowingusdown. Ifyoutakealookattheprofilingsectioninthestandardlibraryofficialdocumentation, youwillseethatthereareacoupleofdifferentimplementationsofthesameprofiling interface:profileandcProfile. cProfileisrecommendedformostusers,it’saCextensionwithreasonable overheadthatmakesitsuitableforprofilinglong-runningprograms profileisapurePythonmodulewhoseinterfaceisimitatedbycProfile,butwhich addssignificantoverheadtoprofiledprograms Thisinterfacedoesdeterministprofiling,whichmeansthatallfunctioncalls,function returnsandexceptioneventsaremonitored,andprecisetimingsaremadefortheintervals betweentheseevents.Anotherapproach,calledstatisticalprofiling,randomlysamples theeffectiveinstructionpointer,anddeduceswheretimeisbeingspent. Thelatterusuallyinvolveslessoverhead,butprovidesonlyapproximateresults. Moreover,becauseofthewaythePythoninterpreterrunsthecode,deterministicprofiling doesn’taddthatasmuchoverheadasonewouldthink,soI’llshowyouasimpleexample usingcProfilefromthecommandline. We’regoingtocalculatePythagoreantriples(Iknow,you’vemissedthem…)usingthe followingcode: profiling/triples.py defcalc_triples(mx): triples=[] forainrange(1,mx+1): forbinrange(a,mx+1): hypotenuse=calc_hypotenuse(a,b) ifis_int(hypotenuse): triples.append((a,b,int(hypotenuse))) returntriples defcalc_hypotenuse(a,b): return(a**2+b**2)**.5 defis_int(n):#nisexpectedtobeafloat returnn.is_integer() triples=calc_triples(1000) Thescriptisextremelysimple;weiterateovertheinterval[1,mx]withaandb(avoiding repetitionofpairsbysettingb>=a)andwecheckiftheybelongtoarighttriangle.We usecalc_hypotenusetogethypotenuseforaandb,andthen,withis_int,wecheckifit WOW! eBook www.wowebook.org isaninteger,whichmeans(a,b,c)isaPythagoreantriple.Whenweprofilethisscript,we getinformationintabularform.Thecolumnsarencalls,tottime,percall,cumtime, percall,andfilename:lineno(function).Theyrepresenttheamountofcallswemade toafunction,howmuchtimewespentinit,andsoon.I’lltrimacoupleofcolumnsto savespace,soifyouruntheprofilingyourself,don’tworryifyougetadifferentresult. $python-mcProfileprofiling/triples.py 1502538functioncallsin0.750seconds Orderedby:standardname ncallstottimepercallfilename:lineno(function) 5005000.4690.000triples.py:14(calc_hypotenuse) 5005000.0870.000triples.py:18(is_int) 10.0000.000triples.py:4(<module>) 10.1630.163triples.py:4(calc_triples) 10.0000.000{built-inmethodexec} 10340.0000.000{method'append'of'list'objects} 10.0000.000{method'disable'of'_lsprof.Profil… 5005000.0320.000{method'is_integer'of'float'objects} Evenwiththislimitedamountofdata,wecanstillinfersomeusefulinformationabout thiscode.Firstly,wecanseethatthetimecomplexityofthealgorithmwehavechosen growswiththesquareoftheinputsize.Theamountoftimeswegetinsidetheinnerloop bodyisexactlymx(mx+1)/2.Werunthescriptwithmx=1000,whichmeansweget 500500timesinsidetheinnerforloop.Threemainthingshappeninsidethatloop,wecall calc_hypotenuse,wecallis_intand,iftheconditionismet,weappendtothetriples list. Takingalookattheprofilingreport,wenoticethatthealgorithmhasspent0.469seconds insidecalc_hypotenuse,whichiswaymorethanthe0.087secondsspentinsideis_int, giventhattheywerecalledthesamenumberoftimes,solet’sseeifwecanboost calc_hypotenusealittle. Asitturnsout,wecan.AsImentionedearlieroninthebook,thepoweroperator**is quiteexpensive,andincalc_hypotenuse,we’reusingitthreetimes.Fortunately,wecan easilytransformtwoofthoseintosimplemultiplications,likethis: profiling/triples.py defcalc_hypotenuse(a,b): return(a*a+b*b)**.5 Thissimplechangeshouldimprovethings.Ifweruntheprofilingagain,weseethatnow the0.469isnowdownto0.177.Notbad!Thismeansnowwe’respendingonlyabout 37%ofthetimeinsidecalc_hypotenuseaswewerebefore. Let’sseeifwecanimproveis_intaswell,bychangingitlikethis: profiling/triples.py defis_int(n): returnn==int(n) Thisimplementationisdifferentandtheadvantageisthatitalsoworkswhennisan integer.Alas,whenweruntheprofilingagainstit,weseethatthetimetakeninsidethe WOW! eBook www.wowebook.org is_intfunctionhasgoneupto0.141seconds.Thismeansthatithasroughlydoubled, comparedtowhatitwasbefore.Inthiscase,weneedtoreverttotheprevious implementation. Thisexamplewastrivial,ofcourse,butenoughtoshowyouhowonecouldprofilean application.Havingtheamountofcallsthatareperformedagainstafunctionhelpsus understandbetterthetimecomplexityofouralgorithms.Forexample,youwouldn’t believehowmanycodersfailtoseethatthosetwoforloopsrunproportionallytothe squareoftheinputsize. Onethingtomention:dependingonwhatsystemyou’reusing,resultsmaybedifferent. Therefore,it’squiteimportanttobeabletoprofilesoftwareonasystemthatisascloseas possibletotheonethesoftwareisdeployedon,ifnotactuallyonthatone. WOW! eBook www.wowebook.org Whentoprofile? Profilingissupercool,butweneedtoknowwhenitisappropriatetodoit,andinwhat measureweneedtoaddresstheresultswegetfromit. DonaldKnuthoncesaidthatprematureoptimizationistherootofalleviland,althoughI wouldn’thaveputitdownsodrastically,Idoagreewithhim.Afterall,whoamIto disagreewiththemanthatgaveusTheArtofComputerProgramming,TeX,andsomeof thecoolestalgorithmsIhaveeverstudiedwhenIwasauniversitystudent? So,firstandforemost:correctness.Youwantyoucodetodelivertheresultcorrectly, thereforewritetests,findedgecases,andstressyourcodeineverywayyouthinkmakes sense.Don’tbeprotective,don’tputthingsinthebackofyourbrainforlaterbecauseyou thinkthey’renotlikelytohappen.Bethorough. Secondly,takecareofcodingbestpractices.Rememberreadability,extensibility,loose coupling,modularity,anddesign.ApplyOOPprinciples:encapsulation,abstraction,single responsibility,open/closed,andsoon.Readupontheseconcepts.Theywillopenhorizons foryou,andtheywillexpandthewayyouthinkaboutcode. Thirdly,refactorlikeabeast!TheBoyScoutsRulesaystoAlwaysleavethecampground cleanerthanyoufoundit.Applythisruletoyourcode. And,finally,whenalloftheabovehasbeentakencareof,thenandonlythen,youtake careofprofiling. Runyourprofilerandidentifybottlenecks.Whenyouhaveanideaofthebottlenecksyou needtoaddress,startwiththeworstonefirst.Sometimes,fixingabottleneckcausesa rippleeffectthatwillexpandandchangethewaytherestofthecodeworks.Sometimes thisisonlyalittle,sometimesabitmore,accordingtohowyourcodewasdesignedand implemented.Therefore,startwiththebiggestissuefirst. OneofthereasonsPythonissopopularisthatitispossibletoimplementitinmany differentways.So,ifyoufindyourselfhavingtroublesboostingupsomepartofyour codeusingsheerPython,nothingpreventsyoufromrollingupyoursleeves,buyinga coupleofhundredlitersofcoffee,andrewritingtheslowpieceofcodeinC.Guaranteed tobefun! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,weexploredtheworldoftesting,exceptions,andprofiling. Itriedtogiveyouafairlycomprehensiveoverviewoftesting,especiallyunittesting, whichisthekindoftestingthatadevelopermostlydoes.IhopeIhavesucceededin channelingthemessagethattestingisnotsomethingthatisperfectlydefinedandthatyou canlearnfromabook.Youneedtoexperimentwithitalotbeforeyougetcomfortable.Of alltheeffortsacodermustmakeintermsofstudyandexperimentation,I’dsaytestingis oneofthosethataremostworthit. We’vebrieflyseenhowwecanpreventourprogramfromdyingbecauseoferrors,called exceptions,thathappenatruntime.And,tosteerawayfromtheusualground,Ihavegiven youanexampleofasomewhatunconventionaluseofexceptionstobreakoutofnested forloops.That’snottheonlycase,andI’msureyou’lldiscoverothersasyougrowasa coder. Intheend,weverybrieflytouchedbaseonprofiling,withasimpleexampleandafew guidelines.Iwantedtotalkaboutprofilingforthesakeofcompleteness,soatleastyou canplayaroundwithit. We’renowabouttoenterChapter8,TheEdges–GUIsandScripts,wherewe’regoingto getourhandsdirtywithscriptsandGUIsand,hopefully,comeupwithsomething interesting. Note IamawarethatIgaveyoualotofpointersinthischapter,withnolinksordirections.I’m afraidthisisbychoice.Asacoder,therewon’tbeasingledayatworkwhenyouwon’t havetolooksomethingupinadocumentationpage,inamanual,onawebsite,andsoon. Ithinkit’svitalforacodertobeabletosearcheffectivelyfortheinformationtheyneed, soIhopeyou’llforgivemeforthisextratraining.Afterall,it’sallforyourbenefit. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter8.TheEdges–GUIsandScripts “Auserinterfaceislikeajoke.Ifyouhavetoexplainit,it’snotthatgood.” —MartinLeBlanc Inthischapter,we’regoingtoworkonaprojecttogether.We’regoingtoprepareavery simpleHTMLpagewithafewimages,andthenwe’regoingtoscrapeit,inordertosave thoseimages. We’regoingtowriteascripttodothis,whichwillallowustotalkaboutafewconcepts thatI’dliketorunbyyou.We’realsogoingtoaddafewoptionstosaveimagesbasedon theirformat,andtochoosethewaywesavethem.And,whenwe’redonewiththescript, we’regoingtowriteaGUIapplicationthatdoesbasicallythesamething,thuskillingtwo birdswithonestone.Havingonlyoneprojecttoexplainwillallowmetoshowawider rangeoftopicsinthischapter. Note Agraphicaluserinterface(GUI)isatypeofinterfacethatallowstheusertointeract withanelectronicdevicethroughgraphicalicons,buttonsandwidgets,asopposedtotextbasedorcommand-lineinterfaces,whichrequirecommandsortexttobetypedonthe keyboard.Inanutshell,anybrowser,anyofficesuitesuchasLibreOffice,and,ingeneral, anythingthatpopsupwhenyouclickonanicon,isaGUIapplication. So,ifyouhaven’talreadydoneso,thiswouldbetheperfecttimetostartaconsoleand positionyourselfinafoldercalledch8intherootofyourprojectforthisbook.Withinthat folder,we’llcreatetwoPythonmodules(scrape.pyandguiscrape.py)andonestandard folder(simple_server).Withinsimple_server,we’llwriteourHTMLpage (index.html)insimple_server.Imageswillbestoredinch8/simple_server/img.The structureinch8shouldlooklikethis: $tree-A . ├──guiscrape.py ├──scrape.py └──simple_server ├──img │├──owl-alcohol.png │├──owl-book.png │├──owl-books.png │├──owl-ebook.jpg │└──owl-rose.jpeg ├──index.html └──serve.sh Ifyou’reusingeitherLinuxorMac,youcandowhatIdoandputthecodetostartthe HTTPserverinaserve.shfile.OnWindows,you’llprobablywanttouseabatchfile. TheHTMLpagewe’regoingtoscrapehasthefollowingstructure: simple_server/index.html WOW! eBook www.wowebook.org <!DOCTYPEhtml> <htmllang="en"> <head><title>CoolOwls!</title></head> <body> <h1>Welcometomyowlgallery</h1> <div> <imgsrc="img/owl-alcohol.png"height="128"/> <imgsrc="img/owl-book.png"height="128"/> <imgsrc="img/owl-books.png"height="128"/> <imgsrc="img/owl-ebook.jpg"height="128"/> <imgsrc="img/owl-rose.jpeg"height="128"/> </div> <p>Doyoulikemyowls?</p> </body> </html> It’sanextremelysimplepage,solet’sjustnotethatwehavefiveimages,threeofwhich arePNGsandtwoareJPGs(notethateventhoughtheyarebothJPGs,oneendswith.jpg andtheotherwith.jpeg,whicharebothvalidextensionsforthisformat). So,PythongivesyouaverysimpleHTTPserverforfreethatyoucanstartwiththe followingcommand(inthesimple_serverfolder): $python-mhttp.server8000 ServingHTTPon0.0.0.0port8000… 127.0.0.1--[31/Aug/201516:11:10]"GET/HTTP/1.1"200- Thelastlineisthelogyougetwhenyouaccesshttp://localhost:8000,whereour beautifulpagewillbeserved.Alternatively,youcanputthatcommandinafilecalled serve.sh,andjustrunthatwiththiscommand(makesureit’sexecutable): $./serve.sh Itwillhavethesameeffect.Ifyouhavethecodeforthisbook,yourpageshouldlook somethinglikethis: WOW! eBook www.wowebook.org Feelfreetouseanyothersetofimages,aslongasyouuseatleastonePNGandoneJPG, andthatinthesrctagyouuserelativepaths,notabsolute.Igotthoselovelyowlsfrom https://openclipart.org/. WOW! eBook www.wowebook.org Firstapproach–scripting Now,let’sstartwritingthescript.I’llgothroughthesourceinthreesteps:importsfirst, thentheargumentparsinglogic,andfinallythebusinesslogic. WOW! eBook www.wowebook.org Theimports scrape.py(Imports) importargparse importbase64 importjson importos frombs4importBeautifulSoup importrequests Goingthroughthemfromthetop,youcanseethatwe’llneedtoparsethearguments. whichwe’llfeedtothescriptitself(argparse).Wewillneedthebase64librarytosave theimageswithinaJSONfile(base64andjson),andwe’llneedtoopenfilesforwriting (os).Finally,we’llneedBeautifulSoupforscrapingthewebpageeasily,andrequeststo fetchitscontent.requestsisanextremelypopularlibraryforperformingHTTPrequests, builttoavoidthedifficultiesandquirksofusingthestandardlibraryurllibmodule.It’s basedonthefasturllib3third-partylibrary. Note WewillexploretheHTTPprotocolandrequestsmechanisminChapter10,Web DevelopmentDoneRightso,fornow,let’sjust(simplistically)saythatweperforman HTTPrequesttofetchthecontentofawebpage.Wecandoitprogrammaticallyusinga librarysuchasrequests,andit’smoreorlesstheequivalentoftypingaURLinyour browserandpressingEnter(thebrowserthenfetchesthecontentofawebpageandalso displaysittoyou). Ofalltheseimports,onlythelasttwodon’tbelongtothePythonstandardlibrary,butthey aresowidelyusedthroughouttheworldthatIdarenotexcludetheminthisbook.Make sureyouhavetheminstalled: $pipfreeze|egrep-i"soup|requests" beautifulsoup4==4.4.0 requests==2.7.0 Ofcourse,theversionnumbersmightbedifferentforyou.Ifthey’renotinstalled,usethis commandtodoso: $pipinstallbeautifulsoup4requests Atthispoint,theonlythingthatIreckonmightconfuseyouisthebase64/jsoncouple,so allowmetospendafewwordsonthat. Aswesawinthepreviouschapter,JSONisoneofthemostpopularformatsfordata exchangebetweenapplications.It’salsowidelyusedforotherpurposestoo,forexample, tosavedatainafile.Inourscript,we’regoingtooffertheusertheabilitytosaveimages asimagefiles,orasaJSONsinglefile.WithintheJSON,we’llputadictionarywithkeys astheimagesnamesandvaluesastheircontent.Theonlyissueisthatsavingimagesin thebinaryformatistricky,andthisiswherethebase64librarycomestotherescue. Base64isaverypopularbinary-to-textencodingschemethatrepresentsbinarydatainan WOW! eBook www.wowebook.org ASCIIstringformatbytranslatingitintoaradix-64representation. Note Theradix-64representationusesthelettersA-Z,a-z,andthedigits0-9,plusthetwo symbols+and/foragrandtotalof64symbolsaltogether.Therefore,notsurprisingly,the Base64alphabetismadeupofthese64symbols. Ifyouthinkyouhaveneverusedit,thinkagain.Everytimeyousendanemailwithan imageattachedtoit,theimagegetsencodedwithBase64beforetheemailissent.Onthe recipientside,imagesareautomaticallydecodedintotheiroriginalbinaryformatsothat theemailclientcandisplaythem. WOW! eBook www.wowebook.org Parsingarguments Nowthatthetechnicalitiesareoutoftheway,let’sseethesecondsectionofourscript(it shouldbeattheendofthescrape.pymodule). scrape.py(Argumentparsingandscrapertriggering) if__name__=="__main__": parser=argparse.ArgumentParser( description='Scrapeawebpage.') parser.add_argument( '-t', '--type', choices=['all','png','jpg'], default='all', help='Theimagetypewewanttoscrape.') parser.add_argument( '-f', '--format', choices=['img','json'], default='img', help='Theformatimagesaresavedto.') parser.add_argument( 'url', help='TheURLwewanttoscrapeforimages.') args=parser.parse_args() scrape(args.url,args.format,args.type) Lookatthatfirstline;itisaverycommonidiomwhenitcomestoscripting.Accordingto theofficialPythondocumentation,thestring'__main__'isthenameofthescopein whichtop-levelcodeexecutes.Amodule’s__name__issetequalto'__main__'when readfromstandardinput,ascript,orfromaninteractiveprompt. Therefore,ifyouputtheexecutionlogicunderthatif,theresultisthatyouwillbeableto usethemoduleasalibraryshouldyouneedtoimportanyofthefunctionsorobjects definedinit,becausewhenimportingitfromanothermodule,__name__won’tbe '__main__'.Ontheotherhand,whenyourunthescriptdirectly,likewe’regoingto, __name__willbe'__main__',sotheexecutionlogicwillrun. Thefirstthingwedothenisdefineourparser.Iwouldrecommendusingthestandard librarymodule,argparse,whichissimpleenoughandquitepowerful.Thereareother optionsoutthere,butinthiscase,argparsewillprovideuswithallweneed. Wewanttofeedourscriptthreedifferentdata:thetypeofimageswewanttosave,the formatinwhichwewanttosavethem,andtheURLforthepagetobescraped. ThetypecanbePNG,JPGorboth(default),whiletheformatcanbeeitherimageor JSON,imagebeingthedefault.URListheonlymandatoryargument. So,weaddthe-toption,allowingalsothelongversion--type.Thechoicesare'all', 'png',and'jpg'.Wesetthedefaultto'all'andweaddahelpmessage. Wedoasimilarprocedurefortheformatargumentallowingboththeshortandlong WOW! eBook www.wowebook.org syntax(-fand--format),andfinallyweaddtheurlargument,whichistheonlyonethat isspecifieddifferentlysothatitwon’tbetreatedasanoption,butratherasapositional argument. Inordertoparseallthearguments,allweneedisparser.parse_args().Verysimple, isn’tit? Thelastlineiswherewetriggertheactuallogic,bycallingthescrapefunction,passing alltheargumentswejustparsed.Wewillseeitsdefinitionshortly. Thenicethingaboutargparseisthatifyoucallthescriptbypassing-h,itwillprinta niceusagetextforyouautomatically.Let’stryitout: $pythonscrape.py-h usage:scrape.py[-h][-t{all,png,jpg}][-f{img,json}]url Scrapeawebpage. positionalarguments: urlTheURLwewanttoscrapeforimages. optionalarguments: -h,--helpshowthishelpmessageandexit -t{all,png,jpg},--type{all,png,jpg} Theimagetypewewanttoscrape. -f{img,json},--format{img,json} Theformatimagesaresavedto. Ifyouthinkaboutit,theonetrueadvantageofthisisthatwejustneedtospecifythe argumentsandwedon’thavetoworryabouttheusagetext,whichmeanswewon’thave tokeepitinsyncwiththearguments’definitioneverytimewechangesomething.Thisis precious. Here’safewdifferentwaystocallourscrape.pyscript,whichdemonstratethattypeand formatareoptional,andhowyoucanusetheshortandlongsyntaxtousethem: $pythonscrape.pyhttp://localhost:8000 $pythonscrape.py-tpnghttp://localhost:8000 $pythonscrape.py--type=jpg-fjsonhttp://localhost:8000 Thefirstoneisusingdefaultvaluesfortypeandformat.Thesecondonewillsaveonly PNGimages,andthethirdonewillsaveonlyJPGs,butinJSONformat. WOW! eBook www.wowebook.org Thebusinesslogic Nowthatwe’veseenthescaffolding,let’sdivedeepintotheactuallogic(ifitlooks intimidatingdon’tworry;we’llgothroughittogether).Withinthescript,thislogiclies aftertheimportsandbeforetheparsing(beforetheif__name__clause): scrape.py(Businesslogic) defscrape(url,format_,type_): try: page=requests.get(url) exceptrequests.RequestExceptionasrex: print(str(rex)) else: soup=BeautifulSoup(page.content,'html.parser') images=_fetch_images(soup,url) images=_filter_images(images,type_) _save(images,format_) def_fetch_images(soup,base_url): images=[] forimginsoup.findAll('img'): src=img.get('src') img_url=( '{base_url}/{src}'.format( base_url=base_url,src=src)) name=img_url.split('/')[-1] images.append(dict(name=name,url=img_url)) returnimages def_filter_images(images,type_): iftype_=='all': returnimages ext_map={ 'png':['.png'], 'jpg':['.jpg','.jpeg'], } return[ imgforimginimages if_matches_extension(img['name'],ext_map[type_]) ] def_matches_extension(filename,extension_list): name,extension=os.path.splitext(filename.lower()) returnextensioninextension_list def_save(images,format_): ifimages: ifformat_=='img': _save_images(images) else: _save_json(images) print('Done') else: print('Noimagestosave.') WOW! eBook www.wowebook.org def_save_images(images): forimginimages: img_data=requests.get(img['url']).content withopen(img['name'],'wb')asf: f.write(img_data) def_save_json(images): data={} forimginimages: img_data=requests.get(img['url']).content b64_img_data=base64.b64encode(img_data) str_img_data=b64_img_data.decode('utf-8') data[img['name']]=str_img_data withopen('images.json','w')asijson: ijson.write(json.dumps(data)) Let’sstartwiththescrapefunction.Thefirstthingitdoesisfetchthepageatthegiven urlargument.Whatevererrormayhappenwhiledoingthis,wetrapitinthe RequestExceptionrexandweprintit.TheRequestExceptionisthebaseexception classforalltheexceptionsintherequestslibrary. However,ifthingsgowell,andwehaveapagebackfromtheGETrequest,thenwecan proceed(elsebranch)andfeeditscontenttotheBeautifulSoupparser.The BeautifulSouplibraryallowsustoparseawebpageinnotime,withouthavingtowrite allthelogicthatwouldbeneededtofindalltheimagesinapage,whichwereallydon’t wanttodo.It’snotaseasyasitseems,andreinventingthewheelisnevergood.Tofetch images,weusethe_fetch_imagesfunctionandwefilterthemwith_filter_images. Finally,wecall_savewiththeresult. Splittingthecodeintodifferentfunctionswithmeaningfulnamesallowsustoreaditmore easily.Evenifyouhaven’tseenthelogicofthe_fetch_images,_filter_images,and _savefunctions,it’snothardtopredictwhattheydo,right? _fetch_imagestakesaBeautifulSoupobjectandabaseURL.Allitdoesislooping throughalloftheimagesfoundonthepageandfillinginthe'name'and'url' informationabouttheminadictionary(oneperimage).Alldictionariesareaddedtothe imageslist,whichisreturnedattheend. Thereissometrickerygoingonwhenwegetthenameofanimage.Whatwedoissplit theimg_url(http://localhost:8000/img/my_image_name.png)stringusing'/'asa separator,andwetakethelastitemastheimagename.Thereisamorerobustwayof doingthis,butforthisexampleitwouldbeoverkill.Ifyouwanttoseethedetailsofeach step,trytobreakthislogicdownintosmallersteps,andprinttheresultofeachofthemto helpyourselfunderstand. Towardstheendofthebook,I’llshowyouanothertechniquetodebuginamuchmore efficientway. Anyway,byjustaddingprint(images)attheendofthe_fetch_imagesfunction,weget this: WOW! eBook www.wowebook.org [{'url':'http://localhost:8000/img/owl-alcohol.png','name':'owlalcohol.png'},{'url':'http://localhost:8000/img/owl-book.png','name': 'owl-book.png'},...] Itruncatedtheresultforbrevity.Youcanseeeachdictionaryhasa'url'and'name' key/valuepair,whichwecanusetofetch,identifyandsaveourimagesaswelike.Atthis point,Ihearyouaskingwhatwouldhappeniftheimagesonthepagewerespecifiedwith anabsolutepathinsteadofarelativeone,right?Goodquestion! Theansweristhatthescriptwillfailtodownloadthembecausethislogicexpectsrelative paths.IwasabouttoaddabitoflogictosolvethisissuewhenIthoughtthat,atthisstage, itwouldbeaniceexerciseforyoutodoit,soI’llleaveituptoyoutofixit. Tip Hint:inspectthestartofthatsrcvariable.Ifitstartswith'http',thenit’sprobablyan absolutepath. Ihopethebodyofthe_filter_imagesfunctionisinterestingtoyou.Iwantedtoshow youhowtocheckonmultipleextensionsbyusingamappingtechnique. Inthisfunction,iftype_is'all',thennofilteringisrequired,sowejustreturnallthe images.Ontheotherhand,whentype_isnot'all',wegettheallowedextensionsfrom theext_mapdictionary,anduseittofiltertheimagesinthelistcomprehensionthatends thefunctionbody.Youcanseethatbyusinganotherhelperfunction, _matches_extension,Ihavemadethelistcomprehensionsimplerandmorereadable. All_matches_extensiondoesissplitthenameoftheimagegettingitsextensionand checkingwhetheritiswithinthelistofallowedones.Canyoufindonemicro improvement(speed-wise)thatcouldbedonetothisfunction? I’msurethatyou’rewonderingwhyIhavecollectedalltheimagesinthelistandthen removedthem,insteadofcheckingwhetherIwantedtosavethembeforeaddingthemto thelist.ThefirstreasonisthatIneeded_fetch_imagesintheGUIappasitisnow.A secondreasonisthatcombining,fetching,andfilteringwouldproducealongerandabit morecomplicatedfunction,andI’mtryingtokeepthecomplexityleveldown.Athird reasonisthatthiscouldbeaniceexerciseforyoutodo.Feelslikewe’repairinghere… Let’skeepgoingthroughthecodeandinspectthe_savefunction.Youcanseethat,when imagesisn’tempty,thisbasicallyactsasadispatcher.Weeithercall_save_imagesor _save_json,dependingonwhichinformationisstoredintheformat_variable. Wearealmostdone.Let’sjumpto_save_images.Weloopontheimageslistandforeach dictionarywefindthereweperformaGETrequestontheimageURLandsaveitscontent inafile,whichwenameastheimageitself.Theoneimportantthingtonotehereishow wesavethatfile. Weuseacontextmanager,representedbythekeywordwith,todothat.Python’swith statementsupportstheconceptofaruntimecontextdefinedbyacontextmanager.Thisis implementedusingapairofmethods(contextmanager.__enter__()and contextmanager.__exit__(exc_type,exc_val,exc_tb))thatallowuser-defined WOW! eBook www.wowebook.org classestodefinearuntimecontextthatisenteredbeforethestatementbodyisexecuted andexitedwhenthestatementends. Inourcase,usingacontextmanager,inconjunctionwiththeopenfunction,givesusthe guaranteethatifanythingbadweretohappenwhilewritingthatfile,theresources involvedintheprocesswillbecleanedupandreleasedproperlyregardlessoftheerror. HaveyouevertriedtodeleteafileonWindows,onlytobepresentedwithanalertthat tellsyouthatyoucannotdeletethefilebecausethereisanotherprocessthatisholdingon toit?We’reavoidingthatsortofveryannoyingthing. Whenweopenafile,wegetahandlerforitand,nomatterwhathappens,wewanttobe surewereleaseitwhenwe’redonewiththefile.Acontextmanageristhetoolweneedto makesureofthat. Finally,let’snowstepintothe_save_jsonfunction.It’sverysimilartothepreviousone. Webasicallyfillinthedatadictionary.Theimagenameisthekey,andtheBase64 representationofitsbinarycontentisthevalue.Whenwe’redonepopulatingour dictionary,weusethejsonlibrarytodumpitintheimages.jsonfile.I’llgiveyouasmall previewofthat: images.json(truncated) { "owl-ebook.jpg":"/9j/4AAQSkZJRgABAQEAMQAxAAD/2wBDAAEBAQ… "owl-book.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEbCAYAAAB… "owl-books.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAElCAYAAA… "owl-alcohol.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEICAYA… "owl-rose.jpeg":"/9j/4AAQSkZJRgABAQEANAA0AAD/2wBDAAEBAQ… } Andthat’sit!Now,beforeproceedingtothenextsection,makesureyouplaywiththis scriptandunderstandwellhowitworks.Tryandmodifysomething,printoutintermediate results,addanewargumentorfunctionality,orscramblethelogic.We’regoingtomigrate itintoaGUIapplicationnow,whichwilladdalayerofcomplexitysimplybecausewe’ll havetobuildtheGUIinterface,soit’simportantthatyou’rewellacquaintedwiththe businesslogic:itwillallowyoutoconcentrateontherestofthecode. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Secondapproach–aGUIapplication ThereareseverallibrariestowriteGUIapplicationsinPython.Themostfamousonesare tkinter,wxPython,PyGTK,andPyQt.Theyallofferawiderangeoftoolsandwidgets thatyoucanusetocomposeaGUIapplication. TheoneI’mgoingtousefortherestofthischapteristkinter.tkinterstandsforTk interfaceanditisthestandardPythoninterfacetotheTkGUItoolkit.BothTkandtkinter areavailableonmostUnixplatforms,MacOSX,aswellasonWindowssystems. Let’smakesurethattkinterisinstalledproperlyonyoursystembyrunningthis command: $python-mtkinter ItshouldopenadialogwindowdemonstratingasimpleTkinterface.Ifyoucanseethat, thenwe’regoodtogo.However,ifitdoesn’twork,pleasesearchfortkinterinthe Pythonofficialdocumentation.Youwillfindseverallinkstoresourcesthatwillhelpyou getupandrunningwithit. We’regoingtomakeaverysimpleGUIapplicationthatbasicallymimicsthebehaviorof thescriptwesawinthefirstpartofthischapter.Wewon’taddtheabilitytosaveJPGsor PNGssingularly,butafteryou’vegonethroughthischapter,youshouldbeabletoplay withthecodeandputthatfeaturebackinbyyourself. So,thisiswhatwe’reaimingfor: Gorgeous,isn’tit?Asyoucansee,it’saverysimpleinterface(thisishowitshouldlook onUbuntu).Thereisaframe(thatis,acontainer)fortheURLfieldandtheFetchinfo button,anotherframefortheListboxtoholdtheimagenamesandtheradiobuttonto controlthewaywesavethem,andfinallythereisaScrape!buttonatthebottom.Wealso haveastatusbar,whichshowsussomeinformation. Inordertogetthislayout,wecouldjustplaceallthewidgetsonarootwindow,butthat WOW! eBook www.wowebook.org wouldmakethelayoutlogicquitemessyandunnecessarilycomplicated.So,instead,we willdividethespaceusingframesandplacethewidgetsinthoseframes.Thiswaywewill achieveamuchnicerresult.So,thisisthedraftforthelayout: WehaveaRootWindow,whichisthemainwindowoftheapplication.Wedivideitinto tworows,thefirstoneinwhichweplacetheMainFrame,andthesecondoneinwhich weplacetheStatusFrame(whichwillholdthestatusbar).TheMainFrameis subsequentlydividedintothreerowsitself.InthefirstoneweplacetheURLFrame, whichholdstheURLwidgets.InthesecondoneweplacetheImgFrame,whichwill holdtheListboxandtheRadioFrame,whichwillhostalabelandtheradiobutton widgets.Andfinallyathirdone,whichwilljustholdtheScrapebutton. Inordertolayoutframesandwidgets,wewillusealayoutmanagercalledgrid,that simplydividesupthespaceintorowsandcolumns,asinamatrix. Now,allthecodeI’mgoingtowritecomesfromtheguiscrape.pymodule,soIwon’t repeatitsnameforeachsnippet,tosavespace.Themoduleislogicallydividedintothree sections,notunlikethescriptversion:imports,layoutlogic,andbusinesslogic.We’re goingtoanalyzethemlinebyline,inthreechunks. WOW! eBook www.wowebook.org Theimports fromtkinterimport* fromtkinterimportttk,filedialog,messagebox importbase64 importjson importos frombs4importBeautifulSoup importrequests We’realreadyfamiliarwithmostofthese.Theinterestingbithereisthosefirsttwolines. Thefirstoneisquitecommonpractice,althoughitisbadpracticeinPythontoimport usingthestarsyntax.Youcanincurinnamecollisionsand,ifthemoduleistoobig, importingeverythingwouldbeexpensive. Afterthat,weimportttk,filedialog,andmessageboxexplicitly,followingthe conventionalapproachusedwiththislibrary.ttkisthenewsetofstyledwidgets.They behavebasicallyliketheoldones,butarecapableofdrawingthemselvescorrectly accordingtothestyleyourOSisseton,whichisnice. Therestoftheimportsiswhatweneedinordertocarryoutthetaskyouknowwellby now.Notethatthereisnothingweneedtoinstallwithpipinthissecondpart,wealready haveeverythingweneed. WOW! eBook www.wowebook.org Thelayoutlogic I’mgoingtopasteitchunkbychunksothatIcanexplainiteasilytoyou.You’llseehow allthosepieceswetalkedaboutinthelayoutdraftarearrangedandgluedtogether.What I’mabouttopaste,aswedidinthescriptbefore,isthefinalpartoftheguiscrape.py module.We’llleavethemiddlepart,thebusinesslogic,forlast. if__name__=="__main__": _root=Tk() _root.title('Scrapeapp') Asyouknowbynow,weonlywanttoexecutethelogicwhenthemoduleisrundirectly, sothatfirstlineshouldn’tsurpriseyou. Inthelasttwolines.wesetupthemainwindow,whichisaninstanceoftheTkclass.We instantiateitandgiveitatitle.NotethatIusetheprependingunderscoretechniqueforall thenamesofthetkinterobjects,inordertoavoidpotentialcollisionswithnamesinthe businesslogic.Ijustfinditcleanerlikethis,butyou’reallowedtodisagree. _mainframe=ttk.Frame(_root,padding='5555') _mainframe.grid(row=0,column=0,sticky=(E,W,N,S)) Here,wesetuptheMainFrame.It’sattk.Frameinstance.Weset_rootasitsparent, andgiveitsomepadding.Thepaddingisameasureinpixelsofhowmuchspaceshould beinsertedbetweentheinnercontentandthebordersinordertoletourlayoutbreathea little,otherwisewehavethesardineeffect,wherewidgetsarepackedtootightly. Thesecondlineismuchmoreinteresting.Weplacethis_mainframeonthefirstrow(0) andfirstcolumn(0)oftheparentobject(_root).Wealsosaythatthisframeneedsto extenditselfineachdirectionbyusingthestickyargumentwithallfourcardinal directions.Ifyou’rewonderingwheretheycamefrom,it’sthefromtkinterimport* magicthatbroughtthemtous. _url_frame=ttk.LabelFrame( _mainframe,text='URL',padding='5555') _url_frame.grid(row=0,column=0,sticky=(E,W)) _url_frame.columnconfigure(0,weight=1) _url_frame.rowconfigure(0,weight=1) Next,westartbyplacingtheURLFramedown.Thistime,theparentobjectis _mainframe,asyouwillrecallfromourdraft.ThisisnotjustasimpleFrame,butit’s actuallyaLabelFrame,whichmeanswecansetthetextargumentandexpectarectangle tobedrawnaroundit,withthecontentofthetextargumentwritteninthetop-leftpartofit (checkoutthepreviouspictureifithelps).Wepositionthisframeat(0,0),andsaythatit shouldexpandtotheleftandtotheright.Wedon’tneedtheothertwodirections. Finally,weuserowconfigureandcolumnconfiguretomakesureitbehavescorrectly, shoulditneedtoresize.Thisisjustaformalityinourpresentlayout. _url=StringVar() _url.set('http://localhost:8000') _url_entry=ttk.Entry( WOW! eBook www.wowebook.org _url_frame,width=40,textvariable=_url) _url_entry.grid(row=0,column=0,sticky=(E,W,S,N),padx=5) _fetch_btn=ttk.Button( _url_frame,text='Fetchinfo',command=fetch_url) _fetch_btn.grid(row=0,column=1,sticky=W,padx=5) Here,wehavethecodetolayouttheURLtextboxandthe_fetchbutton.Atextboxin thisenvironmentiscalledEntry.Weinstantiateitasusual,setting_url_frameasits parentandgivingitawidth.Also,andthisisthemostinterestingpart,wesetthe textvariableargumenttobe_url._urlisaStringVar,whichisanobjectthatisnow connectedtoEntryandwillbeusedtomanipulateitscontent.Therefore,wedon’tmodify thetextinthe_url_entryinstancedirectly,butbyaccessing_url.Inthiscase,wecall thesetmethodonittosettheinitialvaluetotheURLofourlocalwebpage. Weposition_url_entryat(0,0),settingallfourcardinaldirectionsforittostickto,and wealsosetabitofextrapaddingontheleftandrightedgesbyusingpadx,whichadds paddingonthex-axis(horizontal).Ontheotherhand,padytakescareofthevertical direction. Bynow,youshouldgetthateverytimeyoucallthe.gridmethodonanobject,we’re basicallytellingthegridlayoutmanagertoplacethatobjectsomewhere,accordingto rulesthatwespecifyasargumentsinthegrid()call. Similarly,wesetupandplacethe_fetchbutton.Theonlyinterestingparameteris command=fetch_url.Thismeansthatwhenweclickthisbutton,weactuallycallthe fetch_urlfunction.Thistechniqueiscalledcallback. _img_frame=ttk.LabelFrame( _mainframe,text='Content',padding='9000') _img_frame.grid(row=1,column=0,sticky=(N,S,E,W)) ThisiswhatwecalledImgFrameinthelayoutdraft.Itisplacedonthesecondrowofits parent_mainframe.ItwillholdtheListboxandtheRadioFrame. _images=StringVar() _img_listbox=Listbox( _img_frame,listvariable=_images,height=6,width=25) _img_listbox.grid(row=0,column=0,sticky=(E,W),pady=5) _scrollbar=ttk.Scrollbar( _img_frame,orient=VERTICAL,command=_img_listbox.yview) _scrollbar.grid(row=0,column=1,sticky=(S,N),pady=6) _img_listbox.configure(yscrollcommand=_scrollbar.set) Thisisprobablythemostinterestingbitofthewholelayoutlogic.Aswedidwiththe _url_entry,weneedtodrivethecontentsofListboxbytyingittoavariable_images. WesetupListboxsothat_img_frameisitsparent,and_imagesisthevariableit’stiedto. Wealsopasssomedimensions. Theinterestingbitcomesfromthe_scrollbarinstance.Notethat,whenweinstantiateit, wesetitscommandto_img_listbox.yview.Thisisthefirsthalfofthecontractbetween aListboxandaScrollbar.Theotherhalfisprovidedbythe_img_listbox.configure method,whichsetstheyscrollcommand=_scrollbar.set. WOW! eBook www.wowebook.org Byprovidingthisreciprocalbond,whenwescrollonListbox,theScrollbarwillmove accordinglyandvice-versa,whenweoperatetheScrollbar,theListboxwillscroll accordingly. _radio_frame=ttk.Frame(_img_frame) _radio_frame.grid(row=0,column=2,sticky=(N,S,W,E)) WeplacetheRadioFrame,readytobepopulated.NotethattheListboxisoccupying(0, 0)on_img_frame,theScrollbar(0,1)andtherefore_radio_framewillgoin(0,2). _choice_lbl=ttk.Label( _radio_frame,text="Choosehowtosaveimages") _choice_lbl.grid(row=0,column=0,padx=5,pady=5) _save_method=StringVar() _save_method.set('img') _img_only_radio=ttk.Radiobutton( _radio_frame,text='AsImages',variable=_save_method, value='img') _img_only_radio.grid( row=1,column=0,padx=5,pady=2,sticky=W) _img_only_radio.configure(state='normal') _json_radio=ttk.Radiobutton( _radio_frame,text='AsJSON',variable=_save_method, value='json') _json_radio.grid(row=2,column=0,padx=5,pady=2,sticky=W) Firstly,weplacethelabel,andwegiveitsomepadding.Notethatthelabelandradio buttonsarechildrenof_radio_frame. AsfortheEntryandListboxobjects,theRadiobuttonisalsodrivenbyabondtoan externalvariable,whichIcalled_save_method.EachRadiobuttoninstancesetsavalue argument,andbycheckingthevalueon_save_method,weknowwhichbuttonisselected. _scrape_btn=ttk.Button( _mainframe,text='Scrape!',command=save) _scrape_btn.grid(row=2,column=0,sticky=E,pady=5) Onthethirdrowof_mainframeweplacetheScrapebutton.Itscommandissave,which savestheimagestobelistedinListbox,afterwehavesuccessfullyparsedawebpage. _status_frame=ttk.Frame( _root,relief='sunken',padding='2222') _status_frame.grid(row=1,column=0,sticky=(E,W,S)) _status_msg=StringVar() _status_msg.set('TypeaURLtostartscraping…') _status=ttk.Label( _status_frame,textvariable=_status_msg,anchor=W) _status.grid(row=0,column=0,sticky=(E,W)) Weendthelayoutsectionbyplacingdownthestatusframe,whichisasimplettk.Frame. Togiveitalittlestatusbareffect,wesetitsreliefpropertyto'sunken'andgiveita uniformpaddingof2pixels.Itneedstosticktothe_rootwindowleft,rightandbottom parts,sowesetitsstickyattributeto(E,W,S). Wethenplacealabelinitand,thistime,wetieittoaStringVarobject,becausewewill WOW! eBook www.wowebook.org havetomodifyiteverytimewewanttoupdatethestatusbartext.Youshouldbe acquaintedtothistechniquebynow. Finally,onthelastline,weruntheapplicationbycallingthemainloopmethodontheTk instance. _root.mainloop() Pleaserememberthatalltheseinstructionsareplacedundertheif__name__== "__main__":clauseintheoriginalscript. Asyoucansee,thecodetodesignourGUIapplicationisnothard.Granted,atthe beginningyouhavetoplayaroundalittlebit.Noteverythingwillworkoutperfectlyat thefirstattempt,butIpromiseyouit’sveryeasyandyoucanfindplentyoftutorialson theweb.Let’snowgettotheinterestingbit,thebusinesslogic. WOW! eBook www.wowebook.org Thebusinesslogic We’llanalyzethebusinesslogicoftheGUIapplicationinthreechunks.Thereisthe fetchinglogic,thesavinglogic,andthealertinglogic. Fetchingthewebpage config={} deffetch_url(): url=_url.get() config['images']=[] _images.set(())#initializedasanemptytuple try: page=requests.get(url) exceptrequests.RequestExceptionasrex: _sb(str(rex)) else: soup=BeautifulSoup(page.content,'html.parser') images=fetch_images(soup,url) ifimages: _images.set(tuple(img['name']forimginimages)) _sb('Imagesfound:{}'.format(len(images))) else: _sb('Noimagesfound') config['images']=images deffetch_images(soup,base_url): images=[] forimginsoup.findAll('img'): src=img.get('src') img_url=( '{base_url}/{src}'.format(base_url=base_url,src=src)) name=img_url.split('/')[-1] images.append(dict(name=name,url=img_url)) returnimages Firstofall,letmeexplainthatconfigdictionary.Weneedsomewayofpassingdata betweentheGUIapplicationandthebusinesslogic.Now,insteadofpollutingtheglobal namespacewithmanydifferentvariables,mypersonalpreferenceistohaveasingle dictionarythatholdsalltheobjectsweneedtopassbackandforth,sothattheglobal namespaceisn’tbecloggedupwithallthosenames,andwehaveonesingle,clean,easy wayofknowingwherealltheobjectsthatareneededbyourapplicationare. Inthissimpleexample,we’lljustpopulatetheconfigdictionarywiththeimageswefetch fromthepage,butIwantedtoshowyouthetechniquesothatyouhaveatleastan example.ThistechniquecomesfrommyexperiencewithJavaScript.Whenyoucodea webpage,youveryoftenimportseveraldifferentlibraries.Ifeachoftheseclutteredthe globalnamespacewithallsortsofvariables,therewouldbesevereissuesinmaking everythingwork,becauseofnameclashesandvariableoverriding.Theymakethecoder’s lifealivinghell. So,it’smuchbettertotryandleavetheglobalnamespaceascleanaswecan.Inthiscase, WOW! eBook www.wowebook.org Ifindthatusingoneconfigvariableismorethanacceptable. Thefetch_urlfunctionisquitesimilartowhatwedidinthescript.Firstly,wegetthe urlvaluebycalling_url.get().Rememberthatthe_urlobjectisaStringVarinstance thatistiedtothe_url_entryobject,whichisanEntry.ThetextfieldyouseeontheGUI istheEntry,butthetextbehindthescenesisthevalueoftheStringVarobject. Bycallingget()on_url,wegetthevalueofthetextwhichisdisplayedin_url_entry. Thenextstepistoprepareconfig['images']tobeanemptylist,andtoemptythe _imagesvariable,whichistiedto_img_listbox.This,ofcourse,hastheeffectof cleaningupalltheitemsin_img_listbox. Afterthispreparationstep,wecantrytofetchthepage,usingthesametry/exceptlogic weadoptedinthescriptatthebeginningofthechapter. Theonedifferenceisintheactionwetakeifthingsgowrong.Wecall_sb(str(rex)). _sbisahelperfunctionwhosecodewe’llseeshortly.Basically,itsetsthetextinthestatus barforus.Notagoodname,right?Ihadtoexplainitsbehaviortoyou:foodforthought. Ifwecanfetchthepage,thenwecreatethesoupinstance,andfetchtheimagesfromit. Thelogicoffetch_imagesisexactlythesameastheoneexplainedbefore,soIwon’t repeatmyselfhere. Ifwehaveimages,usingaquicktuplecomprehension(whichisactuallyagenerator expressionfedtoatupleconstructor)wefeedthe_imagesStringVarandthishasthe effectofpopulatingour_img_listboxwithalltheimagenames.Finally,weupdatethe statusbar. Iftherewerenoimages,westillupdatethestatusbar,andattheendofthefunction, regardlessofhowmanyimageswerefound,weupdateconfig['images']toholdthe imageslist.Inthisway,we’llbeabletoaccesstheimagesfromotherfunctionsby inspectingconfig['images']withouthavingtopassthatlistaround. Savingtheimages Thelogictosavetheimagesisprettystraightforward.Hereitis: defsave(): ifnotconfig.get('images'): _alert('Noimagestosave') return if_save_method.get()=='img': dirname=filedialog.askdirectory(mustexist=True) _save_images(dirname) else: filename=filedialog.asksaveasfilename( initialfile='images.json', filetypes=[('JSON','.json')]) _save_json(filename) def_save_images(dirname): ifdirnameandconfig.get('images'): WOW! eBook www.wowebook.org forimginconfig['images']: img_data=requests.get(img['url']).content filename=os.path.join(dirname,img['name']) withopen(filename,'wb')asf: f.write(img_data) _alert('Done') def_save_json(filename): iffilenameandconfig.get('images'): data={} forimginconfig['images']: img_data=requests.get(img['url']).content b64_img_data=base64.b64encode(img_data) str_img_data=b64_img_data.decode('utf-8') data[img['name']]=str_img_data withopen(filename,'w')asijson: ijson.write(json.dumps(data)) _alert('Done') WhentheuserclickstheScrapebutton,thesavefunctioniscalledusingthecallback mechanism. Thefirstthingthatthisfunctiondoesischeckwhetherthereareactuallyanyimagestobe saved.Ifnot,italertstheuseraboutit,usinganotherhelperfunction,_alert,whosecode we’llseeshortly.Nofurtheractionisperformediftherearenoimages. Ontheotherhand,iftheconfig['images']listisnotempty,saveactsasadispatcher, anditcalls_save_imagesor_save_json,accordingtowhichvalueisheldby _same_method.Remember,thisvariableistiedtotheradiobuttons,thereforeweexpectits valuetobeeither'img'or'json'. Thisdispatcherisabitdifferentfromtheoneinthescript.Accordingtowhichmethodwe haveselected,adifferentactionmustbetaken. Ifwewanttosavetheimagesasimages,weneedtoasktheusertochooseadirectory.We dothisbycallingfiledialog.askdirectoryandassigningtheresultofthecalltothe variabledirname.Thisopensupanicedialogwindowthatasksustochooseadirectory. Thedirectorywechoosemustexist,asspecifiedbythewaywecallthemethod.Thisis donesothatwedon’thavetowritecodetodealwithapotentiallymissingdirectorywhen savingthefiles. Here’showthisdialogshouldlookonUbuntu: WOW! eBook www.wowebook.org Ifwecanceltheoperation,dirnamewillbesettoNone. Beforefinishinganalyzingthelogicinsave,let’squicklygothrough_save_images. It’sverysimilartotheversionwehadinthescriptsojustnotethat,atthebeginning,in ordertobesurethatweactuallyhavesomethingtodo,wecheckonbothdirnameandthe presenceofatleastoneimageinconfig['images']. Ifthat’sthecase,itmeanswehaveatleastoneimagetosaveandthepathforit,sowecan proceed.Thelogictosavetheimageshasalreadybeenexplained.Theonethingwedo differentlythistimeistojointhedirectory(whichmeansthecompletepath)totheimage name,bymeansofos.path.join.Intheos.pathmodulethere’splentyofusefulmethods toworkwithpathsandfilenames. Attheendof_save_images,ifwesavedatleastoneimage,wealerttheuserthatwe’re done. Let’sgobacknowtotheotherbranchinsave.Thisbranchisexecutedwhentheuser selectstheAsJSONradiobuttonbeforepressingtheScrapebutton.Inthiscase,wewant tosaveafile;therefore,wecannotjustaskforadirectory.Wewanttogivetheuserthe abilitytochooseafilenameaswell.Hence,wefireupadifferentdialog: filedialog.asksaveasfilename. Wepassaninitialfilename,whichisproposedtotheuserwiththeabilitytochangeitif theydon’tlikeit.Moreover,becausewe’resavingaJSONfile,we’reforcingtheuserto WOW! eBook www.wowebook.org usethecorrectextensionbypassingthefiletypesargument.Itisalistwithanynumber of2-tuples(description,extension)thatrunsthelogicofthedialog. Here’showthisdialogshouldlookonUbuntu: Oncewehavechosenaplaceandafilename,wecanproceedwiththesavinglogic,which isthesameasitwasinthepreviousscript.WecreateaJSONobjectfromaPython dictionary(data)thatwepopulatewithkey/valuepairsmadebytheimagesnameand Base64encodedcontent. In_save_jsonaswell,wehavealittlecheckatthebeginningthatmakessurethatwe don’tproceedunlesswehaveafilenameandatleastoneimagetosave. ThisensuresthatiftheuserpressestheCancelbutton,nothingbadhappens. Alertingtheuser Finally,let’sseethealertinglogic.It’sextremelysimple. def_sb(msg): _status_msg.set(msg) def_alert(msg): messagebox.showinfo(message=msg) That’sit!Tochangethestatusbarmessageallweneedtodoistoaccess_status_msg StringVar,asit’stiedtothe_statuslabel. WOW! eBook www.wowebook.org Ontheotherhand,ifwewanttoshowtheuseramorevisiblemessage,wecanfireupa messagebox.Here’showitshouldlookonUbuntu: Themessageboxobjectcanalsobeusedtowarntheuser(messagebox.showwarning)orto signalanerror(messagebox.showerror).Butitcanalsobeusedtoprovidedialogsthat askusifwe’resurethatwewanttoproceedorifwereallywanttodeletethatfile,andso on. Ifyouinspectmessageboxbysimplyprintingoutwhatdir(messagebox)returns,you’ll findmethodslikeaskokcancel,askquestion,askretrycancel,askyesno,and askyesnocancel,aswellasasetofconstantstoverifytheresponseoftheuser,suchas CANCEL,NO,OK,OKCANCEL,YES,YESNOCANCEL,andsoon.Youcancomparethesetothe user’schoicesothatyouknowwhatthenextactiontoexecutewhenthedialogcloses. WOW! eBook www.wowebook.org Howtoimprovetheapplication? Nowthatyou’reaccustomedtothefundamentalsofdesigningaGUIapplication,I’dlike togiveyousomesuggestionsonhowtomakeoursbetter. Wecanstartfromthecodequality.Doyouthinkthiscodeisgoodenough,orwouldyou improveit?Ifso,how?Iwouldtestit,andmakesureit’srobustandcatersforallthe variousscenariosthatausermightcreatebyclickingaroundontheapplication.Iwould alsomakesurethebehavioriswhatIwouldexpectwhenthewebsitewe’rescrapingis downforanyreason. Anotherthingthatwecouldimproveisthenaming.Ihaveprudentlynamedallthe componentswithaleadingunderscore,bothtohighlighttheirsomewhat“private”nature, andtoavoidhavingnameclasheswiththeunderlyingobjectstheyarelinkedto.Butin retrospect,manyofthosecomponentscoulduseabettername,soit’sreallyuptoyouto refactoruntilyoufindtheformthatsuitsyoubest.Youcouldstartbygivingabettername tothe_sbfunction! Forwhatconcernstheuserinterface,youcouldtryandresizethemainapplication.See whathappens?Thewholecontentstaysexactlywhereitis.Emptyspaceisaddedifyou expand,orthewholewidgetssetdisappearsgraduallyifyoushrink.Thisbehaviorisn’t exactlynice,thereforeonequicksolutioncouldbetomaketherootwindowfixed(thatis, unabletoresize). Anotherthingthatyoucoulddotoimprovetheapplicationistoaddthesamefunctionality wehadinthescript,tosaveonlyPNGsorJPGs.Inordertodothis,youcouldplacea comboboxsomewhere,withthreevalues:All,PNGs,JPGs,orsomethingsimilar.The usershouldbeabletoselectoneofthoseoptionsbeforesavingthefiles. Evenbetter,youcouldchangethedeclarationofListboxsothatit’spossibletoselect multipleimagesatthesametime,andonlytheselectedoneswillbesaved.Ifyoumanage todothis(it’snotashardasitseems,believeme),thenyoushouldconsiderpresentingthe Listboxabitbetter,maybeprovidingalternatingbackgroundcolorsfortherows. Anothernicethingyoucouldaddisabuttonthatopensupadialogtoselectafile.Thefile mustbeoneoftheJSONfilestheapplicationcanproduce.Onceselected,youcouldrun somelogictoreconstructtheimagesfromtheirBase64-encodedversion.Thelogictodo thisisverysimple,sohere’sanexample: withopen('images.json','r')asf: data=json.loads(f.read()) for(name,b64val)indata.items(): withopen(name,'wb')asf: f.write(base64.b64decode(b64val)) Asyoucansee,weneedtoopenimages.jsoninreadmode,andgrabthedatadictionary. Oncewehaveit,wecanloopthroughitsitems,andsaveeachimagewiththeBase64 decodedcontent.I’llleaveituptoyoutotiethislogictoabuttonintheapplication. WOW! eBook www.wowebook.org Anothercoolfeaturethatyoucouldaddistheabilitytoopenupapreviewpanethat showsanyimageyouselectfromtheListbox,sothattheusercantakeapeekatthe imagesbeforedecidingtosavethem. Finally,onelastsuggestionforthisapplicationistoaddamenu.Maybeevenasimple menuwithFileand?toprovidetheusualHelporAbout.Justforfun.Addingmenusis notthatcomplicated;youcanaddtext,keyboardshortcuts,images,andsoon. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Wheredowegofromhere? IfyouareinterestedindiggingdeeperintotheworldofGUIs,thenI’dliketoofferyou thefollowingsuggestions. WOW! eBook www.wowebook.org Thetkinter.tixmodule Exploringtkinteranditsthemedwidgetset,tkinter.ttk,willtakeyousometime. There’smuchtolearnandplaywith.Anotherinterestingmoduletoexplore,whenyou’ll befamiliarwiththistechnology,istkinter.tix. Thetkinter.tix(TkInterfaceExtension)moduleprovidesanadditionalveryrichset ofwidgets.TheneedforthemstemsfromthefactthatthewidgetsinthestandardTk libraryarefarfromcomplete. Thetkinter.tixlibraryallowsustosolvethisproblembyprovidingwidgetslikeHList, ComboBox,Control(orSpinBox),andvariousscrollablewidgets.Altogether,thereare morethan40widgets.Theyallowyoutointroducedifferentinteractiontechniquesand paradigmsintoyourapplications,thusimprovingtheirqualityandusability. WOW! eBook www.wowebook.org Theturtlemodule Theturtlemoduleisanextendedreimplementationoftheeponymousmodulefromthe PythonstandarddistributionuptoversionPython2.5.It’saverypopularwaytointroduce childrentoprogramming. It’sbasedontheideaofanimaginaryturtlestartingat(0,0)intheCartesianplane.You canprogrammaticallycommandtheturtletomoveforwardandbackwards,rotate,andso on.andbycombiningtogetherallthepossiblemoves,allsortsofintricateshapesand imagescanbedrawn. It’sdefinitelyworthcheckingout,ifonlytoseesomethingdifferent. WOW! eBook www.wowebook.org wxPython,PyQt,andPyGTK Afteryouhaveexploredthevastnessofthetkinterrealm,I’dsuggestyoutoexplore otherGUIlibraries:wxPython,PyQt,andPyGTK.Youmayfindoutoneoftheseworks betterforyou,oritmakeseasierforyoutocodetheapplicationyouneed. Ibelievethatcoderscanrealizetheirideasonlywhentheyareconsciousaboutwhattools theyhaveavailable.Ifyourtoolsetistoonarrow,yourideasmayseemimpossibleor extremelyhardtorealize,andtheyriskremainingexactlywhattheyare,justideas. Ofcourse,thetechnologicalspectrumtodayishumongous,soknowingeverythingisnot possible;therefore,whenyouareabouttolearnanewtechnologyoranewsubject,my suggestionistogrowyourknowledgebyexploringbreadthfirst. Investigateseveralthingsnottoodeeply,andthengodeepwiththeoneorthefewthat lookedmostpromising.Thiswayyou’llbeabletobeproductivewithatleastonetool, andwhenthetoolnolongerfitsyourneeds,you’llknowwheretodigdeeper,thanksto yourpreviousexploration. WOW! eBook www.wowebook.org Theprincipleofleastastonishment Whendesigninganinterface,therearemanydifferentthingstobearinmind.Oneofthem, whichformeisthemostimportant,isthelaworprincipleofleastastonishment.It basicallystatesthatifinyourdesignanecessaryfeaturehasahighastonishingfactor,it maybenecessarytoredesignyourapplication.Togiveyouoneexample,whenyou’re usedtoworkingwithWindows,wherethebuttonstominimize,maximizeandclosea windowareonthetop-rightcorner,it’squitehardtoworkonLinux,wheretheyareatthe top-leftcorner.You’llfindyourselfconstantlygoingtothetop-rightcorneronlyto discoveroncemorethatthebuttonsareontheotherside. Ifacertainbuttonhasbecomesoimportantinapplicationsthatit’snowplacedinaprecise locationbydesigners,pleasedon’tinnovate.Justfollowtheconvention.Userswillonly becomefrustratedwhentheyhavetowastetimelookingforabuttonthatisnotwhereit’s supposedtobe. ThedisregardforthisruleisthereasonwhyIcannotworkwithproductslikeJira.Ittakes meminutestodosimplethingsthatshouldrequireseconds. WOW! eBook www.wowebook.org Threadingconsiderations Thistopicisbeyondthescopeofanintroductorybooklikethis,butIdowanttomention it.Inanutshell,athreadofexecutionisthesmallestsequenceofprogrammed instructionsthatcanbemanagedindependentlybyascheduler.Thereasonwehavethe perceptionthatmoderncomputerscandomanythingsatthesametimeisnotonlydueto thefactthattheyhavemultipleprocessors.Theyalsosubdividetheworkindifferent threads,whicharethenworkedoninsequence.Iftheirlifecycleissufficientlyshort, threadscanbeworkedoninonesinglego,buttypically,whathappensisthattheOS worksonathreadforalittletime,thenswitchestoanotherone,thentoanotherone,then backtothefirstone,andsoon.Theorderinwhichtheyareworkedondependson differentfactors.Theendresultisthat,becausecomputersareextremelyfastindoingthis switching,weperceivemanythingshappeningatthesametime. IfyouarecodingaGUIapplicationthatneedstoperformalongrunningoperationwhena buttonisclicked,youwillseethatyourapplicationwillprobablyfreezeuntiltheoperation hasbeencarriedout.Inordertoavoidthis,andmaintaintheapplication’sresponsiveness, youmayneedtorunthattime-expensiveoperationinadifferentthreadsothattheOSwill beabletodedicatealittlebitoftimetotheGUIeverynowandthen,tokeepitresponsive. Threadsareanadvancedtopic,especiallyinPython.Gainagoodgraspofthe fundamentalsfirst,andthenhavefunexploringthem! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,weworkedonaprojecttogether.Wehavewrittenascriptthatscrapesa verysimplewebpageandacceptsoptionalcommandsthatalteritsbehaviorindoingso. WealsocodedaGUIapplicationtodothesamethingbyclickingbuttonsinsteadof typingonaconsole.IhopeyouenjoyedreadingitandfollowingalongasmuchasI enjoyedwritingit. Wesawmanydifferentconceptslikecontextmanagers,workingwithfiles,performing HTTPrequests,andwe’vetalkedaboutguidelinesforusabilityanddesign. Ihaveonlybeenabletoscratchthesurface,buthopefully,youhaveagoodstartingpoint fromwhichtoexpandyourexploration. Throughoutthechapter,Ihavepointedyouinseveraldifferentwaysonhowtoimprove theapplication,andIhavechallengedyouwithafewexercisesandquestions.Ihopeyou havetakenthetimetoplaywiththoseideas.Onecanlearnalotjustbyplayingaround withfunapplicationsliketheonewe’vecodedtogether. Inthenextchapter,we’regoingtotalkaboutdatascience,oratleastaboutthetoolsthata Pythonprogrammerhaswhenitcomestofacingthissubject. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter9.DataScience “Ifwehavedata,let’slookatdata.Ifallwehaveareopinions,let’sgowithmine.” —JimBarksdale,formerNetscapeCEO Datascienceisaverybroadterm,andcanassumeseveraldifferentmeaningsaccording tocontext,understanding,tools,andsoon.Therearecountlessbooksaboutthissubject, whichisnotsuitableforthefaint-hearted. Inordertodoproperdatascience,youneedtoknowmathematicsandstatisticsatthevery least.Then,youmaywanttodigintoothersubjectssuchaspatternrecognitionand machinelearningand,ofcourse,thereisaplethoraoflanguagesandtoolsyoucanchoose from. UnlessItransformintoTheAmazingFabriziointhenextfewminutes,Iwon’tbeableto talkabouteverything;Iwon’tevengetclosetoit.Therefore,inordertorenderthis chaptermeaningful,we’regoingtoworkonacoolprojecttogether. About3yearsago,Iwasworkingforatop-tiersocialmediacompanyinLondon.Istayed therefor2years,andIwasprivilegedtoworkwithseveralpeoplewhosebrillianceIcan onlystarttodescribe.WewerethefirstintheworldtohaveaccesstotheTwitterAdsAPI, andwewerepartnerswithFacebookaswell.Thatmeansalotofdata. Ouranalystsweredealingwithahugenumberofcampaignsandtheywerestruggling withtheamountofworktheyhadtodo,sothedevelopmentteamIwasapartoftriedto helpbyintroducingthemtoPythonandtothetoolsPythongivesyoutodealwithdata.It wasaveryinterestingjourneythatledmetomentorseveralpeopleinthecompanyand eventuallytoManilawhere,for2weeks,IgaveintensivetraininginPythonanddata sciencetoouranalyststhere. Theprojectwe’regoingtodotogetherinthischapterisalightweightversionofthefinal exampleIpresentedtomyManilastudents.Ihaverewrittenittoasizethatwillfitthis chapter,andmadeafewadjustmentshereandthereforteachingpurposes,butallthemain conceptsarethere,soitshouldbefunandinstructionalforyoutocodealong. Onourjourney,we’regoingtomeetafewofthetoolsyoucanfindinthePython ecosystemwhenitcomestodealingwithdata,solet’sstartbytalkingaboutRomangods. WOW! eBook www.wowebook.org IPythonandJupyternotebook In2001,FernandoPerezwasagraduatestudentinphysicsatCUBoulder,andwastrying toimprovethePythonshellsothathecouldhavesomenicetieslikethosehewasusedto whenhewasworkingwithtoolssuchasMathematicaandMaple.Theresultofthateffort tookthenameIPython. Inanutshell,thatsmallscriptbeganasanenhancedversionofthePythonshelland, throughtheeffortofothercodersandeventuallyproperfundingfromseveraldifferent companies,itbecamethewonderfulandsuccessfulprojectitistoday.Some10yearsafter itsbirth,anotebookenvironmentwascreated,poweredbytechnologieslikeWebSockets, theTornadowebserver,jQuery,CodeMirror,andMathJax.TheZeroMQlibrarywasalso usedtohandlethemessagesbetweenthenotebookinterfaceandthePythoncorethatlies behindit. TheIPythonnotebookhasbecomesopopularandwidelyusedthateventually,allsortsof goodieshavebeenaddedtoit.Itcanhandlewidgets,parallelcomputing,allsortsofmedia formats,andmuch,muchmore.Moreover,atsomepoint,itbecamepossibletocodein languagesotherthanPythonfromwithinthenotebook. Thishasledtoahugeprojectthatonlyrecentlyhasbeensplitintotwo:IPythonhasbeen strippeddowntofocusmoreonthekernelpartandtheshell,whilethenotebookhas becomeabrandnewprojectcalledJupyter.Jupyterallowsinteractivescientific computationstobedoneinmorethan40languages. Thischapter’sprojectwillallbecodedandruninaJupyternotebook,soletmeexplainin afewwordswhatanotebookis. Anotebookenvironmentisawebpagethatexposesasimplemenuandthecellsinwhich youcanrunPythoncode.Eventhoughthecellsareseparateentitiesthatyoucanrun individually,theyallsharethesamePythonkernel.Thismeansthatallthenamesthatyou defineinacell(thevariables,functions,andsoon)willbeavailableinanyothercell. Note Simplyput,aPythonkernelisaprocessinwhichPythonisrunning.Thenotebookweb pageisthereforeaninterfaceexposedtotheuserfordrivingthiskernel.Thewebpage communicatestoitusingaveryfastmessagingsystem. Apartfromallthegraphicaladvantages,thebeautytohavesuchanenvironmentconsists intheabilityofrunningaPythonscriptinchunks,andthiscanbeatremendous advantage.Takeascriptthatisconnectingtoadatabasetofetchdataandthenmanipulate thatdata.Ifyoudoitintheconventionalway,withaPythonscript,youhavetofetchthe dataeverytimeyouwanttoexperimentwithit.Withinanotebookenvironment,youcan fetchthedatainacellandthenmanipulateandexperimentwithitinothercells,so fetchingiteverytimeisnotnecessary. Thenotebookenvironmentisalsoextremelyhelpfulfordatasciencebecauseitallowsfor step-by-stepintrospection.Youdoonechunkofworkandthenverifyit.Youthendo WOW! eBook www.wowebook.org anotherchunkandverifyagain,andsoon. It’salsoinvaluableforprototypingbecausetheresultsarethere,rightinfrontofyour eyes,immediatelyavailable. Ifyouwanttoknowmoreaboutthesetools,pleasecheckouthttp://ipython.org/and http://jupyter.org/. Ihavecreatedaverysimpleexamplenotebookwithafibonaccifunctionthatgivesyou thelistofallFibonaccinumberssmallerthanagivenN.Inmybrowser,itlookslikethis: EverycellhasanIn[]label.Ifthere’snothingbetweenthebraces,itmeansthatcellhas neverbeenexecuted.Ifthereisanumber,itmeansthatthecellhasbeenexecuted,andthe numberrepresentstheorderinwhichthecellwasexecuted.Finally,a*meansthatthe celliscurrentlybeingexecuted. YoucanseeinthepicturethatinthefirstcellIhavedefinedthefibonaccifunction,andI haveexecutedit.Thishastheeffectofplacingthefibonaccinameintheglobalframe associatedwiththenotebook,thereforethefibonaccifunctionisnowavailabletothe othercellsaswell.Infact,inthesecondcell,Icanrunfibonacci(100)andseetheresults inOut[2].Inthethirdcell,Ihaveshownyouoneoftheseveralmagicfunctionsyoucan findinanotebookinthesecondcell.%timeitrunsthecodeseveraltimesandprovides youwithanicebenchmarkforit.Allthemeasurementsforthelistcomprehensionsand generatorsIdidinChapter5,SavingTimeandMemorywerecarriedoutwiththisnice feature. Youcanexecuteacellasmanytimesasyouwant,andchangetheorderinwhichyourun them.Cellsareverymalleable,youcanalsoputinmarkdowntextorrenderthemas WOW! eBook www.wowebook.org headers. Note Markdownisalightweightmarkuplanguagewithplaintextformattingsyntaxdesigned sothatitcanbeconvertedtoHTMLandmanyotherformats. Also,whateveryouplaceinthelastrowofacellwillbeautomaticallyprintedforyou. Thisisveryhandybecauseyou’renotforcedtowriteprint(...)explicitly. Feelfreetoexplorethenotebookenvironment;onceyou’refriendswithit,it’salonglastingrelationship,Ipromise. Inordertorunthenotebook,youhavetoinstallahandfuloflibraries,eachofwhich collaborateswiththeotherstomakethewholethingwork.Alternatively,youcanjust installJupyteranditwilltakecareofeverythingforyou.Forthischapter,thereareafew otherdependenciesthatweneedtoinstall,sopleaserunthefollowingcommand: $pipinstalljupyterpandasmatplotlibfake-factorydeloreanxlwt Don’tworry,I’llintroduceyoutoeachofthesegradually.Now,whenyou’redone installingtheselibraries(itmaytakeafewminutes),youcanstartthenotebook: $jupyternotebook Thiswillopenapageinyourbrowseratthisaddress:http://localhost:8888/. Gotothatpageandcreateanewnotebookusingthemenu.Whenyouhaveitandyou’re comfortablewithit,we’rereadytogo. Tip Ifyouexperienceanyissuessettingupthenotebookenvironment,pleasedon’tget discouraged.Ifyougetanerror,it’susuallyjustamatterofsearchingalittlebitonthe webandyou’llenduponapagewheresomeoneelsehashadthesameissue,andthey haveexplainedhowtofixit.Tryyourbesttohavethenotebookenvironmentupand runningbeforecontinuingwiththechapter. Ourprojectwilltakeplaceinanotebook,thereforeIwilltageachcodesnippetwiththe cellnumberitbelongsto,sothatyoucaneasilyreproducethecodeandfollowalong. Tip Ifyoufamiliarizeyourselfwiththekeyboardshortcuts(lookinthenotebook’shelp section),youwillbeabletomovebetweencellsandhandletheircontentwithouthaving toreachforthemouse.Thiswillmakeyoumoreproficientandwayfasterwhenyouwork inanotebook. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Dealingwithdata Typically,whenyoudealwithdata,thisisthepathyougothrough:youfetchit,youclean andmanipulateit,thenyouinspectitandpresentresultsasvalues,spreadsheets,graphs, andsoon.Iwantyoutobeinchargeofallthreestepsoftheprocesswithouthavingany externaldependencyonadataprovider,sowe’regoingtodothefollowing: 1. We’regoingtocreatethedata,simulatingthefactthatitcomesinaformatwhichis notperfectorreadytobeworkedon. 2. We’regoingtocleanitandfeedittothemaintoolwe’lluseintheproject: DataFrameofpandas. 3. We’regoingtomanipulatethedataintheDataFrame. 4. We’regoingtosavetheDataFrametoafileindifferentformats. 5. Finally,we’regoingtoinspectthedataandgetsomeresultsoutofit. WOW! eBook www.wowebook.org Settingupthenotebook Firstthingsfirst,weneedtosetupthenotebook.Thismeansimportsandabitof configuration. #1 importjson importcalendar importrandom fromdatetimeimportdate,timedelta importfaker importnumpyasnp frompandasimportDataFrame fromdeloreanimportparse importpandasaspd #makethegraphsnicer pd.set_option('display.mpl_style','default') Cell#1takescareoftheimports.Therearequiteafewnewthingshere:thecalendar, randomanddatetimemodulesarepartofthestandardlibrary.Theirnamesareselfexplanatory,solet’slookatfaker.Thefake-factorylibrarygivesyouthismodule, whichyoucanusetopreparefakedata.It’sveryusefulintests,whenyouprepareyour fixtures,togetallsortsofthingssuchasnames,e-mailaddresses,phonenumbers,credit carddetails,andmuchmore.Itisallfake,ofcourse. numpyistheNumPylibrary,thefundamentalpackageforscientificcomputingwith Python.I’llspendafewwordsonitlateroninthechapter. pandasistheverycoreuponwhichthewholeprojectisbased.ItstandsforPythonData AnalysisLibrary.Amongmanyothers,itprovidestheDataFrame,amatrix-likedata structurewithadvancedprocessingcapabilities.It’scustomarytoimporttheDataFrame separatelyandthendoimportpandasaspd. deloreanisanicethird-partylibrarythatspeedsupdealingwithdatesdramatically. Technically,wecoulddoitwiththestandardlibrary,butIseenoreasonnottoexpanda bittherangeoftheexampleandshowyousomethingdifferent. Finally,wehaveaninstructiononthelastlinethatwillmakeourgraphsattheendalittle bitnicer,whichdoesn’thurt. WOW! eBook www.wowebook.org Preparingthedata Wewanttoachievethefollowingdatastructure:we’regoingtohavealistofuserobjects. Eachuserobjectwillbelinkedtoanumberofcampaignobjects. InPython,everythingisanobject,soI’musingthisterminagenericway.Theuserobject maybeastring,adict,orsomethingelse. Acampaigninthesocialmediaworldisapromotionalcampaignthatamediaagency runsonsocialmedianetworksonbehalfofaclient. Rememberthatwe’regoingtopreparethisdatasothatit’snotinperfectshape(butit won’tbesobadeither…). #2 fake=faker.Faker() Firstly,weinstantiatetheFakerthatwe’llusetocreatethedata. #3 usernames=set() usernames_no=1000 #populatethesetwith1000uniqueusernames whilelen(usernames)<usernames_no: usernames.add(fake.user_name()) Thenweneedusernames.Iwant1,000uniqueusernames,soIloopoverthelengthofthe usernamessetuntilithas1,000elements.Asetdoesn’tallowduplicatedelements, thereforeuniquenessisguaranteed. #4 defget_random_name_and_gender(): skew=.6#60%ofuserswillbefemale male=random.random()>skew ifmale: returnfake.name_male(),'M' else: returnfake.name_female(),'F' defget_users(usernames): users=[] forusernameinusernames: name,gender=get_random_name_and_gender() user={ 'username':username, 'name':name, 'gender':gender, 'email':fake.email(), 'age':fake.random_int(min=18,max=90), 'address':fake.address(), } users.append(json.dumps(user)) returnusers WOW! eBook www.wowebook.org users=get_users(usernames) users[:3] Here,wecreatealistofusers.Eachusernamehasnowbeenaugmentedtoafull-blown userdict,withotherdetailssuchasname,gender,e-mail,andsoon.Eachuserdictis thendumpedtoJSONandaddedtothelist.Thisdatastructureisnotoptimal,ofcourse, butwe’resimulatingascenariowhereuserscometouslikethat. Notetheskeweduseofrandom.random()tomake60%ofusersfemale.Therestofthe logicshouldbeveryeasyforyoutounderstand. Notealsothelastline.Eachcellautomaticallyprintswhat’sonthelastline;therefore,the outputofthisisalistwiththefirstthreeusers: Out#4 ['{"gender":"F","age":48,"email":"[email protected]", "address":"2006SawaynTrailApt.207\\nHyattview,MO27278","username": "darcy00","name":"VirgiaHilpert"}', '{"gender":"F","age":58,"email":"[email protected]","address": "5176AndresPlainsApt.040\\nLakinside,GA92446","username": "renner.virgie","name":"MissClarabelleKertzmannMD"}', '{"gender":"M","age":33,"email":"[email protected]", "address":"1218JacobsonFort\\nNorthDoctor,OK04469","username": "hettinger.alphonsus","name":"LudwigProsacco"}'] Note Ihopeyou’refollowingalongwithyourownnotebook.Ifyoudo,pleasenotethatalldata isgeneratedusingrandomfunctionsandvalues;therefore,youwillseedifferentresults. Theywillchangeeverytimeyouexecutethenotebook. #5 #campaignnameformat: #InternalType_StartDate_EndDate_TargetAge_TargetGender_Currency defget_type(): #justsomegibberishinternalcodes types=['AKX','BYU','GRZ','KTR'] returnrandom.choice(types) defget_start_end_dates(): duration=random.randint(1,2*365) offset=random.randint(-365,365) start=date.today()-timedelta(days=offset) end=start+timedelta(days=duration) def_format_date(date_): returndate_.strftime("%Y%m%d") return_format_date(start),_format_date(end) defget_age(): age=random.randint(20,45) age-=age%5 WOW! eBook www.wowebook.org diff=random.randint(5,25) diff-=diff%5 return'{}-{}'.format(age,age+diff) defget_gender(): returnrandom.choice(('M','F','B')) defget_currency(): returnrandom.choice(('GBP','EUR','USD')) defget_campaign_name(): separator='_' type_=get_type() start_end=separator.join(get_start_end_dates()) age=get_age() gender=get_gender() currency=get_currency() returnseparator.join( (type_,start_end,age,gender,currency)) In#5,wedefinethelogictogenerateacampaignname.Analystsusespreadsheetsallthe timeandtheycomeupwithallsortsofcodingtechniquestocompressasmuch informationaspossibleintothecampaignnames.TheformatIchoseisasimpleexample ofthattechnique:thereisacodethattellsthecampaigntype,thenstartandenddates,then thetargetageandgender,andfinallythecurrency.Allvaluesareseparatedbyan underscore. Intheget_typefunction,Iuserandom.choice()togetonevaluerandomlyoutofa collection.Probablymoreinterestingisget_start_end_dates.First,Igetthedurationfor thecampaign,whichgoesfrom1dayto2years(randomly),thenIgetarandomoffsetin timewhichIsubtractfromtoday’sdateinordertogetthestartdate.Giventhattheoffset isarandomnumberbetween-365and365,wouldanythingbedifferentifIaddeditto today’sdateinsteadofsubtractingit? WhenIhaveboththestartandenddates,Ireturnastringifiedversionofthem,joinedby anunderscore. Then,wehaveabitofmodulartrickerygoingonwiththeagecalculation.Ihopeyou rememberthemodulooperator(%)fromChapter2,Built-inDataTypes. WhathappenshereisthatIwantadaterangethathasmultiplesof5asextremes.So,there aremanywaystodoit,butwhatIdoistogetarandomnumberbetween20and45forthe leftextreme,andremovetheremainderofthedivisionby5.So,if,forexample,Iget28,I willremove28%5=3toit,getting25.Icouldhavejustusedrandom.randrange(),but it’shardtoresistmodulardivision. Therestofthefunctionsarejustsomeotherapplicationsofrandom.choice()andthelast one,get_campaign_name,isnothingmorethanacollectorforallthesepuzzlepiecesthat returnsthefinalcampaignname. #6 defget_campaign_data(): WOW! eBook www.wowebook.org name=get_campaign_name() budget=random.randint(10**3,10**6) spent=random.randint(10**2,budget) clicks=int(random.triangular(10**2,10**5,0.2*10**5)) impressions=int(random.gauss(0.5*10**6,2)) return{ 'cmp_name':name, 'cmp_bgt':budget, 'cmp_spent':spent, 'cmp_clicks':clicks, 'cmp_impr':impressions } In#6,wewriteafunctionthatcreatesacompletecampaignobject.Iusedafewdifferent functionsfromtherandommodule.random.randint()givesyouanintegerbetweentwo extremes.Theproblemwithitisthatitfollowsauniformprobabilitydistribution,which meansthatanynumberintheintervalhasthesameprobabilityofcomingup. Therefore,whendealingwithalotofdata,ifyoudistributeyourfixturesusingauniform distribution,theresultsyouwillgetwillalllooksimilar.Forthisreason,Ichosetouse triangularandgauss,forclicksandimpressions.Theyusedifferentprobability distributionssothatwe’llhavesomethingmoreinterestingtoseeintheend. Justtomakesurewe’reonthesamepagewiththeterminology:clicksrepresentsthe numberofclicksonacampaignadvertisement,budgetisthetotalamountofmoney allocatedforthecampaign,spentishowmuchofthatmoneyhasalreadybeenspent,and impressionsisthenumberoftimesthecampaignhasbeenfetched,asaresource,fromits source,regardlessoftheamountofclicksthatwereperformedonthecampaign. Normally,theamountofimpressionsisgreaterthantheamountofclicks. Nowthatwehavethedata,it’stimetoputitalltogether: #7 defget_data(users): data=[] foruserinusers: campaigns=[get_campaign_data() for_inrange(random.randint(2,8))] data.append({'user':user,'campaigns':campaigns}) returndata Asyoucansee,eachitemindataisadictwithauserandalistofcampaignsthatare associatedwiththatuser. WOW! eBook www.wowebook.org Cleaningthedata Let’sstartcleaningthedata: #8 rough_data=get_data(users) rough_data[:2]#let'stakeapeek Wesimulatefetchingthedatafromasourceandtheninspectit.Thenotebookisthe perfecttooltoinspectyoursteps.Youcanvarythegranularitytoyourneeds.Thefirst iteminrough_datalookslikethis: [{'campaigns':[{'cmp_bgt':130532, 'cmp_clicks':25576, 'cmp_impr':500001, 'cmp_name':'AKX_20150826_20170305_35-50_B_EUR', 'cmp_spent':57574}, ...omit… {'cmp_bgt':884396, 'cmp_clicks':10955, 'cmp_impr':499999, 'cmp_name':'KTR_20151227_20151231_45-55_B_GBP', 'cmp_spent':318887}], 'user':'{"age":44,"username":"jacob43", "name":"HollandStrosin", "email":"[email protected]", "address":"1038RunolfsdottirParks\\nElmapo…", "gender":"M"}'}] So,wenowstartworkingwithit. #9 data=[] fordatuminrough_data: forcampaignindatum['campaigns']: campaign.update({'user':datum['user']}) data.append(campaign) data[:2]#let'stakeanotherpeek ThefirstthingweneedtodoinordertobeabletofeedaDataFramewiththisdataisto denormalizeit.Thismeanstransformingthedataintoalistwhoseitemsarecampaign dicts,augmentedwiththeirrelativeuserdict.Userswillbeduplicatedineachcampaign theybelongto.Thefirstitemindatalookslikethis: [{'cmp_bgt':130532, 'cmp_clicks':25576, 'cmp_impr':500001, 'cmp_name':'AKX_20150826_20170305_35-50_B_EUR', 'cmp_spent':57574, 'user':'{"age":44,"username":"jacob43", "name":"HollandStrosin", "email":"[email protected]", "address":"1038RunolfsdottirParks\\nElmaport…", "gender":"M"}'}] WOW! eBook www.wowebook.org Youcanseethattheuserobjecthasbeenbroughtintothecampaigndictwhichwas repeatedforeachcampaign. WOW! eBook www.wowebook.org CreatingtheDataFrame Nowit’stimetocreatetheDataFrame: #10 df=DataFrame(data) df.head() Finally,wewillcreatetheDataFrameandinspectthefirstfiverowsusingthehead method.Youshouldseesomethinglikethis: Jupyterrenderstheoutputofthedf.head()callasHTMLautomatically.Inordertohave atext-basedoutput,simplywrapdf.head()inaprintcall. TheDataFramestructureisverypowerful.Itallowsustodoagreatdealofmanipulation onitscontents.Youcanfilterbyrows,columns,aggregateondata,andmanyother operations.Youcanoperatewithrowsorcolumnswithoutsufferingthetimepenaltyyou wouldhavetopayifyouwereworkingondatawithpurePython.Thishappensbecause, underthecovers,pandasisharnessingthepoweroftheNumPylibrary,whichitselfdraws itsincrediblespeedfromthelow-levelimplementationofitscore.NumPystandsfor NumericPython,anditisoneofthemostwidelyusedlibrariesinthedatascience environment. UsingaDataFrameallowsustocouplethepowerofNumPywithspreadsheet-like capabilitiessothatwe’llbeabletoworkonourdatainafashionthatissimilartowhatan analystcoulddo.Only,wedoitwithcode. Butlet’sgobacktoourproject.Let’sseetwowaystoquicklygetabird’seyeviewofthe data: #11 df.count() countyieldsacountofallthenon-emptycellsineachcolumn.Thisisgoodtohelpyou understandhowsparseyourdatacanbe.Inourcase,wehavenomissingvalues,sothe outputis: cmp_bgt4974 cmp_clicks4974 cmp_impr4974 cmp_name4974 WOW! eBook www.wowebook.org cmp_spent4974 user4974 dtype:int64 Nice!Wehave4,974rows,andthedatatypeisintegers(dtype:int64meanslong integersbecausetheytake64bitseach).Giventhatwehave1,000usersandtheamount ofcampaignsperuserisarandomnumberbetween2and8,we’reexactlyinlinewith whatIwasexpecting. #12 df.describe() describeisaniceandquickwaytointrospectabitfurther: cmp_bgtcmp_clickscmp_imprcmp_spent count4974.0000004974.0000004974.0000004974.000000 mean503272.70687640225.764978499999.495979251150.604343 std289393.74746521910.6319502.035355220347.594377 min1250.000000609.000000499992.000000142.000000 25%253647.50000022720.750000499998.00000067526.750000 50%508341.00000036561.500000500000.000000187833.000000 75%757078.25000055962.750000500001.000000385803.750000 max999631.00000098767.000000500006.000000982716.000000 Asyoucansee,itgivesusseveralmeasuressuchascount,mean,std(standarddeviation), min,max,andshowshowdataisdistributedinthevariousquadrants.Thankstothis method,wecouldalreadyhavearoughideaofhowourdataisstructured. Let’sseewhicharethethreecampaignswiththehighestandlowestbudgets: #13 df.sort_index(by=['cmp_bgt'],ascending=False).head(3) Thisgivesthefollowingoutput(truncated): cmp_bgtcmp_clickscmp_imprcmp_name 465599963115343499997AKX_20160814_20180226_40 370899960645367499997KTR_20150523_20150527_35 199599944512580499998AKX_20141102_20151009_30 And(#14)acallto.tail(3),showsustheoneswiththelowestbudget. Unpackingthecampaignname Nowit’stimetoincreasethecomplexityupabit.Firstofall,wewanttogetridofthat horriblecampaignname(cmp_name).Weneedtoexplodeitintopartsandputeachpartin onededicatedcolumn.Inordertodothis,we’llusetheapplymethodoftheSeriesobject. Thepandas.core.series.Seriesclassisbasicallyapowerfulwrapperaroundanarray (thinkofitasalistwithaugmentedcapabilities).WecanextrapolateaSeriesobjectfrom aDataFramebyaccessingitinthesamewaywedowithakeyinadict,andwecancall applyonthatSeriesobject,whichwillrunafunctionfeedingeachitemintheSeriesto it.WecomposetheresultintoanewDataFrame,andthenjointhatDataFramewithour beloveddf. WOW! eBook www.wowebook.org #15 defunpack_campaign_name(name): #veryoptimisticmethod,assumesdataincampaignname #isalwaysingoodstate type_,start,end,age,gender,currency=name.split('_') start=parse(start).date end=parse(end).date returntype_,start,end,age,gender,currency campaign_data=df['cmp_name'].apply(unpack_campaign_name) campaign_cols=[ 'Type','Start','End','Age','Gender','Currency'] campaign_df=DataFrame( campaign_data.tolist(),columns=campaign_cols,index=df.index) campaign_df.head(3) Withinunpack_campaign_name,wesplitthecampaignnameinparts.Weuse delorean.parse()togetaproperdateobjectoutofthosestrings(deloreanmakesit reallyeasytodoit,doesn’tit?),andthenwereturntheobjects.Aquickpeekatthelast linereveals: TypeStartEndAgeGenderCurrency 0KTR2016-06-162017-01-2420-30MEUR 1BYU2014-10-252015-07-3135-50BUSD 2BYU2015-10-262016-03-1735-50MEUR Nice!Oneimportantthing:evenifthedatesappearasstrings,theyarejustthe representationoftherealdateobjectsthatarehostedintheDataFrame. Anotherveryimportantthing:whenjoiningtwoDataFrameinstances,it’simperativethat theyhavethesameindex,otherwisepandaswon’tbeabletoknowwhichrowsgowith which.Therefore,whenwecreatecampaign_df,wesetitsindextotheonefromdf.This enablesustojointhem.WhencreatingthisDataFrame,wealsopassthecolumnsnames. #16 df=df.join(campaign_df) Andafterthejoin,wetakeapeek,hopingtoseematchingdata(outputtruncated): #17 df[['cmp_name']+campaign_cols].head(3) Gives: cmp_nameTypeStartEnd 0KTR_20160616_20170124_20-30_M_EURKTR2016-06-162017-01-24 1BYU_20141025_20150731_35-50_B_USDBYU2014-10-252015-07-31 2BYU_20151026_20160317_35-50_M_EURBYU2015-10-262016-03-17 Asyoucansee,thejoinwassuccessful;thecampaignnameandtheseparatecolumns exposethesamedata.Didyouseewhatwedidthere?We’reaccessingtheDataFrame usingthesquarebracketssyntax,andwepassalistofcolumnnames.Thiswillproducea brandnewDataFrame,withthosecolumns(inthesameorder),onwhichwethencall WOW! eBook www.wowebook.org head(). Unpackingtheuserdata WenowdotheexactsamethingforeachpieceofuserJSONdata.Wecallapplyonthe userSeries,runningtheunpack_user_jsonfunction,whichtakesaJSONuserobjectand transformsitintoalistofitsfields,whichwecantheninjectintoabrandnewDataFrame user_df.Afterthat,we’lljoinuser_dfbackwithdf,likewedidwithcampaign_df. #18 defunpack_user_json(user): #veryoptimisticaswell,expectsuserobjects #tohaveallattributes user=json.loads(user.strip()) return[ user['username'], user['email'], user['name'], user['gender'], user['age'], user['address'], ] user_data=df['user'].apply(unpack_user_json) user_cols=[ 'username','email','name','gender','age','address'] user_df=DataFrame( user_data.tolist(),columns=user_cols,index=df.index) Verysimilartothepreviousoperation,isn’tit?Weshouldalsonoteherethat,when creatinguser_df,weneedtoinstructDataFrameaboutthecolumnnamesand,very important,theindex.Let’sjoin(#19)andtakeaquickpeek(#20): df=df.join(user_df) df[['user']+user_cols].head(2) Theoutputshowsusthateverythingwentwell.We’regood,butwe’renotdoneyet. Ifyoucalldf.columnsinacell,you’llseethatwestillhaveuglynamesforourcolumns. Let’schangethat: #21 better_columns=[ 'Budget','Clicks','Impressions', 'cmp_name','Spent','user', 'Type','Start','End', 'TargetAge','TargetGender','Currency', 'Username','Email','Name', 'Gender','Age','Address', ] df.columns=better_columns Good!Now,withtheexceptionof'cmp_name'and'user',weonlyhavenicenames. CompletingthedatasetNextstepwillbetoaddsomeextracolumns.Foreachcampaign, WOW! eBook www.wowebook.org wehavetheamountofclicksandimpressions,andwehavethespent.Thisallowsusto introducethreemeasurementratios:CTR,CPC,andCPI.TheystandforClickThrough Rate,CostPerClick,andCostPerImpression,respectively. Thelasttwoareeasytounderstand,butCTRisnot.Sufficeittosaythatitistheratio betweenclicksandimpressions.Itgivesyouameasureofhowmanyclickswere performedonacampaignadvertisementperimpression:thehigherthisnumber,themore successfultheadvertisementisinattractinguserstoclickonit. #22 defcalculate_extra_columns(df): #ClickThroughRate df['CTR']=df['Clicks']/df['Impressions'] #CostPerClick df['CPC']=df['Spent']/df['Clicks'] #CostPerImpression df['CPI']=df['Spent']/df['Impressions'] calculate_extra_columns(df) Iwrotethisasafunction,butIcouldhavejustwrittenthecodeinthecell.It’snot important.WhatIwantyoutonoticehereisthatwe’readdingthosethreecolumnswith onelineofcodeeach,buttheDataFrameappliestheoperationautomatically(thedivision, inthiscase)toeachpairofcellsfromtheappropriatecolumns.So,eveniftheyare maskedasthreedivisions,theseareactually4974*3divisions,becausetheyare performedforeachrow.Pandasdoesalotofworkforus,andalsodoesaverygoodjobin hidingthecomplexityofit. Thefunction,calculate_extra_columns,takesaDataFrame,andworksdirectlyonit. Thismodeofoperationiscalledin-place.Doyourememberhowlist.sort()was sortingthelist?Itisthesamedeal. Wecantakealookattheresultsbyfilteringontherelevantcolumnsandcallinghead. #23 df[['Spent','Clicks','Impressions', 'CTR','CPC','CPI']].head(3) Thisshowsusthatthecalculationswereperformedcorrectlyoneachrow: SpentClicksImpressionsCTRCPCCPI 057574255765000010.0511522.2510950.115148 1226319612474999990.1224943.6951850.452639 24354155825000040.0311640.2794250.008708 Now,Iwanttoverifytheaccuracyoftheresultsmanuallyforthefirstrow: #24 clicks=df['Clicks'][0] impressions=df['Impressions'][0] spent=df['Spent'][0] CTR=df['CTR'][0] CPC=df['CPC'][0] CPI=df['CPI'][0] WOW! eBook www.wowebook.org print('CTR:',CTR,clicks/impressions) print('CPC:',CPC,spent/clicks) print('CPI:',CPI,spent/impressions) Ityieldsthefollowingoutput: CTR:0.05115189769620.0511518976962 CPC:2.251094776352.25109477635 CPI:0.1151477697040.115147769704 Thisisexactlywhatwesawinthepreviousoutput.Ofcourse,Iwouldn’tnormallyneedto dothis,butIwantedtoshowyouhowcanyouperformcalculationsthisway.Youcan accessaSeries(acolumn)bypassingitsnametotheDataFrame,insquarebrackets,and thenyouaccesseachrowbyitsposition,exactlyasyouwouldwitharegularlistortuple. We’realmostdonewithourDataFrame.Allwearemissingnowisacolumnthattellsus thedurationofthecampaignandacolumnthattellsuswhichdayoftheweekcorresponds tothestartdateofeachcampaign.Thisallowsmetoexpandonhowtoplaywithdate objects. #25 defget_day_of_the_week(day): number_to_day=dict(enumerate(calendar.day_name,1)) returnnumber_to_day[day.isoweekday()] defget_duration(row): return(row['End']-row['Start']).days df['DayofWeek']=df['Start'].apply(get_day_of_the_week) df['Duration']=df.apply(get_duration,axis=1) Weusedtwodifferenttechniquesherebutfirst,thecode. get_day_of_the_weektakesadateobject.Ifyoucannotunderstandwhatitdoes,please takeafewmomentstotryandunderstandforyourselfbeforereadingtheexplanation.Use theinside-outtechniquelikewe’vedoneafewtimesbefore. So,asI’msureyouknowbynow,ifyouputcalendar.day_nameinalistcall,youget ['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday', 'Sunday'].Thismeansthat,ifweenumeratecalendar.day_namestartingfrom1,weget pairssuchas(1,'Monday'),(2,'Tuesday'),andsoon.Ifwefeedthesepairstoadict, wegetamappingbetweenthedaysoftheweekasnumbers(1,2,3,…)andtheirnames. Whenthemappingiscreated,inordertogetthenameofaday,wejustneedtoknowits number.Togetit,wecalldate.isoweekday(),whichtellsuswhichdayoftheweekthat dateis(asanumber).Youfeedthatintothemappingand,boom!Youhavethenameof theday. get_durationisinterestingaswell.First,noticeittakesanentirerow,notjustasingle value.Whathappensinitsbodyisthatweperformasubtractionbetweenacampaignend andstartdates.Whenyousubtractdateobjectstheresultisatimedeltaobject,which representsagivenamountoftime.Wetakethevalueofits.daysproperty.Itisassimple asthat. WOW! eBook www.wowebook.org Now,wecanintroducethefunpart,theapplicationofthosetwofunctions. ThefirstapplicationisperformedonaSeriesobject,likewedidbeforefor'user'and 'cmp_name',thereisnothingnewhere. ThesecondoneisappliedtothewholeDataFrameand,inordertoinstructPandasto performthatoperationontherows,wepassaxis=1. Wecanverifytheresultsveryeasily,asshownhere: #26 df[['Start','End','Duration','DayofWeek']].head(3) Yields: StartEndDurationDayofWeek 02015-08-262017-03-05557Wednesday 12014-10-152014-12-1965Wednesday 22015-02-222016-01-14326Sunday So,wenowknowthatbetweenthe26thofAugust2015andthe5thofMarch2017there are557days,andthatthe26thofAugust2015wasaWednesday. Ifyou’rewonderingwhatthepurposeofthisis,I’llprovideanexample.Imaginethatyou haveacampaignthatistiedtoasportseventthatusuallytakesplaceonaSunday.You maywanttoinspectyourdataaccordingtothedayssothatyoucancorrelatethemtothe variousmeasurementsyouhave.We’renotgoingtodoitinthisproject,butitwasuseful tosee,ifonlyforthedifferentwayofcallingapply()onaDataFrame. Cleaningeverythingup Nowthatwehaveeverythingwewant,it’stimetodothefinalcleaning:rememberwestill havethe'cmp_name'and'user'columns.Thoseareuselessnow,sotheyhavetogo. Also,IwanttoreorderthecolumnsintheDataFramesothatitismorerelevanttothedata itnowcontains.Inordertodothis,wejustneedtofilterdfonthecolumnlistwewant. We’llgetbackabrandnewDataFramethatwecanreassigntodfitself. #27 final_columns=[ 'Type','Start','End','Duration','DayofWeek','Budget', 'Currency','Clicks','Impressions','Spent','CTR','CPC', 'CPI','TargetAge','TargetGender','Username','Email', 'Name','Gender','Age' ] df=df[final_columns] Ihavegroupedthecampaigninformationatthebeginning,thenthemeasurements,and finallytheuserdataattheend.NowourDataFrameiscleanandreadyforustoinspect. Beforewestartgoingcrazywithgraphs,whatabouttakingasnapshotofourDataFrame sothatwecaneasilyreconstructitfromafile,ratherthanhavingtoredoallthestepswe didtogethere.Someanalystsmaywanttohaveitinspreadsheetform,todoadifferent kindofanalysisthantheonewewanttodo,solet’sseehowtosaveaDataFrametoafile. WOW! eBook www.wowebook.org It’seasierdonethansaid. WOW! eBook www.wowebook.org SavingtheDataFrametoafile WecansaveaDataFrameinmanydifferentways.Youcantypedf.to_andthenpress Tabtomakeauto-completionpopup,toseeallthepossibleoptions. We’regoingtosaveourDataFrameinthreedifferentformats,justforfun:commaseparatedvalues(CSV),JSON,andExcelspreadsheet. #28 df.to_csv('df.csv') #29 df.to_json('df.json') #30 df.to_excel('df.xls') TheCSVfilelookslikethis(outputtruncated): Type,Start,End,Duration,DayofWeek,Budget,Currency,Clicks,Impres 0,GRZ,2015-03-15,2015-11-10,240,Sunday,622551,GBP,35018,500002,787 1,AKX,2016-06-19,2016-09-19,92,Sunday,148219,EUR,45185,499997,6588 2,BYU,2014-09-25,2016-07-03,647,Thursday,537760,GBP,55771,500001,3 AndtheJSONonelikethis(again,outputtruncated): { "Type":{ "0":"GRZ", "1":"AKX", "2":"BYU", So,it’sextremelyeasytosaveaDataFrameinmanydifferentformats,andthegoodnews isthattheoppositeisalsotrue:it’sveryeasytoloadaspreadsheetintoaDataFrame.The programmersbehindPandaswentalongwaytoeaseourtasks,somethingtobegrateful for. WOW! eBook www.wowebook.org Visualizingtheresults Finally,thejuicybits.Inthissection,we’regoingtovisualizesomeresults.Fromadata scienceperspective,I’mnotveryinterestedingoingdeepintoanalysis,especiallybecause thedataiscompletelyrandom,butnonetheless,thiscodewillgetyoustartedwithgraphs andotherfeatures. SomethingIlearnedinmylife—andthismaycomeasasurprisetoyou—isthatlooks alsocountssoit’sveryimportantthatwhenyoupresentyourresults,youdoyourbestto makethempretty. Iwon’ttrytoprovetoyouhowtruthfulthatlaststatementis,butIreallydobelieveinit. Ifyourecallthelastlineofcell#1: #makethegraphsnicer pd.set_option('display.mpl_style','default') Itspurposeistomakethegraphswewilllookatinthissectionalittlebitprettier. Okay,so,firstofallwehavetoinstructthenotebookthatwewanttousematplotlib inline.ThismeansthatwhenweaskPandastoplotsomething,wewillhavetheresult renderedinthecelloutputframe.Inordertodothis,wejustneedonesimpleinstruction: #31 %matplotlibinline Youcanalsoinstructthenotebooktodothiswhenyoustartitfromtheconsolebypassing aparameter,butIwantedtoshowyouthiswaytoo,sinceitcanbeannoyingtohaveto restartthenotebookjustbecauseyouwanttoplotsomething.Inthisway,youcandoiton theflyandthenkeepworking. Next,we’regoingtosetsomeparametersonpylab.Thisisforplottingpurposesandit willremoveawarningthatafonthasn’tbeenfound.Isuggestthatyoudonotexecutethis lineandkeepgoing.Ifyougetawarningthatafontismissing,comebacktothiscelland runit. #32 importpylab pylab.rcParams.update({'font.family':'serif'}) ThisbasicallytellsPylabtousethefirstavailableseriffont.Itissimplebuteffective,and youcanexperimentwithotherfontstoo. NowthattheDataFrameiscomplete,let’srundf.describe()(#33)again.Theresults shouldlooksomethinglikethis: WOW! eBook www.wowebook.org Thiskindofquickresultisperfecttosatisfythosemanagerswhohave20secondsto dedicatetoyouandjustwantroughnumbers. Note Onceagain,pleasekeepinmindthatourcampaignshavedifferentcurrencies,sothese numbersareactuallymeaningless.ThepointhereistodemonstratetheDataFrame capabilities,nottogettoacorrectordetailedanalysisofrealdata. Alternatively,agraphisusuallymuchbetterthanatablewithnumbersbecauseit’smuch easiertoreaditanditgivesyouimmediatefeedback.So,let’sgraphoutthefourpiecesof informationwehaveoneachcampaign:budget,spent,clicks,andimpressions. #34 df[['Budget','Spent','Clicks','Impressions']].hist( bins=16,figsize=(16,6)); Weextrapolatethosefourcolumns(thiswillgiveusanotherDataFramemadewithonly thosecolumns)andcallthehistogramhist()methodonit.Wegivesomemeasurements onthebinsandfiguresizes,butbasicallyeverythingisdoneautomatically. Oneimportantthing:sincethisinstructionistheonlyoneinthiscell(whichalsomeans, it’sthelastone),thenotebookwillprintitsresultbeforedrawingthegraph.Tosuppress thisbehaviorandhaveonlythegraphdrawnwithnoprinting,justaddasemicolonatthe end(youthoughtIwasreminiscingaboutJava,didn’tyou?).Herearethegraphs: WOW! eBook www.wowebook.org Theyarebeautiful,aren’tthey?Didyounoticetheseriffont?Howaboutthemeaningof thosefigures?Ifyougobackto#6andtakealookatthewaywegeneratethedata,you willseethatallthesegraphsmakeperfectsense. Budgetissimplyarandomintegerinaninterval,thereforewewereexpectingauniform distribution,andtherewehaveit;it’spracticallyaconstantline. Spentisauniformdistributionaswell,butthehighendofitsintervalisthebudget,which ismoving,thismeansweshouldexpectsomethinglikeaquadratichyperbolethat decreasestotheright.Andthereitisaswell. Clickswasgeneratedwithatriangulardistributionwithmeanroughly20%oftheinterval size,andyoucanseethatthepeakisrightthere,atabout20%totheleft. Finally,ImpressionswasaGaussiandistribution,whichistheonethatassumesthe famousbellshape.Themeanwasexactlyinthemiddleandwehadstandarddeviationof 2.Youcanseethatthegraphmatchesthoseparameters. Good!Let’splotoutthemeasureswecalculated: #35 df[['CTR','CPC','CPI']].hist( bins=20,figsize=(16,6)); WOW! eBook www.wowebook.org Wecanseethatthecostperclickishighlyskewedtotheleft,meaningthatmostofthe CPCvaluesareverylow.Thecostperimpressionhasasimilarshape,butlessextreme. Now,allthisisnice,butifyouwantedtoanalyzeonlyaparticularsegmentofthedata, howwouldyoudoit?WecanapplyamasktoaDataFrame,sothatwegetanotherone withonlytherowsthatsatisfythemaskcondition.It’slikeapplyingaglobalrow-wiseif clause. #36 mask=(df.Spent>0.75*df.Budget) df[mask][['Budget','Spent','Clicks','Impressions']].hist( bins=15,figsize=(16,6),color='g'); Inthiscase,Ipreparedamasktofilteroutalltherowsforwhichthespentislessthanor equalto75%ofthebudget.Inotherwords,we’llincludeonlythosecampaignsforwhich wehavespentatleastthreequartersofthebudget.NoticethatinthemaskIamshowing youanalternativewayofaskingforaDataFramecolumn,byusingdirectpropertyaccess (object.property_name),insteadofdict-likeaccess(object['property_name']).If property_nameisavalidPythonname,youcanusebothwaysinterchangeably (JavaScriptworkslikethisaswell). Themaskisappliedinthesamewaythatweaccessadictwithakey.Whenyouapplya masktoaDataFrame,yougetbackanotheroneandweselectonlytherelevantcolumns onthis,andcallhist()again.Thistime,justforfun,wewanttheresultstobepainted green: WOW! eBook www.wowebook.org Notethattheshapesofthegraphshaven’tchangedmuch,apartfromthespent,whichis quitedifferent.Thereasonforthisisthatwe’veaskedonlyfortherowswherespentisat least75%ofthebudget.Thismeansthatwe’reincludingonlytherowswherespentis closetothebudget.Thebudgetnumberscomefromauniformdistribution.Therefore,itis quiteobviousthatthespentisnowassumingthatkindofshape.Ifyoumaketheboundary eventighter,andaskfor85%ormore,you’llseespentbecomemoreandmorelike budget. Let’snowaskforsomethingdifferent.Howaboutthemeasureofspent,click,and impressionsgroupedbydayoftheweek? #37 df_weekday=df.groupby(['DayofWeek']).sum() df_weekday[['Impressions','Spent','Clicks']].plot( figsize=(16,6),subplots=True); ThefirstlinecreatesanewDataFrame,df_weekday,byaskingforagroupingby'Dayof Week'ondf.Thefunctionusedtoaggregatethedataisaddition. Thesecondlinegetsasliceofdf_weekdayusingalistofcolumnnames,somethingwe’re accustomedtobynow.Ontheresultwecallplot(),whichisabitdifferenttohist(). Theoptionsubplots=Truemakesplotdrawthreeindependentgraphs: WOW! eBook www.wowebook.org Interestinglyenough,wecanseethatmostoftheactionhappensonThursdays.Ifthis weremeaningfuldata,thiswouldpotentiallybeimportantinformationtogivetoour clients,andthisisthereasonI’mshowingyouthisexample. Notethatthedaysaresortedalphabetically,whichscramblesthemupabit.Canyouthink ofaquicksolutionthatwouldfixtheissue?I’llleaveittoyouasanexercisetocomeup withsomething. Let’sfinishthispresentationsectionwithacouplemorethings.First,asimple aggregation.Wewanttoaggregateon'TargetGender'and'TargetAge',andshow 'Impressions'and'Spent'.Forboth,wewanttoseethemeanandthestandard deviation. #38 agg_config={ 'Impressions':{ 'MeanImpr':'mean', 'StdImpr':'std', }, 'Spent':['mean','std'], } df.groupby(['TargetGender','TargetAge']).agg(agg_config) It’sveryeasytodoit.Wewillprepareadictionarythatwe’lluseasaconfiguration.I’m showingyoutwooptionstodoit.Weuseanicerformatfor'Impressions',wherewe passanesteddictwithdescription/functionaskey/valuepairs.Ontheotherhand,for 'Spent',wejustuseasimplerlistwithjustthefunctionnames. Then,weperformagroupingonthe'TargetGender'and'TargetAge'columns,and wepassourconfigurationdicttotheagg()method.Theresultistruncatedandrearranged alittlebittomakeitfit,andshownhere: ImpressionsSpent WOW! eBook www.wowebook.org MeanImprStdImprmeanstd TargetTarget GenderAge B20-255000002.189102239882209442.168488 20-305000002.245317271285236854.155720 20-355000001.886396243725174268.898935 20-404999992.100786247740211540.133771 20-455000001.772811148712118603.932051 ............... M20-255000002.022023212520215857.323228 20-305000002.111882292577231663.713956 20-354999991.965177255651222790.960907 20-404999991.932473282515250023.393334 20-454999991.905746271077219901.462405 Thisisthetextualrepresentation,ofcourse,butyoucanalsohavetheHTMLone.You canseethatSpenthasthemeanandstdcolumnswhoselabelsaresimplythefunction names,whileImpressionsfeaturesthenicetitlesweaddedtotheconfigurationdict. Let’sdoonemorethingbeforewewrapthischapterup.Iwanttoshowyousomething calledapivottable.It’skindofabuzzwordinthedataenvironment,soanexamplesuch asthisone,albeitverysimple,isamust. #39 pivot=df.pivot_table( values=['Impressions','Clicks','Spent'], index=['TargetAge'], columns=['TargetGender'], aggfunc=np.sum ) pivot Wecreateapivottablethatshowsusthecorrelationbetweenthetargetageand impressions,clicks,andspent.Theselastthreewillbesubdividedaccordingtothetarget gender.Theaggregationfunctionusedtocalculatetheresultsisthenumpy.sumfunction (numpy.meanwouldbethedefault,hadInotspecifiedanything). Aftercreatingthepivottable,wesimplyprintitwiththelastlineinthecell,andhere’sa cropoftheresult: WOW! eBook www.wowebook.org It’sprettyclearandprovidesveryusefulinformationwhenthedataismeaningful. That’sit!I’llleaveyoutodiscovermoreaboutthewonderfulworldofIPython,Jupyter, anddatascience.Istronglyencourageyoutogetcomfortablewiththenotebook environment.It’smuchbetterthanaconsole,it’sextremelypracticalandfuntouse,and youcanevendoslidesanddocumentswithit. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Wheredowegofromhere? Datascienceisindeedafascinatingsubject.AsIsaidintheintroduction,thosewhowant todelveintoitsmeandersneedtobewelltrainedinmathematicsandstatistics.Working withdatathathasbeeninterpolatedincorrectlyrendersanyresultaboutituseless.The samegoesfordatathathasbeenextrapolatedincorrectlyorsampledwiththewrong frequency.Togiveyouanexample,imagineapopulationofindividualsthatarealignedin aqueue.If,forsomereason,thegenderofthatpopulationalternatedbetweenmaleand female,thequeuewouldbesomethinglikethis:F-M-F-M-F-M-F-M-F… Ifyousampledittakingonlytheevenelements,youwoulddrawtheconclusionthatthe populationwasmadeuponlyofmales,whilesamplingtheoddoneswouldtellyou exactlytheopposite. Ofcourse,thiswasjustasillyexample,Iknow,butbelievemeit’sveryeasytomake mistakesinthisfield,especiallywhendealingwithbigdatawheresamplingismandatory andtherefore,thequalityoftheintrospectionyoumakedepends,firstandforemost,onthe qualityofthesamplingitself. WhenitcomestodatascienceandPython,thesearethemaintoolsyouwanttolookat: NumPy(http://www.numpy.org/):Thisisthefundamentalpackageforscientific computingwithPython.ItcontainsapowerfulN-dimensionalarrayobject, sophisticated(broadcasting)functions,toolsforintegratingC/C++andFortrancode, usefullinearalgebra,Fouriertransform,randomnumbercapabilities,andmuch more. Scikit-Learn(http://scikit-learn.org/stable/):Thisisprobablythemostpopular machinelearninglibraryinPython.Ithassimpleandefficienttoolsfordatamining anddataanalysis,accessibletoeverybody,andreusableinvariouscontexts.It’sbuilt onNumPy,SciPy,andMatplotlib. Pandas(http://pandas.pydata.org/):Thisisanopensource,BSD-licensedlibrary providinghigh-performance,easy-to-usedatastructures,anddataanalysistools. We’veuseditthroughoutthiswholechapter. IPython(http://ipython.org/)/Jupyter(http://jupyter.org/):Theseprovidearich architectureforinteractivecomputing. Matplotlib(http://matplotlib.org/):ThisisaPython2Dplottinglibrarythatproduces publication-qualityfiguresinavarietyofhardcopyformatsandinteractive environmentsacrossplatforms.MatplotlibcanbeusedinPythonscripts,thePython andIPythonshellandnotebook,webapplicationservers,andsixgraphicaluser interfacetoolkits. Numba(http://numba.pydata.org/):Thisgivesyouthepowertospeedupyour applicationswithhighperformancefunctionswrittendirectlyinPython.Withafew annotations,array-orientedandmath-heavyPythoncodecanbejust-in-time compiledtonativemachineinstructions,similarinperformancetoC,C++,and Fortran,withouthavingtoswitchlanguagesorPythoninterpreters. Bokeh(http://bokeh.pydata.org/en/latest/):It’saPython-interactivevisualization WOW! eBook www.wowebook.org librarythattargetsmodernwebbrowsersforpresentation.Itsgoalistoprovide elegant,conciseconstructionofnovelgraphicsinthestyleofD3.js,butalsodeliver thiscapabilitywithhigh-performanceinteractivityoververylargeorstreaming datasets. Otherthanthesesinglelibraries,youcanalsofindecosystemssuchasSciPy (http://scipy.org/)andAnaconda(https://www.continuum.io/),whichbundleseveral differentpackagesinordertogiveyousomethingthatjustworksinan“out-of-the-box” fashion. Installingallthesetoolsandtheirseveraldependenciesishardonsomesystems,soI suggestthatyoutryoutecosystemsaswellandseeifyouarecomfortablewiththem.It maybeworthit. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,wetalkedaboutdatascience.Ratherthanattemptingtoexplainanything aboutthisextremelywidesubject,wedelvedintoaproject.Wefamiliarizedourselves withtheJupyternotebook,andwithdifferentlibrariessuchasPandas,Matplotlib,NumPy. Ofcourse,havingtocompressallthisinformationintoonesinglechaptermeansIcould onlytouchbrieflyonthesubjectsIpresented.Ihopetheprojectwe’vegonethrough togetherhasbeencomprehensiveenoughtogiveyouagoodideaaboutwhatcould potentiallybetheworkflowyoumightfollowwhenworkinginthisfield. Thenextchapterisdedicatedtowebdevelopment.So,makesureyouhaveabrowser readyandlet’sgo! WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter10.WebDevelopmentDone Right “Don’tbelieveeverythingyoureadontheWeb.” —Confucius Inthischapter,we’regoingtoworkonawebsitetogether.Byworkingonasmallproject, myaimistoopenawindowforyoutotakeapeekonwhatwebdevelopmentis,along withthemainconceptsandtoolsyoushouldknowifyouwanttobesuccessfulwithit. WOW! eBook www.wowebook.org WhatistheWeb? TheWorldWideWeb,orsimplyWeb,isawayofaccessinginformationthroughtheuse ofamediumcalledtheInternet.TheInternetisahugenetworkofnetworks,a networkinginfrastructure.Itspurposeistoconnectbillionsofdevicestogether,allaround theglobe,sothattheycancommunicatewithoneanother.Informationtravelsthroughthe Internetinarichvarietyoflanguagescalledprotocols,whichallowdifferentdevicesto speakthesametongueinordertosharecontent. TheWebisaninformation-sharingmodel,builtontopoftheInternet,whichemploysthe HypertextTransferProtocol(HTTP)asabasisfordatacommunication.TheWeb, therefore,isjustoneoftheseveraldifferentwaysinformationcanbeexchangedoverthe Internet:e-mail,instantmessaging,newsgroups,andsoon,theyallrelyondifferent protocols. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org HowdoestheWebwork? Inanutshell,HTTPisanasymmetricrequest-responseclient-serverprotocol.AnHTTP clientsendsarequestmessagetoanHTTPserver.Theserver,inturn,returnsaresponse message.Inotherwords,HTTPisapullprotocolinwhichtheclientpullsinformation fromtheserver(asopposedtoapushprotocolinwhichtheserverpushesinformation downtotheclient).Takealookatthefollowingimage: HTTPisbasedonTCP/IP(TransmissionControlProtocol/InternetProtocol),which providesthetoolsforareliablecommunicationexchange. AnimportantfeatureoftheHTTPprotocolisthatit’sstateless.Thismeansthatthe currentrequesthasnoknowledgeaboutwhathappenedinpreviousrequests.Thisisa limitation,butyoucanbrowseawebsitewiththeillusionofbeingloggedin.Underthe coversthough,whathappensisthat,onlogin,atokenofuserinformationissaved(most oftenontheclientside,inspecialfilescalledcookies)sothateachrequesttheusermakes carriesthemeansfortheservertorecognizetheuserandprovideacustominterfaceby showingtheirname,keepingtheirbasketpopulated,andsoon. Eventhoughit’sveryinteresting,we’renotgoingtodelveintotherichdetailsofHTTP andhowitworks.However,we’regoingtowriteasmallwebsite,whichmeanswe’llhave towritethecodetohandleHTTPrequestsandreturnHTTPresponses.Iwon’tkeep prependingHTTPtothetermsrequestandresponsefromnowon,asItrusttherewon’tbe anyconfusion. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org TheDjangowebframework Forourproject,we’regoingtouseoneofthemostpopularwebframeworksyoucanfind inthePythonecosystem:Django. Awebframeworkisasetoftools(libraries,functions,classes,andsoon)thatwecanuse tocodeawebsite.Weneedtodecidewhatkindofrequestswewanttoallowtobeissued againstourwebserverandhowwerespondtothem.Awebframeworkistheperfecttool todothatbecauseittakescareofmanythingsforussothatwecanconcentrateonlyon theimportantbitswithouthavingtoreinventthewheel. Note Therearedifferenttypesofframeworks.Notallofthemaredesignedforwritingcodefor theweb.Ingeneral,aframeworkisatoolthatprovidesfunctionalitiestofacilitatethe developmentofsoftwareapplications,productsandsolutions. WOW! eBook www.wowebook.org Djangodesignphilosophy Djangoisdesignedaccordingtothefollowingprinciples: DRY:Asin,Don’tRepeatYourself.Don’trepeatcode,andcodeinawaythat makestheframeworkdeduceasmuchaspossiblefromaslittleaspossible. Loosecoupling:Thevariouslayersoftheframeworkshouldn’tknowabouteach other(unlessabsolutelynecessaryforwhateverreason).Loosecouplingworksbest whenparalleledwithhighcohesion.ToquoteRobertMartin:puttingtogetherthings whichchangeforthesamereason,andspreadingapartthosewhichchangefor differentreasons. Lesscode:Applicationsshouldusetheleastpossibleamountofcode,andbewritten inawaythatfavorsreuseasmuchaspossible. Consistency:WhenusingtheDjangoframework,regardlessofwhichlayeryou’re codingagainst,yourexperiencewillbeveryconsistentwiththedesignpatternsand paradigmsthatwerechosentolayouttheproject. Theframeworkitselfisdesignedaroundthemodel-template-view(MTV)pattern,which isavariantofmodel-view-controller(MVC),whichiswidelyemployedbyother frameworks.Thepurposeofsuchpatternsistoseparateconcernsandpromotecodereuse andquality. Themodellayer Ofthethreelayers,thisistheonethatdefinesthestructureofthedatathatishandledby theapplication,anddealswithdatasources.Amodelisaclassthatrepresentsadata structure.ThroughsomeDjangomagic,modelsaremappedtodatabasetablessothatyou canstoreyourdatainarelationaldatabase. Note Arelationaldatabasestoresdataintablesinwhicheachcolumnisapropertyofthedata andeachrowrepresentsasingleitemorentryinthecollectionrepresentedbythattable. Throughtheprimarykeyofeachtable,whichisthatpartofthedatathatallowsto uniquelyidentifyeachitem,itispossibletoestablishrelationshipsbetweenitems belongingtodifferenttables,thatis,toputthemintorelation. Thebeautyofthissystemisthatyoudon’thavetowritedatabase-specificcodeinorderto handleyourdata.Youwilljusthavetoconfigureyourmodelscorrectlyandsimplyuse them.TheworkonthedatabaseisdoneforyoubytheDjangoobject-relationalmapping (ORM),whichtakescareoftranslatingoperationsdoneonPythonobjectsintoalanguage thatarelationaldatabasecanunderstand:SQL(StructuredQueryLanguage). Onebenefitofthisapproachisthatyouwillbeabletochangedatabaseswithoutrewriting yourcodesinceallthedatabasespecificcodeisproducedbyDjangoonthefly,according towhichdatabaseit’sconnectedto.RelationaldatabasesspeakSQL,buteachofthemhas itsownuniqueflavorofit;therefore,nothavingtohardcodeanySQLinourapplicationis atremendousadvantage. WOW! eBook www.wowebook.org Djangoallowsyoutomodifyyourmodelsatanytime.Whenyoudo,youcanruna commandthatcreatesamigration,whichisthesetofinstructionsneededtoportthe databaseinastatethatrepresentsthecurrentdefinitionofyourmodels. Tosummarize,thislayerdealswithdefiningthedatastructuresyouneedtohandleinyour websiteandgivesyouthemeanstosaveandloadthemfromandtothedatabaseby simplyaccessingthemodels,whicharePythonobjects. Theviewlayer Thefunctionofaviewishandlingarequest,performingwhateveractionneedstobe carriedout,andeventuallyreturningaresponse.Forexample,ifyouopenyourbrowser andrequestapagecorrespondingtoacategoryofproductsinane-commerceshop,the viewwilllikelytalktothedatabase,askingforallthecategoriesthatarechildrenofthe selectedcategory(forexample,todisplaytheminanavigationsidebar)andforallthe productsthatbelongtotheselectedcategory,inordertodisplaythemonthepage. Therefore,theviewisthemechanismthroughwhichwecanfulfillarequest.Itsresult,the responseobject,canassumeseveraldifferentforms:aJSONpayload,text,anHTML page,andsoon.Whenyoucodeawebsite,yourresponsesusuallyconsistofHTMLor JSON. Note TheHypertextMarkupLanguage,orHTML,isthestandardmarkuplanguageusedto createwebpages.WebbrowsersrunenginesthatarecapableofinterpretingHTMLcode andrenderitintowhatweseewhenweopenapageofawebsite. Thetemplatelayer Thisisthelayerthatprovidesthebridgebetweenbackendandfrontenddevelopment. WhenaviewhastoreturnHTML,itusuallydoesitbypreparingacontextobject(adict) withsomedata,andthenitfeedsthiscontexttoatemplate,whichisrendered(thatisto say,transformedintoHTML)andreturnedtothecallerintheformofaresponse(more precisely,thebodyoftheresponse).Thismechanismallowsformaximumcodereuse.If yougobacktothecategoryexample,it’seasytoseethat,ifyoubrowseawebsitethat sellsproducts,itdoesn’treallymatterwhichcategoryyouclickonorwhattypeofsearch youperform,thelayoutoftheproductspagedoesn’tchange.Whatdoeschangeisthedata withwhichthatpageispopulated. Therefore,thelayoutofthepageisdefinedbyatemplate,whichiswritteninamixtureof HTMLandDjangotemplatelanguage.Theviewthatservesthatpagecollectsallthe productstobedisplayedinthecontextdict,andfeedsittothetemplatewhichwillbe renderedintoanHTMLpagebytheDjangotemplateengine. WOW! eBook www.wowebook.org TheDjangoURLdispatcher ThewayDjangoassociatesaUniformResourceLocator(URL)withaviewisthrough matchingtherequestedURLwiththepatternsthatareregisteredinaspecialfile.AURL representsapageinawebsiteso,forexample,http://mysite.com/categories?id=123 wouldprobablypointtothepageforthecategorywithID123onmywebsite,while https://mysite.com/loginwouldprobablybetheuserloginpage. Tip ThedifferencebetweenHTTPandHTTPSisthatthelatteraddsencryptiontotheprotocol sothatthedatathatyouexchangewiththewebsiteissecured.Whenyouputyourcredit carddetailsonawebsite,orloginanywhere,ordoanythingaroundsensitivedata,you wanttomakesurethatyou’reusingHTTPS. Regularexpressions ThewayDjangomatchesURLstopatternsisthrougharegularexpression.Aregular expressionisasequenceofcharactersthatdefinesasearchpatternwithwhichwecan carryoutoperationssuchaspatternandstringmatching,find/replace,andsoon. Regularexpressionshaveaspecialsyntaxtoindicatethingslikedigits,letters,spaces,and soon,aswellashowmanytimesweexpectacharactertoappear,andmuchmore.A completeexplanationofthistopicisbeyondofthescopeofthebook.However,itisa veryimportanttopic,sotheprojectwe’regoingtoworkontogetherwillevolvearoundit, inthehopethatyouwillbestimulatedtofindthetimetoexploreitabitmoreonyour own. Togiveyouaquickexample,imaginethatyouwantedtospecifyapatterntomatchadate suchas"26-12-1947".Thisstringconsistsoftwodigits,onedash,twodigits,onedash, andfinallyfourdigits.Therefore,wecouldwriteitlikethis:r'[0-9]{2}-[0-9]{2}-[0-9] {4}'.Wecreatedaclassbyusingsquarebrackets,andwedefinedarangeofdigitsinside, from0to9,henceallthepossibledigits.Then,betweencurlybraces,wesaythatwe expecttwoofthem.Thenadash,thenwerepeatthispatternonceasitis,andoncemore, bychanginghowmanydigitsweexpect,andwithoutthefinaldash.Havingaclasslike [0-9]issuchacommonpatternthataspecialnotationhasbeencreatedasashortcut: '\d'.Therefore,wecanrewritethepatternlikethis:r'\d{2}-\d{2}-\d{4}'anditwill workexactlythesame.Thatrinfrontofthestringstandsforraw,anditspurposeisto alterthewayeverybackslash'\'isinterpretedbytheregularexpressionengine. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Aregexwebsite So,hereweare.We’llcodeawebsitethatstoresregularexpressionssothatwe’llbeable toplaywiththemalittlebit. Note Beforeweproceedcreatingtheproject,I’dliketospendawordaboutCSS.CSS (CascadingStyleSheets)arefilesinwhichwespecifyhowthevariouselementsonan HTMLpagelook.Youcansetallsortsofpropertiessuchasshape,size,color,margins, borders,fonts,andsoon.Inthisproject,Ihavetriedmybesttoachieveadecentresulton thepages,butI’mneitherafrontenddevelopernoradesigner,sopleasedon’tpaytoo muchattentiontohowthingslook.Tryandfocusonhowtheywork. WOW! eBook www.wowebook.org SettingupDjango OntheDjangowebsite(https://www.djangoproject.com/),youcanfollowthetutorial, whichgivesyouaprettygoodideaofDjango’scapabilities.Ifyouwant,youcanfollow thattutorialfirstandthencomebacktothisexample.So,firstthingsfirst;let’sinstall Djangoinyourvirtualenvironment: $pipinstalldjango Whenthiscommandisdone,youcantestitwithinaconsole(trydoingitwithbpython,it givesyouashellsimilartoIPythonbutwithniceintrospectioncapabilities): >>>importdjango >>>django.VERSION (1,8,4,'final',0) NowthatDjangoisinstalled,we’regoodtogo.We’llhavetodosomescaffolding,soI’ll quicklyguideyouthroughthat. Startingtheproject Chooseafolderinthebook’senvironmentandchangeintothat.I’llusech10.Fromthere, westartaDjangoprojectwiththefollowingcommand: $django-adminstartprojectregex ThiswillpreparetheskeletonforaDjangoprojectcalledregex.Changeintotheregex folderandrunthefollowing: $pythonmanage.pyrunserver Youshouldbeabletogotohttp://127.0.0.1:8000/withyourbrowserandseetheIt worked!defaultDjangopage.Thismeansthattheprojectiscorrectlysetup.Whenyou’ve seenthepage,killtheserverwithCtrl+C(orwhateveritsaysintheconsole).I’llpaste thefinalstructurefortheprojectnowsothatyoucanuseitasareference: $tree-Aregex#fromthech10folder regex ├──db.sqlite3 ├──entries │├──admin.py │├──forms.py │├──__init__.py │├──migrations ││├──0001_initial.py ││└──__init__.py │├──models.py │├──static ││└──entries ││└──css ││└──main.css │├──templates ││└──entries ││├──base.html WOW! eBook www.wowebook.org ││├──footer.html ││├──home.html ││├──insert.html ││└──list.html │└──views.py ├──manage.py └──regex ├──__init__.py ├──settings.py ├──urls.py └──wsgi.py Don’tworryifyou’remissingfiles,we’llgetthere.ADjangoprojectistypicallya collectionofseveraldifferentapplications.Eachapplicationismeanttoprovidea functionalityinaself-contained,reusablefashion.We’llcreatejustone,calledentries: $pythonmanage.pystartappentries Withintheentriesfolderthathasbeencreated,youcangetridofthetests.pymodule. Now,let’sfixtheregex/settings.pyfileintheregexfolder.Weneedtoaddour applicationtotheINSTALLED_APPStuplesothatwecanuseit(additatthebottomofthe tuple): INSTALLED_APPS=( ...djangoapps… 'entries', ) Then,youmaywanttofixthelanguageandtimezoneaccordingtoyourpersonal preference.IliveinLondon,soIsetthemlikethis: LANGUAGE_CODE='en-gb' TIME_ZONE='Europe/London' Thereisnothingelsetodointhisfile,soyoucansaveandcloseit. Nowit’stimetoapplythemigrationstothedatabase.Djangoneedsdatabasesupportto handleusers,sessions,andthingslikethat,soweneedtocreateadatabaseandpopulateit withthenecessarydata.Luckily,thisisveryeasilydonewiththefollowingcommand: $pythonmanage.pymigrate Note Forthisproject,weuseaSQLitedatabase,whichisbasicallyjustafile.Onarealproject, youwouldprobablyuseadifferentdatabaseenginelikeMySQLorPostgreSQL. Creatingusers Nowthatwehaveadatabase,wecancreateasuperuserusingtheconsole. $pythonmanage.pycreatesuperuser Afterenteringusernameandotherdetails,wehaveauserwithadminprivileges.Thisis enoughtoaccesstheDjangoadminsection,sotryandstarttheserver: WOW! eBook www.wowebook.org $pythonmanage.pyrunserver ThiswillstarttheDjangodevelopmentserver,whichisaveryusefulbuilt-inwebserver thatyoucanusewhileworkingwithDjango.Nowthattheserverisrunning,wecan accesstheadminpageathttp://localhost:8000/admin/.Iwillshowyouascreenshot ofthissectionlater.Ifyouloginwiththecredentialsoftheuseryoujustcreatedandhead totheAuthenticationandAuthorizationsection,you’llfindUsers.Openthatandyou willbeabletoseethelistofusers.Youcaneditthedetailsofanyuseryouwantasan admin.Inourcase,makesureyoucreateadifferentonesothatthereareatleasttwousers inthesystem(we’llneedthemlater).I’llcallthefirstuserFabrizio(username:fab)and thesecondoneAdriano(username:adri)inhonorofmyfather. Bytheway,youshouldseethattheDjangoadminpanelcomesforfreeautomatically.You defineyourmodels,hookthemup,andthat’sit.Thisisanincredibletoolthatshowshow advancedDjango’sintrospectioncapabilitiesare.Moreover,itiscompletelycustomizable andextendable.It’strulyanexcellentpieceofwork. WOW! eBook www.wowebook.org AddingtheEntrymodel Nowthattheboilerplateisoutoftheway,andwehaveacoupleofusers,we’rereadyto code.WestartbyaddingtheEntrymodeltoourapplicationsothatwecanstoreobjectsin thedatabase.Here’sthecodeyou’llneedtoadd(remembertousetheprojecttreefor reference): entries/models.py fromdjango.dbimportmodels fromdjango.contrib.auth.modelsimportUser fromdjango.utilsimporttimezone classEntry(models.Model): user=models.ForeignKey(User) pattern=models.CharField(max_length=255) test_string=models.CharField(max_length=255) date_added=models.DateTimeField(default=timezone.now) classMeta: verbose_name_plural='entries' Thisisthemodelwe’llusetostoreregularexpressionsinoursystem.We’llstorea pattern,ateststring,areferencetotheuserwhocreatedtheentry,andthemomentof creation.Youcanseethatcreatingamodelisactuallyquiteeasy,butnonetheless,let’sgo throughitlinebyline. Firstweneedtoimportthemodelsmodulefromdjango.db.Thiswillgiveusthebase classforourEntrymodel.Djangomodelsarespecialclassesandmuchisdoneforus behindthesceneswhenweinheritfrommodels.Model. Wewantareferencetotheuserwhocreatedtheentry,soweneedtoimporttheUser modelfromDjango’sauthorizationapplicationandwealsoneedtoimportthetimezone modeltogetaccesstothetimezone.now()function,whichprovidesuswithatimezoneawareversionofdatetime.now().Thebeautyofthisisthatit’shookedupwiththe TIME_ZONEsettingsIshowedyoubefore. Asfortheprimarykeyforthisclass,ifwedon’tsetoneexplicitly,Djangowilladdonefor us.AprimarykeyisakeythatallowsustouniquelyidentifyanEntryobjectinthe database(inthiscase,Djangowilladdanauto-incrementingintegerID). So,wedefineourclass,andwesetupfourclassattributes.WehaveaForeignKey attributethatisourreferencetotheUsermodel.WealsohavetwoCharFieldattributes thatholdthepatternandteststringsforourregularexpressions.Wealsohavea DateTimeField,whosedefaultvalueissettotimezone.now.Notethatwedon’tcall timezone.nowrightthere,it’snow,notnow().So,we’renotpassingaDateTimeinstance (setatthemomentintimewhenthatlineisparsed)rather,we’repassingacallable,a functionthatiscalledwhenwesaveanentryinthedatabase.Thisissimilartothe callbackmechanismweusedinChapter8,TheEdges–GUIsandScripts,whenwewere assigningcommandstobuttonclicks. WOW! eBook www.wowebook.org Thelasttwolinesareveryinteresting.WedefineaclassMetawithintheEntryclassitself. TheMetaclassisusedbyDjangotoprovideallsortsofextrainformationforamodel. Djangohasagreatdealoflogicunderthehoodtoadaptitsbehavioraccordingtothe informationweputintheMetaclass.Inthiscase,intheadminpanel,thepluralized versionofEntrywouldbeEntrys,whichiswrong,thereforeweneedtomanuallysetit. Wespecifythepluralalllowercase,asDjangotakescareofcapitalizingitforuswhen needed. Nowthatwehaveanewmodel,weneedtoupdatethedatabasetoreflectthenewstateof thecode.Inordertodothis,weneedtoinstructDjangothatitneedstocreatethecodeto updatethedatabase.Thiscodeiscalledmigration.Let’screateitandexecuteit: $pythonmanage.pymakemigrationsentries $pythonmanage.pymigrate Afterthesetwoinstructions,thedatabasewillbereadytostoreEntryobjects. Note Therearetwodifferentkindsofmigrations:dataandschemamigration.Datamigrations portdatafromonestatetoanotherwithoutalteringitsstructure.Forexample,adata migrationcouldsetallproductsforacategoryasoutofstockbyswitchingaflagtoFalse or0.Aschemamigrationisasetofinstructionsthatalterthestructureofthedatabase schema.Forexample,thatcouldbeaddinganagecolumntoaPersontable,orincreasing themaximumlengthofafieldtoaccountforverylongaddresses.Whendevelopingwith Django,it’squitecommontohavetoperformbothkindsofmigrationsoverthecourseof development.Dataevolvescontinuously,especiallyifyoucodeinanagileenvironment. WOW! eBook www.wowebook.org Customizingtheadminpanel ThenextstepistohooktheEntrymodelupwiththeadminpanel.Youcandoitwithone lineofcode,butinthiscase,Iwanttoaddsomeoptionstocustomizeabitthewaythe adminpanelshowstheentries,bothinthelistviewofallentryitemsinthedatabaseand intheformviewthatallowsustocreateandmodifythem. Allweneedtodoistoaddthefollowingcode: entries/admin.py fromdjango.contribimportadmin from.modelsimportEntry @admin.register(Entry) classEntryAdmin(admin.ModelAdmin): fieldsets=[ ('RegularExpression', {'fields':['pattern','test_string']}), ('OtherInformation', {'fields':['user','date_added']}), ] list_display=('pattern','test_string','user') list_filter=['user'] search_fields=['test_string'] Thisissimplybeautiful.Myguessisthatyouprobablyalreadyunderstandmostofit,even ifyou’renewtoDjango. So,westartbyimportingtheadminmoduleandtheEntrymodel.Becausewewantto fostercodereuse,weimporttheEntrymodelusingarelativeimport(there’sadotbefore models).Thiswillallowustomoveorrenametheappwithouttoomuchtrouble.Then, wedefinetheEntryAdminclass,whichinheritsfromadmin.ModelAdmin.Thedecoration ontheclasstellsDjangotodisplaytheEntrymodelintheadminpanel,andwhatweput intheEntryAdminclasstellsDjangohowtocustomizethewayithandlesthismodel. Firstly,wespecifythefieldsetsforthecreate/editpage.Thiswilldividethepageinto twosectionssothatwegetabettervisualizationofthecontent(patternandteststring) andtheotherdetails(userandtimestamp)separately. Then,wecustomizethewaythelistpagedisplaystheresults.Wewanttoseeallthefields, butnotthedate.Wealsowanttobeabletofilterontheusersothatwecanhavealistof alltheentriesbyjustoneuser,andwewanttobeabletosearchontest_string. Iwillgoaheadandaddthreeentries,oneformyselfandtwoonbehalfofmyfather.The resultisshowninthenexttwoimages.Afterinsertingthem,thelistpagelookslikethis: WOW! eBook www.wowebook.org IhavehighlightedthethreepartsofthisviewthatwecustomizedintheEntryAdminclass. Wecanfilterbyuser,wecansearchandwehaveallthefieldsdisplayed.Ifyouclickona pattern,theeditviewopensup. Afterourcustomization,itlookslikethis: Noticehowwehavetwosections:RegularExpressionandOtherInformation,thanks toourcustomEntryAdminclass.Haveagowithit,addsomeentriestoacoupleof differentusers,getfamiliarwiththeinterface.Isn’titnicetohaveallthisforfree? WOW! eBook www.wowebook.org Creatingtheform Everytimeyoufillinyourdetailsonawebpage,you’reinsertingdatainformfields.A formisapartoftheHTMLDocumentObjectModel(DOM)tree.InHTML,you createaformbyusingtheformtag.Whenyouclickonthesubmitbutton,yourbrowser normallypackstheformdatatogetherandputsitinthebodyofaPOSTrequest.As opposedtoGETrequests,whichareusedtoaskthewebserverforaresource,aPOST requestnormallysendsdatatothewebserverwiththeaimofcreatingorupdatinga resource.Forthisreason,handlingPOSTrequestsusuallyrequiresmorecarethanGET requests. WhentheserverreceivesdatafromaPOSTrequest,thatdataneedstobevalidated. Moreover,theserverneedstoemploysecuritymechanismstoprotectagainstvarious typesofattacks.Oneattackthatisverydangerousisthecross-siterequestforgery (CSRF)attack,whichhappenswhendataissentfromadomainthatisnottheonethe userisauthenticatedon.Djangoallowsyoutohandlethisissueinaveryelegantway. So,insteadofbeinglazyandusingtheDjangoadmintocreatetheentries,I’mgoingto showyouhowtodoitusingaDjangoform.Byusingthetoolstheframeworkgivesyou, yougetaverygooddegreeofvalidationworkalreadydone(infact,wewon’tneedtoadd anycustomvalidationourselves). TherearetwokindsofformclassesinDjango:FormandModelForm.Youusetheformer tocreateaformwhoseshapeandbehaviordependsonhowyoucodetheclass,whatfields youadd,andsoon.Ontheotherhand,thelatterisatypeofformthat,albeitstill customizable,infersfieldsandbehaviorfromamodel.SinceweneedaformfortheEntry model,we’llusethatone. entries/forms.py fromdjango.formsimportModelForm from.modelsimportEntry classEntryForm(ModelForm): classMeta: model=Entry fields=['pattern','test_string'] Amazinglyenough,thisisallwehavetodotohaveaformthatwecanputonapage.The onlynotablethinghereisthatwerestrictthefieldstoonlypatternandtest_string. Onlylogged-inuserswillbeallowedaccesstotheinsertpage,andthereforewedon’t needtoaskwhotheuseris:weknowthat.Asforthedate,whenwesaveanEntry,the date_addedfieldwillbesetaccordingtoitsdefault,thereforewedon’tneedtospecify thataswell.We’llseeintheviewhowtofeedtheuserinformationtotheformbefore saving.So,allthebackgroundworkisdone,allweneedistheviewsandthetemplates. Let’sstartwiththeviews. WOW! eBook www.wowebook.org Writingtheviews Weneedtowritethreeviews.Weneedoneforthehomepage,onetodisplaythelistofall entriesforauser,andonetocreateanewentry.Wealsoneedviewstologinandlogout. ButthankstoDjango,wedon’tneedtowritethem.I’llpasteallthecode,andthenwe’ll gothroughittogether,stepbystep. entries/views.py importre fromdjango.contrib.auth.decoratorsimportlogin_required fromdjango.contrib.messages.viewsimportSuccessMessageMixin fromdjango.core.urlresolversimportreverse_lazy fromdjango.utils.decoratorsimportmethod_decorator fromdjango.views.genericimportFormView,TemplateView from.formsimportEntryForm from.modelsimportEntry classHomeView(TemplateView): template_name='entries/home.html' @method_decorator( login_required(login_url=reverse_lazy('login'))) defget(self,request,*args,**kwargs): context=self.get_context_data(**kwargs) returnself.render_to_response(context) classEntryListView(TemplateView): template_name='entries/list.html' @method_decorator( login_required(login_url=reverse_lazy('login'))) defget(self,request,*args,**kwargs): context=self.get_context_data(**kwargs) entries=Entry.objects.filter( user=request.user).order_by('-date_added') matches=(self._parse_entry(entry)forentryinentries) context['entries']=list(zip(entries,matches)) returnself.render_to_response(context) def_parse_entry(self,entry): match=re.search(entry.pattern,entry.test_string) ifmatchisnotNone: return( match.group(), match.groups()orNone, match.groupdict()orNone ) returnNone classEntryFormView(SuccessMessageMixin,FormView): template_name='entries/insert.html' form_class=EntryForm success_url=reverse_lazy('insert') success_message="Entrywascreatedsuccessfully" WOW! eBook www.wowebook.org @method_decorator( login_required(login_url=reverse_lazy('login'))) defget(self,request,*args,**kwargs): returnsuper(EntryFormView,self).get( request,*args,**kwargs) @method_decorator( login_required(login_url=reverse_lazy('login'))) defpost(self,request,*args,**kwargs): returnsuper(EntryFormView,self).post( request,*args,**kwargs) defform_valid(self,form): self._save_with_user(form) returnsuper(EntryFormView,self).form_valid(form) def_save_with_user(self,form): self.object=form.save(commit=False) self.object.user=self.request.user self.object.save() Let’sstartwiththeimports.Weneedtheremoduletohandleregularexpressions,thenwe needafewclassesandfunctionsfromDjango,andfinally,weneedtheEntrymodeland theEntryFormform. Thehomeview ThefirstviewisHomeView.ItinheritsfromTemplateView,whichmeansthattheresponse willbecreatedbyrenderingatemplatewiththecontextwe’llcreateintheview.Allwe havetodoisspecifythetemplate_nameclassattributetopointtothecorrecttemplate. Djangopromotescodereusetoapointthatifwedidn’tneedtomakethisviewaccessible onlytologged-inusers,thefirsttwolineswouldhavebeenallweneeded. However,wewantthisviewtobeaccessibleonlytologged-inusers;therefore,weneedto decorateitwithlogin_required.Now,historicallyviewsinDjangousedtobefunctions; therefore,thisdecoratorwasdesignedtoacceptafunctionnotamethodlikewehavein thisclass.We’reusingDjangoclass-basedviewsinthisprojectso,inordertomakethings work,weneedtotransformlogin_requiredsothatitacceptsamethod(thedifference beinginthefirstargument:self).Wedothisbypassinglogin_requiredto method_decorator. Wealsoneedtofeedthelogin_requireddecoratorwithlogin_urlinformation,andhere comesanotherwonderfulfeatureofDjango.Asyou’llseeafterwe’redonewiththe views,inDjango,youtieaviewtoaURLthroughapattern,consistingofaregular expressionandotherinformation.Youcangiveanametoeachentryintheurls.pyfileso thatwhenyouwanttorefertoaURL,youdon’thavetohardcodeitsvalueintoyourcode. AllyouhavetodoisgetDjangotoreverse-engineerthatURLfromthenamewegaveto theentryinurls.pydefiningtheURLandtheviewthatistiedtoit.Thismechanismwill becomeclearerlater.Fornow,justthinkofreverse('...')asawayofgettingaURL fromanidentifier.Inthisway,youonlywritetheactualURLonce,intheurls.pyfile, WOW! eBook www.wowebook.org whichisbrilliant.Intheviews.pycode,weneedtousereverse_lazy,whichworks exactlylikereversewithonemajordifference:itonlyfindstheURLwhenweactually needit(inalazyfashion).Thisisneededwhentheurls.pyfilehasn’tbeenloadedyet whenthereversefunctionisused. Thegetmethod,whichwejustdecorated,simplycallsthegetmethodoftheparentclass. Ofcourse,thegetmethodisthemethodthatDjangocallswhenaGETrequestis performedagainsttheURLtiedtothisview. Theentrylistview Thisviewismuchmoreinterestingthanthepreviousone.Firstofall,wedecoratetheget methodaswedidbefore.Insideofit,weneedtopreparealistofEntryobjectsandfeedit tothetemplate,whichshowsittotheuser.Inordertodoso,westartbygettingthe contextdictlikewe’resupposedtodo,bycallingtheget_context_datamethodofthe TemplateViewclass.Then,weusetheORMtogetalistoftheentries.Wedothisby accessingtheobjectsmanager,andcallingafilteronit.Wefiltertheentriesaccordingto whichuserisloggedin,andweaskforthemtobesortedinadescendingorder(that'-' infrontofthenamespecifiesthedescendingorder).Theobjectsmanageristhedefault managereveryDjangomodelisaugmentedwithoncreation,itallowsustointeractwith thedatabasethroughitsmethods. Weparseeachentrytogetalistofmatches(actually,Icodeditsothatmatchesisa generatorexpression).Finally,weaddtothecontextan'entries'keywhosevalueisthe couplingofentriesandmatches,sothateachEntryinstanceispairedwiththeresulting matchofitspatternandteststring. Onthelastline,wesimplyaskDjangotorenderthetemplateusingthecontextwe created. Takealookatthe_parse_entrymethod.Allitdoesisperformasearchonthe entry.test_stringwiththeentry.pattern.IftheresultingmatchobjectisnotNone,it meansthatwefoundsomething.Ifso,wereturnatuplewiththreeelements:theoverall group,thesubgroups,andthegroupdictionary.Ifyou’renotfamiliarwiththeseterms, don’tworry,you’llseeascreenshotsoonwithanexample.WereturnNoneifthereisno match. Theformview Finally,let’sexamineEntryFormView.Thisisparticularlyinterestingforafewreasons. Firstly,itshowsusaniceexampleofPython’smultipleinheritance.Wewanttodisplaya messageonthepage,afterhavinginsertedanEntry,soweinheritfrom SuccessMessageMixin.Butwewanttohandleaformaswell,sowealsoinheritfrom FormView. Note Notethat,whenyoudealwithmixinsandinheritance,youmayhavetoconsidertheorder inwhichyouspecifythebaseclassesintheclassdeclaration. WOW! eBook www.wowebook.org Inordertosetupthisviewcorrectly,weneedtospecifyafewattributesatthebeginning: thetemplatetoberendered,theformclasstobeusedtohandlethedatafromthePOST request,theURLweneedtoredirecttheusertointhecaseofsuccess,andthesuccess message. AnotherinterestingfeatureisthatthisviewneedstohandlebothGETandPOSTrequests. Whenwelandontheformpageforthefirsttime,theformisempty,andthatistheGET request.Ontheotherhand,whenwefillintheformandwanttosubmittheEntry,we makeaPOSTrequest.Youcanseethatthebodyofgetisconceptuallyidenticalto HomeView.Djangodoeseverythingforus. Thepostmethodisjustlikeget.Theonlyreasonweneedtocodethesetwomethodsis sothatwecandecoratethemtorequirelogin. WithintheDjangoformhandlingprocess(intheFormViewclass),thereareafewmethods thatwecanoverrideinordertocustomizetheoverallbehavior.Weneedtodoitwiththe form_validmethod.Thismethodwillbecalledwhentheformvalidationissuccessful.Its purposeistosavetheformsothatanEntryobjectiscreatedoutofit,andthenstoredin thedatabase. Theonlyproblemisthatourformismissingtheuser.Weneedtointerceptthatmomentin thechainofcallsandputtheuserinformationinourselves.Thisisdonebycallingthe _save_with_usermethod,whichisverysimple. Firstly,weaskDjangotosavetheformwiththecommitargumentsettoFalse.This createsanEntryinstancewithoutattemptingtosaveittothedatabase.Savingit immediatelywouldfailbecausetheuserinformationisnotthere. ThenextlineupdatestheEntryinstance(self.object),addingtheuserinformationand, onthelastline,wecansafelysaveit.ThereasonIcalleditobjectandsetitonthe instancelikethatwastofollowwhattheoriginalFormViewclassdoes. We’refiddlingwiththeDjangomechanismhere,soifwewantthewholethingtowork, weneedtopayattentiontowhenandhowwemodifyitsbehavior,andmakesurewe don’talteritincorrectly.Forthisreason,it’sveryimportanttoremembertocallthe form_validmethodofthebaseclass(weusesuperforthat)attheendofourown customizedversion,tomakesurethateveryotheractionthatmethodusuallyperformsis carriedoutcorrectly. Notehowtherequestistiedtoeachviewinstance(self.request)sothatwedon’tneed topassitthroughwhenwerefactorourlogicintomethods.Notealsothattheuser informationhasbeenaddedtotherequestautomaticallybyDjango.Finally,notethatthe reasonwhyalltheprocessissplitintoverysmallmethodsliketheseissothatwecanonly overridethosethatweneedtocustomize.Allthisremovestheneedtowritealotofcode. Nowthatwehavetheviewscovered,let’sseehowwecouplethemtotheURLs. WOW! eBook www.wowebook.org TyingupURLsandviews Intheurls.pymodule,wetieeachviewtoaURL.Therearemanywaysofdoingthis.I chosethesimplestone,whichworksperfectlyfortheextentofthisexercise,butyoumay wanttoexplorethisargumentmoredeeplyifyouintendtoworkwithDjango.Thisisthe corearoundwhichthewholewebsitelogicwillrevolve;therefore,youshouldtrytogetit downcorrectly.Notethattheurls.pymodulebelongstotheprojectfolder. regex/urls.py fromdjango.conf.urlsimportinclude,url fromdjango.contribimportadmin fromdjango.contrib.authimportviewsasauth_views fromdjango.core.urlresolversimportreverse_lazy fromentries.viewsimportHomeView,EntryListView,EntryFormView urlpatterns=[ url(r'^admin/',include(admin.site.urls)), url(r'^entries/$',EntryListView.as_view(),name='entries'), url(r'^entries/insert$', EntryFormView.as_view(), name='insert'), url(r'^login/$', auth_views.login, kwargs={'template_name':'admin/login.html'}, name='login'), url(r'^logout/$', auth_views.logout, kwargs={'next_page':reverse_lazy('home')}, name='logout'), url(r'^$',HomeView.as_view(),name='home'), ] Asyoucansee,themagiccomesfromtheurlfunction.Firstly,wepassitaregular expression;thentheview;andfinally,aname,whichiswhatwewilluseinthereverse andreverse_lazyfunctionstorecovertheURL. Notethat,whenusingclass-basedviews,wehavetotransformthemintofunctions,which iswhaturlisexpecting.Todothat,wecalltheas_view()methodonthem. Notealsothatthefirsturlentry,fortheadmin,isspecial.InsteadofspecifyingaURL andaview,itspecifiesaURLprefixandanotherurls.pymodule(fromtheadmin.site package).Inthisway,DjangowillcompletealltheURLsfortheadminsectionby prepending'admin/'toalltheURLsspecifiedinadmin.site.urls.Wecouldhavedone thesameforourentriesapp(andweshouldhave),butIfeelitwouldhavebeenabittoo muchforthissimpleproject. Intheregularexpressionlanguage,the'^'and'$'symbolsrepresentthestartandendof astring.Notethatifyouusetheinclusiontechnique,asfortheadmin,the'$'ismissing. Ofcourse,thisisbecause'admin/'isjustaprefix,whichneedstobecompletedbyallthe definitionsintheincludedurlsmodule. WOW! eBook www.wowebook.org Somethingelseworthnoticingisthatwecanalsoincludethestringifiedversionofapath toaview,whichwedofortheloginandlogoutviews.Wealsoaddinformationabout whichtemplatestousewiththekwargsargument.Theseviewscomestraightfromthe django.contrib.authpackage,bytheway,sothatwedon’tneedtowriteasinglelineof codetohandleauthentication.Thisisbrilliantandsavesusalotoftime. Eachurldeclarationmustbedonewithintheurlpatternslistandonthismatter,it’s importanttoconsiderthat,whenDjangoistryingtofindaviewforaURLthathasbeen requested,thepatternsareexercisedinorder,fromtoptobottom.Thefirstonethat matchesistheonethatwillprovidetheviewforitso,ingeneral,youhavetoputspecific patternsbeforegenericones,otherwisetheywillnevergetachancetobecaught.For example,'^shop/categories/$'needstocomebefore'^shop'(notetheabsenceofthe '$'inthelatter),otherwiseitwouldneverbecalled.Ourexamplefortheentriesworks finebecauseIthoroughlyspecifiedURLsusingthe'$'attheend. So,models,forms,admin,viewsandURLsarealldone.Allthatislefttodoistakecare ofthetemplates.I’llhavetobeverybriefonthispartbecauseHTMLcanbeveryverbose. WOW! eBook www.wowebook.org Writingthetemplates Alltemplatesinheritfromabaseone,whichprovidestheHTMLstructureforallothers, inaveryOOPtypeoffashion.Italsospecifiesafewblocks,whichareareasthatcanbe overriddenbychildrensothattheycanprovidecustomcontentforthoseareas.Let’sstart withthebasetemplate: entries/templates/entries/base.html {%loadstaticfromstaticfiles%} <!DOCTYPEhtml> <htmllang="en"> <head> {%blockmeta%} <metacharset="utf-8"> <metaname="viewport" content="width=device-width,initial-scale=1.0"> {%endblockmeta%} {%blockstyles%} <linkhref="{%static"entries/css/main.css"%}" rel="stylesheet"> {%endblockstyles%} <title>{%blocktitle%}Title{%endblocktitle%}</title> </head> <body> <divid="page-content"> {%blockpage-content%} {%endblockpage-content%} </div> <divid="footer"> {%blockfooter%} {%endblockfooter%} </div> </body> </html> Thereisagoodreasontorepeattheentriesfolderfromthetemplatesone.Whenyou deployaDjangowebsite,youcollectallthetemplatefilesunderonefolder.Ifyoudon’t specifythepathslikeIdid,youmaygetabase.htmltemplateintheentriesapp,anda base.htmltemplateinanotherapp.Thelastonetobecollectedwilloverrideanyother filewiththesamename.Forthisreason,byputtingtheminatemplates/entriesfolder andusingthistechniqueforeachDjangoappyouwrite,youavoidtheriskofname collisions(thesamegoesforanyotherstaticfile). Thereisnotmuchtosayaboutthistemplate,really,apartfromthefactthatitloadsthe statictagsothatwecangeteasyaccesstothestaticpathwithouthardcodingitinthe templatebyusing{%static…%}.Thecodeinthespecial{%...%}sectionsiscodethat defineslogic.Thecodeinthespecial{{...}}representsvariablesthatwillberendered onthepage. WOW! eBook www.wowebook.org Wedefinethreeblocks:title,page-content,andfooter,whosepurposeistoholdthe title,thecontentofthepage,andthefooter.Blockscanbeoptionallyoverriddenbychild templatesinordertoprovidedifferentcontentwithinthem. Here’sthefooter: entries/templates/entries/footer.html <divclass="footer"> Goback<ahref="{%url"home"%}">home</a>. </div> Itgivesusanicelinktothehomepage. Thehomepagetemplateisthefollowing: entries/templates/entries/home.html {%extends"entries/base.html"%} {%blocktitle%}WelcometotheEntrywebsite.{%endblocktitle%} {%blockpage-content%} <h1>Welcome{{user.first_name}}!</h1> <divclass="home-option">Toseethelistofyourentries pleaseclick<ahref="{%url"entries"%}">here.</a> </div> <divclass="home-option">Toinsertanewentrypleaseclick <ahref="{%url"insert"%}">here.</a> </div> <divclass="home-option">Tologinasanotheruserpleaseclick <ahref="{%url"logout"%}">here.</a> </div> <divclass="home-option">Togototheadminpanel pleaseclick<ahref="{%url"admin:index"%}">here.</a> </div> {%endblockpage-content%} Itextendsthebase.htmltemplate,andoverridestitleandpage-content.Youcansee thatbasicallyallitdoesisprovidefourlinkstotheuser.Thesearethelistofentries,the insertpage,thelogoutpage,andtheadminpage.Allofthisisdonewithouthardcodinga singleURL,throughtheuseofthe{%url…%}tag,whichisthetemplateequivalentof thereversefunction. ThetemplateforinsertinganEntryisasfollows: entries/templates/entries/insert.html {%extends"entries/base.html"%} {%blocktitle%}InsertanewEntry{%endblocktitle%} {%blockpage-content%} {%ifmessages%} {%formessageinmessages%} <pclass="{{message.tags}}">{{message}}</p> {%endfor%} {%endif%} WOW! eBook www.wowebook.org <h1>InsertanewEntry</h1> <formaction="{%url"insert"%}"method="post"> {%csrf_token%}{{form.as_p}} <inputtype="submit"value="Insert"> </form><br> {%endblockpage-content%} {%blockfooter%} <div><ahref="{%url"entries"%}">Seeyourentries.</a></div> {%include"entries/footer.html"%} {%endblockfooter%} Thereissomeconditionallogicatthebeginningtodisplaymessages,ifany,andthenwe definetheform.Djangogivesustheabilitytorenderaformbysimplycalling{{ form.as_p}}(alternatively,form.as_ulorform.as_table).Thiscreatesallthe necessaryfieldsandlabelsforus.Thedifferencebetweenthethreecommandsisinthe waytheformislaidout:asaparagraph,asanunorderedlistorasatable.Weonlyneedto wrapitinformtagsandaddasubmitbutton.Thisbehaviorwasdesignedforour convenience;weneedthefreedomtoshapethat<form>tagaswewant,soDjangoisn’t intrusiveonthat.Also,notethat{%csrf_token%}.Itwillberenderedintoatokenby Djangoandwillbecomepartofthedatasenttotheserveronsubmission.Thisway Djangowillbeabletoverifythattherequestwasfromanallowedsource,thusavoiding theaforementionedcross-siterequestforgeryissue.Didyouseehowwehandledthe tokenwhenwewrotetheviewfortheEntryinsertion?Exactly.Wedidn’twriteasingle lineofcodeforit.Djangotakescareofitautomaticallythankstoamiddlewareclass (CsrfViewMiddleware).PleaserefertotheofficialDjangodocumentationtoexplorethis subjectfurther. Forthispage,wealsousethefooterblocktodisplayalinktothehomepage.Finally,we havethelisttemplate,whichisthemostinterestingone. entries/templates/entries/list.html {%extends"entries/base.html"%} {%blocktitle%}Entrieslist{%endblocktitle%} {%blockpage-content%} {%ifentries%} <h1>Yourentries({{entries|length}}found)</h1> <div><ahref="{%url"insert"%}">Insertnewentry.</a></div> <tableclass="entries-table"> <thead> <tr><th>Entry</th><th>Matches</th></tr> </thead> <tbody> {%forentry,matchinentries%} <trclass="entries-list{%cycle'light-gray''white'%}"> <td> Pattern:<codeclass="code"> "{{entry.pattern}}"</code><br> TestString:<codeclass="code"> WOW! eBook www.wowebook.org "{{entry.test_string}}"</code><br> Added:{{entry.date_added}} </td> <td> {%ifmatch%} Group:{{match.0}}<br> Subgroups: {{match.1|default_if_none:"none"}}<br> GroupDict:{{match.2|default_if_none:"none"}} {%else%} Nomatchesfound. {%endif%} </td> </tr> {%endfor%} </tbody> </table> {%else%} <h1>Youhavenoentries</h1> <div><ahref="{%url"insert"%}">Insertnewentry.</a></div> {%endif%} {%endblockpage-content%} {%blockfooter%} {%include"entries/footer.html"%} {%endblockfooter%} Itmaytakeyouawhiletogetusedtothetemplatelanguage,butreally,allthereistoitis thecreationofatableusingaforloop.Westartbycheckingifthereareanyentriesand, ifso,wecreateatable.Therearetwocolumns,onefortheEntry,andtheotherforthe match. IntheEntrycolumn,wedisplaytheEntryobject(apartfromtheuser)andintheMatches column,wedisplaythat3-tuplewecreatedintheEntryListView.Notethattoaccessthe attributesofanobject,weusethesamedotsyntaxweuseinPython,forexample{{ entry.pattern}}or{{entry.test_string}},andsoon. Whendealingwithlistsandtuples,wecannotaccessitemsusingthesquarebrackets syntax,soweusethedotoneaswell({{match.0}}isequivalenttomatch[0],andso on.).Wealsouseafilter,throughthepipe(|)operatortodisplayacustomvalueifa matchisNone. TheDjangotemplatelanguage(whichisnotproperlyPython)iskeptsimpleforaprecise reason.Ifyoufindyourselflimitedbythelanguage,itmeansyou’reprobablytryingtodo somethinginthetemplatethatshouldactuallybedoneintheview,wherethatlogicis morerelevant. Allowmetoshowyouacoupleofscreenshotsofthelistandinserttemplates.Thisis whatthelistofentrieslookslikeformyfather: WOW! eBook www.wowebook.org Notehowtheuseofthecycletagalternatesthebackgroundcoloroftherowsfromwhite tolightgray.Thoseclassesaredefinedinthemain.cssfile. TheEntryinsertionpageissmartenoughtoprovideafewdifferentscenarios.Whenyou landonitatfirst,itpresentsyouwithjustanemptyform.Ifyoufillitincorrectly,itwill displayanicemessageforyou(seethefollowingpicture).However,ifyoufailtofillin bothfields,itwilldisplayanerrormessagebeforethem,alertingyouthatthosefieldsare required. Notealsothecustomfooter,whichincludesbothalinktotheentrieslistandalinktothe homepage: WOW! eBook www.wowebook.org Andthat’sit!YoucanplayaroundwiththeCSSstylesifyouwish.Downloadthecodefor thebookandhavefunexploringandextendingthisproject.Addsomethingelsetothe model,createandapplyamigration,playwiththetemplates,there’slotstodo! Djangoisaverypowerfulframework,andofferssomuchmorethanwhatI’vebeenable toshowyouinthischapter,soyoudefinitelywanttocheckitout.Thebeautyofitisthat it’sPython,soreadingitssourcecodeisaveryusefulexercise. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Thefutureofwebdevelopment Computerscienceisaveryyoungsubject,comparedtootherbranchesofsciencethat haveexistedalongsidehumankindforcenturiesormore.Oneofitsmaincharacteristicsis thatitmovesextremelyfast.Itleapsforwardwithsuchspeedthat,injustafewyears,you canseechangesthatarecomparabletorealworldchangesthattookacenturytohappen. Therefore,asacoder,youmustpayattentiontowhathappensinthisworld,allthetime. Somethingthatishappeningnowisthatbecausepowerfulcomputersarenowquitecheap andalmosteveryonehasaccesstothem,thetrendistotryandavoidputtingtoomuch workloadonthebackend,andletthefrontendhandlepartofit.Therefore,inthelastfew years,JavaScriptframeworksandlibrarieslikejQueryandBackbonehavebecomevery popularandwebdevelopmenthasshiftedfromaparadigmwherethebackendtakescare ofhandlingdata,preparingit,andservingittothefrontendtodisplayit,toaparadigm wherethebackendissometimesjustusedasanAPI,asheerdataprovider.Thefrontend fetchesthedatafromthebackendwithanAPIcall,andthenittakescareoftherest.This shiftfacilitatestheexistenceofparadigmslikeSingle-PageApplication(SPA),where, ideally,thewholepageisloadedonceandthenevolves,basedonthecontentthatusually comesfromthebackend.E-commercewebsitesthatloadtheresultsofasearchinapage thatdoesn’trefreshthesurroundingstructure,aremadewithsimilartechniques.Browsers canperformasynchronouscalls(AJAX)thatcanreturndatawhichcanberead, manipulatedandinjectedbackintothepagewithJavaScriptcode. So,ifyou’replanningtoworkonwebdevelopment,Istronglysuggestyoutoget acquaintedwithJavaScript(ifyou’renotalready),andalsowithAPIs.Inthelastfew pagesofthischapter,I’llgiveyouanexampleofhowtomakeasimpleAPIusingtwo differentPythonmicroframeworks:FlaskandFalcon. WOW! eBook www.wowebook.org WritingaFlaskview Flask(http://flask.pocoo.org/)isaPythonmicroframework.Itprovidesfewerfeaturesthan Django,butit’ssupposedlyfasterandquickertogetupandrunning.Tobehonest,getting Djangoupandrunningnowadaysisalsoveryquicklydone,butFlaskissopopularthat it’sgoodtoseeanexampleofit,nonetheless. Inyourch10folder,createaflaskfolderwiththefollowingstructure: $tree-Aflask#fromthech10folder flask ├──main.py └──templates └──main.html Basically,we’regoingtocodetwosimplefiles:aFlaskapplicationandanHTML template.FlaskusesJinja2astemplateengine.It’sextremelypopularandveryfast,and justrecentlyevenDjangohasstartedtooffernativesupportforit,whichissomethingthat Pythoncodershavelongedfor,foralongtime. flask/templates/main.html <!doctypehtml> <title>HellofromFlask</title> <h1> {%ifname%} Hello{{name}}! {%else%} Helloshyperson! {%endif%} </h1> Thetemplateisalmostoffensivelysimple;allitdoesistochangethegreetingaccording tothepresenceofthenamevariable.AbitmoreinterestingistheFlaskapplicationthat rendersit: flask/main.py fromflaskimportFlask,render_template app=Flask(__name__) @app.route('/') @app.route('/<name>') defhello(name=None): returnrender_template('main.html',name=name) if__name__=='__main__': app.run() Wecreateanappobject,whichisaFlaskapplication.Weonlyfeedthefully-qualified nameofthemodule,whichisstoredin__name__. Then,wewriteasimplehelloview,whichtakesanoptionalnameargument.Inthebody oftheview,wesimplyrenderthemain.htmltemplate,passingtoitthenameargument, WOW! eBook www.wowebook.org regardlessofitsvalue. What’sinterestingistherouting.DifferentlyfromDjango’swayoftyingupviewsand URLs(theurls.pymodule),inFlaskyoudecorateyourviewwithoneormore @app.routedecorators.Inthiscase,weacceptboththerootURLwithoutanythingelse, orwithnameinformation. Changeintotheflaskfolderandtype(makesureyouhaveFlaskinstalledwith$pip installflask): $pythonmain.py Youcanopenabrowserandgotohttp://127.0.0.1:5000/.ThisURLhasnoname information;therefore,youwillseeHelloshyperson!Itiswrittenallniceandbig.Tryto addsomethingtothatURLlikehttp://127.0.0.1:5000/Adriano.HitEnterandthepage willchangetoHelloAdriano!. Ofcourse,Flaskoffersyoumuchmorethanthisbutwedon’thavetheroomtoseeamore complexexample.It’sdefinitelyworthexploring,though.Severalprojectsuseit successfullyandit’sfunanditisnicetocreatewebsitesorAPIswithit.Flask’sauthor, ArminRonacher,isasuccessfulandveryprolificcoder.Healsocreatedorcollaborated onseveralotherinterestingprojectslikeWerkzeug,Jinja2,Click,andSphinx. WOW! eBook www.wowebook.org BuildingaJSONquoteserverinFalcon Falcon(http://falconframework.org/)isanothermicroframeworkwritteninPython,which wasdesignedtobelight,fastandflexible.Ithinkthisrelativelyyoungprojectwillevolve tobecomesomethingreallypopularduetoitsspeed,whichisimpressive,soI’mhappyto showyouatinyexampleusingit. We’regoingtobuildaviewthatreturnsarandomlychosenquotefromtheBuddha. Inyourch10folder,createanewonecalledfalcon.We’llhavetwofiles:quotes.pyand main.py.Torunthisexample,installFalconandGunicorn($pipinstallfalcon gunicorn).Falconistheframework,andGunicorn(GreenUnicorn)isaPythonWSGI HTTPServerforUnix(which,inlaymanterms,meansthetechnologythatisusedtorun theserver).Whenyou’reallsetup,startbycreatingthequotes.pyfile. falcon/quotes.py quotes=[ "Thousandsofcandlescanbelightedfromasinglecandle," "andthelifeofthecandlewillnotbeshortened." "Happinessneverdecreasesbybeingshared.", ... "Peacecomesfromwithin.Donotseekitwithout.", ] Youwillfindthecompletelistofquotesinthesourcecodeforthisbook.Ifyoudon’thave it,youcanalsofillinyourfavoritequotes.Notethatnoteverylinehasacommaatthe end.InPython,it’spossibletoconcatenatestringslikethat,aslongastheyareinbrackets (orbraces).It’scalledimplicitconcatenation. Thecodeforthemainappisnotlong,butitisinteresting: falcon/main.py importjson importrandom importfalcon fromquotesimportquotes classQuoteResource: defon_get(self,req,resp): quote={ 'quote':random.choice(quotes), 'author':'TheBuddha' } resp.body=json.dumps(quote) api=falcon.API() api.add_route('/quote',QuoteResource()) Let’sstartwiththeclass.InDjangowehadagetmethod,inFlaskwedefinedafunction, andherewewriteanon_getmethod,anamingstylethatremindsmeofC#event handlers.Ittakesarequestandaresponseargument,bothautomaticallyfedbythe framework.Initsbody,wedefineadictwitharandomlychosenquote,andtheauthor WOW! eBook www.wowebook.org information.ThenwedumpthatdicttoaJSONstringandsettheresponsebodytoits value.Wedon’tneedtoreturnanything,Falconwilltakecareofitforus. Attheendofthefile,wecreatetheFalconapplication,andwecalladd_routeonittotie thehandlerwehavejustwrittentotheURLwewant. Whenyou’reallsetup,changetothefalconfolderandtype: $gunicornmain:api Then,makearequest(orsimplyopenthepagewithyourbrowser)to http://127.0.0.1:8000/quote.WhenIdidit,IgotthisJSONinresponse: { quote:"Themindiseverything.Whatyouthinkyoubecome.", author:"TheBuddha" } Whatevertheframeworkyouendupusingforyourwebdevelopment,tryandkeep yourselfinformedaboutotherchoicestoo.Sometimesyoumaybeinsituationswherea differentframeworkistherightwaytogo,andhavingaworkingknowledgeofdifferent toolswillgiveyouanadvantage. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,wecaughtaglimpseofwebdevelopment.Wetalkedaboutimportant conceptsliketheDRYphilosophyandtheconceptofaframeworkasatoolthatprovides uswithmanythingsweneedinordertowritecodetoserverequests.Wealsotalkedabout theMTVpattern,andhownicelythesethreelayersplaytogethertorealizearequestresponsepath. Lateron,webrieflyintroducedregularexpressions,whichisasubjectofparamount importance,andit’sthelayerwhichprovidesthetoolsforURLrouting. Therearemanydifferentframeworksoutthere,andDjangoisdefinitelyoneofthebest andmostwidelyused,soit’sdefinitelyworthexploring,especiallyitssourcecode,which isverywellwritten. Thereareotherveryinterestingandimportantframeworkstoo,likeFlask.Theyprovide fewerfeaturesbut,ingeneral,theyarefaster,bothinexecutiontimeandtosetup.One thatisextremelyfastistherelativelyyoungFalconproject,whosebenchmarksare outstanding. It’simportanttogetasolidunderstandingofhowtherequest-responsemechanismworks, andhowtheWebingeneralworks,sothateventuallyitwon’tmattertoomuchwhich frameworkyouhavetouse.Youwillbeabletopickitupquicklybecauseitwillonlybea matterofgettingfamiliarwithawayofdoingsomethingyoualreadyknowalotabout. Exploreatleastthreeframeworksandtrytocomeupwithdifferentusecasestodecide whichoneofthemcouldbetheidealchoice.Whenyouareabletomakethatchoice,you willknowyouhaveagoodenoughunderstandingofthem. Thenextchapterisaboutdebuggingandtroubleshooting.We’lllearnhowtodealwith errorsandissuessothatifyougetintroublewhencoding(don’tworry,normallyitonly happensaboutallthetime),youwillbeabletoquicklyfindthesolutiontoyourproblem andmoveon. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter11.Debuggingand Troubleshooting “Ifdebuggingistheprocessofremovingsoftwarebugs,thenprogrammingmustbetheprocessofputtingthemin.” —EdsgerW.Dijkstra Inthelifeofaprofessionalcoder,debuggingandtroubleshootingtakeupasignificant amountoftime.Evenifyouworkonthemostbeautifulcodebaseeverwrittenbyman, therewillstillbebugsinit,thatisguaranteed. Wespendanawfullotoftimereadingotherpeople’scodeand,inmyopinion,agood softwaredeveloperissomeonewhokeepstheirattentionhigh,evenwhenthey’rereading codethatisnotreportedtobewrongorbuggy. Beingabletodebugcodeefficientlyandquicklyisaskillthatanycoderneedstokeep improving.Somethinkthatbecausetheyhavereadthemanual,they’refine,butthereality is,thenumberofvariablesinthegameissobigthatthereisnomanual.Thereare guidelinesthatonecanfollow,butthereisnomagicbookthatwillteachyoueverything youneedtoknowinordertobecomegoodatthis. Ifeelthatonthisparticularsubject,Ihavelearnedthemostfrommycolleagues.Itamazes metoobservesomeoneveryskilledattackingaproblem.Ienjoyseeingthestepsthey take,thethingstheyverifytoexcludepossiblecauses,andthewaytheyconsiderthe suspectsthateventuallyleadthemtothesolutiontotheproblem. Everycolleagueweworkwithcanteachussomething,orsurpriseuswithafantastic guessthatturnsouttobetherightone.Whenthathappens,don’tjustremainin wonderment(orworse,inenvy),butseizethemomentandaskthemhowtheygottothat guessandwhy.Theanswerwillallowyoutoseeifthereissomethingyoucanstudyin deeplateronsothat,maybenexttime,you’llbetheonewhowillcatchthebug. Somebugsareveryeasytospot.Theycomeoutofcoarsemistakesand,onceyouseethe effectsofthosemistakes,it’seasytofindasolutionthatfixestheproblem. Butthereareotherbugswhicharemuchmoresubtle,muchmoreslippery,andrequire trueexpertise,andagreatdealofcreativityandout-of-the-boxthinking,tobedealtwith. Theworstofall,atleastforme,arethenondeterministicones.Thesesometimeshappen, andsometimesdon’t.SomehappenonlyinenvironmentAbutnotinenvironmentB,even thoughAandBaresupposedtobeexactlythesame.Thosebugsarethetrueevilones, andtheycandriveyoucrazy. Andofcourse,bugsdon’tjusthappeninthesandbox,right?Withyourbosstellingyou “don’tworry!takeyourtimetofixthis,havelunchfirst!“.Nope.TheyhappenonaFriday athalfpastfive,whenyourbrainiscookedandyoujustwanttogohome.It’sinthose moments,wheneveryoneisgettingupsetinasplitsecond,whenyourbossisbreathingon yourneck,thatyouhavetobeabletokeepcalm.AndIdomeanit.That’sthemost WOW! eBook www.wowebook.org importantskilltohaveifyouwanttobeabletofightbugseffectively.Ifyouallowyour mindtogetstressed,saygoodbyetocreativethinking,tologicdeduction,andto everythingyouneedatthatmoment.Sotakeadeepbreath,sitproperly,andfocus. Inthischapter,Iwilltrytodemonstratesomeusefultechniquesthatyoucanemploy accordingtotheseverityofthebug,andafewsuggestionsthatwillhopefullyboostyour weaponsagainstbugsandissues. WOW! eBook www.wowebook.org Debuggingtechniques Inthispart,I’llpresentyouwiththemostcommontechniques,theonesIusemostoften, however,pleasedon’tconsiderthislisttobeexhaustive. WOW! eBook www.wowebook.org Debuggingwithprint Thisisprobablytheeasiesttechniqueofall.It’snotveryeffective,itcannotbeused everywhereanditrequiresaccesstoboththesourcecodeandaterminalthatwillrunit (andthereforeshowtheresultsoftheprintfunctioncalls). However,inmanysituations,thisisstillaquickandusefulwaytodebug.Forexample,if youaredevelopingaDjangowebsiteandwhathappensinapageisnotwhatwouldyou expect,youcanfilltheviewwithprintsandkeepaneyeontheconsolewhileyoureload thepage.I’veprobablydoneitamilliontimes. Whenyouscattercallstoprintinyourcode,younormallyendupinasituationwhere youduplicatealotofdebuggingcode,eitherbecauseyou’reprintingatimestamp(likewe didwhenweweremeasuringhowfastlistcomprehensionsandgeneratorswere),or becauseyouhavetosomehowbuildastringofsomesortthatyouwanttodisplay. Anotherissueisthatit’sextremelyeasytoforgetcallstoprintinyourcode. So,forthesereasons,ratherthanusingabarecalltoprint,Isometimesprefertocodea customfunction.Let’sseehow. WOW! eBook www.wowebook.org Debuggingwithacustomfunction Havingacustomfunctioninasnippetthatyoucanquicklygrabandpasteintothecode, andthenusetodebug,canbeveryuseful.Ifyou’refast,youcanalwayscodeoneonthe fly.Theimportantthingistocodeitinawaythatitwon’tleavestuffaroundwhenyou eventuallyremovethecallsanditsdefinition,thereforeit’simportanttocodeitinaway thatiscompletelyself-contained.Anothergoodreasonforthisrequirementisthatitwill avoidpotentialnameclasheswiththerestofthecode. Let’sseeanexampleofsuchafunction. custom.py defdebug(*msg,print_separator=True): print(*msg) ifprint_separator: print('-'*40) debug('Datais…') debug('Different','Strings','Arenotaproblem') debug('Afterwhileloop',print_separator=False) Inthiscase,Iamusingakeyword-onlyargumenttobeabletoprintaseparator,whichisa lineof40dashes. Thefunctionisverysimple,Ijustredirectwhateverisinmsgtoacalltoprintand,if print_separatorisTrue,Iprintalineseparator.Runningthecodewillshow: $pythoncustom.py Datais… ---------------------------------------DifferentStringsArenotaproblem ---------------------------------------Afterwhileloop Asyoucansee,thereisnoseparatorafterthelastline. Thisisjustoneeasywaytosomehowaugmentasimplecalltotheprintfunction.Let’s seehowwecancalculateatimedifferencebetweencalls,usingoneofPython’stricky featurestoouradvantage. custom_timestamp.py fromtimeimportsleep defdebug(*msg,timestamp=[None]): print(*msg) fromtimeimporttime#localimport iftimestamp[0]isNone: timestamp[0]=time()#1 else: now=time() print('Timeelapsed:{:.3f}s'.format( now-timestamp[0])) timestamp[0]=now#2 WOW! eBook www.wowebook.org debug('Enteringnastypieceofcode…') sleep(.3) debug('Firststepdone.') sleep(.5) debug('Secondstepdone.') Thisisabittrickier,butstillquitesimple.Firstnoticeweimportthetimefunctionfrom thetimemodulefromthedebugfunction.Thisallowsustoavoidhavingtoaddthat importoutsideofthefunction,andmaybeforgetitthere. TakealookathowIdefinedtimestamp.It’salist,ofcourse,butwhat’simportanthereis thatitisamutableobject.ThismeansthatitwillbesetupwhenPythonparsesthe functionanditwillretainitsvaluethroughoutdifferentcalls.Therefore,ifweputa timestampinitaftereachcall,wecankeeptrackoftimewithouthavingtouseanexternal globalvariable.Iborrowedthistrickfrommystudiesonclosures,atechniquethatI encourageyoutoreadaboutbecauseit’sveryinteresting. Right,so,afterhavingprintedwhatevermessagewehadtoprintandimportingtime,we theninspectthecontentoftheonlyitemintimestamp.IfitisNone,wehavenoprevious reference,thereforewesetthevaluetothecurrenttime(#1). Ontheotherhand,ifwehaveapreviousreference,wecancalculateadifference(which wenicelyformattothreedecimaldigits)andthenwefinallyputthecurrenttimeagainin timestamp(#2).It’sanicetrick,isn’tit? Runningthiscodeshowsthisresult: $pythoncustom_timestamp.py Enteringnastypieceofcode… Firststepdone. Timeelapsed:0.300s Secondstepdone. Timeelapsed:0.501s Whateverisyoursituation,havingaselfcontainedfunctionlikethiscanbeveryuseful. WOW! eBook www.wowebook.org Inspectingthetraceback WebrieflytalkedaboutthetracebackinChapter7,Testing,Profiling,andDealingwith Exceptionswhenwesawseveraldifferentkindsofexceptions.Thetracebackgivesyou informationaboutwhathappenedinyourapplicationthatwentwrong.Yougetagreat helpfromreadingit.Let’sseeaverysmallexample: traceback_simple.py d={'some':'key'} key='some-other' print(d[key]) Wehaveadictandwehavetriedtoaccessakeywhichisn’tinit.Youshouldremember thatthiswillraiseaKeyErrorexception.Let’srunthecode: $pythontraceback_simple.py Traceback(mostrecentcalllast): File"traceback_simple.py",line3,in<module> print(d[key]) KeyError:'some-other' Youcanseethatwegetalltheinformationweneed:themodulename,thelinethatcaused theerror(boththenumberandtheinstruction),andtheerroritself.Withthisinformation, youcangobacktothesourcecodeandtryandunderstandwhat’sgoingwrong. Let’snowcreateamoreinterestingexamplethatbuildsonthis,andexercisesafeature thatisonlyavailableinPython3.Imaginethatwe’revalidatingadict,workingon mandatoryfields,thereforeweexpectthemtobethere.Ifnot,weneedtoraiseacustom ValidationError,thatwewilltrapfurtherupstreamintheprocessthatrunsthevalidator (whichisnotshownhere,itcouldbeanything,really).Itshouldbesomethinglikethis: traceback_validator.py classValidatorError(Exception): """RaisedwhenaccessingadictresultsinKeyError.""" d={'some':'key'} mandatory_key='some-other' try: print(d[mandatory_key]) exceptKeyError: raiseValidatorError( '`{}`notfoundind.'.format(mandatory_key)) Wedefineacustomexceptionthatisraisedwhenthemandatorykeyisn’tthere.Notethat itsbodyconsistsofitsdocumentationstringsowedon’tneedtoaddanyotherstatements. Verysimply,wedefineadummydictandtrytoaccessitusingmandatory_key.Wetrap theKeyErrorandraiseValidatorErrorwhenthathappens.Thepurposeofdoingthisis thatwemayalsowanttoraiseValidatorErrorinothercircumstances,notnecessarilyas aconsequenceofamandatorykeybeingmissing.Thistechniqueallowsustorunthe validationinasimpletry/exceptthatonlycaresaboutValidatorError. WOW! eBook www.wowebook.org Thethingis,inPython2,thiscodewouldjustdisplaythelastexception (ValidatorError),whichmeanswewouldlosetheinformationabouttheKeyErrorthat precedesit.InPython3,thisbehaviorhaschangedandexceptionsarenowchainedsothat youhaveamuchbetterinformationreportwhensomethinghappens.Thecodeproduces thisresult: $pythontraceback_validator.py Traceback(mostrecentcalllast): File"traceback_validator.py",line7,in<module> print(d[mandatory_key]) KeyError:'some-other' Duringhandlingoftheaboveexception,anotherexceptionoccurred: Traceback(mostrecentcalllast): File"traceback_validator.py",line10,in<module> '`{}`notfoundind.'.format(mandatory_key)) __main__.ValidatorError:`some-other`notfoundind. Thisisbrilliant,becausewecanseethetracebackoftheexceptionthatledustoraise ValidationError,aswellasthetracebackfortheValidationErroritself. Ihadanicediscussionwithoneofmyreviewersaboutthetracebackyougetfromthepip installer.Hewashavingtroublesettingeverythingupinordertoreviewthecodefor Chapter9,DataScience.HisfreshUbuntuinstallationwasmissingafewlibrariesthat wereneededbythepippackagesinordertoruncorrectly. Thereasonhewasblockedwasthathewastryingtofixtheerrorsdisplayedinthe tracebackstartingfromthetopone.Isuggestedthathestartedfromthebottomone instead,andfixthat.Thereasonwasthat,iftheinstallerhadgottentothatlastline,Iguess thatbeforethat,whatevererrormayhaveoccurred,itwasstillpossibletorecoverfromit. Onlyafterthelastline,pipdecideditwasn’tpossibletocontinueanyfurther,and thereforeIstartedfixingthatone.Oncethelibrariesrequiredtofixthaterrorhadbeen installed,everythingelsewentsmoothly. Readingatracebackcanbetricky,andmyfriendwaslackingthenecessaryexperienceto addressthisproblemcorrectly,therefore,ifyouendupinthesamesituation,don’tbe discouraged,andtrytoshakethingsupabit,don’ttakeanythingforgranted. Pythonhasahugeandwonderfulcommunityandit’sveryunlikelythat,whenyou encounteraproblem,you’rethefirstonetoseeit,soopenabrowserandsearch.Bydoing so,yoursearchingskillswillalsoimprovebecauseyouwillhavetotrimtheerrordownto theminimumbutessentialsetofdetailsthatwillmakeyoursearcheffective. Ifyouwanttoplayandunderstandthetracebackabitbetter,inthestandardlibrarythere isamodulecalled,surprisesurprise,tracebackthatyoucanuse.Itprovidesastandard interfacetoextract,format,andprintstacktracesofPythonprograms,mimickingexactly thebehaviorofthePythoninterpreterwhenitprintsastacktrace. WOW! eBook www.wowebook.org UsingthePythondebugger AnotherveryeffectivewayofdebuggingPythonistousethePythondebugger:pdb.If youareaddictedtotheIPythonconsole,likeme,youshoulddefinitelycheckouttheipdb library.ipdbaugmentsthestandardpdbinterfacelikeIPythondoeswiththePython console. Thereareseveraldifferentwaysofusingthisdebugger(whicheverversion,itisnot important),butthemostcommononeconsistsofsimplysettingabreakpointandrunning thecode.WhenPythonreachesthebreakpoint,executionissuspendedandyouget consoleaccesstothatpointsothatyoucaninspectallthenames,andsoon.Youcanalso alterdataontheflytochangetheflowoftheprogram. Asatoyexample,let’spretendwehaveaparserthatisraisingaKeyErrorbecauseakey ismissinginadict.ThedictisfromaJSONpayloadthatwecannotcontrol,andwejust want,forthetimebeing,tocheatandpassthatcontrol,sincewe’reinterestedinwhat comesafterwards.Let’sseehowwecouldinterceptthismoment,inspectthedata,fixit andgettothebottom,withipdb. ipdebugger.py #dcomesfromaJSONpayloadwedon'tcontrol d={'first':'v1','second':'v2','fourth':'v4'} #keysalsocomesfromaJSONpayloadwedon'tcontrol keys=('first','second','third','fourth') defdo_something_with_value(value): print(value) forkeyinkeys: do_something_with_value(d[key]) print('Validationdone.') Asyoucansee,thiscodewillbreakwhenkeygetsthevalue'third',whichismissingin thedict.Remember,we’repretendingthatbothdandkeyscomedynamicallyfroma JSONpayloadwedon’tcontrol,soweneedtoinspecttheminordertofixdandpassthe forloop.Ifwerunthecodeasitis,wegetthefollowing: $pythonipdebugger.py v1 v2 Traceback(mostrecentcalllast): File"ipdebugger.py",line10,in<module> do_something_with_value(d[key]) KeyError:'third' Soweseethatthatkeyismissingfromthedict,butsinceeverytimewerunthiscodewe maygetadifferentdictorkeystuple,thisinformationdoesn’treallyhelpus.Let’sinjecta calltoipdb. ipdebugger_ipdb.py WOW! eBook www.wowebook.org #dcomesfromaJSONpayloadwedon'tcontrol d={'first':'v1','second':'v2','fourth':'v4'} #keysalsocomesfromaJSONpayloadwedon'tcontrol keys=('first','second','third','fourth') defdo_something_with_value(value): print(value) importipdb ipdb.set_trace()#weplaceabreakpointhere forkeyinkeys: do_something_with_value(d[key]) print('Validationdone.') Ifwenowrunthiscode,thingsgetinteresting(notethatyouroutputmayvaryalittleand thatallthecommentsinthisoutputwereaddedbyme): $pythonipdebugger_ipdb.py >/home/fab/srv/l.p/ch11/ipdebugger_ipdb.py(12)<module>() 11 --->12forkeyinkeys:#thisiswherethebreakpointcomes 13do_something_with_value(d[key]) ipdb>keys#let'sinspectthekeystuple ('first','second','third','fourth') ipdb>!d.keys()#nowthekeysofd dict_keys(['first','fourth','second'])#wemiss'third' ipdb>!d['third']='somethingdarkside…'#let'sputitin ipdb>c#...andcontinue v1 v2 somethingdarkside… v4 Validationdone. Thisisveryinteresting.First,notethat,whenyoureachabreakpoint,you’reserveda consolethattellsyouwhereyouare(thePythonmodule)andwhichlineisthenextoneto beexecuted.Youcan,atthispoint,performabunchofexploratoryactions,suchas inspectingthecodebeforeandafterthenextline,printingastacktrace,interactingwith theobjects,andsoon.PleaseconsulttheofficialPythondocumentationonpdbtolearn moreaboutthis.Inourcase,wefirstinspectthekeystuple.Afterthat,weinspectthekeys ofd. HaveyounoticedthatexclamationmarkIprependedtod?It’sneededbecausedisa commandinthepdbinterfacethatmovestheframe(d)own. Note Iindicatecommandswithintheipdbshellwiththisnotation:eachcommandisactivated byoneletter,whichtypicallyisthefirstletterofthecommandname.So,dfordown,nfor next,andsforstepbecome,moreconcisely,(d)own,(n)extand(s)tep. Iguessthisisagoodenoughreasontohavebetternames,right?Indeed,butIneededto WOW! eBook www.wowebook.org showyouthis,soIchosetoused.Inordertotellpdbthatwe’renotyieldinga(d)own command,weput“!”infrontofdandwe’refine. Afterseeingthekeysofd,weseethat'third'ismissing,soweputitinourselves(could thisbedangerous?thinkaboutit).Finally,nowthatallthekeysarein,wetypec,which means(c)ontinue. pdbalsogivesyoutheabilitytoproceedwithyourcodeonelineatatimeusing(n)ext,to (s)tepintoafunctionfordeeperanalysis,orhandlingbreakswith(b)reak.Foracomplete listofcommands,pleaserefertothedocumentationortype(h)elpintheconsole. Youcanseefromtheoutputthatwecouldfinallygettotheendofthevalidation. pdb(oripdb)areinvaluabletoolsthatIuseeveryday,Icouldn’tlivewithoutthem.So,go andhavefun,setabreakpointsomewhereandtryandinspect,followtheofficial documentationandtrythecommandsinyourcodetoseetheireffectandlearnthemwell. WOW! eBook www.wowebook.org Inspectinglogfiles Anotherwayofdebuggingamisbehavingapplicationistoinspectitslogfiles.Logfiles arespecialfilesinwhichanapplicationwritesdownallsortsofthings,normallyrelatedto what’sgoingoninsideofit.Ifanimportantprocedureisstarted,Iwouldtypicallyexpect alineforthatinthelogs.Itisthesamewhenitfinishes,andpossiblyforwhathappens insideofit. Errorsneedtobeloggedsothatwhenaproblemhappenswecaninspectwhatwentwrong bytakingalookattheinformationinthelogfiles. TherearemanydifferentwaystosetupaloggerinPython.Loggingisverymalleableand youcanconfigureit.Inanutshell,therearenormallyfourplayersinthegame:loggers, handlers,filters,andformatters: Loggersexposetheinterfacethattheapplicationcodeusesdirectly Handlerssendthelogrecords(createdbyloggers)totheappropriatedestination Filtersprovideafinergrainedfacilityfordeterminingwhichlogrecordstooutput Formattersspecifythelayoutofthelogrecordsinthefinaloutput LoggingisperformedbycallingmethodsoninstancesoftheLoggerclass.Eachlineyou loghasalevel.Thelevelsnormallyusedare:DEBUG,INFO,WARNING,ERROR,andCRITICAL. Youcanimportthemfromtheloggingmodule.Theyareinorderofseverityandit’svery importanttousethemproperlybecausetheywillhelpyoufilterthecontentsofalogfile basedonwhatyou’researchingfor.Logfilesusuallybecomeextremelybigsoit’svery importanttohavetheinformationinthemwrittenproperlysothatyoucanfinditquickly whenitmatters. Youcanlogtoafilebutyoucanalsologtoanetworklocation,toaqueue,toaconsole, andsoon.Ingeneral,ifyouhaveanarchitecturethatisdeployedononemachine,logging toafileisacceptable,butwhenyourarchitecturespansovermultiplemachines(suchasin thecaseofservice-orientedarchitectures),it’sveryusefultoimplementacentralized solutionforloggingsothatalllogmessagescomingfromeachservicecanbestoredand investigatedinasingleplace.Ithelpsalot,otherwiseyoucanreallygocrazytryingto correlategiantfilesfromseveraldifferentsourcestofigureoutwhatwentwrong. Note Aservice-orientedarchitecture(SOA)isanarchitecturalpatterninsoftwaredesignin whichapplicationcomponentsprovideservicestoothercomponentsviaa communicationsprotocol,typicallyoveranetwork.Thebeautyofthissystemisthat, whencodedproperly,eachservicecanbewritteninthemostappropriatelanguageto serveitspurpose.Theonlythingthatmattersisthecommunicationwiththeother services,whichneedstohappenviaacommonformatsothatdataexchangecanbedone. Here,Iwillpresentyouwithaverysimpleloggingexample.Wewilllogafewmessages toafile: log.py WOW! eBook www.wowebook.org importlogging logging.basicConfig( filename='ch11.log', level=logging.DEBUG,#minimumlevelcaptureinthefile format='[%(asctime)s]%(levelname)s:%(message)s', datefmt='%m/%d/%Y%I:%M:%S%p') mylist=[1,2,3] logging.info('Startingtoprocess`mylist`...') forpositioninrange(4): try: logging.debug('Valueatposition{}is{}'.format( position,mylist[position])) exceptIndexError: logging.exception('Faultyposition:{}'.format(position)) logging.info('Doneparsing`mylist`.') Let’sgothroughitlinebyline.First,weimporttheloggingmodule,thenwesetupa basicconfiguration.Ingeneral,aproductionloggingconfigurationismuchmore complicatedthanthis,butIwantedtokeepthingsaseasyaspossible.Wespecifya filename,theminimumlogginglevelwewanttocaptureinthefile,andthemessage format.We’lllogthedateandtimeinformation,thelevel,andthemessage. Iwillstartbylogginganinfomessagethattellsmewe’reabouttoprocessourlist.Then, Iwilllog(thistimeusingtheDEBUGlevel,byusingthedebugfunction)whichisthevalue atsomeposition.I’musingdebugherebecauseIwanttobeabletofilterouttheselogsin thefuture(bysettingtheminimumleveltologging.INFOormore),becauseImighthave tohandleverybiglistsandIdon’twanttologallthevalues. IfwegetanIndexError(andwedo,sinceI’mloopingoverrange(4)),wecall logging.exception(),whichisthesameaslogging.error(),butitalsoprintsthe traceback. Attheendofthecode,Iloganotherinfomessagesayingwe’redone.Theresultisthis: [10/08/201504:17:06PM]INFO:Startingtoprocess`mylist`... [10/08/201504:17:06PM]DEBUG:Valueatposition0is1 [10/08/201504:17:06PM]DEBUG:Valueatposition1is2 [10/08/201504:17:06PM]DEBUG:Valueatposition2is3 [10/08/201504:17:06PM]ERROR:Faultyposition:3 Traceback(mostrecentcalllast): File"log.py",line15,in<module> position,mylist[position])) IndexError:listindexoutofrange [10/08/201504:17:06PM]INFO:Doneparsing`mylist`. Thisisexactlywhatweneedtobeabletodebuganapplicationthatisrunningonabox, andnotonourconsole.Wecanseewhatwenton,thetracebackofanyexceptionraised, andsoon. Note WOW! eBook www.wowebook.org Theexamplepresentedhereonlyscratchesthesurfaceoflogging.Foramorein-depth explanation,youcanfindaveryniceintroductioninthehowto (https://docs.python.org/3.4/howto/logging.html)sectionoftheofficialPython documentation. Loggingisanart,youneedtofindagoodbalancebetweenloggingeverythingand loggingnothing.Ideally,youshouldloganythingthatyouneedtomakesureyour applicationisworkingcorrectly,andpossiblyallerrorsorexceptions. WOW! eBook www.wowebook.org Othertechniques Inthisfinalsection,I’dliketodemonstratebrieflyacoupleoftechniquesthatyoumay finduseful. Profiling WetalkedaboutprofilinginChapter7,Testing,Profiling,andDealingwithExceptions, andI’monlymentioningitherebecauseprofilingcansometimesexplainweirderrorsthat areduetoacomponentbeingtooslow.Especiallywhennetworkingisinvolved,having anideaofthetimingsandlatenciesyourapplicationhastogothroughisveryimportantin ordertounderstandwhatmaybegoingonwhenproblemsarise,thereforeIsuggestyou getacquaintedwithprofilingtechniquesalsoforatroubleshootingperspective. Assertions Assertionsareanicewaytomakeyourcodeensureyourassumptionsareverified.Ifthey are,allproceedsregularlybut,iftheyarenot,yougetaniceexceptionthatyoucanwork with.Sometimes,insteadofinspecting,it’squickertodropacoupleofassertionsinthe codejusttoexcludepossibilities.Let’sseeanexample: assertions.py mylist=[1,2,3]#thisideallycomesfromsomeplace assert4==len(mylist)#thiswillbreak forpositioninrange(4): print(mylist[position]) Thiscodesimulatesasituationinwhichmylistisn’tdefinedbyuslikethat,ofcourse,but we’reassumingithasfourelements.Soweputanassertionthere,andtheresultisthis: $pythonassertions.py Traceback(mostrecentcalllast): File"assertions.py",line3,in<module> assert4==len(mylist) AssertionError Thistellsusexactlywheretheproblemis. WOW! eBook www.wowebook.org Wheretofindinformation InthePythonofficialdocumentation,thereisasectiondedicatedtodebuggingand profiling,whereyoucanreadupaboutthebdbdebuggerframework,andaboutmodules suchasfaulthandler,timeit,trace,tracemallock,andofcoursepdb.Justheadtothe standardlibrarysectioninthedocumentationandyou’llfindallthisinformationvery easily. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Troubleshootingguidelines Inthisshortsection,I’llliketogiveyouafewtipsthatcomefrommytroubleshooting experience. WOW! eBook www.wowebook.org Usingconsoleeditors First,getcomfortableusingvimornanoasaneditor,andlearnthebasicsoftheconsole. Whenthingsbreakbadyoudon’thavetheluxuryofyoureditorwithallthebellsand whistlesthere.Youhavetoconnecttoaboxandworkfromthere.Soit’saverygoodidea tobecomfortablebrowsingyourproductionenvironmentwithconsolecommands,andbe abletoeditfilesusingconsole-basededitorssuchasvi,vim,ornano.Don’tletyourusual developmentenvironmentspoilyou,becauseyou’llhavetopayapriceifyoudo. WOW! eBook www.wowebook.org Wheretoinspect Mysecondsuggestionisonwheretoplaceyourdebuggingbreakpoints.Itdoesn’tmatter ifyouareusingprint,acustomfunction,oripdb,youstillhavetochoosewheretoplace thecallsthatprovideyouwiththeinformation,right? Well,someplacesarebetterthanothers,andtherearewaystohandlethedebugging progressionthatarebetterthanothers. Inormallyavoidplacingabreakpointinanifclausebecause,ifthatclauseisnot exercised,IlosethechanceofgettingtheinformationIwanted.Sometimesit’snoteasyor quicktogettothebreakpoint,sothinkcarefullybeforeplacingthem. Anotherimportantthingiswheretostart.Imaginethatyouhave100linesofcodethat handleyourdata.Datacomesinatline1,andsomehowit’swrongatline100.Youdon’t knowwherethebugis,sowhatdoyoudo?Youcanplaceabreakpointatline1and patientlygothroughallthelines,checkingyourdata.Intheworstcasescenario,99lines later(andmanycoffeecups)youspotthebug.So,considerusingadifferentapproach. Youstartatline50,andinspect.Ifthedataisgood,itmeansthebughappenslater,in whichcaseyouplaceyournextbreakpointatline75.Ifthedataatline50isalreadybad, yougoonbyplacingabreakpointatline25.Then,yourepeat.Eachtime,youmove eitherbackwardsorforwards,byhalfthejumpyoudidlasttime. Inourworstcasescenario,yourdebuggingwouldgofrom1,2,3,…,99to50,75,87,93, 96,…,99whichiswayfaster.Infact,it’slogarithmic.Thissearchingtechniqueiscalled binarysearch,it’sbasedonadivideandconquerapproachandit’sveryeffective,sotry tomasterit. WOW! eBook www.wowebook.org Usingteststodebug DoyourememberChapter7,Testing,Profiling,andDealingwithExceptions,abouttests? Well,ifwehaveabugandalltestsarepassing,itmeanssomethingiswrongormissingin ourtestcodebase.So,oneapproachistomodifythetestsinsuchawaythattheycaterfor thenewedgecasethathasbeenspotted,andthenworkyourwaythroughthecode.This approachcanbeverybeneficial,becauseitmakessurethatyourbugwillbecoveredbya testwhenit’sfixed. WOW! eBook www.wowebook.org Monitoring Monitoringisalsoveryimportant.Softwareapplicationscangocompletelycrazyand havenon-deterministichiccupswhentheyencounteredgecasesituationssuchasthe networkbeingdown,aqueuebeingfull,anexternalcomponentbeingunresponsive,and soon.Inthesecases,it’simportanttohaveanideaofwhatwasthebigpicturewhenthe problemhappenedandbeabletocorrelateittosomethingrelatedtoitinasubtle,perhaps mysteriousway. YoucanmonitorAPIendpoints,processes,webpagesavailabilityandloadtime,and basicallyalmosteverythingthatyoucancode.Ingeneral,whenstartinganapplication fromscratch,itcanbeveryusefultodesignitkeepinginmindhowyouwanttomonitor it. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthisshortchapter,wesawdifferenttechniquesandsuggestionstodebugand troubleshootourcode.Debuggingisanactivitythatisalwayspartofasoftware developer’swork,soit’simportanttobegoodatit. Ifapproachedwiththecorrectattitude,itcanbefunandrewarding. Wesawtechniquestoinspectourcodebaseonfunctions,logging,debuggers,traceback information,profiling,andassertions.Wesawsimpleexamplesofmostofthemandwe alsotalkedaboutasetofguidelinesthatwillhelpwhenitcomestofacethefire. Justremembertoalwaysstaycalmandfocused,anddebuggingwillbeeasieralready. Thistoo,isaskillthatneedstobelearnedandit’sthemostimportant.Anagitatedand stressedmindcannotworkproperly,logicallyandcreatively,therefore,ifyoudon’t strengthenit,itwillbehardforyoutoputallofyourknowledgetogooduse. Inthenextchapter,wewillendthebookwithanothersmallprojectwhosegoalistoleave youmorethirstythanyouwerewhenyoustartedthisjourneywithme. Ready? WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Chapter12.SummingUp–AComplete Example “Donotdwellinthepast,donotdreamofthefuture,concentratethemindonthepresentmoment.” —TheShakyamuniBuddha Inthischapter,Iwillshowyouonelastproject.Ifyou’veworkedwellintherestofthe book,thisexampleshouldbeeasy.Itriedmybesttocraftitinawaythatitwillneitherbe toohardforthosewhohaveonlyreadthebook,nortoosimpleforthosewhoalsotookthe timetoworkontheexamples,andmaybehavereaduponthelinksandtopicsIsuggested. WOW! eBook www.wowebook.org Thechallenge Oneproblemthatweallhavethesedaysisrememberingpasswords.Wehavepasswords foreverything:websites,phones,cards,bankaccounts,andsoon.Theamountof informationwehavetomemorizeisjusttoomuch,somanypeopleendupusingthesame passwordoverandoveragain.Thisisverybad,ofcourse,soatsomepoint,toolswere inventedtoalleviatethisproblem.OneofthesetoolsiscalledKeepassX,andbasicallyit workslikethis:youstartthesoftwarebysettingupaspecialpasswordcalledmaster password.Onceinside,youstorearecordforeachpasswordyouneedtomemorize,for example,youre-mailaccount,thebankwebsite,creditcardinformation,andsoon.When youclosethesoftware,itencryptsthedatabaseusedtostoreallthatinformation,sothat thedatacanonlybeaccessedbytheownerofthemasterpassword.Therefore,kindofina LordofTheRingsfashion,byjustowningonepassword,yourulethemall. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Ourimplementation Ourgoalinthischapteristocreatesomethingsimilarbutweb-based,andthewayIwant toimplementitisbywritingtwoapplications. OnewillbeanAPIwritteninFalcon.Itspurposewillbetwofold,itwillbeabletoboth generateandvalidatepasswords.Itwillprovidethecallerwithinformationaboutthe validityandascorewhichshouldindicatehowstrongthepasswordis. ThesecondapplicationisaDjangowebsite,whichwillprovidetheinterfacetohandle records.Eachrecordwillretaininformationsuchastheusername,e-mail,password,URL, andsoon.Itwillshowalistofalltherecords,anditwillallowtheusertocreate,update anddeletethem.Passwordswillbeencryptedbeforebeingstoredinthedatabase. Thepurposeofthewholeprojectis,therefore,tomimicthewayKeepassXworks,even thoughitisinamuchsimplerfashion.Itwillbeuptoyou,ifyoulikethisidea,todevelop itfurtherinordertoaddotherfeaturesandmakeitmoresecure.Iwillmakesuretogive yousomesuggestionsonhowtoextendit,towardstheend. Thischapterwillthereforebequitedense,code-wise.It’sthepriceIhavetopayforgiving youaninterestingexampleinarestrictedamountofspace. Beforewestart,pleasemakesureyouarecomfortablewiththeprojectspresentedin Chapter10,WebDevelopmentDoneRightsothatyou’refamiliarwiththebasicsofweb development.Makesurealsothatyouhaveinstalledallthepippackagesneededforthis project:django,falcon,cryptography,andnose-parameterized.Ifyoudownloadthe sourcecodeforthebook,you’llfindeverythingyouneedtoinstallintherequirements folder,whilethecodeforthischapterwillbeinch12. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org ImplementingtheDjangointerface Ihopeyou’recomfortablewiththeconceptspresentedinChapter10,WebDevelopment DoneRightwhichwasmostlyaboutDjango.Ifyouhaven’treadit,thisisprobablyagood time,beforereadingonhere. WOW! eBook www.wowebook.org Thesetup Inyourrootfolder(ch12,forme),whichwillcontaintherootfortheinterfaceandtheroot fortheAPI,startbyrunningthiscommand: $django-adminstartprojectpwdweb ThiswillcreatethestructureforaDjangoproject,whichweknowwellbynow.I’llshow youthefinalstructureoftheinterfaceprojecthere: $tree-Apwdweb pwdweb ├──db.sqlite3 ├──manage.py ├──pwdweb │├──__init__.py │├──settings.py │├──urls.py │└──wsgi.py └──records ├──admin.py ├──forms.py ├──__init__.py ├──migrations │├──0001_initial.py │└──__init__.py ├──models.py ├──static │└──records │├──css ││└──main.css │└──js │├──api.js │└──jquery-2.1.4.min.js ├──templates │└──records │├──base.html │├──footer.html │├──home.html │├──list.html │├──messages.html │├──record_add_edit.html │└──record_confirm_delete.html ├──templatetags │└──record_extras.py ├──urls.py └──views.py Asusual,don’tworryifyoudon’thaveallthefiles,we’lladdthemgradually.Changeto thepwdwebfolder,andmakesureDjangoiscorrectlysetup:$pythonmanage.py runserver(ignorethewarningaboutunappliedmigrations). Shutdowntheserverandcreateanapp:$pythonmanage.pystartapprecords.Thatis excellent,nowwecanstartcoding.Firstthingsfirst,let’sopenpwdweb/settings.pyand WOW! eBook www.wowebook.org startbyadding'records',attheendoftheINSTALLED_APPtuple(notethatthecommais includedinthecode).Then,goaheadandfixtheLANGUAGE_CODEandTIME_ZONEsettings accordingtoyourpreferenceandfinally,addthefollowinglineatthebottom: ENCRYPTION_KEY=b'qMhPGx-ROWUDr4veh0ybPRL6viIUNe0vcPDmy67x6CQ=' ThisisacustomencryptionkeythathasnothingtodowithDjangosettings,butwewill needitlateron,andthisisthebestplaceforittobe.Don’tworryfornow,we’llgetback toit. WOW! eBook www.wowebook.org Themodellayer Weneedtoaddjustonemodelfortherecordsapplication:Record.Thismodelwill representeachrecordwewanttostoreinthedatabase: records/models.py fromcryptography.fernetimportFernet fromdjango.confimportsettings fromdjango.dbimportmodels classRecord(models.Model): DEFAULT_ENCODING='utf-8' title=models.CharField(max_length=64,unique=True) username=models.CharField(max_length=64) email=models.EmailField(null=True,blank=True) url=models.URLField(max_length=255,null=True,blank=True) password=models.CharField(max_length=2048) notes=models.TextField(null=True,blank=True) created=models.DateTimeField(auto_now_add=True) last_modified=models.DateTimeField(auto_now=True) defencrypt_password(self): self.password=self.encrypt(self.password) defdecrypt_password(self): self.password=self.decrypt(self.password) defencrypt(self,plaintext): returnself.cypher('encrypt',plaintext) defdecrypt(self,cyphertext): returnself.cypher('decrypt',cyphertext) defcypher(self,cypher_func,text): fernet=Fernet(settings.ENCRYPTION_KEY) result=getattr(fernet,cypher_func)( self._to_bytes(text)) returnself._to_str(result) def_to_str(self,bytes_str): returnbytes_str.decode(self.DEFAULT_ENCODING) def_to_bytes(self,s): returns.encode(self.DEFAULT_ENCODING) Firstly,wesettheDEFAULT_ENCODINGclassattributeto'utf-8',whichisthemostpopular typeofencodingfortheweb(andnotonlytheweb).Wesetthisattributeontheclassto avoidhardcodingastringinmorethanoneplace. Then,weproceedtosetupallthemodel’sfields.Asyoucansee,Djangoallowsyouto specifyveryspecificfields,suchasEmailFieldandURLField.Thereasonwhyit’sbetter tousethesespecificfieldsinsteadofaplainandsimpleCharFieldiswe’llgete-mailand URLvalidationforfreewhenwecreateaformforthismodel,whichisbrilliant. WOW! eBook www.wowebook.org Alltheoptionsarequitestandard,andwesawtheminChapter10,WebDevelopment DoneRightbutIwanttopointoutafewthingsanyway.Firstly,titleneedstobeunique sothateachRecordobjecthasauniquetitleandwedon’twanttoriskhavingdoubles. Eachdatabasetreatsstringsalittlebitdifferently,accordingtohowitissetup,which engineitruns,andsoon,soIhaven’tmadethetitlefieldtheprimarykeyforthismodel, whichwouldhavebeenthenaturalthingtodo.Iprefertoavoidthepainofhavingtodeal withweirdstringerrorsandIamhappywithlettingDjangoaddaprimarykeytothe modelautomatically. Anotheroptionyoushouldunderstandisthenull=True,blank=Truecouple.Theformer allowsthefieldtobeNULL,whichmakesitnon-mandatory,whilethesecondallowsitto beblank(thatistosay,anemptystring).TheiruseisquitepeculiarinDjango,soIsuggest youtotakealookattheofficialdocumentationtounderstandexactlyhowtousethem. Finally,thedates:createdneedstohaveauto_add_now=True,whichwillsetthecurrent momentintimeontheobjectwhenit’screated.Ontheotherhand,last_modifiedneeds tobeupdatedeverytimewesavethemodel,hencewesetauto_now=True. Afterthefielddefinitions,thereareafewmethodsforencryptinganddecryptingthe password.Itisalwaysaverybadideatosavepasswordsastheyareinadatabase, thereforeyoushouldalwaysencryptthembeforesavingthem. Normally,whensavingapassword,youencryptitusingaonewayencryptionalgorithm (alsoknownasaonewayhashfunction).Thismeansthat,onceyouhavecreatedthe hash,thereisnowayforyoutorevertitbacktotheoriginalpassword. Thiskindofencryptionisnormallyusedforauthentication:theuserputstheirusername andpasswordinaformand,onsubmission,thecodefetchesthehashfromtheuserrecord inthedatabaseandcomparesitwiththehashofthepasswordtheuserhasjustputinthe form.Ifthetwohashesmatch,itmeansthattheywereproducedbythesamepassword, thereforeauthenticationisgranted. Inthiscasethough,weneedtobeabletorecoverthepasswords,otherwisethiswhole applicationwouldn’tbeveryuseful.Therefore,wewilluseaso-calledsymmetric encryptionalgorithmtoencryptthem.Thewaythisworksisverysimple:thepassword (calledplaintext)ispassedtoanencryptfunction,alongwithasecretkey.Thealgorithm producesanencryptedstring(calledcyphertext)outofthem,whichiswhatyoustorein thedatabase.Whenyouwanttorecoverthepassword,youwillneedthecyphertextand thesecretkey.Youfeedthemtoadecryptfunction,andyougetbackyouroriginal password.Thisisexactlywhatweneed. Inordertoperformsymmetricencryption,weneedthecryptographypackage,whichis whyIinstructedyoutoinstallit. AllthemethodsintheRecordclassareverysimple.encrypt_passwordand decrypt_passwordareshortcutstoencryptanddecryptthepasswordfieldandreassign theresulttoitself. Theencryptanddecryptmethodsaredispatchersforthecyphermethod,and_to_str WOW! eBook www.wowebook.org and_to_bytesarejustacoupleofhelpers.Thecryptographylibraryworkswithbytes objects,soweneedthosehelperstogobackandforthbetweenbytesandstrings,usinga commonencoding. Theonlyinterestinglogicisinthecyphermethod.Icouldhavecodeditdirectlyinthe encryptanddecryptones,butthatwouldhaveresultedinabitofredundancy,andI wouldn’thavehadthechancetoshowyouadifferentwayofaccessinganobject’s attribute,solet’sanalyzethebodyofcypher. WestartbycreatinganinstanceoftheFernetclass,whichprovidesuswiththesymmetric encryptionfunctionalityweneed.Wesettheinstanceupbypassingthesecretkeyinthe settings(ENCRYPTION_KEY).Aftercreatingfernet,weneedtouseit.Wecanuseitto eitherencryptordecrypt,accordingtowhatvalueisgiventothecypher_funcparameter. Weusegetattrtogetanattributefromanobjectgiventheobjectitselfandthenameof theattribute.Thistechniqueallowsustofetchanyattributefromanobjectdynamically. Theresultofgetattr(fernet,cypher_func),withcyper_funcbeing'encrypt',for example,isthesameasfernet.encrypt.Thegetattrfunctionreturnsamethod,which wethencallwiththebytesrepresentationofthetextargument.Wethenreturntheresult, instringformat. Here’swhatthisfunctionisequivalenttowhenit’scalledbytheencryptdispatcher: defcypher_encrypt(self,text): fernet=Fernet(settings.ENCRYPTION_KEY) result=fernet.encrypt( self._to_bytes(text)) returnself._to_str(result) Whenyoutakethetimetounderstanditproperly,you’llseeit’snotasdifficultasit sounds. So,wehaveourmodel,henceit’stimetomigrate(Ihopeyourememberthatthiswill createthetablesinthedatabaseforyourapplication): $pythonmanage.pymakemigrations $pythonmanage.pymigrate Nowyoushouldhaveanicedatabasewithallthetablesyouneedtoruntheinterface application.Goaheadandcreateasuperuser($pythonmanage.pycreatesuperuser). Bytheway,ifyouwanttogenerateyourownencryptionkey,itisaseasyasthis: >>>fromcryptography.fernetimportFernet >>>Fernet.generate_key() WOW! eBook www.wowebook.org Asimpleform WeneedaformfortheRecordmodel,sowe’llusetheModelFormtechniquewesawin Chapter10,WebDevelopmentDoneRight. records/forms.py fromdjango.formsimportModelForm,Textarea from.modelsimportRecord classRecordForm(ModelForm): classMeta: model=Record fields=['title','username','email','url', 'password','notes'] widgets={'notes':Textarea( attrs={'cols':40,'rows':4})} WecreateaRecordFormclassthatinheritsfromModelForm,sothattheformiscreated automaticallythankstotheintrospectioncapabilitiesofDjango.Weonlyspecifywhich modeltouse,whichfieldstodisplay(weexcludethedates,whicharehandled automatically)andweprovideminimalstylingforthedimensionsofthenotesfield, whichwillbedisplayedusingaTextarea(whichisamultilinetextfieldinHTML). WOW! eBook www.wowebook.org Theviewlayer Thereareatotaloffivepagesintheinterfaceapplication:home,recordlist,record creation,recordupdate,andrecorddeleteconfirmation.Hence,therearefiveviewsthat wehavetowrite.Asyou’llseeinamoment,Djangohelpsusalotbygivingusviewswe canreusewithminimumcustomization.Allthecodethatfollowsbelongstothe records/views.pyfile. Importsandhomeview Justtobreaktheice,herearetheimportsandtheviewforthehomepage: fromdjango.contribimportmessages fromdjango.contrib.messages.viewsimportSuccessMessageMixin fromdjango.core.urlresolversimportreverse_lazy fromdjango.views.genericimportTemplateView fromdjango.views.generic.editimport( CreateView,UpdateView,DeleteView) from.formsimportRecordForm from.modelsimportRecord classHomeView(TemplateView): template_name='records/home.html' WeimportafewtoolsfromDjango.Thereareacoupleofmessaging-relatedobjects,a URLlazyreverser,andfourdifferenttypesofview.WealsoimportourRecordmodeland RecordForm.Asyoucansee,theHomeViewclassconsistsofonlytwolinessinceweonly needtospecifywhichtemplatewewanttouse,therestjustreusesthecodefrom TemplateView,asitis.It’ssoeasy,italmostfeelslikecheating. Listingallrecords Afterthehomeview,wecanwriteaviewtolistalltheRecordinstancesthatwehavein thedatabase. classRecordListView(TemplateView): template_name='records/list.html' defget(self,request,*args,**kwargs): context=self.get_context_data(**kwargs) records=Record.objects.all().order_by('title')#1 forrecordinrecords: record.plaintext=record.decrypt(record.password)#2 context['records']=records returnself.render_to_response(context) Allweneedtodoissub-classTemplateViewagain,andoverridethegetmethod.Weneed todoacoupleofthings:wefetchalltherecordsfromthedatabaseandsortthembytitle (#1)andthenparsealltherecordsinordertoaddtheattributeplaintext(#2)ontoeach ofthem,toshowtheoriginalpasswordonthepage.Anotherwayofdoingthiswouldbe toaddaread-onlypropertytotheRecordmodel,todothedecryptiononthefly.I’llleave ittoyou,asafunexercise,toamendthecodetodoit. WOW! eBook www.wowebook.org Afterrecoveringandaugmentingtherecords,weputtheminthecontextdictandfinish asusualbyinvokingrender_to_response. Creatingrecords Here’sthecodeforthecreationview: classEncryptionMixin: defform_valid(self,form): self.encrypt_password(form) returnsuper(EncryptionMixin,self).form_valid(form) defencrypt_password(self,form): self.object=form.save(commit=False) self.object.encrypt_password() self.object.save() classRecordCreateView( EncryptionMixin,SuccessMessageMixin,CreateView): template_name='records/record_add_edit.html' form_class=RecordForm success_url=reverse_lazy('records:add') success_message='Recordwascreatedsuccessfully' Apartofitslogichasbeenfactoredoutinordertobereusedlateronintheupdateview. Let’sstartwithEncryptionMixin.Allitdoesisoverridetheform_validmethodsothat, priortosavinganewRecordinstancetothedatabase,wemakesurewecall encrypt_passwordontheobjectthatresultsfromsavingtheform.Inotherwords,when theusersubmitstheformtocreateanewRecord,iftheformvalidatessuccessfully,then theform_validmethodisinvoked.Withinthismethodwhatusuallyhappensisthatan objectiscreatedoutoftheModelForminstance,likethis: self.object=form.save() Weneedtointerferewiththisbehaviorbecauserunningthiscodeasitiswouldsavethe recordwiththeoriginalpassword,whichisn’tencrypted.Sowechangethistocallsave ontheformpassingcommit=False,whichcreatestheRecordinstanceoutoftheform,but doesn’tattempttosaveitinthedatabase.Immediatelyafterwards,weencryptthe passwordonthatinstanceandthenwecanfinallycallsaveonit,actuallycommittingitto thedatabase. Sinceweneedthisbehaviorbothforcreatingandupdatingrecords,Ihavefactoreditout inamixin. Note Perhaps,abettersolutionforthispasswordencryptionlogicistocreateacustomField (inheritingfromCharFieldistheeasiestwaytodoit)andaddthenecessarylogictoit,so thatwhenwehandleRecordinstancesfromandtothedatabase,theencryptionand decryptionlogicisperformedautomaticallyforus.Thoughmoreelegant,thissolution needsmetodigressandexplainalotmoreaboutDjangointernals,whichistoomuchfor theextentofthisexample.Asusual,youcantrytodoityourself,ifyoufeellikea WOW! eBook www.wowebook.org challenge. AftercreatingtheEncryptionMixinclass,wecanuseitintheRecordCreateViewclass. Wealsoinheritfromtwootherclasses:SuccessMessageMixinandCreateView.The messagemixinprovidesuswiththelogictoquicklysetupamessagewhencreationis successful,andtheCreateViewgivesusthenecessarylogictocreateanobjectfroma form. Youcanseethatallwehavetocodeissomecustomization:thetemplatename,theform class,andthesuccessmessageandURL.Everythingelseisgracefullyhandledforusby Django. Updatingrecords ThecodetoupdateaRecordinstanceisonlyatinybitmorecomplicated.Wejustneedto addsomelogictodecryptthepasswordbeforewepopulatetheformwiththerecorddata. classRecordUpdateView( EncryptionMixin,SuccessMessageMixin,UpdateView): template_name='records/record_add_edit.html' form_class=RecordForm model=Record success_message='Recordwasupdatedsuccessfully' defget_context_data(self,**kwargs): kwargs['update']=True returnsuper( RecordUpdateView,self).get_context_data(**kwargs) defform_valid(self,form): self.success_url=reverse_lazy( 'records:edit', kwargs={'pk':self.object.pk} ) returnsuper(RecordUpdateView,self).form_valid(form) defget_form_kwargs(self): kwargs=super(RecordUpdateView,self).get_form_kwargs() kwargs['instance'].decrypt_password() returnkwargs Inthisview,westillinheritfrombothEncryptionMixinandSuccessMessageMixin,but theviewclassweuseisUpdateView. Thefirstfourlinesarecustomizationasbefore,wesetthetemplatename,theformclass, theRecordmodel,andthesuccessmessage.Wecannotsetthesuccess_urlasaclass attributebecausewewanttoredirectasuccessfuledittothesameeditpageforthatrecord and,inordertodothis,weneedtheIDoftheinstancewe’reediting.Noworries,we’lldo itanotherway. First,weoverrideget_context_datainordertoset'update'toTrueinthekwargs argument,whichmeansthatakey'update'willendupinthecontextdictthatispassed tothetemplateforrenderingthepage.Wedothisbecausewewanttousethesame WOW! eBook www.wowebook.org templateforcreatingandupdatingarecord,thereforewewillusethisvariableinthe contexttobeabletounderstandinwhichsituationweare.Thereareotherwaystodothis butthisoneisquickandeasyandIlikeitbecauseit’sexplicit. Afteroverridingget_context_data,weneedtotakecareoftheURLredirection.Wedo thisintheform_validmethodsinceweknowthat,ifwegetthere,itmeanstheRecord instancehasbeensuccessfullyupdated.Wereversethe'records:edit'view,whichis exactlytheviewwe’reworkingon,passingtheprimarykeyoftheobjectinquestion.We takethatinformationfromself.object.pk. Oneofthereasonsit’shelpfultohavetheobjectsavedontheviewinstanceisthatwecan useitwhenneededwithouthavingtoalterthesignatureofthemanymethodsintheview inordertopasstheobjectaround.Thisdesignisveryhelpfulandallowsustoachievea lotwithveryfewlinesofcode. Thelastthingweneedtodoistodecryptthepasswordontheinstancebeforepopulating theformfortheuser.It’ssimpleenoughtodoitintheget_form_kwargsmethod,where youcanaccesstheRecordinstanceinthekwargsdict,andcalldecrypt_passwordonit. Thisisallweneedtodotoupdatearecord.Ifyouthinkaboutit,theamountofcodewe hadtowriteisreallyverylittle,thankstoDjangoclass-basedviews. Tip Agoodwayofunderstandingwhichisthebestmethodtooverride,istotakealookatthe Djangoofficialdocumentationor,evenbetterinthiscase,checkoutthesourcecodeand lookattheclass-basedviewssection.You’llbeabletoappreciatehowmuchworkhas beendonetherebyDjangodeveloperssothatyouonlyhavetotouchthesmallestamounts ofcodetocustomizeyourviews. Deletingrecords Ofthethreeactions,deletingarecordisdefinitelytheeasiestone.Allweneedisthe followingcode: classRecordDeleteView(SuccessMessageMixin,DeleteView): model=Record success_url=reverse_lazy('records:list') defdelete(self,request,*args,**kwargs): messages.success( request,'Recordwasdeletedsuccessfully') returnsuper(RecordDeleteView,self).delete( request,*args,**kwargs) WeonlyneedtoinheritfromSuccessMessageMixinandDeleteView,whichgivesusall weneed.WesetupthemodelandthesuccessURLasclassattributes,andthenwe overridethedeletemethodonlytoaddanicemessagethatwillbedisplayedinthelist view(whichiswhereweredirecttoafterdeletion). Wedon’tneedtospecifythetemplatename,sincewe’lluseanamethatDjangoinfersby default:record_confirm_delete.html. WOW! eBook www.wowebook.org Withthisfinalview,we’reallsettohaveaniceinterfacethatwecanusetohandleRecord instances. WOW! eBook www.wowebook.org SettinguptheURLs Beforewemoveontothetemplatelayer,let’ssetuptheURLs.Thistime,Iwanttoshow youtheinclusiontechniqueItalkedaboutinChapter10,WebDevelopmentDoneRight. pwdweb/urls.py fromdjango.conf.urlsimportinclude,url fromdjango.contribimportadmin fromrecordsimporturlsasrecords_url fromrecords.viewsimportHomeView urlpatterns=[ url(r'^admin/',include(admin.site.urls)), url(r'^records/',include(records_url,namespace='records')), url(r'^$',HomeView.as_view(),name='home'), ] ThesearetheURLsforthemainproject.Wehavetheusualadmin,ahomepage,andthen fortherecordssection,weincludeanotherurls.pyfile,whichwedefineintherecords application.Thistechniqueallowsforappstobereusableandself-contained.Notethat, whenincludinganotherurls.pyfile,youcanpassnamespaceinformation,whichyoucan thenuseinfunctionssuchasreverse,ortheurltemplatetag.Forexample,we’veseen thatthepathtotheRecordUpdateViewwas'records:edit'.Thefirstpartofthatstringis thenamespace,andthesecondisthenamethatwehavegiventotheview,asyoucansee inthefollowingcode: records/urls.py fromdjango.conf.urlsimportinclude,url fromdjango.contribimportadmin from.viewsimport(RecordCreateView,RecordUpdateView, RecordDeleteView,RecordListView) urlpatterns=[ url(r'^add/$',RecordCreateView.as_view(),name='add'), url(r'^edit/(?P<pk>[0-9]+)/$',RecordUpdateView.as_view(), name='edit'), url(r'^delete/(?P<pk>[0-9]+)/$',RecordDeleteView.as_view(), name='delete'), url(r'^$',RecordListView.as_view(),name='list'), ] Wedefinefourdifferenturlinstances.Thereisoneforaddingarecord,whichdoesn’t needprimarykeyinformationsincetheobjectdoesn’texistyet.Thenwehavetwourl instancesforupdatinganddeletingarecord,andforthoseweneedtoalsospecifyprimary keyinformationtobepassedtotheview.SinceRecordinstanceshaveintegerIDs,wecan safelypassthemontheURL,followinggoodURLdesignpractice.Finally,wedefineone urlinstanceforthelistofrecords. Allurlinstanceshavenameinformationwhichisusedinviewsandtemplates. WOW! eBook www.wowebook.org Thetemplatelayer Let’sstartwiththetemplatewe’lluseasthebasisfortherest: records/templates/records/base.html {%loadstaticfromstaticfiles%} <!DOCTYPEhtml> <htmllang="en"> <head> <metacharset="utf-8"> <metaname="viewport" content="width=device-width,initial-scale=1.0"> <linkhref="{%static"records/css/main.css"%}" rel="stylesheet"> <title>{%blocktitle%}Title{%endblocktitle%}</title> </head> <body> <divid="page-content"> {%blockpage-content%}{%endblockpage-content%} </div> <divid="footer">{%blockfooter%}{%endblockfooter%}</div> {%blockscripts%} <script src="{%static"records/js/jquery-2.1.4.min.js"%}"> </script> {%endblockscripts%} </body> </html> It’sverysimilartotheoneIusedinChapter10,WebDevelopmentDoneRightalthoughit isabitmorecompressedandwithonemajordifference.WewillimportjQueryinevery page. Note jQueryisthemostpopularJavaScriptlibraryoutthere.Itallowsyoutowritecodethat worksonallthemainbrowsersanditgivesyoumanyextratoolssuchastheabilityto performasynchronouscalls(AJAX)fromthebrowseritself.We’llusethislibraryto performthecallstotheAPI,bothtogenerateandvalidateourpasswords.Youcan downloaditathttps://jquery.com/,andputitinthepwdweb/records/static/records/js/ folder(youmayhavetoamendtheimportinthetemplate). Ihighlightedtheonlyinterestingpartofthistemplateforyou.Notethatweloadthe JavaScriptlibraryattheend.Thisiscommonpractice,asJavaScriptisusedtomanipulate thepage,soloadinglibrariesattheendhelpsinavoidingsituationssuchasJavaScript codefailingbecausetheelementneededhadn’tbeenrenderedonthepageyet. Homeandfootertemplates Thehometemplateisverysimple: records/templates/records/home.html WOW! eBook www.wowebook.org {%extends"records/base.html"%} {%blocktitle%}WelcometotheRecordswebsite.{%endblock%} {%blockpage-content%} <h1>Welcome{{user.first_name}}!</h1> <divclass="home-option">Tocreatearecordclick <ahref="{%url"records:add"%}">here.</a> </div> <divclass="home-option">Toseeallrecordsclick <ahref="{%url"records:list"%}">here.</a> </div> {%endblockpage-content%} Thereisnothingnewherewhencomparedtothehome.htmltemplatewesawinChapter 10,WebDevelopmentDoneRight.Thefootertemplateisactuallyexactlythesame: records/templates/records/footer.html <divclass="footer"> Goback<ahref="{%url"home"%}">home</a>. </div> Listingallrecords Thistemplatetolistallrecordsisfairlysimple: records/templates/records/list.html {%extends"records/base.html"%} {%loadrecord_extras%} {%blocktitle%}Records{%endblocktitle%} {%blockpage-content%} <h1>Records</h1><spanname="top"></span> {%include"records/messages.html"%} {%forrecordinrecords%} <divclass="record{%cycle'row-light-blue''row-white'%}" id="record-{{record.pk}}"> <divclass="record-left"> <divclass="record-list"> <spanclass="record-span">Title</span>{{record.title}} </div> <divclass="record-list"> <spanclass="record-span">Username</span> {{record.username}} </div> <divclass="record-list"> <spanclass="record-span">Email</span>{{record.email}} </div> <divclass="record-list"> <spanclass="record-span">URL</span> <ahref="{{record.url}}"target="_blank"> {{record.url}}</a> </div> <divclass="record-list"> <spanclass="record-span">Password</span> WOW! eBook www.wowebook.org {%hide_passwordrecord.plaintext%} </div> </div> <divclass="record-right"> <divclass="record-list"> <spanclass="record-span">Notes</span> <textarearows="3"cols="40"class="record-notes" readonly>{{record.notes}}</textarea> </div> <divclass="record-list"> <spanclass="record-span">Lastmodified</span> {{record.last_modified}} </div> <divclass="record-list"> <spanclass="record-span">Created</span> {{record.created}} </div> </div> <divclass="record-list-actions"> <ahref="{%url"records:edit"pk=record.pk%}">»edit</a> <ahref="{%url"records:delete"pk=record.pk%}">»delete </a> </div> </div> {%endfor%} {%endblockpage-content%} {%blockfooter%} <p><ahref="#top">Gobacktotop</a></p> {%include"records/footer.html"%} {%endblockfooter%} Forthistemplateaswell,IhavehighlightedthepartsI’dlikeyoutofocuson.Firstly,I loadacustomtagsmodule,record_extras,whichwe’llneedlater.Ihavealsoaddedan anchoratthetop,sothatwe’llbeabletoputalinktoitatthebottomofthepage,toavoid havingtoscrollallthewayup. Then,IincludedatemplatetoprovidemewiththeHTMLcodetodisplayDjango messages.It’saverysimpletemplatewhichI’llshowyoushortly. Then,wedefinealistofdivelements.EachRecordinstancehasacontainerdiv,inwhich therearetwoothermaindivelements:record-leftandrecord-right.Inorderto displaythemsidebyside,Ihavesetthisclassinthemain.cssfile: .record-left{float:left;width:300px;} Theoutermostdivcontainer(theonewithclassrecord),hasanidattribute,whichIhave usedasananchor.Thisallowsustoclickoncancelontherecorddeletepage,sothatif wechangeourmindsanddon’twanttodeletetherecord,wecangetbacktothelistpage, andattherightposition. Eachattributeoftherecordisthendisplayedindivelementswhoseclassisrecord-list. Mostoftheseclassesarejusttheretoallowmetosetabitofpaddinganddimensionson WOW! eBook www.wowebook.org theHTMLelements. Thenextinterestingbitisthehide_passwordtag,whichtakestheplaintext,whichisthe unencryptedpassword.Thepurposeofthiscustomtagistodisplayasequenceof'*' characters,aslongastheoriginalpassword,sothatifsomeoneispassingbywhileyou’re onthepage,theywon’tseeyourpasswords.However,hoveringonthatsequenceof'*' characterswillshowyoutheoriginalpasswordinthetooltip.Here’sthecodeforthe hide_passwordtag: records/templatetags/record_extras.py fromdjangoimporttemplate fromdjango.utils.htmlimportescape register=template.Library() @register.simple_tag defhide_password(password): return'<spantitle="{0}">{1}</span>'.format( escape(password),'*'*len(password)) Thereisnothingfancyhere.Wejustregisterthisfunctionasasimpletagandthenwecan useitwhereverwewant.Ittakesapasswordandputsitasatooltipofaspanelement, whosemaincontentisasequenceof'*'characters.Justnoteonething:weneedto escapethepassword,sothatwe’resureitwon’tbreakourHTML(thinkofwhatmight happenifthepasswordcontainedadouble-quote`"`,forexample). Asfarasthelist.htmltemplateisconcerned,thenextinterestingbitisthatwesetthe readonlyattributetothetextareaelement,soasnottogivetheimpressiontotheuser thattheycanmodifynotesonthefly. Then,wesetacoupleoflinksforeachRecordinstance,rightatthebottomofthe containerdiv.Thereisonefortheeditpage,andanotherforthedeletepage.Notethatwe needtopasstheurltagnotonlythenamespace:namestring,butalsotheprimarykey information,asrequiredbytheURLsetupwemadeintheurls.pymoduleforthose views. Finally,weimportthefooterandsetthelinktotheanchorontopofthepage. Now,aspromised,hereisthecodeforthemessages: records/templates/records/messages.html {%ifmessages%} {%formessageinmessages%} <pclass="{{message.tags}}">{{message}}</p> {%endfor%} {%endif%} Thiscodetakescareofdisplayingmessagesonlywhenthereisatleastonetodisplay.We givetheptagclassinformationtodisplaysuccessmessagesingreenanderrormessages inred. Ifyougrabthemain.cssfilefromthesourcecodeforthebook,youwillnowbeableto WOW! eBook www.wowebook.org visualizethelistpage(yourswillbeblank,youstillneedtoinsertdataintoit),andit shouldlooksomethinglikethis: Asyoucansee,Ihavetworecordsinthedatabaseatthemoment.I’mhoveringonthe passwordofthefirstone,whichismyplatformaccountatmysister’sschool,andthe passwordisdisplayedinthetooltip.Thedivisionintwodivelements,leftandright,helps inmakingrowssmallersothattheoverallresultismorepleasingtotheeye.The importantinformationisontheleftandtheancillaryinformationisontheright.Therow coloralternatesbetweenaverylightshadeofblueandwhite. Eachrowhasaneditanddeletelink,atitsbottomleft.We’llshowthepagesforthosetwo linksrightafterweseethecodeforthetemplatesthatcreatethem. TheCSScodethatholdsalltheinformationforthisinterfaceisthefollowing: records/static/records/css/main.css html,body,*{ font-family:'TrebuchetMS',Helvetica,sans-serif;} a{color:#333;} .record{ clear:both;padding:1em;border-bottom:1pxsolid#666;} .record-left{float:left;width:300px;} .record-list{padding:2px0;} .fieldWrapper{padding:5px;} .footer{margin-top:1em;color:#333;} .home-option{padding:.6em0;} WOW! eBook www.wowebook.org .record-span{font-weight:bold;padding-right:1em;} .record-notes{vertical-align:top;} .record-list-actions{padding:4px0;clear:both;} .record-list-actionsa{padding:04px;} #pwd-info{padding:06px;font-size:1.1em;font-weight:bold;} #id_notes{vertical-align:top;} /*Messages*/ .success,.errorlist{font-size:1.2em;font-weight:bold;} .success{color:#25B725;} .errorlist{color:#B12B2B;} /*colors*/ .row-light-blue{background-color:#E6F0FA;} .row-white{background-color:#fff;} .green{color:#060;} .orange{color:#FF3300;} .red{color:#900;} Pleaseremember,I’mnotaCSSgurusojusttakethisfileasitis,afairlynaivewayto providestylingtoourinterface. Creatingandeditingrecords Nowfortheinterestingpart.Creatingandupdatingarecord.We’llusethesametemplate forboth,soweexpectsomedecisionallogictobetherethatwilltellusinwhichofthe twosituationsweare.Asitturnsout,itwillnotbethatmuchcode.Themostexcitingpart ofthistemplate,however,isitsassociatedJavaScriptfilewhichwe’llexamineright afterwards. records/templates/records/record_add_edit.html {%extends"records/base.html"%} {%loadstaticfromstaticfiles%} {%blocktitle%} {%ifupdate%}Update{%else%}Create{%endif%}Record {%endblocktitle%} {%blockpage-content%} <h1>{%ifupdate%}Updatea{%else%}Createanew{%endif%} Record </h1> {%include"records/messages.html"%} <formaction="."method="post">{%csrf_token%} {{form.non_field_errors}} <divclass="fieldWrapper">{{form.title.errors}} {{form.title.label_tag}}{{form.title}}</div> <divclass="fieldWrapper">{{form.username.errors}} {{form.username.label_tag}}{{form.username}}</div> <divclass="fieldWrapper">{{form.email.errors}} {{form.email.label_tag}}{{form.email}}</div> <divclass="fieldWrapper">{{form.url.errors}} {{form.url.label_tag}}{{form.url}}</div> WOW! eBook www.wowebook.org <divclass="fieldWrapper">{{form.password.errors}} {{form.password.label_tag}}{{form.password}} <spanid="pwd-info"></span></div> <buttontype="button"id="validate-btn"> ValidatePassword</button> <buttontype="button"id="generate-btn"> GeneratePassword</button> <divclass="fieldWrapper">{{form.notes.errors}} {{form.notes.label_tag}}{{form.notes}}</div> <inputtype="submit" value="{%ifupdate%}Update{%else%}Insert{%endif%}"> </form> {%endblockpage-content%} {%blockfooter%} <br>{%include"records/footer.html"%}<br> Goto<ahref="{%url"records:list"%}">therecordslist</a>. {%endblockfooter%} {%blockscripts%} {{block.super}} <scriptsrc="{%static"records/js/api.js"%}"></script> {%endblockscripts%} Asusual,Ihavehighlightedtheimportantparts,solet’sgothroughthiscodetogether. Youcanseethefirstbitofdecisionlogicinthetitleblock.Similardecisionlogicisalso displayedlateron,intheheaderofthepage(theh1HTMLtag),andinthesubmitbutton attheendoftheform. Apartfromthislogic,whatI’dlikeyoutofocusonistheformandwhat’sinsideit.Weset theactionattributetoadot,whichmeansthispage,sothatwedon’tneedtocustomizeit accordingtowhichviewisservingthepage.Also,weimmediatelytakecareofthecrosssiterequestforgerytoken,asexplainedinChapter10,WebDevelopmentDoneRight. Notethat,thistime,wecannotleavethewholeformrenderinguptoDjangosincewe wanttoaddinacoupleofextrathings,sowegodownonelevelofgranularityandask Djangotorendereachindividualfieldforus,alongwithanyerrors,alongwithitslabel. Thiswaywestillsavealotofeffort,andatthesametime,wecanalsocustomizetheform aswelike.Insituationslikethis,it’snotuncommontowriteasmalltemplatetorendera field,inordertoavoidrepeatingthosethreelinesforeachfield.Inthiscasethough,the formissosmallIdecidedtoavoidraisingthecomplexitylevelupanyfurther. Thespanelement,pwd-info,containstheinformationaboutthepasswordthatweget fromtheAPI.Thetwobuttonsafterthat,validate-btnandgenerate-btn,arehookedup withtheAJAXcallstotheAPI. Attheendofthetemplate,inthescriptsblock,weneedtoloadtheapi.jsJavaScript filewhichcontainsthecodetoworkwiththeAPI.Wealsoneedtouseblock.super, WOW! eBook www.wowebook.org whichwillloadwhatevercodeisinthesameblockintheparenttemplate(forexample, jQuery).block.superisbasicallythetemplateequivalentofacalltosuper(ClassName, self)inPython.It’simportanttoloadjQuerybeforeourlibrary,sincethelatterisbased ontheformer. TalkingtotheAPI Let’snowtakealookatthatJavaScript.Idon’texpectyoutounderstandeverything. Firstly,thisisaPythonbookandsecondly,you’resupposedtobeabeginner(thoughby now,ninjatrained),sofearnot.However,asJavaScripthas,bynow,becomeessentialif you’redealingwithawebenvironment,havingaworkingknowledgeofitisextremely importantevenforaPythondeveloper,sotryandgetthemostoutofwhatI’maboutto showyou.We’llseethepasswordgenerationfirst: records/static/records/js/api.js varbaseURL='http://127.0.0.1:5555/password'; vargetRandomPassword=function(){ varapiURL='{url}/generate'.replace('{url}',baseURL); $.ajax({ type:'GET', url:apiURL, success:function(data,status,request){ $('#id_password').val(data[1]); }, error:function(){alert('Unexpectederror');} }); } $(function(){ $('#generate-btn').click(getRandomPassword); }); Firstly,wesetavariableforthebaseAPIURL:baseURL.Then,wedefinethe getRandomPasswordfunction,whichisverysimple.Atthebeginning,itdefinesthe apiURLextendingbaseURLwithareplacementtechnique.Evenifthesyntaxisdifferent fromthatofPython,youshouldn’thaveanyissuesunderstandingthisline. AfterdefiningtheapiURL,theinterestingbitcomesup.Wecall$.ajax,whichisthe jQueryfunctionthatperformstheAJAXcalls.That$isashortcutforjQuery.Asyoucan seeinthebodyofthecall,it’saGETrequesttoapiURL.Ifitsucceeds(success:…),an anonymousfunctionisrun,whichsetsthevalueoftheid_passwordtextfieldtothe secondelementofthereturneddata.We’llseethestructureofthedatawhenweexamine theAPIcode,sodon’tworryaboutthatnow.Ifanerroroccurs,wesimplyalerttheuser thattherewasanunexpectederror. Note ThereasonwhythepasswordfieldintheHTMLhasid_passwordastheIDisduetothe wayDjangorendersforms.Youcancustomizethisbehaviorusingacustomprefix,for example.Inthiscase,I’mhappywiththeDjangodefaults. WOW! eBook www.wowebook.org Afterthefunctiondefinition,werunacoupleoflinesofcodetobindtheclickeventon thegenerate-btnbuttontothegetRandomPasswordfunction.Thismeansthat,afterthis codehasbeenrunbythebrowserengine,everytimeweclickthegenerate-btnbutton, thegetRandomPasswordfunctioniscalled. Thatwasn’tsoscary,wasit?Solet’sseewhatweneedforthevalidationpart. Nowthereisavalueinthepasswordfieldandwewanttovalidateit.Weneedtocallthe APIandinspectitsresponse.Sincepasswordscanhaveweirdcharacters,Idon’twantto passthemontheURL,thereforeIwilluseaPOSTrequest,whichallowsmetoputthe passwordinitsbody.Todothis,Ineedthefollowingcode: varvalidatePassword=function(){ varapiURL='{url}/validate'.replace('{url}',baseURL); $.ajax({ type:'POST', url:apiURL, data:JSON.stringify({'password':$('#id_password').val()}), contentType:"text/plain",//AvoidCORSpreflight success:function(data,status,request){ varvalid=data['valid'],infoClass,grade; varmsg=(valid?'Valid':'Invalid')+'password.'; if(valid){ varscore=data['score']['total']; grade=(score<10?'Poor':(score<18?'Medium':'Strong')); infoClass=(score<10?'red':(score<18?'orange':'green')); msg+='(Score:{score},{grade})' .replace('{score}',score).replace('{grade}',grade); } $('#pwd-info').html(msg); $('#pwd-info').removeClass().addClass(infoClass); }, error:function(data){alert('Unexpectederror');} }); } $(function(){ $('#validate-btn').click(validatePassword); }); Theconceptisthesameasbefore,onlythistimeit’sforthevalidate-btnbutton.The bodyoftheAJAXcallissimilar.WeuseaPOSTinsteadofaGETrequest,andwedefine thedataasaJSONobject,whichistheequivalentofusingjson.dumps({'password': 'some_pwd'})inPython. ThecontentTypelineisaquickhacktoavoidproblemswiththeCORSpreflightbehavior ofthebrowser.Cross-originresourcesharing(CORS)isamechanismthatallows restrictedresourcesonawebpagetoberequestedfromanotherdomainoutsideofthe domainfromwhichtherequestoriginated.Inanutshell,sincetheAPIislocatedat 127.0.0.1:5555andtheinterfaceisrunningat127.0.0.1:8000,withoutthishack,the browserwouldn’tallowustoperformthecalls.Inaproductionenvironment,youmay wanttocheckthedocumentationforJSONP,whichisamuchbetter(albeitmore WOW! eBook www.wowebook.org complex)solutiontothisissue. Thebodyoftheanonymousfunctionwhichisrunifthecallsucceedsisapparentlyonlya bitcomplicated.Allweneedtodoisunderstandifthepasswordisvalid(from data['valid']),andassignitagradeandaCSSclassbasedonitsscore.Validityand scoreinformationcomefromtheAPIresponse. TheonlytrickybitinthiscodeistheJavaScriptternaryoperator,solet’sseea comparativeexampleforit: #Python error='critical'iferror_level>50else'medium' //JavaScriptequivalent error=(error_level>50?'critical':'medium'); Withthisexample,youshouldn’thaveanyissuereadingtherestofthelogicinthe function.Iknow,Icouldhavejustusedaregularif(...),butJavaScriptcodersusethe ternaryoperatorallthetime,soyoushouldgetusedtoit.It’sgoodtrainingtoscratchour headsabitharderinordertounderstandcode. Lastly,I’dlikeyoutotakealookattheendofthatfunction.Wesetthehtmlofthepwdinfospanelementtothemessageweassembled(msg),andthenwestyleit.Inoneline, weremovealltheCSSclassesfromthatelement(removeClass()withnoparameters doesthat),andweaddtheinfoClasstoit.infoClassiseither'red','orange',or 'green'.Ifyougobacktothemain.cssfile,you’llseethematthebottom. Nowthatwe’veseenboththetemplatecodeandtheJavaScripttomakethecalls,let’ssee ascreenshotofthepage.We’regoingtoeditthefirstrecord,theoneaboutmysister’s school. WOW! eBook www.wowebook.org Inthepicture,youcanseethatIupdatedthepasswordbyclickingontheGenerate Passwordbutton.Then,Isavedtherecord(soyoucouldseethenicemessageontop), and,finally,IclickedontheValidatePasswordbutton. Theresultisshowningreenontheright-handsideofthePasswordfield.It’sstrong(23is actuallythemaximumscorewecanget)sothemessageisdisplayedinaniceshadeof green. Deletingrecords Todeletearecord,gotothelistandclickonthedeletelink.You’llberedirectedtoapage thatasksyouforconfirmation;youcanthenchoosetoproceedanddeletethepoorrecord, ortocanceltherequestandgobacktothelistpage.Thetemplatecodeisthefollowing: records/templates/records/record_confirm_delete.html {%extends"records/base.html"%} {%blocktitle%}Deleterecord{%endblocktitle%} {%blockpage-content%} <h1>ConfirmRecordDeletion</h1> <formaction="."method="post">{%csrf_token%} <p>Areyousureyouwanttodelete"{{object}}"?</p> <inputtype="submit"value="Confirm"/> <ahref="{%url"records:list"%}#record-{{object.pk}}"> »cancel</a> </form> {%endblockpage-content%} SincethisisatemplateforastandardDjangoview,weneedtousethenaming conventionsadoptedbyDjango.Therefore,therecordinquestioniscalledobjectinthe template.The{{object}}tagdisplaysastringrepresentationfortheobject,whichis notexactlybeautifulatthemoment,sincethewholelinewillread:Areyousureyou wanttodelete“Recordobject”?. Thisisbecausewehaven’taddeda__str__methodtoourModelclassyet,whichmeans thatPythonhasnoideaofwhattoshowuswhenweaskforastringrepresentationofan instance.Let’schangethisbycompletingourmodel,addingthe__str__methodatthe bottomoftheclassbody: records/models.py classRecord(models.Model): ... def__str__(self): return'{}'.format(self.title) Restarttheserverandnowthepagewillread:Areyousureyouwanttodelete“Some Bank”?whereSomeBankisthetitleoftherecordwhosedeletelinkIclickedon. Wecouldhavejustused{{object.title}},butIprefertofixtherootoftheproblem, notjusttheeffect.Addinga__str__methodisinfactsomethingthatyououghttodofor WOW! eBook www.wowebook.org allofyourmodels. Theinterestingbitinthislasttemplateisactuallythelinkforcancelingtheoperation.We usetheurltagtogobacktothelistview(records:list),butweaddanchorinformation toitsothatitwilleventuallyreadsomethinglikethis(thisisforpk=2): http://127.0.0.1:8000/records/#record-2 ThiswillgobacktothelistpageandscrolldowntothecontainerdivthathasIDrecord2, whichisnice. Thisconcludestheinterface.Eventhoughthissectionwassimilartowhatwesawin Chapter10,WebDevelopmentDoneRight,we’vebeenabletoconcentratemoreonthe codeinthischapter.We’veseenhowusefulDjangoclass-basedviewsare,andweeven touchedonsomecoolJavaScript.Run$pythonmanage.pyrunserverandyour interfaceshouldbeupandrunningathttp://127.0.0.1:8000. Note Ifyouarewondering,127.0.0.1meansthelocalhost—yourcomputer—while8000is theporttowhichtheserverisbound,tolistenforincomingrequests. Nowit’stimetospicethingsupabitwiththesecondpartofthisproject. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org ImplementingtheFalconAPI ThestructureoftheFalconprojectwe’reabouttocodeisnowherenearasextendedasthe interfaceone.We’llcodefivefilesaltogether.Inyourch12folder,createanewonecalled pwdapi.Thisisitsfinalstructure: $tree-Apwdapi/ pwdapi/ ├──core │├──handlers.py │└──passwords.py ├──main.py └──tests └──test_core ├──test_handlers.py └──test_passwords.py TheAPIwasallcodedusingTDD,sowe’realsogoingtoexplorethetests.However,I thinkit’sgoingtobeeasierforyoutounderstandthetestsifyoufirstseethecode,so we’regoingtostartwiththat. WOW! eBook www.wowebook.org Themainapplication ThisisthecodefortheFalconapplication: main.py importfalcon fromcore.handlersimport( PasswordValidatorHandler, PasswordGeneratorHandler, ) validation_handler=PasswordValidatorHandler() generator_handler=PasswordGeneratorHandler() app=falcon.API() app.add_route('/password/validate/',validation_handler) app.add_route('/password/generate/',generator_handler) AsintheexampleinChapter10,WebDevelopmentDoneRight,westartbycreatingone instanceforeachofthehandlersweneed,thenwecreateafalcon.APIobjectand,by callingitsadd_routemethod,wesetuptheroutingtotheURLsofourAPI.We’llgetto thedefinitionsofthehandlersinamoment.Firstly,weneedacoupleofhelpers. WOW! eBook www.wowebook.org Writingthehelpers Inthissection,wewilltakealookatacoupleofclassesthatwe’lluseinourhandlers.It’s alwaysgoodtofactoroutsomelogicfollowingtheSingleResponsibilityPrinciple. Note InOOP,theSingleResponsibilityPrinciple(SRP)statesthateveryclassshouldhave responsibilityforasinglepartofthefunctionalityprovidedbythesoftware,andthat responsibilityshouldbeentirelyencapsulatedbytheclass.Allofitsservicesshouldbe narrowlyalignedwiththatresponsibility. TheSingleResponsibilityPrincipleistheSinS.O.L.I.D.,anacronymforthefirstfive OOPandsoftwaredesignprinciplesintroducedbyRobertMartin. Iheartilysuggestyoutoopenabrowserandreaduponthissubject,itisveryimportant. Allthecodeinthehelperssectionbelongstothecore/passwords.pymodule.Here’show itbegins: frommathimportceil fromrandomimportsample fromstringimportascii_lowercase,ascii_uppercase,digits punctuation='!#$%&()*+-?@_|' allchars=''.join( (ascii_lowercase,ascii_uppercase,digits,punctuation)) We’llneedtohandlesomerandomizedcalculationsbutthemostimportantparthereisthe allowedcharacters.Wewillallowletters,digits,andasetofpunctuationcharacters.To easewritingthecode,wewillmergethosepartsintotheallcharsstring. Codingthepasswordvalidator ThePasswordValidatorclassismyfavoritebitoflogicinthewholeAPI.Itexposesan is_validandascoremethod.Thelatterrunsalldefinedvalidators(“private”methodsin thesameclass),andcollectsthescoresintoasingledictwhichisreturnedasaresult.I’ll writethisclassmethodbymethodsothatitdoesnotgettoocomplicated: classPasswordValidator: def__init__(self,password): self.password=password.strip() Itbeginsbysettingpassword(withnoleadingortrailingspaces)asaninstanceattribute. Thiswaywewon’tthenhavetopassitaroundfrommethodtomethod.Allthemethods thatwillfollowbelongtothisclass. defis_valid(self): return(len(self.password)>0and all(charinallcharsforcharinself.password)) Apasswordisvalidwhenitslengthisgreaterthan0andallofitscharactersbelongtothe allcharsstring.Whenyoureadtheis_validmethod,it’spracticallyEnglish(that’show WOW! eBook www.wowebook.org amazingPythonis).allisabuilt-infunctionthattellsyouifalltheelementsofthe iterableyoufeedtoitareTrue. defscore(self): result={ 'length':self._score_length(), 'case':self._score_case(), 'numbers':self._score_numbers(), 'special':self._score_special(), 'ratio':self._score_ratio(), } result['total']=sum(result.values()) returnresult Thisistheothermainmethod.It’sverysimple,itjustpreparesadictwithalltheresults fromthevalidators.Theonlyindependentbitoflogichappensattheend,whenwesum thegradesfromeachvalidatorandassignittoa'total'keyinthedict,justfor convenience. Asyoucansee,wescoreapasswordbylength,bylettercase,bythepresenceofnumbers, andspecialcharacters,and,finally,bytheratiobetweenlettersandnumbers.Lettersallow acharactertobebetween26*2=52differentpossiblechoices,whiledigitsallowonly 10.Therefore,passwordswhoseletterstodigitsratioishigheraremoredifficulttocrack. Let’sseethelengthvalidator: def_score_length(self): scores_list=([0]*4)+([1]*4)+([3]*4)+([5]*4) scores=dict(enumerate(scores_list)) returnscores.get(len(self.password),7) Weassign0pointstopasswordswhoselengthislessthanfourcharacters,1pointfor thosewhoselengthislessthan8,3foralengthlessthan12,5foralengthlessthan16, and7foralengthof16ormore. Inordertoavoidawaterfallofif/elifclauses,Ihaveadoptedafunctionalstylehere.I preparedascore_list,whichisbasically[0,0,0,0,1,1,1,1,3,...].Then, byenumeratingit,Igota(length,score)pairforeachlengthlessthan16.Iputthosepairs intoadict,whichgivesmetheequivalentindictform,soitshouldlooklikethis:{0:0, 1:0,2:0,3:0,4:1,5:1,...}.Ithenperformagetonthisdictwiththelengthofthe password,settingavalueof7asthedefault(whichwillbereturnedforlengthsof16or more,whicharenotinthedict). Ihavenothingagainstif/elifclauses,ofcourse,butIwantedtotaketheopportunityto showyoudifferentcodingstylesinthisfinalchapter,tohelpyougetusedtoreadingcode whichdeviatesfromwhatyouwouldnormallyexpect.It’sonlybeneficial. def_score_case(self): lower=bool(set(ascii_lowercase)&set(self.password)) upper=bool(set(ascii_uppercase)&set(self.password)) returnint(lowerorupper)+2*(lowerandupper) Thewaywevalidatethecaseisagainwithanicetrick.lowerisTruewhenthe WOW! eBook www.wowebook.org intersectionbetweenthepasswordandalllowercasecharactersisnon-empty,otherwise it’sFalse.upperbehavesinthesameway,onlywithuppercasecharacters. Tounderstandtheevaluationthathappensonthelastline,let’susetheinside-out techniqueoncemore:lowerorupperisTruewhenatleastoneofthetwoisTrue.When it’sTrue,itwillbeconvertedtoa1bytheintclass.Thisequatestosaying,ifthereisat leastonecharacter,regardlessofthecasing,thescoregets1point,otherwiseitstaysat0. Nowforthesecondpart:lowerandupperisTruewhenbothofthemareTrue,which meansthatwehaveatleastonelowercaseandoneuppercasecharacter.Thismeansthat, tocrackthepassword,abrute-forcealgorithmwouldhavetoloopthrough52letters insteadofjust26.Therefore,whenthat’sTrue,wegetanextratwopoints. Thisvalidatorthereforeproducesaresultintherange(0,1,3),dependingonwhatthe passwordis. def_score_numbers(self): return2if(set(self.password)&set(digits))else0 Scoringonthenumbersissimpler.Ifwehaveatleastonenumber,wegettwopoints, otherwiseweget0.Inthiscase,Iusedaternaryoperatortoreturntheresult. def_score_special(self): return4if( set(self.password)&set(punctuation))else0 Thespecialcharactersvalidatorhasthesamelogicasthepreviousonebut,sincespecial charactersaddquiteabitofcomplexitywhenitcomestocrackingapassword,wehave scoredfourpointsinsteadofjusttwo. Thelastonevalidatestheratiobetweenthelettersandthedigits. def_score_ratio(self): alpha_count=sum( 1ifc.lower()inascii_lowercaseelse0 forcinself.password) digits_count=sum( 1ifcindigitselse0forcinself.password) ifdigits_count==0: return0 returnmin(ceil(alpha_count/digits_count),7) Ihighlightedtheconditionallogicintheexpressionsinthesumcalls.Inthefirstcase,we geta1foreachcharacterwhoselowercaseversionisinascii_lowercase.Thismeans thatsummingallthose1’supgivesusexactlythecountofalltheletters.Then,wedothe sameforthedigits,onlyweusethedigitsstringforreference,andwedon’tneedto lowercasethecharacter.Whendigits_countis0,alpha_count/digits_countwould causeaZeroDivisionError,thereforewecheckondigits_countandwhenit’s0we return0.Ifwehavedigits,wecalculatetheceilingoftheletters:digitsratio,andreturnit, cappedat7. Ofcourse,therearemanydifferentwaystocalculateascoreforapassword.Myaimhere isnottogiveyouthefinestalgorithmtodothat,buttoshowyouhowyoucouldgoabout WOW! eBook www.wowebook.org implementingit. Codingthepasswordgenerator Thepasswordgeneratorisamuchsimplerclassthanthevalidator.However,Ihavecoded itsothatwewon’tneedtocreateaninstancetouseit,justtoshowyouyetagaina differentcodingstyle. classPasswordGenerator: @classmethod defgenerate(cls,length,bestof=10): candidates=sorted([ cls._generate_candidate(length) forkinrange(max(1,bestof)) ]) returncandidates[-1] @classmethod def_generate_candidate(cls,length): password=cls._generate_password(length) score=PasswordValidator(password).score() return(score['total'],password) @classmethod def_generate_password(cls,length): chars=allchars*(ceil(length/len(allchars))) return''.join(sample(chars,length)) Ofthethreemethods,onlythefirstoneismeanttobeused.Let’sstartouranalysiswith thelastone:_generate_password. Thismethodsimplytakesalength,whichisthedesiredlengthforthepasswordwewant, andcallsthesamplefunctiontogetapopulationoflengthelementsoutofthechars string.Thereturnvalueofthesamplefunctionisalistoflengthelements,andweneedto makeitastringusingjoin. Beforewecancallsample,thinkaboutthis,whatifthedesiredlengthexceedsthelength ofallchars?ThecallwouldresultinValueError:Samplelargerthanthe population. Becauseofthis,wecreatethecharsstringinawaythatitismadebyconcatenatingthe allcharsstringtoitselfjustenoughtimestocoverthedesiredlength.Togiveyouan example,let’ssayweneedapasswordof27characters,andlet’spretendallcharsis10 characterslong.length/len(allchars)gives2.7,which,whenpassedtotheceil function,becomes3.Thismeansthatwe’regoingtoassigncharstoatripleconcatenation oftheallcharsstring,hencecharswillbe10*3=30characterslong,whichisenough tocoverourrequirements. Notethat,inorderforthesemethodstobecalledwithoutcreatinganinstanceofthisclass, weneedtodecoratethemwiththeclassmethoddecorator.Theconventionisthentocall thefirstargument,cls,insteadofself,becausePython,behindthescenes,willpassthe classobjecttothecall. WOW! eBook www.wowebook.org Thecodefor_generate_candidateisalsoverysimple.Wejustgenerateapasswordand, giventhelength,wecalculateitsscore,andreturnatuple(score,password). Wedothissothatinthegeneratemethodwecangenerate10(bydefault)passwordseach timethemethodiscalledandreturntheonethathasthehighestscore.Sinceour generationlogicisbasedonarandomfunction,it’salwaysagoodwaytoemploya techniquelikethistoavoidworstcasescenarios. Thisconcludesthecodeforthehelpers. WOW! eBook www.wowebook.org Writingthehandlers Asyoumayhavenoticed,thecodeforthehelpersisn’trelatedtoFalconatall.Itisjust purePythonthatwecanreusewhenweneedit.Ontheotherhand,thecodeforthe handlersisofcoursebasedonFalcon.Thecodethatfollowsbelongstothe core/handlers.pymoduleso,aswedidbefore,let’sstartwiththefirstfewlines: importjson importfalcon from.passwordsimportPasswordValidator,PasswordGenerator classHeaderMixin: defset_access_control_allow_origin(self,resp): resp.set_header('Access-Control-Allow-Origin','*') Thatwasverysimple.Weimportjson,falcon,andourhelpers,andthenwesetupa mixinwhichwe’llneedinbothhandlers.TheneedforthismixinistoallowtheAPIto serverequeststhatcomefromsomewhereelse.ThisistheothersideoftheCORScointo whatwesawintheJavaScriptcodefortheinterface.Inthiscase,weboldlygowhereno securityexpertwouldeverdare,andallowrequeststocomefromanydomain('*').We dothisbecausethisisanexerciseand,inthiscontext,itisfine,butdon’tdoitin production,okay? Codingthepasswordvalidatorhandler ThishandlerwillhavetorespondtoaPOSTrequest,thereforeIhavecodedanon_post method,whichisthewayyoureacttoaPOSTrequestinFalcon. classPasswordValidatorHandler(HeaderMixin): defon_post(self,req,resp): self.process_request(req,resp) password=req.context.get('_body',{}).get('password') ifpasswordisNone: resp.status=falcon.HTTP_BAD_REQUEST returnNone result=self.parse_password(password) resp.body=json.dumps(result) defparse_password(self,password): validator=PasswordValidator(password) return{ 'password':password, 'valid':validator.is_valid(), 'score':validator.score(), } defprocess_request(self,req,resp): self.set_access_control_allow_origin(resp) body=req.stream.read() ifnotbody: WOW! eBook www.wowebook.org raisefalcon.HTTPBadRequest('Emptyrequestbody', 'AvalidJSONdocumentisrequired.') try: req.context['_body']=json.loads( body.decode('utf-8')) except(ValueError,UnicodeDecodeError): raisefalcon.HTTPError( falcon.HTTP_753,'MalformedJSON', 'JSONincorrectornotutf-8encoded.') Let’sstartwiththeon_postmethod.Firstofall,wecalltheprocess_requestmethod, whichdoesasanitycheckontherequestbody.Iwon’tgointofinestdetailbecauseit’s takenfromtheFalcondocumentation,andit’sastandardwayofprocessingarequest. Let’sjustsaythat,ifeverythinggoeswell(thehighlightedpart),wegetthebodyofthe request(alreadydecodedfromJSON)inreq.context['_body'].Ifthingsgobadlyfor anyreason,wereturnanappropriateerrorresponse. Let’sgobacktoon_post.Wefetchthepasswordfromtherequestcontext.Atthispoint, process_requesthassucceeded,butwestilldon’tknowifthebodywasinthecorrect format.We’reexpectingsomethinglike:{'password':'my_password'}. Soweproceedwithcaution.Wegetthevalueforthe'_body'keyand,ifthatisnot present,wereturnanemptydict.Wegetthevaluefor'password'fromthat.Weuseget insteadofdirectaccesstoavoidKeyErrorissues. IfthepasswordisNone,wesimplyreturna400error(badrequest).Otherwise,we validateitandcalculateitsscore,andthensettheresultasthebodyofourresponse. Youcanseehoweasyitistovalidateandcalculatethescoreofthepasswordinthe parse_passwordmethod,byusingourhelpers. Wereturnadictwiththreepiecesofinformation:password,valid,andscore.The passwordinformationistechnicallyredundantbecausewhoevermadetherequestwould knowthepasswordbut,inthiscase,Ithinkit’sagoodwayofprovidingenough informationforthingssuchaslogging,soIaddedit. WhathappensiftheJSON-decodedbodyisnotadict?Iwillleaveituptoyoutofixthe code,addingsomelogictocaterforthatedgecase. Codingthepasswordgeneratorhandler ThegeneratorhandlerhastohandleaGETrequestwithonequeryparameter:thedesired passwordlength. classPasswordGeneratorHandler(HeaderMixin): defon_get(self,req,resp): self.process_request(req,resp) length=req.context.get('_length',16) resp.body=json.dumps( PasswordGenerator.generate(length)) defprocess_request(self,req,resp): self.set_access_control_allow_origin(resp) WOW! eBook www.wowebook.org length=req.get_param('length') iflengthisNone: return try: length=int(length) assertlength>0 req.context['_length']=length except(ValueError,TypeError,AssertionError): raisefalcon.HTTPBadRequest('Wrongqueryparameter', '`length`mustbeapositiveinteger.') Wehaveasimilarprocess_requestmethod.Itdoesasanitycheckontherequest,even thoughabitdifferentlyfromtheprevioushandler.Thistime,weneedtomakesurethatif thelengthisprovidedonthequerystring(whichmeans,forexample,http://our-apiurl/?length=23),it’sinthecorrectformat.Thismeansthatlengthneedstobeapositive integer. So,tovalidatethat,wedoanintconversion(req.get_param('length')returnsa string),thenweassertthatlengthisgreaterthanzeroand,finally,weputitincontext underthe'_length'key. Doingtheintconversionofastringwhichisnotasuitablerepresentationforaninteger raisesValueError,whileaconversionfromatypethatisnotastringraisesTypeError, thereforewecatchthosetwointheexceptclause. WealsocatchAssertionError,whichisraisedbytheassertlength>0linewhen lengthisnotapositiveinteger.Wecanthensafelyguaranteethatthelengthisasdesired withonesingletry/exceptblock. Tip Notethat,whencodingatry/exceptblock,youshouldusuallytryandbeasspecificas possible,separatinginstructionsthatwouldraisedifferentexceptionsifaproblemarose. Thiswouldallowyoumorecontrolovertheissue,andeasierdebugging.Inthiscase though,sincethisisasimpleAPI,it’sfinetohavecodewhichonlyreactstoarequestfor whichlengthisnotintherightformat. Thecodefortheon_getmethodisquitestraightforward.Itstartsbyprocessingthe request,thenthelengthisfetched,fallingbackto16(thedefaultvalue)whenit’snot passed,andthenapasswordisgeneratedanddumpedtoJSON,andthensettobethe bodyoftheresponse. WOW! eBook www.wowebook.org RunningtheAPI Inordertorunthisapplication,youneedtorememberthatwesetthebaseURLinthe interfacetohttp://127.0.0.1:5555.Therefore,weneedthefollowingcommandtostart theAPI: $gunicorn-b127.0.0.1:5555main:app Runningthatwillstarttheappdefinedinthemainmodule,bindingtheserverinstanceto port5555onlocalhost.FormoreinformationaboutGunicorn,pleaserefertoeither Chapter10,WebDevelopmentDoneRightordirectlytotheproject’shomepage (http://gunicorn.org/). ThecodefortheAPIisnowcompletesoifyouhaveboththeinterfaceandtheAPI running,youcantrythemouttogether.Seeifeverythingworksasexpected. WOW! eBook www.wowebook.org TestingtheAPI Inthissection,let’stakealookatthetestsIwroteforthehelpersandforthehandlers. Testsforthehelpersareheavilybasedonthenose_parameterizedlibrary,asmyfavorite testingstyleisinterfacetesting,withaslittlepatchingaspossible.Using nose_parameterizedallowsmetowriteteststhatareeasiertoreadbecausethetestcases areveryvisible. Ontheotherhand,testsforthehandlershavetofollowthetestingconventionsforthe Falconlibrary,sotheywillbeabitdifferent.Thisis,ofcourse,idealsinceitallowsmeto showyouevenmore. DuetothelimitedamountofpagesIhaveleft,I’llshowyouonlyapartofthetests,so makesureyoucheckthemoutinfullinthesourcecode. Testingthehelpers Let’sseethetestsforthePasswordGeneratorclass: tests/test_core/test_passwords.py classPasswordGeneratorTestCase(TestCase): deftest__generate_password_length(self): forlengthinrange(300): assert_equal( length, len(PasswordGenerator._generate_password(length)) ) deftest__generate_password_validity(self): forlengthinrange(1,300): password=PasswordGenerator._generate_password( length) assert_true(PasswordValidator(password).is_valid()) deftest__generate_candidate(self): score,password=( PasswordGenerator._generate_candidate(42)) expected_score=PasswordValidator(password).score() assert_equal(expected_score['total'],score) @patch.object(PasswordGenerator,'_generate_candidate') deftest__generate(self,_generate_candidate_mock): #checks`generate`returnsthehighestscorecandidate _generate_candidate_mock.side_effect=[ (16,'&a69Ly+0H4jZ'), (17,'UXaF4stRfdlh'), (21,'aB4Ge_KdTgwR'),#thewinner (12,'IRLT*XEfcglm'), (16,'$P92-WZ5+DnG'), (18,'Xi#36jcKA_qQ'), (19,'?p9avQzRMIK0'), (17,'4@sY&bQ9*H!+'), (12,'Cx-QAYXG_Ejq'), WOW! eBook www.wowebook.org (18,'C)RAV(HP7j9n'), ] assert_equal( (21,'aB4Ge_KdTgwR'),PasswordGenerator.generate(12)) Withintest__generate_password_lengthwemakesurethe_generate_password methodhandlesthelengthparametercorrectly.Wegenerateapasswordforeachlengthin therange[0,300),andverifythatithasthecorrectlength. Inthetest__generate_password_validitytest,wedosomethingsimilarbut,thistime, wemakesurethatwhateverlengthweaskfor,thegeneratedpasswordisvalid.Weusethe PasswordValidatorclasstocheckforvalidity. Finally,weneedtotestthegeneratemethod.Thepasswordgenerationisrandom, therefore,inordertotestthisfunction,weneedtomock_generate_candidate,thus controllingitsoutput.Wesettheside_effectargumentonitsmocktobealistof10 candidates,fromwhichweexpectthegeneratemethodtochoosetheonewiththehighest score.Settingside_effectonamocktoalistcausesthatmocktoreturntheelementsof thatlist,oneatatime,eachtimeit’scalled.Toavoidambiguity,thehighestscoreis21, andonlyonecandidatehasscoredthathigh.Wecallthemethodandmakesurethatthat particularoneisthecandidatewhichisreturned. Note IfyouarewonderingwhyIusedthosedoubleunderscoresinthetestnames,it’svery simple:thefirstoneisaseparatorandthesecondoneistheleadingunderscorethatispart ofthenameofthemethodundertest. TestingthePasswordValidatorclassrequiresmanymorelinesofcode,soI’llshowonly aportionofthesetests: pwdapi/tests/test_core/test_passwords.py fromunittestimportTestCase fromunittest.mockimportpatch fromnose_parameterizedimportparameterized,param fromnose.toolsimport( assert_equal,assert_dict_equal,assert_true) fromcore.passwordsimportPasswordValidator,PasswordGenerator classPasswordValidatorTestCase(TestCase): @parameterized.expand([ (False,''), (False,''), (True,'abcdefghijklmnopqrstuvwxyz'), (True,'ABCDEFGHIJKLMNOPQRSTUVWXYZ'), (True,'0123456789'), (True,'!#$%&()*+-?@_|'), ]) deftest_is_valid(self,valid,password): validator=PasswordValidator(password) assert_equal(valid,validator.is_valid()) WOW! eBook www.wowebook.org Westartbytestingtheis_validmethod.WetestwhetherornotitreturnsFalsewhenit’s fedanemptystring,aswellasastringmadeupofonlyspaces,whichmakessurewe’re testingwhetherwe’recalling.strip()whenweassignthepassword. Then,weuseallthecharactersthatwewanttobeacceptedtomakesurethefunction acceptsthem. Iunderstandthesyntaxbehindtheparameterize.expanddecoratorcanbechallengingat firstbutreally,allthereistoitisthateachtupleconsistsofanindependenttestcase which,inturn,meansthatthetest_is_validtestisrunindividuallyforeachtuple,and thatthetwotupleelementsarepassedtothemethodasarguments:validandpassword. Wethentestforinvalidcharacters.Weexpectthemalltofailsoweuseparam.explicit, whichrunsthetestforeachofthecharactersinthatweirdstring. @parameterized.expand( param.explicit(char)forcharin'>]{<`\\;,[^/"\'~:}=.' ) deftest_is_valid_invalid_chars(self,password): validator=PasswordValidator(password) assert_equal(False,validator.is_valid()) TheyallevaluatetoFalse,sowe’regood. @parameterized.expand([ (0,''),#0-3:score0 (0,'a'),#0-3:score0 (0,'aa'),#0-3:score0 (0,'aaa'),#0-3:score0 (1,'aaab'),#4-7:score1 ... (5,'aaabbbbccccddd'),#12-15:score5 (5,'aaabbbbccccdddd'),#12-15:score5 ]) deftest__score_length(self,score,password): validator=PasswordValidator(password) assert_equal(score,validator._score_length()) Totestthe_score_lengthmethod,Icreated16testcasesforthelengthsfrom0to15. Thebodyofthetestsimplymakessurethatthescoreisassignedappropriately. deftest__score_length_sixteen_plus(self): #allpasswordwhoselengthis16+score7points password='x'*255 forlengthinrange(16,len(password)): validator=PasswordValidator(password[:length]) assert_equal(7,validator._score_length()) Theprecedingtestisforlengthsfrom16to254.Weonlyneedtomakesurethatany lengthafter15gets7asascore. Iwillskipoverthetestsfortheotherinternalmethodsandjumpdirectlytotheoneforthe scoremethod.Inordertotestit,Iwanttocontrolexactlywhatisreturnedbyeachofthe _score_*methodssoImockthemoutandinthetest,Isetareturnvalueforeachofthem. WOW! eBook www.wowebook.org Notethattomockmethodsofaclass,weuseavariantofpatch:patch.object.When yousetreturnvaluesonmocks,it’snevergoodtohaverepetitionsbecauseyoumaynotbe surewhichmethodreturnedwhat,andthetestwouldn’tfailinthecaseofaswap.So, alwaysreturndifferentvalues.Inmycase,Iamusingthefirstfewprimenumberstobe surethereisnopossibilityofconfusion. @patch.object(PasswordValidator,'_score_length') @patch.object(PasswordValidator,'_score_case') @patch.object(PasswordValidator,'_score_numbers') @patch.object(PasswordValidator,'_score_special') @patch.object(PasswordValidator,'_score_ratio') deftest_score( self, _score_ratio_mock, _score_special_mock, _score_numbers_mock, _score_case_mock, _score_length_mock): _score_ratio_mock.return_value=2 _score_special_mock.return_value=3 _score_numbers_mock.return_value=5 _score_case_mock.return_value=7 _score_length_mock.return_value=11 expected_result={ 'length':11, 'case':7, 'numbers':5, 'special':3, 'ratio':2, 'total':28, } validator=PasswordValidator('') assert_dict_equal(expected_result,validator.score()) Iwanttopointoutexplicitlythatthe_score_*methodsaremocked,soIsetupmy validatorinstancebypassinganemptystringtotheclassconstructor.Thismakesiteven moreevidenttothereaderthattheinternalsoftheclasshavebeenmockedout.Then,I justcheckiftheresultisthesameaswhatIwasexpecting. ThislasttestistheonlyoneinthisclassinwhichIusedmocks.Alltheothertestsforthe _score_*methodsareinaninterfacestyle,whichreadsbetterandusuallyproducesbetter results. Testingthehandlers Let’sbrieflyseeoneexampleofatestforahandler: pwdapi/tests/test_core/test_handlers.py importjson fromunittest.mockimportpatch fromnose.toolsimportassert_dict_equal,assert_equal WOW! eBook www.wowebook.org importfalcon importfalcon.testingastesting fromcore.handlersimport( PasswordValidatorHandler,PasswordGeneratorHandler) classPGHTest(PasswordGeneratorHandler): defprocess_request(self,req,resp): self.req,self.resp=req,resp returnsuper(PGHTest,self).process_request(req,resp) classPVHTest(PasswordValidatorHandler): defprocess_request(self,req,resp): self.req,self.resp=req,resp returnsuper(PVHTest,self).process_request(req,resp) BecauseofthetoolsFalcongivesyoutotestyourhandlers,Icreatedachildforeachof theclassesIwantedtotest.TheonlythingIchanged(byoverridingamethod)isthatin theprocess_requestmethod,whichiscalledbybothclasses,beforeprocessingthe requestImakesureIsetthereqandrespargumentsontheinstance.Thenormalbehavior oftheprocess_requestmethodisthusnotalteredinanyotherway.Bydoingthis, whateverhappensoverthecourseofthetest,I’llbeabletocheckagainstthoseobjects. It’squitecommontousetrickslikethiswhentesting.Weneverchangethecodetoadapt foratest,itwouldbebadpractice.Wefindawayofadaptingourteststosuitourneeds. classTestPasswordValidatorHandler(testing.TestBase): defbefore(self): self.resource=PVHTest() self.api.add_route('/password/validate/',self.resource) ThebeforemethodiscalledbytheFalconTestBaselogic,anditallowsustosetupthe resourcewewanttotest(thehandler)andarouteforit(whichisnotnecessarilythesame astheoneweuseinproduction). deftest_post(self): self.simulate_request( '/password/validate/', body=json.dumps({'password':'abcABC0123#&'}), method='POST') resp=self.resource.resp assert_equal('200OK',resp.status) assert_dict_equal( {'password':'abcABC0123#&', 'score':{'case':3,'length':5,'numbers':2, 'special':4,'ratio':2,'total':16}, 'valid':True}, json.loads(resp.body)) Thisisthetestforthehappypath.AllitdoesissimulateaPOSTrequestwithaJSON payloadasbody.Then,weinspecttheresponseobject.Inparticular,weinspectitsstatus anditsbody.Wemakesurethatthehandlerhascorrectlycalledthevalidatorandreturned itsresults. WOW! eBook www.wowebook.org Wealsotestthegeneratorhandler: classTestPasswordGeneratorHandler(testing.TestBase): defbefore(self): self.resource=PGHTest() self.api.add_route('/password/generate/',self.resource) @patch('core.handlers.PasswordGenerator') deftest_get(self,PasswordGenerator): PasswordGenerator.generate.return_value=(7,'abc123') self.simulate_request( '/password/generate/', query_string='length=7', method='GET') resp=self.resource.resp assert_equal('200OK',resp.status) assert_equal([7,'abc123'],json.loads(resp.body)) Forthisoneaswell,Iwillonlyshowyouthetestforthehappypath.Wemockoutthe PasswordGeneratorclassbecauseweneedtocontrolwhichpassworditwillgenerateand, unlesswemock,wewon’tbeabletodoit,asitisarandomprocess. Oncewehavecorrectlysetupitsreturnvalue,wecansimulatetherequestagain.Inthis case,it’saGETrequest,withadesiredlengthof7.Weuseatechniquesimilartotheone weusedfortheotherhandler,andchecktheresponsestatusandbody. ThesearenottheonlytestsyoucouldwriteagainsttheAPI,andthestylecouldbe differentaswell.Somepeoplemockoften,ItendtomockonlywhenIreallyhaveto.Just trytoseeifyoucanmakesomesenseoutofthem.Iknowthey’renotreallyeasybut they’llbegoodtrainingforyou.Testsareextremelyimportantsogiveityourbestshot. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Wheredoyougofromhere? Ifyoulikedthisprojectandyoufeellikeexpandingit,hereareafewsuggestions: ImplementtheencryptioninthemechanismofacustomDjangofield. Amendthetemplatefortherecordlistsothatyoucansearchforaparticularrecord. AmendtheJavaScripttouseJSONPwithacallbacktoovercometheCORSissue. AmendtheJavaScripttofirethevalidationcallwhenthepasswordfieldchanges. WriteaDjangocommandthatallowsyoutoencryptanddecryptthedatabasefile. Whenyoudoitfromthecommandline,incorporatethatbehaviorintothewebsite, possiblyonthehomepage,sothatyoudon’thaveaccesstotherecordsunlessyou areauthenticated.Thisisdefinitelyahardchallengeasitrequireseitheranother databasewithanauthenticationpasswordstoredproperlywithaonewayhash,or someseriousreworkingofthedatastructureusedtoholdtherecordmodeldata. Evenifyoudon’thavethemeanstodoitnow,tryandthinkabouthowyouwould solvethisproblem. SetupPostgreSQLonyourmachineandswitchtousingitinsteadoftheSQLitefile thatisthedefault. Addtheabilitytoattachafiletoarecord. Playwiththeapplication,trytofindoutwhichfeaturesyouwanttoaddorchange, andthendoit. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Summary Inthischapter,we’veworkedonafinalprojectthatinvolvesaninterfaceandanAPI.We haveusedtwodifferentframeworkstoaccomplishourgoal:DjangoandFalcon.Theyare verydifferentandhaveallowedustoexploredifferentconceptsandtechniquestocraft oursoftwareandmakethisfunapplicationcomealive. Wehaveseenanexampleofsymmetricencryptionandexploredcodethatwaswrittenina morefunctionalstyle,asopposedtoamoreclassiccontrolflow-orientedapproach.We havereusedandextendedtheDjangoclass-basedviews,reducingtoaminimumthe amountofcodewehadtowrite. WhencodingtheAPI,wedecoupledhandlingrequestsfrompasswordmanagement.This wayit’smucheasiertoseewhichpartofthecodedependsontheFalconframeworkand whichisindependentfromit. Finally,wesawafewtestsforthehelpersandhandlersoftheAPI.Wehavebriefly touchedonatechniquethatIusetoexpandclassesundertestinordertobeabletotest againstthosepartsofthecodewhichwouldnotnormallybeavailable. Myaiminthischapterwastoprovideyouwithaninterestingexamplethatcouldbe expandedandimprovedindifferentways.Ialsowantedtogiveyouafewexamplesof differentcodingstylesandtechniques,whichiswhyIchosetospreadthingsapartanduse differentframeworks. WOW! eBook www.wowebook.org WOW! eBook www.wowebook.org Awordoffarewell Ihopethatyouarestillthirstyandthatthisbookwillbejustthefirstofmanystepsyou taketowardsPython.It’satrulywonderfullanguage,wellworthlearningdeeply. Ihopethatyouenjoyedthisjourneywithme,Ididmybesttomakeitinterestingforyou. Itsurewasforme,Ihadsuchagreattimewritingthesepages. Pythonisopensource,sopleasekeepsharingitandconsidersupportingthewonderful communityaroundit. Tillnexttime,myfriend,farewell! WOW! eBook www.wowebook.org Index A adhocpolymorphism about/Polymorphism–abriefoverview AJAX about/Thefutureofwebdevelopment,Thetemplatelayer Anaconda URL/Wheredowegofromhere? anonymousfunctions about/Anonymousfunctions API testing/TestingtheAPI helpers,testing/Testingthehelpers handlers,testing/Testingthehandlers application testing/Testingyourapplication applicationprogramminginterface(API) about/Writingaunittest assertion about/Assertions WOW! eBook www.wowebook.org B Base64 about/Theimports baseclass about/Inheritanceandcomposition binarysearch about/Wheretoinspect Bitbucket URL/Exceptions black-boxtests about/Testingyourapplication Bokeh URL/Wheredowegofromhere? boundaries about/Aspecializedelse:elif boundary about/Boundariesandgranularity built-inexceptionshierarchy URL/Exceptions built-infunctions about/Built-infunctions built-inscope about/Scopes businesslogic,GUIapplication webpage,fetching/Thebusinesslogic,Fetchingthewebpage images,saving/Savingtheimages user,alerting/Alertingtheuser WOW! eBook www.wowebook.org C callback about/Thelayoutlogic campaign about/Preparingthedata classes about/Objectandclasses,Object-orientedprogramming classmethods about/Classmethods code writing,guidelines/Guidelinesonhowtowritegoodcode documenting/Documentingyourcode collectionsmodule about/Thecollectionsmodule namedtuples/Namedtuples defaultdict/Defaultdict ChainMap/ChainMap comma-separatedvalues(CSV) about/SavingtheDataFrametoafile commandprompt about/SettingupthePythoninterpreter composition about/Inheritanceandcomposition comprehensions about/map,zip,andfilter,Comprehensions nestedcomprehensions/Nestedcomprehensions filtering/Filteringacomprehension dictcomprehensions/dictcomprehensions setcomprehensions/setcomprehensions andgenerators/Don’toverdocomprehensionsandgenerators conditionalprogramming about/Conditionalprogramming elif/Aspecializedelse:elif ternaryoperator/Theternaryoperator considerations threading/Threadingconsiderations console about/SettingupthePythoninterpreter,Yourfriend,theconsole constructor about/Initializinganinstance containerdatatypes defining/Thecollectionsmodule contextmanager WOW! eBook www.wowebook.org using/Thebusinesslogic contextobject preparing/Thetemplatelayer cookies about/HowdoestheWebwork? CPC(CostPerClick) about/Unpackingtheuserdata CPI(CostPerImpression) about/Unpackingtheuserdata Cron about/RunningPythonscripts Cross-originresourcesharing(CORS) about/TalkingtotheAPI cross-siterequestforgery(CSRF)attack about/Creatingtheform CSS(CascadingStyleSheets) about/Aregexwebsite CTR(ClickThroughRate) about/Unpackingtheuserdata customexceptions writing/Exceptions customiterator writing/Writingacustomiterator cypertext about/Themodellayer WOW! eBook www.wowebook.org D data dealingwith/Dealingwithdata notebook,settingup/Settingupthenotebook preparing/Preparingthedata cleaning/Cleaningthedata DataFrame,creating/CreatingtheDataFrame DataFrame,savingtofile/SavingtheDataFrametoafile results,visualizing/Visualizingtheresults DataFrame campaignname,unpacking/Unpackingthecampaignname userdata,unpacking/Unpackingtheuserdata defining/Cleaningeverythingup datamigrations about/AddingtheEntrymodel datastructures selecting/Howtochoosedatastructures debuggingtechniques about/Debuggingtechniques debugging,withprint/Debuggingwithprint debugging,withcustomfunction/Debuggingwithacustomfunction traceback,inspecting/Inspectingthetraceback Pythondebugger,using/UsingthePythondebugger logfiles,inspecting/Inspectinglogfiles profiling/Profiling assertions/Assertions information,finding/Wheretofindinformation decorate-sort-undecorate about/map URL/map decoration about/Decorators decorationpoint about/Decorators decorator about/Decorators decoratorfactory about/Adecoratorfactory decorators about/Decorators defaultvalues about/Keywordargumentsanddefaultvalues deterministprofiling WOW! eBook www.wowebook.org about/ProfilingPython diamond about/Multipleinheritance dictionaries about/Mappingtypes–dictionaries discounts applying/Example2–applyingdiscounts dispatcher about/Example2–applyingdiscounts Django settingup/SettingupDjango URL/SettingupDjango project,starting/Startingtheproject users,creating/Creatingusers about/Writingthetemplates Djangointerface implementing/Ourimplementation,ImplementingtheDjangointerface setup/Thesetup modellayer/Themodellayer simpleform/Asimpleform viewlayer/Theviewlayer URLs,settingup/SettinguptheURLs templatelayer/Thetemplatelayer DjangoURLdispatcher defining/TheDjangoURLdispatcher regularexpression/Regularexpressions Djangowebframework defining/TheDjangowebframework modellayer/Themodellayer viewlayer/Theviewlayer templatelayer/Thetemplatelayer docstrings about/Documentingyourcode DRY(Don’tRepeatYourself)principle/Howdoweusemodulesandpackages dunder about/Goingbeyondnext WOW! eBook www.wowebook.org E enclosingscope about/Scopes entries about/Startingtheproject environment settingup/Settinguptheenvironment equalities about/Aspecializedelse:elif Euclid’salgorithm about/Don’toverdocomprehensionsandgenerators Euclideanformula about/Don’toverdocomprehensionsandgenerators exception about/Aspecialelseclause exceptions about/Exceptions executionmodel,Python about/Python’sexecutionmodel WOW! eBook www.wowebook.org F factorial about/Howdoweusemodulesandpackages Falcon JSONquoteserver,building/BuildingaJSONquoteserverinFalcon about/BuildingaJSONquoteserverinFalcon URL/BuildingaJSONquoteserverinFalcon FalconAPI implementing/ImplementingtheFalconAPI mainapplication/Themainapplication helpers,writing/Writingthehelpers handlers,writing/Writingthehandlers API,running/RunningtheAPI API,testing/TestingtheAPI features,Python portability/Portability coherence/Coherence developerproductivity/Developerproductivity extensivelibrary/Anextensivelibrary softwarequality/Softwarequality softwareintegration/Softwareintegration satisfactionandenjoyment/Satisfactionandenjoyment Fibonaccisequenceexample about/Onelastexample filter about/map,zip,andfilter defining/filter Flask URL/WritingaFlaskview about/WritingaFlaskview Flaskview writing/WritingaFlaskview floatingpointnumbers URL/Reals form about/Creatingtheform framework about/TheDjangowebframework function about/Howdoweusemodulesandpackages functionattributes about/Functionattributes functions WOW! eBook www.wowebook.org using/Whyusefunctions? codeduplication,reducing/Reducecodeduplication complextask,splitting/Splittingacomplextask implementationdetails,hiding/Hideimplementationdetails readability,improving/Improvereadability traceability,improving/Improvetraceability writing,tips/Afewusefultips example/Onefinalexample WOW! eBook www.wowebook.org G generationbehavior,inbuilt-ins about/Generationbehaviorinbuilt-ins generatorexpressions about/Generatorexpressions generatorfunctions about/Generatorfunctions generatorobjects about/Goingbeyondnext generators about/map,zip,andfilter,Generators generatorfunctions/Generators generatorexpressions/Generators andcomprehensions/Don’toverdocomprehensionsandgenerators getter about/Thepropertydecorator gettersandsetters about/Thepropertydecorator Git about/Guidelinesonhowtowritegoodcode GitHub URL/Exceptions globalscope about/Scopes gray-boxtesting about/Testingyourapplication greatestcommondivisor(GCD) about/Don’toverdocomprehensionsandgenerators Greenphase about/Test-drivendevelopment GUI(GraphicalUserInterface)/RunningPythonasaGUIapplication GUIapplication Python,runningas/RunningPythonasaGUIapplication about/Secondapproach–aGUIapplication imports/Secondapproach–aGUIapplication,Theimports layoutlogic/Thelayoutlogic businesslogic/Thebusinesslogic improving/Howtoimprovetheapplication? Gunicorn URL/RunningtheAPI Gunicorn(GreenUnicorn) about/BuildingaJSONquoteserverinFalcon WOW! eBook www.wowebook.org H handlers writing/Writingthehandlers passwordvalidatorhandler,coding/Codingthepasswordvalidatorhandler passwordgeneratorhandler,coding/Codingthepasswordgeneratorhandler Hashability about/Settypes helpers writing/Writingthehelpers passwordvalidator,coding/Codingthepasswordvalidator passwordgenerator,coding/Codingthepasswordgenerator HTMLDocumentObjectModel(DOM) about/Creatingtheform HypertextMarkupLanguage(HTML) about/Theviewlayer HypertextTransferProtocol(HTTP) about/WhatistheWeb? WOW! eBook www.wowebook.org I IDLE(IntegratedDeveLopmentEnvironment)/RunningthePythoninteractiveshell immutable about/Aproperintroduction,Mutableorimmutable?Thatisthequestion immutablesequences about/Immutablesequences stringsandbytes/Stringsandbytes tuples/Tuples implicitconcatenation about/BuildingaJSONquoteserverinFalcon in-place about/Unpackingtheuserdata indexing about/Aboutindexingandslicing inequalities about/Aspecializedelse:elif infiniteloop about/Thewhileloop inheritance about/Inheritanceandcomposition initializer about/Objectandclasses,Initializinganinstance inner about/Aspecializedelse:elif inputparameters about/Inputparameters argumentpassing/Argumentpassing assignment,toargumentnames/Assignmenttoargumentnamesdon’taffectthe caller mutable,changing/Changingamutableaffectsthecaller specifying/Howtospecifyinputparameters positionalarguments/Positionalarguments keywordarguments/Keywordargumentsanddefaultvalues defaultvalues/Keywordargumentsanddefaultvalues variablepositionalarguments/Variablepositionalarguments variablekeywordarguments/Variablekeywordarguments keyword-onlyarguments/Keyword-onlyarguments ,combining/Combininginputparameters mutabledefaults/Avoidthetrap!Mutabledefaults inside-outtechnique about/Iteratingoverasequence installing Python/InstallingPython WOW! eBook www.wowebook.org instanceattributes about/Classandobjectnamespaces instancesofclasses about/Objectandclasses integerdivision(//) about/Integers IntegratedDevelopmentEnvironments(IDEs)/AnoteontheIDEs about/Thepropertydecorator interfacetesting about/Interfacetesting Internet about/WhatistheWeb? ipdblibrary about/UsingthePythondebugger IPython URL/Wheredowegofromhere? Ipython about/IPythonandJupyternotebook URL/IPythonandJupyternotebook iterable about/Writingacustomiterator iterator about/Iteratorsanditerables,Writingacustomiterator itertoolsmodule about/Aquickpeekattheitertoolsmodule infiniteiterators/Infiniteiterators terminating,onshortestinputsequence/Iteratorsterminatingontheshortest inputsequence combinatoricgenerators/Combinatoricgenerators WOW! eBook www.wowebook.org J jQuery about/Thetemplatelayer URL/Thetemplatelayer JSON(JavaScriptObjectNotation) about/Exceptions JSONquoteserver building,inFalcon/BuildingaJSONquoteserverinFalcon Jupyternotebook about/IPythonandJupyternotebook URL/IPythonandJupyternotebook WOW! eBook www.wowebook.org K KeepassX about/Thechallenge keyword-onlyparameter about/Keyword-onlyarguments keywordarguments about/Keywordargumentsanddefaultvalues WOW! eBook www.wowebook.org L lambdas about/Anonymousfunctions library about/Howdoweusemodulesandpackages listcomprehension about/Lists local,enclosing,global,built-in(LEGB)/Scopes localscope about/Scopes logfiles about/Inspectinglogfiles loggers/Inspectinglogfiles handlers/Inspectinglogfiles filters/Inspectinglogfiles formatters/Inspectinglogfiles logging URL/Inspectinglogfiles loop iterating,overrange/Iteratingoverarange iterating,oversequence/Iteratingoverasequence looping about/Looping forloop/Theforloop iteratorsanditerables/Iteratorsanditerables iterating,overmultiplesequences/Iteratingovermultiplesequences whileloop/Thewhileloop breakandcontinuestatements/Thebreakandcontinuestatements elseclause/Aspecialelseclause WOW! eBook www.wowebook.org M magicmethod about/Objectandclasses magicmethods about/Goingbeyondnext map about/map,zip,andfilter defining/map markdown about/IPythonandJupyternotebook masterpassword about/Thechallenge Matplotlib URL/Wheredowegofromhere? Mercurial about/Guidelinesonhowtowritegoodcode mergeandinsertionsort about/Lists metaclasses about/Objectandclasses,ThesimplestPythonclass metaprogramming about/ThesimplestPythonclass method about/Goingbeyondnext methodresolutionorder(MRO) about/Methodresolutionorder methods about/Aproperintroduction middlewareclass about/Writingthetemplates migration about/AddingtheEntrymodel migrations applying/Startingtheproject mixins about/Multipleinheritance mocks about/Mockobjectsandpatching model about/Themodellayer model-template-view(MTV)pattern about/Djangodesignphilosophy model-view-controller(MVC) WOW! eBook www.wowebook.org about/Djangodesignphilosophy modules using/Howdoweusemodulesandpackages mutable about/Aproperintroduction,Mutableorimmutable?Thatisthequestion mutablesequences about/Mutablesequences lists/Lists bytearrays/Bytearrays WOW! eBook www.wowebook.org N NameErrorexception/Scopes namelocalization about/Namelocalization namemangling about/Privatemethodsandnamemangling names about/Namesandnamespaces defining/Aboutthenames namespaces about/Namesandnamespaces nano about/Usingconsoleeditors negativeindexing about/Aboutindexingandslicing nose-parameterized URL/Amoreinterestingexample Numba URL/Wheredowegofromhere? numbers about/Numbers integers/Integers booleans/Booleans realnumbers/Reals complexnumbers/Complexnumbers fractionsanddecimals/Fractionsanddecimals NumericPython about/CreatingtheDataFrame NumPy URL/Wheredowegofromhere? WOW! eBook www.wowebook.org O object defining/Everythingisanobject mutable/Mutableorimmutable?Thatisthequestion immutable/Mutableorimmutable?Thatisthequestion object-orientedprogramming about/Object-orientedprogramming Pythonclass/ThesimplestPythonclass classandobjectnamespaces/Classandobjectnamespaces attributeshadowing/Attributeshadowing selfvariable,using/I,me,andmyself–usingtheselfvariable instance,initializing/Initializinganinstance code,reusing/OOPisaboutcodereuse inheritanceandcomposition/Inheritanceandcomposition baseclass,accessing/Accessingabaseclass multipleinheritance/Multipleinheritance methodresolutionorder/Methodresolutionorder classandstaticmethods/Staticandclassmethods privatemethods/Privatemethodsandnamemangling namemangling/Privatemethodsandnamemangling propertydecorator/Thepropertydecorator operatoroverloading/Operatoroverloading polymorphism/Polymorphism–abriefoverview object-relationalmapping(ORM) about/Themodellayer objects about/Aproperintroduction,Objectandclasses,Object-orientedprogramming importing/Importingobjects relativeimports/Relativeimports onewayencryptionalgorithm about/Themodellayer onewayhashfunction about/Themodellayer operatingsystem(OS) about/Portability operatoroverloading about/Lists,Operatoroverloading outer about/Aspecializedelse:elif WOW! eBook www.wowebook.org P package about/HowisPythoncodeorganized packages using/Howdoweusemodulesandpackages Pandas URL/Wheredowegofromhere? patching about/Mockobjectsandpatching URL/Aclassicunittestexample pdb about/UsingthePythondebugger PEP328 about/Relativeimports performanceconsiderations about/Someperformanceconsiderations pivottable about/Visualizingtheresults plaintext about/Themodellayer polymorphism/Polymorphism–abriefoverview primarykey about/Themodellayer,AddingtheEntrymodel primegenerator about/Example1–aprimegenerator primitive about/Don’toverdocomprehensionsandgenerators principleofleastastonishment about/Theprincipleofleastastonishment principles,Djangowebframework DRY/Djangodesignphilosophy Loosecoupling/Djangodesignphilosophy Lesscode/Djangodesignphilosophy Consistency/Djangodesignphilosophy profiling about/Exceptions,Whentoprofile? properties about/Aproperintroduction protocols about/WhatistheWeb? pullprotocol about/HowdoestheWebwork? pushprotocol WOW! eBook www.wowebook.org about/HowdoestheWebwork? PyGTK about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK PyPy URL/Whatarethedrawbacks? PyQt about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK pytest URL/Amoreinterestingexample Pythagoreantriple about/Filteringacomprehension Python about/EnterthePython features/AboutPython,Softwareintegration drawbacks/Whatarethedrawbacks? users/WhoisusingPythontoday? installing/InstallingPython references/SettingupthePythoninterpreter running,asservice/RunningPythonasaservice running,asGUIapplication/RunningPythonasaGUIapplication executionmodel/Python’sexecutionmodel profiling/ProfilingPython Python2 versusPython3/Python2versusPython3–thegreatdebate Pythoncode organizing/HowisPythoncodeorganized Pythonculture about/ThePythonculture PythonEnhancementProposal(PEP)/Guidelinesonhowtowritegoodcode Pythoninteractiveshell running/RunningthePythoninteractiveshell Pythoninterpreter settingup/SettingupthePythoninterpreter Pythonmodule reference/Everythingisanobject PythonPackageIndex(PyPI)/Anextensivelibrary Pythonprogram running/HowyoucanrunaPythonprogram Pythonscripts running/RunningPythonscripts WOW! eBook www.wowebook.org Q qualityassurance(QA) about/Testingyourapplication WOW! eBook www.wowebook.org R radix-64 about/Theimports recursivefunctions about/Recursivefunctions Redphase about/Test-drivendevelopment Refactor about/Test-drivendevelopment regexwebsite defining/Aregexwebsite Django,settingup/SettingupDjango Entrymodel,adding/AddingtheEntrymodel adminpanel,customizing/Customizingtheadminpanel form,creating/Creatingtheform views,writing/Writingtheviews URLsandviews,using/TyingupURLsandviews templates,writing/Writingthetemplates regularexpression/Regularexpressions relationaldatabase about/Themodellayer relativeimports URL/Relativeimports request-responseclient-serverprotocol about/HowdoestheWebwork? returnvalues about/Returnvalues multiplevalues,returning/Returningmultiplevalues WOW! eBook www.wowebook.org S scheduler about/Threadingconsiderations schemamigration about/AddingtheEntrymodel Schwartziantransform about/map Scikit-Learn URL/Wheredowegofromhere? SciPy URL/Wheredowegofromhere? scopes about/Scopes local/Scopes enclosing/Scopes global/Scopes built-in/Scopes scopesandnameresolution defining/Scopesandnameresolution globalandnonlocalstatements/Theglobalandnonlocalstatements scripting about/Firstapproach–scripting imports/Firstapproach–scripting,Theimports arguments,parsing/Parsingarguments businesslogic/Thebusinesslogic service Python,runningas/RunningPythonasaservice service-orientedarchitecture(SOA) about/Inspectinglogfiles service-orientedarchitectures about/Inspectinglogfiles setter about/Thepropertydecorator settypes about/Settypes Single-PageApplication(SPA) about/Thefutureofwebdevelopment SingleResponsibilityPrinciple(SRP) about/Writingthehelpers slicing about/Aboutindexingandslicing smallvaluescaching about/Smallvaluescaching WOW! eBook www.wowebook.org space about/Someperformanceconsiderations Sphinx about/Documentingyourcode SQL(StructuredQueryLanguage) about/Themodellayer staticmethods about/Staticmethods statisticalprofiling about/ProfilingPython strings encoding/Encodinganddecodingstrings decoding/Encodinganddecodingstrings indexing/Indexingandslicingstrings slicing/Indexingandslicingstrings strobjects about/Stringsandbytes symmetricencryptionalgorithm about/Themodellayer system-exitingexceptions about/Exceptions WOW! eBook www.wowebook.org T Tcl(ToolCommandLanguage)/RunningPythonasaGUIapplication TCP/IP(TransmissionControlProtocol/InternetProtocol) about/HowdoestheWebwork? templatelayer about/Thetemplatelayer homeandfootertemplates/Homeandfootertemplates records,listing/Listingallrecords records,creating/Creatingandeditingrecords records,editing/Creatingandeditingrecords API,defining/TalkingtotheAPI records,deleting/Deletingrecords terminal about/SettingupthePythoninterpreter ternaryoperator about/Theternaryoperator test defining/Theanatomyofatest preparation/Theanatomyofatest execution/Theanatomyofatest verification/Theanatomyofatest failing/Makingatestfail test-drivendevelopment(TDD) about/Test-drivendevelopment benefits/Test-drivendevelopment disadvantages/Test-drivendevelopment testingguidelines defining/Testingguidelines tests front-endtests/Testingyourapplication scenariotests/Testingyourapplication integrationtests/Testingyourapplication smoketests/Testingyourapplication Acceptancetests/Testingyourapplication functionaltests/Testingyourapplication destructivetests/Testingyourapplication performancetests/Testingyourapplication usabilitytests/Testingyourapplication securityandpenetrationtests/Testingyourapplication unittests/Testingyourapplication regressiontests/Testingyourapplication comparing,withmocks/Comparingtestswithandwithoutmocks comparing,withoutmocks/Comparingtestswithandwithoutmocks WOW! eBook www.wowebook.org boundaries/Boundariesandgranularity granularity/Boundariesandgranularity example/Amoreinterestingexample thread about/Threadingconsiderations time about/Someperformanceconsiderations Timsort about/Lists Tk/RunningPythonasaGUIapplication tkinter about/Secondapproach–aGUIapplication Tkinter/RunningPythonasaGUIapplication tkinter.tix(TkInterfaceExtension)module about/Thetkinter.tixmodule tkinter.tixmodule about/Thetkinter.tixmodule Tkinterface about/Secondapproach–aGUIapplication triangulation about/Comparingtestswithandwithoutmocks troubleshootingguidelines about/Troubleshootingguidelines consoleeditors,using/Usingconsoleeditors inspecting/Wheretoinspect testsused,fordebugging/Usingteststodebug monitoring/Monitoring truedivision(/) about/Integers tuple about/Tuples turtlemodule about/Theturtlemodule WOW! eBook www.wowebook.org U unicodecodepoints about/Stringsandbytes UniformResourceLocator(URL) about/TheDjangoURLdispatcher unittest about/Unittesting writing/Writingaunittest mockobjects/Mockobjectsandpatching patching/Mockobjectsandpatching assertions/Assertions example/Aclassicunittestexample unittesting defining/Unittesting unpacking about/Variablepositionalarguments Upcasting about/Booleans useracceptancetesting(UAT) about/Testingyourapplication userexperience(UX) about/Testingyourapplication Utf-8 about/Encodinganddecodingstrings WOW! eBook www.wowebook.org V viewlayer about/Theviewlayer importsandhomeview/Importsandhomeview records,listing/Listingallrecords records,creating/Creatingrecords records,updating/Updatingrecords records,deleting/Deletingrecords views writing/Writingtheviews homeview/Thehomeview entrylistview/Theentrylistview formview/Theformview vim about/Usingconsoleeditors virtualenv about/Aboutvirtualenv referencelink/Aboutvirtualenv virtualenvironment creating/Yourfirstvirtualenvironment WOW! eBook www.wowebook.org W wastedtime about/Aboutthenames Web(WorldWideWeb) defining/WhatistheWeb? working/HowdoestheWebwork? webdevelopment defining/Thefutureofwebdevelopment webframework about/TheDjangowebframework white-boxtests about/Testingyourapplication wxPython about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK WOW! eBook www.wowebook.org Y yieldfromexpression about/Theyieldfromexpression WOW! eBook www.wowebook.org Z zip about/map,zip,andfilter defining/zip WOW! eBook www.wowebook.org